Data processing method, device, electronic device and storage medium

Abstract

The present invention discloses a data processing method, device, electronic device and storage medium, wherein the method comprises: connecting to a target database according to preset configuration parameters, and obtaining metadata of a data source or a data destination in the target database; determining a matching relationship between the data source and the data destination according to the metadata; and generating a data extraction execution code according to the matching relationship and the database type of the target database. The embodiment of the present invention realizes the automatic execution of data extraction, can reduce the cost of business code generation, reduces the repetitive workload of personnel, and can improve the efficiency of data extraction.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer application technology, and in particular, to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the development of computer technology, we are gradually entering an era of coexistence with data. Data is usually stored in data tables, and the use of data requires processing and analysis. The use of data involves the timeliness of data. In response to the timeliness of data, streaming computing is often used to achieve real-time processing of data. At present, when using streaming computing to extract data, it is necessary to write corresponding business codes according to the definition of the data table to achieve the mapping of data source and data purpose, so as to achieve data extraction. However, the writing of business codes requires manpower investment, and the cost of business code writing will increase as the scale of business projects increases, resulting in a large waste of manpower costs and reducing the efficiency of data extraction. Summary of the invention

[0003] The present invention provides a data processing method, device, electronic device and storage medium to achieve efficient data extraction, reduce the redundancy of business codes, and reduce the time cost and labor cost of data processing.

[0004] In a first aspect, an embodiment of the present invention further provides a data processing method, wherein the method comprises:

[0005] Connecting to a target database according to preset configuration parameters, and obtaining metadata of a data source or a data destination in the target database;

[0006] Determining a matching relationship between the data source and the data destination according to the meta information;

[0007] A data extraction execution code is generated according to the matching relationship and the database type of the target database.

[0008] In a second aspect, an embodiment of the present invention further provides a data processing device, wherein the device includes:

[0009] An information acquisition module, used to connect to a target database according to preset configuration parameters and obtain meta information of a data source or a data destination in the target database;

[0010] A relationship matching module, used to determine a matching relationship between the data source and the data destination according to the meta information;

[0011] An execution code module is used to generate a data extraction execution code according to the matching relationship and the database type of the target database.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, wherein the electronic device includes:

[0013] one or more processors;

[0014] a memory for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the embodiments of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data processing method as described in any one of the embodiments of the present invention.

[0017] The embodiment of the present invention connects to the target database according to preset configuration parameters, obtains metadata of the data source or data destination in the target database, determines the matching relationship between the data source and the data destination according to the source information, and generates data extraction execution code according to the matching relationship and the database type of the target database. This can realize automatic generation of data extraction, reduce the generation cost of business execution code, reduce the repetitive workload of personnel, and improve data extraction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a data processing method provided by Embodiment 1 of the present invention;

[0019] Figure 2 is a flow chart of another data processing method provided by Embodiment 2 of the present invention;

[0020] Figure 3 is an example diagram of a matching relationship provided by Embodiment 2 of the present invention;

[0021] Figure 4 is an example diagram of a data processing method provided by Embodiment 3 of the present invention;

[0022] Figure 5 is a structural diagram of a data processing device provided by Embodiment 4 of the present invention;

[0023] Figure 6 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention are shown in the accompanying drawings, rather than all structures. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0025] Embodiment 1

[0026] Figure 1 is a flow chart of a data processing method provided in the first embodiment of the present invention. This embodiment is applicable to the case of automatic extraction of streaming data. The method can be executed by a data processing device, which can be implemented in hardware and / or software. Figure 1 The method provided in the embodiment of the present invention specifically includes the following steps:

[0027] Step 110: Connect to the target database according to preset configuration parameters, and obtain metadata of the data source or data destination in the target database.

[0028] The preset configuration parameters may be parameter information pre-configured in the stream computing for connecting to the database, which may include the database address and database login verification information, etc. The preset configuration parameters may be specifically database drivers, and it is understandable that different preset configuration parameters may exist for different types of databases. The target database may be a database connected when data is extracted, which may include a database to which the data source belongs and a database to which the data destination belongs. Meta information may refer to the definition information of a database or a data table, such as the definition of information such as field names and field types.

[0029] In an embodiment of the present invention, preset configuration parameters can be read and used to connect to a target database. The target database can be input by a user through a visual interface or configured by parameters in a configuration file. After the connection to the target database is established, the metadata of the data source or the data destination in the target database can be read through the connection, wherein the data source can be the data source of data extraction, and the data destination can be the calculation target of data extraction.

[0030] Step 120: Determine the matching relationship between the data source and the data destination according to the meta information.

[0031] Among them, the matching relationship can be the corresponding relationship between the data source and the data destination, that is, the data source needs to be stored in the data destination relationship. The matching relationship can include one-to-one, one-to-many, many-to-one and many-to-many.

[0032] Specifically, after obtaining the metadata of the data source or the data destination, a matching relationship between the corresponding data source and the data destination can be established by determining whether the metadata has the same parameters or the same characteristics. For example, when the same parameters exist in the metadata of the data source A and the data destination B, a matching relationship between the data source and the data destination can be established. It can be understood that the matching relationship can refer to the matching relationship between the entire data table of the data source and the data destination, or it can also be a matching relationship between a certain data table field in the data source or the data destination.

[0033] Step 130: Generate data extraction execution code according to the matching relationship and the database type of the target database.

[0034] The database type may be information reflecting the type of database, and the database type may include MySQL database, SQL Server database, Oracle database, DB2 database, SyBase database, Informix database, PostgreSQL database, etc. The execution code may be a data extraction program code executable in the streaming technology.

[0035] In the implementation of the present invention, the grammatical logic or program template of the generated execution code can be determined according to the database type, and the corresponding data source and data destination processing rules can be generated as the execution code according to the matching relationship and the grammatical logic or program template.

[0036] The embodiment of the present invention connects to the target database according to preset configuration parameters, obtains metadata of the data source or data destination in the target database, determines the matching relationship between the data source and the data destination according to the source information, and generates data extraction execution code according to the matching relationship and the database type of the target database. This can realize automatic generation of data extraction, reduce the generation cost of business execution code, reduce the repetitive workload of personnel, and improve data extraction efficiency.

[0037] Embodiment 2

[0038] Figure 2 is a flow chart of another data processing method provided by Embodiment 2 of the present invention. This embodiment of the present invention is a specific implementation of the above-mentioned embodiment of the present invention. Figure 2 The method provided in the embodiment of the present invention specifically includes the following steps:

[0039] Step 210: Read preset configuration parameters, wherein the preset configuration parameters at least include: user name, host, port and database.

[0040] In the embodiment of the present invention, preset configuration parameters including user name, host, port and database may be read, the preset configuration parameters may correspond to one or more databases, and the preset configuration parameters may be pre-stored in a configuration file.

[0041] Step 220: Call the database connection function according to the preset configuration parameters to establish a connection with the target database.

[0042] Specifically, a connection function can be called and a connection with a target database can be established through preset configuration parameters. The connection can be used to read data in the target database. It can be understood that separate threads can be used between different target databases, and different data connection functions can respectively maintain a connection to a target database, which can improve the data extraction efficiency under streaming computing.

[0043] Further on the basis of the above-mentioned implementation of the invention, the database connection function includes: GORM library function.

[0044] Specifically, the GORM library function is a Golang ORM library that can connect to a relational database.

[0045] Step 230: Read the metadata corresponding to the data source or data destination in the target database based on the connection, wherein the metadata includes the field name and field type.

[0046] Specifically, the data source or data destination field name and field type in the target database can be read as meta information. For example, when the target database is a MySQL database, the information_schema.columns table in the target database can be queried and the TABLE_NAME field can be extracted as the table name, COLUMN_NAME:id as the field name and the DATA_TYPE:int field as the field type.

[0047] Step 240: Determine whether the field names and field types in the data source and data destination meta-information are the same.

[0048] In an embodiment of the present invention, the metadata of the data source and the data destination may be compared to determine whether the field name of the data source is the same as the field name of the data destination and whether the field type of the data source is the same as the field type of the data source. Figure 3 , there are fields ID and field ID1 in the data source table TS, and there are fields ID, ID1 and ID2 in the data destination table TD. The field name of TS's ID is the same as that of TD, and the field type is the same, both of which are int type.

[0049] Step 250: If yes, a matching relationship is established between the data source and the data destination; if no, the data source and the data destination are not processed.

[0050] Specifically, if the field name and field type of the data source and the data destination are the same, a matching relationship is established between the data source and the data destination. Otherwise, if there is no matching relationship between the data source and the data destination, there is no need to establish a matching relationship, and a new data source and data destination can be reselected for matching.

[0051] Step 260: extract the database type of the target database, and obtain the execution code template corresponding to the database type.

[0052] Among them, the execution code module can be a code template with streaming computing processing logic. The execution code template can be composed of syntax that conforms to the corresponding database type, and the execution code template may lack specific parameter values ​​of the data source and data destination.

[0053] In the implementation of the present invention, the database type of the target database can be extracted, and the execution code template matching the database type can be searched according to the database type. It can be understood that the match can refer to the existence of the same identification information as the database type or the execution code module has the operation syntax of the corresponding database type.

[0054] Step 270: Extract connection parameter values ​​of the data source and the data target respectively.

[0055] The connection parameter value may be a parameter value specifically used to connect the data source and the data target, and may include specific values ​​of information such as user name, host, port, and database.

[0056] In the embodiment of the present invention, a user-input or pre-configured connection parameter value may be extracted, and the connection parameter value may be used to execute the function execution of the execution code template.

[0057] Step 280: Associate each connection parameter value with the execution code template according to the matching relationship to generate a data extraction execution code.

[0058] Specifically, the connection parameter values ​​may be filled into the execution code template according to the matching relationship between the data source and the data destination, so that the execution code template can implement data extraction through the connection parameter values ​​of the data source and the data destination.

[0059] In an embodiment of the present invention, by reading preset configuration parameters, a database connection function is called based on the preset configuration parameters to establish a connection with a target database, metadata of a data source or a data destination is read through the connection, and if it is determined that the field name and field type of the data source and the data destination are the same, a matching relationship is established between the data source and the data destination, otherwise no processing is performed, an execution code template corresponding to the database type of the template database is extracted, and the connection parameter values ​​of the data source and the data destination are filled into the execution code template according to the matching relationship to generate an execution code, thereby realizing automatic execution of data extraction, reducing the cost of business code generation, reducing the repetitive workload of personnel, and improving data extraction efficiency.

[0060] Further, based on the above-mentioned embodiment of the invention, establishing the matching relationship between the data source and the data destination includes: adding an associated identification number to the data source and the data destination.

[0061] Specifically, the association identification number may be identification information that identifies that the data source and the data destination have a matching relationship, and the management identification number may be composed of letters, numbers, and special symbols. When establishing a matching relationship between a data source and a data destination, an association identification number may be added for the data source and the data destination, for example, an association identification number may be added for a field in the data source and a field in the data destination, or an association identification number may be added for a data table in the data source or a data table in the data destination.

[0062] Further, based on the above-mentioned embodiment of the invention, the metadata corresponding to the data source or data destination in the target database is read based on the connection, including:

[0063] The data source or data destination structure information table is queried in the target database; the field name stored in the name field and the field type stored in the type field are extracted from the structure information table as meta-information.

[0064] In the implementation of the present invention, the structural information table corresponding to the data source or data destination can be searched in each target database, and the field name and field type can be extracted from the structural information table as metadata. It can be understood that the number of target databases can be multiple, and the data source or data destination can be a field or a data table in the target database respectively.

[0065] Embodiment 3

[0066] Figure 4 is an example diagram of a data processing method provided by Embodiment 3 of the present invention, see Figure 4, by obtaining the data source connection and the data destination connection, establishing a database connection, and querying the metadata such as the field name and field type of the table through the connection with the data source and data destination database respectively, and realizing information field matching based on the metadata, the field name can be used to establish a one-to-one matching relationship between the data source and the data destination, and the data corresponding to the matching relationship can be automatically generated into an execution code that can be recognized by streaming computing.

[0067] Embodiment 4

[0068] Figure 5 301 is a schematic diagram of a data processing device provided in the fourth embodiment of the present invention, which can execute the data processing method provided in any embodiment of the present invention and has the corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software and / or hardware, and specifically includes: an information acquisition module 301, a relationship matching module 302 and an execution code module 303.

[0069] The information acquisition module 301 is used to connect to the target database according to preset configuration parameters and obtain the meta information of the data source or data destination in the target database.

[0070] The relationship matching module 302 is used to determine the matching relationship between the data source and the data destination according to the meta information.

[0071] The execution code module 303 is used to generate a data extraction execution code according to the matching relationship and the database type of the target database.

[0072] In the embodiment of the present invention, an information acquisition module is connected to a target database according to preset configuration parameters, metadata of a data source or a data destination is acquired in the target database, a relationship matching module determines a matching relationship between a data source and a data destination according to the source information, and an execution code module generates a data extraction execution code according to the matching relationship and a database type of the target database. This can realize automatic generation of data extraction, reduce the generation cost of business execution codes, reduce repetitive workload of personnel, and improve data extraction efficiency.

[0073] Further, based on the above-mentioned embodiment of the invention, the information acquisition module 301 includes:

[0074] The parameter reading unit is used to read the preset configuration parameters, wherein the preset configuration parameters at least include: user name, host, port and database.

[0075] The connection establishing unit is used to call the database connection function according to the preset configuration parameters to establish a connection with the target database.

[0076] An information reading unit is used to read the meta information corresponding to the data source or the data destination in the target database based on the connection, wherein the meta information includes a field name and a field type.

[0077] Further, based on the above-mentioned embodiment of the invention, the database connection function in the connection establishment unit includes: GORM library function.

[0078] Further, based on the above-mentioned embodiment of the invention, the relationship matching module 302 includes:

[0079] The matching judgment unit is used to judge whether the field names and field types in the metadata of the data source and the data destination are the same.

[0080] The relationship matching unit is used to establish the matching relationship between the data source and the data destination if yes; if no, not process the data source and the data destination.

[0081] Further, based on the above-mentioned embodiment of the invention, the relationship matching unit is specifically used to: add an associated identification number to the data source and the data destination.

[0082] Further, based on the above-mentioned embodiment of the invention, the execution code module 303 includes:

[0083] The type extraction unit is used to extract the database type of the target database and obtain the execution code template corresponding to the database type.

[0084] The parameter extraction unit is used to extract the connection parameter values ​​of the data source and the data target respectively.

[0085] A code filling unit is used to associate and fill each of the connection parameter values ​​into the execution code template according to the matching relationship to generate the data extraction execution code.

[0086] Further, based on the above-mentioned embodiments of the invention, the information reading unit is specifically used to: query the data source or the data destination structure information table in the target database; extract the field name stored in the name field and the field type stored in the type field in the structure information table as the meta-information.

[0087] Embodiment 5

[0088] Figure 6 is a structural diagram of an electronic device provided by Embodiment 5 of the present invention, Figure 6 A block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention is shown. Figure 6The computer device 312 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The device 312 is a typical computing device for implementing the data processing method.

[0089] like Figure 6 As shown, the computer device 312 is in the form of a general-purpose computing device. The components of the computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 connecting different system components (including the storage device 328 and the processor 316).

[0090] Bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0091] The computer device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 312, including volatile and non-volatile media, removable and non-removable media.

[0092] The storage device 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. The computer device 312 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 334 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 318 via one or more data medium interfaces. The storage device 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0093] A program 336 having a set (at least one) of program modules 326 may be stored, for example, in a storage device 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 326 generally perform the functions and / or methods of the embodiments described herein.

[0094] The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 312, and / or communicate with any device that enables the computer device 312 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 322. In addition, the computer device 312 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 320. As shown, the network adapter 320 communicates with other modules of the computer device 312 through a bus 318. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computer device 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0095] The processor 316 executes various functional applications and data processing by running the programs stored in the storage device 328, such as implementing the data processing method provided in the above-mentioned embodiment of the present invention.

[0096] Embodiment 6

[0097] The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processing device, a data processing method as in the embodiment of the present invention is implemented. The above-mentioned computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0098] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0099] The computer-readable medium may be included in the computer device, or may exist independently without being incorporated into the computer device.

[0100] The computer-readable medium carries one or more programs. When the one or more programs are executed by the computer device, the computer device: connects to the target database according to preset configuration parameters, and obtains the metadata of the data source or data purpose in the target database; determines the matching relationship between the data source and the data purpose according to the metadata; generates data extraction execution code according to the matching relationship and the database type of the target database. The computer program code for performing the operation of the present disclosure can be written in one or more programming languages ​​or a combination thereof, and the programming languages ​​include but are not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0101] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0103] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0104] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Connecting to a target database according to preset configuration parameters and obtaining meta information of a data source or a data destination in the target database includes: reading the preset configuration parameters; calling a database connection function according to the preset configuration parameters to establish a connection with the target database; reading meta information corresponding to the data source or the data destination in the target database based on the connection, including: querying a structure information table of the data source or the data destination in the target database, and extracting a field name stored in a name field and a field type stored in a type field in the structure information table as the meta information; Determining the matching relationship between the data source and the data destination according to the meta information includes: judging whether the field names and field types in the meta information of the data source and the data destination are the same, if so, establishing the matching relationship between the data source and the data destination, if not, not processing the data source and the data destination, wherein the matching relationship includes at least one-to-one, one-to-many, many-to-one and many-to-many; The establishing the matching relationship between the data source and the data destination includes: adding an associated identification number to the data source and the data destination; Generate a data extraction execution code according to the matching relationship and the database type of the target database, including: extracting the database type of the target database, and obtaining an execution code template corresponding to the database type, respectively extracting connection parameter values ​​of the data source and the data destination, and associating and filling each of the connection parameter values ​​into the execution code template according to the matching relationship to generate the data extraction execution code.

2. The method according to claim 1, characterized in that: The preset configuration parameters include at least: user name, host, port and database.

3. The method according to claim 1, characterized in that: The database connection function includes: GORM library function.

4. A data processing device, characterized in that: The device comprises: An information acquisition module, used to connect to a target database according to preset configuration parameters and acquire meta information of a data source or a data destination in the target database; A relationship matching module, used to determine a matching relationship between the data source and the data destination according to the meta information, wherein the matching relationship includes at least one-to-one, one-to-many, many-to-one and many-to-many; An execution code module, used for generating a data extraction execution code according to the matching relationship and the database type of the target database; The information acquisition module includes: a parameter reading unit, used to read the preset configuration parameters; a connection establishment unit, used to call a database connection function according to the preset configuration parameters to establish a connection with the target database; an information reading unit, used to read the metadata corresponding to the data source or the data destination in the target database based on the connection; The information reading unit is specifically used to: query the data source or the data destination structure information table in the target database, and extract the field name stored in the name field and the field type stored in the type field in the structure information table as the meta information; The relationship matching module includes: a matching judgment unit, used to judge whether the field names and field types in the meta information of the data source and the data destination are the same; a relationship matching unit, used to establish the matching relationship between the data source and the data destination if yes, and not process the data source and the data destination if no; The relationship matching unit is specifically used to: add an associated identification number to the data source and the data destination; The execution code module includes: a type extraction unit, used to extract the database type of the target database and obtain the execution code template corresponding to the database type; a parameter extraction unit, used to extract the connection parameter values ​​of the data source and the data destination respectively; a code filling unit, used to associate each of the connection parameter values ​​with the execution code template according to the matching relationship to generate the data extraction execution code.

5. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 3 is implemented.

Images