Data acquisition method, system, equipment and medium

Abstract

The invention discloses a data acquisition method, system and device and a medium. The data acquisition method comprises the following steps: acquiring a source data layer, a character set conversion program and a data acquisition program; collecting target data from the source data layer by using a data collection program; synchronizing the target data to a character set conversion program and performing character code conversion processing; and transferring the character set conversion program subjected to the code conversion processing to a transfer application server directory through an FTP (File Transfer Protocol) tool. According to the method, the character set is converted into the standard character set when the database data is collected, so that the problem of Chinese messy codes caused by different character sets on two sides when the data in Oracle databases of different companies are collected is thoroughly solved, and the delay of research and development of other programs is effectively avoided.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data acquisition method, system, device and medium. Background Art

[0002] Currently, in the existing Oracle databases, due to historical and production environment reasons, different data manufacturers have different requirements for the database, resulting in inconsistent character sets of the Oracle databases of different data manufacturers.

[0003] When using FlinkCDC to collect data from Oracle databases of different companies, different character sets on both sides lead to garbled Chinese characters, which delays the development of other programs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the defect in the prior art that when collecting data from Oracle databases of different companies, different character sets on both sides lead to garbled Chinese characters, delaying the development of other programs, and provide a data collection method, system, device and medium.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] In a first aspect, the present invention provides a data collection method, the data collection method comprising:

[0007] Obtain source data layer, character set conversion program and data collection program;

[0008] Using the data acquisition program to acquire target data from the source data layer;

[0009] Synchronizing the target data to the character set conversion program and performing character encoding conversion processing;

[0010] The character set conversion program after the encoding conversion process is transferred to the application server directory through the FTP tool.

[0011] Preferably, the step of synchronizing the target data to the character set conversion program and performing character encoding conversion processing comprises:

[0012] The target data is transcoded for the first time to obtain first encoded data in UTF-16 encoding format;

[0013] The first encoded data is transcoded for a second time to obtain second encoded data in GBK encoding format or UTF-8 encoding format.

[0014] Preferably, the step of collecting target data from the source data layer using the data collection program includes:

[0015] The data acquisition program is used to collect data in US7ASCII encoding format from a low-version database in real time as the target data.

[0016] Preferably, the source data layer includes an Oracle database, the character set conversion program includes a Java character set conversion program, and the data acquisition program includes a FlinkCDC data acquisition program.

[0017] In a second aspect, the present invention provides a data acquisition system, the data acquisition system comprising:

[0018] The acquisition module is used to acquire the source data layer, the character set conversion program and the data acquisition program;

[0019] A collection module, used for collecting target data from the source data layer using the data collection program;

[0020] A conversion module, used for synchronizing the target data to the character set conversion program and performing character encoding conversion processing;

[0021] The transfer module is used to transfer the character set conversion program after the encoding conversion process to the application server directory through the FTP tool.

[0022] Preferably, the conversion module comprises:

[0023] A first conversion unit, configured to perform a first transcoding on the target data to obtain first encoded data in UTF-16 encoding format;

[0024] The second conversion unit is used to perform a second transcoding on the first encoded data to obtain second encoded data in GBK encoding format or UTF-8 encoding format.

[0025] Preferably, the acquisition module is specifically used for:

[0026] The data acquisition program is used to collect data in US7ASCII encoding format from a low-version database in real time as the target data.

[0027] Preferably, the source data layer includes an Oracle database, the character set conversion program includes a Java character set conversion program, and the data acquisition program includes a FlinkCDC data acquisition program.

[0028] In a third aspect, the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the data acquisition method as described above when executed by the processor.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data acquisition method as described above is implemented.

[0030] The positive and progressive effects of the present invention are: obtaining a source data layer, a character set conversion program and a data acquisition program; using the data acquisition program to collect target data from the source data layer; synchronizing the target data to the character set conversion program and performing character encoding conversion processing; transferring the character set conversion program after encoding conversion processing to the application server directory through an FTP tool. The present invention completely solves the problem of Chinese garbled characters caused by different character sets on both sides when collecting data from Oracle databases of different companies, and effectively avoids delays in the development of other programs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the data acquisition method of embodiment 1 of the present invention.

[0032] Figure 2 This is a module schematic diagram of a data acquisition system according to Embodiment 2 of the present invention.

[0033] Figure 3 This is a schematic diagram of the structure of an electronic device for implementing a data collection method according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0035] Example 1

[0036] The data collection method of this embodiment is as follows: Figure 1 As shown, the data collection method includes:

[0037] S11, obtaining source data layer, character set conversion program and data acquisition program;

[0038] S12, using a data acquisition program to collect target data from a source data layer;

[0039] S13, synchronizing the target data to the character set conversion program and performing character encoding conversion processing;

[0040] S14, the character set conversion program after the encoding conversion process is transferred to the application server directory through the FTP tool.

[0041] The source data layer includes an Oracle database, the character set conversion program includes a Java character set conversion program, and the data collection program includes a FlinkCDC data collection program.

[0042] In this embodiment, the data source library is an Oracle database, a Java character set conversion program, and a FlinkCDC data collection program; the character set conversion program written in Java is connected to the database through the JDBC data source connection technology. The FlinkCDC data collection program collects data from the Oracle database into the data character conversion program; the written Java conversion program is encoded into a program package through Maven, and the program package is uploaded to the application server directory through the FTP tool. This method converts the Oracle character set into a standard character set when collecting Oracle database data, which effectively solves the problem of garbled Chinese characters due to different character sets on both sides in the process of collecting target data from Oracle databases of different companies.

[0043] In an optional implementation, step S12 specifically includes:

[0044] The data acquisition program is used to collect data in US7ASCII encoding format from the low-version database in real time as the target data.

[0045] Wherein, step S13 comprises:

[0046] S131, performing a first transcoding on the target data to obtain first encoded data in UTF-16 encoding format.

[0047] S132. Perform a second transcoding on the first encoded data to obtain second encoded data in GBK encoding format or UTF-8 encoding format.

[0048] In this embodiment, by passing the character set encoding (such as GBK, UTF-8 and other formats) as a parameter to the character set conversion program written in Java language, this method can more flexibly convert, test and apply the character set encoding, shortening the development time of other programs. First, the data collected by FlinkCDC is transcoded for the first time, and the data in US7ASCII encoding format is converted into data in UTF-16 encoding format; then the data converted into UTF-16 encoding format is transcoded for the second time, and the data in UTF-16 encoding format is converted into GBK or UTF-8 encoding format. This application performs a second transcoding to collect data in US7ASCII encoding format from an older version of Oracle database. The first transcoding cannot solve the data in Chinese garbled format, and the second transcoding must be solved to convert it into the encoding format data consistent with the target library format. It should be emphasized that the fields containing Chinese data are not filtered, and all fields are transcoded, which reduces the screening of fields containing Chinese data and enhances operability.

[0049] This embodiment provides a data collection method. When collecting database data, the character set is converted into a standard character set according to set rules, which effectively solves the problem of garbled Chinese characters caused by different character sets on both sides during the process of collecting target data from Oracle databases of different companies.

[0050] Example 2

[0051] The data acquisition system of this embodiment is as follows: Figure 2 As shown, the data acquisition system includes: an acquisition module 310 , a collection module 320 , a conversion module 330 and a transmission module 340 .

[0052] Wherein, the acquisition module 310 is used to acquire the source data layer, the character set conversion program and the data acquisition program;

[0053] The acquisition module 320 is used to acquire target data from the source data layer using a data acquisition program;

[0054] The conversion module 330 is used to synchronize the target data to the character set conversion program and perform character encoding conversion processing;

[0055] The transfer module 340 is used to transfer the character set conversion program after the encoding conversion process to the application server directory through the FTP tool.

[0056] The source data layer includes an Oracle database, the character set conversion program includes a Java character set conversion program, and the data collection program includes a FlinkCDC data collection program.

[0057] In this embodiment, the data source library is an Oracle database, a Java character set conversion program, and a FlinkCDC data collection program; the character set conversion program written in Java is connected to the database through the JDBC data source connection technology. The FlinkCDC data collection program collects data from the Oracle database into the data character conversion program; the written Java conversion program is encoded into a program package through Maven, and the program package is uploaded to the application server directory through the FTP tool. This method converts the Oracle character set into a standard character set when collecting Oracle database data, which effectively solves the problem of garbled Chinese characters due to different character sets on both sides in the process of collecting target data from Oracle databases of different companies.

[0058] In an optional implementation, the acquisition module 320 is specifically used for:

[0059] The data acquisition program is used to collect data in US7ASCII encoding format from the low-version database in real time as the target data.

[0060] Wherein, the conversion module 330, such as Figure 2 As shown, including:

[0061] The first conversion unit 331 is used to perform a first transcoding on the target data to obtain first encoded data in UTF-16 encoding format.

[0062] The second conversion unit 332 is used to perform a second transcoding on the first encoded data to obtain second encoded data in GBK encoding format or UTF-8 encoding format.

[0063] In this embodiment, by passing the character set encoding (such as GBK, UTF-8 and other formats) as a parameter to the character set conversion program written in Java language, this method can more flexibly convert, test and apply the character set encoding, shortening the development time of other programs. First, the data collected by FlinkCDC is transcoded for the first time, and the data in US7ASCII encoding format is converted into data in UTF-16 encoding format; then the data converted into UTF-16 encoding format is transcoded for the second time, and the data in UTF-16 encoding format is converted into GBK or UTF-8 encoding format. This application performs a second transcoding to collect data in US7ASCII encoding format from an older version of Oracle database. The first transcoding cannot solve the data in Chinese garbled format, and the second transcoding must be solved to convert it into the encoding format data consistent with the target library format. It should be emphasized that the fields containing Chinese data are not filtered, and all fields are transcoded, which reduces the screening of fields containing Chinese data and enhances operability.

[0064] This embodiment provides a data collection system. When collecting database data, the conversion module converts the character set into a standard character set according to the set rules, which effectively solves the problem of garbled Chinese characters caused by the different character sets on both sides during the process of collecting target data from Oracle databases of different companies.

[0065] Example 3

[0066] Figure 3 The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the data collection method of the first embodiment is implemented. Figure 3 The electronic device 90 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0067] like Figure 3As shown, the electronic device 90 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including the memory 92 and the processor 91).

[0068] The bus 93 includes a data bus, an address bus, and a control bus.

[0069] The memory 92 may include a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read-only memory (ROM) 923 .

[0070] The memory 92 may also include a program / utility 925 having a set (at least one) of program modules 924, such program modules 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0071] The processor 91 executes various functional applications and data processing by running the computer programs stored in the memory 92, such as the data acquisition method of embodiment 1 of the present invention.

[0072] The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the model generating device 90 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 96. Figure 3 As shown, the network adapter 96 communicates with other modules of the model-generated device 90 via the bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0073] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0074] Example 4

[0075] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the data acquisition method of Embodiment 1 are implemented.

[0076] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0077] In a possible implementation manner, the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps in the data collection method of embodiment 1.

[0078] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0079] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A data collection method, characterized in that: The data collection method comprises: Obtain source data layer, character set conversion program and data collection program; Using the data acquisition program to acquire target data from the source data layer; Synchronizing the target data to the character set conversion program and performing character encoding conversion processing; The character set conversion program after the encoding conversion process is transferred to the application server directory through the FTP tool.

2. The data collection method according to claim 1, characterized in that: The step of synchronizing the target data to the character set conversion program and performing character encoding conversion processing includes: The target data is transcoded for the first time to obtain first encoded data in UTF-16 encoding format; The first encoded data is transcoded for a second time to obtain second encoded data in GBK encoding format or UTF-8 encoding format.

3. The data collection method according to claim 1, characterized in that: The step of collecting target data from the source data layer using the data collection program includes: The data acquisition program is used to collect data in US7ASCII encoding format from a low-version database in real time as the target data.

4. The data collection method according to claim 1, characterized in that: The source data layer includes an Oracle database, the character set conversion program includes a Java character set conversion program, and the data acquisition program includes a FlinkCDC data acquisition program.

5. A data acquisition system, characterized in that: The data acquisition system comprises: The acquisition module is used to acquire the source data layer, the character set conversion program and the data acquisition program; A collection module, used for collecting target data from the source data layer using the data collection program; A conversion module, used for synchronizing the target data to the character set conversion program and performing character encoding conversion processing; The transfer module is used to transfer the character set conversion program after the encoding conversion process to the application server directory through the FTP tool.

6. The data acquisition system according to claim 5, characterized in that: The conversion module comprises: A first conversion unit, configured to perform a first transcoding on the target data to obtain first encoded data in UTF-16 encoding format; The second conversion unit is used to perform a second transcoding on the first encoded data to obtain second encoded data in GBK encoding format or UTF-8 encoding format.

7. The data acquisition system according to claim 5, characterized in that: The acquisition module is specifically used for: The data acquisition program is used to collect data in US7ASCII encoding format from a low-version database in real time as the target data.

8. The data acquisition system according to claim 5, characterized in that: The source data layer includes an Oracle database, the character set conversion program includes a Java character set conversion program, and the data acquisition program includes a FlinkCDC data acquisition program.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data acquisition method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data collection method according to any one of claims 1 to 4 is implemented.

Images