Data processing method, device and system

Abstract

The present invention discloses a data processing method, device and system, which relates to the field of big data technology, wherein the method comprises: in response to the source data table structure update, the source data stored in the database is acquired according to the predetermined table structure field information; according to the predetermined single file size, at least one binary file is generated from the acquired source data; a spliced ​​file is generated according to the at least one binary file, the file name of the binary file, and the table structure field information; before executing the data management operation, the spliced ​​file is transmitted to a predetermined position in the data management system according to the predetermined data mapping information. Through the present invention, the error rate of the managed data can be reduced and the success rate of file transmission can be improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device and system. Background Art

[0002] At present, big data management requires the prior registration of information such as table structure to generate the lake entry table in advance, and the management process is achieved by reading data and sending pre-registered file contents from the other party's server. This results in the inability to synchronize the table structure corresponding to the managed tables of each application to the big data warehouse in real time when it changes, resulting in management errors. At the same time, due to the size limit of the transferred files, it is also impossible to cope with the situation where the files are too large due to the exponential increase in data volume, resulting in the inability to transfer files. Summary of the invention

[0003] In view of this, the present invention provides a data processing method, device and system to solve at least one of the above-mentioned problems.

[0004] According to a first aspect of the present invention, there is provided a data processing method, the method comprising:

[0005] In response to the source data table structure being updated, the source data stored in the database is acquired according to the predetermined table structure field information;

[0006] Generate at least one binary file from the acquired source data according to a predetermined single file size;

[0007] Generate a spliced ​​file according to the at least one binary file, the file name of the binary file, and the table structure field information;

[0008] Before executing the data management operation, the spliced ​​file is transmitted to a predetermined location in the data management system according to predetermined data mapping information.

[0009] According to a second aspect of the present invention, there is provided a data processing device, the device comprising:

[0010] A source data acquisition unit, configured to acquire source data stored in a database according to predetermined table structure field information in response to source data table structure update;

[0011] A binary file generating unit, used to generate at least one binary file from the acquired source data according to a predetermined single file size;

[0012] A splicing file generating unit, used to generate a splicing file according to the at least one binary file, the file name of the binary file, and the table structure field information;

[0013] The splicing file transmission unit is used to transmit the splicing file to a predetermined location in the data management system according to predetermined data mapping information before executing the data management operation.

[0014] According to a third aspect of the present invention, there is provided a data processing system, the system comprising: the above-mentioned data processing device, a database and a data management system.

[0015] According to a fourth aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0016] According to a fifth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0017] It can be seen from the above technical solution that when the source data table structure is updated, the source data is obtained according to the predetermined table structure field information, and at least one binary file is generated from the obtained source data according to the predetermined single file size, and then a spliced ​​file is generated according to the binary file, its file name and the table structure field information, and then before executing the data management operation, the spliced ​​file is transmitted to a predetermined location in the data management system, thereby overcoming the problem of management errors caused by the inability to synchronize to the data management system in real time when the source table structure changes in the prior art. By generating multiple binary files, the problem of being unable to transmit files due to excessive file size can also be overcome. Through this technical solution, the error rate of managed data can be reduced and the success rate of file transmission can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention;

[0020] Figure 2 is a structural block diagram of a data processing system according to an embodiment of the present invention;

[0021] Figure 3 is a structural block diagram of a data processing device according to an embodiment of the present invention;

[0022] Figure 4is an exemplary structural block diagram of a data processing system according to an embodiment of the present invention;

[0023] Figure 5 is a working principle diagram of an exemplary data processing system according to an embodiment of the present invention;

[0024] Figure 6 It is a schematic block diagram of the system structure of the electronic device 600 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In view of the current data management solution, when the source table structure changes, it cannot be synchronized to the data management system in real time, resulting in management error problems. At the same time, due to the size limit of the transmitted file, it is also impossible to cope with the situation where the file is too large and cannot be transmitted. Based on this, an embodiment of the present invention provides a data processing solution, which can solve the above defects, reduce the error rate of managed data, and reduce the problem of being unable to transmit files due to large files. The following is a detailed description of the embodiment of the present invention in conjunction with the accompanying drawings.

[0027] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the process includes:

[0028] Step 101, in response to the source data table structure being updated, the source data stored in the database is acquired according to the predetermined table structure field information.

[0029] The table structure field information here can be: table name, file name, etc.

[0030] Step 102: Generate at least one binary file (BIN file) from the acquired source data according to a predetermined single file size.

[0031] Specifically, according to a predetermined single file size, the acquired source data may be sequentially generated into a plurality of binary files according to predetermined sequence numbers.

[0032] In one embodiment, a corresponding verification file may be generated according to the file size of the at least one binary file to verify whether the binary file is complete.

[0033] Step 103: Generate a concatenated file according to the at least one binary file, the file name of the binary file (for example, including: source table name and file sequence number, etc.), and table structure field information.

[0034] Step 104: before executing the data management operation, the spliced ​​file is transmitted to a predetermined location in the data management system according to predetermined data mapping information.

[0035] The mapping information here may include: table name, file name, and the association between the file names in the management system.

[0036] When there is no data corresponding to the table name and file name of the spliced ​​file in the data management system, it indicates that the spliced ​​file is stored in the data management system for the first time, and the spliced ​​file can be directly stored in the data management system. When there is historical data corresponding to the table name and file name of the spliced ​​file in the data management system, it indicates that the spliced ​​file is updated data, and the spliced ​​file is stored in the location corresponding to the historical data, and the historical data is updated, including but not limited to: deletion, replacement, addition, etc.

[0037] Afterwards, the above data mapping information can be updated according to the splicing file.

[0038] By acquiring source data according to predetermined table structure field information when the source data table structure is updated, generating at least one binary file from the acquired source data according to a predetermined single file size, and then generating a spliced ​​file according to the binary file, its file name and table structure field information, and then before executing the data management operation, transmitting the spliced ​​file to a predetermined location in the data management system, the problem of management errors caused by the inability to synchronize to the data management system in real time when the source table structure changes in the prior art can be overcome. By generating multiple binary files, the problem of being unable to transmit files due to excessive file size can also be overcome. Through the embodiments of the present invention, the management data error rate can be reduced and the file transmission success rate can be improved.

[0039] In one embodiment, after at least one binary file is generated in step 102, the at least one binary file may be directly transferred to the data management system via a file transfer protocol (FTP).

[0040] In FTP transmission, regardless of whether the file transmission is successful or failed, the transmitted file is preferably backed up.

[0041] In actual operation, data index information may be generated in advance, and the data index information includes: table structure field information; thereafter, when a query operation is required, a query operation may be performed on the database according to the data index information.

[0042] Based on similar inventive concepts, an embodiment of the present invention further provides a data processing system, such as Figure 2 As shown, the system includes: a data processing device 1, a database 2 and a data management system 3, wherein the data processing device 1 can preferably be used to implement the process in the above method embodiment.

[0043] Figure 3 is a structural block diagram of the data processing device 1, such as Figure 3 As shown, the data processing device 1 includes: a source data acquisition unit 11, a binary file generation unit 12, a splicing file generation unit 13 and a splicing file transmission unit 14, wherein:

[0044] The source data acquisition unit 11 is used to acquire the source data stored in the database according to the predetermined table structure field information in response to the source data table structure update.

[0045] The binary file generating unit 12 is used to generate at least one binary file from the acquired source data according to a predetermined single file size.

[0046] Specifically, the binary file generating unit generates a plurality of binary files in sequence according to a predetermined sequence number from the acquired source data according to a predetermined single file size.

[0047] The splicing file generating unit 13 is used to generate a splicing file according to the at least one binary file, the file name of the binary file, and the table structure field information.

[0048] The splicing file transmission unit 14 is used to transmit the splicing file to a predetermined location in the data management system according to predetermined data mapping information before executing the data management operation.

[0049] When the source data table structure is updated, the source data acquisition unit 11 acquires the source data according to the predetermined table structure field information, the binary file generation unit 12 generates at least one binary file from the acquired source data according to the predetermined single file size, and then the splicing file generation unit 13 generates a splicing file according to the binary file and its file name and table structure field information. Then, before executing the data management operation, the splicing file transmission unit 14 transmits the splicing file to a predetermined location in the data management system, thereby overcoming the problem of management errors in the prior art caused by the inability to synchronize to the data management system in real time when the source table structure changes. By generating multiple binary files, the problem of being unable to transmit files due to excessive file size can also be overcome. Through the embodiments of the present invention, the management data error rate can be reduced and the file transmission success rate can be improved.

[0050] In one embodiment, the data processing device 1 further includes: a binary file transmission unit, configured to transmit the at least one binary file to the data management system via a file transfer protocol.

[0051] In the specific implementation process, the above-mentioned data processing device 1 also includes: an index information generation unit and a query unit, wherein: the index information generation unit is used to pre-generate data index information, and the data index information includes: table structure field information; the query unit is used to perform query operations on the database according to the data index information.

[0052] Preferably, the above-mentioned data processing device 1 further comprises: a verification file generating unit, configured to generate a verification file of the at least one binary file according to the file size of the at least one binary file.

[0053] Furthermore, the above-mentioned data processing device 1 also includes: a mapping information updating unit, which is used to update the data mapping information according to the splicing file.

[0054] The specific execution process of the above-mentioned units and modules can be found in the description of the above-mentioned method embodiment, which will not be repeated here.

[0055] In actual operation, the above-mentioned units and modules may be provided in combination or individually, and the present invention is not limited thereto.

[0056] In order to better understand the present invention, the following Figure 4 A data processing example system is described in detail.

[0057] like Figure 4 As shown, the example system includes: a data acquisition system, a data transmission system and a data management system, wherein:

[0058] The data collection system is used to obtain the source database table structure and generate files. The file content is text that is spliced ​​by filling in spaces according to the number of characters in the table structure fields. It supports conditional queries and multiple data sources. When the table structure in the database is not unified, such as the addition, deletion, and modification of table structure fields, an alarm will be issued to facilitate subsequent processing by staff.

[0059] The data transmission system is used to realize data interaction between the data collection system and the data management system. The data transmission system can be a KAFKA (a high-throughput distributed publish-subscribe messaging system) system, which can transmit the information and files obtained by the data collection system to the data management system. If a timeout occurs, an alarm will be issued and a log will be output.

[0060] Data management system: uploads the acquired data files to the Hadoop (a distributed system infrastructure) distributed file system according to the predetermined logic.

[0061] The following combination Figure 5 Let's describe in detail how the example system works.

[0062] like Figure 5 As shown, the working principle of the example system includes:

[0063] 1. Preprocessing

[0064] Step 1-1, pre-register the lake entry table data, generate the association relationship between the source table and the Hadoop table structure, which can be a mapping relationship table sql in the form of a table, to facilitate subsequent query, data acquisition and other operations. Generate the lake entry information table, including the Hadoop library name, lake entry date (first lake entry date), lake entry status, and set the data status to 0 (0: not entered the lake, 1: entering the lake, 2: entered the lake).

[0065] Step 1-2: Enable the firewall of the file server of the data acquisition system and the data management system to send files using FTP.

[0066] 2. Working principle of data acquisition system

[0067] In step 2-1, multiple data sources can use information.schema to query the table structure and index of the database. The specific sample code is as follows:

[0068] show keys from'${tableName}'

[0069] select column_name,column_type,column_comment,is_nullable frominformation_schema.COLUMNS where table_schema=DATABASE()and table_name='${tableName}'

[0070] Step 2-2, compare the field information and indexes of the same table in multiple data sources. If the field information and indexes are inconsistent, it means that the tables of the multiple data sources are different, and an alarm is issued. If they are consistent, execute step 2-3.

[0071] Step 2-3, query the database data condition configuration, use the mapping relationship table sql in the preprocessing to obtain the table structure field information and index, and place it in xml (extensible markup language) as the select query condition for splicing, and the generated file name is table name_file serial number_entry date.

[0072] Sort the query table structure fields by orderby and add "," to concatenate them as the database fields to query the data of the source database and generate a BIN file (that is, the binary file mentioned above). The sample code for querying sql is as follows:

[0073] select${columnName}from${tableName}where PARTID BETWEEN${minPartid}AND${maxPartid}order by${primaryKey}LIMIT${beginRow},${line}

[0074] Step 2-4, in actual operation, you can set the maximum capacity of a single file, customize the file size, and configure it in the table name xml for easy query by the program. Specifically, you can use the field length to calculate the number of data rows allowed to be written into a single file in advance. File size = file size / query row size / number of query rows written each time. If the number of data rows allowed to be written into a single file exceeds the file size, the next file will be generated according to the sequence number.

[0075] Step 2-5, generate a CHK check file for each BIN file, and store the check file in a fixed directory on the local server. The CHK check file name is the same as the BIN file name, but the file name suffix is ​​different. The CHK check file can be generated based on the file size of the BIN file and used to verify the integrity of the BIN file later.

[0076] Step 2-6: Transfer the BIN file to the data management system via the FTP file transfer method in the data transmission system.

[0077] Step 2-7, concatenate the first entry date, table name, index, field name and field length of the corresponding field in preprocessing, and the file name of the source table with the concatenation character "**" for later use.

[0078] 3. Working principle of data transmission system

[0079] Step 3-1, for the KAFKA system, is responsible for transmitting the spliced ​​text of step 2-7 of the data collection system to the data management system. The push mode is asynchronous. If the message is not consumed for more than 3 days, an alarm is issued and the unconsumed information is written to the log table.

[0080] Step 3-2, for FTP file transfer, you can use the automatic trigger method to regularly scan the sending directory of the data acquisition system. When the file is completely generated in the folder of the local file server, the file is automatically sent to the file server of the data management system. If the file is sent successfully, the file will be moved to the success directory (the sending directory is added with .succ). If the file fails to be sent, the file will be moved to the failure directory (the sending directory is added with .fail) for backup.

[0081] 4. Working principle of data management system

[0082] Step 4-1, periodically read the KAFKA system to obtain the concatenated text of step 2-7, and intercept the obtained data with the concatenation symbol "**" for backup.

[0083] Step 4-2, compare the database field information, specifically, query the field information table according to the acquired data, if the data field information is different from the database field information, execute step 4-3, otherwise, execute step 4-4.

[0084] Step 4-3, when querying the field information table, if the field information table status is 1 (that is, the table field has changed), the old table data can be migrated to the history table, and then the corresponding table structure of HIVE (HIVE is a sql parsing engine based on Hadoopd, which can translate sql statements into MR relations and then execute them in Hadoop) is migrated. In actual operation, the data files of HDFS (Hadoop Distributed File System) will not be modified, but only the table definition in the metadata will be modified, that is, the MR (object-relational) data mapping of the data file content of HDFS will be performed in sequence again, and then the field status will be set to 0.

[0085] Step 4-4, query the field information table according to the acquired data. If the field information is the same, put the data directly into the predetermined data table and set the table status field information to 0 (that is, the table field has not changed). At the same time, update the first entry date, table name, index and entry file name to the entry information association mapping table for storage.

[0086] Step 4-5, read the table file name of the lake entry information association mapping table, and scan the files in the local server directory regularly according to the file name.

[0087] In step 4-6, use the jar package provided by Hadoop (hadoop-client.jar, hadoop-common.jar, hadoop-hadfs.jar, hadoop-mapreduce-client-core.jar) to stream the local BIN file to the server of the distributed file system HDFS, and set the data status to 1 (entering the lake) and 2 (entered the lake) in sequence. If successful, set the entry date to the next day.

[0088] Step 4-7, based on HIVE query table structure information table field information, and HDFS data file content in sequence mapped into a table (TABLE).

[0089] The example system provided in this embodiment is based on KAFKA for dynamic management of big data. On the one hand, when the source table changes temporarily, since the table structure of Hadoopd is achieved by consuming the field information pushed to KAFKA from the data acquisition system every day, HDFS can be migrated after the source table structure is updated and before the data is managed to avoid management errors. In addition, for the situation where the business volume is increasing, dynamically adding files by sequence number to write data can well solve the problem of being unable to transmit due to a single file being too large, thereby reducing maintenance costs. On the other hand, the system configures the table name, query conditions, data source and other configurations, thereby improving the ease of operation of the system.

[0090] This embodiment also provides an electronic device, which may be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited thereto. In this embodiment, the electronic device may be implemented with reference to the above method embodiment and the data processing device / system embodiment, the contents of which are incorporated herein, and the repeated parts are not repeated.

[0091] Figure 6 FIG. 6 is a schematic block diagram of a system structure of an electronic device 600 according to an embodiment of the present invention. Figure 6 As shown, the electronic device 600 may include a central processor 100 and a memory 140; the memory 140 is coupled to the central processor 100. It should be noted that this figure is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0092] In one embodiment, the data processing function may be integrated into the central processing unit 100. The central processing unit 100 may be configured to perform the following control:

[0093] In response to the source data table structure being updated, the source data stored in the database is acquired according to the predetermined table structure field information;

[0094] Generate at least one binary file from the acquired source data according to a predetermined single file size;

[0095] Generate a spliced ​​file according to the at least one binary file, the file name of the binary file, and the table structure field information;

[0096] Before executing the data management operation, the spliced ​​file is transmitted to a predetermined location in the data management system according to predetermined data mapping information.

[0097] From the above description, it can be seen that the electronic device provided in the embodiment of the present application obtains source data according to predetermined table structure field information when the source data table structure is updated, and generates at least one binary file from the obtained source data according to a predetermined single file size, and then generates a spliced ​​file according to the binary file, its file name and table structure field information, and then before executing the data management operation, the spliced ​​file is transmitted to a predetermined location in the data management system, thereby overcoming the problem of management errors in the prior art caused by the inability to synchronize to the data management system in real time when the source table structure changes. By generating multiple binary files, the problem of being unable to transmit files due to excessive file size can also be overcome. Through the embodiments of the present invention, the error rate of managed data can be reduced and the success rate of file transmission can be improved.

[0098] In another embodiment, the data processing device / system may be configured separately from the central processing unit 100. For example, the data processing device / system may be configured as a chip connected to the central processing unit 100, and the data processing function is implemented under the control of the central processing unit.

[0099] like Figure 6 As shown, the electronic device 600 may further include: a communication module 110, an input unit 120, an audio processing unit 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily have to include Figure 6 In addition, the electronic device 600 may also include Figure 6 For components not shown, reference may be made to the prior art.

[0100] like Figure 6 As shown, the central processor 100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The central processor 100 receives inputs and controls the operations of various components of the electronic device 600 .

[0101] The memory 140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 100 may execute the program stored in the memory 140 to implement information storage or processing.

[0102] The input unit 120 provides input to the CPU 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0103] The memory 140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 140 may also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 600 through the central processor 100.

[0104] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0105] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processor 100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0106] Based on different communication technologies, multiple communication modules 110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide an audio output via the speaker 131 and receive an audio input from the microphone 132, thereby realizing a common telecommunication function. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 130 is also coupled to the central processor 100, so that the sound can be recorded on the local machine through the microphone 132, and the sound stored on the local machine can be played through the speaker 131.

[0107] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0108] In summary, the embodiments of the present invention provide a data processing solution, which is mainly used to solve the defect that when the source table undergoes temporary structural changes, it cannot be synchronized to the big data management platform in real time. When the amount of data surges, it can dynamically cut and generate corresponding files according to the serial number to prevent the phenomenon that the file cannot be transmitted because it is too large. Therefore, the purpose of entering the lake can be achieved by only setting the configuration file.

[0109] The preferred embodiments of the present invention are described above with reference to the accompanying drawings. Many features and advantages of these embodiments are clear from this detailed description, and therefore the claims are intended to cover all such features and advantages of these embodiments that fall within their true spirit and scope. In addition, since many modifications and changes are easily conceivable to those skilled in the art, it is not intended to limit the embodiments of the present invention to the precise structure and operation illustrated and described, but all suitable modifications and equivalents falling within its scope may be covered.

[0110] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: In response to the source data table structure being updated, the source data stored in the database is acquired according to the predetermined table structure field information; Generate at least one binary file from the acquired source data according to a predetermined single file size; Generate a spliced ​​file according to the at least one binary file, the file name of the binary file, and the table structure field information; Before executing the data management operation, the spliced ​​file is transmitted to a predetermined location in the data management system according to predetermined data mapping information.

2. The method according to claim 1, characterized in that After generating at least one binary file from the acquired source data, the method further includes: The at least one binary file is transferred to the data management system through a file transfer protocol.

3. The method according to claim 1, characterized in that: The method further comprises: Pre-generate data index information, the data index information includes: table structure field information; A query operation is performed on the database according to the data index information.

4. The method according to claim 1, characterized in that After generating at least one binary file from the acquired source data, the method further includes: A checksum file of the at least one binary file is generated according to the file size of the at least one binary file.

5. The method according to claim 1, characterized in that Generating at least one binary file from the acquired source data according to a predetermined single file size includes: According to the predetermined single file size, the acquired source data is sequentially generated into multiple binary files according to the predetermined sequence numbers.

6. The method according to claim 1, characterized in that After transmitting the spliced ​​file to a predetermined location in the data management system according to the predetermined data mapping information, the method further includes: The data mapping information is updated according to the splicing file.

7. A data processing device, characterized in that: The device comprises: A source data acquisition unit, configured to acquire source data stored in a database according to predetermined table structure field information in response to source data table structure update; A binary file generating unit, used to generate at least one binary file from the acquired source data according to a predetermined single file size; A splicing file generating unit, used to generate a splicing file according to the at least one binary file, the file name of the binary file, and the table structure field information; The splicing file transmission unit is used to transmit the splicing file to a predetermined location in the data management system according to predetermined data mapping information before executing the data management operation.

8. The device according to claim 7, characterized in that The device also includes: A binary file transmission unit is used to transmit the at least one binary file to the data management system through a file transfer protocol.

9. The device according to claim 7, characterized in that The device also includes: An index information generating unit, used to generate data index information in advance, wherein the data index information includes: table structure field information; A query unit is used to perform a query operation on the database according to the data index information.

10. The device according to claim 7, characterized in that The device also includes: The verification file generating unit is used to generate a verification file of the at least one binary file according to the file size of the at least one binary file.

11. The device according to claim 7, characterized in that The binary file generation unit is specifically used for: According to the predetermined single file size, the acquired source data is sequentially generated into multiple binary files according to the predetermined sequence numbers.

12. The device according to claim 7, characterized in that The device also includes: A mapping information updating unit is used to update the data mapping information according to the splicing file.

13. A data processing system, characterized in that: The system comprises: a data processing device, a database and a data management system as claimed in any one of claims 7 to 12.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images