Data processing method, apparatus, device, medium, and product

Abstract

The application discloses a data processing method and device, equipment, medium and product, and relates to the technical field of big data processing. The method comprises the following steps: receiving a data loading request sent by a target object, wherein the data loading request comprises a target table name; querying whether a target file path matched with the target table name exists in a data file mapping library based on a pre-established mapping relationship, wherein the mapping relationship is a corresponding relationship between a table name of a data file and a file path of the data file; in response to the existence of the target file path, determining a target data file according to the target file path; creating a first historical table and loading full-amount data in the target data file into the first historical table; and switching a positive table and the first historical table to complete data loading. According to the embodiment of the application, the generality of data loading can be realized, the complexity of data loading and the later maintenance cost are reduced, and the data loading efficiency is improved.

Description

Technical Field

[0001] This application belongs to the field of big data processing technology, and in particular relates to a data processing method, apparatus, equipment, medium and product. Background Technology

[0002] As the business grows, the demand for large amounts of data sharing between various systems is increasing, thus requiring higher and higher data processing efficiency.

[0003] In related technologies, data loading is usually achieved by writing shell scripts. However, different shell scripts need to be written for different data loading, which means that scripts need to be added continuously to adapt to new data loading requirements. This leads to an increasing number of loading scripts, which in turn increases the workload and cost of script maintenance and affects data loading efficiency. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, medium, and product that can achieve the universality of data loading and improve data loading efficiency.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Receive a data loading request sent by the target object, the data loading request including the target table name;

[0007] Based on the pre-established mapping relationship, the system queries the data file mapping library to see if there is a target file path that matches the target table name. The mapping relationship is the correspondence between the table name of the data file and the file path of the data file.

[0008] In response to the existence of the target file path, the target data file is determined based on the target file path;

[0009] Create a first history table and load all data from the target data file into the first history table;

[0010] Switch the current table to the first historical table to complete the data loading.

[0011] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0012] The receiving module is used to receive a data loading request sent by the target object, wherein the data loading request includes the target table name;

[0013] The query module is used to query the data file mapping library for whether there is a target file path that matches the target table name, based on a pre-established mapping relationship. The mapping relationship is the correspondence between the table name of the data file and the file path of the data file.

[0014] A determination module is configured to determine a target data file based on the target file path in response to the existence of the target file path;

[0015] A loading module is used to create a first history table and load all data from the target data file into the first history table.

[0016] The switching module is used to switch the current table with the first historical table to complete the data loading.

[0017] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the steps of the data processing method as described in any embodiment of the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the data processing method as described in any embodiment of the first aspect.

[0019] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the steps of the data processing method as described in any embodiment of the first aspect.

[0020] The data processing method and related equipment in this application embodiment can obtain the target data file by querying the target file path that matches the target table name based on the data file mapping library. Then, the full data in the target data file is loaded into the historical table, and the data loading is completed by switching between the current table and the historical table. This achieves the universality of data loading, without having to add a new script for each data loading requirement, reducing the complexity of data loading and the later maintenance cost, and improving the efficiency of data loading. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0023] Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0024] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0029] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0030] Shell scripts are program files that use Linux / Unix commands. They contain various commands pre-loaded into a single file for easy execution. Shell scripting is a scripting language used in Linux / Unix systems, allowing for complex operations.

[0031] A database is a repository for organizing, storing, and managing data according to a data structure. It is a large collection of organized, shareable, and uniformly managed data that is stored permanently in a computer.

[0032] CTL control files are used to represent a binary file that can be used to record information such as the database name and the location of the database data files.

[0033] As the business grows, the demand for large amounts of data sharing between various systems is increasing, thus requiring higher and higher data processing efficiency.

[0034] In related technologies, data loading is usually achieved by writing shell scripts. However, different shell scripts need to be written for different data loading, which means that scripts need to be added continuously to adapt to new data loading requirements. This leads to an increasing number of loading scripts, which in turn increases the workload and cost of script maintenance and affects data loading efficiency.

[0035] To address the problems in the related technologies, embodiments of this application provide a data processing method, apparatus, device, medium, and product.

[0036] The data processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0037] Figure 1 A flowchart illustrating a data processing method 100 according to an embodiment of this application is shown. Figure 1 As shown, the data processing method 100 may specifically include the following steps:

[0038] S101. Receive a data loading request sent by the target object, wherein the data loading request includes the target table name;

[0039] S102. Based on the pre-established mapping relationship, query the data file mapping library to see if there is a target file path that matches the target table name. The mapping relationship is the correspondence between the table name of the data file and the file path of the data file.

[0040] S103. In response to the existence of the target file path, determine the target data file based on the target file path;

[0041] S104. Create a first history table and load all the data in the target data file into the first history table;

[0042] S105. Switch the current table with the first historical table to complete the data loading.

[0043] Therefore, based on the data file mapping library, the target file path matching the target table name can be queried to obtain the target data file. Then, the full data in the target data file is loaded into the historical table, and the data loading is completed by switching between the current table and the historical table. This achieves the universality of data loading, without having to add a new script for each data loading requirement, reducing the complexity of data loading and the later maintenance cost, and improving the efficiency of data loading.

[0044] The specific implementation methods for each of the above steps are described below.

[0045] It should be noted that the table name in this embodiment is unique for each data file; that is, each data file has only one corresponding table name. Alternatively, a unique identifier for a data file can also be based on "database name + table name," but this embodiment does not impose specific limitations on this.

[0046] In some embodiments, a data file mapping library is established before step S101. Specifically, this data file mapping library stores the table names of data files, the file paths of data files, and mapping relationships, wherein the mapping relationship is the correspondence between the table names of data files and the file paths of data files.

[0047] In practice, a mapping relationship is established between table names and file paths in the data files. Then, the table names, file paths, and mapping relationships are stored in a data file mapping library. Furthermore, table names are unique, meaning there is a one-to-one correspondence between data files and table names, and therefore, the mapping relationship between table names and data file paths is also unique.

[0048] In this way, after obtaining the target table name, the unique target file path corresponding to the target table name can be found in the data file mapping library based on the mapping relationship, so as to obtain the target data file according to the target file path.

[0049] It should be understood that the establishment of mapping relationships is based on communication confirmation with the target object. Specifically, after confirming the paths of the several data files that need to be processed with the target object, the mapping relationship between table names and file paths is configured. Thus, when data from another data file needs to be loaded, the data file mapping library can be updated by adding a new mapping relationship, and the corresponding file path can be obtained by querying the data file mapping library with the passed table name as a parameter.

[0050] In some embodiments, based on a database, after configuring environment variables, one can directly connect to the database in a shell script using the sqlplus command to perform corresponding operations.

[0051] In some embodiments, in step S101, when a data loading request sent by the target object is received, the data loading request can be parsed to obtain the target table name. The target table name may include the identifier or index corresponding to the target data file.

[0052] Thus, the file path of the target file can be determined based on the parsed target table name. Specifically, in some embodiments, in step S102, based on the pre-established mapping relationship between the table names and file paths of data files, a query can be performed in the data file mapping library to see if a target file path matching the target table name exists. Furthermore, in step S103, after determining the target file path, the corresponding target data file can be obtained.

[0053] The target data file can be a .dat data file, and the fields in the .dat data file can be separated by a uniform string |@|, and the predefined order of the fields can be displayed. For example, field1|@|field2|@|field3|@|.

[0054] In another embodiment, if no target file path matching the target table name is found in the data file mapping library, i.e., the target file path does not exist in the data file mapping library, the data file mapping library is updated by confirming with the target object.

[0055] Furthermore, in some embodiments, before step S104, it is determined whether a second history table exists. Specifically, in response to the existence of a second history table, the second history table is deleted. For details, please refer to... Figure 2 Another data processing method 200. In this method, the second history table can store all historical data from the data loading process.

[0056] Thus, by determining whether a second historical table exists, if a second historical table exists, the second historical table is deleted and a first historical table is created so that all data in the target data file can be loaded into the first historical table subsequently; if a second historical table does not exist, the first historical table is created directly.

[0057] In related technologies, data loading is typically achieved by sequentially searching for each target data entry in a temporary table containing all backed-up data, and then storing that target data in a historical table. However, for these data processing solutions, directly manipulating the database poses certain security risks, and querying and storing single or incremental data may lead to issues such as duplicate data processing or missing data entries.

[0058] Therefore, in this embodiment, the entire data in the target data file is loaded into the first historical table, and the data loading is completed by switching between the current table and the first historical table. Specifically, DDL and DML operations are used to load, delete, and rename data in the historical table, temporary table, and current table, respectively.

[0059] In some embodiments, a data program control file may be generated before step S104, so that in step S104, the full data in the target data file is loaded into the first history table using the data program control file.

[0060] In practice, the target field can be determined by receiving instructions from the target object, and then the type and length of the target field can be defined. Based on the type and length of the target field, a data program control file can be generated. This data program control file can be a ctl data program control file.

[0061] In practice, the field separator |@| is used as the delimiter for fields terminated by. A CTL data loading program is written according to the predefined target fields, and the type, length, and date format of the target fields are determined.

[0062] In this way, because the generated data program control files are differentiated, different data loading requirements can be met by using these data program control files for data loading.

[0063] Therefore, in some embodiments, in step S105, the first historical table, which already contains the full data corresponding to the target data file, is switched with the positive table (containing the full data of the data file corresponding to the previous stage, i.e., the full data corresponding to the previous data processing) to complete the loading of the target data. Specifically, this switching can be achieved by renaming the positive table to a temporary table and renaming the first historical table to the target positive table. That is, by changing the table names, a temporary table and a target positive table are generated respectively, and the temporary table is used as a tool table to complete the loading of the target data into the target positive table, thereby achieving the corresponding data processing and improving data processing efficiency.

[0064] Furthermore, in some embodiments, the temporary table is renamed a third historical table. This stores the full data corresponding to the previous data processing (i.e., historical data) in the third historical table. Because historical data is retained, it can be queried based on this third historical table, enabling data tracing. Moreover, the third historical table will be deleted during the next data processing to create a new historical table for loading data into the next target data file. This allows only the full data corresponding to the current stage and the previous stage to be stored and retained, reducing data redundancy and storage burden.

[0065] As can be seen, data processing method 100 only needs to maintain the mapping relationship between table names and data file paths and generate data program control files, rather than using a script-per-requirement approach as in related technologies, where a new shell script needs to be written each time new table data is loaded. This solves the problems of low data loading efficiency and script errors leading to missing data in related technologies, reducing the complexity and cost of data loading and making it universally applicable. In another embodiment, data loading can be completed by renaming the current table to the fourth historical table, renaming the pre-generated alternative historical table to the fifth historical table, and loading all data from the target data file into the target current table.

[0066] In practice, since the main table stores the full data of the data file corresponding to the previous stage—that is, the full data corresponding to the previous data processing—the main table needs to be renamed to load the target data into the target main table. Thus, the fourth history table stores the full data corresponding to the previous stage for data tracing.

[0067] In practice, the alternative historical table saved in the previous stage can be left undeleted. Instead, to differentiate the processing, this alternative historical table is renamed the fifth historical table. This allows for tracing, searching, and retrieving historical data from different stages based on both the fourth and fifth historical tables.

[0068] In practice, due to the renaming and switching process of the main table, it is necessary to create a new main table for loading the target data in the current data processing. Specifically, the full data (i.e., the target data) in the target data file can be loaded into the target main table through a pre-generated data control file to complete the data loading process.

[0069] In addition, refer to Figure 3 This is a flowchart illustrating another data processing method 300.

[0070] like Figure 3 As shown, in step S301, the old historical table is deleted; in step S302, a new historical table is created, and all data from the target data file is loaded into the historical table according to the table name and the pre-generated data program control file; in step S303, the old positive table is renamed to a temporary table; in step S304, the historical table is renamed to a positive table, and the data loading is complete; in step S305, the temporary table is renamed to a historical table, so that it will be deleted again during the next data loading, thus forming a closed loop with step S301.

[0071] Specifically, the passed-in table name is used as a placeholder parameter, with temporary table name = ${tablename}_tmp ​​and historical table name = ${tablename}_his.

[0072] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides a data processing apparatus 400.

[0074] like Figure 4 As shown, the data processing device 400 may include:

[0075] Receiving module 401 is used to receive a data loading request sent by the target object, wherein the data loading request includes the target table name;

[0076] The query module 402 is used to query the data file mapping library for whether there is a target file path that matches the target table name based on a pre-established mapping relationship. The mapping relationship is the correspondence between the table name of the data file and the file path of the data file.

[0077] The determination module 403 is configured to determine the target data file based on the target file path in response to the existence of the target file path;

[0078] Loading module 404 is used to create a first history table and load all data from the target data file into the first history table;

[0079] The switching module 405 is used to switch the current table with the first historical table to complete the data loading.

[0080] In some embodiments, the loading module 404 is specifically used to load all data in the target data file into the first history table using a pre-generated data program control file.

[0081] In some embodiments, the switching module 405 is specifically used to rename the main table to a temporary table; rename the first historical table to a target main table; and rename the temporary table to a third historical table to complete data loading.

[0082] In some embodiments, the data processing apparatus 400 further includes a generation module ( Figure 4 (Not shown in the image). This generation module is used to receive instructions from the target object to determine target fields; define the type and length of the target fields; and generate a data program control file based on the type and length of the target fields.

[0083] In some embodiments, the data processing device 400 further includes a determination module ( Figure 4 (Not shown in the image). This judgment module is used to determine whether a second historical table exists, and in response to the existence of a second historical table, to delete the second historical table.

[0084] In some embodiments, the data processing apparatus 400 further includes a setup module. Figure 4 (Not shown in the image). This creation module is used to establish the mapping relationship between the table name and the file path of the data file; and to store the table name, file path and mapping relationship in the data file mapping library.

[0085] It should be noted that, for ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0086] The apparatus of the above embodiments is used to implement the corresponding data processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0087] Based on the same technical concept, corresponding to any of the above embodiments, this application also provides an electronic device.

[0088] Figure 5 A schematic diagram of a more specific electronic device hardware structure provided in this embodiment is shown.

[0089] The electronic device 500 may include a processor 501 and a memory 502 storing computer program instructions.

[0090] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0091] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0092] In certain embodiments, the memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0093] The processor 501 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0094] In some examples, the electronic device 500 may also include a communication interface 503 and a bus 510. For example, Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0095] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0096] Bus 510 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, bus 510 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0097] For example, the electronic device 500 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0098] Based on the same technical concept, corresponding to any of the methods in the above embodiments, this application also provides a non-transitory computer-readable storage medium. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the data processing methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0099] Based on the same technical concept, corresponding to any of the above-described embodiments, this application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processors to perform the data processing method. Corresponding to the execution entity for each step in each embodiment of the data processing method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0100] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0101] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0102] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0103] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0104] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Receive a data loading request sent by the target object, the data loading request including the target table name; Based on the pre-established mapping relationship, the system queries the data file mapping library to see if there is a target file path that matches the target table name. The mapping relationship is the correspondence between the table name of the data file and the file path of the data file. In response to the existence of the target file path, the target data file is determined based on the target file path; Create a first history table, and use a pre-generated data program control file to load all the data in the target data file into the first history table; Switch the current table to the first historical table to complete the data loading; Before receiving the data loading request sent by the target object, the method further includes: Receive instructions from the target object to determine the target fields; Define the type and length of the target field; Generate the data program control file based on the type and length of the target field; Before receiving the data loading request sent by the target object, the method further includes: Establish the mapping relationship between the table names of the data files and the file paths of the data files; The table name, file path, and mapping relationship are stored in the data file mapping library; Specifically, when it is necessary to load data from another data file, the data file mapping library can be updated by adding a new mapping relationship, thereby querying the data file mapping library with the passed table name as a parameter to obtain the corresponding file path.

2. The method according to claim 1 or 1, characterized in that, Before loading all data from the target data file into the first history table, the method further includes: Determine if a second historical table exists. If a second historical table exists, delete the second historical table.

3. The method according to claim 1, characterized in that, The step of switching between the current table and the first historical table to complete data loading includes: Rename the regular table to a temporary table; Rename the first history table to the target positive table; The temporary table is renamed to the third historical table to complete the data loading.

4. A data processing apparatus, characterized in that, The device includes: The receiving module is used to receive a data loading request sent by the target object, wherein the data loading request includes the target table name; The query module is used to query the data file mapping library for whether there is a target file path that matches the target table name, based on a pre-established mapping relationship. The mapping relationship is the correspondence between the table name of the data file and the file path of the data file. A determination module is configured to determine a target data file based on the target file path in response to the existence of the target file path; A loading module is used to create a first history table and load all the data in the target data file into the first history table using a pre-generated data program control file; The switching module is used to switch the current table with the first historical table to complete the data loading. Before receiving the data loading request sent by the target object, the apparatus further includes: a generation module, configured to receive instructions from the target object to determine a target field before receiving the data loading request sent by the target object; define the type and length of the target field; and generate the data program control file according to the type and length of the target field; Before receiving the data loading request sent by the target object, the device further includes: A module is established to create a mapping relationship between the table name and the file path of the data file before receiving a data loading request from the target object; the table name, file path, and mapping relationship are stored in the data file mapping library; wherein, when it is necessary to load data from another data file, the data file mapping library can be updated by adding a new mapping relationship, thereby querying the data file mapping library with the passed table name as a parameter to obtain the corresponding file path.

5. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the data processing method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-3.

7. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data processing method as described in any one of claims 1-3.

Images