A method and device for data processing

Abstract

The present invention discloses a method and device for data processing, and relates to the field of computer technology. A specific implementation of the method includes: performing an inner join operation on a master data table and a data table to be associated to obtain an associated master data table; performing a left join operation on an incremental data table of the master data and an associated master data table to obtain a target data table for data processing. This implementation can significantly save computing resources, reduce the computing costs of an enterprise, and improve timeliness. At the same time, it can also ensure that the amount of data processed every day is basically the same, and the resources to be consumed remain stable, avoiding the potential risk of computing failure due to the daily increase in resource consumption.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data processing. Background Art

[0002] With the rapid development of Internet technology, data has exploded and is growing exponentially. Facing such a large amount of data, traditional data warehouse systems have long been powerless. The emergence of Hive (a data warehouse tool based on Hadoop, which provides a SQL-style query language to query the file system stored on Hadoop) built on the Hadoop (a distributed system infrastructure developed by the Apache Foundation) cluster provides a solution for the processing of massive data.

[0003] However, as time goes by, the amount of data that Hive needs to process is increasing, the resources it consumes are increasing, and the processing time is getting longer and longer. If this trend continues, it will one day be overwhelmed. Once this happens, the consequences will be unimaginable. At the same time, due to increasingly fierce competition, the time required for response to decisions based on data is getting shorter and shorter, and real-time technology is still unable to handle complex data associations. This requires Hive to accurately handle complex association logic while responding as quickly as possible.

[0004] Data association is a basic operation of relational databases. Hive based on MapReduce (distributed computing) also supports data association operations on massive data sets. Usually, when Hive performs DW (Data Warehouse) data association, it is performed by two or more massive data sets. Common data processing modes in the e-commerce industry include: billions of user traffic and billions of goods and billions of users are associated to obtain user behavior characteristics or common paths for users to reach goods, etc., hundreds of millions of order data sets are associated with billions of goods and billions of users to obtain hot-selling goods, high-value users and other content. The main characteristics of this type of operation are: first, the incremental data is used as the main table to associate with other full data, and the amount of associated data is huge; second, it consumes more resources; third, it takes a long time to process.

[0005] The existing method of using incremental tables as the main table to perform association operations with other full tables is acceptable in terms of resource consumption and processing timeliness in the initial stage when the amount of data processed is within a certain range. However, as time goes by, the amount of data in the full table increases day by day, and the amount of data becomes larger and larger, and the resources consumed during the operation also increase, and the operation speed also tends to gradually slow down. At the same time, there are extreme situations where the operation fails due to exceeding the Hive operation limit. This situation is absolutely unacceptable.

[0006] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0007] 1) The resources consumed in the computing process increase every day, and a lot of unnecessary data is involved in the computing process, wasting scarce resources and consuming resource costs;

[0008] 2) The operation time is long, which cannot meet the timeliness requirements of the data warehouse, affecting decision-making efficiency and causing unnecessary losses to the enterprise;

[0009] 3) There is a risk of task failure due to exceeding the data processing limit of current technology, resulting in the inability to calculate the data, affecting front-end applications such as decision-making, production system recommendations, and warehouse intelligent replenishment, affecting user experience, and causing immeasurable losses to the company. Summary of the invention

[0010] In view of this, an embodiment of the present invention provides a method and device for data processing, which can significantly save computing resources, reduce the computing costs of an enterprise, and improve timeliness. At the same time, it can also ensure that the amount of data processed every day is basically the same, and the resources required to be consumed remain stable, thereby avoiding the potential risk of computing failure due to the daily increase in resource consumption.

[0011] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for data processing is provided.

[0012] A data processing method includes: performing inner join operation on a main data table and a data table to be associated to obtain an associated main data table; performing left join operation on an incremental data table of main data and the associated main data table to obtain a target data table for data processing.

[0013] Optionally, performing a left join operation on the incremental data table of the main data and the associated main data table to obtain a target data table includes: taking the incremental data table of the main data as the main table, and the associated main data table as the auxiliary table, and performing a left join operation on the incremental data table of the main data and the associated main data table to obtain the target data table.

[0014] Optionally, the main data table and the data table to be associated are full data tables.

[0015] Optionally, the master data is obtained by screening business data according to business requirements and filtering out invalid data.

[0016] According to another aspect of an embodiment of the present invention, a data processing device is provided.

[0017] A data processing device includes: a first processing module, used to perform an inner join operation on a main data table and a data table to be associated to obtain an associated main data table; a second processing module, used to perform a left join operation on an incremental data table of the main data and the associated main data table to obtain a target data table for data processing.

[0018] Optionally, the second processing module is further used to: use the incremental data table of the main data as the main table, the associated main data table as the auxiliary table, and perform a left join operation on the incremental data table of the main data and the associated main data table to obtain a target data table.

[0019] Optionally, the main data table and the data table to be associated are full data tables.

[0020] Optionally, the master data is obtained by screening business data according to business requirements and filtering out invalid data.

[0021] According to yet another aspect of the embodiments of the present invention, an electronic device for data processing is provided.

[0022] An electronic device for data processing comprises: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by an embodiment of the present invention.

[0023] According to still another aspect of the embodiments of the present invention, a computer readable medium is provided.

[0024] A computer readable medium stores a computer program, which, when executed by a processor, implements a data processing method provided by an embodiment of the present invention.

[0025] An embodiment of the above invention has the following advantages or beneficial effects: by performing an inner join operation on the main data table and the data table to be associated to obtain the associated main data table; performing a left join operation on the incremental data table of the main data and the associated main data table to obtain the target data table for data processing, the number of data records involved in the operation and the file size can be greatly reduced, thereby greatly saving computing resources and reducing the computing costs of the enterprise; after the amount of data to be processed is reduced, the internal processing steps are reduced, the processing efficiency is improved, and the timeliness is greatly improved; in addition, it can also ensure that the amount of data processed every day is basically the same, the resources required to be consumed remain stable, and the potential risk of computing failure due to the daily increase in resource consumption is avoided.

[0026] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0028] Figure 1 is a schematic diagram of main steps of a method for data processing according to an embodiment of the present invention;

[0029] Figure 2 It is a schematic diagram of the principle of the connection type of data association;

[0030] Figure 3 is a schematic diagram of a data processing process according to an embodiment of the present invention;

[0031] Figure 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0032] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0033] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0035] In order to solve the problems existing in the prior art, the present invention provides a method and device for data processing based on master data, which adopts an operation strategy based on master data to perform associated operations, and can eliminate useless data in advance before participating in the operation, saving operation resources and reducing operation costs; improving operation efficiency, completing operations earlier to meet the timeliness requirements of the data warehouse level, obtaining data earlier, and formulating strategies earlier to avoid losses caused by untimely decision-making; avoiding the situation where data cannot be calculated, preventing problems before they occur, and avoiding greater losses to the enterprise.

[0036] Figure 1 FIG. 1 is a schematic diagram of the main steps of the data processing method according to an embodiment of the present invention. Figure 1 As shown, the data processing method of the embodiment of the present invention mainly includes the following steps S101 and S102.

[0037] Step S101: perform inner join operation on the main data table and the data table to be associated to obtain an associated main data table;

[0038] Step S102: Perform a left join operation on the incremental data table of the master data and the associated master data table to obtain a target data table for data processing.

[0039] Master data refers to data that is shared between business systems (operational / transactional application systems and analytical systems, etc.) across the entire enterprise. For example, it can be data related to customers, accounts, and organizational units. Master data usually needs to be consistent, complete, and controlled across the entire enterprise. Data to be associated is data that is associated with master data. Depending on the actual business, there may be multiple different copies of data.

[0040] In a data warehouse, data is generally stored in data tables. Accordingly, master data is stored in the master data table, and data to be associated is stored in the data table to be associated. There is generally only one master data table. The data table to be associated stores specific business data, and there may be multiple tables depending on the actual business. Taking the product table as an example, the master data table stores the main information of the product (such as product name, product type, etc.), and the data table to be associated may include multiple tables such as the product extension table (such as product supplier number, product supplier name, etc.), the product category table (such as product first-level category number, product first-level category name, etc.).

[0041] According to one embodiment of the present invention, the master data stored in the master data table is obtained by screening the business data according to business needs and filtering out invalid data. Specifically, the business data can be screened according to the business situation to determine the content of the master data. By screening the business data, invalid data can be filtered out in advance, which can save computing resources and improve the efficiency of data processing. The creation time of the data table to be associated is relatively flexible. It can be created from the beginning of the business generation, or it can be created when it is necessary to add records of the data to be associated, which does not affect the implementation of the technical solution of the present invention.

[0042] In the process of data processing, in order to distinguish data tables, the tables that need to be associated with data can be divided into main tables and auxiliary tables, that is, the main table and the auxiliary table are corresponding concepts, and the data in the main table and the auxiliary table are to be associated. In addition, in the embodiment of the present invention, a full table and an incremental table are also involved. The full table and the incremental table are corresponding concepts. The full table stores all data from the initial generation of the business to the present, and the incremental table stores the data of the business changes every day (with "day" as the unit time for data statistics, which can be flexibly set according to needs).

[0043] According to the technical solution of the present invention, both the master data and the data to be associated are set as full data, that is, all data from the beginning of the business to the present, so as to avoid data loss and ensure the accuracy of data processing results.

[0044] According to a technical solution of an embodiment of the present invention, in step S101, when the main data table and the data table to be associated are inner-joined to obtain the associated main data table, the main data table can be used as the main table and the data table to be associated as the auxiliary table, and the main data table and the data table to be associated are inner-joined to obtain the associated main data table; or the data table to be associated is used as the main table and the main data table is used as the auxiliary table, and the main data table and the data table to be associated are inner-joined to obtain the associated main data table; and in step S102, when the incremental data table of the main data and the associated main data table are left-joined to obtain the target data, the incremental data table of the main data can be used as the main table and the associated main data table as the auxiliary table, and the incremental data table of the main data and the associated main data table are left-joined to obtain the target data table.

[0045] Figure 2 This is a schematic diagram of the principle of the connection type of data association. Figure 2 As shown, the connection methods of data association include inner join, left join, right join and full join. The result of inner join operation of two data tables is the intersection of the two data tables, that is, the content related to the common parts of the two data tables; the result of left join operation of two data tables is to retain only the relevant content of the part existing in the left data table; the result of right join operation of two data tables is to retain only the relevant content of the part existing in the right data table; the result of full join operation of data tables is the union of the two data tables, that is, the content obtained after deduplication of all the contents existing in the two data tables. Taking inner join operation as an example, suppose a record in the main data table is: "Product name AA, product type XX"; the first-level category number of the product is recorded in the data table 1 to be associated. When the data table 1 to be associated is inner-joined with the main data table, if the data table 1 to be associated contains data corresponding to "product name AA", the data table obtained after the inner-join operation will contain a record "product name AA, product type XX, first-level category number of product AA"; otherwise, the data table obtained after the inner-join operation will not contain data corresponding to "product name AA".

[0046] In step S101, by performing an inner join operation on the main data table and the data table to be associated, the intersection content of the main data table and the data table to be associated can be obtained, so that a large amount of irrelevant data in the data table to be associated can be eliminated to avoid participating in the operation in the subsequent operation process, thereby saving resources and improving the timeliness of data processing.

[0047] Afterwards, in step S102, by performing a left join operation on the incremental data table of the master data and the associated master data table, only the data-related content contained in the left data table (the incremental data table of the master data) can be retained, so that the amount of data processed every day can be basically consistent, thereby avoiding the risk of operation failure caused by the infinite increase in processing data reaching the technical bottleneck point of data processing volume; at the same time, under the premise of ensuring that the operation result is completely consistent with the result of processing the incremental data table of the master data using all the full tables, the data resources required are greatly reduced, the operation process is simplified, and the operation efficiency is improved.

[0048] Figure 3 FIG. 1 is a schematic diagram of a data processing process according to an embodiment of the present invention. Figure 3 As shown, when the data warehouse processes data, in order to process data efficiently, a buffer data layer and a backup data layer are set up, and according to the primary key of the data (the meaning of the primary key is the field information that can uniquely identify a record), the basic data is obtained by zipping the data in the buffer data layer and the backup data layer (the zipper table is defined for the way tables store data in data warehouse design. As the name suggests, the so-called zipper records history. It records all the changes of a thing from the beginning to the current state) and saves it to the basic data layer for backup. In the introduction of the embodiments of the present invention, the data saved in each data table refers to the data saved in the basic data layer.

[0049] like Figure 3 As shown, the data processing process of the embodiment of the present invention generally involves the following modules:

[0050] 1. Create table module: In Hive, create the master data table, associated master data table and target data table. The master data table is used to store the master data that needs to be associated; the associated master data table is used to store the associated master data obtained after the inner join operation between the master data and the data to be associated; the target data table is used to store the target data obtained after the data processing is completed and participate in subsequent operations;

[0051] 2. Generate master data module: filter data according to business conditions to determine the master data content, and store the filtered master data in the created master data table;

[0052] 3. Association operation module: perform inner join operation on the data table to be associated (one or more tables created in advance to store data used to associate with the master data) and the generated master data table to obtain the associated master data information, and store the associated master data in the associated master data table created in Hive. The specific implementation principle is: perform inner join operation on the two tables, and the result is the intersection of the two tables, that is, the related content of the common parts of the two tables. In this way, a large amount of irrelevant data in the data table to be associated can be eliminated to avoid participating in the operation in the next link, thereby saving resources and improving the operation time efficiency;

[0053] 4. Generate target data module: perform left join operation on the incremental data table of the master data and the associated master data table to obtain the target data information, and store the target data in the target data table created in Hive. The specific implementation principle is: perform left join operation on the two tables, and the result is to retain only the relevant content of the existing part of the left table. In this way, the amount of data processed every day can be basically consistent, thereby avoiding the risk of operation failure caused by the unlimited increase of processing data reaching the bottleneck point of technical processing data volume; at the same time, under the premise of ensuring that the operation result is completely consistent with the result of processing the incremental data table of the master data and using all the full tables for data processing, the required data resources are greatly reduced, the operation process is simplified, and the operation efficiency is improved.

[0054] By performing data processing based on the master data method introduced in the embodiment of the present invention, the amount of data processed and the number of data records are greatly reduced, the operation time efficiency is greatly improved, and the improvement effect is very significant.

[0055] Figure 4 Schematic diagram of the main modules of the data processing device according to the embodiment of the present invention. Figure 4 As shown, the data processing device 400 according to the embodiment of the present invention mainly includes a first processing module 401 and a second processing module 402 .

[0056] The first processing module 401 is used to perform an inner join operation on the main data table and the data table to be associated to obtain an associated main data table;

[0057] The second processing module 402 is used to perform a left join operation on the incremental data table of the main data and the associated main data table to obtain a target data table for data processing.

[0058] According to an embodiment of the present invention, the second processing module 402 may also be used for:

[0059] The incremental data table of the main data is used as the main table, and the associated main data table is used as the auxiliary table. The incremental data table of the main data and the associated main data table are left-joined to obtain the target data table.

[0060] According to the technical solution of the embodiment of the present invention, the main data table and the data table to be associated are full data tables.

[0061] According to another embodiment of the present invention, the master data is obtained by screening the business data according to business requirements and filtering out invalid data.

[0062] According to the technical solution of the embodiment of the present invention, by performing an inner join operation on the main data table and the data table to be associated to obtain the associated main data table; performing a left join operation on the incremental data table of the main data and the associated main data table to obtain the target data table for data processing, the number of data records involved in the operation and the file size can be greatly reduced, thereby greatly saving computing resources and reducing the computing cost of the enterprise; after the amount of data to be processed is reduced, the internal processing steps are reduced, the processing efficiency is improved, and the timeliness is greatly improved; in addition, it can also ensure that the amount of data processed every day is basically the same, the resources required to be consumed remain stable, and the potential risk of operation failure due to the daily increase in resource consumption is avoided.

[0063] Figure 5 An exemplary system architecture 500 is shown to which the data processing method or data processing apparatus according to the embodiment of the present invention can be applied.

[0064] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0065] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0066] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0067] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0068] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data processing device is generally set in the server 505 .

[0069] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0070] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 6 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0071] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0072] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0073] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0074] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0075] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0076] The units or modules involved in the embodiments of the present invention may be implemented by software or hardware. The units or modules described may also be arranged in a processor, for example, they may be described as: a processor includes a first processing module and a second processing module. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves, for example, the first processing module may also be described as "a module for performing an inner join operation on a master data table and a data table to be associated to obtain an associated master data table".

[0077] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: performing an inner join operation on the main data table and the data table to be associated to obtain an associated main data table; performing a left join operation on the incremental data table of the main data and the associated main data table to obtain a target data table for data processing.

[0078] According to the technical solution of the embodiment of the present invention, by performing an inner join operation on the main data table and the data table to be associated to obtain the associated main data table; performing a left join operation on the incremental data table of the main data and the associated main data table to obtain the target data table for data processing, the number of data records involved in the operation and the file size can be greatly reduced, thereby greatly saving computing resources and reducing the computing cost of the enterprise; after the amount of data to be processed is reduced, the internal processing steps are reduced, the processing efficiency is improved, and the timeliness is greatly improved; in addition, it can also ensure that the amount of data processed every day is basically the same, the resources required to be consumed remain stable, and the potential risk of operation failure due to the daily increase in resource consumption is avoided.

[0079] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data processing, characterized in that: include: Perform an inner join operation on the master data table and the data table to be associated to obtain an associated master data table; the master data table and the data table to be associated are full data tables, and the master data stored in the master data table is data to be shared between various systems; Performing a left join operation on the incremental data table of the master data and the associated master data table to obtain a target data table for data processing; The method of performing a left join operation on the incremental data table of the main data and the associated main data table to obtain a target data table includes: taking the incremental data table of the main data as the main table, the associated main data table as the auxiliary table, and performing a left join operation on the incremental data table of the main data and the associated main data table to obtain the target data table.

2. The method according to claim 1, characterized in that The master data is obtained by screening business data according to business requirements and filtering out invalid data.

3. A data processing device, characterized in that: include: The first processing module is used to perform an inner join operation on the master data table and the data table to be associated to obtain an associated master data table; the master data table and the data table to be associated are full data tables, and the master data stored in the master data table is data to be shared among various systems; A second processing module is used to perform a left join operation on the incremental data table of the master data and the associated master data table to obtain a target data table for data processing; The second processing module is also used to: use the incremental data table of the main data as the main table, the associated main data table as the auxiliary table, and perform a left join operation on the incremental data table of the main data and the associated main data table to obtain a target data table.

4. The device according to claim 3, characterized in that The master data is obtained by screening business data according to business requirements and filtering out invalid data.

5. An electronic device for data processing, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in claim 1 or 2.

6. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to claim 1 or 2 is implemented.

Images