A data processing method and apparatus

Abstract

Embodiments of the present application disclose a data processing method and device, each source data table associated with a to-be-processed task is acquired, each source data table is split according to a participant to which each source data belongs, and an intermediate data table corresponding to each participant is obtained, a to-be-mapped intermediate data table corresponding to the to-be-processed task is determined from the intermediate data table corresponding to each participant, the to-be-mapped intermediate data table is mapped using a mapping template corresponding to the to-be-processed task, and a data file corresponding to the to-be-processed task is obtained. By setting the mapping template corresponding to the to-be-processed task, the mapping template can be directly called to map the to-be-mapped intermediate data table into a data source file, without the need to customize and develop a corresponding initial data table for each participant, so that the operation is simpler and the data processing efficiency is higher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Fintech, and in particular to a data processing method and device. BACKGROUND

[0002] With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to Fintech. However, due to the safety and real-time requirements of the financial industry, the financial industry also puts forward higher requirements for technology. In the financial industry, multiple participants can jointly complete the same business. If a participant wants to combine the business data in other participants for modeling and other operations, it usually needs to obtain the business data in other participants. However, the business data in each participant exists in different storage forms, and how to accurately obtain the business data that meets the needs of the participants is extremely important for the modeling operation of the participants.

[0003] At present, the method of custom development is usually used to obtain the business data that meets the needs of the participants, such as setting an initial data table corresponding to the participant according to the needs of the participant, traversing the business data of each participant to obtain the business data required by the participant, placing the business data in the initial data table, obtaining a target data table and sending it to the participant. However, the problem of this method is that the method of custom development needs to set an initial data table corresponding to each participant, and needs to perform the operation of traversing the business data of each participant once for each participant, which is relatively complex and has low data processing efficiency.

[0004] In summary, there is an urgent need for a data processing method to solve the technical problem of low data processing efficiency caused by the method of custom development used in the prior art to obtain the target data table of each participant. SUMMARY

[0005] The embodiments of the present application provide a data processing method and device to solve the technical problem of low data processing efficiency caused by the method of custom development used in the prior art to obtain the target data table of each participant.

[0006] In a first aspect, the embodiments of the present application provide a data processing method, comprising:

[0007] The method comprises the following steps: obtaining each source data table associated with a to-be-processed task, splitting each source data table according to a participant to which each source data belongs, obtaining an intermediate data table corresponding to each participant, determining a to-be-mapped intermediate data table corresponding to the to-be-processed task from the intermediate data table corresponding to each participant, mapping the to-be-mapped intermediate data table by using a mapping template corresponding to the to-be-processed task, and obtaining a data file corresponding to the to-be-processed task; the data file is used for a participant corresponding to the to-be-processed task to parse a target data table meeting a requirement of the participant.

[0008] In the embodiment of the application, the mapping template corresponding to the to-be-processed task is set, the mapping template is directly called to map the to-be-mapped intermediate data table into a data source file, and a corresponding initial data table is not developed for each participant, so that the operation is simpler, and the efficiency of data processing is higher; and the intermediate data table corresponding to each participant is obtained by splitting each source data table, the intermediate data table of the corresponding participant is directly obtained according to the mapping relationship of the to-be-processed task for mapping operation, and the traversal operation of each source data table is not performed for each participant, so that useless resource consumption is reduced, and the efficiency of data processing is further improved.

[0009] In a possible implementation, the method is applied to a data processing device, and the data processing device is connected with each application server; the obtaining of each source data table associated with the to-be-processed task comprises: determining each target application server performing a business corresponding to the to-be-processed task from each application server according to business association of each application server, and obtaining the business corresponding to each source data table from each target application server.

[0010] In the implementation, the data processing device is used to perform the data processing operation, and the application server is not used to perform the data processing operation, so that resource consumption of the application server is reduced, the normal application business is decoupled from the data processing process, the efficiency of data processing is improved, and the influence of data processing on the normal application business is reduced.

[0011] In a possible implementation, the splitting of each source data table according to a participant to which each source data belongs comprises: constructing a general intermediate table corresponding to a type by using source data belonging to the same type in each source data table; the general intermediate table corresponding to any type is used to store source data of each participant under the type; further, source data belonging to any participant is split from the general intermediate table corresponding to any type, and an intermediate data table of the participant under the type is constructed by using the source data of the participant.

[0012] In a possible implementation, the source data belonging to the participant is split from the universal intermediate table corresponding to any type, and the intermediate data table of the participant under the type is constructed using the source data of the participant, including: when any source data is split, the participant to which the source data belongs is determined, if the partition of the participant under the type exists, the source data is placed in the partition of the participant under the type, if the partition of the participant under the type does not exist, the partition of the participant under the type is first created, and then the source data is placed in the partition of the participant under the type.

[0013] In the implementation, the intermediate data table of each participant is split from the universal intermediate table by using the dynamic partitioning, so that the intermediate data table corresponding to each participant can be obtained through one iteration operation on the universal intermediate table, thereby reducing resource consumption and development cost, and improving the efficiency of data processing.

[0014] In a possible implementation, the mapping of the intermediate data table to be mapped is performed using the mapping template corresponding to the task to be processed, including: when the intermediate data table to be mapped includes one, if the dictionary mapping is included in the mapping template, the original field of the dictionary mapping is obtained from the intermediate data table to be mapped, and the original field is mapped to the updated field of the dictionary mapping; correspondingly, if the field mapping is included in the mapping template, the field located at the original position of the field mapping is obtained from the intermediate data table to be mapped, and the position of the field is mapped to the updated position of the field mapping; correspondingly, if the type mapping is included in the mapping template, the field corresponding to the original type of the type mapping is obtained from the intermediate data table to be mapped, and the type of the field is mapped to the updated type of the type mapping.

[0015] In the implementation, the mapping manner in the mapping template is set in advance, the mapping manner in the mapping template can be directly called to map the intermediate data table to be mapped, and the data file is obtained, without creating a mapping data table and distributing the mapping data table, thereby saving the memory occupation, ensuring the mapping when the data is exported, and reducing the memory consumption of the system.

[0016] In a possible implementation, the mapping of the intermediate data table to be mapped using the mapping template corresponding to the task to be processed comprises: when the intermediate data table to be mapped comprises two or more, merging the intermediate data tables to be mapped to obtain a mapping view table, and then mapping the mapping view table using the mapping template corresponding to the task to be processed; wherein the mapping view table comprises data in each intermediate data table to be mapped.

[0017] In the implementation, when there are multiple intermediate data tables to be mapped, the memory occupation can be saved and the system loss can be reduced by setting the mapping view table instead of an actual logical table; and the operation flexibility can be improved by establishing the association relationship for the multiple intermediate data tables to be mapped, so that the intermediate data tables to be mapped can be directly processed according to the association relationship.

[0018] In a second aspect, an embodiment of the present application provides a data processing apparatus, which comprises:

[0019] The acquisition module is configured to acquire each source data table associated with a task to be processed.

[0020] The splitting module is configured to split each source data table according to a participant to which each source data in the source data table belongs, to obtain an intermediate data table corresponding to each participant.

[0021] The mapping module is configured to determine an intermediate data table to be mapped corresponding to the task to be processed from the intermediate data tables corresponding to each participant, map the intermediate data table to be mapped using a mapping template corresponding to the task to be processed, and obtain a data file corresponding to the task to be processed; the data file is used by a participant corresponding to the task to be processed to parse a target data table meeting a requirement of the participant.

[0022] In a possible implementation, the acquisition module is specifically configured to determine, according to business association of each application server, each target application server performing a business corresponding to the task to be processed from the application servers, and acquire the source data tables corresponding to the business from the target application servers.

[0023] In a possible implementation, the splitting module is specifically configured to construct a general intermediate table corresponding to a type using source data belonging to the same type in the source data tables; the general intermediate table corresponding to any type is used to store source data of each participant under the type; further, for any participant, source data belonging to the participant is split from the general intermediate table corresponding to any type, and an intermediate data table of the participant under the type is constructed using the source data of the participant.

[0024] In a possible implementation, the splitting module is specifically configured to: when splitting any source data, determine a participant to which the source data belongs, if a partition of the participant under the type already exists, place the source data in the partition of the participant under the type, and if the partition of the participant under the type does not exist, first create the partition of the participant under the type, and then place the source data in the partition of the participant under the type.

[0025] In a possible implementation, the mapping module is specifically configured to: when the intermediate data table to be mapped includes one, if a dictionary mapping is included in the mapping template, obtain an original field of the dictionary mapping from the intermediate data table to be mapped, and map the original field to an updated field of the dictionary mapping; correspondingly, if a field mapping is included in the mapping template, obtain a field located at an original position of the field mapping from the intermediate data table to be mapped, and map a position of the field to an updated position of the field mapping; correspondingly, if a type mapping is included in the mapping template, obtain a field corresponding to an original type of the type mapping from the intermediate data table to be mapped, and map a type of the field to an updated type of the type mapping.

[0026] In a possible implementation, the mapping module is specifically configured to: when the intermediate data tables to be mapped include two or more, first merge the intermediate data tables to be mapped to obtain a mapping view table, and then map the mapping view table using the mapping template corresponding to the to-be-processed task; the mapping view table includes data in each intermediate data table to be mapped.

[0027] In a third aspect, an embodiment of the present application provides a computing device, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes the data processing method in any of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a computing device, and when the program runs on the computing device, the computing device executes the data processing method in any of the first aspect.

[0029] These aspects and other aspects of the present application will be more apparent in the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without creative labor.

[0031] Figure 1 A possible system architecture schematic diagram provided for the embodiments of the present application is shown in FIG. 1.

[0032] Figure 2 A flowchart corresponding to the data processing method provided for the embodiments of the present application is shown in FIG. 2.

[0033] Figure 3 A mapping manner schematic diagram provided for the embodiments of the present application is shown in FIG. 3.

[0034] Figure 4 A structure schematic diagram of the data processing apparatus provided for the embodiments of the present application is shown in FIG. 4.

[0035] Figure 5 A structure schematic diagram of the computing device provided for the embodiments of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments only constitute some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0037] Figure 1 A possible system architecture schematic diagram provided for the embodiments of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the system architecture includes a data processing device 110 and at least one application server, such as an application server 121, an application server 122 and an application server 123. The data processing device 110 can be connected with any application server, such as through wired connection or through wireless connection, which is not limited in particular.

[0038] In the embodiments of the present application, each application server refers to an application server participating in the same joint service, each application server is responsible for completing part of the service operation in the joint service, and the service data of each application server includes the service data of each participant in the service operation corresponding to the application server.

[0039] For example, in a joint loan business, a plurality of banks jointly provide a joint loan to a borrowing institution. If the auditing task, the transaction task and the risk control task in the joint loan business are executed by the application server 121, the application server 122 and the application server 123 respectively, the application server 121 is configured to audit the loan application of the borrowing institution, and the business data of the application server 121 includes the relevant information and the loan auditing information of the borrowing institution, such as the name, the address, the credit rating, the legal person and the asset amount of the borrowing institution. Correspondingly, the application server 122 is configured to execute the transaction operation with the borrowing institution, and the business data of the application server 122 can include the transaction data of the plurality of banks and the borrowing institution, such as the number of loans provided by the plurality of banks to the borrowing institution, the loan amount of each time, the loan time and the loan proportion. Correspondingly, the application server 123 is configured to manage the repayment operation of the borrowing institution, and the business data of the application server 123 can include the collection data of the plurality of banks, such as the amount of the repayment of the borrowing institution to each bank, the overdue repayment time length, the overdue repayment amount, the collection call frequency and the repayment intention.

[0040] It should be noted that, Figure 1 is only an example and does not constitute a limitation on the present solution. In specific implementation, the data processing device 110 can refer to a data processing server, or a process in the data processing server, and the specific implementation is not limited.

[0041] In the embodiment of the present application, the data processing method can be executed by any application server or the data processing device 110. However, if the application server is used to execute the data processing operation, the application server needs to obtain the source data table from other application servers respectively, and then distribute to each participant after association splitting, which leads to high communication consumption and resource consumption between the application servers.

[0042] Therefore, in the embodiment of the present application, the data processing device 110 is used to execute the data processing operation, so as to decouple the normal application business and the data processing process in the application server, avoid the influence of the data processing operation on the normal application business, and reduce the communication consumption between the application servers and improve the efficiency of the data processing.

[0043] Based on Figure 1 the system architecture shown in the figure, Figure 2 a flow diagram of a data processing method provided by the embodiment of the present application, the method is applicable to the data processing device 110, and the method comprises the following steps:

[0044] In step 201, each source data table associated with a to-be-processed task is obtained.

[0045] In the embodiment of the present application, the data processing device 110 can acquire the source data tables associated with the to-be-processed task in various ways. For example, the data processing device 110 can first determine the application servers associated with the to-be-processed task, and then send an acquisition instruction to the associated application servers to acquire the source data tables reported by the associated application servers, or the data processing device 110 can receive the source data tables reported by the application servers, and then acquire the source data tables associated with the to-be-processed task from the source data tables, without any limitation.

[0046] In a possible implementation, the data processing device 110 can acquire the source data tables associated with the to-be-processed task by the following steps:

[0047] In step a, the data processing device 110 acquires the business source data tables reported by the application servers.

[0048] In a specific implementation, a data reporting task can be configured in any application server, and the trigger condition of the data reporting task can be the completion of a batch task. In this way, the application server can report the business source data table corresponding to the batch task after completing the batch task, so that the data processing device 110 can perform data processing operations in batches, thereby improving the unity and integrity of data processing. Alternatively, the trigger condition of the data reporting task can be a fixed period, so that the application server can report the business source data table generated in the fixed period, thereby improving the orderliness of data processing. Alternatively, the trigger condition of the data reporting task can also be business data change, so that the application server can report the business data corresponding to the task after completing the task, thereby improving the real-time performance and accuracy of data processing.

[0049] In an example, the application server and the data processing device 110 can also be deployed with an Extract-Transform-Load (ETL) component, and the ETL component is built-in with a metadata function class function. The metadata function class function is used to store the mapping relationship between the source end and the destination end of the business source data table and the transformation rule. In addition, the ETL component can provide a powerful connection function to connect the source end and the destination end of the business source data table, and perform extraction, transformation and loading operations on the business source data table at the source end through the metadata function class function. In the case where the business source data table is compatible with the destination end, the business source data table is sent from the source end to the destination end, so that the destination end acquires the business source data table consistent with the source end.

[0050] In the above example, by embedding the ETL component in the application server and the data processing device 110, the developer no longer needs to re-write the script file of data transmission, and does not need to have deep development capability, thereby reducing the development difficulty and improving the generality of data processing.

[0051] In the embodiment of the application, the data processing device 110 can be provided with a scheduling platform and a big data platform, the application server is connected with the big data platform via the scheduling platform, the scheduling platform is provided with an ETL component, the scheduling platform can transmit the business source data table of the application server to the big data platform through the ETL component, and the big data platform can be scheduled to execute the to-be-processed task when the to-be-processed task is triggered. Correspondingly, the big data platform is used to manage the business source data table reported by each application server, and can associate, split and map the business source data table under the scheduling of the scheduling platform.

[0052] The type of the big data platform can be set by the person skilled in the art according to experience, for example, can be set as any one of Hadoop platform, Spark platform, Storm platform, Apache Drill platform and Rapid Miner platform, and the specific type is not limited.

[0053] In another example, the data reporting task of each application server can also be uniformly managed by the scheduling platform, for example, when the scheduling platform detects that the batch task processing in an application server is completed, the scheduling platform can send a reporting instruction to the application server, thereby triggering the application server to report the business source data table of the batch task to the scheduling platform, so as to transmit the business source data table to the big data platform via the ETL component of the scheduling platform.

[0054] It should be noted that the above is only an example of simple description, and does not constitute a limitation on the scheme, in the specific implementation, the scheduling platform can be a software development kit (Software Development Kit, SDK) component deployed in the data processing device 110, or can be a device independent of the data processing device 110, and the specific type is not limited.

[0055] Step b, according to the business association of each application server, determining each target application server for executing the to-be-processed task from each application server.

[0056] In one example, the data processing device 110 can take each application server performing the business corresponding to the to-be-processed task as a target application server. For example, if the first participant, the second participant and the third participant jointly perform a joint loan business, and the first participant and the fourth participant jointly perform a joint credit card business, the data processing device 110 can take the first participant, the second participant and the third participant as target application servers when the to-be-processed task corresponds to the joint loan business, and take the first participant and the fourth participant as target application servers when the to-be-processed task corresponds to the joint credit card business.

[0057] In another example, the data processing device 110 can also take the user-configured to-be-associated application server as a target application server. As shown in the figure, the data processing device 110 can also be connected with the client 130, and the data processing device 110 can previously provide a task configuration interface to the client 130 and display the task configuration interface to the user via the client 130, so that the user inputs the configuration information of the to-be-processed task on the task configuration interface. Accordingly, after obtaining the configuration information input by the user, the data processing device 110 can also send the to-be-processed task and the configuration information to the scheduling platform, so that the scheduling platform maintains each to-be-processed task. Figure 1

[0058] In a specific implementation, the data processing device 110 can support the user to customize the processing subtask, the splitting subtask and the distribution subtask corresponding to the to-be-processed task, and the user can input the configuration information of the processing subtask, the configuration information of the splitting subtask and the configuration information of the distribution subtask on the task configuration interface. The processing subtask is used to perform an association operation on the to-be-processed data of the user-configured to-be-associated application server to obtain a general intermediate table, the splitting subtask is used to perform a splitting operation on the general intermediate table to obtain an intermediate data table corresponding to the user-configured splitting participant, and the distribution subtask is used to generate a data file corresponding to the splitting participant according to the intermediate data table corresponding to the splitting participant and the user-configured distribution rule, and send the data file to the splitting participant.

[0059] In the embodiment of the application, the configuration information of the processing subtask can include the to-be-associated application server and the trigger condition of the processing subtask. The to-be-associated application server can be set by the user according to the to-be-processed task. For example, if a to-be-processed task is to send an application form in the second participant to the first participant at 12:00 every day, the user can set the first participant and the second participant as the to-be-associated application server, and set the trigger condition as 12:00 every day. In this way, for this processing subtask, if the scheduling platform determines that the current time is 12:00, the big data platform can be scheduled to perform an association and splitting operation on the business source data table reported by the first participant and the business source data table reported by the second participant.​

[0060] In the above example, by supporting user customization of the to-be-processed task, the data processing operation can be made to meet the actual business scenario requirement more, and the accuracy of data processing and user satisfaction are improved.

[0061] Step c, the business source data table reported by each target application server is taken as each source data table associated with the to-be-processed task.

[0062] In the embodiment of the application, each to-be-processed task configured by the user can be uniformly managed by the scheduling platform, and different to-be-processed tasks can correspond to the same processing subtask, the same splitting subtask and different distribution subtasks.

[0063] As an example, Table 1 is a schematic table of a to-be-processed task provided by the embodiment of the application.

[0064] Table 1: Schematic of a to-be-processed task

[0065]

[0066] As shown in Table 1, the to-be-processed task 1, the to-be-processed task 2 and the to-be-processed task 3 correspond to the same processing subtask and the same splitting subtask, but correspond to different distribution subtasks. Among them, the processing subtasks corresponding to the three to-be-processed tasks are all associated with the application server 121, the application server 122 and the application server 123, that is, the general intermediate table is constructed according to the source data table of the application server 121, the source data table of the application server 122 and the source data table of the application server 123; the splitting subtasks corresponding to the three to-be-processed tasks are all splitting the first participant, the second participant, the third participant and the fourth participant, that is, the first participant's intermediate data table, the second participant's intermediate data table, the third participant's intermediate data table and the fourth participant's intermediate data table are respectively split from the general intermediate table.

[0067] Correspondingly, the distribution subtask of the to-be-processed task 1 is to distribute the source data table of the first participant to the second participant, that is, to convert the first participant's intermediate data table into a data file meeting the format requirement of the second participant, and then distribute it to the second participant; the distribution subtask of the to-be-processed task 2 is to distribute the source data table of the second participant to the fourth participant, that is, to convert the second participant's intermediate data table into a data file meeting the format requirement of the fourth participant, and then distribute it to the fourth participant; the distribution subtask of the to-be-processed task 3 is to distribute the source data table of the third participant to the second participant, that is, to convert the third participant's intermediate data table into a data file meeting the format requirement of the second participant, and then distribute it to the third participant.

[0068] It should be noted that Table 1 is only an exemplary simple description, and the listed to-be-processed tasks are only for facilitating the description of the scheme and do not constitute a limitation on the scheme. In a specific implementation, one to-be-processed task can also include multiple distribution sub-tasks at the same time, such as to-be-processed task 1, to-be-processed task 2, and to-be-processed task 3 as one to-be-processed task.

[0069] In a specific implementation, for any to-be-processed task, the scheduling platform can monitor whether the triggering condition of the processing sub-task in the to-be-processed task is met at the current time in real time. If the triggering condition is met, the scheduling platform can send a task processing instruction to the big data platform, and carry the identifier of the to-be-associated application server in the task processing instruction. Correspondingly, after receiving the task processing instruction, the big data platform can obtain the business source data table reported by the to-be-associated application server from the business source data table reported by each application server according to the identifier of the to-be-associated application server, as the to-be-associated source data table corresponding to the to-be-processed task.

[0070] In step 202, the plurality of source data tables are split according to the participants to which each source data in the plurality of source data tables belongs, to obtain the intermediate data table corresponding to each participant.

[0071] In a possible implementation, the data processing device 110 can obtain the intermediate data table corresponding to each participant by the following steps:

[0072] In step one, a common intermediate table corresponding to each type is constructed using the source data of the same type in each source data table. The common intermediate table corresponding to any type is used to store the source data of each participant under the type.

[0073] In the embodiment of the application, the big data platform can be provided with a data table set corresponding to each business scenario, and the data table set stores each type of data table format related to the business scenario. For example, in the joint loan business, the data table set can store each type of data table format related to the loan, such as customer table format, account table format, application table format, plan table format, loan receipt table format, and collection table format. The data table format is used to indicate the row and column information of the data table. For example, in the account table format, each row of the account table is used to indicate each account, and each column of the account table is the name, age, loan amount, loan time, and the like arranged in sequence by column.

[0074] Accordingly, after determining the to-be-associated source data tables, the data processing device 110 can first determine the business scenarios to which the source data tables belong, and then generate each initial intermediate table according to the data table format of each data table in the data table set corresponding to the business scenario, each initial intermediate table indicating a type. Further, for each source data in any source data table to be associated, the data processing device 110 can place the source data in the initial intermediate table of the corresponding type. In this way, when the traversal of each source data table to be associated is completed, the data processing device 110 can generate a plurality of general intermediate tables, and each general intermediate table includes source data of each participant under the same type.

[0075] In the above example, by setting various types of data table formats corresponding to business scenarios, the general intermediate table can better meet the needs of the business scenario, thereby being more adaptable.

[0076] Step two, for any participant, the source data belonging to the participant is split from the general intermediate table corresponding to any type, and the source data of the participant is used to construct an intermediate data table of the participant under the type.

[0077] In one example, the data processing device 110 can automatically split each participant in the general intermediate table. In a specific implementation, the data processing device 110 can split each source data in the general intermediate table corresponding to each type in order. When splitting any source data, the data processing device 110 can first determine the participant to which the source data belongs, and then determine whether a partition of the participant under the type already exists. If it exists, the source data can be directly placed in the partition of the participant under the type. If it does not exist, the data processing device 110 can first create a partition of the participant under the type, and then place the source data in the partition of the participant under the type.

[0078] In this way, when the traversal of the source data in the general intermediate table corresponding to any type is completed, the data processing device 110 can obtain a partition of each participant under the type, and the partition is an intermediate data table of the participant under the type. Further, the intermediate data table of each participant under the type is consistent with the format of the general intermediate table corresponding to the type.

[0079] The prior art needs to traverse the data tables in each application server once for each participant, while the above example can obtain the intermediate data table of each participant through a general traversal process, without the need to traverse multiple times for each participant. Compared with the prior art, the present scheme can reduce resource consumption, reduce development cost, and improve data processing efficiency by using dynamic partitioning to split the intermediate data table of each participant from the general intermediate table.

[0080] In one example, the data processing device 110 can also split according to the split participant configured by the user. In a specific implementation, the user can set the split participant corresponding to the split subtask in advance in the task configuration interface. Thus, when the data processing device 110 splits each source data in the general intermediate table in sequence, if it is determined that the participant to which the source data belongs is not the split participant, the data processing device 110 can directly jump to the next source data. If the participant corresponding to the source data belongs to the split participant, the data processing device 110 can split according to the split manner described above.

[0081] In this example, the data processing device can only split the intermediate data table of the split participant configured by the user, and no longer split the intermediate data table of other participants, thereby saving useless operation steps, reducing memory occupation, and improving the performance of the system.

[0082] In step 203, the data processing device 110 determines the intermediate data table to be mapped corresponding to the to-be-processed task from the intermediate data table corresponding to each participant, maps the intermediate data table to be mapped using the mapping template corresponding to the to-be-processed task, and obtains the data file corresponding to the to-be-processed task. The data file is used by the participant corresponding to the to-be-processed task to parse the target data table meeting the requirements of the participant.

[0083] In a specific implementation, the data processing device 110 can be provided with a mapping template library, which is used to store the mapping template corresponding to each participant. The participant and the mapping template can be matched based on the identifier of the participant. Thus, for any distribution subtask corresponding to a participant, when the data processing device 110 obtains the intermediate data table to be mapped corresponding to the distribution subtask, the data processing device 110 can obtain the mapping template matching the identifier of the participant from the mapping template library, and use the matching mapping template as the mapping template corresponding to the distribution task.

[0084] In one possible implementation, if the intermediate data table to be mapped includes only one, the data processing device 110 can directly call the mapping template corresponding to the to-be-processed task to map the intermediate data table to be mapped. If the intermediate data table to be mapped includes two or more, the data processing device 110 can first combine the intermediate data tables to be mapped to obtain a mapping view table, and then call the mapping template corresponding to the to-be-processed task to map the mapping view table. The mapping view table is a view table and does not belong to the logical table in the actual sense. The mapping view table includes all the data in each intermediate data table to be mapped.

[0085] In this embodiment of the invention, the mapping template can be a mapping function, such as a user-defined function (UDF). The mapping template can be set based on the mapping requirements of the participants. The mapping template includes various types of mappings, such as field mapping, type mapping, and dictionary mapping. Field mapping refers to mapping a field located at one position in an intermediate data table to another position. Type mapping refers to mapping a field type in an intermediate data table to another field type, such as mapping the field type "string" (string type) to "bigint" (integer type) and the character field "long" (long integer type) to "float" (floating-point type). Dictionary mapping refers to mapping a field in an intermediate data table to another field, such as mapping "customer 001" to "customer A" and "customer 002" to "customer B".

[0086] In practice, if the mapping template corresponding to the task to be processed includes dictionary mapping, the original fields of the dictionary mapping can be obtained from the intermediate data table or mapping view table to be mapped, and the original fields can be mapped to the updated fields of the dictionary mapping; if the mapping template includes field mapping, the fields located at the original positions of the field mapping can be obtained from the intermediate data table or mapping view table to be mapped, and the positions of these fields can be mapped to the updated positions of the field mapping; if the mapping template includes type mapping, the fields corresponding to the original types of the type mapping can be obtained from the intermediate data table or mapping view table to be mapped, and the types of these fields can be mapped to the updated types of the type mapping.

[0087] It should be noted that if the mapping template includes two or more of the following mapping types: field mapping, type mapping, and dictionary mapping, the data processing device can perform multiple mappings in any order without limitation.

[0088] To make it easier to understand, a specific example is given below.

[0089] The first institution and the second institution conduct joint lending business. The first institution's business source data table on day T includes customer table, account table, application table, plan table, loan receipt table, and collection table.

[0090] When performing data modeling, the second agency needs to obtain the first agency's business source data table on day T, and the requirements are as follows:

[0091] (1) The difference between the account table and the account table of the first institution is that the account table needs the identity identifier field from the customer table of the first institution;

[0092] 2) the loan table, the difference between the loan table of the first institution is that the loan table needs to be marked whether it is overdue and the collection identification dictionary is inconsistent with the collection identification dictionary in the collection table of the first institution;

[0093] 4) the application table, the difference between the application table of the first institution is that the order of each field in the application table is different, and the type of the field is also different.

[0094] In a specific implementation, the data processing device 110 can first obtain each source data table associated with the to-be-processed task, i.e., the account table, the customer table, the loan table, the collection table, and the application table of the first institution on the T-th day, and can generate a data file corresponding to each table in the following manner:

[0095] For the data file corresponding to the account table of the second institution, since the account table of the second institution contains the contents in the account table and the customer table of the first institution, the intermediate data table to be mapped corresponding to the account table of the second institution includes the account table and the customer table (two) of the first institution. In this way, the data processing device can first construct a first mapping view table based on the account table and the customer table of the first institution, and then call the mapping template corresponding to the to-be-processed task to map the first mapping view table to obtain the data file corresponding to the account table of the second institution; wherein the first mapping view table includes all data in the account table and the customer table of the first institution.

[0096] And, since the account table of the second institution also includes the identity identification field in the customer table of the first institution, the mapping template can use the field mapping to generate the data file corresponding to the account table, the field mapping is used to extract the identity identification field and each field in the account table of the first institution from the first mapping view table, and then combine each field according to the order of each field in the account table of the second institution.

[0097] For the data file corresponding to the loan table of the second institution, since the loan table of the second institution contains the contents in the loan table and the collection table of the first institution, the intermediate data table to be mapped corresponding to the loan table of the second institution includes the loan table and the collection table (two) of the first institution. In this way, the data processing device can first construct a second mapping view table based on the loan table and the collection table of the first institution, and then call the mapping template corresponding to the to-be-processed task to map the second mapping view table to obtain the data file corresponding to the loan table of the second institution; wherein the second mapping view table includes all data in the loan table and the collection table of the first institution.

[0098] And, since the collection identifier dictionary in the loan table of the second institution is inconsistent with the collection identifier dictionary in the collection table of the first institution, the mapping template can also use dictionary mapping to map the collection identifier in the second mapping view table to the collection identifier of the second institution; correspondingly, since the overdue collection status is also included in the loan table of the second institution, the mapping template can also use field mapping to extract the expected collection status field from the second mapping view table and each field in the loan table of the first institution, and then combine the above-mentioned fields according to the order of each field in the loan table of the second institution.

[0099] For the data file corresponding to the application table of the second institution, since the loan table of the second institution only contains the content of the application table of the first institution, the intermediate data table to be mapped corresponding to the loan table of the second institution includes the application table (one) of the first institution. In this way, the data processing device can directly call the mapping template corresponding to the to-be-processed task to map the application table of the first institution to obtain the data file corresponding to the application table of the second institution.

[0100] And, since the field order and type in the application table of the second institution are inconsistent with the field order and type in the application table of the first institution, the mapping template can use field mapping to map the order of each field in the application table of the first institution to the order of each field in the application table of the second institution, and can use type mapping to map the type and precision of each field in the application table of the first institution to the type and precision of each field in the application table of the second institution.

[0101] In the above example, the first mapping view table and the second mapping view table exist in the form of views and are not stored in the form of logical tables; correspondingly, the dictionary mapping, the field mapping and the type mapping are all executed in the distribution subtask stage, that is, the data processing device can directly map each intermediate data table to be mapped to a data file and send it to the second institution, without the need to generate a target data table again, thereby reducing resource consumption and improving the efficiency of data processing.

[0102] Figure 3 A mapping flow diagram provided by an embodiment of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, the mapping process maps the source data table 1 and the source data table 2 to a data file, and the mapping process includes field mapping, dictionary mapping and type mapping.

[0103] In a specific implementation, since the source data tables to be mapped include two, source data table 1 and source data table 2 can be combined to obtain a mapping view table, which includes all data in source data table 1 and all data in source data table 2. Further, field 1 of source data table 2 in the mapping view table is mapped to the first position of the data file, field 2 of source data table 2 in the mapping view table is mapped to the third position of the data file, and the positions of field B and field D of source data table 1 in the mapping view table remain unchanged. In this way, dictionary mapping and / or type mapping can also be performed on field 1, field B, field 2 and field D to directly obtain the data file by mapping in the export stage, without generating a target data table.

[0104] In the above embodiment of the present application, each source data table associated with a to-be-processed task is obtained, each source data table is split according to a participant to which each source data in the source data table belongs, to obtain an intermediate data table corresponding to each participant, a to-be-mapped intermediate data table corresponding to the to-be-processed task is determined from the intermediate data table corresponding to each participant, the to-be-mapped intermediate data table is mapped using a mapping template corresponding to the to-be-processed task, and a data file corresponding to the to-be-processed task is obtained. The data file is used by a participant corresponding to the to-be-processed task to parse a target data table that meets the requirements of the participant. In the embodiment of the present application, by setting the mapping template corresponding to the to-be-processed task, the mapping template can be directly called to map the to-be-mapped intermediate data table into a data source file, without the need to develop a corresponding initial data table for each participant, so that the operation is simpler and the efficiency of data processing is higher. Further, by splitting each participant's corresponding intermediate data table from each source data table, the intermediate data table of the corresponding participant can be directly obtained for mapping operation according to the mapping relationship of the to-be-processed task, without the need to perform a traversal operation on each source data table for each participant, thereby reducing useless resource consumption and further improving the efficiency of data processing.

[0105] For the above method flow, an embodiment of the present application further provides a data processing device, and specific contents of the device can be referred to the above method implementation.

[0106] Figure 4 A structural schematic diagram of a data processing device provided by an embodiment of the present application includes:

[0107] The acquisition module 401 is configured to acquire each source data table associated with a to-be-processed task.

[0108] The splitting module 402 is configured to split each source data table according to a participant to which each source data in the source data table belongs, to obtain an intermediate data table corresponding to each participant.

[0109] The mapping module 403 is configured to determine a to-be-mapped intermediate data table corresponding to the to-be-processed task from the intermediate data table corresponding to each participant, map the to-be-mapped intermediate data table using a mapping template corresponding to the to-be-processed task, and obtain a data file corresponding to the to-be-processed task; the data file is used by a participant corresponding to the to-be-processed task to parse a target data table meeting a requirement of the participant.

[0110] Optionally, the acquisition module 401 is specifically configured to:

[0111] acquire a business source data table reported by each application server;

[0112] determine, according to a business correlation of each application server, each target application server that performs a business corresponding to the to-be-processed task from the application servers;

[0113] report, as the source data tables associated with the to-be-processed task, the business source data tables reported by the target application servers.

[0114] Optionally, the splitting module 402 is specifically configured to:

[0115] construct a general intermediate table corresponding to each type using source data belonging to the same type in the source data tables; and each general intermediate table corresponding to a type is configured to store source data of each participant under the type;

[0116] for any participant, split source data belonging to the participant from the general intermediate table corresponding to any type, and construct an intermediate data table of the participant under the type using the source data of the participant.

[0117] Optionally, the splitting module 402 is specifically configured to:

[0118] split each source data in the general intermediate table corresponding to the type in sequence;

[0119] wherein, when any source data is split, a participant to which the source data belongs is determined, if a partition of the participant under the type exists, the source data is placed in the partition of the participant under the type, and if the partition of the participant under the type does not exist, the partition of the participant under the type is first created, and then the source data is placed in the partition of the participant under the type.

[0120] Optionally, the mapping module 403 is specifically configured to:

[0121] When the intermediate data table to be mapped includes one, if a dictionary mapping is included in the mapping template, an original field of the dictionary mapping is obtained from the intermediate data table to be mapped, and the original field is mapped to an updated field of the dictionary mapping; if a field mapping is included in the mapping template, a field located at an original position of the field mapping is obtained from the intermediate data table to be mapped, and a position of the field is mapped to an updated position of the field mapping; if a type mapping is included in the mapping template, a field corresponding to an original type of the type mapping is obtained from the intermediate data table to be mapped, and a type of the field is mapped to an updated type of the type mapping.

[0122] Optionally, the mapping module 403 is specifically configured to:

[0123] When the intermediate data tables to be mapped include two or more, the intermediate data tables to be mapped are first combined to obtain a mapping view table, and then the mapping view table is mapped using the mapping template corresponding to the task to be processed; wherein the mapping view table includes data in each intermediate data table to be mapped.

[0124] As can be seen from the above, in the above embodiments of the present application, each source data table associated with a task to be processed is obtained, each source data in the source data tables is split according to a participant to which the source data belongs, intermediate data tables corresponding to each participant are obtained, intermediate data tables to be mapped corresponding to the task to be processed are determined from the intermediate data tables corresponding to each participant, the intermediate data tables to be mapped are mapped using a mapping template corresponding to the task to be processed, and a data file corresponding to the task to be processed is obtained; the data file is used by a participant corresponding to the task to be processed to parse a target data table meeting a requirement of the participant. In the embodiments of the present application, by setting a mapping template corresponding to a task to be processed, the mapping template can be directly called to map an intermediate data table to be mapped into a data source file, without the need to customize and develop an initial data table for each participant, so that the operation is simpler and the efficiency of data processing is higher; and by splitting each participant corresponding intermediate data tables from each source data table, the intermediate data tables of the corresponding participants can be directly obtained according to a mapping relationship of the task to be processed for mapping operation, without the need to perform a traversal operation on each source data table for each participant, so that useless resource consumption is reduced and the efficiency of data processing is further improved.

[0125] Based on the same inventive concept, the embodiments of the present application also provide a computing device, and based on the same technical concept, the embodiments of the present application provide a computing device, such as Figure 5As shown, the apparatus includes at least one processor 501 and a memory 502 connected to the at least one processor. In the embodiments of the present application, the specific connection medium between the processor 501 and the memory 502 is not limited, Figure 5 For example, the processor 501 and the memory 502 are connected through a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0126] In the embodiments of the present application, the memory 502 stores instructions executable by the at least one processor 501. The at least one processor 501 can execute the steps included in the foregoing data processing method by executing the instructions stored in the memory 502.

[0127] The processor 501 is the control center of the computing device, and can connect various parts of the computing device through various interfaces and lines. The processor 501 can realize data processing by running or executing instructions stored in the memory 502 and calling data stored in the memory 502. Optionally, the processor 501 can include one or more processing units. The processor 501 can integrate an application processor and a modem processor. The application processor mainly processes an operating system, a user interface, an application program, etc. The modem processor mainly processes the issued instructions. It can be understood that the foregoing modem processor can also be not integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 can be implemented on the same chip. In some embodiments, they can also be respectively implemented on independent chips.

[0128] The processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the data processing embodiments can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0129] The memory 502, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 502 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 502 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0130] Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium storing a computer program executable by a computing device, which, when running on the computing device, causes the computing device to execute Figure 2 Any of the data processing methods.

[0131] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0132] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions described in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart

[0133] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart Figure 1 one or more processes and / or blocks Figure 1 one or more blocks or steps

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the flowchart Figure 1 one or more processes and / or blocks Figure 1 one or more blocks or steps

[0135] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such additional variations and modifications as fall within the scope of the application. What is claimed is:

[0136] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve the various source data tables associated with the task to be processed; Based on the participants to which each source data belongs in each source data table, the source data tables are split to obtain intermediate data tables corresponding to each participant; The intermediate data table to be mapped corresponding to the task to be processed is determined from the intermediate data tables corresponding to each participant. The intermediate data table to be mapped is mapped using the mapping template corresponding to the task to be processed to obtain the data file corresponding to the task to be processed. The data file is used by the participant corresponding to the task to be processed to parse the target data table that meets the requirements of the participant. The step of splitting the source data tables according to the participants to which each source data belongs, to obtain intermediate data tables corresponding to each participant, includes: A general intermediate table corresponding to the type is constructed using source data of the same type from each of the source data tables; the general intermediate table corresponding to any type is used to store the source data of each participant under the type. For any participant, source data belonging to the participant is extracted from the general intermediate table corresponding to any type, and the intermediate data table of the participant under the type is constructed using the source data of the participant. The step of mapping the intermediate data table to be mapped using the mapping template corresponding to the task to be processed includes: When there are two or more intermediate data tables to be mapped, the intermediate data tables to be mapped are first merged to obtain a mapping view table, and then the mapping view table is mapped using the mapping template corresponding to the task to be processed; wherein, the mapping view table includes data from each intermediate data table to be mapped.

2. The method according to claim 1, characterized in that, The method is applied to a data processing device, which is connected to various application servers. The step of obtaining the various source data tables associated with the task to be processed includes: Obtain the business source data tables reported by each application server; Based on the business association of each application server, the target application servers that execute the business corresponding to the task to be processed are determined from each application server. The business source data tables reported by each target application server are used as the source data tables associated with the task to be processed.

3. The method according to claim 1, characterized in that, The step of splitting the source data belonging to the participant from the general intermediate table corresponding to any type, and constructing the intermediate data table of the participant under the type using the source data of the participant, includes: Split the source data in the general intermediate table corresponding to the type in sequence; Specifically, when splitting any source data, the participant to which the source data belongs is determined. If a partition of the participant under the type already exists, the source data is placed in the partition of the participant under the type. If a partition of the participant under the type does not exist, the partition of the participant under the type is created first, and then the source data is placed in the partition of the participant under the type.

4. The method according to any one of claims 1 to 3, characterized in that, The step of mapping the intermediate data table to be mapped using the mapping template corresponding to the task to be processed includes: When the intermediate data table to be mapped includes one, if the mapping template includes a dictionary mapping, then the original field of the dictionary mapping is obtained from the intermediate data table to be mapped, and the original field is mapped to the updated field of the dictionary mapping; if the mapping template includes a field mapping, then the field located at the original position of the field mapping is obtained from the intermediate data table to be mapped, and the position of the field is mapped to the updated position of the field mapping; if the mapping template includes a type mapping, then the field corresponding to the original type of the type mapping is obtained from the intermediate data table to be mapped, and the type of the field is mapped to the updated type of the type mapping.

5. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the various source data tables associated with the task to be processed; The splitting module is used to split the source data tables according to the participants to which each source data belongs, and obtain the intermediate data tables corresponding to each participant. The mapping module is used to determine the intermediate data table to be mapped corresponding to the task to be processed from the intermediate data tables corresponding to each participant, and to map the intermediate data table to be mapped using the mapping template corresponding to the task to be processed to obtain the data file corresponding to the task to be processed; the data file is used by the participant corresponding to the task to be processed to parse the target data table that meets the requirements of the participant. Specifically, the splitting module is used for: A general intermediate table corresponding to the type is constructed using source data of the same type from each of the source data tables; the general intermediate table corresponding to any type is used to store the source data of each participant under the type. For any participant, source data belonging to the participant is extracted from the general intermediate table corresponding to any type, and the intermediate data table of the participant under the type is constructed using the source data of the participant. The mapping module is specifically used for: When there are two or more intermediate data tables to be mapped, the intermediate data tables to be mapped are first merged to obtain a mapping view table, and then the mapping view table is mapped using the mapping template corresponding to the task to be processed; wherein, the mapping view table includes data from each intermediate data table to be mapped.

6. The apparatus according to claim 5, characterized in that, The acquisition module is specifically used for: Obtain the business source data tables reported by each application server; Based on the business association of each application server, the target application servers that execute the business corresponding to the task to be processed are determined from each application server. The business source data tables reported by each target application server are used as the source data tables associated with the task to be processed.

7. The apparatus according to claim 5, characterized in that, The splitting module is specifically used for: Split the source data in the general intermediate table corresponding to the type in sequence; Specifically, when splitting any source data, the participant to which the source data belongs is determined. If a partition of the participant under the type already exists, the source data is placed in the partition of the participant under the type. If a partition of the participant under the type does not exist, the partition of the participant under the type is created first, and then the source data is placed in the partition of the participant under the type.

8. The apparatus according to any one of claims 5 to 7, characterized in that, The mapping module is specifically used for: When the intermediate data table to be mapped includes one, if the mapping template includes a dictionary mapping, then the original field of the dictionary mapping is obtained from the intermediate data table to be mapped, and the original field is mapped to the updated field of the dictionary mapping; If the mapping template includes field mapping, then the field located at the original position of the field mapping is obtained from the intermediate data table to be mapped, and the position of the field is mapped to the updated position of the field mapping; If the mapping template includes a type mapping, then the field corresponding to the original type of the type mapping is obtained from the intermediate data table to be mapped, and the type of the field is mapped to the updated type of the type mapping.

9. A computing device, characterized in that, The method includes at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computing device, which, when run on the computing device, causes the computing device to perform the method described in any one of claims 1 to 4.

Images