A federated task statistics processing method and apparatus

Abstract

The application provides a federal task statistical processing method and device, the method comprises the following steps: processing and aggregating a plurality of sub-tasks of a task of a first client to obtain an aggregated processing result of the first client, and storing the aggregated processing result in a preset directory; reading the aggregated processing result of the first client from the preset directory and storing the aggregated processing result in a statistical directory; obtaining a task processing result of a second client; in the statistical directory, based on a pre-set task statistical requirement format, performing encrypted comparison and statistics on the aggregated processing result and the task processing result to obtain a statistical result, and feeding back the statistical result to the first client and the second client, which can solve the problem that federal learning cannot be applied to task statistics in related technologies, reduce the degree of human participation, improve the task statistical efficiency, and also reduce the data leakage risk caused by task comparison.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a federated task statistical processing method and device. BACKGROUND

[0002] The prior art federated learning structure is complex, although it can complete information aggregation and dependency scheduling between tasks, it is biased towards federated scheduling of machine learning models and is not suitable for statistical tasks commonly used in the financial field.

[0003] For the problem that federated learning in the related art cannot be applied to task statistics, no solution has been proposed. SUMMARY

[0004] The embodiments of the present application provide a federated task statistical processing method and device to at least solve the problem that federated learning in the related art cannot be applied to task statistics.

[0005] According to an embodiment of the present application, a federated task statistical processing method is provided, comprising:

[0006] processing and aggregating a plurality of sub-tasks of a task of a first client to obtain an aggregated processing result of the first client, and storing the aggregated processing result in a preset directory;

[0007] reading the aggregated processing result of the first client from the preset directory and storing the aggregated processing result in a statistical directory;

[0008] obtaining a task processing result of a second client;

[0009] In the statistical directory, based on a pre-set task statistical requirement format, the aggregated processing result of the first client and the task processing result of the second client are statistically compared and encrypted to obtain a statistical result, and the statistical result is fed back to the first client and the second client.

[0010] Optionally, processing and aggregating a plurality of sub-tasks of a task of a first client to obtain an aggregated processing result of the first client comprises:

[0011] splitting the task of the first client to obtain the plurality of sub-tasks;

[0012] generating a configuration file required for federation for the plurality of sub-tasks;

[0013] generating a total configuration file according to the plurality of sub-tasks and the corresponding configuration files, wherein the total configuration file comprises the plurality of sub-tasks and the corresponding configuration files, and the configuration file is used to indicate the type of sub-task and the required data;

[0014] processing the total configuration file to obtain a plurality of scheduling results;

[0015] aggregating the plurality of scheduling results to obtain an aggregated processing result of the first client.

[0016] Optionally, before the task of the first client is split to obtain the plurality of sub-tasks, the method further comprises:

[0017] obtaining the task of the first client from a preset directory;

[0018] detecting a file format of the task of the first client;

[0019] in a case where the detection result is that the task of the first client meets a preset format, detecting a data requirement of the task of the first client;

[0020] determining that the task of the first client meets a preset requirement.

[0021] Optionally, splitting the task of the first client to obtain the plurality of sub-tasks comprises:

[0022] respectively reading a user identifier, a geographical classification, whether the first client is a preset type of client, and a statistical date of the first client;

[0023] performing MD5 encryption on the user identifier, and performing data splitting according to the geographical classification and whether the first client is a preset type of client to obtain a plurality of sub-tasks, and writing the encrypted user identifier and the statistical date into files corresponding to the plurality of sub-tasks.

[0024] Optionally, the method further comprises:

[0025] using a data mapping file to record a mapping relationship of a file name, a geographical classification to which a file belongs, and whether the first client is a preset type of client.

[0026] Optionally, based on a preset task statistical requirement format, comparing and statistically analyzing the aggregated processing result of the first client and the task processing result of the second client to obtain a statistical result comprises:

[0027] adapting the aggregated processing result to the data mapping file through a landmark file and a statistical date in the aggregated processing result to obtain a task adaptation file of the first client;

[0028] using a statistical script to compare and summarize the task adaptation file and the task processing result according to the task statistical requirement format to obtain the statistical result.

[0029] Optionally, processing the total configuration file to obtain a plurality of scheduling results comprises:

[0030] The total configuration file and the corresponding configuration file of the plurality of tasks are respectively processed as follows: submitting a task, dependency scheduling, data uploading, data downloading, and data interaction.

[0031] According to another embodiment of the present application, a federal task statistical processing device is further provided, comprising:

[0032] An aggregation module is configured to process and aggregate a plurality of sub-tasks of a task of a first client to obtain an aggregated processing result of the first client, and store the aggregated processing result in a preset directory.

[0033] A reading module is configured to read the aggregated processing result of the first client from the preset directory and store the aggregated processing result in a statistical directory.

[0034] An obtaining module is configured to obtain a task processing result of a second client.

[0035] A statistical module is configured to, in the statistical directory, based on a pre-set task statistical requirement format, perform encrypted comparison and statistics on the aggregated processing result of the first client and the task processing result of the second client to obtain a statistical result, and feed back the statistical result to the first client and the second client.

[0036] Optionally, the aggregation module comprises:

[0037] A splitting sub-module is configured to split the task of the first client to obtain the plurality of sub-tasks.

[0038] A first generation sub-module is configured to generate a federal required configuration file for the plurality of sub-tasks.

[0039] A second generation sub-module is configured to generate a total configuration file according to the plurality of sub-tasks and the corresponding configuration file, wherein the total configuration file comprises the plurality of sub-tasks and the corresponding configuration file, and the configuration file is used to indicate the type of a sub-task and required data.

[0040] A processing sub-module is configured to process the total configuration file to obtain a plurality of scheduling results.

[0041] An aggregation sub-module is configured to aggregate the plurality of scheduling results to obtain the aggregated processing result of the first client.

[0042] Optionally, the device further comprises:

[0043] An obtaining sub-module is configured to obtain the task of the first client from a preset directory.

[0044] a first detecting sub-module, configured to detect a file format of a task of the first client;

[0045] a second detecting sub-module, configured to detect a data requirement of the task of the first client when the detection result is that the task of the first client meets a preset format;

[0046] a determining sub-module, configured to determine that the task of the first client meets a preset requirement.

[0047] Optionally, the splitting sub-module is further configured to read a user identifier of the first client, a regional classification, whether the first client is a preset type of client and a statistical date respectively; perform MD5 encryption on the user identifier, and perform data splitting according to the regional classification and whether the first client is the preset type of client to obtain a plurality of sub-tasks, and write the encrypted user identifier and the statistical date into files corresponding to the plurality of sub-tasks.

[0048] Optionally, the apparatus further comprises:

[0049] a recording sub-module, configured to record, using a data mapping file, a mapping relationship among file names of the plurality of sub-tasks, regional classifications to which the plurality of sub-tasks belong and whether the plurality of sub-tasks are preset types of clients.

[0050] Optionally, the statistical module is further configured to adapt the data mapping file by aggregating a landmark file in a processing result and a statistical date to obtain a task adaptation file of the first client; and compare and summarize the task adaptation file and the processing result of the task according to a task statistical requirement format by using a statistical script to obtain the statistical result.

[0051] Optionally, the processing sub-module is further configured to perform the following processing on the plurality of tasks and corresponding configuration files in the total configuration file respectively: submitting a task, scheduling a dependency, uploading data, downloading data and interacting data.

[0052] According to still another embodiment of the present application, a computer readable storage medium is provided, and the storage medium stores a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0053] According to still another embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to execute the steps in any of the above method embodiments.

[0054] The plurality of sub-tasks of the task of the first client are processed and aggregated to obtain an aggregated processing result of the first client, and the aggregated processing result is stored in a preset directory; the aggregated processing result of the first client is read from the preset directory and stored in a statistical directory; a task processing result of a second client is obtained; in the statistical directory, the aggregated processing result of the first client and the task processing result of the second client are statistically compared based on a pre-set task statistical requirement format, a statistical result is obtained, and the statistical result is fed back to the first client and the second client, which can solve the problem that federated learning cannot be applied to task statistics in the related art, can integrate the statistical result into a statistical format required by the user, reduce the degree of manual participation, improve the task statistical efficiency, and reduce the risk of data leakage caused by task comparison through the encrypted comparison and statistics. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0056] Figure 1 is a hardware structure block diagram of a mobile terminal of a federated task statistical processing method according to an embodiment of the application;

[0057] Figure 2 is a flowchart of a federated task statistical processing method according to an embodiment of the application;

[0058] Figure 3 is a flowchart of a federated task statistical processing method according to an optional embodiment of the application;

[0059] Figure 4 is a block diagram of federated scheduling automation configuration generation and numerical encryption comparison according to an embodiment of the application;

[0060] Figure 5 is a block diagram of a federated task statistical processing device according to an embodiment of the application. DETAILED DESCRIPTION

[0061] Hereinafter, the application will be described in detail with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that the embodiments and features in the embodiments in the present application can be combined with each other without conflict.

[0062] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0063] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking its operation on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for the federated task statistical processing method according to an embodiment of the present invention, as shown below. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0064] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the federated task statistical processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0065] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0066] This embodiment provides a federated task statistical processing method that runs on the aforementioned mobile terminal or network architecture. Figure 2 This is a flowchart of a federal task statistical processing method according to an embodiment of the present invention, such as...Figure 2 As shown in the figure, the flow includes the following steps:

[0067] Step S202, processing and aggregating a plurality of sub-tasks of the task of the first client to obtain an aggregated processing result of the first client, and storing the aggregated processing result in a preset directory;

[0068] Step S204, reading the aggregated processing result of the first client from the preset directory and storing the aggregated processing result in a statistical directory;

[0069] Step S206, obtaining the task processing result of the second client;

[0070] Step S208, in the statistical directory, based on a pre-set task statistical requirement format, performing encrypted comparison statistics on the aggregated processing result of the first client and the task processing result of the second client to obtain a statistical result, and feeding back the statistical result to the first client and the second client.

[0071] The embodiment of the application can be implemented based on the fate platform, and through the above steps S202 to S208, the problem that federated learning cannot be applied to task statistics in the related art can be solved, the statistical result can be integrated into the statistical format of the requirement, the manual participation degree is reduced, the task statistical efficiency is improved, and through the encrypted comparison statistics, the data leakage risk caused by task comparison can also be reduced.

[0072] Figure 3 is a flowchart of a federated task statistical processing method according to an optional embodiment of the application, as Figure 3 As shown in the figure, the above step S202 can specifically include:

[0073] S302, splitting the task of the first client to obtain a plurality of sub-tasks;

[0074] S304, generating a configuration file required for federation for the plurality of sub-tasks;

[0075] S306, generating a total configuration file according to the plurality of sub-tasks and the corresponding configuration files, wherein the total configuration file includes the plurality of sub-tasks and the corresponding configuration files, and the configuration file is used to indicate the type of the sub-task and the required data;

[0076] S308, federated scheduling of the total configuration file to obtain a plurality of scheduling results;

[0077] Further, the above step S308 can specifically include: respectively performing the following processing on the plurality of tasks and the corresponding configuration files in the total configuration file: submitting a task, dependency scheduling, data uploading, data downloading, and data interaction.

[0078] S310, aggregating the plurality of scheduling results to obtain an aggregated processing result of the first client.

[0079] In an optional embodiment, before the step S302, the method further comprises: obtaining the task of the first client from a preset directory; detecting a file format of the task of the first client; in a case that the detection result is that the task of the first client satisfies a preset format, detecting a data requirement of the task of the first client; and determining that the task of the first client satisfies a preset requirement.

[0080] In the embodiment, the step S302 can specifically include: respectively reading a user identifier, a geographical classification, whether the first client is a preset type client, and a statistical date; performing MD5 encryption on the user identifier, and performing data splitting according to the geographical classification and whether the first client is the preset type client to obtain a plurality of sub-tasks, and writing the encrypted user identifier and the statistical date into files corresponding to the plurality of sub-tasks.

[0081] Further, the method can further include: using a data mapping file to record a mapping relationship of a file name of the plurality of sub-tasks, a geographical classification to which the file belongs, and whether the first client is the preset type client.

[0082] Correspondingly, the step S208 can specifically include: adapting the data mapping file by using a landmark file in the aggregated processing result and the statistical date to obtain a task adaptation file of the first client; and comparing and summarizing the task adaptation file and the task processing result by using a statistical script according to a task statistical requirement format to obtain the statistical result.

[0083] The embodiment of the application can realize federal full-process automation, construct a scheduling architecture, only need to put data sets and configuration files into a specified file directory, and add a task scheduling file, and the whole scheduling can be automatically completed according to the order of the scheduling file, and for the case that there is dependency between tasks, the output of the response file can be directly referenced through eggroll, without uploading the file again, which can greatly reduce the task time. And by obtaining the output results of each task of the fate platform, the statistical format required is integrated, the manual participation is reduced, and the working efficiency of the fate platform is improved. On the basis of the fate platform, encrypted communication of both parties of the task is constructed, the comparison of the numerical size of both parties of the task is completed in a homomorphic encryption manner, and the total amount of comparison of both parties is recorded. And a plurality of parameters such as abnormal value, null value judgment processing, numerical format normalization, numerical encryption mode selection, threshold limit and the like are added to more conveniently process a plurality of complex tasks. Complete automation can be realized, and manual uploading of data, downloading of data, aggregation of data, statistics of data and submission of training tasks and the like are not needed, and a federal numerical encryption comparison scheme is also proposed, and the deficiency of the fate framework for the module is improved.

[0084] The embodiment of the application provides configuration generation and numerical encryption comparison of a federal scheduling automation system, Figure 4 is a block diagram of configuration generation and numerical encryption comparison of federal scheduling automation according to the embodiment of the application, like Figure 4As shown, it comprises the following modules: data checking module 42, data processing module 44, federal scheduling module 46 and federal statistics module 48, the data checking module 42 is used for monitoring the transmission of files in the specified directory, and periodically detecting the correctness of the obtained files; the data processing module 44 automatically divides and puts the data into the specified directory according to the demand, generates different dsl and conf files required by the federal algorithm according to the task demand, and generates the configuration file job_config of the overall scheduling task, which is used to specify the type of each federal task and the required data; the federal scheduling module 46 is a set of scheduling architecture based on the federal learning platform, which can start multiple federal tasks at the same time without human intervention, and can complete multiple operations such as task submission, dependency scheduling, data upload, data download and data interaction, and numerical encryption comparison is a two-party encrypted communication numerical comparison scheme based on paillier algorithm, which improves the defect that the federal learning platform fate cannot perform secure numerical comparison; then the data aggregation and arrangement of the output of multiple tasks of the federal platform are carried out, the data aggregation result is downloaded to the multi-party statistics directory for statistics through monitoring sftp. The statistics module is used for obtaining the output of the federal learning platform, and automatically performing statistics on the results and arranging them into the required format. The embodiment of the application reduces the consumption of human resources and the risk of error in the whole federal scheduling process, and is suitable for various different data. The whole federal scheduling process is fully automated, which greatly improves the work efficiency, and solves the data leakage risk caused by two-party numerical comparison.

[0085] A specified scanning file directory and a local data processing directory are provided, and through keyword and suffix name indexing, the required tar package and xml file are extracted, the tar package is automatically decompressed into gzip format, and then the final file is obtained through a second layer of decompression, and the file is downloaded to the local directory, and the cache and intermediate files in the processing process are cleared.

[0086] The data checking module 42 checks whether the downloaded file meets the file format and quantity requirements.

[0087] The data processing module 44 first divides the read data according to the separator, reads the user id, regional classification, whether it is an important customer and the statistical date, md5 encrypts the user id, and divides the data according to the regional classification and whether it is an important customer, and writes the encrypted user id and date into different files. Since the federal needs information security, the naming of the divided file is incremented from 1, and a mapping file is used to record the mapping relationship between the file name and the file belonging to the region and whether it is an important customer.

[0088] After the data is cut, the configuration file required by each data is generated, including the conf file and the dsl file, and the program can automatically generate different types of configuration files according to the identification of different user tasks, including the psi task and the comparison task. Then each data, the conf file and the dsl file are packaged and placed in the corresponding folder, and the task is sorted, and the order of the task is filled in the custom configuration file job_config, which mainly includes: the table_name and namespace required by each task; the output result of other tasks referred by each task, which needs to be named in the task sequence number and component name of the referred task in the job_config, without downloading the eggroll data, improving the work efficiency; the local processing script required by different tasks can be automatically placed in the task sequence folder, reducing the manual participation.

[0089] The federal scheduling developed by the application can be migrated to different environments, and the application of the whole scheduling can be realized, and the generalization is good. After the task double system is migrated, the folder is packaged according to the automatic processing program in the fourth step and stored in the scheduling designated directory, so that the federal task can be started, and the task can be started through the landmark file without the participation of technical personnel.

[0090] The comparison module for numerical encryption comparison is developed based on the fate platform in the embodiment of the application, the deficiency of fate on one side is supplemented, and the numerical comparison in the financial field has great value. In the case that the task double does not know the size of the numerical value of the other party, the size comparison can be completed. And the comparison module is developed into the fate architecture mode, and is applied to the fate platform by configuring the conf file and the dsl file.

[0091] After the federal task is completed, there may be hundreds of tasks involved, and the model result is aggregated to the same file through data aggregation in the embodiment of the application. The sftp monitoring file scans and monitors the data aggregation file in the specified directory, and downloads the file to the data statistics directory. The multi-party statistics module automatically adapts to the generated data mapping file through the landmark file in the aggregation file and the statistical date, and then automatically induces the statistical result according to the task statistical requirement format, and uploads the statistical result to the task double.

[0092] According to the embodiment of the present application, data segmentation can be automatically completed according to different tasks, configuration files required for federation are generated, and mapping with data is completed, and the task sequence is kept secret for both parties to ensure security. Based on the scheduling architecture developed by the federal learning platform fate, the task mounting can be automatically completed throughout. Based on the numerical encryption comparison algorithm developed by the federal learning platform fate, the defect that fate cannot perform secure numerical comparison is improved. Data aggregation statistics eliminates manual checking of fateboard, and in the case of a large amount of tasks, the efficiency will be greatly improved.

[0093] According to another embodiment of the present application, a federal task statistical processing device is also provided, Figure 5 is a block diagram of a federal task statistical processing device according to an embodiment of the present application, as Figure 5 shown, comprising:

[0094] An aggregation module 52 is configured to process and aggregate a plurality of sub-tasks of a task of a first client to obtain an aggregation processing result of the first client, and store the aggregation processing result in a preset directory;

[0095] A reading module 54 is configured to read the aggregation processing result of the first client from the preset directory and store the aggregation processing result in a statistical directory;

[0096] An acquisition module 56 is configured to acquire a task processing result of a second client;

[0097] A statistical module 58 is configured to perform encrypted comparison statistics on the aggregation processing result of the first client and the task processing result of the second client based on a pre-set task statistical requirement format in the statistical directory to obtain a statistical result, and feed back the statistical result to the first client and the second client.

[0098] Optionally, the aggregation module 52 comprises:

[0099] A segmentation sub-module is configured to segment the task of the first client to obtain the plurality of sub-tasks;

[0100] A first generation sub-module is configured to generate configuration files required for federation for the plurality of sub-tasks;

[0101] A second generation sub-module is configured to generate a total configuration file according to the plurality of sub-tasks and corresponding configuration files, wherein the total configuration file comprises the plurality of sub-tasks and corresponding configuration files, and the configuration file is used to indicate the type of sub-task and required data;

[0102] A processing sub-module is configured to process the total configuration file to obtain a plurality of scheduling results;

[0103] An aggregating submodule is configured to aggregate the plurality of scheduling results to obtain an aggregated processing result of the first client.

[0104] Optionally, the apparatus further comprises:

[0105] An obtaining submodule is configured to obtain the task of the first client from a preset directory.

[0106] A first detecting submodule is configured to detect a file format of the task of the first client.

[0107] A second detecting submodule is configured to, in a case where the detecting result is that the task of the first client meets a preset format, detect a data requirement of the task of the first client.

[0108] A determining submodule is configured to determine that the task of the first client meets a preset requirement.

[0109] Optionally, the splitting submodule is further configured to read a user identifier, a geographical classification, whether the first client is a preset type of client, and a statistical date respectively; perform MD5 encryption on the user identifier; and perform data splitting according to the geographical classification and whether the first client is a preset type of client to obtain a plurality of subtasks, and write the encrypted user identifier and the statistical date into files corresponding to the plurality of subtasks.

[0110] Optionally, the apparatus further comprises:

[0111] A recording submodule is configured to record, using a data mapping file, a mapping relationship among a file name of the plurality of subtasks, a geographical classification to which the file belongs, and whether the first client is a preset type of client.

[0112] Optionally, the statistical module 58 is further configured to, by using a landmark file in the aggregated processing result and a statistical date, adapt the data mapping file to obtain a task adaptation file of the first client; and by using a statistical script, compare and summarize the task adaptation file and the task processing result according to a task statistical requirement format to obtain the statistical result.

[0113] Optionally, the processing submodule is further configured to perform the following processing on the plurality of tasks and corresponding configuration files in the total configuration file respectively: submitting a task, scheduling a dependency, uploading data, downloading data, and data interaction.

[0114] It should be noted that each of the above modules can be implemented by software or hardware, and for the latter, the following implementation manners can be used, but are not limited thereto: all the above modules are located in the same processor; or the above modules are located in different processors in any combination.

[0115] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is configured to execute the steps in any of the method embodiments when running.

[0116] Optionally, in the embodiment, the storage medium is configured to store the computer program for executing the following steps.

[0117] S1, processing and aggregating a plurality of subtasks of a task of a first client to obtain an aggregated processing result of the first client, and storing the aggregated processing result under a preset directory;

[0118] S2, reading the aggregated processing result of the first client from the preset directory, and storing the aggregated processing result in a statistical directory;

[0119] S3, obtaining a task processing result of a second client;

[0120] S4, in the statistical directory, performing encrypted comparison and statistics on the aggregated processing result of the first client and the task processing result of the second client based on a pre-set task statistical requirement format, obtaining a statistical result, and feeding back the statistical result to the first client and the second client.

[0121] Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage media that can store computer programs.

[0122] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to execute the steps in any of the method embodiments.

[0123] Optionally, the electronic device can further comprise a transmission device and an input and output device, wherein the transmission device is connected with the processor, and the input and output device is connected with the processor.

[0124] Optionally, in the embodiment, the processor is configured to execute the following steps through the computer program:

[0125] S1, processing and aggregating a plurality of subtasks of a task of a first client to obtain an aggregated processing result of the first client, and storing the aggregated processing result under a preset directory;

[0126] S2, reading the aggregation processing result of the first client from the preset directory, and storing the aggregation processing result into a statistics directory;

[0127] S3, obtaining the task processing result of the second client;

[0128] S4, in the statistics directory, based on a preset task statistics requirement format, performing encrypted comparison statistics on the aggregation processing result of the first client and the task processing result of the second client, obtaining a statistics result, and feeding back the statistics result to the first client and the second client.

[0129] Optionally, specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0130] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned application can be realized by a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, each module or each step can be realized by program codes executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or each module or each step can be manufactured into an individual integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the application is not limited to any specific combination of hardware and software.

[0131] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for statistical processing of federal tasks, characterized in that, Numerical comparisons applied in the financial field include: The task of the first client is divided into multiple subtasks; The task of the first client is divided into multiple sub-tasks, including: reading the user identifier, regional classification, whether it is a preset type of customer, and statistical date of the first client respectively; encrypting the user identifier with MD5, and dividing the data according to the regional classification and whether it is a preset type of customer to obtain multiple sub-tasks; and writing the encrypted user identifier and the statistical date into the files corresponding to the multiple sub-tasks. The data mapping file records the filenames, geographical categories, and mapping relationships of the multiple subtasks as well as whether they belong to a preset type of customer. Generate the configuration files required for the federation of the multiple subtasks; A master configuration file is generated based on the multiple subtasks and their corresponding configuration files. The master configuration file includes the multiple subtasks and their corresponding configuration files, and the configuration files are used to indicate the type of the subtasks and the required data. The overall configuration file is processed to obtain multiple scheduling results; The multiple scheduling results are aggregated to obtain the aggregated processing result of the first client, and the aggregated processing result is stored in a preset directory; Read the aggregation processing results of the first client from the preset directory and store the aggregation processing results in the statistics directory; Obtain the task processing results from the second client; In the statistics directory, based on the pre-set task statistics requirement format, the aggregation processing result of the first client and the task processing result of the second client are encrypted and compared in a homomorphic encryption manner to obtain the statistics result, and the statistics result is fed back to the first client and the second client; Specifically, based on a pre-set task statistics requirement format, the aggregation processing results of the first client and the task processing results of the second client are compared and statistically analyzed to obtain the statistical results. This includes: matching the data mapping file with the identifier file and statistical date in the aggregation processing results to obtain the task adaptation file for the first client; and using a statistical script to compare and summarize the task adaptation file with the task processing results according to the task statistics requirement format to obtain the statistical results.

2. The method according to claim 1, characterized in that, Before dividing the task of the first client into the plurality of subtasks, the method further includes: Retrieve the task from the first client in the preset directory; The file format of the task on the first client is detected; If the detection result indicates that the task of the first client meets the preset format, the data requirements of the task of the first client are detected. It is determined that the task of the first client meets the preset requirements.

3. The method according to claim 1, characterized in that, The overall configuration file is processed to obtain several scheduling results, including: The following processes are performed on the multiple subtasks and their corresponding configuration files in the overall configuration file: task submission, dependency scheduling, data upload, data download, and data interaction.

4. A federal mission statistics processing device, characterized in that, Numerical comparisons applied in the financial field include: The sub-task module is used to divide the task of the first client into multiple sub-tasks. Dividing the task of the first client into multiple sub-tasks includes: reading the user identifier, region classification, whether it is a preset type of customer, and statistical date of the first client; encrypting the user identifier using MD5; dividing the data according to the region classification and whether it is a preset type of customer to obtain multiple sub-tasks; and writing the encrypted user identifier and the statistical date into the files corresponding to the multiple sub-tasks. The recording submodule is used to record the file names, file geographical categories, and mapping relationships of the multiple subtasks using a data mapping file; The first generation submodule is used to generate the configuration files required for the federation of the multiple subtasks; The second generation submodule is used to generate a total configuration file based on the multiple subtasks and their corresponding configuration files. The total configuration file includes the multiple subtasks and their corresponding configuration files, and the configuration files are used to indicate the type of the subtasks and the required data. The processing submodule is used to process the overall configuration file to obtain multiple scheduling results; An aggregation submodule is used to aggregate the multiple scheduling results to obtain the aggregation processing result of the first client, and store the aggregation processing result in a preset directory; a reading module is used to read the aggregation processing result of the first client from the preset directory, and store the aggregation processing result in a statistics directory; The acquisition module is used to obtain the task processing results from the second client. The statistics module is used to perform encrypted comparison and statistical analysis of the aggregation processing results of the first client and the task processing results of the second client in a homomorphic encryption manner within the statistics directory, based on a pre-set task statistics requirement format, to obtain statistical results, and then feed the statistical results back to the first client and the second client. Specifically, comparing and analyzing the aggregation processing results of the first client and the task processing results of the second client based on the pre-set task statistics requirement format to obtain statistical results includes: matching the data mapping file with the identifier file and statistical date in the aggregation processing results to obtain the task adaptation file for the first client; and using a statistical script to compare and summarize the task adaptation file with the task processing results according to the task statistics requirement format to obtain the statistical results.

5. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 3 when it is run.

6. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1 to 3.

Images