Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, which can be applied to the field of financial science and technology, and comprises the following steps: loading a data submission failure message and successfully uploaded candidate financial data of each first target period one by one from a financial database, and in the loading process, when the loading of each first target period is completed, completing the data submission failure message and the successfully uploaded candidate financial data; respectively clearing a first temporary table and a second temporary table which are correspondingly created for the candidate financial data and the data submission failure message of the N first target periods before loading; performing correlation analysis on the plurality of candidate financial data and the plurality of data submission failure messages, and determining first target financial data which is submitted successfully; and performing data difference comparison on the first partition table pre-loaded with the target uploading data and the plurality of pieces of first target financial data, determining second target financial data which fails to be submitted, and performing re-submission processing. According to the embodiment of the invention, the consistency between the financial data submitted by the financial platform and the financial data obtained by the financial supervision platform can be improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] In related technologies, in order to maintain financial stability, financial platforms need to regularly report financial data to financial regulatory platforms. Generally, they will first initialize and push all data to the reporting platform. After that, the data will be transmitted at a certain frequency based on the agreement between the two parties.

[0003] However, due to various reasons, such as data delays, dirty data, abnormal errors, etc., there will be inconsistencies between the financial data reported by the financial platform and the financial data obtained by the financial regulatory platform. As time goes by, this difference will gradually widen, which is not conducive to the supervision of the financial regulatory platform. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data processing method, device, electronic device and storage medium, aiming to improve the consistency between the financial data reported by the financial platform and the financial data obtained by the financial regulatory platform.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a data processing method, comprising the following steps: Loading the data submission failure message and the successfully uploaded candidate financial data of each first target period one by one from a preset financial database, wherein, during the loading process, after each candidate financial data and the data submission failure message of the first target period are loaded, clearing the first temporary table created for loading the candidate financial data of the first N first target periods, and clearing the second temporary table created for loading the data submission failure messages of the first N first target periods, wherein N is a positive integer; Performing correlation analysis on the loaded multiple candidate financial data and the multiple data submission failure messages, and determining the first target financial data that has been submitted successfully from the multiple candidate financial data; By comparing data differences between a first partition table pre-loaded with target uploaded data and a plurality of the first target financial data, second target financial data that failed to be reported is determined from the target reported data; The determined second target financial data is resubmitted.

[0006] In one embodiment, for each of the candidate financial data, the step of performing correlation analysis on the loaded multiple candidate financial data and the multiple data submission failure messages to determine the first target financial data that is successfully submitted among the multiple candidate financial data includes: If the current candidate financial data corresponds to the data submission failure message in the multiple loaded data submission failure messages, traverse the candidate financial data after the first target period where the current candidate financial data is located; If repeated financial data identical to the current candidate financial data is traversed, and the last repeated financial data traversed does not correspond to the data submission failure message in the multiple data submission failure messages, the current candidate financial data is determined as the first target financial data.

[0007] In one embodiment, the step of performing correlation analysis on the loaded plurality of candidate financial data and the plurality of data submission failure messages to determine the first target financial data that has been submitted successfully among the plurality of candidate financial data includes: Obtaining the batch number to which each candidate financial data and each data submission failure message belongs; Performing period division processing on the plurality of candidate financial data and the plurality of data reporting failure messages respectively with the second target period as a unit, to obtain the plurality of candidate financial data and the data reporting failure messages of the second target period, wherein the second target period is longer than the first target period; For each second target cycle, the candidate financial data and the data submission failure messages with the same batch number in the second target cycle, the second half cycle of the second target cycle before the second target cycle, and the first half cycle of the second target cycle after the second target cycle are correlated and compared, and the first target financial data are determined from the candidate financial data with the same batch number.

[0008] In one embodiment, the resubmitting the determined second target financial data includes: For each batch of the determined second target financial data with the same batch number, determine, by using the loaded candidate financial data, a first proportion and a second proportion of the second target financial data that failed to be uploaded and the second target financial data that successfully uploaded but failed to be submitted, respectively, relative to the second target financial data of the current batch; A re-submission strategy analysis is performed on each batch of the second target financial data with the same batch number with respect to the first proportion and the second proportion to determine a re-submission strategy for each batch of the second target financial data with the same batch number.

[0009] In one embodiment, the resubmitting the determined second target financial data includes: For each of the data reporting failure messages, if the data reporting failure message does not correspond to any of the candidate financial data, traverse the first partition table, and determine the data corresponding to the data reporting failure message in the target uploaded data as the third target financial data; An upload test is performed on the determined third target financial data, and a re-submission strategy for the third target financial data is determined according to the upload test result.

[0010] In one embodiment, during the loading process, after clearing the first temporary table and the second temporary table respectively, the method further includes: Merging a plurality of first log table files created for loading the candidate financial data of the first N target periods to obtain a first merged log file; If the first merged log file is generated before the current first target cycle, performing a merge optimization process on the first merged log file generated in the current first target cycle and the first merged log file generated before the current first target cycle; Merging a plurality of second log table files created for loading the data submission failure messages of the first N target cycles to obtain a second merged log file; If the second merged log file is generated before the current first target cycle, the second merged log file generated in the current first target cycle and the second merged log file generated before the current first target cycle are merged and optimized.

[0011] In one embodiment, the candidate financial data and the data submission failure message both correspond to a transaction identifier, a plurality of the candidate financial data are respectively stored in a plurality of second partition tables, and a plurality of the data submission failure messages are respectively stored in a plurality of third partition tables; For each of the first target periods, the step of loading the data submission failure messages and the candidate financial data of each first target period that are successfully uploaded one by one from the preset financial database includes: Performing a modulo process based on a first preset value on the transaction identifier of each candidate financial data belonging to the first target period to obtain a first modulo result of each candidate financial data belonging to the first target period, wherein the first modulo result corresponds to the second partition table; Performing a modulo processing based on a second preset value on the transaction identifier of each of the data delivery failure messages belonging to the first target period, to obtain a second modulo result of each of the data delivery failure messages belonging to the first target period, wherein the second modulo result corresponds to the third partition table; Loading each of the candidate financial data belonging to the first target period in parallel into the second partition table corresponding to the respective first modulo result; Each of the data delivery failure messages belonging to the first target cycle is loaded in parallel into the third partition table corresponding to the respective second modulo results.

[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data processing device, including: a data loading module, configured to load the data submission failure message and the successfully uploaded candidate financial data of each first target period one by one from a preset financial database, wherein during the loading process, after each candidate financial data and the data submission failure message of the first target period are loaded, a first temporary table created for loading the candidate financial data of the first target periods before N is cleared, and a second temporary table created for loading the data submission failure messages of the first target periods before N is cleared, where N is a positive integer; A data association analysis module, configured to perform association analysis on the loaded plurality of candidate financial data and the plurality of data submission failure messages, and determine the first target financial data that has been submitted successfully from the plurality of candidate financial data; A data difference comparison module, configured to compare data differences between a first partition table pre-loaded with target uploaded data and a plurality of the first target financial data, and determine second target financial data that fails to be reported from the target reported data; The resubmission module is used to resubmit the determined second target financial data.

[0013] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0014] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0015] The present application proposes a data processing method, device, electronic device and storage medium, the method may include: loading data submission failure messages and successfully uploaded candidate financial data of each first target period one by one from a preset financial database, wherein, during the loading process, after each candidate financial data and data submission failure message of the first target period are loaded, clearing the first temporary table created for loading the candidate financial data of the first N first target periods, and clearing the second temporary table created for loading the data submission failure messages of the first N first target periods, wherein N is a positive integer; performing correlation analysis on the loaded multiple candidate financial data and the multiple data submission failure messages, and determining the successfully submitted first target financial data from the multiple candidate financial data; performing data difference comparison between the first partition table pre-loaded with the target uploaded data and the multiple first target financial data, and determining the failed second target financial data from the target reported data; and performing resubmission processing on the determined second target financial data. The first target financial data that is successfully uploaded and reported can be determined through the candidate financial data and the data reporting failure message, and then the first target financial data and the target uploaded data of the first partition table can be used to determine the successfully reported financial data, so that the financial data that is not successfully reported can be re-reported, which is conducive to maintaining the data consistency between the financial supervision platform and the financial platform. On this basis, by loading the candidate financial data and data reporting failure message of each first target period one by one, the number of temporary tables generated during the loading process can be limited, thereby reducing the occupation of storage resources by temporary tables, which is conducive to improving the loading efficiency of the candidate financial data and data reporting failure message of each first target period. In addition, after loading the candidate financial data and data reporting failure message of a first target period, the temporary tables generated when loading the first N first target periods are deleted. In this way, the loading of the candidate financial data and data reporting failure message of each first target period can use sufficient storage resources, further improving the loading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present application; Figure 2 yes Figure 1 A flowchart of an embodiment of a sub-step of step 120; Figure 3 yes Figure 1 A flowchart of another sub-step embodiment of step 120; Figure 4 yes Figure 1 A flowchart of an embodiment of a sub-step of step 140; Figure 5 yes Figure 1 A flowchart of another sub-step embodiment of step 140; Figure 6 is a schematic diagram of the specific flow of the loading process provided in the embodiment of the present application; Figure 7 yes Figure 1 A flowchart of an embodiment of a sub-step of step 110; Figure 8 is a structural schematic diagram of a data processing device provided in an embodiment of the present application; Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] In related technologies, in order to maintain financial stability, financial platforms need to regularly report financial data to financial regulatory platforms. Generally, they will first initialize and push all data to the reporting platform. After that, the data will be transmitted at a certain frequency based on the agreement between the two parties.

[0021] However, due to various reasons, such as data delays, dirty data, abnormal errors, etc., there will be inconsistencies between the financial data reported by the financial platform and the financial data obtained by the financial regulatory platform. As time goes by, this difference will gradually widen, which is not conducive to the supervision of the financial regulatory platform.

[0022] In order to improve the consistency between the financial data reported by the financial platform and the financial data obtained by the financial regulatory platform, the embodiments of the present application provide a data processing method, a data processing device, an electronic device and a computer-readable storage medium. The method can first load the data reporting failure message and the candidate financial data uploaded successfully for each first target period from a preset financial database one by one. In the loading process, after each candidate financial data and data reporting failure message of the first target period are loaded, the first temporary table created for loading the candidate financial data of the first N first target periods is cleared, and the second temporary table created for loading the data reporting failure messages of the first N first target periods is cleared, where N is a positive integer; the loaded multiple candidate financial data are associated with the multiple data reporting failure messages, and the successfully reported first target financial data is determined from the multiple candidate financial data; the first partition table pre-loaded with the target uploaded data is compared with the multiple first target financial data for data differences, and the second target financial data that failed to be reported is determined from the target reported data; and the determined second target financial data is re-reported. The first target financial data that is successfully uploaded and successfully submitted can be determined through the candidate financial data and the data submission failure message, and then the first target financial data and the target uploaded data of the first partition table can be used to determine the successfully submitted financial data, so that the financial data that is not successfully submitted can be resubmitted, which is conducive to improving the data consistency between the financial supervision platform and the financial platform. On this basis, by loading the candidate financial data and data submission failure message of each first target period one by one, the number of temporary tables generated during the loading process can be limited, thereby reducing the occupation of storage resources by temporary tables, which is conducive to improving the loading efficiency of the candidate financial data and data submission failure message of each first target period. In addition, after loading the candidate financial data and data submission failure message of a first target period, the temporary tables generated when loading the first N first target periods are deleted. In this way, the loading of the candidate financial data and data submission failure message of each first target period can use sufficient storage resources, further improving the loading efficiency.

[0023] A data processing method, device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the data processing method in the embodiments of the present application is described.

[0024] The data processing method provided in the embodiment of the present application relates to the field of financial technology. The data processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data processing method, etc., but is not limited to the above forms.

[0025] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0026] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0027] See also Figure 1 , Figure 1 The flow chart of a data processing method provided by an embodiment of the present invention is shown. In the embodiment of the present application, the data processing method may include the following steps 110 to 140.

[0028] Step 110: Loading the data submission failure message and the successfully uploaded candidate financial data of each first target period from the preset financial database period by period, wherein during the loading process, after each candidate financial data and data submission failure message of the first target period are loaded, clearing the first temporary table created for loading the candidate financial data of the first N first target periods, and clearing the second temporary table created for loading the data submission failure messages of the first N first target periods, where N is a positive integer; Step 120: performing correlation analysis on the loaded multiple candidate financial data and multiple data submission failure messages, and determining the first target financial data that has been successfully submitted among the multiple candidate financial data; Step 130: performing data difference comparison between the first partition table pre-loaded with the target uploaded data and the plurality of first target financial data, and determining the second target financial data that fails to be reported in the target reported data; Step 140: resubmit the determined second target financial data.

[0029] In one embodiment, the data reporting failure message refers to a response message indicating the data reporting failure returned to the financial platform that reported the financial data after the financial regulatory platform failed to receive a specific financial data.

[0030] In one embodiment, loading the data submission failure message and the candidate financial data uploaded successfully for each first target period one by one from the preset financial database means first loading the data submission failure message and the candidate financial data uploaded successfully for a first target period from the financial database, and then loading the data submission failure message and the candidate financial data uploaded successfully for the next first target period after the loading is completed, and so on, until the data submission failure message and the candidate financial data uploaded successfully for the last first target period are loaded. Among them, the candidate financial data and the data submission failure message can be loaded into the same Hive data warehouse, or they can be loaded into different Hive data warehouses. In addition, among the multiple candidate financial data loaded, multiple of the multiple candidate financial data can be the same (for example, the data has been re-reported once before), or they can be different, which is not specifically limited here. In addition, the first target period can be one day, one week, one month, etc., which is not specifically limited here.

[0031] In one embodiment, in loading the data submission failure messages and the successfully uploaded candidate financial data of each first target period one by one from a preset financial database, specifically, firstly, a data file recording all the successfully uploaded candidate financial data and a log file recording the data submission failure messages can be obtained from a preset financial file library, and then, the data file and the log file are respectively decompressed to obtain a first directory in the data file and a second directory in the log file, and then, according to the first directory, each successfully uploaded candidate financial data of the first target period is loaded one by one from the financial database, and according to the second directory, each data submission failure message of the first target period is loaded one by one from the financial database.

[0032] In one embodiment, the temporary table refers to a table used to store temporary data generated during the loading process of data or information, wherein the temporary table may include a loading table, a log table, etc., which are not specifically limited here.

[0033] In one embodiment, the operation of associating and analyzing the loaded multiple candidate financial data with multiple data reporting failure messages and determining the first target financial data that has been successfully reported from the multiple candidate financial data refers to determining whether each candidate financial data has a corresponding data reporting failure message in the multiple data reporting failure messages, and determining the candidate financial data that does not have a corresponding data reporting failure message as the first target financial data. Specifically, since the data reporting failure message can indicate whether a financial data has failed to be reported, when a candidate financial data has a corresponding data reporting failure message in the multiple data reporting failure messages, it proves that the candidate financial data has been successfully uploaded and the financial regulatory platform has failed to receive the candidate financial data. On the contrary, when a candidate financial data has no corresponding data reporting failure message in the multiple data reporting failure messages, it proves that the candidate financial data has been successfully uploaded and the financial regulatory platform has successfully received the candidate financial data. Among them, the multiple candidate financial data and the multiple data reporting failure messages can be associated by time relationship, or by business ID when sending data, etc., which are not specifically limited here.

[0034] In one embodiment, determining the second target financial data that failed to be reported in the target reported data by comparing the data difference between the first partition table pre-loaded with the target uploaded data and multiple first target financial data refers to comparing the target uploaded data on the first partition table with multiple first target financial data, and determining all data in the target uploaded data except the first target financial data as the second target financial data. Among them, the target uploaded data is the full amount of data that needs to be uploaded to the financial regulatory platform loaded from the financial database. In addition, the first partition table can be loaded into the same hive as the candidate financial data, or loaded into a different hive from the candidate financial data, or loaded into the same hive as the data reporting failure message, or loaded into a different hive from the data reporting failure message, etc., which is not specifically limited here. In addition, the second target financial data can be financial data that failed to be reported, or data that failed to be re-reported, etc., which is not specifically limited here.

[0035] In one embodiment, the resubmission process refers to the operation of resubmitting the second target financial data to the financial regulatory platform based on a preset resubmission strategy. The resubmission strategy may be to merge the unsubmitted financial data with all the second target financial data in a new first target cycle, and submit the merged data in the second target cycle, or to batch all the second target financial data, and submit each batch of the second target financial data together with the unsubmitted financial data in different new first target cycles, etc., which are not specifically limited here. In addition, the resubmitted second target financial data may be submitted successfully, or may fail to be submitted, and the next resubmission may be performed, which are not specifically limited here.

[0036] See also Figure 2 , Figure 2 Shows Figure 1 In the flow of a sub-step embodiment of step 120, in one embodiment, for each candidate financial data, step 120 may include the following steps.

[0037] Step 210: If the current candidate financial data corresponds to a data submission failure message in the multiple loaded data submission failure messages, traverse the candidate financial data after the first target period where the current candidate financial data is located; Step 220: If repeated financial data identical to the current candidate financial data is traversed, and the last repeated financial data traversed does not correspond to a data submission failure message in the multiple data submission failure messages, the current candidate financial data is determined as the first target financial data.

[0038] In one embodiment, the re-reported financial data refers to candidate financial data that is the same as the current candidate financial data but is reported after the current candidate financial data.

[0039] In one embodiment, traversing the candidate financial data after the first target period where the current candidate financial data is located refers to not traversing other candidate financial data in the first target period where the current candidate financial data is located, but traversing the candidate financial data in the next first target period of the first target period and thereafter.

[0040] In one embodiment, if the current candidate financial data does not correspond to a data reporting failure message in the loaded multiple data reporting failure messages, the current candidate financial data is determined as the first target financial data.

[0041] In one embodiment, if the repeated financial data identical to the current candidate financial data is not traversed, it is determined that the current candidate financial data has failed to be submitted.

[0042] In one embodiment, if repeated financial data identical to the current candidate financial data is traversed, but the last repeated financial data traversed corresponds to a data submission failure message among multiple data submission failure messages, it is determined that the current candidate financial data and the traversed repeated financial data have failed to be submitted.

[0043] By traversing the candidate financial data of the first target period after the corresponding data reporting failure message appears in the current candidate financial data, it can be determined whether the candidate financial data is re-reported during the subsequent first target period. If the candidate financial data that is re-reported is traversed, the reporting result of the last re-reported financial data traversed is used to determine whether the candidate financial data is the first target financial data. In this way, it is possible to avoid the contradictory situation that the current candidate financial data is determined as the financial data that fails to be reported, and then the re-reported financial data is determined as the first target financial data that is successfully reported, thereby improving the accuracy of the judgment of the first target financial data, which is conducive to reducing the situation of erroneous re-reported data, and further conducive to improving the accuracy and reliability of the financial data re-reporting process.

[0044] See also Figure 3 , Figure 3 Shows Figure 1 In the process of another sub-step embodiment of step 120, in one embodiment, step 120 may include the following steps.

[0045] Step 310: Obtain the batch number to which each candidate financial data and each data submission failure message belongs; Step 320: performing period division processing on the plurality of candidate financial data and the plurality of data reporting failure messages respectively with the second target period as a unit, to obtain a plurality of candidate financial data and data reporting failure messages of the second target period, wherein the second target period is longer than the first target period; Step 330: For each second target cycle, the candidate financial data and data submission failure messages with the same batch number in the second target cycle, the second half of the second target cycle before the second target cycle, and the first half of the second target cycle after the second target cycle are correlated and compared, and the first target financial data are determined from the candidate financial data with the same batch number.

[0046] In one embodiment, the batch number refers to an identifier used to indicate the batch of candidate financial data to be submitted to the financial regulatory platform.

[0047] In one embodiment, the multiple candidate financial data and multiple data reporting failure messages are respectively divided into periods with the second target period as the unit, and the candidate financial data and data reporting failure messages of the multiple second target periods are obtained, which means that the candidate financial data are divided into the corresponding second target period according to the storage time of each candidate financial data, and the data reporting failure messages corresponding to the candidate financial data are divided into the corresponding second target period according to the situation that the candidate financial data are divided into the second target period. Among them, the second target period can be one month, half a year, one year, etc., which is not specifically limited here. In addition, the candidate financial data of the same batch (that is, the same batch number) can be divided into the same second target period, or divided into different second target periods, which is not specifically limited here.

[0048] Since financial data is reported in batches, and the data of the same batch is uploaded continuously, if a batch-based data reporting failure analysis is conducted, specific problems in the reporting process of the same batch of data can be easily discovered, thereby making it easier for the data in the same batch that needs to be re-reported to be successfully re-reported. However, when the time span of the same batch is large, the data reporting failure analysis process for the same batch needs to process a large amount of data. When the amount of data is large, it will occupy a larger cache, which may result in different analysis and processing efficiencies for different batches of financial data. For this reason, by dividing according to the second target period, the amount of data for the same batch can be limited by the second target period and its semi-periods before and after it, thereby limiting the cache generated by the data reporting failure analysis process for the same batch, which is conducive to improving the efficiency of the data reporting failure analysis process for the same batch.

[0049] See also Figure 4 , Figure 4 Shows Figure 1In the process of a sub-step embodiment of step 140, in one embodiment, step 140 may include the following steps.

[0050] Step 410: for each batch of second target financial data with the same batch number in the determined second target financial data, determine, through the loaded candidate financial data, the first proportion and the second proportion of the second target financial data that failed to be uploaded and the second target financial data that successfully uploaded but failed to be reported, respectively, relative to the second target financial data of the current batch; Step 420: Perform resubmission strategy analysis on the first proportion and the second proportion for each batch of second target financial data with the same batch number, and determine the resubmission strategy for each batch of second target financial data with the same batch number.

[0051] In one embodiment, determining, through the loaded candidate financial data, the first proportion and the second proportion of the second target financial data that failed to be uploaded and the second target financial data that successfully uploaded but failed to be reported, respectively, relative to the second target financial data of the current batch, means that the second target financial data of the current batch and the loaded candidate financial data are first subjected to data overlap analysis to determine the overlapping portion of the second target financial data of the current batch with the loaded candidate financial data, then determining the second proportion of the overlapping portion of the second target financial data of the current batch with the loaded candidate financial data in the current batch, and then determining the first proportion based on the second proportion.

[0052] In one embodiment, for the second target financial data of the same batch, in the process of analyzing the re-submission strategy for the first proportion and the second proportion for each batch of the second target financial data with the same batch number, and determining the re-submission strategy for each batch of the second target financial data with the same batch number, specifically, when the first proportion is greater than the second proportion, the strategy of submitting the second target financial data of the batch as a separate batch can be determined as the re-submission strategy for the second target financial data of the batch.

[0053] In one embodiment, for the second target financial data of the same batch, in the process of analyzing the re-submission strategy for the first proportion and the second proportion for each batch of second target financial data with the same batch number, and determining the re-submission strategy for each batch of second target financial data with the same batch number, specifically, when the second proportion is greater than the first proportion, the reporting connection channel used for reporting the second target financial data of the batch will be first determined, and then the reporting connection channel will be refreshed, and then the strategy for reporting the second target financial data of the batch through the refreshed reporting connection channel will be determined as the re-submission strategy for the second target financial data of the batch, wherein the second target financial data of the batch can be re-submitted in one batch through the refreshed reporting connection channel, or the second target financial data of the batch can be re-submitted in multiple batches through the refreshed reporting connection channel, which is not specifically limited here.

[0054] By calculating the first proportion and the second proportion for the same batch, the reason for the failure in reporting the second financial data of the same batch can be determined based on the first proportion and the second proportion of the second target financial data of the same batch. Therefore, according to the reason for the failure in reporting the second target financial data of different batches, a more reliable re-submission strategy can be determined for the second target financial data of each batch, which is conducive to improving the success rate of re-submission.

[0055] See also Figure 5 , Figure 5 Shows Figure 1 In the process of another sub-step embodiment of step 140, in one embodiment, step 140 may include the following steps.

[0056] Step 510: For each data delivery failure message, if the data delivery failure message does not correspond to any candidate financial data, traverse the first partition table and determine the data corresponding to the data delivery failure message in the target uploaded data as the third target financial data; Step 520: Perform an upload test on the determined third target financial data, and determine a re-submission strategy for the third target financial data according to the upload test result.

[0057] In one embodiment, traversing the first partition table and determining the data in the target uploaded data corresponding to the data reporting failure message as the third target financial data means traversing the target uploaded data to determine whether there is data in the target uploaded data corresponding to the data reporting failure message that does not correspond to the candidate financial data, and determining such data that failed to be uploaded and reported as the third target financial data.

[0058] In one embodiment, the upload test refers to a process of simulating submission of the determined third target financial data to determine whether the third target financial data is successfully uploaded and whether it can be submitted successfully.

[0059] In one embodiment, in the process of determining the re-submission strategy of the third target financial data based on the upload test results, if the upload test of the determined third target financial data fails, the data format of the third target financial data is determined. If the data format of the third target financial data does not conform to the data format required by the financial regulatory platform, the data format of the third target financial data is adjusted, and the strategy of re-submitting the third target financial data with a new data format is determined as the re-submission strategy; if the data format of the third target financial data conforms to the data format required by the financial regulatory platform, each third target financial data is traversed, and data compression processing is performed on the third target financial data that exceeds the single data upload capacity limit, and the strategy of re-submitting the third target financial data that does not need data compression processing and has been data compressed processing is determined as the re-submission strategy.

[0060] Since in the process of data reporting, there may be situations where the uploaded data is erroneous and the financial regulatory platform is receiving the data, resulting in upload failure and being able to receive messages indicating data reporting failure, therefore, by determining the third target financial data where these situations occur and performing upload tests on these third financial data, the source of the reporting failure can be determined between the financial regulatory platform and the financial platform, and then a more reliable re-reporting strategy can be selected for the third financial data, which is beneficial to improving the success rate of re-reporting of the third financial data.

[0061] See also Figure 6 , Figure 6 The specific flow of the loading process provided by the embodiment of the present application is shown. In one embodiment, after clearing the first temporary table and the second temporary table respectively, the loading process may further include the following steps.

[0062] Step 610: merging multiple first log table files created for loading candidate financial data of the first N first target periods to obtain a first merged log file; Step 620: If a first merged log file is generated before the current first target cycle, merge and optimize the first merged log file generated in the current first target cycle with the first merged log file generated before the current first target cycle; Step 630: merging multiple second log table files created for loading the data submission failure messages of the first N first target cycles to obtain a second merged log file; Step 640: If a second merged log file is generated before the current first target cycle, the second merged log file generated in the current first target cycle and the second merged log file generated before the current first target cycle are merged and optimized.

[0063] In one embodiment, the first merged log file refers to a file that records the candidate financial data loading log records of each first target period in the first N first target periods.

[0064] In one embodiment, performing a merge optimization process on the first merged log file generated in the current first target cycle and the first merged log file generated before the current first target cycle refers to an operation of merging the log records recorded in each first merged log file into one log file. Since this process is performed for each first time period, this process can actually perform an incremental merge of the log records generated by loading the candidate financial data of each first target cycle.

[0065] In one embodiment, if the first merged log file is not generated before the current first target period, the merge optimization process is not performed on the first merged log file.

[0066] In one embodiment, the second merged log file refers to a file that records the data reporting failure message loading log records of each first target period in the first N first target periods.

[0067] In one embodiment, performing a merge optimization process on the second merged log file generated in the current first target cycle and the second merged log file generated before the current first target cycle refers to an operation of merging the log records recorded in each second merged log file into one log file. Among them, since this process is performed for each first time period, this process can actually perform incremental merging of the log records generated by the data reporting failure message for each first target cycle.

[0068] In one embodiment, if no second merged log file is generated before the current first target period, the second merged log file is not subjected to merge optimization processing.

[0069] Since log records are generally not called during the loading process except in the case of loading errors, the candidate financial data and data submission failure messages of a first target period in the loading are merged, and the log records of the data submission failure messages of the first N first target periods are merged. In this way, a large number of small log table files generated during loading can be cleared, thereby reducing the load pressure of a large number of small log table files on the storage space and reducing the difficulty of managing log records, which is conducive to saving storage resources and improving the efficiency of log record query during loading errors.

[0070] See also Figure 7 , Figure 7 Shows Figure 1 In the process of a sub-step embodiment of step 110, in one embodiment, both the candidate financial data and the data submission failure message can correspond to the transaction identifier, multiple candidate financial data are respectively stored in multiple second partition tables, and multiple data submission failure messages are respectively stored in multiple third partition tables; for each first target period, step 110 may include the following steps.

[0071] Step 710: performing a modulo process based on a first preset value on the transaction identifier of each candidate financial data belonging to the first target period, to obtain a first modulo result of each candidate financial data belonging to the first target period, wherein the first modulo result corresponds to the second partition table; Step 720: performing a modulo process based on a preset value on the transaction identifier of each data delivery failure message belonging to the first target period, and obtaining a second modulo result of each data delivery failure message belonging to the first target period, wherein the second modulo result corresponds to the third partition table; Step 730: Load each candidate financial data belonging to the first target period in parallel into the second partition table corresponding to the respective first modulo result; Step 740: Load each data delivery failure message belonging to the first target period in parallel into the third partition table corresponding to the respective second modulo result.

[0072] In one embodiment, performing a modulo processing based on a first preset value on the transaction identifier of each candidate financial data belonging to the first target period to obtain a first modulo result of each candidate financial data belonging to the first target period refers to converting the transaction identifier of each candidate financial data belonging to the first target period into a numerical value, performing a modulo processing on the numerical value corresponding to the transaction identifier of each candidate financial data belonging to the first target period relative to the first preset value, and obtaining the first modulo result of each candidate financial data belonging to the first target period. The first modulo result can be used to indicate the second partition table to be loaded with the corresponding candidate financial data.

[0073] In one embodiment, performing a modulo processing based on a second preset value on the transaction identifier of each data delivery failure message belonging to the first target period, and obtaining a first modulo result of each data delivery failure message belonging to the first target period, refers to converting the transaction identifier of each candidate financial data belonging to the first target period into a numerical value, performing a modulo processing on the numerical value corresponding to the transaction identifier of each data delivery failure message belonging to the first target period relative to the second preset value, and obtaining a second modulo result of each data delivery failure message belonging to the first target period. The second modulo result can be used to indicate the third partition table to be loaded for the corresponding data delivery failure message. The second preset value can be the same as or different from the first preset value, and is not specifically limited here.

[0074] By determining the second partition table to which each candidate financial data belongs, the candidate financial data of a first target period can be quickly loaded using parallel loading. Similarly, by determining the third partition table to which each data submission failure message belongs, the data submission failure message of a first target period can be quickly loaded using parallel loading. Since the candidate financial data and data submission failure messages of each first target period are loaded in parallel, the candidate financial data and data submission failure messages with large data volumes can be greatly shortened, thereby effectively improving the efficiency of loading the full amount of data.

[0075] See also Figure 8 The embodiment of the present application further provides a data processing device, which can implement the data processing method of the first aspect above, and the data processing device 800 includes: The data loading module 810 may be used to load the data submission failure message and the successfully uploaded candidate financial data of each first target period one by one from a preset financial database, wherein during the loading process, after each candidate financial data and data submission failure message of the first target period are loaded, the first temporary table created for loading the candidate financial data of the first N first target periods is cleared, and the second temporary table created for loading the data submission failure messages of the first N first target periods is cleared, where N is a positive integer; The data association analysis module 820 may be used to perform association analysis on the loaded multiple candidate financial data and multiple data submission failure messages, and determine the first target financial data that has been successfully submitted from the multiple candidate financial data; The data difference comparison module 830 may be used to compare the data differences between the first partition table pre-loaded with the target uploaded data and the plurality of first target financial data, and determine the second target financial data that failed to be reported from the target reported data; The resubmission module 840 may be used to perform resubmission processing on the determined second target financial data.

[0076] The specific implementation of the data processing device is basically the same as the specific implementation of the above-mentioned data processing method, and will not be repeated here.

[0077] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0078] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated. The electronic device 900 includes: The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data processing method of the embodiments of this application; Input / output interface 903, used to implement information input and output; Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904); The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0079] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data processing method when executed by a processor.

[0080] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0082] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0083] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0085] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0086] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0091] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data processing method, characterized in that: The following steps are involved: Loading the data submission failure message and the successfully uploaded candidate financial data of each first target period one by one from a preset financial database, wherein, during the loading process, after each candidate financial data and the data submission failure message of the first target period are loaded, clearing the first temporary table created for loading the candidate financial data of the first N first target periods, and clearing the second temporary table created for loading the data submission failure messages of the first N first target periods, wherein N is a positive integer; Performing correlation analysis on the loaded multiple candidate financial data and the multiple data submission failure messages, and determining the first target financial data that has been submitted successfully from the multiple candidate financial data; By comparing data differences between a first partition table pre-loaded with target uploaded data and a plurality of the first target financial data, second target financial data that failed to be reported is determined from the target reported data; The determined second target financial data is resubmitted.

2. The method according to claim 1, characterized in that For each of the candidate financial data, the step of performing correlation analysis on the loaded multiple candidate financial data and the multiple data submission failure messages to determine the first target financial data that is successfully submitted among the multiple candidate financial data includes: If the current candidate financial data corresponds to the data submission failure message in the multiple loaded data submission failure messages, traverse the candidate financial data after the first target period where the current candidate financial data is located; If repeated financial data identical to the current candidate financial data is traversed, and the last repeated financial data traversed does not correspond to the data submission failure message in the multiple data submission failure messages, the current candidate financial data is determined as the first target financial data.

3. The method according to claim 1, characterized in that The step of performing correlation analysis on the loaded plurality of candidate financial data and the plurality of data submission failure messages to determine the first target financial data that has been successfully submitted among the plurality of candidate financial data includes: Obtaining the batch number to which each candidate financial data and each data submission failure message belongs; Performing period division processing on the plurality of candidate financial data and the plurality of data reporting failure messages respectively with the second target period as a unit, to obtain the plurality of candidate financial data and the data reporting failure messages of the second target period, wherein the second target period is longer than the first target period; For each second target cycle, the candidate financial data and the data submission failure messages with the same batch number in the second target cycle, the second half cycle of the second target cycle before the second target cycle, and the first half cycle of the second target cycle after the second target cycle are correlated and compared, and the first target financial data are determined from the candidate financial data with the same batch number.

4. The method according to claim 3, characterized in that The resubmitting the determined second target financial data includes: For each batch of the determined second target financial data with the same batch number, determine, by using the loaded candidate financial data, a first proportion and a second proportion of the second target financial data that failed to be uploaded and the second target financial data that successfully uploaded but failed to be submitted, respectively, relative to the second target financial data of the current batch; A re-submission strategy analysis is performed on each batch of the second target financial data with the same batch number with respect to the first proportion and the second proportion to determine a re-submission strategy for each batch of the second target financial data with the same batch number.

5. The method according to claim 2, characterized in that: The resubmitting the determined second target financial data includes: For each of the data reporting failure messages, if the data reporting failure message does not correspond to any of the candidate financial data, traverse the first partition table, and determine the data corresponding to the data reporting failure message in the target uploaded data as the third target financial data; An upload test is performed on the determined third target financial data, and a re-submission strategy for the third target financial data is determined according to the upload test result.

6. The method according to claim 1, characterized in that During the loading process, after clearing the first temporary table and the second temporary table respectively, the method further includes: Merging a plurality of first log table files created for loading the candidate financial data of the first N target periods to obtain a first merged log file; If the first merged log file is generated before the current first target cycle, performing a merge optimization process on the first merged log file generated in the current first target cycle and the first merged log file generated before the current first target cycle; Merging a plurality of second log table files created for loading the data submission failure messages of the first N target cycles to obtain a second merged log file; If the second merged log file is generated before the current first target cycle, the second merged log file generated in the current first target cycle and the second merged log file generated before the current first target cycle are merged and optimized.

7. The method according to claim 1, characterized in that The candidate financial data and the data submission failure message both correspond to the transaction identifier, a plurality of the candidate financial data are respectively stored in a plurality of second partition tables, and a plurality of the data submission failure messages are respectively stored in a plurality of third partition tables; For each of the first target periods, the step of loading the data submission failure messages and the candidate financial data of each first target period that are successfully uploaded one by one from the preset financial database includes: Performing a modulo process based on a first preset value on the transaction identifier of each candidate financial data belonging to the first target period to obtain a first modulo result of each candidate financial data belonging to the first target period, wherein the first modulo result corresponds to the second partition table; Performing a modulo processing based on a second preset value on the transaction identifier of each of the data delivery failure messages belonging to the first target period, to obtain a second modulo result of each of the data delivery failure messages belonging to the first target period, wherein the second modulo result corresponds to the third partition table; Loading each of the candidate financial data belonging to the first target period in parallel into the second partition table corresponding to the respective first modulo result; Each of the data delivery failure messages belonging to the first target cycle is loaded in parallel into the third partition table corresponding to the respective second modulo results.

8. A data processing device, characterized in that: include: a data loading module, configured to load the data submission failure message and the successfully uploaded candidate financial data of each first target period one by one from a preset financial database, wherein during the loading process, after each candidate financial data and the data submission failure message of the first target period are loaded, a first temporary table created for loading the candidate financial data of the first target periods before N is cleared, and a second temporary table created for loading the data submission failure messages of the first target periods before N is cleared, where N is a positive integer; A data association analysis module, configured to perform association analysis on the loaded plurality of candidate financial data and the plurality of data submission failure messages, and determine the first target financial data that has been submitted successfully from the plurality of candidate financial data; A data difference comparison module, configured to compare data differences between a first partition table pre-loaded with target uploaded data and a plurality of the first target financial data, and determine second target financial data that fails to be reported from the target reported data; The resubmission module is used to resubmit the determined second target financial data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images