Data file processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to the technical field of data processing, and particularly discloses a data file processing method and device, computer equipment and a readable storage medium. The method comprises the following steps: when current time is switched from a previous period to a current period, determining a plurality of adjacent historical periods corresponding to the current period, and obtaining respective file receiving time of each product data file sent to a downstream system by each upstream system in each historical period; determining receiving time distribution information based on the receiving time of each file; determining target processing time of the batch processing tasks in the current period according to the receiving time distribution information; accessing a file storage directory according to the target processing time, and when a directory corresponding to the unprocessed product data file exists in the file storage directory, loading the unprocessed product data file to a temporary data table; and calling a preset online transaction according to the temporary data table so as to synchronize the product data in the temporary data table to a product data formal table. By adopting the method, computer resources can be saved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data file processing method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] As banking systems become more complex and diversified, the interaction between systems is becoming more frequent. The upstream and downstream systems usually interact through data files. The upstream system performs preliminary processing on the product parameters reported by each subsidiary, and then synchronizes the product development components of the downstream system. Due to the different timeliness of data reported by each subsidiary, the time of reporting to the downstream system is also different.

[0003] The downstream system generally defines dependent batch jobs to monitor whether files have arrived. When the data file is detected to have arrived, the next step of processing is carried out. If the data file has not arrived, continuous monitoring is required. However, the uncertainty of the arrival time of the upstream file will cause the predecessor job of the dependent batch job to monitor for a long time, thereby wasting computer resources of the downstream system. Summary of the invention

[0004] Based on this, it is necessary to provide a data file processing method, device, computer equipment, computer-readable storage medium and computer program product that can save computer resources in response to the above technical problems.

[0005] In a first aspect, the present application provides a data file processing method, comprising:

[0006] When the current time switches from the previous cycle to the current cycle, multiple adjacent historical cycles corresponding to the current cycle are determined;

[0007] According to the respective time identifiers of the respective historical periods, the respective file receiving times of the respective product data files sent by the respective upstream systems to the downstream systems in the respective historical periods are obtained from the preset receiving time data table;

[0008] Determine the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period;

[0009] Determine the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determine the target processing time of the batch processing task in the current period according to the distribution type;

[0010] Accessing the file storage directory according to the target processing time, and when there is a directory corresponding to the unprocessed product data file in the file storage directory, loading the unprocessed product data file into a temporary data table;

[0011] The data in the temporary data table is assembled to obtain an online transaction input parameter object, and the online transaction input parameter object is used to call a preset online transaction to synchronize the product data in the temporary data table to the formal product data table.

[0012] In one embodiment, determining the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period includes:

[0013] Divide a time period into multiple time periods;

[0014] For each time period, counting the number of product data files whose file receiving time in the multiple historical periods is within the time period;

[0015] According to the number obtained by statistics in each time period, the receiving time distribution information corresponding to each product data file is determined.

[0016] In one embodiment, determining the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determining the target processing time of the batch processing task in the current period according to the distribution type, includes:

[0017] When the reception time distribution information indicates that the statistical quantity of one or more time periods accounts for a proportion greater than a preset threshold in the total quantity of product data files, determining that the distribution type of the file reception time in each historical period is a concentrated distribution type;

[0018] Determine a first target time period corresponding to the concentrated distribution type from the one or more time periods; the first target time period is the time period with the largest number of statistics;

[0019] A target processing time for the batch processing task in the current cycle is determined according to the first target time period.

[0020] In one embodiment, determining the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determining the target processing time of the batch processing task in the current period according to the distribution type, includes:

[0021] When the reception time distribution information indicates that the statistical quantities corresponding to the respective time periods account for a similar proportion in the total quantity of the product data files, it is determined that the distribution type of the file reception time in each historical period is a non-centralized distribution type;

[0022] Determine a second target time period corresponding to the non-centralized distribution type from each of the time periods; the second target time period is the latest time period in which the product data file is received;

[0023] According to the second target time period, a target processing time for the batch processing task in the current cycle is determined.

[0024] In one embodiment, the method further comprises:

[0025] When a product data file is received through the downstream system, determining a target upstream system to which the received product data file belongs;

[0026] Calling the interceptor corresponding to the target upstream system to intercept the received product data file, and verifying the received product data file through the interceptor corresponding to the target upstream system;

[0027] When the verification result indicates that the verification is passed, the received product data file is written into the storage space corresponding to the file storage directory.

[0028] In one of the embodiments, when there is a directory corresponding to the unprocessed product data file in the file storage directory, loading the unprocessed product data file into the temporary data table includes:

[0029] When any file directory in the file storage directory does not contain a field indicating completion, determining the product data file corresponding to the file target as an unprocessed product data file; or

[0030] When there is a folder directory containing a field indicating incompleteness in the file storage directory and there is a product data file in a folder corresponding to the folder directory, the product data file in the folder corresponding to the folder directory is determined as an unprocessed product data file.

[0031] In a second aspect, the present application also provides a data file processing device, comprising:

[0032] The file receiving time acquisition module is used to determine the multiple adjacent historical periods corresponding to the current period when the current time switches from the previous period to the current period; according to the time identifier of each historical period, obtain the file receiving time of each product data file sent by each upstream system to the downstream system in each historical period from the preset receiving time data table;

[0033] A receiving time distribution information determination module, used to determine the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period;

[0034] A target processing time determination module, used to determine the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determine the target processing time of the batch processing task in the current period according to the distribution type;

[0035] A file loading module, configured to access a file storage directory according to the target processing time, and when a date directory corresponding to an unprocessed product data file exists in the file storage directory, load the unprocessed product data file into a temporary data table;

[0036] The product data synchronization module is used to assemble the data in the temporary data table, obtain the online transaction input object, and use the online transaction input object to call the preset online transaction to synchronize the product data in the temporary data table to the formal product data table.

[0037] In one of the embodiments, the receiving time distribution information determination module is also used to: divide a time period into multiple time periods; for each time period, count the number of product data files whose file receiving time in the multiple adjacent historical periods is within the time period, and obtain the statistical number corresponding to the time period; determine the receiving time distribution information corresponding to each product data file based on the statistical number corresponding to each time period.

[0038] In one of the embodiments, the receiving time distribution information determination module is also used to: when the receiving time distribution information indicates that the statistical quantity of one or more time periods accounts for a proportion greater than a preset threshold in the total quantity of product data files, determine that the distribution type of the file receiving time in each historical period is a concentrated distribution type; from the one or more time periods, determine the first target time period corresponding to the concentrated distribution type; the first target time period is the time period with the largest number of statistics; and determine the target processing time of the batch task in the current period based on the first target time period.

[0039] In one of the embodiments, the receiving time distribution information determination module is also used for: when the receiving time distribution information indicates that the statistical quantity corresponding to each time period accounts for a similar proportion in the total number of product data files, determining that the distribution type of the file receiving time in each historical period is a non-concentrated distribution type; determining a second target time period corresponding to the non-concentrated distribution type from each of the time periods; the second target time period is the latest time period for receiving the product data file; and determining the target processing time of the batch task in the current period based on the second target time period.

[0040] In one of the embodiments, the device also includes a verification module, which is used to: when a product data file is received through the downstream system, determine the target upstream system to which the received product data file belongs; call the interceptor corresponding to the target upstream system to intercept the received product data file, and verify the received product data file through the interceptor corresponding to the target upstream system; when the verification result indicates that the verification is passed, write the received product data file into the storage space corresponding to the file storage directory.

[0041] In one of the embodiments, the file loading module is further used to: when any file directory in the file storage directory does not contain a field indicating completion, determine the product data file corresponding to the file target as an unprocessed product data file; or when there is a folder directory in the file storage directory containing a field indicating incomplete and there is a product data file in the folder corresponding to the folder directory, determine the product data file in the folder corresponding to the folder directory as an unprocessed product data file.

[0042] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the data file processing method in the above embodiment when executing the computer program.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the data file processing method in the above embodiment are implemented.

[0044] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data file processing method in the above embodiment.

[0045] The above-mentioned data file processing method, device, computer equipment, computer-readable storage medium and computer program product, when the current time switches from the previous cycle to the current cycle, by obtaining the respective file reception time of each product data file sent by each upstream system to the downstream system in multiple adjacent historical cycles corresponding to the current cycle, based on the position of each file reception time in the respective historical cycle, determine the reception time distribution information corresponding to each product data file, the reception time distribution information can be dynamically determined in different cycles, and can reflect the approximate law of the arrival time distribution of product data files sent by each upstream system in the recent period, and determine the distribution type of the file reception time in each historical cycle according to the reception time distribution information, By determining the target processing time of the batch task in the current cycle according to the distribution type and accessing the file storage directory according to the target processing time, the product data files can be processed through scheduled batch processing, thereby eliminating the need to continuously monitor the arrival time of the product data files, saving computer resources of the downstream system. In addition, since the target processing time of the scheduled batch task is dynamically determined according to the current cycle, the timeliness of data file processing can be improved. In addition, in the scheduled batch task, the unprocessed product data files are loaded into a temporary data table, the data in the temporary data table is assembled, and the online transaction input object is obtained. The preset online transaction is called using the online transaction input object, which can also improve the processing efficiency of the data file. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 An application environment diagram of a data file processing method in an embodiment;

[0048] Figure 2 A schematic diagram of a flow chart of a data file processing method in one embodiment;

[0049] Figure 3 A schematic diagram of a flow chart of steps for determining reception time distribution information in one embodiment;

[0050] Figure 4 is a flow chart of a data file processing method in another embodiment;

[0051] Figure 5 A schematic diagram of a data file verification process in an embodiment;

[0052] Figure 6 A schematic diagram of the overall process of a data file processing method in one embodiment;

[0053] Figure 7 is a structural block diagram of a data file processing device in one embodiment;

[0054] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0056] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0057] The data file processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the downstream system 102 communicates with multiple upstream systems 104 through the network. Each upstream system pushes the relevant product data files to the file storage directory specified by the downstream system through nft by submitting .dat files at irregular intervals (generally under the nas path agreed upon by the upstream and downstream systems). After the file arrives at the downstream system, the downstream system defines a scheduled batch processing task through a general batch processing framework, such as CtrlM, Luban or other methods. The scheduled batch processing task can execute the shell script in the specified file storage directory, and the following steps are performed through the shell script: When the current time switches from the previous cycle to the current cycle, determine the multiple adjacent historical cycles corresponding to the current cycle; according to the time identifiers of each historical cycle, obtain the preset receiving time data table. The respective file receiving time of each product data file sent by each upstream system to the downstream system in each historical period; determining the receiving time distribution information corresponding to each product data file based on the position of each file receiving time in the respective historical period; determining the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determining the target processing time of the batch task in the current period according to the distribution type; accessing the file storage directory according to the target processing time, and when there is a directory corresponding to the unprocessed product data file in the file storage directory, loading the unprocessed product data file into the temporary data table; assembling the data in the temporary data table to obtain the online transaction input object, and using the online transaction input object to call the preset online transaction to synchronize the product data in the temporary data table to the formal product data table.

[0058] The downstream system 102 and the upstream system 104 may be implemented through a server. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0059] In an exemplary embodiment, Figure 2 As shown, a data file processing method is provided, which is applied to Figure 1 Taking the downstream system 102 in the example as an example, the method includes the following steps 202 to 210. Among them:

[0060] Step 202: When the current time switches from the previous cycle to the current cycle, a plurality of adjacent historical cycles corresponding to the current cycle are determined.

[0061] Step 204 , according to the respective time identifiers of the respective historical periods, the respective file receiving times of the respective product data files sent by the respective upstream systems to the downstream systems in the respective historical periods are obtained from the preset receiving time data table.

[0062] Among them, the current cycle refers to the cycle in which the data file needs to be processed at present. The cycle here is the time cycle. The specific time length can be customized. For example, a cycle can be one day, and the current cycle can be the same day. File reception time, that is, the time when the downstream system successfully receives the product data file sent by the upstream system. The product data file refers to the ".dat" file related to the product. In the embodiment of the present application, the product here can be a bank-related product, for example, it can be various financial products, loan products, etc. The product data file contains product-related parameter information. The relevant parameter information in the product data file can be recorded in a data table. Each product data file can contain a main table and multiple attached tables associated with the main table. The main table is used to store the main parameter information of the product, and the attached table can store the accessory parameter information related to the product. The historical cycle can be the cycle before the current cycle. Multiple adjacent historical cycles are multiple historical cycles adjacent to the current cycle and before the current cycle. Here, the number of multiple historical cycles can be set as needed. Taking one cycle as one day as an example, the multiple adjacent historical cycles corresponding to the current cycle can be the consecutive days before the current day. For example, if the current day is October 15, the multiple adjacent historical cycles can be the previous 5 days, that is, October 10 to October 14. The preset receiving time data table is a data table used to store file receiving time. The time identifier of each historical period in the data table is stored in association with the file receiving time corresponding to each historical period. The time identifier of the historical period is used to identify the historical period. For example, when a period is one day, the time identifier can be a specific date.

[0063] Specifically, when the downstream system receives a data file sent by the upstream system, it can record the file receiving time of the data file in a preset receiving time data table, and the file receiving time of each cycle is stored corresponding to the time identifier of the cycle. Then, when a cycle switch is detected, the downstream system can first determine the multiple adjacent historical cycles corresponding to the current cycle, and then obtain the time identifier of each historical cycle, and then traverse the preset receiving time data table to query and obtain the file receiving time of each product data file sent by each upstream system to the downstream system in each historical cycle.

[0064] Step 206: Determine the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period.

[0065] Among them, the receiving time distribution information is information used to characterize the distribution of the file receiving time of each product data file. Exemplarily, the receiving time distribution information can specifically be the distribution of the file receiving time of each product data file in each time period. Exemplarily, the receiving time distribution information corresponding to each product data file can be the receiving time distribution information corresponding to all upstream systems. Alternatively, in other embodiments, the receiving time distribution information corresponding to each upstream system can be determined respectively according to the file receiving time of the product file data of different upstream systems, that is, each upstream system corresponds to a receiving time distribution information. The position of the file receiving time in a cycle, that is, the position of the file receiving time in the entire time of the cycle, for example, assuming that a cycle is a day, then the time period where the file receiving time is located in a cycle.

[0066] Specifically, after obtaining the file reception time of each product data file sent by each upstream system to the downstream system in multiple adjacent historical periods corresponding to the current period, the downstream system can perform statistical analysis on these file reception times to further determine the reception time distribution information corresponding to each product data file.

[0067] Exemplarily, the downstream system may divide a time period into multiple time periods, and for each time period, count the number of product data files whose file reception time falls within the time period in multiple historical periods, and determine the reception time distribution information corresponding to each product data file based on the number obtained in each time period.

[0068] Step 208: determine the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determine the target processing time of the batch processing task in the current period according to the distribution type.

[0069] Among them, the batch processing task can be a scheduled batch processing task, and the target processing time can be the scheduled time of the scheduled batch processing task, that is, the time when the scheduled batch task starts to execute. The downstream system can access the file storage directory according to the target processing time to batch process the unprocessed product data files therein.

[0070] Specifically, after determining the receiving time distribution information, the downstream system can determine the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and then determine the approximate time pattern of the product data files arriving at the downstream system in multiple historical periods according to the distribution type. Since the file receiving time of the product data files in multiple historical periods adjacent to the current period is obtained, the determined time pattern has a high correlation with the current period and can be used to predict the file receiving time of each product data file in the current period. The downstream system can then determine the target processing time of the batch processing task in the current period to ensure the timeliness of the product data file processing.

[0071] Optionally, considering that the data files of the same upstream system are more likely to arrive at the same time in different cycles, in a specific implementation, the data files under the target agreed upon by the upstream and downstream systems can be partitioned and stored according to different upstream systems, so that the file reception time can be counted separately for the data files of each upstream system, and the reception time distribution information can be determined for each upstream system based on the file reception time of the data files of different upstream systems. Then, the target processing time of the batch processing tasks in the current cycle of each upstream system can be determined based on the reception time distribution information corresponding to each upstream system. When accessing data files, the files of each upstream system are accessed according to their respective target processing times, thereby further improving the processing timeliness of the data processing files.

[0072] Step 210 , access the file storage directory according to the target processing time, and when there is a directory corresponding to the unprocessed product data file in the file storage directory, load the unprocessed product data file into the temporary data table.

[0073] The file storage directory is a directory agreed upon by the upstream system and the downstream system, and the storage space corresponding to the directory stores processed product data files and unprocessed product data files. The temporary data table is a data table used to temporarily store product parameter information before the product parameter information in the product data file is synchronized and implemented. The temporary data table can have the same structure as the data table included in the product data file.

[0074] Exemplarily, the file directory corresponding to the processed product data file and the file directory corresponding to the unprocessed product data file can be indicated by different fields. For example, when the product data file arrives at the storage space corresponding to the file storage directory, it is in an unprocessed state, and a field indicating "unprocessed" can be added to its corresponding file target, such as "1". When the product data file is successfully processed, the field can be updated to indicate "processed", such as "1". It can be understood that the field used to indicate whether it is processed here can also be in other forms, for example, the field indicating "processed" can be "done", and the present application does not limit this.

[0075] Exemplarily, processed product data files and unprocessed product data files may be stored in different folders, respectively, and different fields may be used to indicate the folder directories corresponding to the folders storing the processed product data files and the folders storing the unprocessed product data files. For example, a field indicating "unprocessed" may be added to the folder directory corresponding to the folder storing the unprocessed product data files.

[0076] Specifically, in the current cycle, when the time reaches the target processing time of the batch task, the downstream system can access the file storage directory to determine whether there is a directory corresponding to the unprocessed product data file in the file storage directory. If so, the unprocessed product data file is loaded into the temporary data table.

[0077] Exemplarily, the directory corresponding to the unprocessed product data file may be a file directory or a folder directory, that is, when any file directory in the file storage directory does not contain a field indicating completion, the product data file corresponding to the file target is determined to be an unprocessed product data file; or when there is a folder directory in the file storage directory containing a field indicating incompleteness and there are product data files in the folder corresponding to the folder directory, the product data files in the folder corresponding to the folder directory are determined to be unprocessed product data files.

[0078] Step 212, assemble the data in the temporary data table, obtain the online transaction input object, and use the online transaction input object to call the preset online transaction to synchronize the product data in the temporary data table to the formal product data table.

[0079] Among them, the preset online transaction is the product data synchronization transaction, which is used to synchronize product data to the product data formal table. The product data formal table is a data table formally used to store product parameter information, which is used to persistently store product parameter information and provide it to product R&D components for use.

[0080] Specifically, when product data enters the temporary table and becomes data to be processed, the downstream system can assemble the temporary table data into an online transaction input object to be called in subsequent steps, and then the assembled object calls the preset online transaction, so that the product data is finally stored in the warehouse.

[0081] Exemplarily, when it is detected that the product data in the temporary data table is successfully synchronized to the formal product data table, the downstream system can update the directory corresponding to the product data file corresponding to the product data, such as adding a processed field to the file target corresponding to the product data file, or moving the product data file from the folder storing unprocessed product data files to the folder storing processed product data files.

[0082] Exemplarily, when it is detected that the product data in the temporary data table is successfully synchronized to the formal product data table, the downstream system may also delete the product data in the temporary data table to further save computer resources and avoid data confusion in subsequent processing.

[0083] In the above data file processing method, when the current time switches from the previous cycle to the current cycle, the file receiving time of each product data file sent by each upstream system to the downstream system in multiple adjacent historical cycles corresponding to the current cycle is obtained, and based on the position of each file receiving time in the respective historical cycle, the receiving time distribution information corresponding to each product data file is determined. The receiving time distribution information can be dynamically determined in different cycles and can reflect the approximate law of the arrival time distribution of product data files sent by each upstream system in the recent period. The distribution type of the file receiving time in each historical cycle is determined according to the receiving time distribution information, and the distribution type of the file receiving time in the current cycle is determined according to the distribution type. The target processing time of the batch task is to access the file storage directory according to the target processing time, so that the product data files can be processed through scheduled batch processing, thereby eliminating the need to continuously monitor the arrival time of the product data files, saving computer resources of the downstream system, and because the target processing time of the scheduled batch task is dynamically determined according to the current cycle, the timeliness of data file processing can be improved. In addition, in the scheduled batch task, by loading the unprocessed product data files into a temporary data table, assembling the data in the temporary data table, obtaining the online transaction input object, and using the online transaction input object to call the preset online transaction, the processing efficiency of the data file can also be improved.

[0084] In an exemplary embodiment, Figure 3 As shown, based on the position of each file receiving time in its respective historical period, the receiving time distribution information corresponding to each product data file is determined, including:

[0085] Step 302: divide a time period into multiple time periods.

[0086] Specifically, in the process of determining the receiving time distribution information, the downstream system may divide a time period into multiple time periods, and the number of the specific divided time periods may be determined as needed. For example, assuming that a time period is one day, every 2 hours may be divided into a time period, so that a time period may be divided into 6 time periods.

[0087] Step 304 , for each time period, count the number of product data files whose file receiving time is within the time period in a plurality of adjacent historical periods to obtain the statistical number corresponding to the time period.

[0088] Specifically, for each time period, the downstream system can count the number of product data files whose file reception time is within the time period in multiple adjacent historical periods corresponding to the current period, thereby obtaining the number of products data files counted in the time period. For example, assuming that in the third time period obtained by division, the number of product data files whose reception time is within the time period in multiple adjacent historical periods corresponding to the current period is 3, 3, 5, 7, and 5, respectively, then the number of products data files counted in the time period is 23.

[0089] Step 306: Determine the receiving time distribution information corresponding to each product data file according to the statistical quantity corresponding to each time period.

[0090] Specifically, after obtaining the statistical data corresponding to each time period, the downstream system can determine the receiving time distribution information corresponding to each product data file according to these statistical quantities.

[0091] In the above embodiment, since a time period can be divided into multiple time periods, for each time period, the number of product data files whose file reception time in multiple adjacent historical periods is within the time period is counted, and then the reception time distribution information corresponding to each product data file can be quickly determined based on the statistical number corresponding to each time period, thereby improving the efficiency of determining the reception time distribution information.

[0092] In an exemplary embodiment, Figure 4 As shown, a data file processing method is provided, which is applied to Figure 1 Taking the downstream system 102 in the example as an example, the following steps are included:

[0093] Step 402, when the current time switches from the previous cycle to the current cycle, a plurality of adjacent historical cycles corresponding to the current cycle are determined.

[0094] Step 404 , according to the respective time identifiers of the respective historical periods, the respective file receiving times of the respective product data files sent by the respective upstream systems to the downstream systems in the respective historical periods are obtained from the preset receiving time data table.

[0095] Step 406, dividing a time period into a plurality of time periods.

[0096] Step 408 , for each time period, count the number of product data files whose file receiving time is within the time period in a plurality of adjacent historical periods to obtain the statistical number corresponding to the time period.

[0097] Step 410: Determine the receiving time distribution information corresponding to each product data file according to the statistical quantity corresponding to each time period.

[0098] It is understandable that the relevant explanations about steps 402 to 410 can be referred to the above embodiments, and the present application will not elaborate on them here.

[0099] Step 412: When the reception time distribution information indicates that the statistical quantity of one or more time periods accounts for a larger proportion of the total quantity of product data files than a preset threshold, it is determined that the distribution type of the file reception time in each historical period is a concentrated distribution type.

[0100] Step 414: Determine a first target time period corresponding to the concentrated distribution type from one or more time periods.

[0101] Among them, concentrated distribution means that the file receiving time of product data files in multiple adjacent historical periods is concentrated in one or more continuous time periods, so that the statistical quantity of one or more time periods accounts for a large proportion of the total number of product data files, for example, greater than a preset threshold. For example, assuming that the total number of product data files received by the downstream system in multiple adjacent historical periods is 100, and there are two consecutive time periods with statistical quantities of 40 and 50 respectively, then the distribution type of file receiving time can be considered as concentrated distribution type. The first target time period is the time period with the largest number of statistics.

[0102] Specifically, after determining the receiving time distribution information corresponding to each product data file, the downstream system can determine the statistical quantity corresponding to each time period based on the receiving time distribution information. If the statistical quantity of one or more time periods accounts for a large proportion of the total number of product data files, it can be determined that the distribution type of the file receiving time in each historical period is a concentrated distribution type, and then the first target time period corresponding to the concentrated distribution type can be determined from one or more time periods. The first target time period is the time period when the product data files are most concentratedly received.

[0103] Step 416: Determine the target processing time of the batch processing task in the current cycle according to the first target time period.

[0104] Specifically, since the first target time period is the time period when product data files are received most intensively, the downstream system can predict, based on the first target time period, that the file reception time of the product data files sent by the upstream system in the current cycle is most likely to be near the first target time period, so that the downstream system can determine the target processing time of the batch processing task in the current cycle. For example, the downstream system can select a time in the first target time period as the target processing time of the batch processing task in the current cycle, or the downstream system can select a time in the time period next to the first target time period as the target processing time of the batch processing task in the current cycle.

[0105] Step 418, access the file storage directory according to the target processing time, and when there is a directory corresponding to the unprocessed product data file in the file storage directory, load the unprocessed product data file into the temporary data table.

[0106] Step 420, calling a preset online transaction based on the temporary data table to synchronize the product data in the temporary data table to the formal product data table.

[0107] Specifically, the downstream system can assemble the data in the temporary data table, obtain the online transaction input object, and use the online transaction input object to call the preset online transaction to synchronize the product data in the temporary data table to the formal product data table.

[0108] It is understandable that the relevant explanations about steps 418 to 420 can be referred to the above embodiments, and the present application will not elaborate on them here.

[0109] In this embodiment, since the first target time period is the time period when product data files are most concentratedly received, the downstream system can roughly predict the time period in the current cycle when the file reception time of the product data files sent by the upstream system is likely to occur, and thus determine the timing of the scheduled batch task according to this time period. Compared with randomly determining the timing time, the timeliness of data file processing can be better guaranteed.

[0110] In an exemplary embodiment, determining the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determining the target processing time of the batch processing task in the current period according to the distribution type includes:

[0111] When the receiving time distribution information indicates that the statistical quantities corresponding to each time period are close to the proportion of the total number of product data files, it is determined that the distribution type of the file receiving time in each historical period is a non-concentrated distribution type; from each time period, determine the second target time period corresponding to the non-concentrated distribution type; the second target time period is the time period in which the product data file is received the latest; according to the second target time period, determine the target processing time of the batch task in the current cycle.

[0112] The non-centralized distribution means that the file receiving time of the product data files in multiple adjacent historical periods is not concentrated in one or more continuous time periods, but is evenly distributed in each time period, for example, the statistical quantity corresponding to each time period may account for a similar proportion in the total quantity of product data files. The second target time period in which the product data file is received the latest is the time period in which the file receiving time of the last product data file in multiple adjacent historical periods is located.

[0113] Specifically, after determining the receiving time distribution information corresponding to each product data file, the downstream system can determine the statistical quantity corresponding to each time period according to the receiving time distribution information. If the statistical quantity corresponding to each time period accounts for a similar proportion of the total number of product data files, it can be determined that the distribution type of the file receiving time in each historical period is a concentrated non-distributed type. When the distribution type is a non-concentrated distribution type, the downstream system can further determine the second target time period in which the product data file is received the latest according to the receiving time distribution information. The second target time period is the time period in which the file receiving time of the last product data file is located. The downstream system can predict, based on the second target time period, that the file receiving time of the last product data file sent by the upstream system in the current period is likely to be near the second target time period, so that the downstream system can determine the target processing time of the batch processing task in the current period. For example, the downstream system can select a time in the second target time period as the target processing time of the batch processing task in the current period, or the downstream system can select a time in the next time period of the second target time period as the target processing time of the batch processing task in the current period.

[0114] In the above embodiment, in this embodiment, due to the time period in which the file reception time of the last product data file is located, the downstream system can roughly predict the time period in which the file reception time of the last product data file sent by the upstream system in the current cycle is likely to occur, thereby determining the timing time of the scheduled batch task according to the time period. Compared with randomly determining the timing time, it can ensure that the product data files received in the current cycle can be processed in the current cycle.

[0115] In an exemplary embodiment, Figure 5 As shown, the data file processing method of the present application also includes a data file verification step, specifically including:

[0116] Step 502: When a product data file is received through a downstream system, determine a target upstream system to which the received product data file belongs.

[0117] Step 504: Call the interceptor corresponding to the target upstream system to intercept the received product data file, and verify the received product data file through the interceptor corresponding to the target upstream system.

[0118] Specifically, taking into account that the quality of data arriving from the upstream system is not the same, the product data file can be verified before storing it in the storage space corresponding to the preset storage directory. Since different upstream systems usually correspond to different products, an interceptor can be configured for each upstream system. The interceptor is configured with verification rules for the products within the business scope of the upstream system. When the downstream system receives the product data file, it can first determine the target upstream system to which the product data file belongs, and then call the interceptor corresponding to the upstream system to intercept the received product data file, and determine the target product to which the product data file belongs through the interceptor corresponding to the target upstream system, and then obtain the verification rules of the target product to verify the intercepted product data file to obtain a verification result, which can be a verification pass or a verification fail.

[0119] Step 506: When the verification result indicates that the verification is passed, the received product data file is written into the storage space corresponding to the file storage directory.

[0120] Specifically, when the verification result indicates that the verification is passed, the downstream system can write the received product data file into the storage space corresponding to the file storage directory. When the verification result indicates that the verification fails, the downstream system can discard the received product data file and return a prompt message of failure to the target upstream system corresponding to the product data file.

[0121] In the above embodiment, since the product data file can be intercepted and verified by the interceptor, product data with poor data quality can be effectively prevented from being stored.

[0122] In an exemplary embodiment, when there is a directory corresponding to an unprocessed product data file in the file storage directory, the unprocessed product data file is loaded into a temporary data table, including: when any file directory in the file storage directory does not contain a field indicating completion, the product data file corresponding to the file target is determined as an unprocessed product data file.

[0123] In this embodiment, processed product data files and unprocessed product data files are distinguished by whether the file directory contains a field indicating completion. When the downstream system accesses the file storage directory, it can determine whether the file directory of each product data file contains a field indicating completion, such as "done". If the file directory does not contain a field indicating completion, the product data file corresponding to the file target is determined to be an unprocessed product data file.

[0124] In the above embodiment, the unprocessed product files can be quickly determined through the fields indicating completion in the file directory, thereby improving the processing efficiency of the product data files.

[0125] In an exemplary embodiment, when there is a directory corresponding to an unprocessed product data file in the file storage directory, the unprocessed product data file is loaded into a temporary data table, including: when there is a folder directory containing a field indicating incompleteness in the file storage directory and there are product data files in the folder corresponding to the folder directory, the product data files in the folder corresponding to the folder directory are determined as unprocessed product data files.

[0126] In this embodiment, processed product data files and unprocessed product data files are stored in different folders, and the folder directory corresponding to the folder storing the unprocessed product data files contains a field indicating incomplete, so that when the downstream system accesses the file storage directory, it can determine whether there is a folder directory containing a field indicating incomplete in the file storage directory. If so, further determine whether there is a product data file in the folder corresponding to the folder directory. If so, determine the product data file in the folder corresponding to the folder directory as an unprocessed product data file.

[0127] In the above embodiment, the unprocessed product files can be quickly determined through the field indicating unfinished in the folder directory, thereby improving the processing efficiency of the product data files.

[0128] In a specific embodiment, Figure 6 FIG. 1 is a schematic diagram of the overall process of the data file processing method of the present application. Figure 6 This solution aims to solve the problem that when the upstream system reports data to the downstream system through the data line (.dat file), the monitoring time of the predecessor job that relies on the batch job is very long due to time uncertainty, thus causing a waste of computer resources in the downstream system.

[0129] Each upstream system pushes the relevant product data files to the file storage directory specified by the downstream system through nft by submitting .dat files at irregular intervals (generally the nas path agreed upon by the upstream and downstream systems). After the files arrive at the downstream system, the downstream system defines a scheduled batch processing task through a general batch processing framework, such as CtrlM, Luban or other methods. The scheduled batch processing task can execute the shell script in the specified file storage directory, and the following steps are performed through the shell script:

[0130] 1. When the current time switches from the previous date to the current date, multiple adjacent historical dates corresponding to the current date are determined. The downstream system obtains the respective file receiving time of each product data file sent by each upstream system to the downstream system within multiple adjacent historical dates before the current date from the preset receiving time data table according to the respective time identifiers of each historical date. Based on the position of each file receiving time within its respective historical period, the receiving time distribution information corresponding to each product data file is determined. According to the receiving time distribution information, the distribution type of the file receiving time within each historical period is determined. According to the distribution type, the target processing time of the scheduled batch processing task within the current date is determined.

[0131] 2. The downstream system accesses the file storage directory agreed with the upstream system according to the target processing time. The product data files in the file storage directory are stored according to the dates under the directories of different upstream systems (multi-entity distinction), such as: / home / ap / nas / input / A1521 / DG004 / 20240501,20240502. When there is an unprocessed date directory under a certain date, the lightning tool is used to load the .dat file into the newly created temporary data table of the downstream system.

[0132] 3. When the data enters the temporary data table, it becomes data to be processed. The data in the temporary data table is assembled and finally assembled into the online transaction input parameter object to be called in the subsequent steps.

[0133] 4. Use the object assembled in step 3 to call the system's preset online transaction (product data synchronization transaction) to synchronize the product data in the temporary data table to the formal product data table, so that the product data can finally be stored in the warehouse.

[0134] 5. The above processing process is serial processing according to the product dimension, that is, processing one product at a time. When a product is successfully processed (the online transaction processing is successfully called), the temporary table data is deleted (indicating that the current product has been successfully processed).

[0135] 6. When all the unprocessed subsidiaries + date directories in this batch are completed, modify the date file name, such as: 20240501->20240501done.

[0136] For the file directories processed by the above methods, regular batch processing is performed to clean up the files to avoid the accumulation of files pushed by the upstream system.

[0137] In this embodiment, the many uncertainties in the data reported by the upstream system are circumvented to avoid the general reliance on batch processing to monitor files for a long time and waste system resources. At the same time, this solution is more flexible and uses a unified method to process data files reported by all different subsidiaries without distinguishing between specific businesses.

[0138] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0139] Based on the same inventive concept, the embodiment of the present application also provides a data file processing device for implementing the data file processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more data file processing device embodiments provided below can refer to the limitations of the data file processing method above, and will not be repeated here.

[0140] In an exemplary embodiment, Figure 7 As shown, a data file processing device 700 is provided, comprising:

[0141] The file receiving time acquisition module 702 is used to determine a plurality of adjacent historical periods corresponding to the current period when the current time switches from the previous period to the current period; and obtain the respective file receiving time of each product data file sent by each upstream system to the downstream system in each historical period from the preset receiving time data table according to the respective time identifiers of each historical period;

[0142] The receiving time distribution information determining module 704 is used to determine the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period;

[0143] The target processing time determination module 706 is used to determine the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determine the target processing time of the batch processing task in the current period according to the distribution type;

[0144] The file loading module 708 is used to access the file storage directory according to the target processing time, and when there is a date directory corresponding to the unprocessed product data file in the file storage directory, load the unprocessed product data file into the temporary data table;

[0145] The product data synchronization module 710 is used to assemble the data in the temporary data table, obtain the online transaction input object, and use the online transaction input object to call the preset online transaction to synchronize the product data in the temporary data table to the product data formal table.

[0146] The above-mentioned data file processing device, when the current time switches from the previous cycle to the current cycle, obtains the respective file receiving times of the respective product data files sent by the upstream systems to the downstream systems in the multiple adjacent historical cycles corresponding to the current cycle, and determines the receiving time distribution information corresponding to the respective product data files based on the positions of the receiving times of the respective files in the respective historical cycles. The receiving time distribution information can be dynamically determined in different cycles and can reflect the approximate law of the arrival time distribution of the product data files sent by the upstream systems in the recent period. The distribution type of the file receiving time in each historical cycle is determined according to the receiving time distribution information, and the distribution type of the file receiving time in the current cycle is determined according to the distribution type. The target processing time of the batch task is to access the file storage directory according to the target processing time, so that the product data files can be processed through scheduled batch processing, thereby eliminating the need to continuously monitor the arrival time of the product data files, saving computer resources of the downstream system, and because the target processing time of the scheduled batch task is dynamically determined according to the current cycle, the timeliness of data file processing can be improved. In addition, in the scheduled batch task, by loading the unprocessed product data files into a temporary data table, assembling the data in the temporary data table, obtaining the online transaction input object, and using the online transaction input object to call the preset online transaction, the processing efficiency of the data file can also be improved.

[0147] In one of the embodiments, the receiving time distribution information determination module is also used to: divide a time period into multiple time periods; for each time period, count the number of product data files whose file receiving time is within the time period in multiple historical periods; and determine the receiving time distribution information corresponding to each product data file based on the number obtained in each time period.

[0148] In one of the embodiments, the module for determining the receiving time distribution information is also used to: when the receiving time distribution information indicates that the statistical quantity of one or more time periods accounts for a proportion greater than a preset threshold in the total quantity of product data files, determine that the distribution type of the file receiving time in each historical period is a concentrated distribution type; determine a first target time period corresponding to the concentrated distribution type from one or more time periods; the first target time period is the time period with the largest number of statistics; and determine the target processing time of the batch task in the current period based on the first target time period.

[0149] In one of the embodiments, the module for determining the receiving time distribution information is also used for: when the receiving time distribution information indicates that the statistical quantity corresponding to each time period accounts for a close proportion of the total number of product data files, determining that the distribution type of the file receiving time in each historical period is a non-concentrated distribution type; determining from each time period a second target time period corresponding to the non-concentrated distribution type; the second target time period is the time period in which the product data file is received the latest; and determining the target processing time of the batch task in the current period based on the second target time period.

[0150] In one of the embodiments, the device also includes a verification module, which is used to: when a product data file is received through a downstream system, determine the target upstream system to which the received product data file belongs; call the interceptor corresponding to the target upstream system to intercept the received product data file, and verify the received product data file through the interceptor corresponding to the target upstream system; when the verification result indicates that the verification is passed, write the received product data file into the storage space corresponding to the file storage directory.

[0151] In one of the embodiments, the file loading module is further used to: when any file directory in the file storage directory does not contain a field indicating completion, determine the product data file corresponding to the file target as an unprocessed product data file; or when there is a folder directory in the file storage directory containing a field indicating incomplete and there are product data files in the folder corresponding to the folder directory, determine the product data files in the folder corresponding to the folder directory as unprocessed product data files.

[0152] Each module in the above data file processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0153] In an exemplary embodiment, a computer device is provided. The computer device may be a server for implementing the downstream system described above. The internal structure diagram thereof may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store product data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the data file processing method provided in the embodiment of the present application is implemented.

[0154] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the data file processing method in any of the above embodiments are implemented.

[0156] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data file processing method in any of the above embodiments are implemented.

[0157] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the data file processing method in any of the above embodiments when executed by a processor.

[0158] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0160] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data file processing method, characterized in that: The method comprises: When the current time switches from the previous cycle to the current cycle, multiple adjacent historical cycles corresponding to the current cycle are determined; According to the respective time identifiers of the respective historical periods, the respective file receiving times of the respective product data files sent by the respective upstream systems to the downstream systems in the respective historical periods are obtained from the preset receiving time data table; Determine the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period; Determine the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determine the target processing time of the batch processing task in the current period according to the distribution type; Accessing the file storage directory according to the target processing time, and when there is a directory corresponding to the unprocessed product data file in the file storage directory, loading the unprocessed product data file into a temporary data table; The data in the temporary data table is assembled to obtain an online transaction input parameter object, and the online transaction input parameter object is used to call a preset online transaction to synchronize the product data in the temporary data table to the formal product data table.

2. The method according to claim 1, characterized in that The determining of the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period includes: Divide a time period into multiple time periods; For each time period, counting the number of product data files whose file receiving time in the plurality of adjacent historical periods is within the time period, to obtain the statistical number corresponding to the time period; According to the statistical quantities corresponding to each time period, the receiving time distribution information corresponding to each product data file is determined.

3. The method according to claim 2, characterized in that The determining the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determining the target processing time of the batch processing task in the current period according to the distribution type, includes: When the reception time distribution information indicates that the statistical quantity of one or more time periods accounts for a proportion greater than a preset threshold in the total quantity of product data files, determining that the distribution type of the file reception time in each historical period is a concentrated distribution type; Determine a first target time period corresponding to the concentrated distribution type from the one or more time periods; the first target time period is the time period with the largest number of statistics; A target processing time for the batch processing task in the current cycle is determined according to the first target time period.

4. The method according to claim 2, characterized in that: The determining the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determining the target processing time of the batch processing task in the current period according to the distribution type, includes: When the reception time distribution information indicates that the statistical quantities corresponding to the respective time periods account for a similar proportion in the total quantity of the product data files, it is determined that the distribution type of the file reception time in each historical period is a non-centralized distribution type; Determine a second target time period corresponding to the non-centralized distribution type from each of the time periods; the second target time period is the latest time period in which the product data file is received; According to the second target time period, a target processing time for the batch processing task in the current cycle is determined.

5. The method according to claim 1, characterized in that: The method further comprises: When a product data file is received through the downstream system, determining a target upstream system to which the received product data file belongs; Calling the interceptor corresponding to the target upstream system to intercept the received product data file, and verifying the received product data file through the interceptor corresponding to the target upstream system; When the verification result indicates that the verification is passed, the received product data file is written into the storage space corresponding to the file storage directory.

6. The method according to claim 1, characterized in that When there is a directory corresponding to the unprocessed product data file in the file storage directory, loading the unprocessed product data file into the temporary data table includes: When any file directory in the file storage directory does not contain a field indicating completion, determining the product data file corresponding to the file target as an unprocessed product data file; or When there is a folder directory containing a field indicating incompleteness in the file storage directory and there is a product data file in a folder corresponding to the folder directory, the product data file in the folder corresponding to the folder directory is determined as an unprocessed product data file.

7. A data file processing device, characterized in that: The device comprises: The file receiving time acquisition module is used to determine the multiple adjacent historical periods corresponding to the current period when the current time switches from the previous period to the current period; according to the time identifier of each historical period, obtain the file receiving time of each product data file sent by each upstream system to the downstream system in each historical period from the preset receiving time data table; A receiving time distribution information determination module, used to determine the receiving time distribution information corresponding to each product data file based on the position of the receiving time of each file in the respective historical period; A target processing time determination module, used to determine the distribution type of the file receiving time in each historical period according to the receiving time distribution information, and determine the target processing time of the batch processing task in the current period according to the distribution type; A file loading module, configured to access a file storage directory according to the target processing time, and when a date directory corresponding to an unprocessed product data file exists in the file storage directory, load the unprocessed product data file into a temporary data table; The product data synchronization module is used to assemble the data in the temporary data table, obtain the online transaction input object, and use the online transaction input object to call the preset online transaction to synchronize the product data in the temporary data table to the formal product data table.

8. The device according to claim 7, characterized in that The receiving time distribution information determination module is also used to: divide a time period into multiple time periods; for each time period, count the number of product data files whose file receiving time in the multiple adjacent historical periods is within the time period, and obtain the statistical number corresponding to the time period; determine the receiving time distribution information corresponding to each product data file based on the statistical number corresponding to each time period.

9. The device according to claim 8, characterized in that The receiving time distribution information determination module is also used to: when the receiving time distribution information indicates that the statistical quantity of one or more time periods accounts for a proportion greater than a preset threshold in the total quantity of product data files, determine that the distribution type of the file receiving time in each historical period is a concentrated distribution type; from the one or more time periods, determine the first target time period corresponding to the concentrated distribution type; the first target time period is the time period with the largest number of statistics; and determine the target processing time of the batch processing task in the current period based on the first target time period.

10. The device according to claim 8, characterized in that The receiving time distribution information determination module is also used for: when the receiving time distribution information indicates that the statistical quantity corresponding to each time period accounts for a similar proportion in the total number of product data files, determining that the distribution type of the file receiving time in each historical period is a non-concentrated distribution type; determining a second target time period corresponding to the non-concentrated distribution type from each of the time periods; the second target time period is the latest time period for receiving the product data file; and determining the target processing time of the batch processing task in the current period based on the second target time period.

11. The device according to claim 7, characterized in that The device also includes a verification module, which is used to: when a product data file is received through the downstream system, determine the target upstream system to which the received product data file belongs; call the interceptor corresponding to the target upstream system to intercept the received product data file, and verify the received product data file through the interceptor corresponding to the target upstream system; when the verification result indicates that the verification is passed, write the received product data file into the storage space corresponding to the file storage directory.

12. The device according to claim 7, characterized in that The file loading module is further used to: when any file directory in the file storage directory does not contain a field indicating completion, determine the product data file corresponding to the file target as an unprocessed product data file; Alternatively, when there is a folder directory containing a field indicating incompleteness in the file storage directory and there is a product data file in a folder corresponding to the folder directory, the product data file in the folder corresponding to the folder directory is determined as an unprocessed product data file.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images