File processing method and device, equipment, storage medium and program product

Abstract

The invention provides a file processing method and device, equipment, a storage medium and a program product, and can be applied to the technical field of computers. The file processing method comprises the following steps: in response to a plurality of received to-be-processed files, according to file attribute information of each to-be-processed file and processing speed information of each processing program in a processing program sequence, determining the processing duration of each processing program for each to-be-processed file, the processing program sequence processes each to-be-processed file according to an arrangement sequence of a plurality of processing programs, and each processing program processes one to-be-processed file at the same moment; based on an optimization algorithm, the processing sequence of the to-be-processed files is determined according to the processing duration of each processing program for each to-be-processed file, and a target function of the optimization algorithm represents the total processing duration for processing the to-be-processed files; and processing the plurality of to-be-processed files by utilizing the processing program sequence according to the processing sequence of the plurality of to-be-processed files.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and more particularly to a file processing method, device, equipment, storage medium and program product. BACKGROUND

[0002] A file processing system receives a large number of files every day. After receiving the files, the file processing system processes the to-be-processed files by using a processing program sequence composed of different multiple processing programs, and loads into a warehouse.

[0003] In the process of implementing the concept of the present application, it is found that at least the following problems exist in the related art: The related file processing method causes a large number of processing programs in the processing program sequence to be in an idle state, which not only leads to low system resource rate of the file processing system, but also leads to low file processing efficiency. SUMMARY

[0004] In view of the above problems, the present application provides a file processing method, device, equipment, storage medium and program product.

[0005] According to a first aspect of the present application, a file processing method is provided, comprising: in response to receiving a plurality of to-be-processed files, determining, according to file attribute information of each of the to-be-processed files and processing speed information of each processing program in a processing program sequence, a processing duration of each of the processing programs for each of the to-be-processed files, the processing program sequence processing each of the to-be-processed files according to an arrangement order of the plurality of processing programs, and each of the processing programs processing one of the to-be-processed files at the same time; determining, based on an optimization algorithm, a processing order of the plurality of to-be-processed files according to the processing duration of each of the processing programs for each of the to-be-processed files, a target function of the optimization algorithm representing a total processing duration of processing the plurality of to-be-processed files; and processing, by using the processing program sequence, the plurality of to-be-processed files according to the processing order of the plurality of to-be-processed files.

[0006] According to an embodiment of the present application, the optimization algorithm includes a constraint condition, the constraint condition including a first limiting term and a second limiting term, the first limiting term being used to limit processing each of the to-be-processed files according to the arrangement order of the plurality of processing programs, and the second limiting term being used to limit each of the processing programs processing one of the to-be-processed files at the same time; and the determining, based on the optimization algorithm, the processing order of the plurality of to-be-processed files according to the processing duration of each of the to-be-processed files in each of the processing programs, comprises: determining, based on the optimization algorithm, the processing order of the plurality of to-be-processed files satisfying the constraint condition with a minimum target function value of the target function as a target.

[0007] According to an embodiment of this application, the above-described determination of the processing order of multiple files to be processed under the constraint conditions, based on the above-described optimization algorithm and with the goal of minimizing the objective function value of the objective function, includes: optimizing the processing start time of each processing program for each file to be processed based on the above-described optimization algorithm and with the goal of minimizing the objective function value of the objective function, and determining the processing start time of each processing program for each file to be processed that satisfies the constraint conditions; and determining the processing order of multiple files to be processed based on the processing start time of the first processing program in the processing program sequence for each file to be processed.

[0008] According to an embodiment of this application, determining the processing time of each processing program for each file to be processed based on the file attribute information of each file to be processed and the processing speed information of each processing program in the processing program sequence includes: evaluating the file complexity of each file to be processed based on the file attribute information of each file to be processed to obtain the file complexity information of each file to be processed; and inputting the file complexity information of the file to be processed and the processing speed information of each processing program into a trained duration prediction model for each file to be processed to obtain the processing time of each processing program for the file to be processed.

[0009] According to an embodiment of this application, the aforementioned files to be processed include a business data table. The process of evaluating the file complexity of each file to be processed based on the aforementioned file attribute information to obtain file complexity information for each file to be processed includes: evaluating the data volume complexity of the business data table based on the number of rows in the aforementioned file attribute information to determine the data volume complexity information of the business data table; evaluating the structural complexity of the business data table based on the number of fields and field types in the aforementioned file attribute information to determine the structural complexity information of the business data table; and determining the file complexity information of the business data table based on the aforementioned data volume complexity information and the aforementioned structural complexity information.

[0010] According to an embodiment of the present application, the processing procedure sequence comprises M processing procedures, the to-be-processed files are N, and processing the to-be-processed files according to the processing order comprises repeatedly performing the following operations until the N to-be-processed files are all processed: in response to completion of processing of an nth to-be-processed file by an mth processing procedure in the processing procedure sequence, sending the nth to-be-processed file to an (m+1)th processing procedure, 1≤m≤M, 1≤n≤N; determining an (n+1)th to-be-processed file according to the processing order; and in a case where it is determined that the (m-1)th processing procedure has completed processing of the (n+1)th to-be-processed file, sending the (n+1)th to-be-processed file to the mth processing procedure.

[0011] According to an embodiment of the present application, the file processing method further comprises: after processing of the to-be-processed files is completed, obtaining an actual processing duration of each processing procedure for each to-be-processed file; and adjusting parameters of the duration prediction model by using file complexity information of each to-be-processed file, processing speed information of each processing procedure, and the actual processing duration of each processing procedure for each to-be-processed file.

[0012] A second aspect of the present application provides a file processing apparatus, characterized in that the apparatus comprises: a determination module configured to, in response to receiving a plurality of to-be-processed files, determine a processing duration of each processing procedure for each to-be-processed file according to file attribute information of each to-be-processed file and processing speed information of each processing procedure in a processing procedure sequence, the processing procedure sequence processing each to-be-processed file according to an arrangement order of the processing procedures, and each processing procedure processing one to-be-processed file at the same time; an optimization module configured to determine a processing order of the to-be-processed files based on an optimization algorithm and the processing duration of each processing procedure for each to-be-processed file, a target function of the optimization algorithm representing a total processing duration of processing the to-be-processed files; and a processing module configured to process the to-be-processed files according to the processing order by using the processing procedure sequence.

[0013] A third aspect of the present application provides an electronic device, comprising: one or more processors; and a memory storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement steps of the method.

[0014] A fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement steps of the method.

[0015] The fifth aspect of the present application further provides a computer program product comprising computer programs or instructions, which, when executed by a processor, implement the steps of the above method.

[0016] According to the embodiments of the present application, by considering the file attribute information of each file and the processing speed information of each processing program, the processing time of each processing program for each to-be-processed file is predicted in a fine-grained manner. Moreover, based on the processing time of each processing program for each to-be-processed file, the processing order of the plurality of to-be-processed files is optimized with the total processing time of the plurality of to-be-processed files as the objective function of the optimization algorithm, the processing order of the plurality of to-be-processed files is flexibly configured, and the idle waiting time of the processing program is reduced, thereby not only improving the system resource utilization rate, but also improving the processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present application will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 An application scenario diagram of a file processing method, apparatus, device, storage medium and program product according to embodiments of the present application is schematically shown;

[0019] Figure 2 A flowchart of a file processing method according to embodiments of the present application is schematically shown;

[0020] Figure 3 A schematic diagram of processing a to-be-processed file according to embodiments of the present application is schematically shown;

[0021] Figure 4 A schematic diagram of complexity evaluation of a business data table according to embodiments of the present application is schematically shown;

[0022] Figure 5 A structural block diagram of a file processing apparatus according to embodiments of the present application is schematically shown; and

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a file processing method according to embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] The file processing system receives a large number of files every day. After receiving the files, the system uses a sequence of multiple processing programs to process them sequentially and load them into a database.

[0029] However, related file processing methods usually take the process sequence as a whole and use the process sequence to process the files to be processed serially. The process sequence can only process the next file after processing the first file. At this time, only one process in the process sequence is working, which will leave a large number of process programs in the process sequence idle. This will not only lead to low system resource utilization of the file processing system, but also low file processing efficiency.

[0030] Embodiments of the present application provide a file processing method, in response to receiving a plurality of to-be-processed files, determining, according to file attribute information of each to-be-processed file and processing speed information of each processing program in a processing program sequence, a processing duration of each processing program for each to-be-processed file, processing each to-be-processed file by the processing program sequence in the arrangement order of a plurality of processing programs, each processing program processing one to-be-processed file at the same time; determining, based on an optimization algorithm, a processing order of the plurality of to-be-processed files according to the processing duration of each processing program for each to-be-processed file, a target function of the optimization algorithm representing a total processing duration of processing the plurality of to-be-processed files; and processing the plurality of to-be-processed files according to the processing order of the plurality of to-be-processed files by using the processing program sequence.

[0031] Embodiments of the present disclosure determine the processing order of the to-be-processed files by taking the total processing duration of the plurality of to-be-processed files as the target function, realize flexible configuration of the processing order of the plurality of files, and process the to-be-processed files by taking the processing program as the minimum file processing unit according to the processing order, which can reduce the idle waiting duration of the processing program, not only improve the system resource utilization rate, but also improve the processing efficiency.

[0032] Figure 1 An application scenario diagram of the file processing method, apparatus, device, storage medium and program product according to embodiments of the present application is schematically shown.

[0033] As shown in Figure 1 The application scenario according to the embodiments can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] A user can use the first terminal device 101, the second terminal device 102, the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0035] The first terminal device 101, the second terminal device 102, the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0036] The server 105 can be a server providing various services, for example, a background management server providing support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back a processing result (for example, a webpage, information, or data, or the like, obtained or generated according to a user request) to a terminal device.

[0037] It should be noted that the file processing method provided in the embodiments of the present application can generally be executed by the server 105. Correspondingly, the file processing apparatus provided in the embodiments of the present application can generally be arranged in the server 105. The file processing method provided in the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the file processing apparatus provided in the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers in

[0039] The file processing method according to the embodiments of the present application will be described in detail below based on the scenario described in Figure 1 Figures 2-4

[0040] Figure 2 An illustrative flowchart of the file processing method according to the embodiments of the present application is shown.

[0041] As shown in Figure 2 , the file processing of this embodiment includes operations S210 to S230.

[0042] At operation S210, in response to receiving a plurality of to-be-processed files, the processing time length of each processing program for each to-be-processed file is determined according to the file attribute information of each to-be-processed file and the processing speed information of each processing program in the processing program sequence.

[0043] At operation S220, based on an optimization algorithm, the processing order of the plurality of to-be-processed files is determined according to the processing time length of each processing program for each to-be-processed file.

[0044] ​​In operation S230, the plurality of files to be processed are processed according to the processing order of the plurality of files to be processed by using the processing program sequence.

[0045] According to embodiments of the present application, when processing files by using a processing program sequence, the processing process is limited by processing rules and program processing capability. The processing rules may, for example, be that the processing program sequence processes each file to be processed according to the arrangement order of the plurality of processing programs, and the program processing capability limit may be that each processing program can only process one file to be processed at the same time.

[0046] In an embodiment, the processing of the file involves multiple processing stages such as decompression, preprocessing, and importing into a database, and each processing stage corresponds to a processing program. These processing programs need to be executed in sequence, and the execution order is the arrangement order corresponding to the multiple processing stages.

[0047] According to embodiments of the present application, under the limitations of processing rules and program processing capability, an unreasonable file processing strategy can cause programs to be idle while files are waiting in the queue for processing, reducing the efficiency of file processing. Therefore, in order to improve the efficiency of file processing, a file processing strategy with shorter time consumption can be determined before processing the plurality of files by using the processing program sequence, and the plurality of files can be processed by using the file processing strategy to reduce the time length consumed by processing the plurality of files.

[0048] According to embodiments of the present application, before determining the file processing strategy, the processing time of each processing program for each file to be processed can be determined first. In embodiments of the present application, the processing time of the processing program for the file to be processed can be predicted according to the file attribute information and the processing speed information of the program. For example, a deep learning model can be used to predict the processing time.

[0049] According to embodiments of the present application, after obtaining the processing time of each processing program for each file to be processed, an optimization algorithm can be used to optimize the processing order of the plurality of files to be processed by taking the total processing time of the plurality of files to be processed as a target function. Specifically, in the optimization process, the goal is to minimize the total processing time to obtain a processing order with the shortest total processing time, thereby improving the efficiency of file processing.

[0050] According to the embodiments of the present application, when a plurality of files are processed by using a processing program sequence, if the processing of one file by using the processing program sequence is completed, the processing of another file is performed, which causes a large number of processing programs to be in an idle waiting state. Therefore, in order to ensure that the total processing time of processing a plurality of to-be-processed files is reduced under the processing rule restriction, the embodiments of the present application take a processing program as a minimum processing unit, and after the processing of one file by a certain processing program in the processing program sequence is completed, the processing of the next file can be performed without processing the files by using the processing program sequence as a whole, so as to reduce the time length of the processing programs in the idle state.

[0051] According to the embodiments of the present application, the processing time of each to-be-processed file by each processing program is predicted in a fine-grained manner by considering the file attribute information of each file and the processing speed information of each processing program. Moreover, based on the processing time of each to-be-processed file by each processing program, the processing order of the plurality of to-be-processed files when the processing program is taken as a minimum file processing unit is optimized by taking the total processing time of the plurality of to-be-processed files as an objective function of an optimization algorithm, so as to realize the flexible configuration of the processing order of the plurality of to-be-processed files, and further reduce the idle waiting time of the processing programs, thereby not only improving the system resource utilization rate, but also improving the processing efficiency.

[0052] According to the embodiments of the present application, the processing program sequence includes M processing programs, and the to-be-processed files are N. The plurality of to-be-processed files are processed according to the processing order of the plurality of to-be-processed files by using the processing program sequence, which includes repeatedly performing the following operations until the N to-be-processed files are all processed: in response to the processing of the nth to-be-processed file by the mth processing program in the processing program sequence being completed, the nth to-be-processed file is sent to the m+1th processing program, 1≤m≤M, and 1≤n≤N; the n+1th to-be-processed file is determined according to the processing order; and in the case where the processing of the n+1th to-be-processed file by the m-1th processing program is completed, the n+1th to-be-processed file is sent to the mth processing program.

[0053] In the case where the processing rule restriction and the program processing capability restriction exist, and the processing process does not allow for queuing, the processing programs can only process the plurality of to-be-processed files in the processing order of the to-be-processed files in turn. For example, the processing order of the to-be-processed files N1, N2 and N3 is N1, N2 and N3, the processing order of the to-be-processed files by the processing programs M1 and M2 is N1, N2 and N3, and the arrangement order of M1 and M2 is M1, M2. M2 can only process N3 after the processing of N3 by M1 is completed.

[0054] In the embodiments of the present application, the file processing unit is the processing program, after the processing of a file is completed by a processing program, the next file can be processed according to the processing order, so as to reduce the idle time of the processing program. Taking the mth processing program as an example, after the mth processing program completes the processing of the nth to-be-processed file, the mth processing program is in an idle state, at this time, the mth processing program can be used to process the (n+1)th to-be-processed file.

[0055] According to the embodiments of the present application, since the file processing process is limited by the processing rule, before the (n+1)th to-be-processed file is processed by the mth processing program, it is necessary to confirm whether the (m-1)th processing program has completed the processing of the (n+1)th to-be-processed file, and only in the case that the (m-1)th processing program has completed the processing of the (n+1)th to-be-processed file, the (n+1)th to-be-processed file can be sent to the mth processing program to process the (n+1)th to-be-processed file by the mth processing program.

[0056] According to the embodiments of the present application, by taking the processing program as the minimum file processing unit, each processing program can process different files in parallel according to the processing order of the to-be-processed files, the idle time of the processing program is reduced, and the file processing efficiency is improved.

[0057] Figure 3 A schematic diagram of processing to-be-processed files according to the embodiments of the present application is shown.

[0058] As shown in Figure 3 , the processing program sequence includes three processing programs, and the arrangement order of the three processing programs is processing program 1, processing program 2, and processing program 3. The processing order of the to-be-processed files is to-be-processed file 1, to-be-processed file 2, and to-be-processed file 3.

[0059] As shown in Figure 3 , in the process that the processing program 2 processes the to-be-processed file 1 and the processing program 1 processes the to-be-processed file 2, the to-be-processed file 3 is in a waiting processing state, and the processing program 3 is in an idle state.

[0060] As shown in Figure 3 , after the processing program 2 completes the processing of the to-be-processed file 1, the to-be-processed file 1 is sent to the processing program 3, and the to-be-processed file 1 is processed by the processing program 3. At this time, according to the processing order, it is confirmed that the next to-be-processed file of the processing program 2 is the to-be-processed file 2, but since the processing program 1 has not completed the processing of the to-be-processed file 2, the to-be-processed file 2 cannot be sent to the processing program 2, the processing program 2 is in an idle state, and the to-be-processed file 3 is in a waiting state.

[0061] As shown in Figure 3As shown, after the processing program 1 completes processing on the to-be-processed file 2, the to-be-processed file 2 is sent to the processing program 2, and the to-be-processed file 2 is processed by the processing program 2, and meanwhile, the to-be-processed file 3 can be sent to the processing program 1, and the to-be-processed file 3 is processed by the processing program 1.

[0062] According to an embodiment of the present application, based on the optimization algorithm, the processing order of the plurality of to-be-processed files is determined according to the processing time length of each to-be-processed file in each processing program, comprising: based on the optimization algorithm, taking the minimization of the objective function value of the objective function as the goal, determining the processing order of the plurality of to-be-processed files that meet the constraint condition.

[0063] According to an embodiment of the present application, since the file processing system has processing rule restrictions and program processing capacity restrictions when processing files by using the processing program sequence. Therefore, when optimizing the processing order of the plurality of to-be-processed files by using the optimization algorithm, the processing rule restrictions and the program processing capacity restrictions can be set as constraint conditions of the optimization algorithm accordingly, so as to improve the accuracy of the determined processing order.

[0064] According to an embodiment of the present application, the optimization algorithm includes a constraint condition, the constraint condition includes a first restriction term and a second restriction term, the first restriction term is used to limit processing each to-be-processed file according to the arrangement order of the plurality of processing programs, and the second restriction term is used to limit each processing program to process one to-be-processed file at the same time. The processing rule restrictions are realized by the first restriction term, and the program processing capacity restrictions are realized by the second restriction term, so that the obtained processing order can meet the processing rule restrictions and the program processing capacity restrictions.

[0065] According to an embodiment of the present application, since the objective function represents the total processing time length of processing the plurality of to-be-processed files, in order to improve the file processing efficiency, the optimization goal of the optimization algorithm can be set as taking the minimization of the objective function value as the goal, and then determining the processing order that meets the constraint condition and has the shortest total processing time length.

[0066] According to an embodiment of the present application, the restrictions when processing the to-be-processed files by using the processing program sequence are optimized as constraint conditions, so as to improve the accuracy of determining the file processing order.

[0067] According to an embodiment of the present application, based on an optimization algorithm, a processing sequence of the plurality of to-be-processed files is determined under the condition of meeting a constraint condition, with a target function value of a target function being minimized as a target, comprising: based on the optimization algorithm, with the target function value of the target function being minimized as the target, a processing start time of each processing program for each to-be-processed file is optimized, and the processing start time of each processing program for each to-be-processed file meeting the constraint condition is determined; and the processing sequence of the plurality of to-be-processed files is determined according to the processing start time of each to-be-processed file at a first processing program in the processing program sequence.

[0068] According to an embodiment of the present application, since the processing time of each processing program for each to-be-processed file has been obtained, when optimizing, the processing start time of each processing program for each to-be-processed file can be optimized to determine the processing sequence of the plurality of to-be-processed files.

[0069] According to an embodiment of the present application, since the target function of the optimization algorithm is the total processing time, and the processing start time of the first processing program for the first to-be-processed file is 00:00, the total processing time can be determined based on the processing completion time of the last processing program for the last to-be-processed file. Specifically, the target function can be shown in the following formula (1):

[0070] (1) ;

[0071] wherein, is the processing start time of the mth program for the nth to-be-processed file, is the processing time of the mth program for the nth to-be-processed file, n is the file size of the nth to-be-processed file, M is the file download speed of the mth program, is the download time of the mth program for the nth to-be-processed file, and max represents the maximum value. By the processing completion time of the mth program for each to-be-processed file is calculated, and the total processing time is determined based on the maximum value in the plurality of processing completion times.

[0072] According to an embodiment of the present application, the constraint condition includes a first limiting term and a second limiting term. Specifically, the first limiting term can be shown in the following formula (2):

[0073] (2) ;

[0074] wherein, is the processing start time of the mth program for the nth to-be-processed file, is the processing time of the mth program for the nth to-be-processed file, a download time length of the mth program for the nth to-be-processed file, a processing start time of an (m+1)th program for the nth to-be-processed file, so that the first restriction item can be used to limit the processing end time of the mth processing program for the nth to-be-processed file to be earlier than the processing start time of the (m+1)th processing program for the nth to-be-processed file.

[0075] According to the embodiments of the present application, since the arrangement order of the nth to-be-processed file can only be before or after the kth to-be-processed file, the second restriction item can be determined by means of 0-1 programming constraints. Specifically, the second restriction item can be shown in the following formula (3):

[0076] (3) ;

[0077] wherein, is a 0-1 programming variable, indicates that the processing order of the ith file is after the kth file, indicates that the processing order of the ith file is before the kth file, Q is a sufficiently large positive real number, and specifically can be 99999, is a processing start time of the mth program for the kth file, is a processing time length of the mth program for the kth file, is a download time length of the mth processing program for the kth file.

[0078] According to the embodiments of the present application, by optimizing the processing start time of each processing program for each to-be-processed file by using an optimization algorithm, the accuracy of the determined processing order can be improved, and further the processing efficiency of processing the plurality of to-be-processed files according to the processing order can be improved.

[0079] According to the embodiments of the present application, the processing time length of each processing program for each to-be-processed file is determined according to the file attribute information of each to-be-processed file and the processing speed information of each processing program in the processing program sequence, including: based on the file attribute information of each to-be-processed file, the file complexity of each to-be-processed file is evaluated to obtain file complexity information of each to-be-processed file; for each to-be-processed file, the file complexity information of the to-be-processed file and the processing speed information of each processing program are input into a trained time length prediction model to obtain the processing time length of each processing program for the to-be-processed file.

[0080] According to an embodiment of the present application, when determining the processing time length of each processing program for each to-be-processed file, the processing speeds of different processing programs are considered to be different, and the processing time length of the same processing program for files with different complexities is also different. Therefore, the file complexity information can be determined according to the file attribute information, and the processing time length of each processing program for the to-be-processed file can be determined according to the file complexity information and the processing speed information of the processing program.

[0081] According to an embodiment of the present application, when determining the file complexity information according to the file attribute information, the complexity of the to-be-processed file can be evaluated according to the file size information, the file type information and other file attribute information, to obtain the file complexity information of the to-be-processed file.

[0082] In an embodiment of the present application, the file complexity information and the processing speed information can be input into a time length prediction model, and the processing time length of the processing program for the to-be-processed file can be determined according to the output of the time length prediction model. Specifically, the time length prediction model can be obtained by training a regression model. The processing speed information of the mth processing program and the file complexity information of the nth to-be-processed file can be input into the time length prediction model, to obtain the processing time length of the mth processing program for the nth to-be-processed file.

[0083] According to an embodiment of the present application, when the regression model is trained to obtain the time length prediction model, the historical complexity information of the historical file and the processing speed information of the processing program can be input into the regression model, to obtain the predicted time length output by the regression model. After obtaining the predicted time length, the historical actual time length of processing the historical file by using the processing program can be obtained, and the parameters of the regression model can be determined based on the residual sum of squares between the historical actual time length and the predicted time length, to obtain the trained time length prediction model.

[0084] According to an embodiment of the present application, by evaluating the complexity of the to-be-processed file according to the file attribute information, and then predicting the processing time length based on the file complexity information and the processing speed information by using the time length prediction model, the accuracy of the predicted processing time length can be improved, the accuracy of the determined processing order can be improved, and the file processing efficiency can be improved.

[0085] According to an embodiment of the present application, the to-be-processed file includes a business data table, file complexity of each to-be-processed file is evaluated based on file attribute information, and file complexity information of each to-be-processed file is obtained, including: data volume complexity of the business data table is evaluated based on the number of rows in the file attribute information, and data volume complexity information of the business data table is determined; structure complexity of the business data table is evaluated based on the number of fields and the field types in the file attribute information, and structure complexity information of the business data table is determined; and file complexity information of the business data table is determined based on the data volume complexity information and the structure complexity information.

[0086] According to an embodiment of the present application, in a file processing system of a bank, the to-be-processed file is mostly a business data table. In the scenario of evaluating complexity of the business data table, file complexity information of the business data table can be determined based on information such as rows and fields of the business data table.

[0087] According to an embodiment of the present application, when complexity of the business data table is evaluated based on row information of the business data table, data volume complexity information of the business data table can be determined based on the number of rows of the business data table in the file attribute information. Specifically, the more the number of rows, the greater the data volume complexity information represents the data volume of the business data table, and the higher the complexity of the business data table.

[0088] According to an embodiment of the present application, when complexity of the business data table is evaluated based on field information of the business data table, structure complexity information of the business data table can be determined based on the number of fields and the field types of the business data table in the file attribute information. Specifically, the more the number of fields and the more the field types, the more complex the structure complexity information represents the structure of the business data table, and the higher the complexity of the business data table.

[0089] According to an embodiment of the present application, the data volume complexity information and the structure complexity information can be in the form of scores. In some embodiments, the data volume complexity information and the structure complexity information can be determined by using a mapping relationship between a preset rule and the scores. For example, when the number of rows is greater than 1000, the data volume complexity score represented by the data volume complexity information is increased by 0.2.

[0090] According to an embodiment of the present application, considering that different types of complexity information have different degrees of influence on file complexity, when the file complexity information is determined based on the data volume complexity information and the structure complexity information, different weights can be set for the data volume complexity information and the structure complexity information, so that the process of evaluating the file complexity is more accurate.

[0091] For example, the data volume complexity information represents the data volume complexity score of the business data table as 0.3, the structure complexity information represents the structure complexity score of the business data table as 0.6, the weight corresponding to the data volume complexity information is 0.4, the weight corresponding to the structure complexity information is 0.6, and finally the file complexity information represents the file complexity score as 0.48.

[0092] According to the embodiments of this application, when the file to be processed is a business data table, the complexity of the business data table can be evaluated by utilizing file attribute information such as the number of rows, the number of fields, and the field types. This can improve the accuracy of the complexity evaluation of the business data table, thereby improving the accuracy of determining the processing time and, consequently, improving the accuracy of determining the processing order of multiple files to be processed.

[0093] Figure 4 The illustration shows a schematic diagram of a complexity assessment of a business data table according to an embodiment of this application.

[0094] like Figure 4 As shown, business data table 400 includes rows 1 to 6 and fields 1 to 3. When evaluating business data table 400, the data volume complexity information 402 of business data table 400 can be determined based on the number of rows 401, and the structural complexity information 405 of business data table 400 can be determined based on the number of fields 403 and the field types 404. Finally, the file complexity information 406 of business data table 400 can be determined based on the data volume complexity information 402 and the structural complexity information 405.

[0095] According to an embodiment of this application, the file processing method further includes: after processing multiple files to be processed, obtaining the actual processing time of each processing program for each file to be processed; and adjusting the parameters of the duration prediction model using the file complexity information of each file to be processed, the processing speed information of each processing program, and the actual processing time of each processing program for each file to be processed.

[0096] According to an embodiment of this application, after processing multiple files to be processed, the actual processing time of each processing program for each file to be processed can be obtained. At this time, the parameters of the time prediction model can be adjusted using the actual processing time to improve the accuracy of the time prediction model.

[0097] According to an embodiment of the present application, when adjusting the parameters of the time prediction model by using the actual processing duration, the processing speed information of the mth processing program and the file complexity information of the nth to-be-processed file can be taken as the input of the time prediction model, the processing duration output by the time prediction model is obtained, the actual processing duration of the mth processing program on the nth to-be-processed file is acquired, and the parameters of the time prediction model are adjusted based on the residual sum of squares between the actual processing duration and the processing duration.

[0098] According to an embodiment of the present application, by adjusting the parameters of the time prediction model by using the actual processing duration of each processing program on each to-be-processed file, the prediction accuracy of the processing duration by the time prediction model can be further improved, so as to improve the accuracy of the processing order and further improve the processing efficiency.

[0099] Based on the above file processing method, the present application further provides a file processing device. The following will be described in detail Figure 5 The device is described in detail.

[0100] Figure 5 The structure block diagram of the file processing device according to an embodiment of the present application is schematically shown.

[0101] As Figure 5 shown, the file processing device 500 of the embodiment includes a determination module 510, an optimization module 520 and a processing module 530.

[0102] The determination module 510 is configured to, in response to receiving a plurality of to-be-processed files, determine the processing duration of each processing program on each to-be-processed file according to the file attribute information of each to-be-processed file and the processing speed information of each processing program in a processing program sequence, the processing program sequence processing each to-be-processed file according to the arrangement order of the plurality of processing programs, and each processing program processing one to-be-processed file at the same time. In an embodiment, the determination module 510 can be configured to perform the operation S210 described above, and details are not described herein again.

[0103] The optimization module 520 is configured to determine the processing order of the plurality of to-be-processed files according to the processing duration of each processing program on each to-be-processed file based on an optimization algorithm, and a target function of the optimization algorithm representing the total processing duration of processing the plurality of to-be-processed files. In an embodiment, the optimization module 520 can be configured to perform the operation S220 described above, and details are not described herein again.

[0104] The processing module 530 is configured to process the plurality of to-be-processed files according to the processing order of the plurality of to-be-processed files by using the processing program sequence. In an embodiment, the processing module 530 can be configured to perform the operation S230 described above, and details are not described herein again.

[0105] According to an embodiment of the present application, the optimization algorithm comprises a constraint condition, the constraint condition comprising a first restriction term and a second restriction term, the first restriction term being used for limiting processing of each to-be-processed file according to the arrangement order of the plurality of processing programs, and the second restriction term being used for limiting that each processing program processes one to-be-processed file at the same time.

[0106] According to an embodiment of the present application, the optimization module 520 comprises an optimization submodule.

[0107] The optimization submodule is configured to determine the processing order of the plurality of to-be-processed files satisfying the constraint condition based on the optimization algorithm, with the objective of minimizing the objective function value of the objective function.

[0108] According to an embodiment of the present application, the optimization submodule comprises a time optimization unit and a sequence determination unit.

[0109] The time optimization unit is configured to optimize the processing start time of each processing program for each to-be-processed file based on the optimization algorithm, with the objective of minimizing the objective function value of the objective function, and determine the processing start time of each processing program for each to-be-processed file satisfying the constraint condition.

[0110] The sequence determination unit is configured to determine the processing order of the plurality of to-be-processed files according to the processing start time of the first processing program in the processing program sequence for each to-be-processed file.

[0111] According to an embodiment of the present application, the determination module 510 comprises a complexity evaluation submodule and a duration determination submodule.

[0112] The complexity evaluation submodule is configured to evaluate the file complexity of each to-be-processed file based on the file attribute information of each to-be-processed file, and obtain file complexity information of each to-be-processed file.

[0113] The duration determination submodule is configured to input the file complexity information of the to-be-processed file and the processing speed information of each processing program into the trained duration prediction model for each to-be-processed file, and obtain the processing duration of each processing program for the to-be-processed file.

[0114] According to an embodiment of the present application, the to-be-processed file comprises a business data table, and the complexity evaluation submodule comprises a data volume evaluation unit, a structure evaluation unit, and a complexity determination unit.

[0115] The data volume evaluation unit is configured to perform data volume complexity evaluation on the business data table according to the number of rows in the file attribute information, and determine data volume complexity information of the business data table.

[0116] The structure evaluation unit is configured to evaluate the structural complexity of the business data table according to the number of fields and the field types in the file attribute information, and determine the structural complexity information of the business data table.

[0117] The complexity determination unit is configured to determine the file complexity information of the business data table according to the data volume complexity information and the structural complexity information.

[0118] According to an embodiment of the present application, the processing program sequence includes M processing programs, and the to-be-processed files are N. The processing module 530 includes a first sending sub-module, a file determination sub-module, and a second sending sub-module.

[0119] The first sending sub-module is configured to send the nth to-be-processed file to the (m+1)th processing program in response to the completion of processing of the nth to-be-processed file by the mth processing program in the processing program sequence, where 1≤m≤M and 1≤n≤N.

[0120] The file determination sub-module is configured to determine the (n+1)th to-be-processed file according to the processing sequence.

[0121] The second sending sub-module is configured to send the (n+1)th to-be-processed file to the mth processing program in the case where the (m-1)th processing program completes processing of the (n+1)th to-be-processed file.

[0122] According to an embodiment of the present application, the determination module 510 further includes an acquisition sub-module and an adjustment sub-module.

[0123] The acquisition sub-module is configured to acquire the actual processing time of each processing program for each to-be-processed file after the processing of the plurality of to-be-processed files is completed.

[0124] The adjustment sub-module is configured to adjust the parameters of the time length prediction model by using the file complexity information of each to-be-processed file, the processing speed information of each processing program, and the actual processing time of each processing program for each to-be-processed file.

[0125] According to an embodiment of the present application, any of the modules 510, 520 and 530 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of the other modules, and implemented in one module. According to an embodiment of the present application, at least one of the modules 510, 520 and 530 can be implemented at least in part as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, etc. in hardware or firmware, or in any one of the three implementation manners of software, hardware and firmware, or in a proper combination of any of the three. Alternatively, at least one of the modules 510, 520 and 530 can be implemented at least in part as a computer program module, which, when executed, can perform the corresponding functions.

[0126] Figure 6 A block diagram of an electronic device suitable for implementing the file processing method according to an embodiment of the present application is schematically shown.

[0127] As shown in Figure 6 The electronic device 600 according to an embodiment of the present application includes a processor 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), etc. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0128] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0129] According to an embodiment of the present application, the electronic device 600 can further include an input / output (I / O) interface 605 that is also connected to the bus 604. The electronic device 600 can further include one or more of the following components connected to the input / output (I / O) interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage part 608 as necessary.

[0130] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present application.

[0131] According to an embodiment of the present application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories such as the ROM 602 and / or the RAM 603 described above and / or one or more memory other than the ROM 602 and the RAM 603.

[0132] The embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the file processing method provided by the embodiments of the present application.

[0133] The above-described functions of the system / device defined in the system / apparatus of the embodiments of the present application are performed when the computer program is executed by the processor 601. According to the embodiments of the present application, the system, apparatus, module, unit, etc. described above can be implemented by the computer program modules.

[0134] In one embodiment, the computer program can be stored in a tangible storage medium, such as an optical, magnetic, or other memory on a hard disk or solid state drive. In another embodiment, the computer program can be transmitted over a network via a modulated signal, and downloaded and installed by the communication portion 609, and / or installed from the removable medium 611. The computer program embodied in the computer program code can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, or any suitable combination of the foregoing.

[0135] In such an embodiment, the computer program can be downloaded and installed from the network via the communication portion 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-described functions of the system defined in the embodiments of the present application are performed. According to the embodiments of the present application, the system, apparatus, device, module, unit, etc. described above can be implemented by the computer program modules.

[0136] According to the embodiments of the present application, the program code for execution of the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, can be implemented using a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. The programming language includes, but is not limited to, a programming language such as Java, C++, Python, "C" language, or a similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A file processing method characterized by, The method includes: In response to receiving multiple files to be processed, the processing time of each processing program for each file to be processed is determined based on the file attribute information of each file to be processed and the processing speed information of each processing program in the processing program sequence. The processing program sequence processes each file to be processed in the order of the multiple processing programs, and each processing program processes one file to be processed at the same time. Based on the optimization algorithm, the processing order of multiple files to be processed is determined according to the processing time of each of the processing programs for each file to be processed. The objective function of the optimization algorithm represents the total processing time for processing multiple files to be processed. The processing sequence is used to process the multiple files to be processed according to their processing order.

2. The method of claim 1, wherein, The optimization algorithm includes constraints, which include a first constraint and a second constraint. The first constraint restricts the processing of each file to be processed according to the order of the plurality of processing programs, and the second constraint restricts each processing program from processing one file to be processed at a time. The step of determining the processing order of multiple files based on an optimization algorithm, according to the processing time of each file in each processing program, includes: Based on the optimization algorithm, with the goal of minimizing the objective function value of the objective function, the processing order of multiple files to be processed that satisfy the constraints is determined.

3. The method of claim 2, wherein, The step of determining the processing order of multiple files to be processed, based on the optimization algorithm and with the objective function minimization as the goal, while satisfying the constraints, includes: Based on the optimization algorithm, with the goal of minimizing the objective function value, the processing start time of each processing program for each file to be processed is optimized to determine the processing start time of each processing program for each file to be processed that satisfies the constraints. The processing order of the multiple files to be processed is determined based on the start time of the first processor in the processing sequence for each file to be processed.

4. The method of claim 1, wherein, The step of determining the processing time of each processing program for each file to be processed based on the file attribute information of each file to be processed and the processing speed information of each processing program in the processing program sequence includes: Based on the file attribute information of each file to be processed, the file complexity of each file to be processed is evaluated to obtain the file complexity information of each file to be processed; For each file to be processed, the file complexity information of the file to be processed and the processing speed information of each processing program are input into the trained duration prediction model to obtain the processing time of each processing program for the file to be processed.

5. The method of claim 4, wherein, The to-be-processed files include business data tables, the file complexity of each to-be-processed file is evaluated based on the file attribute information, and file complexity information of each to-be-processed file is obtained, including: According to the number of rows in the file attribute information, the data volume complexity of the business data table is evaluated, and data volume complexity information of the business data table is determined; According to the number of fields and the field types in the file attribute information, the structural complexity of the business data table is evaluated, and structural complexity information of the business data table is determined; According to the data volume complexity information and the structural complexity information, the file complexity information of the business data table is determined.

6. The method of claim 1, wherein, The processing program sequence includes M processing programs, the to-be-processed files are N, and the to-be-processed files are processed according to the processing order of the to-be-processed files by using the processing program sequence, including repeating the following operations until N to-be-processed files are processed: In response to the mth processing program in the processing program sequence completing processing of the nth to-be-processed file, the nth to-be-processed file is sent to the m+1th processing program, 1≤m≤M, and 1≤n≤N; According to the processing order, the n+1th to-be-processed file is determined; In the case where the m-1th processing program completes processing of the n+1th to-be-processed file, the n+1th to-be-processed file is sent to the mth processing program.

7. The method of claim 4, wherein, The method further includes: After processing of the to-be-processed files is completed, the actual processing time of each processing program for each to-be-processed file is obtained; The parameters of the time length prediction model are adjusted by using the file complexity information of each to-be-processed file, the processing speed information of each processing program, and the actual processing time of each processing program for each to-be-processed file.

8. A file processing apparatus characterized by comprising: The device includes: A determination module is configured to, in response to receiving a plurality of to-be-processed files, determine the processing time of each processing program for each to-be-processed file according to the file attribute information of each to-be-processed file and the processing speed information of each processing program in a processing program sequence, the processing program sequence processes each to-be-processed file according to the arrangement order of a plurality of processing programs, and each processing program processes one to-be-processed file at the same time; An optimization module is configured to determine the processing order of a plurality of to-be-processed files based on an optimization algorithm and the processing time of each processing program for each to-be-processed file, and a target function of the optimization algorithm represents the total processing time of processing a plurality of to-be-processed files; A processing module is configured to process a plurality of to-be-processed files according to the processing order of the to-be-processed files by using the processing program sequence.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images