A data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. A specific implementation of the method includes: establishing a long connection channel with a terminal, loading a data file to be processed; dynamically calculating the maximum number of data items in each group of the data file, and performing business processing on the data file in groups; after processing each group of data, obtaining progress data according to the processing result of the data file, and pushing the progress data to the terminal through the long connection channel. This implementation can solve the technical problem that the progress bar displayed by the terminal cannot accurately reflect the real progress of the data processing on the server side.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] Importing data using data files (such as Excel files) is a common way to enter data into the system in large quantities. The progress of data processing is related to many factors such as business complexity, data volume, and current server busyness. This also leads to uncertainty in the time required to complete the final processing of the data file.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] Currently, data import adopts a synchronous blocking method, that is, after clicking the import button, the terminal blocks and waits until the server completes the processing and returns a response. During the waiting process, the progress bar displayed by the terminal is also a pseudo-progress, which cannot accurately reflect the real progress of the server's data processing. The terminal will not render the progress bar to the completion state until the server responds and processes. Therefore, the user experience is extremely poor. In addition, when the amount of data is large and the business processing is more complex, it is bound to increase the processing time of the server. If the server delays returning a response to the terminal, it will eventually lead to risks such as request timeout and browser crash. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data processing method and device to solve the technical problem that a progress bar displayed by a terminal cannot accurately reflect the actual progress of data processing on a server.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0007] Establish a long connection channel with the terminal and load the data files to be processed;

[0008] Dynamically calculate the maximum number of data items in each group of the data file, and perform business processing on the data file in groups;

[0009] After processing each packet of data, progress data is obtained according to the processing result of the data file, and the progress data is pushed to the terminal through the long connection channel.

[0010] Optionally, dynamically calculating the maximum number of data items in each group of the data file, and performing business processing on the data file in groups, includes:

[0011] Calculate the total number of data items in the data file;

[0012] The maximum number of data items of the next packet data is calculated according to the preset expected response time, the actual number of data items of the previous packet data and the actual processing time, the next packet data is service processed, and the actual processing time of the next packet data is recorded.

[0013] Optionally, calculating the maximum number of data items in the next data group according to a preset expected response time, an actual number of data items in the previous data group, and an actual processing time includes:

[0014] The maximum number of data items of the next packet data is obtained by multiplying the preset expected response time by the actual number of data items of the previous packet data and dividing the result by the actual processing time of the previous packet data.

[0015] Optionally, performing service processing on the next packet data includes:

[0016] Subtract the number of data items of processed data from the total number of data items in the data file to obtain the number of data items of the data to be processed;

[0017] If the maximum number of data items of the next packet data is less than the number of data items of the data to be processed, the maximum number of data items of the next packet data is used as the actual number of data items of the next packet data for service processing;

[0018] If the maximum number of data items of the next packet data is greater than or equal to the number of data items of the data to be processed, the number of data items of the data to be processed is used as the actual number of data items of the next packet data for service processing.

[0019] Optionally, after pushing the progress data to the terminal through the long connection channel, the method further includes:

[0020] If the maximum number of data items of the next data packet is greater than or equal to the number of data items of the data to be processed, the long connection channel is closed after the service processing of the next data packet is completed.

[0021] Optionally, after processing each packet of data, obtaining progress data according to the processing result of the data file, and pushing the progress data to the terminal through the long connection channel includes:

[0022] After each data packet is processed, progress data is calculated according to the total number of data items in the data file and the number of data items of the processed data, and the progress data is pushed to the terminal through the long connection channel.

[0023] Optionally, dynamically calculating the maximum number of data items in each group of the data file, and performing business processing on the data file in groups, includes:

[0024] Calculating the total number of data items in the data file, and pushing the total number of data items in the data file to the terminal;

[0025] The maximum number of data items of the next packet data is calculated according to the preset expected response time, the actual number of data items of the previous packet data and the actual processing time, the next packet data is service processed, and the actual processing time of the next packet data is recorded.

[0026] Optionally, after processing each packet of data, obtaining progress data according to the processing result of the data file, and pushing the progress data to the terminal through the long connection channel includes:

[0027] After each data packet is processed, the number of data items of the processed data is pushed to the terminal through the long connection channel, so that the terminal calculates progress data according to the total number of data items of the data file and the number of data items of the processed data.

[0028] In addition, according to another aspect of an embodiment of the present invention, there is provided a data processing device, including:

[0029] The loading module is used to establish a long connection channel with the terminal and load the data file to be processed;

[0030] A processing module, used for dynamically calculating the maximum number of data items in each group of the data file, and performing business processing on the data file in groups;

[0031] The push module is used to obtain progress data according to the processing result of the data file after processing each group of data, and push the progress data to the terminal through the long connection channel.

[0032] Optionally, the processing module is further used for:

[0033] Calculate the total number of data items in the data file;

[0034] The maximum number of data items of the next packet data is calculated according to the preset expected response time, the actual number of data items of the previous packet data and the actual processing time, the next packet data is service processed, and the actual processing time of the next packet data is recorded.

[0035] Optionally, the processing module is further used for:

[0036] The maximum number of data items of the next packet data is obtained by multiplying the preset expected response time by the actual number of data items of the previous packet data and dividing the result by the actual processing time of the previous packet data.

[0037] Optionally, the processing module is further used for:

[0038] Subtract the number of data items of processed data from the total number of data items in the data file to obtain the number of data items of the data to be processed;

[0039] If the maximum number of data items of the next packet data is less than the number of data items of the data to be processed, the maximum number of data items of the next packet data is used as the actual number of data items of the next packet data for service processing;

[0040] If the maximum number of data items of the next packet data is greater than or equal to the number of data items of the data to be processed, the number of data items of the data to be processed is used as the actual number of data items of the next packet data for service processing.

[0041] Optionally, the push module is further used for:

[0042] After the progress data is pushed to the terminal through the long connection channel, if the maximum number of data items of the next group data is greater than or equal to the number of data items of the data to be processed, the long connection channel is closed after the business processing of the next group data is completed.

[0043] Optionally, the push module is further used for:

[0044] After each data packet is processed, progress data is calculated according to the total number of data items in the data file and the number of data items of the processed data, and the progress data is pushed to the terminal through the long connection channel.

[0045] Optionally, the processing module is further used for:

[0046] Calculating the total number of data items in the data file, and pushing the total number of data items in the data file to the terminal;

[0047] The maximum number of data items of the next packet data is calculated according to the preset expected response time, the actual number of data items of the previous packet data and the actual processing time, the next packet data is service processed, and the actual processing time of the next packet data is recorded.

[0048] Optionally, the push module is further used for:

[0049] After each data packet is processed, the number of data items of the processed data is pushed to the terminal through the long connection channel, so that the terminal calculates progress data according to the total number of data items of the data file and the number of data items of the processed data.

[0050] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including:

[0051] one or more processors;

[0052] a storage device for storing one or more programs,

[0053] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.

[0054] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0055] One embodiment of the above invention has the following advantages or beneficial effects: because it adopts a technical means of dynamically calculating the maximum number of data items in each group of the data file, and performing business processing on the data file in groups, and pushing progress data to the terminal through a long connection channel after processing each group of data, it overcomes the technical problem that the progress bar displayed by the terminal in the prior art cannot accurately reflect the real progress of the data processing on the server. The embodiment of the present invention dynamically calculates the maximum number of data items in each group of the data file, and performs business processing in groups, and pushes progress data to the terminal in real time according to the processing results. The terminal only needs to render the progress bar according to the progress data returned by the server, so as to truly and accurately reflect the progress of the data processing on the server, and improve the user experience; at the same time, when the amount of data is large, there will be no risk of request timeout or browser crash due to increased processing time; the embodiment of the present invention uses a long connection method to convert synchronous waiting into asynchronous waiting, and the server can promptly and accurately inform the terminal each time it processes a part of the data.

[0056] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0058] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0059] Figure 2 is a schematic diagram of the main flow of a data processing method according to a reference embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of the main flow of a data processing method according to another reference embodiment of the present invention;

[0061] Figure 4 is a schematic diagram of the main flow of a data processing method according to yet another reference embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0063] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0064] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0066] Figure 1 Schematic diagram of the main flow of the data processing method according to an embodiment of the present invention. Figure 1 As shown, the data processing method may include:

[0067] Step 101, establish a long connection channel with the terminal and load the data file to be processed.

[0068] After the user clicks the import button on the terminal, the terminal transfers the data file (such as an Excel file) to the server in a streaming manner. At the same time, the terminal and the server establish a websocket long connection channel, which is mainly used to push progress data to the terminal in real time. After the server and the terminal establish a websocket long connection channel, the data file to be processed uploaded by the terminal is loaded.

[0069] Websocket long connection: Websocket is a protocol for full-duplex communication on a single TCP connection. After the connection channel is successfully established, both ends can actively initiate requests to one end.

[0070] Step 102: dynamically calculate the maximum number of data items in each group of the data file, and perform business processing on the data file in groups.

[0071] After the server loads the data file to be processed, it dynamically calculates the maximum number of data items in each group of the data file, and then processes the data file in groups. After processing each group of data, it recalculates the maximum number of data items in the next group of the data file, and processes the data file in groups until all the data in the data file is processed.

[0072] Optionally, step 102 may include: calculating the total number of data items in the data file; calculating the maximum number of data items in the next packet of data according to the preset expected response time, the actual number of data items in the previous packet of data, and the actual processing time, performing business processing on the next packet of data, and recording the actual processing time of the next packet of data. In an embodiment of the present invention, it is first necessary to calculate the total number of data items S in the data file, and then calculate the maximum number of data items in the next packet of data according to the preset expected response time T, the actual number of data items in the previous packet of data, and the actual processing time, then perform business processing on the next packet of data according to the maximum number of data items in the next packet of data, and record the actual processing time of the next packet of data.

[0073] Optionally, the following code can be used to calculate the total number of data items S in the data file:

[0074] / / Get the excel file stream

[0075] InputStream is=file.getInputStream();

[0076] / / Get the worksheet

[0077] Workbook workbook=WorkbookFactory.create(is);

[0078] Sheet sheet=workbook.getSheetAt(0);

[0079] / / Get the row number of the last row of data in the worksheet

[0080] int lastRowNum=sheet.getLastRowNum();

[0081] Optionally, in an embodiment of the present invention, business processing of data may include validating the data, such as non-empty verification of required fields, date format conversion, length verification, etc.

[0082] Optionally, the maximum number of data items in the next group of data is calculated based on a preset expected response time, the actual number of data items in the previous group of data, and the actual processing time, including: multiplying the preset expected response time by the actual number of data items in the previous group of data and dividing the result by the actual processing time of the previous group of data to obtain the maximum number of data items in the next group of data.

[0083] The server calculates the maximum number of data items M for the next packet based on the preset expected response time T. n When grouping data files, the size of each group of data and the processing time of each group of data need to be considered. In an embodiment of the present invention, the maximum number of data items in each group of data is dynamically allocated based on the actual processing time of the previous group of data. Optionally, the following formula can be used to calculate the maximum number of data items M for the next group of data: n :

[0084]

[0085] Where n≥2, M n Round to the nearest integer. If M n <1, then set M n =1.

[0086] M n : The maximum number of data items in the nth group data;

[0087] C n : The actual number of data items in the nth group data. If the data in the data file has not been processed completely, the actual number of data items in the nth group data is C n =M n , otherwise C n =S-∑C n-1 ;

[0088] T: configuration item, the expected response time to the terminal. If you expect to respond to the terminal once a second, then T = 1 second.

[0089] t n : The actual processing time of the nth packet data.

[0090] The purpose of this formula is to estimate the data volume of the next packet of data based on the set response time and the processing speed of the previous packet of data, so as to ensure that the actual processing time of each packet of data is close to the set expected response time T.

[0091] Therefore, the embodiment of the present invention reasonably and dynamically calculates the maximum number of data items in each group according to the configured expected response time, and the group size is determined according to the server's recent processing capacity, thereby avoiding request timeouts due to delayed responses caused by excessive data volume.

[0092] Optionally, performing business processing on the next packet of data includes: subtracting the number of data items of the processed data from the total number of data items of the data file to obtain the number of data items of the data to be processed; if the maximum number of data items of the next packet of data is less than the number of data items of the data to be processed, then using the maximum number of data items of the next packet of data as the actual number of data items of the next packet of data for business processing; if the maximum number of data items of the next packet of data is greater than or equal to the number of data items of the data to be processed, then using the number of data items of the data to be processed as the actual number of data items of the next packet of data for business processing. In an embodiment of the present invention, the number of data items of the data to be processed = S - ∑C n-1 , if M n <S-∑C n-1 , then C n =M n , indicating that the last group data has not been calculated, continue to calculate; if M n ≥S-∑C n-1 , then C n =S-∑C n-1 , indicating that the last group of data has been calculated and the calculation is finished.

[0093] Step 103: After processing each packet of data, progress data is obtained according to the processing result of the data file, and the progress data is pushed to the terminal through the long connection channel.

[0094] After processing each group, the server returns the progress data to the terminal through the persistent connection channel, so that the terminal renders the progress bar according to the progress data sent by the server until the server completes data processing and closes the persistent connection channel.

[0095] Optionally, after the progress data is pushed to the terminal through the persistent connection channel, the method further includes: if the maximum number of data items of the next group of data is greater than or equal to the number of data items of the data to be processed, closing the persistent connection channel after completing the business processing of the next group of data. n ≥S-∑C n-1 , indicating that the last group data has been calculated, the actual number of data items of the last group data C n =S-∑C n-1 After completing the business processing of the last packet data, end the business processing of the data file and close the long connection channel.

[0096] Optionally, step 103 may include: after processing each packet of data, calculating progress data according to the total number of data items in the data file and the number of data items of the processed data, and pushing the progress data to the terminal through the long connection channel. In some embodiments of the present invention, the progress data is calculated by the server, so when the terminal receives the progress data, it can directly render the progress bar according to the progress data transmitted by the server.

[0097] According to the various embodiments described above, it can be seen that the embodiment of the present invention solves the technical problem in the prior art that the progress bar displayed by the terminal cannot accurately reflect the real progress of the data processing on the server side by dynamically calculating the maximum number of data items in each group of the data file, performing business processing on the data file in groups, and pushing progress data to the terminal through a long connection channel after each group of data is processed. The embodiment of the present invention dynamically calculates the maximum number of data items in each group of the data file, performs business processing in groups, and pushes progress data to the terminal in real time according to the processing results. The terminal only needs to render the progress bar according to the progress data returned by the server, thereby truly and accurately reflecting the progress of the data processing on the server side and improving the user experience; at the same time, when the amount of data is large, there will be no risk of request timeout or browser crash due to increased processing time; the embodiment of the present invention uses a long connection method to convert synchronous waiting into asynchronous waiting, and the server can promptly and accurately inform the terminal each time it processes a portion of data.

[0098] Figure 2 Schematic diagram of the main flow of a data processing method according to a reference embodiment of the present invention. As another embodiment of the present invention, Figure 2 As shown, step 102 may specifically include the following steps:

[0099] Step 201: Obtain the total number of data items S and the preset expected response time T of the data file to be processed.

[0100] Step 202: Assuming that the first packet data is a fixed value, that is, setting C1=1, the actual processing time length of the first packet data is t1.

[0101] Step 203, the maximum number of data items M2 of the second group data can be obtained according to the following formula and the known data T, C1, and t1:

[0102]

[0103] Where n≥2, M n Round to the nearest integer. If M n <1, then set M n =1.

[0104] M n: The maximum number of data items in the nth group data;

[0105] C n : The actual number of data items in the nth group data. If the data in the data file has not been processed completely, the actual number of data items in the nth group data is C n =M n , otherwise C n =S-∑C n-1 ;

[0106] T: configuration item, the expected response time to the terminal. If you expect to respond to the terminal once a second, then T = 1 second.

[0107] t n : The actual processing time of the nth packet data.

[0108] Step 204: perform service processing on the packet data.

[0109] Step 205, determine whether all the data in the data file has been processed; if so, end; if not, execute step 203.

[0110] If M n <S-∑C n-1 , then C n =M n , indicating that the last group data has not been calculated, and the calculation continues.

[0111] If M n S-∑C n-1 , then C n =M n , indicating that the last group of data has been calculated and the calculation is finished.

[0112] If M n >S-∑C n-1 >0, then C n =S-∑C n-1 , indicating that the last group of data has been calculated and the calculation is finished.

[0113] In addition, the specific implementation content of the data processing method in a reference embodiment of the present invention has been described in detail in the above-mentioned data processing method, so the repeated content will not be described again here.

[0114] Figure 3 Schematic diagram of the main flow of a data processing method according to another reference embodiment of the present invention. As another embodiment of the present invention, Figure 3 As shown, the data processing method may include:

[0115] Step 301: Establish a long connection channel with the terminal and load the data file to be processed.

[0116] Step 302: Calculate the total number of data items in the data file.

[0117] Step 303: multiply the preset expected response time by the actual number of data items in the previous packet data and divide the result by the actual processing time of the previous packet data to obtain the maximum number of data items in the next packet data.

[0118] Step 304: perform service processing on the next packet data and record the actual processing time of the next packet data.

[0119] Step 305: After processing each packet of data, the progress data is calculated according to the total number of data items in the data file and the number of data items of the processed data, and the progress data is pushed to the terminal through the long connection channel.

[0120] In this embodiment, the server calculates the progress data and then pushes the progress data to the terminal through a long connection channel. When the terminal receives the progress data, it can directly render the progress bar according to the progress data transmitted from the server.

[0121] Step 306, subtract the number of data items of the processed data from the total number of data items of the data file to obtain the number of data items of the data to be processed; if the maximum number of data items of the next group of data is greater than or equal to the number of data items of the data to be processed, then after completing the business processing of the next group of data, close the long connection channel.

[0122] In addition, the specific implementation content of the data processing method in another reference embodiment of the present invention has been described in detail in the above-mentioned data processing method, so the repeated content will not be described again here.

[0123] Figure 4 Schematic diagram of the main flow of a data processing method according to another reference embodiment of the present invention. As another embodiment of the present invention, Figure 4 As shown, the data processing method may include:

[0124] Step 401: Establish a long connection channel with the terminal and load the data file to be processed.

[0125] Step 402: Calculate the total number of data items in the data file, and push the total number of data items in the data file to the terminal.

[0126] Step 403: multiply the preset expected response time by the actual number of data items in the previous packet data and divide the result by the actual processing time of the previous packet data to obtain the maximum number of data items in the next packet data.

[0127] Step 404: perform service processing on the next packet data and record the actual processing time of the next packet data.

[0128] Step 405: After processing each packet of data, the number of processed data items is pushed to the terminal through the long connection channel, so that the terminal calculates progress data according to the total number of data items in the data file and the number of processed data items.

[0129] In this embodiment, after the server completes the business processing, it directly pushes the number of processed data to the terminal through the long connection channel, and the terminal calculates the progress data and then renders the progress bar according to the calculated progress data.

[0130] Step 406, subtract the number of data items of the processed data from the total number of data items of the data file to obtain the number of data items of the data to be processed; if the maximum number of data items of the next group of data is greater than or equal to the number of data items of the data to be processed, then after completing the business processing of the next group of data, close the long connection channel.

[0131] In addition, the specific implementation content of the data processing method in another reference embodiment of the present invention has been described in detail in the above-mentioned data processing method, so the repeated content will not be described here.

[0132] Figure 5 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention, such as Figure 5 As shown, the data processing device 500 includes a loading module 501, a processing module 502 and a pushing module 503; wherein the loading module 501 is used to establish a long connection channel with the terminal and load the data file to be processed; the processing module 502 is used to dynamically calculate the maximum number of data items in each group of the data file, and perform business processing on the data file in groups; the pushing module 503 is used to obtain progress data according to the processing result of the data file after processing a group of data, and push the progress data to the terminal through the long connection channel.

[0133] Optionally, the processing module 502 is further configured to:

[0134] Calculate the total number of data items in the data file;

[0135] The maximum number of data items of the next packet data is calculated according to the preset expected response time, the actual number of data items of the previous packet data and the actual processing time, the next packet data is service processed, and the actual processing time of the next packet data is recorded.

[0136] Optionally, the processing module 502 is further configured to:

[0137] The maximum number of data items of the next packet data is obtained by multiplying the preset expected response time by the actual number of data items of the previous packet data and dividing the result by the actual processing time of the previous packet data.

[0138] Optionally, the processing module 502 is further configured to:

[0139] Subtract the number of data items of processed data from the total number of data items in the data file to obtain the number of data items of the data to be processed;

[0140] If the maximum number of data items of the next packet data is less than the number of data items of the data to be processed, the maximum number of data items of the next packet data is used as the actual number of data items of the next packet data for service processing;

[0141] If the maximum number of data items of the next packet data is greater than or equal to the number of data items of the data to be processed, the number of data items of the data to be processed is used as the actual number of data items of the next packet data for service processing.

[0142] Optionally, the push module 503 is further used for:

[0143] After the progress data is pushed to the terminal through the long connection channel, if the maximum number of data items of the next group data is greater than or equal to the number of data items of the data to be processed, the long connection channel is closed after the business processing of the next group data is completed.

[0144] Optionally, the push module 503 is further used for:

[0145] After each data packet is processed, progress data is calculated according to the total number of data items in the data file and the number of data items of the processed data, and the progress data is pushed to the terminal through the long connection channel.

[0146] Optionally, the processing module 502 is further configured to:

[0147] Calculating the total number of data items in the data file, and pushing the total number of data items in the data file to the terminal;

[0148] The maximum number of data items of the next packet data is calculated according to the preset expected response time, the actual number of data items of the previous packet data and the actual processing time, the next packet data is service processed, and the actual processing time of the next packet data is recorded.

[0149] Optionally, the push module 503 is further used for:

[0150] After each data packet is processed, the number of data items of the processed data is pushed to the terminal through the long connection channel, so that the terminal calculates progress data according to the total number of data items of the data file and the number of data items of the processed data.

[0151] It should be noted that the specific implementation content of the data processing device of the present invention has been described in detail in the above-mentioned data processing method, so the repeated content will not be described again here.

[0152] Figure 6 An exemplary system architecture 600 is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0153] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0154] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0155] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0156] The server 605 may be a server that provides various services, such as a backend management server (for example only) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as the item information query request, and feed back the processing results to the terminal device.

[0157] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data processing device is generally arranged in the server 605 .

[0158] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0159] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0160] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0161] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0162] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0163] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0164] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as follows: a processor includes a loading module, a processing module, and a push module, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases.

[0166] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device implements the following method: establish a long connection channel with the terminal and load the data file to be processed; dynamically calculate the maximum number of data items in each group of the data file, and perform business processing on the data file in groups; after processing each group of data, obtain progress data according to the processing result of the data file, and push the progress data to the terminal through the long connection channel.

[0167] According to the technical solution of the embodiment of the present invention, because it adopts a technical means of dynamically calculating the maximum number of data items in each group of the data file, performing business processing on the data file in units of groups, and pushing progress data to the terminal through a long connection channel after each group of data is processed, it overcomes the technical problem that the progress bar displayed by the terminal in the prior art cannot accurately reflect the real progress of the data processing on the server. The embodiment of the present invention dynamically calculates the maximum number of data items in each group of the data file, performs business processing in units of groups, and pushes progress data to the terminal in real time according to the processing results. The terminal only needs to render the progress bar according to the progress data returned by the server, so as to truly and accurately reflect the progress of the data processing on the server and improve the user experience; at the same time, when the amount of data is large, there will be no risk of request timeout or browser crash due to increased processing time; the embodiment of the present invention uses a long connection method to convert synchronous waiting into asynchronous waiting, and the server can promptly and accurately inform the terminal each time it processes a part of the data.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Establish a long connection channel with the terminal and load the data files to be processed; Dynamically calculate the maximum number of data items in each group of the data file, and perform business processing on the data file in groups; After processing each packet of data, progress data is obtained according to the processing result of the data file, and the progress data is pushed to the terminal through the long connection channel; Dynamically calculating the maximum number of data items in each group of the data file, and performing business processing on the data file in groups, including: Calculate the total number of data items in the data file; The preset expected response time is multiplied by the actual number of data items in the previous packet data and divided by the actual processing time of the previous packet data to obtain the maximum number of data items in the next packet data, perform service processing on the next packet data, and record the actual processing time of the next packet data.

2. The method according to claim 1, characterized in that Performing service processing on the next packet data includes: Subtract the number of data items of processed data from the total number of data items in the data file to obtain the number of data items of the data to be processed; If the maximum number of data items of the next packet data is less than the number of data items of the data to be processed, the maximum number of data items of the next packet data is used as the actual number of data items of the next packet data for service processing; If the maximum number of data items of the next packet data is greater than or equal to the number of data items of the data to be processed, the number of data items of the data to be processed is used as the actual number of data items of the next packet data for service processing.

3. The method according to claim 2, characterized in that After pushing the progress data to the terminal through the long connection channel, the method further includes: If the maximum number of data items of the next data packet is greater than or equal to the number of data items of the data to be processed, the long connection channel is closed after the service processing of the next data packet is completed.

4. The method according to claim 1, characterized in that: After processing each packet of data, progress data is obtained according to the processing result of the data file, and the progress data is pushed to the terminal through the long connection channel, including: After each data packet is processed, progress data is calculated according to the total number of data items in the data file and the number of data items of the processed data, and the progress data is pushed to the terminal through the long connection channel.

5. The method according to claim 1, characterized in that Dynamically calculating the maximum number of data items in each group of the data file, and performing business processing on the data file in groups, including: Calculating the total number of data items in the data file, and pushing the total number of data items in the data file to the terminal; The maximum number of data items of the next packet data is calculated according to the preset expected response time, the actual number of data items of the previous packet data and the actual processing time, the next packet data is service processed, and the actual processing time of the next packet data is recorded.

6. The method according to claim 5, characterized in that After processing each packet of data, progress data is obtained according to the processing result of the data file, and the progress data is pushed to the terminal through the long connection channel, including: After each data packet is processed, the number of data items of the processed data is pushed to the terminal through the long connection channel, so that the terminal calculates progress data according to the total number of data items of the data file and the number of data items of the processed data.

7. A data processing device, characterized in that: include: The loading module is used to establish a long connection channel with the terminal and load the data file to be processed; A processing module, used to calculate the total number of data items in the data file; The maximum number of data items of the next packet data is obtained by multiplying the preset expected response time by the actual number of data items of the previous packet data and dividing the result by the actual processing time of the previous packet data, performing service processing on the next packet data, and recording the actual processing time of the next packet data; The push module is used to obtain progress data according to the processing result of the data file after processing each group of data, and push the progress data to the terminal through the long connection channel.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images