Data processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: receiving a data processing request corresponding to a to-be-processed table file, and then determining the data size of a worksheet in the table file according to the data processing request; and finally, processing the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and a preset mapping relation to obtain a processing result. Wherein the preset mapping relationship comprises a corresponding relationship among the data category, the target category standard and the business data acquisition entry. According to the method, the multiple worktables in the table file are processed in batches by combining the capacity of the local memory, the problem that the local memory is insufficient due to the fact that a single table file is too large or a large amount of data is loaded at a time can be solved, and therefore the local memory is more fully utilized, and then the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of information technology, Excel files are applied to various scenarios. Due to the diversity of Excel file versions and data complexity, the file size and data volume of Excel in many scenarios are large. Therefore, how to efficiently read and parse Excel files becomes crucial.

[0003] Currently, Excel files are usually exported using methods such as EasyExcel or JExcel API library, etc. However, the above Excel file export methods have the problem of low data processing efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, device, equipment, storage medium and program product that can improve data processing efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, the method comprising:

[0006] Receiving a data processing request corresponding to a table file to be processed;

[0007] Determine the data volume of a worksheet in a spreadsheet file according to a data processing request;

[0008] The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0009] In one embodiment, the table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result, including:

[0010] According to the data volume of the worksheet in the spreadsheet file and the capacity of the local memory, a target worksheet is determined from all the worksheets in the spreadsheet file;

[0011] The target worksheet is stored in the local memory, and the target worksheet is processed in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result;

[0012] The other worksheets except the target worksheet are stored in the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, a new target worksheet is determined from the other worksheets, and the step of storing the target worksheet in the local memory is returned according to the new target worksheet.

[0013] In one embodiment, the target worksheet is processed according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain the processing result, including:

[0014] Determine the target processing type of the target worksheet and the data category of the target worksheet;

[0015] If the target processing type is a read processing type, the target worksheet is verified according to the data category and the preset mapping relationship, and when the verification result indicates that the verification is passed, the data in the target worksheet is read;

[0016] If the target processing type is a write processing type, the target business data is obtained according to the data category and the preset mapping relationship, and the target business data is written to the target worksheet.

[0017] In one embodiment, verifying the target worksheet according to the data category and the preset mapping relationship includes:

[0018] Searching for a target category standard corresponding to the data category in a preset mapping relationship; the target category standard includes at least one of a worksheet header type, a number of cells corresponding to each header, and a data format of each cell;

[0019] Determine whether the data category is consistent with the target category standard. If consistent, determine that the verification result indicates that the verification is passed. If inconsistent, determine that the verification result indicates that the verification is failed.

[0020] In one embodiment, obtaining target business data according to the data category and the preset mapping relationship includes:

[0021] Find the business data acquisition path corresponding to the data category in the preset mapping relationship;

[0022] Get the target business data from the business data acquisition path.

[0023] In one embodiment, according to the data amount of the worksheet in the spreadsheet file and the capacity of the local memory, determining the target worksheet from all the worksheets in the spreadsheet file includes:

[0024] The worksheet whose total amount of data in the spreadsheet files does not exceed the capacity of the local memory is determined as the target worksheet.

[0025] In a second aspect, the present application further provides a data processing device, the device comprising:

[0026] A receiving module, used for receiving a data processing request corresponding to a table file to be processed;

[0027] A determination module, used for determining the data amount of a worksheet in a table file according to a data processing request;

[0028] The processing module is used to process the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0029] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0030] Receiving a data processing request corresponding to a table file to be processed;

[0031] Determine the data volume of the worksheet in the table file according to the data processing request;

[0032] The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0034] Receiving a data processing request corresponding to a table file to be processed;

[0035] Determine the data volume of the worksheet in the table file according to the data processing request;

[0036] The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0037] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0038] Receiving a data processing request corresponding to a table file to be processed;

[0039] Determine the data volume of the worksheet in the table file according to the data processing request;

[0040] The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0041] The above-mentioned data processing method, device, equipment, storage medium and program product, the method receives a data processing request corresponding to the table file to be processed, then determines the data volume of the worksheet in the table file according to the data processing request, and finally processes the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result. Among them, the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry. In the above method, by processing multiple worksheets in the table file in batches in combination with the capacity of the local memory, the problem of insufficient local memory caused by a single table file being too large or a large amount of data being loaded at one time can be solved, so that the local memory can be more fully utilized, thereby improving the data processing efficiency. Moreover, the business data can be quickly acquired through the preset mapping relationship, so when the table file is processed based on the business data in the future, the processing efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is an internal structure diagram of a computer device in one embodiment;

[0043] Figure 2 is a flow chart of a data processing method in one embodiment;

[0044] Figure 3 is a flow chart of a data processing method in another embodiment;

[0045] Figure 4 is a flow chart of a data processing method in another embodiment;

[0046] Figure 5 is a flow chart of a data processing method in another embodiment;

[0047] Figure 6 is a flow chart of a data processing method in another embodiment;

[0048] Figure 7 FIG. 4 is a structural block diagram of a data processing device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] With the rapid development of information technology, Excel files are applied to a variety of scenarios. Due to the diversity of Excel file versions and data complexity, the file size and data volume of Excel in many scenarios are large. Therefore, how to efficiently read and parse Excel files becomes crucial. At present, Excel files are usually exported using methods such as EasyExcel or JExcelAPI library. However, the above Excel file export method has the problem of low data processing efficiency.

[0051] The present application provides a data processing method, aiming to solve the above-mentioned technical problems. The following embodiments will specifically illustrate the data processing method described in the present application.

[0052] The data processing method provided in the embodiment of the present application can be applied to Figure 1 In the computer device shown in the figure, the computer device can be a terminal or a server, and its internal structure diagram can be as shown in FIG. Figure 1 As shown, the computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and a local internal memory. The non-volatile storage medium stores an operating system and a computer program. The local internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data processing method is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0053] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0054] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 The computer device in the example is used to illustrate, including the following steps:

[0055] S201, receiving a data processing request corresponding to a table file to be processed.

[0056] Among them, the table file can be an Excel file or other types of table files. The format of the table file can be .xls format or .xlsx format. The table file to be processed can be one table file or multiple table files. Each table file can include one or more worksheets, each worksheet can include one or more cells, and each worksheet can include multiple column headers and / or row headers. The data processing request is used to indicate that the table file to be processed is to be read or written. The data processing request includes one or more of the identifier of the table file, the identifier of each worksheet in the table file, the data amount of the table file, the data amount of each worksheet, and the data of each worksheet.

[0057] In an embodiment of the present application, a computer device may receive a data processing request sent by another device. Optionally, a user may trigger a processing instruction on a user interface, upload a table file to be processed through a preset interface, and generate a data processing request according to the processing instruction and the table file to be processed, and the computer device receives the data processing request.

[0058] S202, determining the data volume of the worksheet in the spreadsheet file according to the data processing request.

[0059] The data volume of a worksheet indicates the size of the memory space occupied by the worksheet.

[0060] In an embodiment of the present application, after the computer device obtains the data processing request based on the above steps, it can parse the table file to determine the amount of data in each worksheet in the table file. Specifically, the format or structure of the table file can be converted to a preset format to obtain a converted table file, and then the amount of data in each worksheet can be determined from the converted table file. For example, a table file in .xls format and / or a table file in .xlsx format can be converted to a table file in .json format. For another example, a binary structure (a structure corresponding to the .xls format) and / or an XML structure (a structure corresponding to the .xlsx format) can be converted to a lightweight text structure corresponding to the .json format.

[0061] S203, processing the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result.

[0062] The capacity of the local memory is the memory capacity of the computer device. The preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entrance. The data category may include one or more of hardware data, software data, business data, amount data and device data. The target category standard includes one or more of the header type of the worksheet, the number of cells corresponding to each header and the data format of each cell. The computer device can obtain business data through the business data acquisition entrance.

[0063] In an embodiment of the present application, the computer device can pre-establish the correspondence between the data category of the worksheet, the target category standard, and the business data acquisition entry as a preset mapping relationship, and store the preset mapping relationship under a preset path. After the computer device obtains the data volume of the worksheet in the table file based on the above steps, it can obtain the capacity of the local memory through the command line tool, and obtain the preset mapping relationship from the preset path, and then process the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result. Specifically, it can first perform preliminary processing according to the data volume of the worksheet and the capacity of the local memory to obtain a table file after preliminary processing, and then use the preset mapping relationship to perform final processing on the table file after preliminary processing to obtain the processing result.

[0064] The data processing method provided in the embodiment of the present application receives a data processing request corresponding to a table file to be processed, then determines the data volume of a worksheet in the table file according to the data processing request, and finally processes the table file according to the data volume of the worksheet in the table file, the capacity of the local memory, and a preset mapping relationship to obtain a processing result. Among them, the preset mapping relationship includes the correspondence between data categories, target category standards, and business data acquisition entrances. In the above method, by processing multiple worksheets in the table file in batches in combination with the capacity of the local memory, the problem of insufficient local memory caused by a single table file being too large or a large amount of data being loaded at one time can be solved, thereby making more full use of the local memory, thereby improving data processing efficiency. Moreover, business data can be quickly acquired through the preset mapping relationship, so when the table file is subsequently processed based on the business data, the processing efficiency can be improved.

[0065] In one embodiment, a specific implementation method for processing a table file is also provided, such as Figure 3 As shown, the above step S203 of "processing the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result" includes:

[0066] S301, determining a target worksheet from all worksheets in the spreadsheet file according to the data volume of the worksheet in the spreadsheet file and the capacity of the local memory.

[0067] The total amount of data in the target worksheets must not exceed the capacity of the local memory.

[0068] In an embodiment of the present application, the computer device can sum the data amount of the worksheets in the table file in chronological order, and compare the sum of the data amount with the capacity of the local memory in real time, and determine the worksheet whose sum of the data amount of the worksheets in the table file does not exceed the capacity of the local memory as the target worksheet. Optionally, the worksheet whose maximum sum of the data amount of the worksheets does not exceed the capacity of the local memory can be determined as the target worksheet. Optionally, if the data amount of one of the worksheets exceeds the capacity of the local memory, the worksheet is split to obtain multiple worksheets whose data amount is less than or equal to the capacity of the local memory. For example, there are 10 worksheets in the table file, and the data amount of each worksheet does not exceed the capacity of the local memory. The data amount of worksheet 1 and worksheet 2 can be added to obtain the sum of the first data amount, and then it is determined whether the sum of the first data amount exceeds the capacity of the local memory. If it exceeds, worksheet 1 is determined to be the target worksheet, and if it does not exceed, worksheet 3 and the sum of the first data amount are added to obtain the sum of the second data amount. Then determine whether the sum of the second data volume exceeds the capacity of the local memory. If so, determine worksheet 1 and worksheet 2 as the target worksheets. If not, add worksheet 4 and the sum of the second data volume, and so on. Finally, determine the worksheet whose sum of data volume does not exceed the capacity of the local memory, that is, the target worksheet.

[0069] S302, storing the target worksheet into the local memory, and processing the target worksheet in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result.

[0070] The processing type includes a read processing type and a write processing type.

[0071] In the embodiment of the present application, the computer device obtains the target worksheet based on the above steps, and can store the target worksheet into the local memory, and then process the target worksheet in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain the processing result. Specifically, it can be determined first whether the processing type of the target worksheet is a read processing type or a write processing type, and then the data category of the target worksheet is determined for different processing types, and the target worksheet is processed in combination with the preset mapping relationship to obtain the processing result.

[0072] S303, storing other worksheets except the target worksheet into the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, determining a new target worksheet from other worksheets, and returning to execute the step of storing the target worksheet into the local memory according to the new target worksheet.

[0073] In an embodiment of the present application, the computer device can determine other worksheets other than the target worksheet based on the above steps, and then store the other worksheets in the local disk, and then monitor the operation of the local memory in real time to determine whether the target worksheet in the local memory has been processed, and after detecting that the target worksheet in the local memory has been processed, determine a new target worksheet from other worksheets, and return to execute the step of storing the target worksheet in the local memory according to the new target worksheet, that is, return to execute step S302.

[0074] The method described in the embodiment of the present application avoids the problem of memory overflow caused by loading too much data at one time by loading the data into the memory in batches, reduces the performance loss caused by frequent disk reading and writing, and improves the overall data processing speed and efficiency. Moreover, the number and size of data processing batches are dynamically adjusted according to the memory size of the current system, so that system resources are used more reasonably.

[0075] In one embodiment, a specific implementation method for processing the target worksheet is also provided, such as Figure 4 As shown, the above step S302 of "processing the target worksheet according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result" includes:

[0076] S401, determining a target processing type of a target worksheet and a data category of the target worksheet.

[0077] The data categories may include one or more of hardware data, software data, business data, amount data and equipment data.

[0078] In the embodiment of the present application, after obtaining the target worksheet, the computer device may determine the target processing type according to the data processing request, and then determine the data category of the target worksheet according to the identifier of the target worksheet (eg, the name of the target worksheet).

[0079] S402: If the target processing type is a read processing type, the target worksheet is verified according to the data category and the preset mapping relationship, and when the verification result indicates that the verification is passed, the data in the target worksheet is read.

[0080] The preset mapping relationship also includes a corresponding relationship between data categories and data storage locations.

[0081] In the embodiment of the present application, if the computer device determines that the target processing type is a read processing type based on the above steps, the target worksheet can be verified according to the data category and the preset mapping relationship to obtain a verification result, and when the verification result indicates that the verification is passed, the data in the target worksheet is read. Specifically, after determining the data category of the target worksheet, the computer device can first determine the target data storage location corresponding to the target worksheet according to the preset mapping relationship, and then read the columns or rows in the target worksheet in turn, and in each column / row, read the data in each cell in turn, and then store the read data in the target data storage location.

[0082] Specifically, Figure 5 As shown, the "verifying the target worksheet according to the data category and the preset mapping relationship" in the above step S402 includes:

[0083] S4021, searching for a target category standard corresponding to the data category in a preset mapping relationship.

[0084] The target category standard includes one or more of the worksheet header type, the number of cells corresponding to each header, and the data format of each cell.

[0085] In the embodiment of the present application, the computer device can search for a target category standard corresponding to the data category in a preset mapping relationship according to the data category of the target worksheet.

[0086] S4022, determining whether the data category is consistent with the target category standard, if consistent, determining the verification result indicates that the verification is passed, if inconsistent, determining the verification result indicates that the verification is failed.

[0087] In an embodiment of the present application, after the computer device obtains the target category standard based on the above steps, it can compare each item in the data category of the target worksheet with each corresponding item in the target category standard to determine whether the data category is consistent with the target category standard. If each item in the data category of the target worksheet is consistent with each corresponding item in the target category standard, it means that the data category of the target worksheet is consistent with the target category standard, and it can be determined that the verification result indicates that the verification is passed. If there is an item in the data category of the target worksheet that is inconsistent with the corresponding item in the target category standard, it means that the data category of the target worksheet is inconsistent with the target category standard, and it can be determined that the verification result indicates that the verification is failed.

[0088] S403: If the target processing type is a write processing type, the target business data is acquired according to the data category and the preset mapping relationship, and the target business data is written into the target work table.

[0089] In the embodiment of the present application, if the computer device determines that the target processing type is a write processing type based on the above steps, the target business data can be obtained according to the data category and the preset mapping relationship, and the target business data can be written to the target worksheet. Specifically, the columns or rows in the target worksheet can be written in sequence, and in each column / row, the data in each cell can be written in sequence.

[0090] Specifically, Figure 6 As shown, the "obtaining target business data according to the data category and the preset mapping relationship" in the above step S403 includes:

[0091] S4031, searching for a business data acquisition path corresponding to the data category in a preset mapping relationship.

[0092] The business data acquisition path may be a data storage address or a data acquisition interface.

[0093] In the embodiment of the present application, the computer device can search for a business data acquisition path corresponding to the data category of the target worksheet in a preset mapping relationship according to the data category of the target worksheet.

[0094] S4032, acquiring target business data from the business data acquisition path.

[0095] In the embodiment of the present application, after the computer device obtains the business data acquisition path based on the above steps, it can obtain the target business data from the business data acquisition path.

[0096] The method described in the embodiment of the present application verifies the data in the worksheet before reading the data, and can check whether the data meets the standard, so as to find and handle abnormal data early in the data processing process. At the same time, by mapping the path for writing data, the process of data writing can be simplified and the efficiency of data writing can be improved.

[0097] Based on all the above embodiments, a data processing method is also provided, which includes:

[0098] S501, receiving a data processing request corresponding to a table file to be processed.

[0099] S502, determining the data volume of the worksheet in the spreadsheet file according to the data processing request.

[0100] S503, determining a worksheet whose total data volume in the table file does not exceed the capacity of the local memory as the target worksheet.

[0101] S504, storing the target worksheet into the local memory, and determining the target processing type of the target worksheet and the data category of the target worksheet in the local memory.

[0102] S505, if the target processing type is a read processing type, execute S506-S508, if the target processing type is a write processing type, execute S509-S510.

[0103] S506, searching for a target category standard corresponding to the data category in the preset mapping relationship, wherein the target category standard includes at least one of the worksheet header type, the number of cells corresponding to each header, and the data format of each cell, and the preset mapping relationship includes the corresponding relationship between the data category, the target category standard, and the business data acquisition entry.

[0104] S507, determining whether the data category is consistent with the target category standard, if consistent, determining the verification result indicates that the verification is passed, if inconsistent, determining the verification result indicates that the verification is failed.

[0105] S508: When the verification result indicates that the verification is passed, read the data in the target worksheet.

[0106] S509: Search the preset mapping relationship for a business data acquisition path corresponding to the data category.

[0107] S510, acquiring target business data from the business data acquisition path, and writing the target business data into a target worksheet.

[0108] S511, save other worksheets except the target worksheet to the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, determine a new target worksheet from other worksheets, and return to execute steps S504-S510 according to the new target worksheet.

[0109] The methods described in the above steps are all described in the above embodiments. Please refer to the above description for details and will not be repeated here.

[0110] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0112] In one embodiment, Figure 7 As shown, a data processing device is provided, comprising:

[0113] A receiving module 11, used for receiving a data processing request corresponding to a table file to be processed;

[0114] A determination module 12, configured to determine the data volume of the worksheet in the table file according to the data processing request;

[0115] The processing module 13 is used to process the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0116] In one embodiment, the processing module includes:

[0117] A determination unit is used to determine a target worksheet from all the worksheets in the table file according to the data volume of the worksheets in the table file and the capacity of the local memory.

[0118] The processing unit is used to store the target worksheet into the local memory, and process the target worksheet in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result.

[0119] A loop unit is used to store other worksheets except the target worksheet into the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, determine a new target worksheet from the other worksheets, and return to execute the step of storing the target worksheet into the local memory according to the new target worksheet.

[0120] In one embodiment, the processing unit includes:

[0121] The determination subunit is used to determine the target processing type of the target worksheet and the data category of the target worksheet.

[0122] The verification subunit is used to verify the target worksheet according to the data category and the preset mapping relationship if the target processing type is the read processing type, and read the data in the target worksheet when the verification result indicates that the verification is passed.

[0123] An acquisition subunit is used to acquire target business data according to the data category and the preset mapping relationship if the target processing type is the write processing type, and write the target business data into the target work table.

[0124] In one embodiment, the above-mentioned verification sub-unit is specifically used to search for a target category standard corresponding to the data category in the preset mapping relationship; the target category standard includes at least one of the header type of the worksheet, the number of cells corresponding to each header, and the data format of each cell; determine whether the data category is consistent with the target category standard, if consistent, determine that the verification result indicates that the verification is passed, if inconsistent, determine that the verification result indicates that the verification is failed.

[0125] In one embodiment, the acquisition subunit is specifically used to search for a business data acquisition path corresponding to the data category in the preset mapping relationship; and acquire the target business data from the business data acquisition path.

[0126] In one embodiment, the determination unit is specifically used to determine a worksheet whose total amount of data in the table file does not exceed the capacity of the local memory as the target worksheet.

[0127] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0129] Receiving a data processing request corresponding to a table file to be processed;

[0130] Determine the data volume of the worksheet in the table file according to the data processing request;

[0131] The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0132] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0133] Determine a target worksheet from all the worksheets in the table file according to the data volume of the worksheets in the table file and the capacity of the local memory;

[0134] The target worksheet is stored in the local memory, and the target worksheet is processed in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result;

[0135] The other worksheets except the target worksheet are stored in the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, a new target worksheet is determined from the other worksheets, and the step of storing the target worksheet in the local memory is returned according to the new target worksheet.

[0136] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0137] Determine the target processing type of the target worksheet and the data category of the target worksheet;

[0138] If the target processing type is the read processing type, verifying the target worksheet according to the data category and the preset mapping relationship, and reading the data in the target worksheet when the verification result indicates that the verification is passed;

[0139] If the target processing type is the write processing type, target business data is acquired according to the data category and the preset mapping relationship, and the target business data is written into the target work table.

[0140] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0141] Searching for a target category standard corresponding to the data category in the preset mapping relationship; the target category standard includes at least one of a header type of the worksheet, a number of cells corresponding to each header, and a data format of each cell;

[0142] Determine whether the data category is consistent with the target category standard. If so, determine that the verification result indicates that the verification is passed. If not, determine that the verification result indicates that the verification is failed.

[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0144] Searching for a business data acquisition path corresponding to the data category in the preset mapping relationship;

[0145] The target business data is acquired from the business data acquisition path.

[0146] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0147] A worksheet whose total amount of data in the table file does not exceed the capacity of the local memory is determined as the target worksheet.

[0148] The above embodiment provides a computer device, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0149] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0150] Receiving a data processing request corresponding to a table file to be processed;

[0151] Determine the data volume of the worksheet in the table file according to the data processing request;

[0152] The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0153] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0154] Determine a target worksheet from all the worksheets in the table file according to the data volume of the worksheets in the table file and the capacity of the local memory;

[0155] The target worksheet is stored in the local memory, and the target worksheet is processed in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result;

[0156] The other worksheets except the target worksheet are stored in the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, a new target worksheet is determined from the other worksheets, and the step of storing the target worksheet in the local memory is returned according to the new target worksheet.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0158] Determine the target processing type of the target worksheet and the data category of the target worksheet;

[0159] If the target processing type is the read processing type, verifying the target worksheet according to the data category and the preset mapping relationship, and reading the data in the target worksheet when the verification result indicates that the verification is passed;

[0160] If the target processing type is the write processing type, target business data is acquired according to the data category and the preset mapping relationship, and the target business data is written into the target work table.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0162] Searching for a target category standard corresponding to the data category in the preset mapping relationship; the target category standard includes at least one of a header type of the worksheet, a number of cells corresponding to each header, and a data format of each cell;

[0163] Determine whether the data category is consistent with the target category standard. If so, determine that the verification result indicates that the verification is passed. If not, determine that the verification result indicates that the verification is failed.

[0164] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0165] Searching for a business data acquisition path corresponding to the data category in the preset mapping relationship;

[0166] The target business data is acquired from the business data acquisition path.

[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0168] A worksheet whose total amount of data in the table file does not exceed the capacity of the local memory is determined as the target worksheet.

[0169] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0170] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0171] Receiving a data processing request corresponding to a table file to be processed;

[0172] Determine the data volume of the worksheet in the table file according to the data processing request;

[0173] The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0175] Determine a target worksheet from all the worksheets in the table file according to the data volume of the worksheets in the table file and the capacity of the local memory;

[0176] The target worksheet is stored in the local memory, and the target worksheet is processed in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result;

[0177] The other worksheets except the target worksheet are stored in the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, a new target worksheet is determined from the other worksheets, and the step of storing the target worksheet in the local memory is returned according to the new target worksheet.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0179] Determine the target processing type of the target worksheet and the data category of the target worksheet;

[0180] If the target processing type is the read processing type, verifying the target worksheet according to the data category and the preset mapping relationship, and reading the data in the target worksheet when the verification result indicates that the verification is passed;

[0181] If the target processing type is the write processing type, target business data is acquired according to the data category and the preset mapping relationship, and the target business data is written into the target work table.

[0182] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0183] Searching for a target category standard corresponding to the data category in the preset mapping relationship; the target category standard includes at least one of a header type of the worksheet, a number of cells corresponding to each header, and a data format of each cell;

[0184] Determine whether the data category is consistent with the target category standard. If so, determine that the verification result indicates that the verification is passed. If not, determine that the verification result indicates that the verification is failed.

[0185] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0186] Searching for a business data acquisition path corresponding to the data category in the preset mapping relationship;

[0187] The target business data is acquired from the business data acquisition path.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] A worksheet whose total amount of data in the table file does not exceed the capacity of the local memory is determined as the target worksheet.

[0190] The above embodiment provides a computer program product, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0192] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Receiving a data processing request corresponding to a table file to be processed; Determine the data volume of the worksheet in the table file according to the data processing request; The table file is processed according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

2. The method according to claim 1, characterized in that: The processing of the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain the processing result includes: Determine a target worksheet from all the worksheets in the table file according to the data volume of the worksheets in the table file and the capacity of the local memory; The target worksheet is stored in the local memory, and the target worksheet is processed in the local memory according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result; The other worksheets except the target worksheet are stored in the local disk, and after detecting that the processing of the target worksheet in the local memory is completed, a new target worksheet is determined from the other worksheets, and the step of storing the target worksheet in the local memory is returned according to the new target worksheet.

3. The method according to claim 2, characterized in that The step of processing the target worksheet according to the processing type of the target worksheet, the data category of the target worksheet and the preset mapping relationship to obtain a processing result includes: Determine the target processing type of the target worksheet and the data category of the target worksheet; If the target processing type is a read processing type, verifying the target worksheet according to the data category and the preset mapping relationship, and reading the data in the target worksheet when the verification result indicates that the verification is passed; If the target processing type is a write processing type, target business data is acquired according to the data category and the preset mapping relationship, and the target business data is written into the target work table.

4. The method according to claim 3, characterized in that The verifying the target worksheet according to the data category and the preset mapping relationship includes: Searching for a target category standard corresponding to the data category in the preset mapping relationship; the target category standard includes at least one of a header type of the worksheet, a number of cells corresponding to each header, and a data format of each cell; Determine whether the data category is consistent with the target category standard. If so, determine that the verification result indicates that the verification is passed. If not, determine that the verification result indicates that the verification is failed.

5. The method according to claim 3, characterized in that: The acquiring target business data according to the data category and the preset mapping relationship includes: Searching for a business data acquisition path corresponding to the data category in the preset mapping relationship; The target business data is acquired from the business data acquisition path.

6. The method according to any one of claims 2 to 5, characterized in that: The step of determining a target worksheet from all worksheets in the table file according to the data amount of the worksheet in the table file and the capacity of the local memory comprises: A worksheet whose total amount of data in the table file does not exceed the capacity of the local memory is determined as the target worksheet.

7. A data processing device, characterized in that: The device comprises: A receiving module, used for receiving a data processing request corresponding to a table file to be processed; A determination module, used for determining the data amount of the worksheet in the table file according to the data processing request; A processing module is used to process the table file according to the data volume of the worksheet in the table file, the capacity of the local memory and the preset mapping relationship to obtain a processing result; the preset mapping relationship includes the correspondence between the data category, the target category standard and the business data acquisition entry.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images