Data Import Method, Data Import Device, Computer Equipment and Storage Medium

Abstract

The data import method, data import device, computer equipment and storage medium proposed in the present application realize the decomposition of the import process by mapping the import process to pending tasks, and the pending tasks include import parameter detection tasks, import data detection tasks and data import tasks, which helps to improve the import efficiency. During the process of the corresponding pending tasks, the task identifier is returned to the front end, so that the front end can query the task progress according to the task identifier, thereby improving the interactivity with the front end. In addition, the present application first performs the import parameter detection task and the import data detection task before the data import task, so that errors can be found and corrected in advance, reducing the possibility of data errors in the actual data import task process, and improving the efficiency of data import. In summary, the embodiments of the present application can improve the efficiency and flexibility of table data import, and also improve the interactivity with the front end.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data import method, a data import device, a computer device, and a storage medium. Background Art

[0002] Currently, in a low-code system, the table fields of each application are customized, the number of fields is not unified, and the data formats are also not unified, with too strong personalized requirements. The data interconnection between systems is usually achieved by exporting the data of System A to a table and then importing it into System B. Since the table formats between different systems are different, solving the data import between different systems and the resulting conversion of different fields and corresponding data formats has become a key research object in the industry. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a data import method, a data import device, a computer device, and a storage medium, which can improve the efficiency and interaction flexibility of data import.

[0004] To achieve the above object, a first aspect of the embodiments of this application proposes a data import method, and the method includes:

[0005] Receiving the to-be-imported table data sent by the front end;

[0006] Obtaining a to-be-processed task according to the to-be-imported table data; wherein, the to-be-processed task includes an import parameter detection task, an import data detection task, and a data import task;

[0007] Performing an import size detection and an import permission detection on the to-be-imported table data according to the import parameter detection task, and returning a first task identifier to the front end, so that the front end obtains the task progress of the import parameter detection task according to the first task identifier;

[0008] Obtaining first task result information of the import parameter detection task;

[0009] If the first task result information is passed, performing data detection on the to-be-imported table data according to the import data detection task, and returning a second task identifier to the front end, so that the front end obtains the task progress of the import data detection task according to the second task identifier;

[0010] Obtaining second task result information of the import data detection task;

[0011] If the second task result information is "passed", import the to-be-imported table data and the preset source table data according to the data import task to obtain target table data, and return a third task identifier to the front end, so that the front end can obtain the task progress of the data import task according to the third task identifier.

[0012] In some embodiments, the importing the to-be-imported table data and the preset source table data according to the data import task to obtain target table data includes:

[0013] Determine the mapping fields of the original fields of the to-be-imported table data according to a preset mapping relationship, where the source table data includes at least two source fields, the mapping field is any one of the at least two source fields, and the mapping relationship is sent by the front end;

[0014] Obtain the data conversion rule of the mapping field;

[0015] Convert the field data of the original field according to the data conversion rule to obtain target conversion data;

[0016] Import the target conversion data and the source table data through a preset import engine to obtain target table data.

[0017] In some embodiments, the importing the target conversion data and the source table data through a preset import engine to obtain target table data further includes:

[0018] During the import process, monitor the status of the target conversion data to obtain import status information;

[0019] If the import status information is "error", record the error information and send the error information to the front end; where the error information includes: error data and the data row number of the error data;

[0020] Receive feedback information from the front end; where the feedback information includes corrected data and the data row number of the corrected data;

[0021] Obtain a to-be-processed task according to the feedback information to obtain an updated processing task;

[0022] Re-import the data according to the updated processing task to update the target table data.

[0023] In some embodiments, the target conversion data includes at least two row data, and each row data includes the field data of the mapping field; the importing the target conversion data and the source table data through a preset import engine to obtain target table data includes:

[0024] If the preset import mode includes an addition mode and an update mode, traverse each line of data to obtain the identification code of each line of data; wherein, the import mode is sent by the front end.

[0025] If the identification code is an update identification code, the field data of the mapping field is update data. During the import processing, perform an update action on the field data of the source field according to the update data to obtain updated field data.

[0026] If the identification code is an addition identification code, the field data of the mapping field is addition data. During the import processing, perform an addition action on the field data of the source field according to the addition data to obtain added field data.

[0027] Update the source table data according to the updated field data and the added field data to obtain target table data.

[0028] In some embodiments, the converting the field data of the original field according to the data conversion rule to obtain target conversion data includes:

[0029] Pack the field data of the original field to obtain a combined data packet.

[0030] Match the combined data packet with a preset character array to obtain a matching character; wherein, the character array includes at least two initial characters, and the matching character is any one of the at least two initial characters.

[0031] Split the combined data packet according to the matching character to obtain data to be processed.

[0032] Convert the data to be processed according to the data conversion rule to obtain target conversion data.

[0033] In some embodiments, the mapping field includes a main table mapping field and a sub-table mapping field, the target conversion data includes at least two lines of data, and each line of data includes the field data of the main table mapping field and the field data of the sub-table mapping field. The method further includes:

[0034] Detect each line of data to obtain a detection result.

[0035] If the detection result is that the field data of the main table mapping field is not empty, the line of data includes main table data and sub-table data, and obtain the latest main table line number according to the line number of the line of data.

[0036] If the field data of the main table mapping field is empty and the field data of the sub-table mapping field is not empty in the detection result, then the row data includes sub-table data;

[0037] Associate the sub-table data with the main table data corresponding to the latest main table row number to update the target table data.

[0038] In some embodiments, the obtaining the target table data by performing an import process on the target conversion data and the source table data through a preset import engine further includes:

[0039] Obtain the data volume size of the table data to be imported according to the import size detection;

[0040] Set the storage threshold of the memory according to the data volume size;

[0041] During the import process, whenever the number of data entries of the target conversion data written into the memory is greater than the storage threshold, obtain a temporary file according to the target conversion data in the memory;

[0042] When the import process ends, merge all the temporary files to update the source table data and obtain the target table data.

[0043] To achieve the above object, a second aspect of the embodiments of the present application proposes a data import device, the device includes:

[0044] A table data receiving module, configured to receive the table data to be imported sent by the front end;

[0045] A task obtaining module, configured to obtain a task to be processed according to the table data to be imported; wherein, the task to be processed includes an import parameter detection task, an import data detection task, and a data import task;

[0046] An import parameter detection module, configured to perform an import size detection and an import permission detection on the table data to be imported according to the import parameter detection task, and return a first task identifier to the front end, so that the front end obtains the task progress of the import parameter detection task according to the first task identifier;

[0047] A first task result obtaining module, configured to obtain the first task result information of the import parameter detection task;

[0048] A data detection module, configured to, if the first task result information is passed, perform data detection on the table data to be imported according to the import data detection task, and return a second task identifier to the front end, so that the front end obtains the task progress of the import data detection task according to the second task identifier;

[0049] The second task result acquisition module is used to acquire the second task result information of the imported data detection task;

[0050] The import processing module is configured to, if the second task result information is passed, perform import processing on the to-be-imported table data and the preset source table data according to the data import task to obtain target table data, and return a third task identifier to the front end, so that the front end can obtain the task progress of the data import task according to the third task identifier.

[0051] To achieve the above object, a third aspect of the embodiments of the present application provides a computer device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect above is implemented.

[0052] To achieve the above object, a fourth aspect of the embodiments of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0053] The data import method, data import device, computer device, and storage medium provided by the present application map the import process into a to-be-processed task, and the to-be-processed task includes an import parameter detection task, an imported data detection task, and a data import task, realizing the decomposition of the import process, which helps to improve the import efficiency. During the process of the corresponding to-be-processed task, a task identifier is returned to the front end, so that the front end can query the task progress according to the task identifier, improving the interactivity with the front end. In addition, before the data import task, the present application first performs the import parameter detection task and the imported data detection task, which can discover and correct errors in advance, reducing the possibility of data errors during the actual data import task, and improving the efficiency of data import. In summary, the embodiments of the present application can improve the efficiency and flexibility of table data import, and also improve the interactivity with the front end by querying the task progress through the task identifier. Description of the Drawings

[0054] Figure 1 is a flowchart of the data import method provided by the embodiments of the present application;

[0055] Figure 2 is Figure 1 a flowchart of step S107 in

[0056] Figure 3 is Figure 2Flowchart of step S203 in

[0057] Figure 4 is Figure 2 Flowchart of step S204 in

[0058] Figure 5 is Figure 2 Another flowchart of step S204 in

[0059] Figure 6 Is the flowchart after step S204 in the embodiment of the present application;

[0060] Figure 7 is Figure 2 Another flowchart of step S204 in

[0061] Figure 8 Is the module structure block diagram of the data import device provided by the embodiment of the present application;

[0062] Figure 9 Is the schematic diagram of the hardware structure of the computer device provided by the embodiment of the present application. Detailed implementation manners

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0064] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the flowchart in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application, and are not intended to limit this application.

[0066] Currently, in a low-code system, the table fields of each application are customized, the number of fields is not uniform, and the data formats are also not uniform, with too strong personalized requirements. The data interconnection between systems is usually achieved by exporting the data of system A to a table and then importing it into system B. Since the table formats between different systems are different, solving the data import between different systems and the resulting conversion of different fields and corresponding data formats has become a key research object in the industry.

[0067] In the related art, the import function applicable to low-code systems on the market is not comprehensive, and the main disadvantages are as follows:

[0068] 1. One-stop import process. Usually, the import process is long. Users only know the result after waiting until the end and cannot intervene midway. Some data that does not conform to the system format cannot be modified and can only be imported again from the beginning, greatly reducing efficiency;

[0069] 2. Basic fields (strings, numbers, dates) can be imported, but the data format support for some fields is insufficient (for example, Chinese identification of ID cards, identification of digital percentages, and dates do not support. / - year / month / day separation);

[0070] 3. Advanced fields are not supported, such as associated data, detail tables, etc.;

[0071] 4. Mixed import is not supported, that is, both new and updated data exist in the same tabular data (for example, excel).

[0072] Based on this, the main purpose of the embodiments of the present application is to propose a data import method, a data import device, a computer device, and a storage medium, aiming to decompose the import process into multiple serial tasks, including an import parameter detection task, an import data detection task, and a data import task, and perform hierarchical processing on the to-be-imported tabular data according to the multiple serial tasks, so as to detect errors existing in the tabular data in advance for the front end to correct in advance, and return a task identifier in each hierarchical processing for the front end to query the task progress. The embodiments of the present invention can improve the efficiency and flexibility of tabular data import, and also improve the interactivity with the front end by querying the task progress through the task identifier. It should be noted that other technical effects of the present application will be discussed one by one below.

[0073] The embodiments of the present application provide a data import method, a data import device, a computer device, and a storage medium, which are specifically described through the following embodiments. First, the data import method in the embodiments of the present application is described.

[0074] Figure 1 is an optional flowchart of the data import method provided by the embodiments of the present application, Figure 1 The method in may include but is not limited to steps S101 to S107.

[0075] Step S101, receiving the to-be-imported tabular data sent by the front end;

[0076] Step S102, obtaining the to-be-processed tasks according to the to-be-imported tabular data; wherein, the to-be-processed tasks include an import parameter detection task, an import data detection task, and a data import task;

[0077] Step S103: Detect the import size and import permissions of the to-be-imported tabular data according to the import parameter detection task, and return the first task identifier to the front end, so that the front end can obtain the task progress of the import parameter detection task according to the first task identifier;

[0078] Step S104: Obtain the first task result information of the import parameter detection task;

[0079] Step S105: If the first task result information is passed, detect the data of the to-be-imported tabular data according to the import data detection task, and return the second task identifier to the front end, so that the front end can obtain the task progress of the import data detection task according to the second task identifier;

[0080] Step S106: Obtain the second task result information of the import data detection task;

[0081] Step S107: If the second task result information is passed, perform import processing on the to-be-imported tabular data and the preset source tabular data according to the data import task to obtain the target tabular data, and return the third task identifier to the front end, so that the front end can obtain the task progress of the data import task according to the third task identifier.

[0082] Steps S101 to S107 shown in the embodiments of the present application map the import process to the to-be-processed tasks, and the to-be-processed tasks include an import parameter detection task, an import data detection task, and a data import task, realizing the decomposition of the import process, which helps to improve the import efficiency. During the process of the corresponding to-be-processed tasks, the task identifier is returned to the front end, so that the front end can query the task progress according to the task identifier, improving the interactivity with the front end. In addition, in the present application, before the data import task, the import parameter detection task and the import data detection task are performed first, which can detect and correct errors in advance, reducing the possibility of data errors during the actual data import task, and improving the efficiency of data import. In summary, the embodiments of the present application can improve the efficiency and flexibility of tabular data import, and also improve the interactivity with the front end by querying the task progress through the task identifier.

[0083] In step S101 of some embodiments, the table data to be imported can be uploaded by the user from the front end to the back end (also referred to as the server side). The table data to be imported can refer to an Excel table. In one embodiment, the steps of creating a table at least include: creating a table name, table label, table type, field name, and data type of the field. In this embodiment, the table name is in pinyin or English, and the table label is in Chinese. The table type includes types such as user table, process table, and detail table. The field name is in Chinese, and the data type of the field includes text type, numeric type, date type, time type, etc. It can be understood that the field names and data types of fields selected for different tables will be different. Therefore, during the import process of the table data to be imported, it is extremely easy to have errors in field names and data types, causing the import process to be interrupted.

[0084] In step S102 of some embodiments, the task to be processed is pre-set in the server. Regardless of the type of table data to be imported submitted by the front end, it needs to undergo import processing through the task to be processed. Among them, the task to be processed includes an import parameter detection task, an import data detection task, and a data import task.

[0085] In step S103 of some embodiments, the import parameter detection task mainly performs routine verification on the table data to be imported, such as performing import size detection and import permission detection. A first task identifier is returned during the detection process so that the front end can obtain the task progress of the import parameter detection task based on the first task identifier. It should be noted that the import size detection is to detect the data volume of the table data to be imported to avoid situations such as out-of-memory overflow or crash due to too large a data volume. The import permission detection is to detect the import mode and permission of the table data to be imported, and this detection is used to avoid the import process being interrupted due to lack of permission for the corresponding import mode. Among them, the import mode includes update mode, new addition mode, and mixed mode. The mixed mode includes the update mode and the new addition mode.

[0086] In step S104 of some embodiments, the first task result information includes passed and not passed. Among them, if it is passed, the import data detection task is performed. If it is not passed, it means that an error occurred in the import parameter detection, resulting in the task being interrupted. An error report can be generated and returned to the front end so that the front end can modify the content of the error report and perform the import parameter detection task again. It should be noted that because the first task identifier is marked, even if the task is interrupted, when the modified report returned by the front end is received, the import parameter detection task continues instead of starting from the beginning.

[0087] In step S105 of some embodiments, the imported data detection task mainly verifies the field data and data types of the fields in the table data to be imported. For example, business rule detection (such as visible column detection, required data detection, duplicate data detection) and data format detection. It should be noted that if there is incorrect data in the imported data detection task, the incorrect data will be written into an error report and returned to the front end so that it can be corrected before data import. In one example, the reasons for the occurrence of incorrect data include unrecognizable data formats (such as text to number conversion, where the text contains English letters) and unmatched business rules (such as setting that field data is not allowed to be repeated, but there are duplicate values in the input table data). Taking duplicate data detection in business rule detection as an example, the repetition rate of names is relatively high. When duplicate names are found by querying the field data, an error report will be written and returned to the front end for correction before data import. Other reasons for errors include that a required result value is not filled in although the field is set as required; setting that submission is not allowed when the data of field A is equal to a certain value, etc. Taking data format detection as another example, the text length exceeds the limit (maximum 2000); there are failures in special field conversions when converting text to numbers, etc.

[0088] In step S106 of some embodiments, the second task result information includes passed and failed. Among them, if it is passed, the data import task is performed. If it is failed, it means that an error occurred in the imported data detection, resulting in task interruption. An error report can be generated and returned to the front end so that the front end can modify the content of the error report and perform the imported data detection task again. It should be noted that since the second task identifier is marked, even if task interruption occurs, when the modified report returned by the front end is received, the imported data detection task continues instead of starting from the beginning.

[0089] In step S107 of some embodiments, after the above two detection tasks pass, the table data to be imported is imported, and finally the target table data is obtained. The import process is executed asynchronously to enhance user interaction and reduce blocking operations; an asynchronous task system is used for concurrent control to ensure that tasks are not repeatedly executed; the Redis cache is used for idempotency control during the import process to avoid unexpected situations such as duplicate data insertion when the task is retried.

[0090] Please refer to Figure 2 , in some embodiments, step S107 may include but is not limited to steps S201 to S204.

[0091] Step S201, determine the mapped fields of the original fields of the table data to be imported according to the preset mapping relationship. Among them, the source table data includes at least two source fields, and the mapped field is any one of the at least two source fields. The mapping relationship is sent by the front end;

[0092] Step S202, obtain the data conversion rule of the mapped field;

[0093] Step S203, convert the field data of the original field according to the data conversion rule to obtain the target conversion data;

[0094] Step S204, perform an import process on the target conversion data and the source table data through a preset import engine to obtain the target table data.

[0095] Steps S201 to S204 illustrated in the embodiments of the present application. The table data to be imported includes multiple original fields and the data of the original fields. The source table data includes multiple source fields and the data of the source fields. The mapping relationship determines the mapped field of the original field, and the mapped field is one of the multiple source fields. In the source table data, each source field has a corresponding data format. Therefore, the corresponding data conversion rule can be determined according to the mapped field, and the field data of the original field can be converted according to the data conversion rule to obtain the target conversion data, and the target table data is obtained after the import. Among them, the mapping relationship is specified by the user. For example, at the front end, the user specifies which original fields are associated with which source fields, and the front end obtains the mapping relationship according to the user's specification, so that the back end will determine the mapped field of the original field of the table data to be imported according to the mapping relationship.

[0096] Please refer to Figure 3 , in some embodiments, step S203 may include but is not limited to steps S301 to S304.

[0097] Step S301, package the field data of the original field to obtain a combined data packet;

[0098] Step S302, match the combined data packet with a preset character array to obtain a matching character; wherein, the character array includes at least two initial characters, and the matching character is any one of the at least two initial characters;

[0099] Step S303, split the combined data packet according to the matching character to obtain the data to be processed;

[0100] Step S304, convert the data to be processed according to the data conversion rule to obtain the target conversion data.

[0101] Steps S301 to S304 shown in the embodiments of the present application. The character array includes multiple initial characters, and the initial characters include special delimiter characters such as "space", "-", ",", " / ", "AM", "PM", "%", etc. Taking the field type of the original field as the date type as an example, if the field data of the original field is "2022 / 07 / 22", then the combined data packet is "2022 / 07 / 22", where the combined data packet can also be referred to as a string set. Matching the combined data packet with the character array, the matching character obtained is " / ", and thus the data to be processed is "2022", "07", and "22". The data conversion rule can be to connect the data with "-" as the connection character, that is, convert the data to be processed according to the data conversion rule to obtain the target conversion data as "2022-07-22". Or, the data conversion rule is to connect the data with "year, month, day" as the connection character, that is, convert the data to be processed according to the data conversion rule to obtain the target conversion data as "July 22, 2022". In some other embodiments, if the field type of the original field is an ID card field, the data conversion rule is to exclude the internal spaces and Chinese characters and perform regular extraction to obtain the target conversion data.

[0102] Please refer to Figure 4 , in some embodiments, step S204 may include but is not limited to steps S401 to S405.

[0103] Step S401, during the import processing, monitor the status of the target conversion data to obtain the import status information;

[0104] Step S402, if the import status information is an error, record the error information and send the error information to the front end; where the error information includes: the error data and the data line number of the error data;

[0105] Step S403, receive the feedback information from the front end; where the feedback information includes the corrected data and the data line number of the corrected data;

[0106] Step S404, obtain the task to be processed according to the feedback information to obtain the updated processing task;

[0107] Step S405, re-import the data according to the updated processing task to update the target table data.

[0108] In the steps S401 to S405 illustrated in the embodiments of the present application, even if the table data to be imported has been detected twice before the data import task, import errors may still occur during the import process. Therefore, the embodiments of the present application determine whether the import status is normal through the import status information. It should be noted that since the table data to be imported has been detected twice before, even if the import status information is incorrect, it does not affect the import process. The corresponding incorrect data can be skipped first. The incorrect data and the corresponding data line numbers are sent to the front end so that the front end can modify it to obtain feedback information. The back end re-obtains the task to be processed according to the feedback information, that is, updates the processing task. The corrected data and the data line numbers of the corrected data are imported through the updated processing task to update the target table data. Among them, the updated processing task also includes an import parameter detection task, an imported data detection task, and a data import task.

[0109] Please refer to Figure 5 , in some other embodiments, the target conversion data includes at least two rows of data, and each row of data includes the mapped field data of the mapped field; step S204 may include but is not limited to steps S501 to S504.

[0110] Step S501, if the preset import mode includes the add mode and the update mode, traverse each row of data to obtain the identification code of each row of data; wherein, the import mode is sent by the front end;

[0111] Step S502, if the identification code is an update identification code, the field data of the mapped field is update data, and during the import process, the update action is performed on the field data of the source field according to the update data to obtain the updated field data;

[0112] Step S503, if the identification code belongs to the add identification code, the field data of the mapped field is add data, and during the import process, the add action is performed on the field data of the source field according to the add data to obtain the added field data;

[0113] Step S504, update the source table data according to the updated field data and the added field data to obtain the target table data.

[0114] In the steps S501 to S504 illustrated in the embodiments of the present application, there are several import modes for importing the to-be-imported tabular data, that is, the import modes include an addition mode, an update mode, and a hybrid mode, and the hybrid mode includes the addition mode and the update mode. That is to say, if the import mode includes the addition mode and the update mode, it indicates that there are updated data and newly added data in the to-be-imported tabular data, but the related art cannot identify them, thus unable to solve the problem of mixed import in the same to-be-imported tabular data. In the embodiments of the present application, first determine whether it is the hybrid mode by receiving the import mode sent by the front end. If the import mode is the hybrid mode, it is necessary to traverse all row data to obtain the identification code of each row of data. Among them, the update identification code is a unique identification code, and the position of the field data of the source field can be determined according to the unique identification code. Therefore, if the identification code is the update identification code, that is, the identification code is the unique identification code, then replace the field data of the source field with the newly added data according to the unique identification code to implement the addition process of the data import process. If the identification code is the new addition identification code, it is necessary to add a new row under the column to which the source field belongs and fill in the newly added data to implement the addition process of the data import process. It can be understood that if the import mode only includes the addition mode, it indicates that all the to-be-imported tabular data are newly added data. After confirming the corresponding source fields, it is possible to perform the addition action on the source tabular data without confirming the identification code to update the source tabular data and finally obtain the target tabular data. If the import mode only includes the update mode, it is the same as the addition mode.

[0115] Please refer to Figure 6 , in some other embodiments, the mapping fields include a main table mapping field and a sub-table mapping field, the target conversion data includes at least two rows of data, and each row of data includes the field data of the main table mapping field and the field data of the sub-table mapping field. After step S204, it may further include but is not limited to steps S601 to S604.

[0116] Step S601, detect each row of data to obtain a detection result;

[0117] Step S602, if the detection result shows that the field data of the main table mapping field is not empty, then the row data includes main table data and sub-table data, and obtain the latest main table row number according to the row number of the row data;

[0118] Step S603, if the detection result shows that the field data of the main table mapping field is empty and the field data of the sub-table mapping field is not empty, then the row data includes sub-table data;

[0119] Step S604, associate the sub-table data with the main table data corresponding to the latest main table row number to update the target tabular data.

[0120] Steps S601 to S604 shown in the embodiments of the present application determine the association relationship between the main table and the sub-table by identifying the main table data and the sub-table data, improving the accuracy of data import. Referring to Table 1, the detailed list is subordinate to the main table, that is, one main table data can be associated with multiple sub-table data, that is, one-to-many. It should be noted that it will be preset which fields belong to the main table mapping fields and which fields belong to the sub-table mapping fields. Taking Table 1 as an example, the order number, enterprise name, and total amount belong to the main table fields, and the rest belong to the sub-table fields, and they are distinguished with the main table fields on the left and the sub-table fields on the right. Among them, the first row of Table 1 is the field name and does not belong to the field data, that is, it is not the row data referred to in the embodiments of the present application.

[0121] When importing the field data of the detailed list (main table and sub-table), the most important thing is to determine whether the row data of the current row number is the main table data or the sub-table data. If it is the sub-table data, it is necessary to know which main table data it is. Therefore, in the embodiments of the present application, the field data of each row data is detected. When an entire row belonging to the main table mapping field has no assignment, it is considered that the field data of this row is empty. If the field data of all the main table mapping fields is empty and the field data of the sub-table fields is not empty, the row data is the sub-table data, and vice versa for the main table data. If the row data is the sub-table data, it is associated with the nearest main table data. For example, for the row data corresponding to 001 in Table 1, it can be recognized that there is 1 main table data and 1 sub-table data, then this sub-table data will be associated with the main table data with the order number 001. Only 1 sub-table data is recognized in the third and fourth rows of Table 1, then it will also be associated with the main table data with the order number 001.

[0122] Table 1:

[0123] Order Number Enterprise Name Total Amount Product Details, Items Product Details, Amount 001 Company A 5200 Computer 5000 Mouse 100 Keyboard 100 002 Company B 315 Book 300 Pen 10 Eraser 15

[0124] Please refer to Figure 7 , in some other embodiments, in step S204, the data import method may further include but is not limited to steps S701 to S704.

[0125] Step S701, detecting the data volume size of the table data to be imported according to the import size;

[0126] Step S702, setting the storage threshold of the memory according to the data volume size;

[0127] Step S703, during the import processing, whenever the number of data entries of the target conversion data written into the memory is greater than the storage threshold, a temporary file is obtained according to the target conversion data in the memory;

[0128] Step S704, when the import processing is completed, all the temporary files are merged to update the source table data and obtain the target table data.

[0129] Steps S701 to S704 shown in the embodiment of the present application take into account that when encountering operations with large amounts of table data to be imported (for example, more than 100M), the service is prone to memory overflow and crash. In view of this situation, the import and export component used in the embodiment of the present application is NPOI, which provides three modes, namely HSSF, XSSF and SXSSF. Among them, HSSF and XSSF will consume more memory for parsing or writing large amounts of table data, because it involves the style, validator, formula, etc. of the table to be rendered. When writing table data, SXSSF will set a storage threshold. If it is set to 100, then every time more than 100 data are written to SXSSF, SXSSF will write temporary files in real time. When all data are written, SXSSF will merge these temporary files into a complete table data file. Since the storage threshold can be configured independently, it can be evaluated to what value the memory consumption is small, thereby achieving the effect of reducing memory usage. It should be noted that in the embodiment of the present application, the storage threshold of the memory is set according to the amount of data, which can avoid the situation of memory crash and improve the import efficiency.

[0130] See also Figure 8 The embodiment of the present application also provides a data import device, which can implement the above data import method. Figure 8It is a block diagram of the module structure of the data import device provided by the embodiment of the present application. The device includes: a table data receiving module 801, a task obtaining module 802, an import parameter detection module 803, a first task result obtaining module 804, a data detection module 805, a second task result obtaining module 806, and an import processing module 807. Among them, the table data receiving module 801 is used to receive the table data to be imported sent by the front end; the task obtaining module 802 is used to obtain the task to be processed according to the table data to be imported; among them, the task to be processed includes an import parameter detection task, an import data detection task, and a data import task; the import parameter detection module 803 is used to perform import size detection and import permission detection on the table data to be imported according to the import parameter detection task, and return a first task identifier to the front end, so that the front end can obtain the task progress of the import parameter detection task according to the first task identifier; the first task result obtaining module 804 is used to obtain the first task result information of the import parameter detection task; the data detection module 805 is used to, if the first task result information is passed, perform data detection on the table data to be imported according to the import data detection task, and return a second task identifier to the front end, so that the front end can obtain the task progress of the import data detection task according to the second task identifier; the second task result obtaining module 806 is used to obtain the second task result information of the import data detection task; the import processing module 807 is used to, if the second task result information is passed, perform import processing on the table data to be imported and the preset source table data according to the data import task to obtain the target table data, and return a third task identifier to the front end, so that the front end can obtain the task progress of the data import task according to the third task identifier.

[0131] It should be noted that the specific implementation manner of this data import device is basically the same as the specific embodiment of the above data import method, and will not be elaborated here.

[0132] The embodiment of the present application also provides a computer device, which includes: a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing the connection and communication between the processor and the memory. When the program is executed by the processor, the above data import method is realized. This computer device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0133] Please refer to Figure 9 , Figure 9 which shows the hardware structure of a computer device in another embodiment. The computer device includes:

[0134] The processor 901 can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0135] The memory 902 can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the data import method of the embodiments of the present application;

[0136] The input / output interface 903 is used to implement information input and output;

[0137] The communication interface 904 is used to implement communication interaction between this device and other devices, and can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0138] The bus 905 transmits information between the various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0139] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 achieve communication connections with each other inside the device through the bus 905.

[0140] The embodiments of the present application also provide a storage medium. The storage medium is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above data import method.

[0141] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include memories remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The data import method, data import device, computer device, and storage medium provided by the embodiments of the present application decompose the import process by mapping the import process into a task to be processed, and the task to be processed includes an import parameter detection task, an imported data detection task, and a data import task, which helps to improve the import efficiency. During the process of the corresponding task to be processed, a task identifier is returned to the front end, so that the front end can query the task progress according to the task identifier, improving the interactivity with the front end. In addition, in the present application, before the data import task, the import parameter detection task and the imported data detection task are first performed, which can detect and correct errors in advance, reducing the possibility of data errors during the actual data import task, and improving the efficiency of data import. In summary, the embodiments of the present application can improve the efficiency and flexibility of tabular data import, and also improve the interactivity with the front end by querying the task progress through the task identifier.

[0143] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0144] Those skilled in the art can understand that Figure 1 - 7 the technical solutions shown do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown, or combine certain steps, or different steps.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or a suitable combination thereof.

[0147] As used in the specification of this application and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0148] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may mean: only A exists, only B exists, and both A and B exist at the same time. Here, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or similar expressions refer to any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0149] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical, or other form.

[0150] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, in each embodiment of this application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0152] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of this application. The foregoing storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0153] The preferred embodiments of the embodiments of this application have been described above with reference to the drawings, and thus do not limit the scope of rights of the embodiments of this application. Any modification, equivalent replacement, and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall be within the scope of rights of the embodiments of this application.

Claims

1. A data import method, characterized in that, the method includes: Receiving the table data to be imported sent by the front end; Obtaining a task to be processed according to the table data to be imported; wherein, the task to be processed includes an import parameter detection task, an import data detection task, and a data import task; Performing an import size detection and an import permission detection on the table data to be imported according to the import parameter detection task, and returning a first task identifier to the front end, so that the front end can obtain the task progress of the import parameter detection task according to the first task identifier; Obtaining first task result information of the import parameter detection task; If the first task result information is passed, performing data detection on the table data to be imported according to the import data detection task, and returning a second task identifier to the front end, so that the front end can obtain the task progress of the import data detection task according to the second task identifier; Obtaining second task result information of the import data detection task; If the second task result information is passed, performing an import process on the table data to be imported and the preset source table data according to the data import task to obtain target table data, and returning a third task identifier to the front end, so that the front end can obtain the task progress of the data import task according to the third task identifier.

2. The method according to claim 1, characterized in that, the performing an import process on the table data to be imported and the preset source table data according to the data import task to obtain target table data includes: Determining a mapped field of the original field of the table data to be imported according to a preset mapping relationship, wherein the source table data includes at least two source fields, the mapped field is any one of the at least two source fields, and the mapping relationship is sent by the front end; Obtaining a data conversion rule of the mapped field; Converting the field data of the original field according to the data conversion rule to obtain target conversion data; Performing an import process on the target conversion data and the source table data through a preset import engine to obtain target table data.

3. The method according to claim 2, characterized in that, the performing an import process on the target conversion data and the source table data through a preset import engine to obtain target table data further includes: During the import process, monitoring the status of the target conversion data to obtain import status information; If the import status information is an error, recording the error information and sending the error information to the front end; wherein, the error information includes: error data and the data row number of the error data; Receiving feedback information from the front end; wherein, the feedback information includes correction data and the data row number of the correction data; Obtaining a task to be processed according to the feedback information to obtain an updated processing task; Performing data import again according to the updated processing task to update the target table data.

4. The method according to claim 2, characterized in that, The target conversion data includes at least two rows of data, and each row of data includes field data of a mapping field; The importing process of the target conversion data and the source table data through a preset importing engine to obtain target table data includes: If the preset importing mode includes an adding mode and an updating mode, traverse each row of data to obtain the identification code of each row of data; wherein, the importing mode is sent by the front end; If the identification code is an update identification code, the field data of the mapping field is update data, and during the importing process, update actions are performed on the field data of the source field according to the update data to obtain updated field data; If the identification code is an adding identification code, the field data of the mapping field is adding data, and during the importing process, adding actions are performed on the field data of the source field according to the adding data to obtain added field data; Update the source table data according to the updated field data and the added field data to obtain target table data.

5. The method according to claim 2, wherein, The conversion of the field data of the original field according to the data conversion rule to obtain target conversion data includes: Pack the field data of the original field to obtain a combined data packet; Match the combined data packet with a preset character array to obtain a matching character; wherein, the character array includes at least two initial characters, and the matching character is any one of the at least two initial characters; Split the combined data packet according to the matching character to obtain data to be processed; Convert the data to be processed according to the data conversion rule to obtain target conversion data.

6. The method according to claim 2, wherein, The mapping field includes a main table mapping field and a sub-table mapping field, the target conversion data includes at least two rows of data, and each row of data includes field data of the main table mapping field and field data of the sub-table mapping field. The method further includes: Detect each row of data to obtain a detection result; If the detection result is that the field data of the main table mapping field is not empty, the row of data includes main table data and sub-table data, and the latest main table row number is obtained according to the row number of the row of data; If the detection result is that the field data of the main table mapping field is empty and the field data of the sub-table mapping field is not empty, the row of data includes sub-table data; Associate the sub-table data with the main table data corresponding to the latest main table row number to update the target table data.

7. The method according to any one of claims 2 to 6, wherein, The importing process of the target conversion data and the source table data through a preset importing engine to obtain target table data further includes: Obtain the data volume size of the table data to be imported according to the import size detection; Set the storage threshold of the memory according to the data volume size; During the import process, whenever the number of data entries of the target conversion data written to the memory is greater than the storage threshold, a temporary file is obtained based on the target conversion data in the memory; When the import process ends, all the temporary files are merged to update the source table data and obtain the target table data.

8. A data import device, characterized in that, the device includes: a table data receiving module, configured to receive the table data to be imported sent by the front end; a task obtaining module, configured to obtain a task to be processed according to the table data to be imported; wherein, the task to be processed includes an import parameter detection task, an import data detection task, and a data import task; an import parameter detection module, configured to perform an import size detection and an import permission detection on the table data to be imported according to the import parameter detection task, and return a first task identifier to the front end, so that the front end obtains the task progress of the import parameter detection task according to the first task identifier; a first task result obtaining module, configured to obtain the first task result information of the import parameter detection task; a data detection module, configured to, if the first task result information is passed, perform data detection on the table data to be imported according to the import data detection task, and return a second task identifier to the front end, so that the front end obtains the task progress of the import data detection task according to the second task identifier; a second task result obtaining module, configured to obtain the second task result information of the import data detection task; an import processing module, configured to, if the second task result information is passed, perform an import process on the table data to be imported and the preset source table data according to the data import task to obtain the target table data, and return a third task identifier to the front end, so that the front end obtains the task progress of the data import task according to the third task identifier.

9. A computer device, characterized in that, the computer device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory. When the program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium, the storage medium is a computer-readable storage medium for computer-readable storage, characterized in that, the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the method according to any one of claims 1 to 7.

Images