Data importing method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to the field of databases, and discloses a data import method, device and equipment and a storage medium, the method comprises the following steps: obtaining a data import request initiated by a user and a corresponding error data threshold value, the data import request being used for representing a target database to which data to be imported needs to be imported, the error data threshold is used for representing the maximum error allowance when the to-be-imported data is imported into the target database; importing the to-be-imported data into a target database based on the data importing request, and counting error information of errors when the to-be-imported data is imported into the target database to obtain an error data size; and comparing the error data volume with an error data threshold value, and feeding back a comparison result to a user. By applying the technical scheme of the invention, the data import experience of the user can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of databases, and in particular to a data import method, device, equipment and storage medium. Background Art

[0002] In the related technology, when using a database for data analysis, it is usually necessary to import various massive data. Since the entire import process takes a long time and data import errors are prone to occur, it is urgent to globally control the erroneous data to prevent the erroneous data from affecting the entire import process and further affecting the user's data import experience. Summary of the invention

[0003] In view of the above problems, the embodiments of the present invention provide a data import method, apparatus, device and storage medium, which are used to solve the problem in the related art that it is difficult to globally control erroneous data during the process of importing data into a database, thereby affecting the user experience.

[0004] According to one aspect of an embodiment of the present invention, a data import method is provided, the method comprising: obtaining a data import request initiated by a user and a corresponding error data threshold, the data import request being used to characterize a target database into which the data to be imported needs to be imported, and the error data threshold being used to characterize a maximum allowable amount of errors when the data to be imported is imported into the target database; importing the data to be imported into the target database based on the data import request, and counting error information when the data to be imported is imported into the target database to obtain an amount of error data; comparing the amount of error data with the error data threshold, and feeding back the comparison result to the user.

[0005] In an optional implementation, based on the data import request, the data to be imported is imported into the target database, and error information of errors that occur when the data to be imported is imported into the target database is counted to obtain the amount of error data, including:

[0006] Divide the data to be imported into multiple data slices to be imported, each data slice to be imported includes multiple character strings to be imported, and the multiple character strings to be imported are separated by a separator;

[0007] Import multiple data shards to be imported into the target database in preset batches;

[0008] Based on the delimiter, the positions of the strings to be imported that have errors when each data slice to be imported is imported into the target database are counted to obtain the amount of error data, and the error information includes the position of the error data.

[0009] In an optional implementation, based on the data import request, the data to be imported is imported into the target database, and error information of errors that occur when the data to be imported is imported into the target database is counted to obtain the amount of error data, further comprising:

[0010] Based on the separator, the amount of error information that occurs when each data shard to be imported is imported into the target database is counted to obtain the amount of error data. The error information includes the amount of error information of the error data. The amount of error information is used to characterize the importance of the error data in the data shard to be imported.

[0011] In an optional implementation, comparing the amount of erroneous data with an erroneous data threshold and feeding back the comparison result to a user includes:

[0012] Compare the amount of error data with the error data threshold to obtain a comparison result;

[0013] If the comparison result indicates that the amount of erroneous data is greater than or equal to the erroneous data threshold, the import of the data to be imported is stopped, and the comparison result is fed back to the user.

[0014] In an optional implementation, comparing the amount of erroneous data with an erroneous data threshold and feeding back the comparison result to a user further includes:

[0015] If the comparison result indicates that the amount of erroneous data is less than the erroneous data threshold, the amount of imported data and the amount of erroneous data corresponding to the data to be imported are counted;

[0016] The amount of imported data and the amount of erroneous data are fed back to the user.

[0017] In an optional implementation, comparing the amount of erroneous data with an erroneous data threshold and feeding back the comparison result to a user further includes:

[0018] Receive error correction instructions from users regarding comparison results;

[0019] The erroneous data is corrected based on the error correction instruction, and the corrected erroneous data is imported into the target database.

[0020] In an optional implementation, obtaining a data import request initiated by a user includes:

[0021] Obtain the import identifier of the data to be imported and the corresponding data content;

[0022] Based on the data content, determine the sharding information of the data to be imported in the target database;

[0023] Generate a data import request based on the import identifier and shard information.

[0024] According to another aspect of an embodiment of the present invention, a data import device is provided, including: a data acquisition module, used to obtain a data import request initiated by a user and a corresponding error data threshold, the data import request is used to characterize a target database into which the data to be imported needs to be imported, and the error data threshold is used to characterize a maximum allowable amount of errors when the data to be imported is imported into the target database; a data statistics module, used to import the data to be imported into the target database based on the data import request, and to count error information when the data to be imported is imported into the target database to obtain an amount of error data; a result feedback module, used to compare the amount of error data with the error data threshold, and to feed back the comparison result to the user.

[0025] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the operation of one of the aforementioned data import methods.

[0026] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a computer device / apparatus to perform an operation of the aforementioned data import method.

[0027] The technical solution provided by the embodiment of the present invention obtains the data import request initiated by the user and the corresponding error data threshold, imports the data to be imported into the target database based on the data import request, and counts the error information of the data to be imported when it is imported into the target database, obtains the amount of error data, compares the amount of error data with the error data threshold, and feeds back the comparison result to the user, thereby realizing global control of error data and preventing the error data from affecting the entire import process. The user can perform more refined data import control based on the progress of the imported data, thereby improving the user's data import experience.

[0028] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0030] Figure 1 A schematic diagram of a process of a data import method provided by the present invention is shown;

[0031] Figure 2 Another schematic diagram of a data import method provided by the present invention is shown;

[0032] Figure 3 Another schematic diagram of a process of a data import method provided by the present invention is shown;

[0033] Figure 4 A schematic structural diagram of a data import device provided by the present invention is shown;

[0034] Figure 5 A schematic structural diagram of a computer device provided by the present invention is shown. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0036] When using a database for data analysis, it is usually necessary to import a variety of massive data. Since the entire import process takes a long time and data import errors are prone to occur, public cloud analytical PostgreSQL is based on greenplum technology and is used for OLAP data analysis. During the data import process, greenplum supports a variety of error handling methods, including skipping erroneous data, setting erroneous data to NULL, and transferring to an error table or error file.

[0037] However, the above processing method allows users to only view error files corresponding to some data at nodes where some data are imported, but cannot globally control the amount of error data, understand the data import progress and error data information, and the user experience is poor.

[0038] Greenplum: is an open source massively parallel processing (MPP) database suitable for big data analysis and data warehouse scenarios.

[0039] OLAP: Online Analytical Processing is a technology used for complex query and data analysis, commonly used in data warehouse and business intelligence applications.

[0040] gpload: A data loading tool provided by Greenplum, used to import data from files into the Greenplum database.

[0041] Figure 1FIG. 1 is a flow chart showing a first embodiment of a data import method of the present invention, which is executed by a cloud database service platform. Figure 1 As shown, the method comprises the following steps:

[0042] Step 110, obtaining the data import request initiated by the user and the corresponding error data threshold.

[0043] The data import request is used to represent the target database into which the data to be imported needs to be imported, and the error data threshold is used to represent the maximum allowable error amount when the data to be imported is imported into the target database.

[0044] In an optional manner, when obtaining a data import request initiated by a user, the import identifier of the data to be imported and the corresponding error information can be obtained first; based on the data content, the sharding information of the data to be imported in the target database is determined; based on the error information and the sharding information, a data import request is generated.

[0045] Specifically, for Greenplum, when users execute queries or insert statements, they send requests to the gpfdist (a tool for accelerating data import) process in parallel through shard nodes such as segment. At this time, the requests belonging to the same transaction (transaction number) are placed in the same session. Since the same transaction can contain multiple SQL commands, there is a possibility that two SQLs need to access the same external table; the same SQL query command may also scan the external table more than once. Therefore, the import identifier corresponding to the data to be imported can be defined through gpfdist, such as: transactionId+commandId+scanNo+path.

[0046] transactionId: transaction number;

[0047] commandId: SQL command number belonging to the same transaction number;

[0048] scanNo: The number of the external table scan planned by a SQL command.

[0049] It can be understood that by writing the above parameters in the data import request, when the segment communicates with gpfdist, this import identifier is passed as the session ID in the http header.

[0050] Furthermore, by determining the sharding information of the data to be imported in the target database based on the data content of the data to be imported, the data to be imported can be divided into multiple data shards and imported into the target database in batches in parallel, thereby improving the efficiency of data import; by generating a data import request based on error information and sharding information, the management node can import the data to be imported into the target database based on the data import request.

[0051] Step 120 , based on the data import request, the data to be imported is imported into the target database, and error information of errors that occur when the data to be imported is imported into the target database is counted to obtain the amount of error data.

[0052] As described above, by counting the error information of the data to be imported during the process of importing it into the target database, the real-time statistics of the amount of erroneous data to be imported are performed, so that users can control the data import behavior more finely and improve the data import experience.

[0053] In an optional manner, the data to be imported is imported into the target database based on the data import request, and the error information of the data to be imported when being imported into the target database is counted, and when the amount of error data is obtained, the data import request can be parsed through a management node such as a master node to obtain the import identifier and shard information corresponding to the data to be imported, and the data to be imported is divided into multiple data shards to be imported based on the import identifier and shard information, and the data shards to be imported include multiple strings to be imported; the multiple data shards to be imported are imported into the target database according to preset batches; the positions of the error data in each data shard to be imported when being imported into the target database are counted, and the amount of error data is obtained, and the error information includes the position of the error data. It can be understood that by processing multiple data shards in parallel and optimizing the use of storage and computing resources, the efficiency and reliability of data import are improved.

[0054] In an optional manner, the data to be imported is imported into the target database based on the data import request, and error information of errors occurring when the data to be imported is imported into the target database is counted. When the amount of erroneous data is obtained, the data import request can also be parsed by a management node such as a master node to obtain an import identifier and shard information corresponding to the data to be imported. Based on the import identifier and the shard information, the data to be imported is divided into multiple data shards to be imported, each of which includes multiple strings to be imported, and the multiple strings to be imported can be separated by a separator. The multiple data shards to be imported can be imported into the target database according to preset batches. At the same time, based on the separator, the positions of the erroneous strings to be imported occurring when each data shard to be imported is imported into the target database are counted to obtain the amount of erroneous data, and the error information includes the position of the erroneous data.

[0055] Specifically, the data to be imported can be first divided into multiple data shards to be imported through the master node in the cloud database service platform. The corresponding character string to be imported in each data shard to be imported can be determined based on the data processing capability of each shard node, or can be determined based on the data information of the data to be imported after horizontal segmentation, and then based on the data processing capability of each corresponding shard node after horizontal segmentation. Then, multiple data shards to be imported are imported into the target database according to preset batches, such as the number of rows of data. Then, based on delimiters such as newline delimiters, the positions of the character strings to be imported that have errors when each data shard to be imported is imported into the target database are counted, that is, the row number of the corresponding data shard to be imported where the character string to be imported has errors, thereby obtaining the amount of error data, that is, the number of rows of error input data.

[0056] In an optional manner, based on the delimiter, the positions of the strings to be imported where errors occur in each data shard to be imported when it is imported into the target database are counted, and when the amount of erroneous data is obtained, for any imported string, the position information of the imported string in the imported data set is identified, and the string to be imported that matches the position information is determined in the data shard to be imported; the imported string and the string to be imported are compared, and based on the comparison result, the position of the string to be imported where the error occurs in the data shard to be imported is determined.

[0057] Specifically, when comparing the imported string with the string to be imported and determining the position of the erroneous string to be imported in the data slice to be imported based on the comparison result, the first character length of the imported string can be compared with the second character length of the matching string to be imported. When the first character length is different from the second character length, it is determined that an error occurred in the import process of the string to be imported. At this time, the position of the erroneous string to be imported in the data slice to be imported can be determined based on the position of the separator adjacent to the string to be imported.

[0058] Furthermore, when comparing the imported string with the string to be imported, and determining the position of the string to be imported with an error in the data slice to be imported based on the comparison result, the target character value of each character in the imported string can also be obtained; the target character value is compared with the original character value carried in the matching string to be imported; if the target character value is different from the original character value, an error occurred in the import process of the string to be imported, and the position of the string to be imported with an error in the data slice to be imported can be determined based on the position of the separator adjacent to the string to be imported. For example, if the imported string is Aefgsh, the target first character of the imported string is A, and the target character value is 65.

[0059] In addition, when comparing the imported string with the string to be imported and determining the position of the erroneous string to be imported in the data shard to be imported based on the comparison result, a hash operation can be performed on the imported string to obtain a first hash value; the first hash value is compared with the second hash value carried in the matching string to be imported; if the first hash value is different from the second hash value, an error occurs in the import process of the string to be imported. At this time, the position of the erroneous string to be imported in the data shard to be imported can be determined based on the position of the separator adjacent to the string to be imported.

[0060] In an optional manner, the data to be imported is imported into the target database based on a data import request, and error information of errors occurring when the data to be imported is imported into the target database is counted to obtain the amount of erroneous data. The amount of error information of errors occurring when each data shard to be imported is imported into the target database can also be counted based on a delimiter to obtain the amount of erroneous data. The error information includes the amount of error information of the erroneous data, and the amount of error information is used to characterize the importance of the erroneous data in the data shard to be imported.

[0061] In an optional method, when calculating the amount of error information of erroneous data, the importance of each character string to be imported in the data slice to be imported in the current data slice to be imported can be calculated to obtain the character importance; the character importance of each character string to be imported in the current data slice to be imported is multiplied by the logarithm of the inverse of the character importance of each character string to be imported in the current data slice to be imported, and the sum operation is performed to obtain the amount of error information of the erroneous data.

[0062] It can be understood that by calculating the amount of error information of the error data, it is possible to determine the impact of the error data on the current data slice to be imported based on the error information amount, and then determine whether to correct or ignore the current error data.

[0063] Step 130, compare the amount of erroneous data with the erroneous data threshold, and feed back the comparison result to the user.

[0064] The comparison result between the error data amount and the error data threshold may include that the error data amount is less than the error data threshold, and that the error data amount is greater than or equal to the error data threshold.

[0065] In an optional manner, when the amount of erroneous data is compared with the erroneous data threshold and the comparison result is fed back to the user, the amount of erroneous data can be compared with the erroneous data threshold to obtain a comparison result; if the comparison result indicates that the amount of erroneous data is greater than or equal to the erroneous data threshold, the import of the data to be imported is stopped, and the comparison result is fed back to the user, such as displayed on the display interface of the terminal where the user is located. Among them, the erroneous data threshold can be set based on the needs of the user, or it can be determined based on the proportion of the amount of erroneous data in the corresponding data shard to be imported, or it can be determined based on the similarity between the erroneous data and the data to be imported.

[0066] The data import method of the embodiment of the present invention obtains the data import request initiated by the user and the corresponding error data threshold, imports the data to be imported into the target database based on the data import request, and counts the error information of the data to be imported when it is imported into the target database, obtains the amount of error data, compares the amount of error data with the error data threshold, and feeds back the comparison result to the user, thereby realizing global control of error data and preventing the error data from affecting the entire import process. The user can perform more refined data import control based on the progress of importing data, thereby improving the user's data import experience.

[0067] Figure 2 FIG. 1 is a flow chart showing another embodiment of the data import method of the present invention, wherein the method is executed by a computer device. Figure 2 As shown, the method comprises the following steps:

[0068] Step 210, obtaining the data import request initiated by the user and the corresponding error data threshold.

[0069] For details, please see Figure 1 Step 110 of the illustrated embodiment will not be described in detail here.

[0070] Step 220 , based on the data import request, the data to be imported is imported into the target database, and error information of errors that occur when the data to be imported is imported into the target database is counted to obtain the amount of error data.

[0071] For details, please see Figure 1 Step 120 of the illustrated embodiment will not be described in detail here.

[0072] Step 230, compare the amount of erroneous data with the erroneous data threshold, and feed back the comparison result to the user.

[0073] Specifically, the above step 230 includes:

[0074] Step 2301, compare the amount of erroneous data with the erroneous data threshold to obtain a comparison result.

[0075] As described above, by comparing the amount of erroneous data with the erroneous data threshold, it is determined based on the comparison result whether to continue importing the remaining data in the above-mentioned data slice to be imported.

[0076] Step 2302: If the comparison result indicates that the amount of erroneous data is less than the erroneous data threshold, the amount of imported data and the amount of erroneous data corresponding to the data to be imported are counted.

[0077] Specifically, when the amount of error data is less than the error data threshold, the import of the data to be imported into the target database continues, and at the same time, the amount of imported data and the amount of error data corresponding to the data to be imported are counted, so as to feed back the amount of imported data and the amount of error data to the user, or the amount of data to be imported, the amount of imported data, the amount of unimported data and the amount of error data are fed back to the user, such as being displayed on the display interface of the user's terminal, so that the user can understand the progress of the data import in a timely manner, and perform corresponding error data processing operations and remaining data import operations based on the number of errors.

[0078] In an optional manner, the amount of imported data and the amount of erroneous data may be counted using a data statistics function or a script.

[0079] For example, you can add the following data structure to the master's shared memory:

[0080] struct StatItem{

[0081] char sessionId

[64] ; / / session ID

[0082] int totalCount; / / Total number of rows

[0083] int totalErrorCount; / / Total number of rows with errors

[0084] int totalLoadCount; / / Number of loaded rows

[0085] int metux;

[0086] }, used to save statistics related to data loading.

[0087] Add pg protocol S. Each segment node uses this protocol to communicate with the master node through libpq (library function). The request data corresponding to this protocol is:

[0088] struct SRequest{

[0089] int type;

[0090] int count; / / Total number of data rows

[0091] int errcount; / / Number of error data rows

[0092] char **errordata;

[0093] }

[0094] When type=0, it means that the segment node reports the total number of rows to the master node; when a new session is started each time data is loaded, each segment node obtains the total number of rows of the corresponding file, reports it once, and the master node updates the totalCount field. When type=1, it means data reporting (progress reporting) during loading / importing. After receiving this message, the master node updates the shared memory information totalErrorCount and totalLoadCount. After receiving the message, the master node deserializes the data and writes it to the object storage. The analysis of the loading process is displayed through the show load process statement. At the same time, the progress information of the load is loaded from the shared memory and calculated using totalLoadCount / totalCount.

[0095] Step 2303, feeding back the amount of imported data and the amount of erroneous data to the user.

[0096] As described above, by feeding back the amount of imported data and the amount of erroneous data to the user, the user can timely understand the progress of data import and perform corresponding erroneous data processing operations and remaining data import operations based on the number of errors.

[0097] The data import method of the embodiment of the present invention obtains the data import request initiated by the user and the corresponding error data threshold, imports the data to be imported into the target database based on the data import request, and counts the error information of the data to be imported when it is imported into the target database, obtains the amount of error data, compares the amount of error data with the error data threshold, and feeds back the comparison result to the user, thereby realizing global control of error data and preventing the error data from affecting the entire import process. The user can perform more refined data import control based on the progress of importing data, thereby improving the user's data import experience.

[0098] Figure 3 FIG. 1 is a flow chart showing another embodiment of the data import method of the present invention, wherein the method is executed by a computer device. Figure 3 As shown, the method comprises the following steps:

[0099] Step 310, obtaining the data import request initiated by the user and the corresponding error data threshold.

[0100] For details, please see Figure 1 Step 110 of the illustrated embodiment will not be described in detail here.

[0101] Step 320: import the data to be imported into the target database based on the data import request, and count the error information of errors that occur when the data to be imported is imported into the target database to obtain the amount of error data.

[0102] For details, please see Figure 1 Step 120 of the illustrated embodiment will not be described in detail here.

[0103] Step 330, compare the amount of erroneous data with the erroneous data threshold, and feed back the comparison result to the user.

[0104] For details, please see Figure 1 Step 130 of the illustrated embodiment will not be described in detail here.

[0105] Step 340: receiving an error correction instruction fed back by the user regarding the comparison result.

[0106] Among them, the above-mentioned error correction instructions can be to re-import the data to be imported, re-import the data slice to be imported, completely replace the row where the erroneous character string to be imported in the data slice to be imported is located, and replace the erroneous character to be imported in the data slice to be imported.

[0107] For example, if an error occurs at the beginning of the import of the data to be imported into the target database, or if there are many consecutive erroneous data at the starting node, the error correction instruction can instruct to re-import the data to be imported. If there are fewer strings to be imported in the data segment to be imported, or if the amount of errors is large, the error correction instruction can instruct to re-import the data segment to be imported. If an error occurs at the beginning of the import of the data segment to be imported into the target database, or if there are many erroneous characters in the entire line, the error correction instruction can instruct to completely replace the line where the string to be imported with errors in the data segment to be imported is located. If there are fewer erroneous characters to be imported in the data segment to be imported, the error correction instruction can instruct to replace the erroneous characters to be imported in the data segment to be imported.

[0108] Step 350, correct the erroneous data based on the error correction instruction, and import the corrected erroneous data into the target database.

[0109] As mentioned above, by analyzing the error data in detail, we can provide users with more in-depth error handling suggestions to help users more effectively solve problems encountered during the data import process.

[0110] Specifically, if the error correction instruction is used to instruct the data to be imported to be re-imported, the data to be imported that has been imported into the above-mentioned target database can be cleared and then re-imported. If the error correction instruction is used to instruct the data segment to be imported to be re-imported, the data segment to be imported that has been imported into the above-mentioned target database can be cleared and then re-imported. If the error correction instruction is used to instruct the row where the string to be imported with an error in the data segment to be imported is located to be completely replaced, the position information of the imported string in the imported data set can be identified, such as based on the position of the separator adjacent to the imported string, the row number where the imported string is located is determined, and the row number where the string to be imported that matches the position information is determined in the data segment to be imported, thereby achieving complete replacement of the row where the string to be imported with an error is located. If the error correction instruction is used to instruct the replacement of the character to be imported with an error in the data segment to be imported, the position of the character to be imported with an error can be determined based on the position of the separator adjacent to the character to be imported with an error, and the corresponding string to be imported can be determined based on the position of the character to be imported with an error, and then replaced.

[0111] The data import method of the embodiment of the present invention obtains the data import request initiated by the user and the corresponding error data threshold, imports the data to be imported into the target database based on the data import request, and counts the error information of the data to be imported when it is imported into the target database, obtains the amount of error data, compares the amount of error data with the error data threshold, and feeds back the comparison result to the user, thereby realizing global control of the error data, preventing the error data from affecting the entire import process, and the user can perform more refined data import control based on the progress of the imported data, thereby improving the user's data import experience. That is, the present invention can more accurately control the number of rows with errors, count the overall progress of the data import, and improve the data quality by correcting the error data, thereby further improving the user's data import experience.

[0112] Figure 4 FIG. 1 is a schematic diagram showing a structure of an embodiment of a data import device of the present invention. Figure 4 As shown, the device comprises:

[0113] The data acquisition module 410 is used to obtain the data import request initiated by the user and the corresponding error data threshold. The data import request is used to represent the target database into which the data to be imported needs to be imported. The error data threshold is used to represent the maximum allowable error when importing the data to be imported into the target database.

[0114] The data statistics module 420 is used to import the data to be imported into the target database based on the data import request, and to count the error information when the data to be imported is imported into the target database to obtain the amount of error data.

[0115] The result feedback module 430 is used to compare the amount of erroneous data with the erroneous data threshold and feed back the comparison result to the user.

[0116] In an optional implementation, the data statistics module 420 includes:

[0117] The data segmentation unit is used to segment the data to be imported into a plurality of data segments to be imported, wherein the data segments to be imported include a plurality of character strings to be imported.

[0118] The data import unit is used to import multiple data shards to be imported into the target database according to preset batches.

[0119] The first data statistics unit is used to count the locations of erroneous data when each data slice to be imported is imported into the target database, and obtain the amount of erroneous data, wherein the error information includes the location of the erroneous data.

[0120] In an optional implementation, the data statistics module 420 further includes:

[0121] The second data statistics unit is used to count the amount of error information that occurs when each data slice to be imported is imported into the target database to obtain the amount of error data. The error information includes the amount of error information of the error data. The amount of error information is used to characterize the importance of the error data in the data slice to be imported.

[0122] In some optional implementations, the result feedback module 430 includes:

[0123] The data comparison unit is used to compare the amount of erroneous data with the erroneous data threshold to obtain a comparison result.

[0124] The first feedback unit is used to stop importing the data to be imported and feed back the comparison result to the user if the comparison result indicates that the amount of erroneous data is greater than or equal to an erroneous data threshold.

[0125] In some optional implementations, the result feedback module 430 further includes:

[0126] The third data statistics unit is used to count the amount of imported data and the amount of erroneous data corresponding to the data to be imported if the comparison result indicates that the amount of erroneous data is less than the erroneous data threshold.

[0127] The second feedback unit is used to feed back the amount of imported data and the amount of erroneous data to the user.

[0128] In some optional implementations, the result feedback module 430 further includes:

[0129] The instruction receiving unit is used to receive the error correction instruction fed back by the user in response to the comparison result.

[0130] The data correction unit is used to correct the erroneous data based on the error correction instruction and import the corrected erroneous data into the target database.

[0131] In some optional implementations, the data acquisition module 410 includes:

[0132] The information acquisition unit is used to acquire the import identification of the data to be imported and the corresponding data content.

[0133] The identification determination unit is used to determine the sharding information of the data to be imported in the target database based on the data content.

[0134] The request generation unit is used to generate a data import request based on the import identifier and the shard information.

[0135] The further functional description of each of the above modules and units is the same as that of the above corresponding method embodiments and will not be repeated here.

[0136] Through the above device and its components, the technical solution provided by the embodiment of the present invention has the following advantages:

[0137] The data import device of the embodiment of the present invention obtains the data import request initiated by the user and the corresponding error data threshold, imports the data to be imported into the target database based on the data import request, and counts the error information of the data to be imported when it is imported into the target database, obtains the amount of error data, compares the amount of error data with the error data threshold, and feeds back the comparison result to the user, thereby realizing global control of error data and preventing error data from affecting the entire import process. The user can also perform more refined data import control based on the progress of importing data, thereby improving the user's data import experience.

[0138] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0139] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0140] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0141] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a server cluster, a mobile communication network, and combinations thereof.

[0142] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0143] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0144] An embodiment of the present invention also provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on a computer device / data import apparatus, the computer device / data import apparatus executes the data import method in any of the above method embodiments.

[0145] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.

[0146] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.

[0147] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.

[0148] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data import method, characterized in that: The method comprises: Acquire a data import request initiated by a user and a corresponding error data threshold, wherein the data import request is used to indicate a target database into which the data to be imported needs to be imported, and the error data threshold is used to indicate a maximum allowable error amount when the data to be imported is imported into the target database; Importing the data to be imported into the target database based on the data import request, and counting error information when the data to be imported is imported into the target database to obtain an amount of error data; The amount of erroneous data is compared with the erroneous data threshold, and the comparison result is fed back to the user.

2. The method according to claim 1, characterized in that The step of importing the data to be imported into the target database based on the data import request, and counting error information of errors that occur when the data to be imported is imported into the target database to obtain an amount of error data includes: Dividing the data to be imported into a plurality of data slices to be imported, wherein the data slices to be imported include a plurality of character strings to be imported; Importing the plurality of data shards to be imported into the target database in preset batches; The positions of erroneous data occurring in each of the to-be-imported data slices when being imported into the target database are counted to obtain the amount of erroneous data, wherein the error information includes the positions of the erroneous data.

3. The method according to claim 2, characterized in that The step of importing the data to be imported into the target database based on the data import request, and counting error information of errors occurring when the data to be imported is imported into the target database to obtain an amount of error data, further includes: The amount of error information of each data slice to be imported when being imported into the target database is counted to obtain the amount of error data, wherein the error information includes the amount of error information of the error data, and the amount of error information is used to characterize the importance of the error data in the data slice to be imported.

4. The method according to claim 1, characterized in that The comparing the amount of erroneous data with the erroneous data threshold and feeding back the comparison result to the user includes: Comparing the amount of erroneous data with the erroneous data threshold to obtain a comparison result; If the comparison result indicates that the amount of erroneous data is greater than or equal to the erroneous data threshold, the import of the data to be imported is stopped, and the comparison result is fed back to the user.

5. The method according to claim 4, characterized in that The comparing the amount of erroneous data with the erroneous data threshold and feeding back the comparison result to the user further includes: If the comparison result indicates that the amount of erroneous data is less than the erroneous data threshold, then counting the amount of imported data and the amount of erroneous data corresponding to the data to be imported; The amount of imported data and the amount of erroneous data are fed back to the user.

6. The method according to claim 1, characterized in that The comparing the amount of erroneous data with the erroneous data threshold and feeding back the comparison result to the user further includes: receiving an error correction instruction fed back by the user regarding the comparison result; The erroneous data is corrected based on the error correction instruction, and the corrected erroneous data is imported into the target database.

7. The method according to claim 1, characterized in that The obtaining of the data import request initiated by the user includes: Obtaining an import identifier of the data to be imported and corresponding data content; Based on the data content, determining the sharding information of the data to be imported in the target database; The data import request is generated based on the import identifier and the shard information.

8. A data import device, characterized in that: The device comprises: A data acquisition module, used to acquire a data import request initiated by a user and a corresponding error data threshold, wherein the data import request is used to indicate a target database into which the data to be imported needs to be imported, and the error data threshold is used to indicate a maximum allowable error amount when the data to be imported is imported into the target database; A data statistics module, used for importing the data to be imported into the target database based on the data import request, and counting error information when the data to be imported is imported into the target database to obtain the amount of error data; The result feedback module is used to compare the amount of erroneous data with the erroneous data threshold and feed back the comparison result to the user.

9. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the data import method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device, the computer device executes the operation of the data import method according to any one of claims 1 to 7.

Images