A data import method and apparatus

Abstract

The application provides a data import method and device, and relates to the field of big data. The method comprises the following steps: importing target data into a target address, wherein the target data comprises data corresponding to at least one primary key, and each primary key corresponds to one or more pieces of data; in the case that the first data in the target data fails to be imported, writing the first data and the data corresponding to the first primary key and located after the first data into a preset address, wherein the preset address is different from the target address, the first primary key is the primary key corresponding to the first data; and importing the data in the preset address into the target address after the target data is imported. By reimporting the first data and the data after the first data, the problem that the order of the data in the target address is inconsistent with the order of the source data can be avoided. In addition, in the case that the first data fails to be imported, the target data does not need to be reimported completely, which is beneficial to reducing the cost and improving the import efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and in particular to a data import method and device. BACKGROUND

[0002] In some special scenarios, such as initializing a new table, data migration, and the like, it is a crucial step to import a large amount of data to a target address. During the data import process, part of the data may fail to be imported due to network jitter and the like. In this case, the data that fails to be imported can be reimported to the target address. However, this may result in inconsistency between the order of the data in the target address and the order of the source data. For example, before the target address is imported, data D1 is located before data D2, but data D1 fails to be imported and needs to be reimported, while data D2 is successfully imported to the target address, and the reimported data D1 can be located after data D2, thereby resulting in inconsistency between the order of the data in the target address and the order of the source data.

[0003] Therefore, it is desirable to provide a data import method that can ensure consistency between the data in the target address and the source data. SUMMARY

[0004] The present application provides a data import method and device to ensure consistency between the data in the target address and the source data.

[0005] In a first aspect, the present application provides a data import method, which can be executed by a data import device.

[0006] Exemplarily, the method comprises: importing target data to a target address, the target data comprising data corresponding to at least one primary key, each primary key corresponding to one or more pieces of data; in the case that a first piece of data in the target data fails to be imported, writing the first piece of data and data corresponding to a first primary key and located after the first piece of data to a preset address, the preset address being different from the target address, the first primary key being a primary key corresponding to the first piece of data; and after the target data is imported, importing the data in the preset address to the target address.

[0007] In the technical solution, in the process of importing the target data into the target address, in the case that the partial data (e.g., the first data) fails to be imported, the data with the same primary key as the first data and located after the first data can be written into a preset address, and after the target data is imported, the data in the preset address can be imported into the target address. Therefore, once the first data fails to be imported, the data located after the first data in the data corresponding to the first primary key is temporarily imported into the preset address instead of the target address. In this way, the problem that the data located after the first data in the data corresponding to the first primary key is imported into the target address earlier than the first data, resulting in inconsistency between the order of the data in the target address and the order of the source data, can be avoided, thereby facilitating to ensure consistency between the data in the target address and the source data, and additionally, without the need to re-import the target data, facilitating to reduce the overhead and improve the import efficiency.

[0008] With reference to the first aspect, in a possible implementation of the first aspect, the importing the target data into the target address comprises: reading the target data from a source address; processing the target data based on an import configuration file to obtain processed target data; wherein the import configuration file is used to define a rule for processing the target data, and the import configuration file is generated based on a predefined import configuration syntax; and writing the processed target data into the target address.

[0009] With reference to the first aspect, in a possible implementation of the first aspect, the import configuration syntax comprises syntax for defining one or more of the following parameters: an encoding type, a table name, a separator between fields, a field name and a field type, a special character processing format, an addable field, and a number of threads allowed to be parallel.

[0010] With reference to the first aspect, in a possible implementation of the first aspect, before reading the target data from the source address, the method further comprises: in response to a user operation, obtaining an address of the import configuration file, the source address, and the target address.

[0011] With reference to the first aspect, in a possible implementation of the first aspect, the method further comprises: after processing the second data in the target data based on the import configuration file, determining whether a second primary key corresponding to the second data is the same as the first primary key; and in the case that the first data fails to be imported, writing the data located after the first data in the data corresponding to the first primary key into a preset address, comprising: in the case that the second primary key corresponding to the second data is the first primary key, writing the processed second data into the preset address.

[0012] With reference to the first aspect, in a possible implementation manner of the first aspect, the method further includes: creating one reading thread, at least one processing thread, and at least one writing thread for the target data, the processing thread is configured to process the target data based on the import configuration file, and the writing thread is configured to write the processed target data to the target address.

[0013] With reference to the first aspect, in a possible implementation manner of the first aspect, data of the same primary key corresponds to the same processing thread, and data of the same processing thread corresponds to the same writing thread.

[0014] With reference to the first aspect, in a possible implementation manner of the first aspect, the method further includes: in a case where a quantity of data processed by each processing thread of the at least one processing thread reaches a preset threshold, recording position offset information of the processed data, the position offset information of the processed data is configured to indicate an order of a last piece of data in the data processed by the each processing thread in this time in the target data.

[0015] With reference to the first aspect, in a possible implementation manner of the first aspect, the method further includes: in a case where the processing of the target data is interrupted, taking a minimum value of the position offset information of the processed data recorded by each thread of the at least one processing thread last time as a starting position, and continuing to import the target data.

[0016] In a second aspect, a data import apparatus is provided, which includes a first processing unit and a second processing unit, the first processing unit is configured to import target data to a target address, the target data includes data corresponding to at least one primary key, and each primary key corresponds to one or more pieces of data; the second processing unit is configured to, in a case where a first piece of data in the target data fails to be imported, write data after the first piece of data in data corresponding to a first primary key to a preset address, the preset address is different from the target address, and the first primary key is a primary key corresponding to the first piece of data; and the first processing unit is further configured to, after the target data is imported, import the data in the preset address to the target address.

[0017] In a third aspect, a data import apparatus is provided, which includes a processor. The processor is coupled with a memory and is configured to execute a computer program in the memory to implement the method in the first aspect and any possible implementation manner of the first aspect.

[0018] Optionally, the data import apparatus in the third aspect further includes the memory.

[0019] Optionally, the data import apparatus in the third aspect further includes a communication interface, and the processor is coupled with the communication interface.

[0020] In a fourth aspect, the present application provides a chip system, which comprises at least one processor for supporting the functions involved in the first aspect and any possible implementation manner of the first aspect, such as receiving or processing the data involved in the above method, etc.

[0021] In a possible design, the chip system further comprises a memory for storing program instructions and data, and the memory is located in or out of the processor.

[0022] The chip system can be composed of a chip, or can comprise a chip and other discrete devices.

[0023] In a fifth aspect, the present application provides a computer readable storage medium, which stores a computer program (also referred to as code or instructions), and when the computer program is run by a processor, the method in the first aspect and any possible implementation manner of the first aspect is executed.

[0024] In a sixth aspect, the present application provides a computer program product, which comprises a computer program (also referred to as code or instructions), and when the computer program is run, the method in the first aspect and any possible implementation manner of the first aspect is executed.

[0025] It should be understood that the second aspect to the sixth aspect of the present application correspond to the technical solution of the first aspect of the present application, and the beneficial effects obtained by each aspect and the corresponding possible implementation manner are similar, which will not be repeated here.

[0026] It should also be understood that the data import method and device provided by the present application can be applied to the field of big data, and can also be applied to other fields. The present application does not limit this. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a scene schematic diagram suitable for the method provided by the embodiments of the present application;

[0028] Figure 2 is a schematic flowchart of the data import method provided by the embodiments of the present application;

[0029] Figure 3 is a flowchart before the data import provided by the embodiments of the present application;

[0030] Figure 4 is a flowchart of importing target data into a target address provided by the embodiments of the present application;

[0031] Figure 5 is a flowchart of continuing to import target data provided by the embodiments of the present application;

[0032] Figure 6 is a flowchart of a failure data redirection provided by an embodiment of the present application;

[0033] Figure 7 is a schematic block diagram of a data import apparatus provided by an embodiment of the present application;

[0034] Figure 8 is still another schematic block diagram of a data import apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, and the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Thus, the exemplary embodiments are not intended to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject technology. It will be appreciated that those aspects of the exemplary embodiments that can be implemented mechanically can be implemented in any combination of devices or structures.

[0036] To better understand the data import method provided by the embodiments of the present application, first, the terms involved in the present application are briefly described.

[0037] 1. Primary key: In a record in a database, there are several attributes. If one attribute group can uniquely identify a record, the attribute group can be referred to as a primary key. For example, a student table is stored in a database, and the student table includes a student number, a name, a gender, and a class. The student number of each student is unique, and thus the student number can be used as a primary key. For another example, a user table includes a user identifier, a password, and a login level. The user identifier is unique, and thus the user identifier can be used as a primary key.

[0038] To facilitate understanding of the data import method provided by the embodiments of the present application, the application scenarios of the data import method provided by the embodiments of the present application will be described below. It can be understood that the application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application.

[0039] 2. Source address and target address: The source address refers to the address where the source data is stored before data import, and the target address refers to the target address of data import. Here, import can refer to migration or replication.

[0040] For example, data on server 1 is migrated to server 2, server 1 can be understood as a source address, and server 2 can be understood as a target address. For another example, data in table 1 on a server is copied to another new table, such as table 2, and table 1 and table 2 are located in different storage locations of the server, the location of table 1 can be understood as a source address, and the location of table 2 can be understood as a target address.

[0041] 3. Source data and target data: In order to facilitate the distinction, the data stored in the source address is referred to as source data in this paper, and the data that the user wants to import into the target address is referred to as target data. It can be understood that the target data is also data read from the source address, but needs to be imported into the target address, rather than stored in the source address, so the target data is different from the source data. The source data in this paper is intended to express the data stored in the source address, and the data read from the source address and needed to be imported into the target address is referred to as target data.

[0042] Figure 1 is a scenario diagram applicable to the method provided by the embodiments of the present application. As shown in Figure 1 , the server 110 can communicate with the server 120 and the server 130. The server 110 can call the import program in response to the operation of the user to import the data stored in the server 120 into the server 130, that is, the user can specify the source address of the target data as the server 120 and the target address as the server 130, wherein the server 110 pre-stores the import program.

[0043] It should be understood that Figure 1 The servers 110 to 130 are exemplarily shown in the figure, but should not constitute any limitation on the embodiments of the present application. For example, more or fewer servers can be deployed under the server 110. For another example, the server 110 can be a server, or a server cluster composed of multiple servers, and the present application does not limit this.

[0044] In addition, Figure 1 The source address and the target address located in different servers in the figure are only one possible example, and in another embodiment, the source address and the target address can also be located in the same server, and the embodiments of the present application do not limit this.

[0045] In some special scenarios, such as initializing a new table, data migration, etc., it is a crucial step to import a large amount of data into the target address. The process of data import is roughly divided into the following three steps: reading data from the source address, processing the data, and writing the data into the target address. For example, Figure 1As shown in FIG. 1, the server 110 reads the target data from the server 120 and processes the target data so as to write the target data into the server 130. It can be understood that during the data import process, the data import may fail due to network jitter or the like. In this case, only the data that fails to be imported can be re-imported into the target address, but this may result in inconsistency between the order of data in the target address and the order of data in the source data. For example, before the data is imported into the target address, the data D1 is located before the data D2, the data D1 fails to be imported, and after the data D2 is successfully imported into the target address, the data D1 is re-imported, which results in that the data D1 is located after the data D2, and further results in inconsistency between the order of data in the target address and the order of data in the source data.

[0046] To solve the above problem, the present application provides a data import method. During the import process of the target data, if there is a case that part of the data fails to be imported, the data import device can not write the data having the same primary key as the data that fails to be imported and located after the data that fails to be imported into the target address, but write the data into a preset address different from the target address, and then import the data in the preset address into the target address after the import of the target data is completed. This avoids the problem of inconsistency between the order of data in the target address and the order of data in the source address.

[0047] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The embodiments shown below describe the method provided by the embodiments of the present application with the data import device as the execution subject. The data import device may be, for example, the server 110 shown in FIG. 1, or a component (such as a chip, a chip system or other functional modules capable of calling and executing programs) in the server 110. The embodiments of the present application do not limit this. Figure 1

[0048] Figure 2 FIG. 2 is a schematic flowchart of the data import method provided by the embodiments of the present application. Figure 2 The data import method 200 shown in FIG. 2 can include steps 210 to 230. Each step in the method 200 will be described in detail below.

[0049] In step 210, the data import device imports the target data into the target address.

[0050] The target data includes data corresponding to at least one primary key, and each primary key corresponds to one or more pieces of data.

[0051] ​In a possible implementation, the data import apparatus reads target data from the source address, processes the target data, such as splitting the data, selecting partial data fields, processing special characters, and so on, and writes the processed target data to the target address. The processing of special characters includes replacing special characters, such as replacing spaces with line breaks, and so on.

[0052] As can be seen from the above, after reading the target data from the source address, the target data needs to be processed. Exemplarily, the data processing apparatus can process the target data in the following ways:

[0053] The data processing apparatus processes the target data based on the import configuration file to obtain the processed target data. The import configuration file can be used to define the rules for processing the target data, or in other words, define the processing rules for the target data.

[0054] By way of example but not limitation, the import configuration file can include, but is not limited to, configuration information such as encoding type, table name, separator between fields, field name and field type, special character processing format, additional fields, and number of threads allowed to run in parallel, and so on.

[0055] Exemplarily, the data processing apparatus can process the target data based on the configuration information in the import configuration file. For example, the data processing apparatus processes the target data based on the separator included in the import configuration file. For another example, the data processing apparatus processes the target data based on the encoding type defined in the import configuration file. For brevity, no further enumeration is given here. Optionally, the import configuration file is generated based on a predefined import configuration syntax. The data processing apparatus can store the predefined import configuration syntax, so as to generate the import configuration file according to the configuration information specified by the user, such as the encoding type, table name, separator between fields, and so on, in accordance with the import configuration syntax in response to the operation of the user, and then used for processing the target data.

[0056] It should be understood that the predefined import configuration syntax can be applicable to different data import scenarios, in other words, different target data import can generate the import configuration file based on the import configuration syntax. Several examples of the import configuration syntax will be given below.

[0057] In an example, the syntax used for defining the processing of special characters can be "replace A to B", where "A" represents the string to be replaced, and "B" represents the string used for replacement. In another example, the syntax used for defining the field that can be added can be "_D char", where the value of the field "_D char" is 1, indicating that the data is deleted. In another example, the syntax used for defining the field that can be added can also be "ignore errors", if the field is not added, the data import fails, and the data that fails to be imported is written to a preset address; if the field is added, the data import fails, and the data that fails to be imported is written to a preset address. In another example, the syntax used for defining the field and the field type can be "[field1] [field1 type]". The syntax used for defining the table name can be "table[table name]". For brevity, the above examples are not listed one by one.

[0058] It should be understood that the above several import configuration syntaxes are only examples and should not constitute any limitation on the embodiments of the present application. The data import apparatus can further define more types of import configuration syntaxes to facilitate the flexible selection of encoding types, special character processing methods, and the like. In addition, the import configuration syntax can be applicable to the import of different data, so that repeated design of the syntax is avoided in the actual development and design process, thereby improving the development efficiency and reducing the maintenance cost.

[0059] Optionally, before reading the target data from the source address, the method further includes: in response to a user operation, obtaining the address of the import configuration file, the source address, and the target address.

[0060] It can be seen that the process of importing the target data into the target address involves reading the target data from the source address, processing the target data based on the import configuration file, and writing the processed target data to the target address, and therefore, before the data import, the data import apparatus needs to obtain the source address, the address of the import configuration file, and the target address.

[0061] A possible design is that, in response to a user operation, for example, in response to a selection, input, or click operation of the user on the above three addresses, the data import apparatus obtains the above three addresses to complete the import operation of the target data.

[0062] The detailed process of the data import apparatus performing the operation of importing the target data into the target address will be described in detail below.

[0063] A possible design is that the data processing apparatus creates a reading thread, at least one processing thread, and at least one writing thread for the target data, the processing thread is used to process the target data based on the import configuration file, and the writing thread is used to write the processed target data to the target address.

[0064] Figure 3 is a flowchart of importing target data into a target address provided by an embodiment of the present application. The following will be described in detail Figure 3 with reference to the accompanying drawings. Figure 3 The flowchart shown is a more detailed description of step 210.

[0065] As shown in FIG. 2, step 210 can include steps 2101-2106. Figure 3

[0066] In step 2101, read threads, processing threads and write threads are initialized.

[0067] After the data processing apparatus creates read threads, processing threads and write threads for the target data, the threads are initialized.

[0068] In step 2102, the read threads read the target data, perform hash calculation on the primary keys corresponding to each piece of data, and distribute the target data to corresponding processing threads.

[0069] The read threads distribute the target data to corresponding threads. One possible design is that data of the same primary key corresponds to the same processing thread, and data of the same processing thread corresponds to the same write thread.

[0070] For example, after the read threads read the target data from the source address, hash calculation is performed on the primary keys corresponding to each piece of data in the target data. The hash values obtained are taken modulo the number of parallel threads, and data with the same remainder are distributed to the same processing thread. The specific steps of hash calculation can refer to known technologies and will not be described here. It can be understood that if data of the same primary key is distributed to different processing threads, the processing speeds of different processing threads can be different, which can disrupt the order of data of the same primary key. For example, data D1 and data D2 corresponding to the same primary key, data D2 is located after data D1, data D1 is distributed to processing thread 1, and data D2 is distributed to processing thread 2, which can result in the order of the final written data being data D1 located after data D2. By distributing data of the same primary key to the same processing thread, the problem of inconsistency between the final imported data and the target data caused by distributing data of the same primary key to different processing threads can be effectively avoided, thereby ensuring data consistency.

[0071] In step 2103, the processing threads process the target data based on the import configuration file, and write the processed target data to the write thread cache queue.

[0072] ​The processing thread processes the target data, such as segmenting the data, selecting part of the data field, processing special characters, and the like, to obtain processed target data, and writes the processed target data into the cache queue of the writing thread, so as to facilitate the writing thread to write the processed target data into the target address.

[0073] In step 2104, the processing thread sends the position offset information of the last data to the writing thread when the number of processed data reaches the preset threshold of the writing thread.

[0074] For example, each of the at least one processing thread records the number of processed data, such as setting a counter, and each time a piece of data is processed, the count of the counter is added by one. When the count reaches the preset threshold of the writing thread, the position offset information of the last data in the processed data (such as the order of the last data in the target data) is sent to the writing thread.

[0075] In step 2105, the writing thread receives the position offset information, writes the data in the cache queue into the target address, and clears the cache queue.

[0076] In step 2106, the writing thread records the position offset information and the primary key in the position offset information file.

[0077] For example, the writing thread can write the position offset information of the last data and the primary key corresponding to the data into the position offset information file, so as to indicate the starting position when the reading thread reads data next time. For example, the reading thread can start reading data from the above position when reading data next time.

[0078] In step 220, the data import device writes the data after the first data in the data corresponding to the first primary key into the preset address in the case that the first data import fails in the target data.

[0079] During the execution of step 210, that is, during the process of importing the target data into the target address by the data device, there may be a case that part of the data (such as the first data) fails to be imported. In the case that the first data fails to be imported, the data import device can write the first data into the preset address, and write the data after the first data in the data corresponding to the first primary key into the preset address.

[0080] How to write the data after the first data in the data corresponding to the first primary key to the preset address, one possible design is, after completing the processing of the second data in the target data, determine whether the second primary key corresponding to the second data and the first primary key are the same. One possible case is that the second primary key corresponding to the second data is not the same as the first primary key, that is, the first data and the second data are data corresponding to different primary keys. Another possible case is that the second primary key corresponding to the second data is the same as the first primary key, that is, the first data and the second data are data corresponding to different primary keys. In the case that the second primary key corresponding to the second data is the same as the first primary key, write the processed second data to the preset address. Wherein, the second data is processed by importing the configuration file to obtain the processed second data. In the case that the second primary key corresponding to the second data is the same as the first primary key, that is, the first data and the second data correspond to the same primary key, the second data is any one of the data corresponding to the first primary key except the first data. That is, the data import device determines whether the primary key of the data is the same as the first primary key after processing each data to determine whether to write the data to the preset address.

[0081] In step 230, the data import device imports the data in the preset address to the target address after the target data import is completed.

[0082] When the data import device imports data, it may import batch data. The data import device reads all target data from the source address until the process of importing all target data is completed, which can be understood as the completion of target data import. However, as known from the foregoing, the data import device does not necessarily import all data into the target address. For example, the first data fails to be imported into the target address, and the first data and the data after the first data in the first primary key can be temporarily stored in the preset address. Therefore, the completion of target data import does not mean the completion of importing all target data into the target address, but indicates the completion of a data import process.

[0083] From another perspective, the completion of target data import can mean that, in the data corresponding to the first primary key, the process of importing other data into the target address is completed except the first data and the subsequent data. In other words, the imported target data does not include the data imported into the preset address.

[0084] Exemplarily, after the completion of target data import, the data import device can take the preset address as the source address and import the data in the preset address to the target address. The specific import process is the same as that described in step 210, which will not be described herein.

[0085] Figure 4 An example of the data import device reimporting the data that fails to be imported is given, and the following will be described in combination withFigure 4 The detailed description data import device performs steps in the case of first data import failure in target data.

[0086] Figure 4 is the flowchart of the failure data re-direct provided by the embodiment of the application.

[0087] As Figure 4 shown, in step 410, the data import device determines whether the first data is written successfully. In the case of first data writing failure, step 420 is performed to write the first data into the preset address and record the first primary key, which is the primary key corresponding to the first data; in the case of first data writing success, the data import device does not need to perform step 420, and directly performs step 430.

[0088] In step 430, the data import device judges whether the second primary key of the second data and the first primary key are the same. Wherein, the second data is any one data in the target data except the first data. If the same, the data import device performs step 440 to write the second data into the preset address, in other words, the second data and the first data are the data corresponding to the same primary key, and are located after the first data, therefore, the second data is written into the preset address in order to import the target address. If not the same, the data import device does not need to perform step 440, and directly performs step 450.

[0089] In step 450, the data import device continues to import the target data into the target address until the import is completed.

[0090] In step 460, the data import device imports the data in the preset address into the target address. Exemplarily, the data import device imports the data in the preset address into the target address after the target data import is completed.

[0091] Optionally, before the data import device imports the target data into the target address, the method further includes: generating an import configuration file; obtaining the import configuration file address, the source address, and the target address; and parsing the import configuration file.

[0092] The following will be combined Figure 5 with the detailed description of the steps performed by the data import device before importing data into the target address. Figure 5 is the flowchart before data import provided by the embodiment of the application. Figure 5 The flowchart shown includes steps 510 to 530. Figure 5 The flowchart shown can be realized by the data import device performing a pre-defined computer program.

[0093] As Figure 5 shown, in step 510, the data import device generates an import configuration file.

[0094] For example, in response to a user's operation of specifying configuration information such as an encoding type, a special character processing mode, and the like, the data import apparatus generates an import configuration file based on a predefined import configuration syntax, for use in processing target data by the data import apparatus.

[0095] In step 520, the data import apparatus acquires an address of the import configuration file, a source address, and a target address.

[0096] In response to a user's operation, the data import apparatus acquires the above three addresses. The data import apparatus can further determine a target data type based on the above addresses, where the target data type refers to whether the source address and the target address of the target data correspond to the same device, in other words, whether the source address and the target address are different addresses of the same device or correspond to different devices.

[0097] In step 530, the data import apparatus parses the import configuration file.

[0098] The data import apparatus parses the import configuration file to acquire configuration information such as an encoding type, a table name, a separator between fields, field names and field types, a special character processing format, addable fields, and a number of threads allowed to be run in parallel.

[0099] After the data import apparatus acquires the above import configuration information, the source address, and the target address, the data import apparatus can perform step 210 to import the target data into the target address. For example, the data import apparatus can call a pre-stored import program to import the target data into the target address. It should be understood that the data import apparatus performs the above import program to complete the operation of importing the target data into the target address. In other words, the data import apparatus performs the above import program, including performing the following steps: reading the target data from the source address; processing the target data to obtain processed target data; and writing the processed target data into the target address.

[0100] Based on the above technical solution, in the process of importing the target data into the target address by the data import apparatus, in the case that the first data in the target data fails to be imported, data having the same primary key as the first data and located after the first data can be written into a preset address, and after the target data is imported, the data in the preset address can be imported into the target address. In this way, by re-importing the first data and the data after the first data, the problem that the order of the data after the first data is inconsistent with the order of the target data is avoided, which is conducive to ensuring the consistency of the data. In addition, in the case that the first data fails to be imported, there is no need to re-import the target data in its entirety, which is conducive to reducing the overhead and improving the import efficiency.

[0101] It's understandable that the import process might be interrupted due to network anomalies, service disruptions, or other reasons, such as the processing thread being interrupted while processing the target data. Alternatively, there might be an increase in the target data stored at the source address. Therefore, once the import process resumes, a continuation method can be used to continue the import operation without needing to re-import the entire target data, thus improving import efficiency and reducing overhead.

[0102] One possible implementation is that, when the amount of data processed by each of the at least one processing thread reaches a preset threshold, the position offset information of the processed data is recorded. This position offset information is used to indicate the order of the last piece of data processed by each processing thread in the target data. If the processing of the target data is interrupted, the minimum value of the position offset information of the most recently recorded processed data by each of the at least one processing thread is used as the starting position to continue importing the target data.

[0103] In this way, if the import process is interrupted, the target data can be resumed, that is, the target data can be imported from the position before the interruption, without having to re-import the entire data, which helps to reduce overhead and improve import efficiency.

[0104] Figure 6 This is a schematic diagram illustrating the process of resuming the import of target data according to an embodiment of this application. The following will describe in detail how the data import device resumes the import of target data in the event of an interruption in the import process.

[0105] like Figure 6 As shown, in step 610, the data import device acquires the position offset information file.

[0106] Depend on Figure 3 As described above, the data import device can record the position offset information of the last data in the processed data in a position offset information file. The data import device obtains this position offset information file and then obtains the sorting of the last data in the target data in the data processed by each processing thread.

[0107] In step 620, the data import device determines the latest position offset information for each processing thread. For example, after obtaining the position offset information file, the data import device determines the latest position offset information for each processing thread based on that file. It can be understood that the position offset information of the last data item processed by each processing thread can be recorded in this position offset information file.

[0108] In step 630, the data import device determines the smallest offset position indicated by the latest position offset information in each processing thread, so that the read thread can start reading data from that position.

[0109] For example, in the event of an interruption in the processing of the target data, upon resumption of normal operation, the data import device uses the minimum value of the position offset information of the processed data most recently recorded by each of the at least one processing thread as the starting position to facilitate the continued import of the target data. For instance, if the position offset information most recently recorded by processing thread 1 indicates a position of 10001, and the position offset information most recently recorded by processing thread 2 indicates a position of 10004, then the data import device uses 10001 as the starting position.

[0110] In step 640, the data import device continues to import the target data.

[0111] Once the data import device determines the minimum value of the position offset information of the most recently recorded processed data for each thread, it can use that position as the starting position to continue importing the target data into the target address.

[0112] Based on the above technical solution, if the data import device experiences an interruption during the process of importing target data to the target address, upon resumption of normal operation, the data import device does not need to re-import all the target data. It only needs to use the minimum value of the position offset information of the most recently recorded processed data in each thread as the starting position and continue importing the target data to the target address. This saves overhead and improves import efficiency.

[0113] Figure 7 This is a schematic block diagram of the data import device provided in the embodiments of this application.

[0114] like Figure 7 As shown, the data import device 700 may include: a first processing unit 710, a second processing unit 720, and an acquisition unit 730. The data import device 700 can be used to implement... Figures 2 to 6 The method described in any one of the embodiments shown.

[0115] For example, when the device 700 is used to implement Figure 2The method 200, the first processing unit 710 is used for importing target data into a target address, the target data including data corresponding to at least one primary key, each primary key corresponding to one or more data; the second processing unit 720 is used for writing data corresponding to a first primary key and located after the first data in the data corresponding to the first primary key into a preset address in the case of failure of importing the first data in the target data, the preset address being different from the target address, the first primary key being a primary key corresponding to the first data; the first processing unit 710 is further used for importing the data in the preset address into the target address after the target data importing is completed.

[0116] Optionally, the first processing unit 710 is specifically used for reading the target data from a source address; processing the target data based on an import configuration file to obtain processed target data; wherein the import configuration file is used for defining a rule of processing the target data, and the import configuration file is generated based on a predefined import configuration syntax; and writing the processed target data into the target address.

[0117] Optionally, the import configuration syntax includes syntax for defining one or more parameters: encoding type, table name, separator between fields, field name and field type, special character processing format, additional fields, and number of threads allowed in parallel.

[0118] Optionally, the apparatus 700 further includes an acquisition unit 730, the acquisition unit 730 is used for acquiring an address of the import configuration file, the source address and the target address in response to a user operation.

[0119] Optionally, the second processing unit 720 is further used for determining whether a second primary key corresponding to the second data is the same as the first primary key after completing processing of the second data in the target data based on the import configuration file; and in the case that the second primary key corresponding to the second data is the same as the first primary key, the second processing unit 720 is specifically used for writing the processed second data into the preset address.

[0120] Optionally, the first processing unit 710 is further used for creating one reading thread, at least one processing thread and at least one writing thread for the target data, the processing thread being used for processing the target data based on the import configuration file, and the writing thread being used for writing the processed target data into the target address.

[0121] Optionally, data corresponding to the same primary key corresponds to the same processing thread, and data corresponding to the same processing thread corresponds to the same writing thread.

[0122] Optionally, the first processing unit 710 is further configured to record the position offset information of the processed data when the amount of data processed by each processing thread in the at least one processing thread reaches a preset threshold. The position offset information of the processed data is used to indicate the sorting of the last piece of data in the target data in the data processed by each processing thread this time.

[0123] Optionally, the first processing unit 710 is further configured to, in the event of an interruption in the processing of the target data, take the minimum value of the position offset information of the processed data most recently recorded by each of the at least one processing thread as the starting position and continue importing the target data.

[0124] It should be understood that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into a single processor, exist as separate physical units, or be integrated into one unit by two or more units. The integrated units described above can be implemented in hardware or as software functional modules.

[0125] Figure 8 This is another schematic block diagram of the data import device provided in the embodiments of this application.

[0126] The data import device 800 can be used to achieve... Figures 2 to 6 The method described in any of the embodiments shown. The data import device 800 can be a chip system. In the embodiments of this application, the chip system can be composed of chips, or it can include chips and other discrete devices.

[0127] like Figure 8 As shown, the data import device 800 may include at least one processor 810 for implementing Figures 2 to 6 The method described in any one of the embodiments shown.

[0128] For example, when the data import device 800 is used to implement the embodiments of this application... Figure 2 In method 200, processor 810 can be used to import target data into a target address. The target data includes data corresponding to at least one primary key, with each primary key corresponding to one or more data entries. If the import of the first data in the target data fails, the data following the first data in the first data and the data corresponding to the first primary key are written to a preset address. The preset address is different from the target address, and the first primary key is the primary key corresponding to the first data. After the target data import is completed, the data in the preset address is re-imported into the target address. See the detailed description in the method example for further details; it will not be repeated here.

[0129] The data import apparatus 800 can further include at least one memory 820, which can be used to store program instructions and / or data. The memory 820 is coupled to the processor 810. The coupling in the embodiments of the present application is indirect coupling or communication connection between apparatuses, units or modules, which can be electrical, mechanical or other forms, for information interaction between apparatuses, units or modules. The processor 810 can operate in cooperation with the memory 820. The processor 810 can execute program instructions stored in the memory 820. At least one of the at least one memory can be included in the processor.

[0130] The data import apparatus 800 can further include a communication interface 830 for communicating with other devices through a transmission medium, so that the data import apparatus 800 can communicate with other devices. The communication interface 830 can be, for example, a transceiver, an interface, a bus, a circuit or an apparatus capable of transceiving. The processor 810 can transceive data and / or information by using the communication interface 830, and be used to implement Figures 2 to 6 the method in any one of the embodiments shown.

[0131] The specific connection medium between the processor 810, the memory 820 and the communication interface 830 in the embodiments of the present application is not limited. In the embodiments of the present application, the processor 810, the memory 820 and the communication interface 830 are connected through a bus 840. Figure 8 The bus 840 is represented by a thick line in the embodiments of the present application, and the connection mode between other components is only schematically illustrated and is not limited. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 8 only one thick line is used in the embodiments of the present application, but it does not mean that there is only one bus or only one type of bus. Figure 8

[0132] The present application further provides a chip system, which includes at least one processor for implementing the method in any one of the embodiments shown. Figures 2 to 6

[0133] In a possible design, the chip system further includes a memory for saving program instructions and data, and the memory is located in the processor or outside the processor.

[0134] The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0135] The present application further provides a computer program product, which includes a computer program (also referred to as code or instructions), which, when executed, causes a computer to perform the method as described above.The present application further provides a computer program product, which includes a computer program (also referred to as code or instructions), which, when executed, causes a computer to perform the method as described above.​Figures 2 to 6 the method described in any one of the embodiments.

[0136] The present application also provides a computer readable storage medium, which stores a computer program (also referred to as code or instructions). When the computer program is executed, the computer is caused to perform the method as Figures 2 to 6 the method described in any one of the embodiments.

[0137] It should be noted that the data import method and device provided in the embodiments of the present application can be applied to the field of big data, and can also be applied to any field other than the field of big data, which is not limited in the present application.

[0138] It should be understood that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method embodiments can be completed by integrated logic circuits or instruction in the form of software in the processor. The processor mentioned above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware code processor execution, or executed by hardware and software module combination in the code processor. The software module can be located in random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method.

[0139] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically EPROM (EEPROM) or flash memory. The volatile memory can be random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRAM). It should be noted that the memory of the system and method described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0140] The terms "unit", "module" and the like used in the specification can be used to represent a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution.

[0141] Those of skill in the art would understand that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or combinations of both. The disclosure is not limited to implementations set forth herein, but the conception could be carried out in various other ways. For example, the described embodiments are provided for illustrating some aspects of the present disclosure, and various modifications are possible in light of what has been described and drawings. For example, the division into modules is merely a logical function division, and actual implementation can be in another way, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be omitted or not implemented. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0142] The modules described as separate components can or can not be physically separate, and the components shown as modules can or can not be physical modules, that is, they can be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0143] In addition, the functional modules in each embodiment of the present application can be integrated into a processing module, or each module can exist physically, or two or more units can be integrated into a module.

[0144] In the above embodiments, the functions of the various functional modules can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, the software can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the whole or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, digital video disc (DVD)), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0145] The functions can be stored in a computer readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the embodiments of the present application. The storage medium mentioned above includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and various media that can store program codes.

[0146] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data import method, characterized by, The method comprises: Importing target data into a target address, the target data comprising data corresponding to at least one primary key, each primary key corresponding to one or more pieces of data; In the case that the importing of first data in the target data fails, writing data corresponding to the first primary key and located after the first data in the target data into a preset address, the preset address being different from the target address, the first primary key being a primary key corresponding to the first data; After the importing of the target data ends, importing the data in the preset address into the target address; Creating a reading thread, at least one processing thread and at least one writing thread for the target data, the reading thread reading the target data, calculating a hash value of a primary key corresponding to each piece of data, and distributing the target data to a corresponding processing thread, the processing thread being configured to process the target data based on an importing configuration file, the writing thread being configured to write the processed target data into the target address; wherein the importing configuration file is configured to define a rule for processing the target data, and the importing configuration file is generated based on a predefined importing configuration syntax; Wherein, data corresponding to a same primary key correspond to a same processing thread, and data corresponding to a same processing thread correspond to a same writing thread; In the case that the number of data processed by each processing thread in the at least one processing thread reaches a preset threshold, recording the position offset information of the processed data and the primary key in a position offset information file, the position offset information of the processed data being configured to indicate the last piece of data in the data processed by the each processing thread this time in the target data; In the case that the processing of the target data is interrupted, obtaining the position offset information file, determining the latest position offset information of each processing thread based on the position offset information file, and taking the smallest offset position indicated by the latest position offset information of each processing thread as a starting position, so that the reading thread continues to import the target data from the starting position.

2. The method of claim 1, wherein, The importing of the target data into the target address comprises: Reading the target data from a source address; Processing the target data based on an importing configuration file to obtain processed target data; Writing the processed target data into the target address.

3. The method of claim 2, wherein, The importing configuration syntax comprises syntax for defining one or more parameters: encoding type, table name, separator between fields, field name and field type, special character processing format, additional fields, and number of threads allowed to be parallel.

4. The method of claim 2, wherein, Before reading the target data from the source address, the method further comprises: In response to a user operation, obtaining an address of the importing configuration file, the source address and the target address.

5. The method of claim 2, wherein, The method further comprises: After processing a second data in the target data based on the importing configuration file, determining whether the second primary key corresponding to the second data and the first primary key are the same; and In the case that the importing of the first data fails, writing data corresponding to the first primary key and located after the first data in the target data into a preset address, comprises: In a case where the second primary key corresponding to the second data is the first primary key, the processed second data is written to the preset address.

6. A data import apparatus characterized by comprising: An apparatus for performing the data import method of any one of claims 1-5, the apparatus comprising: a first processing unit configured to import target data into a target address, the target data including data corresponding to at least one primary key, each primary key corresponding to one or more pieces of data; a second processing unit configured to, in a case where import of first data in the target data fails, write data corresponding to the first primary key and located after the first data in the data corresponding to the first primary key to a preset address, the preset address being different from the target address, the first primary key being a primary key corresponding to the first data; the first processing unit is further configured to, after the import of the target data ends, import the data in the preset address into the target address.

7. A data import apparatus characterized by comprising: A computer program product comprising a computer readable medium storing instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer program product comprising a computer readable medium storing instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 5.

9. A computer program product, characterised in that, A computer program product comprising a computer readable medium storing instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 5.

Images