A method, device and electronic device for batch importing data

Abstract

The present application relates to a method, device and electronic device for batch importing data, wherein the method includes: obtaining data to be imported and a reference database. The data to be imported is data to be imported into a target database, the reference database contains multiple first positions, the first position corresponding to the stored data in the reference database is stored with a specified mark, the stored data is data stored in the target database, and the correspondence between the first position and the stored data is determined in a specified manner; for each data to be imported, the first position corresponding to it in the reference database is determined in a specified manner; if the specified mark is not stored in the first position corresponding to the data to be imported, the data to be imported is imported into the target database. Through the process in this specification, the problem of consuming a lot of time in the process of finding duplicate time when performing data import for batches of data to be imported in the related art can be effectively solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device and electronic device for batch importing data. Background Art

[0002] In the related art, the method of importing batches of data into a target database is usually as follows: obtain a piece of data to be imported, traverse the stored data in the target database to find out whether the target database has stored data identical to the data to be imported. If so, it indicates that the data to be stored is duplicate data relative to the data in the target database, and the import operation is not performed on it. If there is a lot of data stored in the target database, it will take a long time to perform the search in the process of determining whether the data to be imported is duplicate data. In the case where there is a lot of data to be imported, the phenomenon of low data import efficiency caused by the search will be more obvious. Summary of the invention

[0003] The present application provides a method, device and electronic device for batch data importing to solve the problem in the prior art that it takes a long time to import batches of data to be imported.

[0004] In a first aspect, the present application provides a method for batch importing data, the method comprising: obtaining data to be imported and a reference database, wherein the data to be imported is data to be imported into a target database, the reference database comprises multiple first positions, a first position in the reference database corresponding to the stored data has a specified mark stored therein, the stored data is data stored in the target database, and the correspondence between the first position and the stored data is determined in a specified manner; for each data to be imported, determining the first position corresponding thereto in the reference database in a specified manner; and if the specified mark is not stored in the first position corresponding to the data to be imported, importing the data to be imported into the target database.

[0005] In an optional embodiment of the present specification, after obtaining the reference database, the method further includes: for each data to be imported, if the data to be imported has a specified mark stored in the first position corresponding to the reference database, searching for each stored data in the target database; if no stored data identical to the data to be imported is found, importing the data to be imported into the target database.

[0006] In an optional embodiment of the present specification, the reference database also includes multiple second positions, each first position corresponds to at least one second position, and the stored data stored in the second position is the stored data corresponding to the first position corresponding to the second position in the target data, wherein, after obtaining the reference database, the method also includes: for each data to be imported, if a specified mark is stored in the first position corresponding to the data to be imported in the reference database, the first position corresponding to the data to be imported in the reference database is determined as the specified position; when the stored data in the second position corresponding to the specified position is different from the data to be imported, the data to be imported is imported into the target database.

[0007] In an optional embodiment of the present specification, importing the data to be imported into the target database includes: saving a designated mark to a first position corresponding to the data to be imported in the reference database, and importing the data to be imported into the target database.

[0008] In an optional embodiment of the present specification, the target database is established in a disk of a storage device, and the storage device also includes a memory, wherein obtaining a reference database includes: determining a specified area in the memory; storing a database template in the specified area; saving a specified mark in the first position of the database template corresponding to the stored data, to obtain a reference database.

[0009] In an optional embodiment of the present specification, a reference database is obtained, including: performing a hash calculation on each stored data, determining a first target superscript according to a first result obtained by the hash calculation, using the first position corresponding to the first target superscript in the data block template as the first position corresponding to the stored data, and saving a designated mark in the first position corresponding to the stored data to obtain a reference database; for each data to be imported, determining the first position corresponding to it in the reference database in a designated manner, including: performing a hash calculation on each data to be imported, determining a second target superscript according to a second result obtained by the hash calculation, and using the first position corresponding to the second target superscript in the data block template as the first position corresponding to it in the reference database.

[0010] In an optional embodiment of the present specification, before obtaining the reference database, the method also includes: determining the primary key used in this batch import of data; obtaining the reference database, including: generating a reference database based on the field of the stored data corresponding to the primary key; using a specified method to determine the first position corresponding to it in the reference database, including: using a specified method to process the field of the imported data corresponding to the primary key to determine the first position corresponding to it in the reference database.

[0011] In an optional embodiment of the present specification, a hash calculation is performed for each stored data, including: performing a hash calculation for each field of the stored data corresponding to the primary key; a hash calculation is performed for each data to be imported, including: performing a hash calculation for each field of the primary key corresponding to the data to be imported.

[0012] In a second aspect, the present application provides a device for batch importing data, the device comprising: a module for acquiring data to be imported, configured to acquire data to be imported, the data to be imported being data to be imported into a target database; a module for acquiring data to be imported, configured to acquire data to be imported, wherein the data to be imported is data to be imported into a target database; a module for acquiring reference database, configured to acquire a reference database, wherein the reference database comprises a plurality of first positions, a first position in the reference database corresponding to the stored data being stored with a specified mark, the stored data being data stored in the target database, and the correspondence between the first position and the stored data being determined in a specified manner; a first position determination module, configured to determine, for each data to be imported, the first position corresponding thereto in the reference database in a specified manner; a first import module, configured to import the data to be imported into the target database if the specified mark is not stored in the first position corresponding to the data to be imported.

[0013] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0014] Memory, used to store computer programs;

[0015] The processor is used to implement the steps of any one of the data processing methods in the first aspect when executing the program stored in the memory.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the data processing methods in the first aspect.

[0017] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0018] The method provided by the embodiment of the present application obtains a reference database before the data to be imported is imported. The first position in the reference database can indicate whether the first position corresponds to stored data in the target database. A specified mark is saved in the first position corresponding to the stored data in the target database, and a specified mark is not saved in the first position that does not correspond to the stored data in the target database. Then, it can be determined whether the data to be imported is duplicate data by whether the specified mark is saved in the first position corresponding to the data to be imported in the reference database. The method in this specification, by establishing a corresponding relationship between the stored data in the target database and the first position of the reference database, and establishing a corresponding relationship between the data to be imported and the first position of the reference database, determines whether the data to be imported is duplicate data according to the corresponding relationship judgment and whether the specified mark is saved in the first position, and performs import for non-duplicate data to be imported, without querying the stored data in the target database in a traversal manner in the judgment process of whether the data to be imported is duplicate data, which can effectively solve the problem of more time spent in the process of finding duplicate time when executing data import for batches of data to be imported in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A schematic diagram of a scenario involved in batch data import provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a batch data import process provided in an embodiment of the present application;

[0023] Figure 3a A schematic diagram of a process for batch importing data provided in an embodiment of the present application, which is a process for establishing or updating a reference database based on stored data in a target database;

[0024] Figure 3b A schematic diagram of a process for determining a corresponding relationship between data to be imported and a reference database in a data batch import process provided in an embodiment of the present application;

[0025] Figure 4aAnother data batch import process provided in an embodiment of the present application is a schematic diagram of a process for establishing or updating a reference database based on stored data in a target database;

[0026] Figure 4b A schematic diagram of another data batch import process provided in an embodiment of the present application, which is a process for determining the corresponding relationship between the data to be imported and the reference database;

[0027] Figure 5 To correspond to Figure 2 Schematic diagram of a data batch import device for the method process;

[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] There are many factors that affect the efficiency of batch data import. Among the factors that affect the efficiency of import, the number of data accesses has a more obvious impact on the efficiency of batch data import.

[0031] Take the case where a batch of data to be imported is imported into a target database and the target database is established on a disk as an example. Assuming that there are 10 million storage locations for storing data in the target database, when a certain data to be imported in the batch of data to be imported is imported into the target database, the target database must be queried 10 million times. If the query results of 10 million times all show that the data to be imported is non-duplicate data, the data to be imported can be imported into the target database. It can be seen that for the data to be imported, the duplication check phase before importing the target data consumes a lot of time. In the related art, the method of querying the target database is such as B-tree traversal query, and the binary search of the B-tree is more time-consuming. When using the B-tree for searching, although the query algorithm complexity of a single data is logN, the algorithm complexity when performing import on batch data is N*logN.

[0032] In addition, in the related art, when executing batch data import, the data on the disk needs to be loaded into the memory, resulting in a high IO cost.

[0033] If there is a lot of data to be imported, the time cost of the aforementioned duplicate checking and / or reading from the disk will increase dramatically.

[0034] In view of this, in order to solve the problem in the prior art that the batch data import process is time-consuming, this specification provides a batch data import method.

[0035] The hardware architecture involved in the execution of the method in this specification may include: a data import tool and a storage device. The data import tool is the execution subject of the batch data import method in this specification. The storage device provides a storage medium for the target database. In an optional embodiment of this specification, the hardware architecture may also include a terminal. The terminal can be regarded as the source of the data to be imported.

[0036] The business scenarios involved in the batch data import process in this specification may be data migration scenarios for batch data, etc. For example, the scenarios involved in the batch data import process in this specification are as follows: Figure 1 As shown, in Figure 1 In the example shown, the data import tool is used to import the data to be imported into the target database.

[0037] like Figure 2 As shown, the data batch import method provided in this specification may include the following steps:

[0038] S200: Acquire data to be imported.

[0039] In this specification, the target database is used to store data. The target database may include multiple storage locations for storing data. The storage location of a certain stored data in the target database is exemplified as follows: Figure 3a In the case where at least part of the storage locations of the target database have not yet stored data (such as Figure 3a The storage location 2 and the storage location 4 shown are empty), and the target database can continue to store data. Among them, the data already stored in the target database is the stored data, such as Figure 3a Shown are stored data 1, stored data 3, and stored data m.

[0040] This specification does not limit the type of the target database. Exemplarily, the target database can be a relational database, a hierarchical database, or a network database, etc. In addition, this specification does not limit the storage medium that the target database relies on. Exemplarily, the storage medium used by the target database is a disk.

[0041] The data to be imported in this description refers to the data that needs to be imported into the target database and has not yet been imported into the target database. There can be multiple data to be imported, such as Figure 4bThe data to be imported 1 to the data to be imported p are shown. The process in this specification is applicable to the scenario of batch data import, and the amount of data to be imported can be relatively large.

[0042] S202: Acquire a reference database.

[0043] In this specification, the reference database includes a plurality of first positions, exemplarily as Figure 3a Different first positions correspond to different stored data. In some optional examples, there may be a situation where a first position corresponds to multiple stored data. Figure 3a In the example shown, the stored data 3 and the stored data m both correspond to the first position n.

[0044] In this specification, on the one hand, the first position is used to indicate whether there is stored data corresponding to the first position stored in the target database. When there is stored data in the target database, a specified mark is saved in the first position. On the other hand, the first position is also used to indicate the corresponding relationship between it and the stored data.

[0045] The designated mark can be a character. The specific designated mark can be determined according to actual needs. For example, "1" can be used as the designated mark, such as Figure 3a Then, 1 (specified mark) is saved in the first position corresponding to the stored data in the target database, and the specified mark is not saved in the first position not corresponding to the stored data in the target database, or other marks except the specified mark, such as mark "0", are saved.

[0046] In an optional embodiment of the present specification, the reference database is established when the target database is empty (no data has been imported into the target database). The established reference database is then stored, and in the subsequent process of importing data into the target database, the reference database is updated according to the imported data.

[0047] In another optional embodiment of the present specification, after the data is first imported into the target database, a reference database is established, and the data stored in the first position of the reference database is modified according to the stored data in the target database to obtain the reference database. The established reference database is then stored. In the subsequent process of importing data into the target database, the reference database is updated according to the imported data.

[0048] In another optional embodiment of the present specification, each time the data to be imported is imported, the reference database can be newly obtained based on the stored data in the target database. After the current data import is completed, the reference database is deleted.

[0049] It can be seen that the timing of generating the reference database in this specification can be determined according to actual needs.

[0050] Optionally, the process of obtaining the reference database may be: processing each stored data in the target database in a specified manner to obtain the first position corresponding to each stored data in the database template. Then, saving a specified mark for the first position corresponding to the stored data in the database template to obtain the reference database.

[0051] In order to make the first position in the reference database that does not correspond to the stored data distinguishable from the first position that does not correspond to the stored data. Figure 3a As shown, the database template may be first obtained, and each first position in the database template may be initialized using a non-specified mark (e.g., "0") (i.e., each first position in the database template may be stored with a non-specified mark). For example, the database template may be as follows: Figure 3a Afterwards, the marks stored in at least part of the first position in the initialized database template are adjusted according to the stored data.

[0052] In this specification, the correspondence between the first position and the stored data is determined in a specified manner. Specifically, each stored data can be processed in a specified manner, and based on the obtained processing results, the first position corresponding to the stored data can be screened out from each first position. In other words, the correspondence between the stored data and the first position determined in a specified manner is consistent with the attributes of the stored data itself (for example, the value of the stored data, in the example Figure 3a In the example shown, the value of the stored data 1 is “A1B1 . . . C1”), then for each stored data, a unique first position corresponding thereto can be determined.

[0053] It can be seen that the reference database in this specification at least includes a column in which the first positions are arranged in order.

[0054] It should be noted that this specification does not limit the execution order of step S200 and step S202.

[0055] S204: For each piece of data to be imported, determine the first position corresponding to it in the reference database in a specified manner.

[0056] The process of determining the first position corresponding to the data to be imported in a specified manner is similar to the process of determining the first position corresponding to the stored data in a specified manner. Specifically, each data to be imported can be processed in a specified manner, and according to the obtained processing results, the first position corresponding to the data to be imported can be selected from each first position. For example, Figure 3b That is to say, the correspondence between the data to be imported and the first position determined by the specified method is related to the attributes of the data to be imported (for example, the value of the stored data), and for each data to be imported, a unique first position corresponding to it can be determined. Figure 3b In the example shown, the data 1 to be imported corresponds to the first position 3 in the reference database.

[0057] Since the first position determined for each data to be imported is unique, in the subsequent steps, it can be determined whether the data to be imported is duplicate data (if the target database already has stored data identical to the data to be imported, the data to be imported is duplicate data) based on the situation where the mark is saved in the first position corresponding to the data to be imported (for example, whether a specified mark is saved). It can be seen that the search performed in the process of determining whether the data to be imported is duplicate data only needs to be performed once under ideal conditions. In other words, compared to the method of traversing all stored data in the target database for querying in the related art, the process in this specification can effectively reduce the number of queries executed during the data import process. In particular, when the data to be imported is batch data, the process in this specification consumes less time for the query link of duplicate data involved in the data import process.

[0058] S206: If no designated mark is stored in the first position corresponding to the data to be imported, import the data to be imported into the target database.

[0059] If the data to be imported has a specified tag stored in the corresponding first position in the reference database, such as Figure 3b As shown, the data to be imported 1 does not have a specified mark stored in the corresponding first position 1 in the reference database, indicating that the data to be imported 1 is not duplicate data and can be imported into the target database. The data to be imported 2 has a specified mark stored in the corresponding first position 3 in the reference database, indicating that the data to be imported 2 may be duplicate data.

[0060] It can be seen that through the batch data import process in this specification, the reference database is obtained before the data to be imported is imported (that is, before step S206). The first position in the reference database can indicate whether the first position corresponds to stored data in the target database. Specifically, a specified mark is saved in the first position corresponding to the stored data in the target database, and a specified mark is not saved in the first position that does not correspond to the stored data in the target database. Then, whether the data to be imported is duplicate data can be determined by whether the specified mark is saved in the first position corresponding to the data to be imported in the reference database.

[0061] Through the process in this specification, a correspondence is established between the stored data in the target database and the first position of the reference database, and a correspondence is established between the data to be imported and the first position of the reference database. Based on the correspondence and whether the specified mark is stored in the first position, it is determined whether the data to be imported is duplicate data, and import is performed on non-duplicate data to be imported. There is no need to query the stored data in the target database in a traversal manner during the judgment process of whether the data to be imported is duplicate data. This can effectively solve the problem in the prior art that a lot of time is spent in the process of searching for duplicate times when performing data import for batches of data to be imported.

[0062] As can be seen from the foregoing, the designation method in this specification can establish a correspondence between the first position and the stored data based on the attributes of the stored data; and establish a correspondence between the first position and the data to be imported based on the attributes of the data to be imported.

[0063] To determine the corresponding relationship, in an optional embodiment of the present specification, the process of generating a reference database based on the corresponding relationship between the first position and the stored data may be: the database template includes multiple first positions, and each first position is distinguished by a superscript (exemplarily, each first position is arranged in order in the database template, and the superscript indicates the order of the first position in the database template). A hash calculation is performed for each stored data, and a first target superscript is determined according to the first result obtained by the hash calculation (exemplarily, the first result can be divided by a specified value, and the remainder obtained is used as the first target superscript. The specified data may be positively correlated with the number of storage positions in the target database), and the first position corresponding to the first target superscript in the data block template (i.e., the first position with the same superscript as the first target superscript) is used as the first position corresponding to the stored data, and the specified mark is saved in the first position corresponding to the stored data to obtain a reference database.

[0064] Correspondingly, the process of determining the correspondence between the data to be imported and the first position can be: performing a hash calculation for each data to be imported, determining the second target corner mark based on the second result obtained by the hash calculation, and using the first position in the data block template corresponding to the second target corner mark (that is, the first position where the corner mark is the same as the second target corner mark) as the first position corresponding to it in the reference database.

[0065] It should be noted that this specification does not impose any specific restrictions on the hash function used in hash calculations. For example, MD4, MD5, SHS and other hash functions can be used as hash functions for hash calculations in this specification if conditions permit.

[0066] It should be noted that, in addition to the processing method based on hash calculation, the specified method in this specification can also determine the corresponding relationship between the stored data and the first position according to the attributes of each dimension of the stored data (for example, the generation time of the data, the size of the data, etc.). Exemplarily, for each stored data, the generation time of the stored data can be encoded as a first value (for example, the generation time is June 16, 2021, and the various bits of the generation time can be added together for encoding, and the first value obtained is 18), and the size of the stored data can be encoded as a second value (for example, the data size is 35k, and the various bits of the data size can be added together for encoding, and the second value obtained is 8). The first value and the second value are combined (for example, the result of combining 18 and 8 can be 188) to obtain the first target subscript.

[0067] Correspondingly, the method of determining the correspondence between the data to be imported and the first position according to the attributes of each dimension of the data to be imported may be similar to the aforementioned method of determining the correspondence between the stored data and the first position, which is not elaborated here.

[0068] In addition, the correspondence between the first position and the stored data, and the correspondence between the first position and the data to be imported may be determined by other specified methods, which are not listed here one by one.

[0069] In a further optional embodiment of the present specification, in order to reduce the amount of data processing performed during hash calculation, the primary key used in the current batch import of data can be determined before obtaining the reference database. Figure 3a In the example shown, the fields of each stored data correspond to key 1 to key k, where key 1 is the primary key. The process of performing hash calculation on the stored data may be: performing hash calculation on the fields of each stored data corresponding to the primary key. The process of performing hash calculation on the data to be imported may be: performing hash calculation on the fields of each data to be imported corresponding to the primary key.

[0070] In this specification, the primary key can be determined according to actual needs. In an optional embodiment, the primary key can be a serial number of data, etc. When the amount of data is large and it is difficult to distinguish two or more data by a primary key, the process in this specification can use multiple primary keys.

[0071] As can be seen from the above content, whether the corresponding relationship with the first position is determined by hash calculation or by data attributes, there may be certain conflicts in the process of batch data import. For example, in the above example of using hash calculation to determine the corresponding relationship, if there is a hash conflict, there are multiple stored data corresponding to the same first position, such as Figure 3a In the example shown, stored data 3 and stored data m both correspond to first position n. At this time, if a designated mark is stored in the first position corresponding to a certain data to be imported determined in the reference database, the target database may not store the same stored data as the data to be imported.

[0072] Therefore, in an optional embodiment of the present specification, for each data to be imported, if the data to be imported is stored with a specified mark in the first position corresponding to the reference database, the stored data in the target database is searched. If the stored data identical to the data to be imported is not found, the data to be imported is imported into the target database. Since the phenomenon of conflict (such as hash conflict) is not a common phenomenon, the process in the present specification requires searching the stored data in the target database, which takes less time than the prior art. Moreover, in the present embodiment, even if there is a conflict, it is possible to accurately determine whether the data to be imported is duplicate data.

[0073] In the related art, the time spent on importing batch data is not only affected by finding duplicate data, but may also be caused by reading data from the disk, that is, high IO cost will also increase the time spent on importing batch data.

[0074] In view of this, in an optional embodiment of the present specification, the target database is established in a disk. The process of generating a reference database may be: determining a designated area in a memory corresponding to the disk. Storing a database template in the designated area. Saving a designated mark for the first position corresponding to the stored data in the database template to obtain a reference database.

[0075] In this embodiment, since the reference database is established in the memory, in the process of determining duplicate data based on the reference database, the access to the disk can be reduced to a large extent, which is conducive to reducing IO costs.

[0076] In the case where the data to be imported is stored with a specified mark in the corresponding first position in the reference database, in order to reduce the IO cost caused by accessing the target database, in an optional embodiment of the present specification, the reference database further includes a plurality of second positions, exemplarily as Figure 4a As shown, each first position corresponds to at least one second position, and the stored data stored in the second position is the stored data corresponding to the first position corresponding to the second position in the target data. For each data to be imported, if a specified mark is stored in the first position corresponding to the data to be imported in the reference database, the first position corresponding to the data to be imported in the reference database is determined as the specified position. It is judged whether the stored data in the second position corresponding to the specified position is the same as the data to be imported. For example, as shown in 3b, the data to be imported 3 has a specified mark 1 stored in the first position n corresponding to the reference database, then it is judged whether the data to be imported is the same as one of the stored data in the second position n1 and the second position n2. If not, the data to be imported is imported into the target database. If so, it indicates that the data to be processed is duplicate data, and the data to be imported is optionally not imported into the target database.

[0077] In this embodiment, even if a conflict occurs and the stored data needs to be queried, there is no need to access the disk, and the data in the second position of the reference database can be directly searched. In addition, there is a corresponding relationship between the first position and the second position in the reference database. In this embodiment, even if the data in the second position of the reference database is searched, there is no need to search the data in all the second positions, and only the second position corresponding to the data to be imported needs to be searched. This can effectively reduce the time consumption caused by the search.

[0078] In the case of a more serious conflict, one first position may correspond to multiple second positions.

[0079] It can be seen from the above content that the process in this book can effectively determine the duplicate data of the data to be imported relative to the stored data. However, in the actual batch data import process, there may be a phenomenon that a certain data to be imported is duplicated with another data to be imported. For example, if the data to be imported 1 and the data to be imported 4 are the same data, the data to be imported 1 has been imported into the target database and has become the stored data in the target database. At this time, if the data to be imported 4 is imported into the target database, the duplicate data will be imported into the target database.

[0080] In view of this, in order to cope with the phenomenon of duplicate data in the data to be imported, in an optional embodiment of the present specification, for each data to be imported, if the first position corresponding to the data to be imported does not store a designated mark, the data to be imported is stored in the first position corresponding to the reference database, and the data to be imported is imported into the target database. On the one hand, it is possible to import non-duplicate data to be imported into the target database, thereby achieving the purpose of the present specification; on the other hand, in the process of importing the data to be imported into the target database, the reference database is updated according to the data to be imported that has been imported into the target database, so that the state of storing the designated mark in each first position of the reference database can also change in time with the change of the data stored in the target database, effectively identifying duplicate data in the data to be imported, so as to avoid importing duplicate data to be imported into the target database.

[0081] It can be seen that the batch data import process in this manual can effectively deduplicate the imported data, and the reasons for the generation of duplicate data in the imported data can be ignored. In some more specific scenarios, the duplicate data in the data to be imported may be caused by the operator's erroneous operation or environmental factors that perform the data import. For example, Zhang San, who performs the data import operation, had a power outage during the process of importing the data to be imported with batch identification A into the target database yesterday, resulting in the termination of the data import operation. At this time, part of the data to be imported has been imported into the target database, while the other part of the data to be imported has not been imported into the target database. Today, Zhang San again imports the data to be imported with batch identification A into the target database. Since part of the data to be imported with batch identification A has already been imported into the target database, this part of the data to be imported is duplicate data.

[0082] Since Zhang San is the operator who performs the import operation for the data to be imported with batch identification A, Zhang San is aware of the interruption of the data import operation that occurred yesterday, and Zhang San can estimate the amount of duplicate data in the data to be imported in the target database. Then in an optional embodiment of the present specification, a data import instruction can be generated by the terminal, and the data import instruction carries the data to be imported and a specified number. The specified number is an integer greater than or equal to 0. The specified number is used to determine the number of second positions corresponding to each first position. When the specified number is 0, the second position is not included in the reference database. The larger the value of the specified number, the more duplicate data there is in the data to be imported. The specified number can be input by operators corresponding to the middle and high ends.

[0083] Then, the terminal sends the data import instruction to the data import tool of this case. The data import tool of this case parses the data to be imported and the specified quantity from the data import instruction. Then, the specified number of second positions are determined, and a database template is generated according to the determined second positions, first positions, and the correspondence between the first position and the second position. Then, a reference database is generated according to the database template.

[0084] It can be seen that the batch data import achieved through this embodiment can predictably determine the number of second positions in the process of generating a reference database, so that the reference database can avoid the phenomenon of insufficient second positions or redundant second positions occupying storage space during use.

[0085] The generation process of the data import instruction is implemented to guide the operator in a further optional embodiment of this specification. Before the terminal sends the data to be imported, the data import tool of this case generates a prompt message and sends it to the terminal. The prompt message includes: a prompt for inputting a specified quantity. After the terminal receives the prompt message, the prompt message is added to the page displayed to the user (for example, the aforementioned operator), so that the operator enters the specified quantity in the terminal according to the prompt message.

[0086] Based on the same idea, this specification further provides a data batch import device, and the data batch import device in this specification is applied to the server, such as Figure 5 As shown, the data batch import device includes one or more of the following modules:

[0087] The to-be-imported data acquisition module 500 is configured to: acquire the to-be-imported data, where the to-be-imported data is data to be imported into a target database;

[0088] The reference database acquisition module 502 is configured to: acquire a reference database, wherein the data to be imported is data to be imported into a target database, the reference database includes a plurality of first positions, a first position in the reference database corresponding to the stored data stores a specified mark, the stored data is data stored in the target database, and the correspondence between the first position and the stored data is determined in a specified manner;

[0089] The first position determination module 504 is configured to: for each data to be imported, determine the first position corresponding to it in the reference database in a specified manner;

[0090] The first import module 506 is configured to: import the data to be imported into the target database when no designated mark is stored in the first position corresponding to the data to be imported.

[0091] In an optional embodiment of the present specification, the data batch importing device may further include: a second importing module 508 .

[0092] The second import module 508 is configured as follows: for each data to be imported, if the data to be imported has a specified mark stored in the corresponding first position in the reference database, then the stored data in the target database is searched; if no stored data identical to the data to be imported is found, then the data to be imported is imported into the target database.

[0093] In an optional embodiment of the present specification, the reference database further includes a plurality of second locations, each first location corresponds to at least one second location, and the stored data stored in the second location is the stored data corresponding to the first location corresponding to the second location in the target data. The data batch import device may further include: a third import module 510.

[0094] The third import module 510 is configured as follows: for each data to be imported, if a specified mark is stored in the first position corresponding to the data to be imported in the reference database, the first position corresponding to the data to be imported in the reference database is determined as the specified position; if the stored data in the second position corresponding to the specified position is different from the data to be imported, the data to be imported is imported into the target database.

[0095] In an optional embodiment of the present specification, the first import module 506 is specifically configured to: save the specified mark to the first position corresponding to the data to be imported in the reference database, and import the data to be imported into the target database.

[0096] In an optional embodiment of the present specification, the target database is established in a disk of a storage device, and the storage device also includes a memory. The reference database acquisition module 502 is specifically configured to: determine a designated area in the memory; store the database template in the designated area; save the designated mark to the first position corresponding to the stored data in the database template, and obtain the reference database.

[0097] In an optional embodiment of the present specification, the reference database acquisition module 502 is specifically configured as follows: performing a hash calculation on each stored data, determining a first target superscript based on a first result obtained by the hash calculation, using the first position corresponding to the first target superscript in the data block template as the first position corresponding to the stored data, and saving the designated mark to the first position corresponding to the stored data to obtain a reference database.

[0098] In an optional embodiment of the present specification, the first position determination module 504 is specifically configured as follows: performing a hash calculation for each data to be imported, determining a second target superscript based on a second result obtained by the hash calculation, and using the first position in the data block template corresponding to the second target superscript as the first position corresponding thereto in the reference database.

[0099] In an optional embodiment of the present specification, the data batch import device may further include: a primary key determination module 512. The primary key determination module 512 is specifically configured to: determine the primary key used in the current data batch import.

[0100] In an optional embodiment of the present specification, the reference database acquisition module 502 is specifically configured to perform a hash calculation on each field of the stored data corresponding to the primary key. The first position determination module 504 is specifically configured to perform a hash calculation on each field of the to-be-imported data corresponding to the primary key.

[0101] like Figure 6 As shown, the embodiment of the present application provides an air conditioner control device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0102] Memory 113, used for storing computer programs;

[0103] In one embodiment of the present application, the processor 111 is used to implement the control method for batch importing data provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113, including:

[0104] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of batch importing data provided in any of the aforementioned method embodiments are implemented.

[0105] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0106] The above is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A method for batch importing data, characterized in that: The method comprises: Acquire data to be imported and a reference database, wherein the data to be imported is data to be imported into a target database, the reference database includes a plurality of first positions, the first positions in the reference database corresponding to the stored data are stored with a specified mark, the stored data is data stored in the target database, and the correspondence between the first positions and the stored data is determined in a specified manner; For each piece of data to be imported, determine the first position corresponding to it in the reference database using the specified method; In the case where the designated mark is not stored in the first position corresponding to the data to be imported, importing the data to be imported into the target database; The reference database further includes a plurality of second positions, each of the first positions corresponds to at least one second position, and the stored data stored in the second position is the stored data corresponding to the first position corresponding to the second position in the target data, wherein after acquiring the reference database, the method further includes: For each of the to-be-imported data, if a designated mark is stored in a first position corresponding to the to-be-imported data in the reference database, the first position corresponding to the to-be-imported data in the reference database is determined as the designated position; In a case where the data stored in the second position corresponding to the designated position is different from the data to be imported, the data to be imported is imported into the target database.

2. The method according to claim 1, characterized in that After acquiring the reference database, the method further includes: For each of the data to be imported, if a designated mark is stored in the first position corresponding to the data to be imported in the reference database, searching for each stored data in the target database; If no stored data identical to the data to be imported is found, the data to be imported is imported into the target database.

3. The method according to claim 1, characterized in that Importing the to-be-imported data into the target database comprises: The designated mark is saved in a first position corresponding to the data to be imported in the reference database, and the data to be imported is imported into the target database.

4. The method according to claim 1, characterized in that: The target database is established in a disk of a storage device, and the storage device also includes a memory, wherein obtaining a reference database includes: In the memory, determining a designated area; Storing the database template in the designated area; The designated mark is saved in the first position corresponding to the stored data in the database template to obtain a reference database.

5. The method according to claim 1, characterized in that Acquiring a reference database includes: performing a hash calculation on each stored data, determining a first target corner mark according to a first result obtained by the hash calculation, using a first position corresponding to the first target corner mark in a data block template as a first position corresponding to the stored data, and saving a designated mark to the first position corresponding to the stored data to obtain a reference database; For each data to be imported, the specified method is used to determine the first position corresponding to it in the reference database, including: performing a hash calculation on each data to be imported, determining a second target corner mark according to a second result obtained by the hash calculation, and using the first position corresponding to the second target corner mark in the data block template as the first position corresponding to it in the reference database.

6. The method according to claim 1, characterized in that Before obtaining the reference database, the method further includes: determining a primary key used in the current batch import of data; Acquiring a reference database includes: generating a reference database according to a field of the stored data corresponding to a primary key; Adopting the specified method to determine the first position corresponding to it in the reference database includes: adopting the specified method to process the field of the to-be-imported data corresponding to the primary key to determine the first position corresponding to it in the reference database.

7. A device for batch importing data, characterized in that: The device comprises: The module for acquiring data to be imported is configured to: acquire data to be imported, wherein the data to be imported is data to be imported into a target database; A reference database acquisition module is configured to: acquire a reference database, wherein the reference database includes a plurality of first positions, the first positions in the reference database corresponding to the stored data are stored with a specified mark, the stored data are data stored in the target database, and the correspondence between the first positions and the stored data is determined in a specified manner; A first position determination module is configured to: for each data to be imported, determine the first position corresponding to it in the reference database by adopting the specified method; A first import module is configured to: import the data to be imported into the target database if no designated mark is stored in the first position corresponding to the data to be imported; The reference database further includes a plurality of second positions, each first position corresponds to at least one second position, and the stored data stored in the second position is the stored data corresponding to the first position corresponding to the second position in the target data, and the data batch import device further includes: The third import module is configured as follows: for each data to be imported, if a designated mark is stored in the first position corresponding to the data to be imported in the reference database, the first position corresponding to the data to be imported in the reference database is determined as the designated position; if the stored data in the second position corresponding to the designated position is different from the data to be imported, the data to be imported is imported into the target database.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the steps of the data processing method according to any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images