Data import method, device and equipment

Abstract

The present invention provides a data import method, device and equipment, which can be applied to the field of data processing technology. The method includes: in response to a data import instruction, establishing a connection relationship between the data to be imported and the target database, the data to be imported includes multiple format types, the data to be imported includes multiple row data, and at least two of the multiple row data are associated with the same format type; according to the read-write control identifier corresponding to the format type, updating the read-write properties of the multiple row data of the data to be imported, obtaining the updated source data, and writing the source data into a shared memory queue; using a parallel parsing thread to parse the source data in the shared memory queue to obtain a tuple corresponding to the source data; and writing the tuple into a target file of the target database.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically to a data import method, device and equipment. Background Art

[0002] In the current era of big data, the data stored in database systems is becoming increasingly large. When database administrators need to manage the database or perform data backup and recovery operations, they will need to import massive amounts of data.

[0003] In the related technology, it is often necessary to convert the data to be imported into structured statements first, then build a data table, or import it into the database after being processed by the database query processor. In addition, the above processing operations cannot realize the parallel import of multiple data, which is inefficient and consumes a lot of system resources. Summary of the invention

[0004] In view of the above problems, the present invention provides a data import method, device and equipment.

[0005] According to a first aspect of the present invention, a data import method is provided, comprising: in response to a data import instruction, establishing a connection relationship between data to be imported and a target database, the data to be imported comprising a plurality of format types, the data to be imported comprising a plurality of row data, at least two of the plurality of row data being associated with the same format type; updating the read-write properties of the plurality of row data of the data to be imported according to a read-write control identifier corresponding to the format type, obtaining updated source data, and writing the source data into a shared memory queue; parsing the source data in the shared memory queue using a parallel parsing thread, obtaining a tuple corresponding to the source data; and writing the tuple into a target file of the target database.

[0006] According to an embodiment of the present invention, a read-write control identifier includes a start identifier, an end identifier and a skip identifier; according to the read-write control identifier corresponding to the format type, the read-write properties of multiple row data of the data to be imported are updated to obtain updated source data, and the source data is written into a shared memory queue, including: when the format type of the data to be imported is a text format type, traversing the line break information corresponding to each row data of the data to be imported, the line break information includes the line break corresponding to each row data and the read-write control identifier corresponding to the line break; when the read-write control identifier is a start identifier, performing a read operation on the row data corresponding to the start identifier; wherein, during the read operation, when the line break corresponding to the skip identifier is identified, the row data corresponding to the skip identifier is skipped; during the read operation, when the line break corresponding to the end identifier is identified, after the reading of the row data corresponding to the end identifier is completed, the read operation is interrupted to obtain the source data; and the source data is written into the shared memory queue.

[0007] According to an embodiment of the present invention, the read and write properties of multiple rows of data to be imported are updated according to the read and write control identifier corresponding to the format type to obtain updated source data, and the source data is written to a shared memory queue, and also includes: when the format type of the data to be imported is a non-text format type, a reading operation is performed on multiple rows of data to be imported to obtain source data; wherein, during the reading operation, when a line break corresponding to a skip identifier is identified, the row data corresponding to the skip identifier is skipped; and the source data is written to a shared memory queue.

[0008] According to an embodiment of the present invention, the shared memory queues include multiple; writing source data into the shared memory queues includes: obtaining a weight corresponding to each shared memory queue; sorting the multiple shared memory queues according to the weights corresponding to the shared memory queues; and writing the source data into the shared memory queues according to the sorting of the multiple shared memory queues.

[0009] According to an embodiment of the present invention, source data includes multiple separators; the source data in a shared memory queue is parsed using parallel parsing threads to obtain tuples corresponding to the source data, including: based on multiple separators, the source data is split and processed to obtain multiple column data; a target data table of a target database is obtained to determine the number of columns of the target data table; the number of column data is compared with the number of columns of the target data table to obtain a comparison result; when the comparison result is consistent, a tuple corresponding to the source data is constructed based on the target data table.

[0010] According to an embodiment of the present invention, based on a target data table, a tuple corresponding to source data is constructed, including: determining attribute information of the target data table, the attribute information including a target table structure and a target format of the target data table; performing format conversion on column data according to the target format to obtain target source data; constructing an initial table structure corresponding to the target table structure; and using the target source data to update the initial table structure to obtain a tuple.

[0011] According to an embodiment of the present invention, source data in a shared memory queue is parsed using parallel parsing threads to obtain tuples corresponding to the source data, and the method also includes: when the comparison result is inconsistent, storing multiple column data corresponding to the source data into an abnormal data file in a target database; analyzing the log associated with the abnormal data file to determine target abnormal data from multiple column data; and pushing the target abnormal data.

[0012] According to an embodiment of the present invention, in response to a data import instruction, a connection relationship is established between the data to be imported and the target database, including: in response to the data import instruction, obtaining connection parameter information from the data to be imported; building a connection relationship between the data to be imported and the target database corresponding to the connection parameter information; and using a sorting component of the target database to receive the data to be imported.

[0013] The second aspect of the present invention provides a data import device, including: an establishment module, which is used to establish a connection relationship between the data to be imported and the target database in response to a data import instruction, the data to be imported includes multiple format types, the data to be imported includes multiple row data, and at least two of the multiple row data are associated with the same format type; an update module, which is used to update the read and write properties of multiple row data of the data to be imported according to the read and write control identifier corresponding to the format type, obtain the updated source data, and write the source data into a shared memory queue; a parsing module, which is used to parse the source data in the shared memory queue using parallel parsing threads to obtain tuples corresponding to the source data; and a writing module, which is used to write the tuples into the target file of the target database.

[0014] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0016] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0017] According to an embodiment of the present invention, in response to a data import instruction, a connection relationship is established between the data to be imported and the target database, the data to be imported includes multiple format types, the data to be imported includes multiple row data, and at least two of the multiple row data are associated with the same format type; according to the read-write control identifier corresponding to the format type, the read-write attributes of the multiple row data of the data to be imported are updated to obtain updated source data, and the source data is written into a shared memory queue; the source data in the shared memory queue is parsed using a parallel parsing thread to obtain a tuple corresponding to the source data; and the tuple is written into a target file of the target database. Due to the technical means of updating the read-write attributes of the data to be imported of multiple format types and parsing the data to be imported using a parallel parsing thread to obtain the corresponding tuple, the problem that the relevant processor needs to perform structured processing before importing the database and cannot adapt to the parallel import of multiple data is avoided, thereby realizing the direct writing of the data to be imported of different format types into the database, improving the read-write efficiency, realizing the parallel writing of multiple data to be imported, and reducing the resource overhead of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0019] Figure 1 An application scenario diagram of a data import method, apparatus, device, medium, and program product according to an embodiment of the present invention is shown;

[0020] Figure 2 A flow chart of a data import method according to an embodiment of the present invention is shown;

[0021] Figure 3 A schematic diagram of a data import method according to an embodiment of the present invention is shown;

[0022] Figure 4 It shows a structural block diagram of a data import device according to an embodiment of the present invention; and

[0023] Figure 5 A block diagram of an electronic device suitable for implementing a data import method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] An embodiment of the present invention provides a data import method, which establishes a connection relationship between the data to be imported and the target database in response to a data import instruction, wherein the data to be imported includes multiple format types, and the data to be imported includes multiple row data, and at least two of the multiple row data are associated with the same format type; according to the read-write control identifier corresponding to the format type, the read-write attributes of the multiple row data of the data to be imported are updated to obtain updated source data, and the source data is written into a shared memory queue; the source data in the shared memory queue is parsed using a parallel parsing thread to obtain a tuple corresponding to the source data; and the tuple is written into a target file of the target database.

[0029] Figure 1 An application scenario diagram of a data import method, apparatus, device, medium, and program product according to an embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may process the received data import instructions and data to be imported, and import the data to be imported into the target database.

[0034] It should be noted that the data import method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data import device provided in the embodiment of the present invention can generally be set in the server 105. The data import method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the data import device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] Figure 2 A flow chart of a data import method according to an embodiment of the present invention is shown.

[0037] like Figure 2 As shown, the data import method of this embodiment includes operations S210 to S240.

[0038] In operation S210, in response to a data import instruction, a connection relationship is established between the data to be imported and the target database, the data to be imported includes multiple format types, the data to be imported includes multiple rows of data, and at least two of the multiple rows of data are associated with the same format type.

[0039] According to an embodiment of the present invention, based on the data import instruction, a target database into which the to-be-imported data is to be imported is confirmed, and a network connection is established with the target database.

[0040] According to an embodiment of the present invention, the format type of the data to be imported may be a text format type, a binary data format type, or the like.

[0041] According to an embodiment of the present invention, data of any format type in the data to be imported may be composed of multiple rows of data for the same data, wherein each row of data serves as row data.

[0042] In operation S220, the read / write properties of multiple rows of data to be imported are updated according to the read / write control identifier corresponding to the format type, updated source data is obtained, and the source data is written into the shared memory queue.

[0043] According to an embodiment of the present invention, the read / write control identifier may be a parameter identifier such as skip or quote, which is used to control the reading of row data of the to-be-imported data corresponding to different format types, so as to speed up the import of the to-be-imported data.

[0044] According to an embodiment of the present invention, the read-write properties of multiple rows of data to be imported are updated to determine multiple rows of data of the same data, so as to obtain source data.

[0045] According to an embodiment of the present invention, the shared memory queue is used to store the above-obtained source data.

[0046] In operation S230 , the source data in the shared memory queue is parsed using parallel parsing threads to obtain tuples corresponding to the source data.

[0047] According to an embodiment of the present invention, a parallel parsing thread may be a thread for parsing source data of multiple shared memory queues, wherein the number of the parallel parsing threads may be adjusted according to the current number of source data in the shared memory queue.

[0048] According to an embodiment of the present invention, a tuple may be a data structure based on source data and used to store data that does not need to be modified.

[0049] In operation S240 , the tuple is written into a target file of a target database.

[0050] According to an embodiment of the present invention, the target file of the target database may be a file for storing tuples. During the writing process, if the number of tuples written exceeds a preset number of rows, the writing is stopped and the data import operation is completed.

[0051] According to an embodiment of the present invention, in response to a data import instruction, a connection relationship is established between the data to be imported and the target database, the data to be imported includes multiple format types, the data to be imported includes multiple row data, and at least two of the multiple row data are associated with the same format type; according to the read-write control identifier corresponding to the format type, the read-write attributes of the multiple row data of the data to be imported are updated to obtain updated source data, and the source data is written into a shared memory queue; the source data in the shared memory queue is parsed using a parallel parsing thread to obtain a tuple corresponding to the source data; and the tuple is written into a target file of the target database. Due to the technical means of updating the read-write attributes of the data to be imported of multiple format types and parsing the data to be imported using a parallel parsing thread to obtain the corresponding tuple, the problem that the relevant processor needs to perform structured processing before importing the database and cannot adapt to the parallel import of multiple data is avoided, thereby realizing the direct writing of the data to be imported of different format types into the database, improving the read-write efficiency, realizing the parallel writing of multiple data to be imported, and reducing the resource overhead of the system.

[0052] According to an embodiment of the present invention, a read-write control identifier includes a start identifier, an end identifier and a skip identifier; according to the read-write control identifier corresponding to the format type, the read-write properties of multiple row data of the data to be imported are updated to obtain updated source data, and the source data is written into a shared memory queue, including: when the format type of the data to be imported is a text format type, traversing the line break information corresponding to each row data of the data to be imported, the line break information includes the line break corresponding to each row data and the read-write control identifier corresponding to the line break; when the read-write control identifier is a start identifier, performing a read operation on the row data corresponding to the start identifier; wherein, during the read operation, when the line break corresponding to the skip identifier is identified, the row data corresponding to the skip identifier is skipped; during the read operation, when the line break corresponding to the end identifier is identified, after the reading of the row data corresponding to the end identifier is completed, the read operation is interrupted to obtain the source data; and the source data is written into the shared memory queue.

[0053] According to an embodiment of the present invention, the text format type may be field text (Comma Separated Values, CSV) including a plurality of separators, which is used to represent table data.

[0054] According to an embodiment of the present invention, each line of data is read in sequence to confirm the line break information corresponding to each line of data. Each line break information includes a read / write control identifier for controlling the reading and writing of the line of data.

[0055] According to an embodiment of the present invention, when a start mark is identified, the corresponding row data is used as the first row data of the source data and a reading operation is started.

[0056] According to an embodiment of the present invention, after the row data corresponding to the start identifier is read, the next row data is read. If the read-write control identifier corresponding to the row data is empty, the row data and the previous row data are the same source data, and the reading operation is continued to read the next row data.

[0057] According to an embodiment of the present invention, if the read / write control identifier corresponding to the row data is an end identifier, the corresponding row data is determined to be the last row of the source data, the read operation is interrupted, and all row data read in the read operation are taken as the same source data.

[0058] According to an embodiment of the present invention, in the above-mentioned reading operation, if a skip mark is identified, the corresponding row data does not need to be read, the row data is skipped, the read / write control mark of the next row data is identified, and the corresponding read / write control is performed until the reading operation is completed.

[0059] According to an embodiment of the present invention, for data to be imported in a text format, a read operation is performed on the data to be imported based on a read-write control identifier, and the obtained multiple source data are written into a shared memory queue.

[0060] According to an embodiment of the present invention, by utilizing read-write control identifiers, data in text format can be directly read and written to bypass the execution engine and storage engine of the database system, thereby reducing the function call level and complexity. In addition, since specified row data can be skipped and unnecessary memory consumption is reduced, the integrity of the source data can be guaranteed while improving writing efficiency.

[0061] According to an embodiment of the present invention, the read and write properties of multiple rows of data to be imported are updated according to the read and write control identifier corresponding to the format type to obtain updated source data, and the source data is written to a shared memory queue, and also includes: when the format type of the data to be imported is a non-text format type, a reading operation is performed on multiple rows of data to be imported to obtain source data; wherein, during the reading operation, when a line break corresponding to a skip identifier is identified, the row data corresponding to the skip identifier is skipped; and the source data is written to a shared memory queue.

[0062] According to an embodiment of the present invention, the non-text format type may be a format type other than the text format type, for example, a binary data format type or the like.

[0063] According to an embodiment of the present invention, for data to be imported that is not in a text format, it is not necessary to perform read-write control processing on each row of data, and each row of data is used as source data.

[0064] According to an embodiment of the present invention, in a read operation, if a skip mark is identified, the corresponding row of data does not need to be read, the row of data is skipped, and the next row of data is read and written.

[0065] According to an embodiment of the present invention, the shared memory queues include multiple; writing source data into the shared memory queues includes: obtaining a weight corresponding to each shared memory queue; sorting the multiple shared memory queues according to the weights corresponding to the shared memory queues; and writing the source data into the shared memory queues according to the sorting of the multiple shared memory queues.

[0066] According to an embodiment of the present invention, the weight corresponding to the shared memory queue may be the amount of source data stored in the shared memory queue.

[0067] According to an embodiment of the present invention, before writing new source data, the order of the current multiple shared memory queues is first determined, and the data writing operation is performed according to the order. For example, if the current number of source data in shared memory queue A is 5, the current number of source data in shared memory queue B is 8, and the current number of source data in shared memory queue C is 3, then the order is shared memory queue C, shared memory queue A, and shared memory queue B, and according to the order, the new source data is written into shared memory queue C.

[0068] According to an embodiment of the present invention, by performing write operations on source data based on the weights corresponding to the shared memory queues, random writes are avoided to avoid excessive source data in a shared memory queue, resulting in data accumulation, thereby improving data import efficiency.

[0069] According to an embodiment of the present invention, source data includes multiple separators; the source data in a shared memory queue is parsed using parallel parsing threads to obtain tuples corresponding to the source data, including: based on multiple separators, the source data is split and processed to obtain multiple column data; a target data table of a target database is obtained to determine the number of columns of the target data table; the number of column data is compared with the number of columns of the target data table to obtain a comparison result; when the comparison result is consistent, a tuple corresponding to the source data is constructed based on the target data table.

[0070] According to an embodiment of the present invention, the source data includes a plurality of separators for separating each column of data in the source data.

[0071] According to an embodiment of the present invention, the source data is split according to the separator to obtain a plurality of column data.

[0072] According to an embodiment of the present invention, a target data table of a target database is obtained, the number of columns of the target data table is determined, and the number of columns is compared with the number of column data obtained above. If they are consistent, it is considered that the source data conforms to the table structure of the target database.

[0073] According to an embodiment of the present invention, source data is processed based on a target database to obtain tuples that are consistent with the attributes of the target database.

[0074] According to an embodiment of the present invention, large amounts of source data are processed using parallel parsing threads, thereby avoiding the problem of large system call overhead caused by each source data call, and the maintenance cost is also reduced accordingly, which can flexibly adapt to application scenarios of different scales.

[0075] According to an embodiment of the present invention, based on a target data table, a tuple corresponding to source data is constructed, including: determining attribute information of the target data table, the attribute information including a target table structure and a target format of the target data table; performing format conversion on column data according to the target format to obtain target source data; constructing an initial table structure corresponding to the target table structure; and using the target source data to update the initial table structure to obtain a tuple.

[0076] According to an embodiment of the present invention, a target format of a target data table is determined, and the target format may be a numerical format type of a target database, such as a floating point type.

[0077] According to an embodiment of the present invention, if the value of the target data table is a floating point type, the column data is formatted and converted into a floating point type to obtain the target source data.

[0078] According to an embodiment of the present invention, a target table structure of a target data table is determined, and an initial table structure consistent with the target data table structure is constructed.

[0079] According to an embodiment of the present invention, the first row of target source data is used as header information of the tuple, and the remaining rows of data are imported into columns corresponding to the header information to update the initial table structure and obtain the tuple.

[0080] According to an embodiment of the present invention, in the above construction process, if the target database has an index, the tuple is constructed into an index tuple based on the index information of the target database.

[0081] According to an embodiment of the present invention, source data in a shared memory queue is parsed using parallel parsing threads to obtain tuples corresponding to the source data, and the method also includes: when the comparison result is inconsistent, storing multiple column data corresponding to the source data into an abnormal data file in a target database; analyzing the log associated with the abnormal data file to determine target abnormal data from multiple column data; and pushing the target abnormal data.

[0082] According to an embodiment of the present invention, when the comparison result is inconsistent, it is considered that the data to be imported is imported incorrectly, and multiple column data corresponding to the source data are stored in an abnormal data file in the target database.

[0083] According to an embodiment of the present invention, the log associated with the abnormal data file is analyzed, each column data is traversed, and compared with each column of the target data table to confirm the target abnormal data inconsistent with the target data table in the multiple column data.

[0084] According to an embodiment of the present invention, after the target abnormal data is determined, the target abnormal data is pushed by communication or the like.

[0085] According to an embodiment of the present invention, in response to a data import instruction, a connection relationship is established between the data to be imported and the target database, including: in response to the data import instruction, obtaining connection parameter information from the data to be imported; building a connection relationship between the data to be imported and the target database corresponding to the connection parameter information; and using a sorting component of the target database to receive the data to be imported.

[0086] According to an embodiment of the present invention, the connection parameters in the data import instruction are confirmed, and based on the connection parameters, the table name information of the target data table into which the to-be-imported data is to be imported is confirmed.

[0087] According to an embodiment of the present invention, the target data table is confirmed in the target database according to the table name information, and a connection relationship is established with the data to be imported.

[0088] According to an embodiment of the present invention, the sorting component of the target database can be a FIFO (First In, First Out) queue component, which obtains the data to be imported in a first-in, first-out form and transmits it to the compose thread used to update the imported data, thereby ensuring the order of the data.

[0089] Figure 3 A schematic diagram of a data import method according to an embodiment of the present invention is shown.

[0090] like Figure 3As shown, in operation S301, corresponding to the data import request, the data to be imported is obtained, and a connection relationship is established with the target database. Then, operation S302 is performed. For the data to be imported of different format types, each row of the data to be imported is read based on the read-write control identifier. For example, for the data to be imported of the text format type, based on the start identifier and the end identifier in the read-write control identifier, the same source data is confirmed in multiple rows of data. For the text format type and other format types, when the skip identifier is identified, the corresponding row data is skipped. After obtaining the source data, operation S303 is performed. According to the weight of the shared memory queue, the target shared memory queue is confirmed in multiple shared memory queues, and the source data is written. In operation S304, the source data in the shared memory queue is split and processed using parallel parsing threads to obtain multiple column data corresponding to each source data. Then, operation S305 is executed to determine whether the number of the column data is consistent with the number of columns of the target data table of the target database. If they are consistent, operation S306 is executed to construct a tuple corresponding to the source data based on the attribute information of the target data table. In operation S307, the obtained tuple is stored in the target file of the target database, and the target file includes the target data table. If it is determined that they are inconsistent in operation S305, operation S308 is executed to confirm the target abnormal data in the multiple column data corresponding to the source data and push it.

[0091] Based on the above data import method, the present invention also provides a data import device. Figure 4 The device is described in detail.

[0092] Figure 4 A structural block diagram of a data importing device according to an embodiment of the present invention is shown.

[0093] like Figure 4 As shown, the data importing device 400 of this embodiment includes a establishing module 410 , an updating module 420 , a parsing module 430 and a writing module 440 .

[0094] The establishment module 410 is used to establish a connection relationship between the data to be imported and the target database in response to the data import instruction, the data to be imported includes multiple format types, the data to be imported includes multiple rows of data, and at least two of the multiple rows of data are associated with the same format type. In one embodiment, the establishment module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0095] The update module 420 is used to update the read / write properties of multiple rows of data to be imported according to the read / write control identifier corresponding to the format type, obtain the updated source data, and write the source data into the shared memory queue. In one embodiment, the update module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0096] The parsing module 430 is used to parse the source data in the shared memory queue using parallel parsing threads to obtain tuples corresponding to the source data. In one embodiment, the parsing module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0097] The writing module 440 is used to write the tuple into the target file of the target database. In one embodiment, the writing module 440 can be used to perform the operation S240 described above, which will not be described in detail here.

[0098] According to an embodiment of the present invention, in response to a data import instruction, a connection relationship is established between the data to be imported and the target database, the data to be imported includes multiple format types, the data to be imported includes multiple row data, and at least two of the multiple row data are associated with the same format type; according to the read-write control identifier corresponding to the format type, the read-write attributes of the multiple row data of the data to be imported are updated to obtain updated source data, and the source data is written into a shared memory queue; the source data in the shared memory queue is parsed using a parallel parsing thread to obtain a tuple corresponding to the source data; and the tuple is written into a target file of the target database. Due to the technical means of updating the read-write attributes of the data to be imported of multiple format types and parsing the data to be imported using a parallel parsing thread to obtain the corresponding tuple, the problem that the relevant processor needs to perform structured processing before importing the database and cannot adapt to the parallel import of multiple data is avoided, thereby realizing the direct writing of the data to be imported of different format types into the database, improving the read-write efficiency, realizing the parallel writing of multiple data to be imported, and reducing the resource overhead of the system.

[0099] According to an embodiment of the present invention, any multiple modules of the establishment module 410, the update module 420, the parsing module 430 and the writing module 440 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the establishment module 410, the update module 420, the parsing module 430 and the writing module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the establishment module 410, the update module 420, the parsing module 430 and the writing module 440 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0100] According to an embodiment of the present invention, the update module 420 includes a traversal submodule, a first reading submodule, a first skipping submodule, an interruption submodule and a first writing submodule.

[0101] The traversal submodule is used to traverse the line break information corresponding to each row of the data to be imported when the format type of the data to be imported is a text format type. The line break information includes the line break corresponding to each row of data and the read-write control identifier corresponding to the line break.

[0102] The first reading submodule is used to perform a reading operation on the row data corresponding to the start identifier when the read / write control identifier is the start identifier.

[0103] The first skip submodule is used to skip the row data corresponding to the skip identifier when a line break character corresponding to the skip identifier is identified during a reading operation.

[0104] The interrupt submodule is used to interrupt the reading operation and obtain the source data when a line feed character corresponding to the end marker is identified during the reading operation and after the row data corresponding to the end marker is read.

[0105] The first writing submodule is used to write source data into the shared memory queue.

[0106] According to an embodiment of the present invention, the update module 420 further includes a second reading submodule, a second skipping submodule and a second writing submodule.

[0107] The second reading submodule is used to perform a reading operation on a plurality of rows of data to be imported to obtain source data when the format type of the data to be imported is a non-text format type.

[0108] The second skip submodule is used to skip the row data corresponding to the skip identifier when a line break character corresponding to the skip identifier is identified during the reading operation.

[0109] The second writing submodule is used to write the source data into the shared memory queue.

[0110] According to an embodiment of the present invention, the first writing submodule includes an acquiring unit, a sorting unit and a writing unit.

[0111] The acquisition unit is used to obtain the weight corresponding to each shared memory queue.

[0112] The sorting unit is used to sort multiple shared memory queues according to weights corresponding to the shared memory queues.

[0113] The writing unit is used to write the source data into the shared memory queue according to the order of the multiple shared memory queues.

[0114] According to an embodiment of the present invention, the parsing module 430 includes a splitting submodule, a first acquiring submodule, a comparing submodule and a first building submodule.

[0115] The split submodule is used to split the source data based on multiple delimiters to obtain multiple column data.

[0116] The first acquisition submodule is used to acquire a target data table of a target database and determine the number of columns of the target data table.

[0117] The comparison submodule is used to compare the number of column data with the number of columns in the target data table to obtain a comparison result.

[0118] The first construction submodule is used to construct a tuple corresponding to the source data based on the target data table when the comparison result is consistent.

[0119] According to an embodiment of the present invention, the construction submodule includes a determination unit, a conversion unit, a construction unit and an update unit.

[0120] The determination unit is used to determine the attribute information of the target data table, where the attribute information includes the target table structure and target format of the target data table.

[0121] The conversion unit is used to convert the format of the column data according to the target format to obtain the target source data.

[0122] A construction unit is used to construct an initial table structure corresponding to the target table structure.

[0123] The update unit is used to update the initial table structure using the target source data to obtain tuples.

[0124] According to the embodiment of the present invention, the parsing module 430 further includes a storage submodule, an analysis submodule and a push submodule.

[0125] The storage submodule is used to store multiple column data corresponding to the source data into an abnormal data file in the target database when the comparison result is inconsistent.

[0126] The analysis submodule is used to analyze the logs associated with the abnormal data file and determine the target abnormal data in the multiple column data.

[0127] The push submodule is used to push target abnormal data.

[0128] According to an embodiment of the present invention, the establishing module 410 includes a second acquiring submodule, a second constructing submodule and a receiving submodule.

[0129] The second acquisition submodule is used to acquire connection parameter information from the data to be imported in response to the data import instruction.

[0130] The second construction submodule is used to establish a connection relationship between the data to be imported and the target database corresponding to the connection parameter information.

[0131] The receiving submodule is used to receive the data to be imported by utilizing the sorting component of the target database.

[0132] Figure 5 A block diagram of an electronic device suitable for implementing a data import method according to an embodiment of the present invention is shown.

[0133] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0134] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0135] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0136] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0137] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0138] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data import method provided by the embodiment of the present invention.

[0139] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0140] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0141] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0142] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in a variety of ways, even if such combinations and / or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in a variety of ways without departing from the spirit and teachings of the present invention. All of these combinations and / or couplings fall within the scope of the present invention.

[0145] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data import method, characterized in that: The method comprises: In response to the data import instruction, establishing a connection relationship between the data to be imported and the target database, the data to be imported includes multiple format types, the data to be imported includes multiple rows of data, and at least two of the multiple rows of data are associated with the same format type; updating the read / write properties of the plurality of rows of data to be imported according to the read / write control identifier corresponding to the format type, obtaining updated source data, and writing the source data into a shared memory queue; Parsing the source data in the shared memory queue using parallel parsing threads to obtain tuples corresponding to the source data; Writing the tuple into a target file of the target database; The read / write control identifier includes a start identifier, an end identifier and a skip identifier; The updating of the read / write properties of the plurality of rows of data to be imported according to the read / write control identifier corresponding to the format type, obtaining updated source data, and writing the source data into a shared memory queue comprises: In the case where the format type of the data to be imported is a text format type, traversing line break information corresponding to each row of data of the data to be imported, the line break information including a line break corresponding to each row of data and the read / write control identifier corresponding to the line break; When the read / write control identifier is the start identifier, performing a read operation on the row data corresponding to the start identifier; Wherein, during the reading operation, when the line break character corresponding to the skip identifier is identified, the row data corresponding to the skip identifier is skipped; During the reading operation, when the line break corresponding to the end marker is identified, after the reading of the row data corresponding to the end marker is completed, the reading operation is interrupted to obtain the source data; Writing the source data into the shared memory queue; Wherein, updating the read / write properties of the plurality of rows of data to be imported according to the read / write control identifier corresponding to the format type, obtaining updated source data, and writing the source data into the shared memory queue further includes: In the case where the format type of the data to be imported is a non-text format type, performing a reading operation on a plurality of rows of the data to be imported to obtain the source data; Wherein, during the reading operation, when a line break character corresponding to the skip identifier is identified, the line data corresponding to the skip identifier is skipped; The source data is written into the shared memory queue.

2. The method according to claim 1, characterized in that The shared memory queue includes multiple; Writing the source data into the shared memory queue comprises: Obtaining a weight corresponding to each of the shared memory queues; sorting the multiple shared memory queues according to the weights corresponding to the shared memory queues; The source data is written into the shared memory queue according to the order of the multiple shared memory queues.

3. The method according to claim 1, characterized in that The source data includes multiple separators; The using parallel parsing threads to parse the source data in the shared memory queue to obtain tuples corresponding to the source data includes: Based on the multiple separators, the source data is split to obtain multiple column data; Acquire a target data table of the target database, and determine the number of columns of the target data table; Comparing the number of the column data with the number of columns of the target data table to obtain a comparison result; When the comparison result is a consistent comparison, the tuple corresponding to the source data is constructed based on the target data table.

4. The method according to claim 3, characterized in that The step of constructing the tuple corresponding to the source data based on the target data table includes: Determine attribute information of the target data table, wherein the attribute information includes a target table structure and a target format of the target data table; According to the target format, the column data is formatted to obtain target source data; Constructing an initial table structure corresponding to the target table structure; The initial table structure is updated using the target source data to obtain the tuple.

5. The method according to claim 3, characterized in that: The using parallel parsing threads to parse the source data in the shared memory queue to obtain tuples corresponding to the source data also includes: When the comparison result is inconsistent, storing the plurality of column data corresponding to the source data into an abnormal data file in the target database; Analyze the log associated with the abnormal data file, and determine the target abnormal data in the plurality of columns of data; The target abnormal data is pushed.

6. The method according to claim 1, characterized in that The step of establishing a connection relationship between the data to be imported and the target database in response to the data import instruction includes: In response to the data import instruction, obtaining connection parameter information from the data to be imported; Establishing a connection relationship between the data to be imported and the target database corresponding to the connection parameter information; The data to be imported is received by utilizing a sorting component of the target database.

7. A data import device, characterized in that: The device comprises: An establishing module, configured to establish a connection relationship between the data to be imported and the target database in response to a data import instruction, wherein the data to be imported includes multiple format types, the data to be imported includes multiple row data, and at least two of the multiple row data are associated with the same format type; An updating module, used for updating the read-write properties of the plurality of rows of data to be imported according to the read-write control identifier corresponding to the format type, obtaining updated source data, and writing the source data into a shared memory queue; a parsing module, configured to parse the source data in the shared memory queue using parallel parsing threads to obtain tuples corresponding to the source data; and A writing module, used for writing the tuple into a target file of the target database; Wherein, the update module includes a traversal submodule, a first reading submodule, a first skipping submodule, an interruption submodule and a first writing submodule; A traversal submodule, used for traversing the line break information corresponding to each row of the data to be imported when the format type of the data to be imported is a text format type, wherein the line break information includes the line break corresponding to each row of data and the read-write control identifier corresponding to the line break; A first reading submodule, configured to perform a reading operation on the row data corresponding to the start identifier when the read / write control identifier is a start identifier; A first skip submodule is used for skipping the row data corresponding to the skip identifier when the line break character corresponding to the skip identifier is identified during the reading operation; An interrupt submodule is used for, in the process of reading operation, when the line break character corresponding to the end mark is identified, interrupting the reading operation after the reading of the row data corresponding to the end mark is completed, so as to obtain the source data; A first writing submodule, used for writing the source data into the shared memory queue; Wherein, the update module further includes a second read submodule, a second skip submodule and a second write submodule; A second reading submodule is used for, when the format type of the data to be imported is a non-text format type, performing a reading operation on the plurality of rows of data to be imported to obtain the source data; A second skip submodule is used to skip the row data corresponding to the skip identifier when a line break character corresponding to the skip identifier is identified during the reading operation; The second writing submodule is used to write the source data into the shared memory queue.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

Images