Method and device for loading data to database based on multithreading concurrence and plug-in mode

Abstract

The embodiment of the invention belongs to the technical field of artificial intelligence and database data processing, and relates to a method for loading data to a database based on multithreading concurrence and plug-in, which comprises the following steps: acquiring database data loading request information which comprises a data loading function for loading database data, the data loading function is loaded into the target database after being connected with a preset data loading tool through a preset database data loading plug-in; acquiring a configuration file according to the data loading function; according to the configuration file, a plurality of analysis threads are started to analyze the data of the source database, and corresponding analysis results form tuples which are put into corresponding tuple queues; and traversing each tuple queue, taking out corresponding tuple data, generating target data and returning the target data to the target client. The invention further provides a device for loading the data to the database based on multi-thread concurrency and plug-in mode, computer equipment and a storage medium. According to the method, the database data loading performance is improved.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence and database data processing, and in particular to a method and device for loading data into a database based on multi-threaded concurrency and plug-in. Background Art

[0002] With the popularity of the Internet, the Internet of Things, and other data generation channels, data has exploded, and the application of databases has become more and more extensive. The total amount of database data and the amount of individual files required to be loaded are also increasing. Therefore, the efficiency of database data loading is a very important indicator of the overall performance of the database. How to quickly load data into the database, especially for the loading of a single large file, has become a research focus in the database field.

[0003] In the prior art, multiple processes are used for concurrency and plug-in technology for database data loading, but this technology has the following defects: 1. Before loading, it is necessary to specify the number of rows to be loaded in advance or perform a pre-scan to calculate the number of rows of the loaded file, and then calculate the starting offset of each concurrent process according to this number of rows during concurrency to perform concurrent execution, thereby reducing the data processing efficiency. 2. The use of multiple process concurrency requires the use of shared memory for data and process state interaction. The parsing process needs to serialize and write the data into the shared memory, and the writing process needs to read and deserialize from the shared memory, which requires two memory copies, serialization and deserialization, reducing the data processing speed. 3. Although the specific loading function of this technical solution is implemented by the server plug-in, there is also a client tool for command line parameter parsing or configuration file parsing, which increases the user's learning cost and maintenance cost, and also increases the probability of problems. 4. With multiple process concurrency, the system has a large overhead for creating processes, and starting the parsing process by simulating the client connection will involve connection security authentication, increasing security risks. It can be seen that there is a problem of poor database data loading performance in the prior art. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a method and device for loading data into a database based on multi-threaded concurrency and plug-in mode, so as to solve the problem of poor database data loading performance in the prior art.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for loading data into a database based on multi-threaded concurrency and plug-in, which adopts the following technical solution:

[0006] Obtaining database data loading request information, wherein the database data loading request information includes a data loading function for loading database data, wherein the data loading function is loaded into a target database after being connected to a preset data loading tool through a preset database data loading plug-in;

[0007] Obtaining a configuration file according to the data loading function;

[0008] According to the configuration file, multiple parsing threads are started to parse the source database data concurrently, and the corresponding parsing results are grouped into tuples and put into the corresponding tuple queue;

[0009] Traverse each tuple queue, take out the corresponding tuple data, generate the target data and return it to the target client.

[0010] In order to solve the above technical problems, the embodiment of the present application also provides a device for loading data into a database based on multi-threaded concurrency and plug-in, which adopts the following technical solution:

[0011] The device for loading data into the database based on multi-threaded concurrency and plug-in type includes:

[0012] A first acquisition module is used to acquire database data loading request information, wherein the database data loading request information includes a data loading function for loading database data, wherein the data loading function is loaded into a target database after being connected to a preset data loading tool through a preset database data loading plug-in;

[0013] A second acquisition module, used for acquiring a configuration file according to the data loading function;

[0014] A parsing module, used to start multiple parsing threads to concurrently parse the source database data according to the configuration file, and to group the corresponding parsing results into tuples and put them into corresponding tuple queues;

[0015] The generation module is used to traverse each tuple queue, take out the corresponding tuple data, and generate target data and return it to the target client.

[0016] Furthermore, the device for loading data into a database based on multi-threaded concurrency and plug-in mode also includes:

[0017] An initialization module, used to obtain a shared memory storing a loading status and initialize the loading status information of the shared memory;

[0018] The updating module is used to update the loading status information in the shared memory after each parsing thread completes parsing.

[0019] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0020] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for loading data into a database based on multi-threaded concurrency and plug-in type described in any one of the above embodiments are implemented.

[0021] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0022] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method for loading data into a database based on multi-threaded concurrency and plug-in type in any one of the above embodiments are implemented.

[0023] Compared with the prior art, the embodiment of the present application mainly has the following beneficial effects: by obtaining database data loading request information, the database data loading request information includes a data loading function for loading database data, and the data loading function is loaded into the target database after connecting with the preset data loading tool through the preset database data loading plug-in; according to the data loading function, a configuration file is obtained; according to the configuration file, multiple parsing threads are started to parse the source database data concurrently, and the corresponding parsing results are grouped into tuples and put into the corresponding tuple queue; each tuple queue is traversed, the corresponding tuple data is taken out, and the target data is generated and returned to the target client. In this way, it is registered to the database server in the form of a database data loading plug-in, and by providing a loaded function for the client to use, the user can directly call it directly through the existing client tool, which realizes the pluggability of the function, enhances the scalability, reduces the coupling with the database server, and reduces the learning and use cost of the user for the new tool. At the same time, using multi-threads for concurrent operations and storing them in the form of tuples and tuple queues reduces the concurrent startup overhead (the creation overhead of multi-threads is lower than that of multi-processes), reduces intermediate memory copies, reduces the single-point pre-reading of the entire file for concurrent task allocation, reduces the authentication of intermediate process connections, avoids security risks, and reduces the use of shared memory and other resources. This improves the database data loading performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0026] Figure 2 A flowchart of an embodiment of a method for loading data into a database based on multi-threaded concurrency and plug-in type according to the present application;

[0027] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0028] Figure 4 is Figure 3 A flowchart of a specific implementation method after step S303;

[0029] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S204;

[0030] Figure 6 It is a structural diagram of an embodiment of a device for loading data into a database based on multi-threaded concurrency and plug-in type according to the present application;

[0031] Figure 7 yes Figure 6 A structural diagram of a specific implementation of the parsing module;

[0032] Figure 8 is Figure 7 A structural schematic diagram of a specific implementation method after the recombination submodule;

[0033] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0035] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0038] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0040] The server 105 may be a server that provides various services, such as a background server that provides support for pages displayed on the terminal devices 101, 102, and 103. For example, a database server that provides database data loading services.

[0041] It should be noted that the method for loading data into a database based on multi-threaded concurrency and plug-in provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the device for loading data into a database based on multi-threaded concurrency and plug-in is generally set in the server / terminal device.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] Continue to refer Figure 2 , Figure 2 A flowchart of an embodiment of a method for loading data into a database according to the present application is shown. The method for loading data into a database based on multi-threaded concurrency and plug-in type includes the following steps:

[0044] Step S201, obtaining database data loading request information.

[0045] In this embodiment, the electronic device (for example, Figure 1 The server / terminal device shown in the figure can receive the database data loading request information through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0046] The database may be a financial database, and the database data may be data in a financial database. The database data loading request information includes a data loading function for loading database data, and the data loading function is loaded into the target database after connecting the preset database data loading plug-in with the preset data loading tool. The preset database data loading plug-in may be an existing data loading plug-in of the database, and adopts an existing plug-in mechanism. The preset data loading tool may be an existing client connection tool. For example, a user uses an existing database client connection tool to connect to the data loading plug-in of the database server, and registers the data loading function to the database server to complete the connection between the client tool and the database server, so as to facilitate the database data loading of the database, wherein the client connection tool may be a rasesql tool, the data loading function may be a rase_bulkload(input string, output string, config_file string) function, and the database may be a RASESQL database. The data loading function includes file path import parameters, target table import parameters, and file name import parameters, etc.

[0047] Specifically, when a user needs to load database data, a connection request can be sent to the database server through the existing client connection tool of the database, and with the help of the existing plug-in mechanism of the database, a new database data loading plug-in for batch loading of database data can be created, which can be named rase_bulkload plug-in. This plug-in mainly registers the data loading function rase_bulkload for loading database data to the database server. By using this function to load data in batches, the interface of the existing server can be fully utilized to load data. At the same time, the database connection is made with the help of the existing client connection tool rasesql tool, and then it is realized by calling the rase_bulkload function. This method can reduce the maintenance and learning cost of the new client tool. Since the existing database overall system adopts a multi-process architecture, a server process will be specially started to handle the connection request of the rasesql tool.

[0048] Step S202, obtaining a configuration file according to a data loading function.

[0049] The data loading function includes file path import parameters, target table import parameters, and file name import parameters. The file path import parameter (input) is the import file path, which is a string type. The file must be uploaded to the database server in advance when importing, and is the path of the database server; the target table import parameter (output) is the imported target table, in the format of [schema_name.]table_name. If the schema is not specified, the first matching table in the search path will be used by default; the file name import parameter (config_file) is the name of the imported configuration file, which is a string type. For details on the options supported by config_file, see the subsequent table.

[0050] The configuration file is a pre-set file used to search for source database data. The configuration file also includes a lot of configuration information, such as multiple parameter options, configuration options, etc.

[0051] In this embodiment, when the user calls the batch import data loading function rase_bulkload through the rasesql tool, the server will determine the configuration file more accurately and quickly according to the parameters provided by the data loading function to facilitate subsequent database data loading.

[0052] In one embodiment of the present invention, step S202 includes:

[0053] Import the configuration file path according to the file path import parameter; import the configuration file target table according to the target table import parameter; import the configuration file name according to the file name import parameter; determine the configuration file according to the configuration file path, configuration file target table and configuration file name.

[0054] In this embodiment, further, the configuration file path, configuration file target table and configuration file name of the configuration file are determined directly through the file path import parameter, target table import parameter and file name import parameter in the data loading function, so that the corresponding configuration file can be found quickly and accurately, further improving the database data loading performance.

[0055] Step S203, according to the configuration file, multiple parsing threads are started to parse the source database data concurrently, and the corresponding parsing results are grouped into tuples and put into the corresponding tuple queue.

[0056] The parsing thread is a thread for reading source database data and parsing the source database data. In this embodiment, the number of parsing threads is not limited, and the number of parsing threads can be set according to actual needs. Each parsing thread operates concurrently.

[0057] The source database data is the database data that the user needs to load. The file format of the source database data can be CSV format.

[0058] Tuple is a basic concept in relational databases. A relation is a table. Each row in the table (i.e., each record in the database) is a tuple, and each column is an attribute. In a two-dimensional table, a tuple is also called a row. Of course, a tuple can also be understood as a container that can store various data of the same or different types. To put it simply, it is to encapsulate multiple unrelated data into a whole, called a tuple. When a parsing thread parses a source database data, it obtains a tuple. A tuple queue is a queue used to store tuples (tuple data). A tuple queue can include multiple tuples.

[0059] One parsing thread corresponds to one tuple queue. Specifically, each parsing thread has its own tuple queue, which can reduce the contention for write locks between threads during data parsing. The parsing thread is responsible for writing, and a lock-free queue can be used to further avoid read-write contention.

[0060] In this embodiment, the number of threads for parallel processing of the specified server is determined by the configuration information of the configuration file. After the number of threads is determined, the corresponding multiple parsing threads are started. Each parsing thread reads and parses the loaded file, converts and forms tuples of the parsed data, and writes the corresponding tuples into the corresponding tuple queue. The currently loaded data file format can be CSV format.

[0061] Step S204, traverse each tuple queue, take out the corresponding tuple data, and generate target data to return to the target client.

[0062] In this embodiment, the target data may be the data obtained after successful loading. The target client may be the user end to which data services are provided, and the target client may connect to the database server through the existing client connection tool rasesql tool. Specifically, step 204 may be executed by the main thread. After each parsing thread parses the corresponding tuple and puts it into the tuple queue, the main thread may directly read the required tuple data from the tuple queue. Then, the tuple data obtained by reading is directly written into the data file of the database, thereby obtaining the corresponding target data. Finally, the obtained target data is returned to the target client, wherein the return form of the target data may be returned in the form of a function value. It should be noted that the main thread of this embodiment is responsible for reading the tuple data of each parsing thread, which can reduce the process of writing logs between each parsing thread, and the main thread directly writes the data file, further speeding up the speed of database data loading. It is further noted that, after the parsing thread processes all the data, a null value (NULL) will be inserted into the tuple queue to mark that all tuples of the current thread have been parsed, and the writing state will be updated only after the parsing thread has finished writing. NULL is a string, which specifies a string representing a null value. This can avoid the space occupied by shared memory for data and process status interaction and improve data processing speed.

[0063] The present application obtains database data loading request information, the database data loading request information includes a data loading function for loading database data, and the data loading function is loaded into the target database after connecting with the preset data loading tool through the preset database data loading plug-in; obtains the configuration file according to the data loading function; according to the configuration file, starts multiple parsing threads to parse the source database data concurrently, and forms the corresponding parsing results into tuples, and puts them into the corresponding tuple queue; traverses each tuple queue, takes out the corresponding tuple data, and generates the target data and returns it to the target client. In this way, it is registered to the database server in the form of a database data loading plug-in, and by providing a loaded function for the client to use, the user can directly call it through the existing client tool, which realizes the pluggability of the function, enhances the scalability, reduces the coupling with the database server, and reduces the user's learning and use costs of the new tool. At the same time, multi-threading is used for concurrent operations and stored in the form of tuples and tuple queues. Compared with the multi-process method, it reduces the concurrent startup overhead (the creation overhead of multi-threading is lower than the creation overhead of multi-process), reduces the intermediate memory copy, reduces the single-point pre-reading of the entire file in advance for concurrent task allocation, reduces the authentication of intermediate process connections to avoid security risks, and reduces the use of resources such as shared memory, thereby improving the database data loading performance.

[0064] In some optional implementations of this embodiment, in step 203, multiple parsing threads are started according to the configuration file to concurrently parse the source database data, and the corresponding parsing results are grouped into tuples and placed into the corresponding tuple queue. The electronic device may further perform the following steps:

[0065] Get the shared memory that stores the loading status and initialize the loading status information of the shared memory; after each parsing thread completes parsing, update the loading status information in the shared memory.

[0066] In this embodiment, the shared memory is used to store the loading status information of the interactive content of the data and thread status. When each parsing thread performs parsing, the loading status information of the shared memory is initialized, and the loading status in the shared memory is updated only after each parsing thread completes the parsing. This ensures that when reading the tuple data in each tuple queue, the parsing status of each parsing thread can be more accurately understood from the loading status in the shared memory, and the tuple data in each tuple queue can be further accurately written into the data file of the database, thereby further improving the database data loading performance.

[0067] In some optional implementations, continue to refer to Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific implementation method of step S203 in step S203, step S203 starts multiple parsing threads to parse the source database data concurrently according to the configuration file, and composes the corresponding parsing results into tuples and puts them into the corresponding tuple queue, including the following steps:

[0068] Step S301, after starting multiple parsing threads, number each parsing thread to obtain a thread number corresponding to each parsing thread.

[0069] In this embodiment, after starting multiple parsing threads, each parsing thread needs to be numbered from 0 to THREADS_COUNT-1, so as to monitor the status of each parsing thread during the data parsing process and avoid confusion.

[0070] Step S302 , reading the source database data by task division in a concurrent remainder manner according to the thread number of each parsing thread, and parsing the row data of the read source database data into a plurality of field data accordingly.

[0071] In this embodiment, each parsing thread will concurrently read CSV files in batches of 8k blocks. According to the parsing parameter skip specified in the configuration, the previous skip line data is directly ignored, and the limit line data is read until the limit line data is read or the entire load file is read. Each parsing thread will not process every piece of data read, but only process the data whose thread number is equal to [read CSV line number mod THREADS_COUNT]. Each parsing thread is divided into different tasks by taking the remainder. The parsing thread will parse the line data of the corresponding CSV file into multiple field data, and finally reorganize the field data into tuples according to the information of the configuration file target table and put them into the corresponding tuple queue. Among them, skip means that this option can be set to skip how many lines of data, and these data are not imported into the database. Limit means that the loading stops when the number of lines is loaded, and there is no limit on the number of loaded lines when the value is 0. THREADS_COUNT means the number of threads that the specified server processes in parallel.

[0072] Step S303: reorganize the field data of each parsing thread into tuples according to the information of the configuration file target table.

[0073] In this embodiment, the tuple is formed by reorganizing the information in the target table of the configuration file.

[0074] Step S304, putting the tuple parsed by each parsing thread into the corresponding tuple queue;

[0075] In this embodiment, the tuples parsed by each parsing thread are placed in their own tuple queue, which can avoid problems such as tuple data storage disorder and large data volume between parsing threads, thereby improving database data loading performance.

[0076] The present application creates multiple parsing threads and performs database data loading and parsing by concurrent remainder extraction, and stores the data in the form of tuples and tuple queues, thereby reducing concurrent startup overhead, reducing memory copies between parsing threads, reducing single-point pre-reading of the entire file in advance for concurrent task allocation, reducing authentication of intermediate process connections to avoid security risks, and reducing the use of resources such as shared memory, thereby improving loading performance.

[0077] In one embodiment of the present invention, Figure 4 As shown, Figure 4 is Figure 3 A flowchart of a specific implementation after step S303, after the field data is reorganized into tuples according to the information of the target table of the configuration file in step S303, the method for loading data into the database based on multi-threaded concurrency and plug-in mode also includes:

[0078] Step S401, preset a filtering function.

[0079] In this embodiment, the preset filter function can be a function pre-set by the user for filtering and transforming data. For example, the SQL or similar PL / SQL function registered by the user is used as the preset filter function, and the preset filter function can be called a FILTER function. The preset filter function is a specified filter function used to transform each line of the input file. If the function name is unique in the database, the parameter type definition of the function can be ignored. If this option is not specified, the input data will be directly parsed into the target table.

[0080] Step S402, calling a filtering function to filter the multiple field data obtained by each parsing thread to obtain the filtered multiple field data.

[0081] In this embodiment, after each parsing thread parses the field data, it can be processed by determining whether the FILTER function is empty. When the FILTER function is not empty, the FILTER function can be used to transform and filter the field data. If the FILTER function is empty, no field transformation will be performed.

[0082] This application uses preset filtering functions to perform data conversion and filtering on the database loading tool, thereby enhancing the functionality and flexibility of the tool and further improving the database data loading performance.

[0083] In this embodiment, if Figure 5 As shown, Figure 5 yes Figure 2 The flowchart of a specific implementation of step S204 is as follows. Step S204 traverses each tuple queue, takes out corresponding tuple data, and generates target data and returns it to the target client, including:

[0084] Step S501, traverse each tuple queue and read a tuple data from each tuple queue.

[0085] In this embodiment, the tuple data in the tuple queue may be read in order.

[0086] Step S502, determining whether the tuple data is a preset null value.

[0087] In this embodiment, the preset null value can be a value inserted into the corresponding tuple queue after each parsing thread completes the parsing, which is used to determine whether the corresponding parsing thread has completed the parsing. If the tuple data is a null value, it means that the corresponding parsing thread has completed the parsing and the tuple queue has been completed. If the tuple data is not a null value, it means that the corresponding parsing thread is still in the parsing process. Among them, each tuple queue has its own preset null value.

[0088] Step S503, if the tuple data is not a preset null value, the tuple data is written into the data file of the database, and the next tuple data is continuously extracted for judgment until all tuple data are written into the data file to generate target data.

[0089] In this embodiment, when the tuple data is not a preset null value, it means that the corresponding parsing thread is still in the parsing process, which means that the corresponding parsing thread will parse out new tuples and put them into the corresponding tuple queue. At this time, it is necessary to continue writing the tuple data in the tuple queue into the data file of the database until the null value tuple data is taken out from the tuple queue, then it is necessary to stop reading the tuple data in the tuple queue, and use the last obtained database file as the target data.

[0090] It should be noted that when writing through the database writing process, logs are written first and then data. However, our main thread reduces the intermediate log writing process and directly writes data files, which speeds up the database data loading speed and further improves the database data loading performance.

[0091] Step S504: if the tuple data is a preset null value, the corresponding tuple queue is marked as a processed queue, and the tuple data of the corresponding tuple queue is no longer written into the data file of the database.

[0092] In this embodiment, when the tuple data is a preset null value, it means that the parsing thread corresponding to the tuple queue has completed the parsing, and no new tuples will be put into the tuple queue, so the tuple data of the tuple queue is stopped from being written into the data file.

[0093] Step S505 , after all tuple queues are marked as processed queues, the obtained data file is determined as target data, and the target data is returned to the target client.

[0094] In this embodiment, after all parsing threads have completed parsing and the tuple data in each tuple queue are written into the data file, the obtained data file is used as the target data, and the target data is returned to the target client. The target data can be returned in the form of a function value. It should be noted that after the target data is returned to the target client, the thread can be closed and the resources can be released.

[0095] This application determines whether each parsing thread has completed processing by judging whether the tuple data in each tuple queue is a preset null value, thereby completing the writing process of each tuple data and obtaining the target data. In this process, the work of each parsing thread is independent and does not affect each other, thereby reducing the information interaction between each parsing thread, thereby improving the efficiency of database data loading and further improving the database data loading performance.

[0096] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0097] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0099] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0100] Further references Figure 6 , as a response to the above Figure 2 The present application provides an embodiment of a device for loading data into a database based on multi-threaded concurrency and plug-in type. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0101] like Figure 6As shown, the device 600 for loading data into a database based on multi-threaded concurrency and plug-in type in this embodiment includes: a first acquisition module 601, a second acquisition module 602, a parsing module 603 and a generating module 604. Among them:

[0102] The first acquisition module 601 is used to acquire database data loading request information, which includes a data loading function for loading database data. The data loading function is loaded into the target database after connecting with a preset data loading tool through a preset database data loading plug-in.

[0103] In this embodiment, the electronic device (for example, Figure 1 The server / terminal device shown in the figure can receive the database data loading request information through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0104] The database may be a financial database, and the database data may be data in a financial database. The database data loading request information includes a data loading function for loading database data, and the data loading function is loaded into the target database after connecting with the preset data loading tool through a preset database data loading plug-in. The preset database data loading plug-in may be an existing data loading plug-in of the database, and adopts an existing plug-in mechanism. The preset data loading tool may be an existing client connection tool. For example, an existing client connection tool of the database is used to connect to the data loading plug-in of the database server, and the data loading function is registered in the database server to complete the connection between the client tool and the database server, so as to facilitate the database data loading of the database, wherein the client connection tool may be a rasesql tool, the data loading function may be a rase_bulkload(input string, output string, config_filestring) function, and the database may be a RASESQL database.

[0105] The data loading function includes file path import parameters, target table import parameters, and file name import parameters.

[0106] Specifically, when the user needs to load database data, a connection request can be sent to the database server through the existing client connection tool of the database, and a new database data loading plug-in for batch loading of database data can be created with the help of the existing plug-in mechanism of the database, named rase_bulkload plug-in. This plug-in mainly registers the data loading function rase_bulkload for loading database data to the database server. By using this function to load data in batches, the interface of the existing server can be fully utilized to load data. At the same time, the database connection is made with the help of the existing client connection tool rasesql tool, and then it is realized by calling the rase_bulkload function. This method can reduce the maintenance and learning cost of the new client tool. Since the existing database overall system adopts a multi-process architecture, a server process will be specially started to handle the connection request of the rasesql tool.

[0107] The second acquisition module 602 is used to acquire the configuration file according to the data loading function.

[0108] In this embodiment, the data loading function includes a file path import parameter, a target table import parameter, and a file name import parameter. The file path import parameter (input) is the import file path, a string type, and the file must be uploaded to the database server in advance when importing, which is the path of the database server; the target table import parameter (output) is the imported target table, in the format of [schema_name.]table_name. If the mode is not specified, the first matching table in the search path will be used by default; the file name import parameter (config_file) is the name of the imported configuration file, a string type, and the options supported by config_file are detailed in the subsequent table.

[0109] The configuration file is a pre-set file used to search for source database data. The configuration file also includes a lot of configuration information, such as multiple parameter options, configuration options, etc.

[0110] In this embodiment, when the user calls the batch import data loading function rase_bulkload through the rasesql tool, the server will determine the configuration file more accurately and quickly according to the parameters provided by the data loading function to facilitate subsequent database data loading.

[0111] In one embodiment of the present invention, the second acquisition module 602 includes:

[0112] The first import submodule is used to import the configuration file path according to the file path import parameters; the second import submodule is used to import the configuration file target table according to the target table import parameters; the third import submodule is used to import the configuration file name according to the file name import parameters; the determination submodule is used to determine the configuration file according to the configuration file path, the configuration file target table and the configuration file name.

[0113] In this embodiment, further, the configuration file path, configuration file target table and configuration file name of the configuration file are determined directly through the file path import parameter, target table import parameter and file name import parameter in the data loading function, so that the corresponding configuration file can be found quickly and accurately, further improving the database data loading performance.

[0114] The parsing module 603 is used to start multiple parsing threads to concurrently parse the source database data according to the configuration file, and to group the corresponding parsing results into tuples and put them into the corresponding tuple queue.

[0115] The parsing thread is a thread for reading source database data and parsing the source database data. In this embodiment, the number of parsing threads is not limited, and the number of parsing threads can be set according to actual needs. Each parsing thread operates concurrently.

[0116] The source database data is the database data that the user needs to load. The file format of the source database data can be CSV format.

[0117] Tuple is a basic concept in relational databases. A relation is a table. Each row in the table (i.e., each record in the database) is a tuple, and each column is an attribute. In a two-dimensional table, a tuple is also called a row. Of course, a tuple can also be understood as a container that can store various data of the same or different types. To put it simply, it is to encapsulate multiple unrelated data into a whole, called a tuple. When a parsing thread parses a source database data, it obtains a tuple. A tuple queue is a queue used to store tuples (tuple data). A tuple queue can include multiple tuples.

[0118] One parsing thread corresponds to one tuple queue. Specifically, each parsing thread has its own tuple queue, which can reduce the contention for write locks between threads during data parsing. The parsing thread is responsible for writing, and a lock-free queue can be used to further avoid read-write contention.

[0119] In this embodiment, the number of threads for parallel processing of the specified server is determined by the configuration information of the configuration file. After the number of threads is determined, the corresponding multiple parsing threads are started. Each parsing thread reads and parses the loaded file, converts and forms tuples of the parsed data, and writes the corresponding tuples into the corresponding tuple queue. The currently loaded data file format can be CSV format.

[0120] The generation module 604 is used to traverse each tuple queue, take out the corresponding tuple data, and generate target data to return to the target client.

[0121] In this embodiment, the target data may be the data obtained after successful loading. The target client may be the user end to which data services are provided, and the target client may connect to the database server through the existing client connection tool rasesql tool. Specifically, step 204 may be executed by the main thread. After each parsing thread parses the corresponding tuple and puts it into the tuple queue, the main thread may directly read the required tuple data from the tuple queue. Then, the tuple data obtained by reading is directly written into the data file of the database, thereby obtaining the corresponding target data. Finally, the obtained target data is returned to the target client, wherein the return form of the target data may be returned in the form of a function value. It should be noted that the main thread of this embodiment is responsible for reading the tuple data of each parsing thread, which can reduce the process of writing logs between each parsing thread, and the main thread directly writes the data file, further speeding up the speed of database data loading. It is further noted that, after the parsing thread processes all the data, a null value (NULL) will be inserted into the tuple queue to mark that all tuples of the current thread have been parsed, and the writing state will be updated only after the parsing thread has finished writing. NULL is a string, which specifies a string representing a null value. This can avoid the space occupied by shared memory for data and process status interaction and improve data processing speed.

[0122] The present application obtains database data loading request information, the database data loading request information includes a data loading function for loading database data, and the data loading function is loaded into the target database after connecting with the preset data loading tool through the preset database data loading plug-in; obtains the configuration file according to the data loading function; according to the configuration file, starts multiple parsing threads to parse the source database data concurrently, and forms the corresponding parsing results into tuples, and puts them into the corresponding tuple queue; traverses each tuple queue, takes out the corresponding tuple data, and generates the target data and returns it to the target client. In this way, it is registered to the database server in the form of a database data loading plug-in, and by providing a loaded function for the client to use, the user can directly call it through the existing client tool, which realizes the pluggability of the function, enhances the scalability, reduces the coupling with the database server, and reduces the user's learning and use costs of the new tool. At the same time, multi-threading is used for concurrent operations and stored in the form of tuples and tuple queues. Compared with the multi-process method, it reduces the concurrent startup overhead (the creation overhead of multi-threading is lower than the creation overhead of multi-process), reduces the intermediate memory copy, reduces the single-point pre-reading of the entire file in advance for concurrent task allocation, reduces the authentication of intermediate process connections to avoid security risks, and reduces the use of resources such as shared memory, thereby improving the database data loading performance.

[0123] In some optional implementations of this embodiment, the above-mentioned device 600 for loading data into a database based on multi-threaded concurrency and plug-in mode further includes:

[0124] The initialization module is used to obtain the shared memory storing the loading status and initialize the loading status information of the shared memory; the update module is used to update the loading status information in the shared memory after each parsing thread completes the parsing.

[0125] In this embodiment, the shared memory is used to store the loading status information of the interactive content of the data and thread status. When each parsing thread performs parsing, the loading status information of the shared memory is initialized, and the loading status in the shared memory is updated only after each parsing thread completes the parsing. This ensures that when reading the tuple data in each tuple queue, the parsing status of each parsing thread can be more accurately understood from the loading status in the shared memory, and the tuple data in each tuple queue can be further accurately written into the data file of the database, thereby further improving the database data loading performance.

[0126] In some optional implementations, such as Figure 7 As shown, the parsing module 603 includes:

[0127] The numbering submodule 6031 is used to number each parsing thread after starting multiple parsing threads, so as to obtain a thread number corresponding to each parsing thread.

[0128] In this embodiment, after starting multiple parsing threads, each parsing thread needs to be numbered from 0 to THREADS_COUNT-1, so as to monitor the status of each parsing thread during the data parsing process and avoid confusion.

[0129] The parsing submodule 6032 is used to read the source database data by performing task division in a concurrent remainder manner according to the thread number of each parsing thread, and parse the row data of the read source database data into multiple field data accordingly.

[0130] In this embodiment, each parsing thread will concurrently read CSV files in batches of 8k blocks. According to the parsing parameter skip specified in the configuration, the previous skip line data is directly ignored, and the limit line data is read until the limit (limit) line data is read or the entire load file is read. Each parsing thread will not process every piece of data read, but only process the data with the thread number equal to [read CSV line number modTHREADS_COUNT]. Each parsing thread is divided into tasks by taking the remainder. The parsing thread will parse the line data of the corresponding CSV file into multiple field data, and finally reorganize the field data into tuples according to the information of the configuration file target table and put them into the corresponding tuple queue. Among them, skip means that this option can be set to skip how many lines of data, and these data are not imported into the database. limit means that the loading stops when the number of lines is loaded, and there is no limit on the number of loaded lines when the value is 0. THREADS_COUNT means the number of threads that the specified server processes in parallel.

[0131] The reorganization submodule 6033 is used to reorganize the field data of each parsing thread into tuples according to the information of the configuration file target table.

[0132] In this embodiment, the tuple is formed by reorganizing the information in the target table of the configuration file.

[0133] The submodule 6034 is used to put the tuples parsed by each parsing thread into the corresponding tuple queue;

[0134] In this embodiment, the tuples parsed by each parsing thread are placed in their own tuple queue, which can avoid problems such as tuple data storage disorder and large data volume between parsing threads, thereby improving database data loading performance.

[0135] The present application creates multiple parsing threads and performs database data loading and parsing by concurrent remainder extraction, and stores the data in the form of tuples and tuple queues, thereby reducing concurrent startup overhead, reducing memory copies between parsing threads, reducing single-point pre-reading of the entire file in advance for concurrent task allocation, reducing authentication of intermediate process connections to avoid security risks, and reducing the use of resources such as shared memory, thereby improving loading performance.

[0136] In one embodiment of the present invention, Figure 8 As shown, after the recombinant module 6033, it also includes:

[0137] The preset submodule 6035 is used to preset the filtering function.

[0138] In this embodiment, the preset filter function can be a function pre-set by the user for filtering and transforming data. For example, the SQL or similar PL / SQL function registered by the user is used as the preset filter function, and the preset filter function can be called a FILTER function. The preset filter function is a specified filter function used to transform each line of the input file. If the function name is unique in the database, the parameter type definition of the function can be ignored. If this option is not specified, the input data will be directly parsed into the target table.

[0139] The filtering submodule 6036 is used to call the filtering function to filter the multiple field data obtained by each parsing thread to obtain the filtered multiple field data.

[0140] In this embodiment, after each parsing thread parses the field data, it can be processed by determining whether the FILTER function is empty. When the FILTER function is not empty, the FILTER function can be used to transform and filter the field data. If the FILTER function is empty, no field transformation will be performed.

[0141] This application uses preset filtering functions to perform data conversion and filtering on the database loading tool, thereby enhancing the functionality and flexibility of the tool and further improving the database data loading performance.

[0142] In this embodiment, the generating module 604 includes:

[0143] The reading submodule is used to traverse each tuple queue and read a tuple data from each tuple queue.

[0144] In this embodiment, the tuple data in the tuple queue may be read in order.

[0145] The judgment submodule is used to judge whether the tuple data is a preset null value.

[0146] In this embodiment, the preset null value can be a value inserted into the corresponding tuple queue after each parsing thread completes the parsing, which is used to determine whether the corresponding parsing thread has completed the parsing. If the tuple data is a null value, it means that the corresponding parsing thread has completed the parsing and the tuple queue has been completed. If the tuple data is not a null value, it means that the corresponding parsing thread is still in the parsing process. Among them, each tuple queue has its own preset null value.

[0147] The writing submodule is used to write the tuple data into the data file of the database if the tuple data is not a preset null value, and continue to extract the next tuple data for judgment until all tuple data are written into the data file to generate target data.

[0148] In this embodiment, when the tuple data is not a preset null value, it means that the corresponding parsing thread is still in the parsing process, which means that the corresponding parsing thread will parse out new tuples and put them into the corresponding tuple queue. At this time, it is necessary to continue writing the tuple data in the tuple queue into the data file of the database until the null value tuple data is taken out from the tuple queue, then it is necessary to stop reading the tuple data in the tuple queue, and use the last obtained database file as the target data.

[0149] It should be noted that when writing through the database writing process, logs are written first and then data. However, our main thread reduces the intermediate log writing process and directly writes data files, which speeds up the database data loading speed and further improves the database data loading performance.

[0150] The marking submodule is used to mark the corresponding tuple queue as a processed queue if the tuple data is a preset null value, and no longer write the tuple data of the corresponding tuple queue into the data file of the database.

[0151] In this embodiment, when the tuple data is a preset null value, it means that the parsing thread corresponding to the tuple queue has completed the parsing, and no new tuples will be put into the tuple queue, so the tuple data of the tuple queue is stopped from being written into the data file.

[0152] The determination submodule is used to determine the obtained data file as the target data after all tuple queues are marked as processed queues, and return the target data to the target client.

[0153] In this embodiment, after all parsing threads have completed parsing and the tuple data in each tuple queue are written into the data file, the obtained data file is used as the target data, and the target data is returned to the target client. The target data can be returned in the form of a function value. It should be noted that after the target data is returned to the target client, the thread can be closed and the resources can be released.

[0154] This application determines whether each parsing thread has completed processing by judging whether the tuple data in each tuple queue is a preset null value, thereby completing the writing process of each tuple data and obtaining the target data. In this process, the work of each parsing thread is independent and does not affect each other, thereby reducing the information interaction between each parsing thread, thereby improving the efficiency of database data loading and further improving the database data loading performance.

[0155] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0156] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 7 with components 71-73, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0157] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0158] The memory 71 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 7. Of course, the memory 71 can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions based on multi-threaded concurrency and plug-in methods for loading data into a database. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or are to be output.

[0159] The processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run computer-readable instructions stored in the memory 71 or process data, such as running computer-readable instructions based on multi-threaded concurrency and plug-in-based data loading methods to a database.

[0160] The network interface 73 may include a wireless network interface or a wired network interface, and the network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0161] The present application obtains database data loading request information, the database data loading request information includes a data loading function for loading database data, and the data loading function is loaded into the target database after connecting with the preset data loading tool through the preset database data loading plug-in; obtains the configuration file according to the data loading function; according to the configuration file, starts multiple parsing threads to parse the source database data concurrently, and forms the corresponding parsing results into tuples, and puts them into the corresponding tuple queue; traverses each tuple queue, takes out the corresponding tuple data, and generates the target data and returns it to the target client. In this way, it is registered to the database server in the form of a database data loading plug-in, and by providing a loaded function for the client to use, the user can directly call it through the existing client tool, which realizes the pluggability of the function, enhances the scalability, reduces the coupling with the database server, and reduces the user's learning and use costs of the new tool. At the same time, multi-threading is used for concurrent operations and stored in the form of tuples and tuple queues. Compared with the multi-process method, it reduces the concurrent startup overhead (the creation overhead of multi-threading is lower than the creation overhead of multi-process), reduces the intermediate memory copy, reduces the single-point pre-reading of the entire file in advance for concurrent task allocation, reduces the authentication of intermediate process connections to avoid security risks, and reduces the use of resources such as shared memory, thereby improving the database data loading performance.

[0162] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the method for loading data into a database based on multi-threaded concurrency and plug-in type as described above.

[0163] The present application obtains database data loading request information, the database data loading request information includes a data loading function for loading database data, and the data loading function is loaded into the target database after connecting with the preset data loading tool through the preset database data loading plug-in; obtains the configuration file according to the data loading function; according to the configuration file, starts multiple parsing threads to parse the source database data concurrently, and forms the corresponding parsing results into tuples, and puts them into the corresponding tuple queue; traverses each tuple queue, takes out the corresponding tuple data, and generates the target data and returns it to the target client. In this way, it is registered to the database server in the form of a database data loading plug-in, and by providing a loaded function for the client to use, the user can directly call it through the existing client tool, which realizes the pluggability of the function, enhances the scalability, reduces the coupling with the database server, and reduces the user's learning and use costs of the new tool. At the same time, multi-threading is used for concurrent operations and stored in the form of tuples and tuple queues. Compared with the multi-process method, it reduces the concurrent startup overhead (the creation overhead of multi-threading is lower than the creation overhead of multi-process), reduces the intermediate memory copy, reduces the single-point pre-reading of the entire file in advance for concurrent task allocation, reduces the authentication of intermediate process connections to avoid security risks, and reduces the use of resources such as shared memory, thereby improving the database data loading performance.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0165] Obviously, the embodiments described above are only a part of the embodiments of the present application, rather than all the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to replace some of the technical features therein with equivalents. All equivalent structures made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, are similarly within the scope of patent protection of this application. The non-company software tools or components that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A method for loading data into a database based on multi-threaded concurrency and plug-in, characterized in that: The steps include: Obtaining database data loading request information, wherein the database data loading request information includes a data loading function for loading database data, wherein the data loading function is loaded into a target database after being connected to a preset data loading tool through a preset database data loading plug-in; Obtaining a configuration file according to the data loading function; According to the configuration file, multiple parsing threads are started to parse the source database data concurrently, and the corresponding parsing results are grouped into tuples and put into the corresponding tuple queue; Traverse each tuple queue, take out the corresponding tuple data, generate the target data and return it to the target client.

2. The method for loading data into a database based on multi-threaded concurrency and plug-in type according to claim 1, characterized in that: Before the step of starting multiple parsing threads to concurrently parse the source database data according to the configuration file, and composing corresponding parsing results into tuples and putting them into corresponding tuple queues, the step further includes: Acquire a shared memory storing a loading status, and initialize the loading status information of the shared memory; After each parsing thread completes parsing, the loading status information in the shared memory is updated.

3. The method for loading data into a database based on multi-threaded concurrency and plug-in type according to claim 1, characterized in that: The data loading function includes a file path import parameter, a target table import parameter, and a file name import parameter. The step of obtaining a configuration file according to the data loading function specifically includes: Import the configuration file path according to the file path import parameter; Importing a configuration file target table according to the target table import parameters; Import the configuration file name according to the file name import parameter; A configuration file is determined according to the configuration file path, the configuration file target table and the configuration file name.

4. The method for loading data into a database based on multi-threaded concurrency and plug-in type according to claim 2, characterized in that: The step of starting multiple parsing threads to concurrently parse the source database data according to the configuration file, and forming corresponding parsing results into tuples and putting them into corresponding tuple queues specifically includes: After starting multiple parsing threads, number each parsing thread to obtain the thread number corresponding to each parsing thread; According to the thread number of each parsing thread, the source database data is read by task division in a concurrent remainder manner, and the row data of the source database data read is parsed into multiple field data accordingly; Reorganize the field data of each parsing thread into tuples according to the information of the configuration file target table; Put the tuple parsed by each parsing thread into the corresponding tuple queue; Among them, one parsing thread corresponds to one tuple queue.

5. The method for loading data into a database based on multi-threaded concurrency and plug-in type according to claim 4, characterized in that: After the step of reorganizing the field data of each parsing thread into tuples according to the information of the configuration file target table, the method for loading data into the database based on multi-thread concurrency and plug-in mode also includes: Preset filter function; The filtering function is called to filter the multiple field data obtained by each parsing thread to obtain the filtered multiple field data.

6. The method for loading data into a database based on multi-threaded concurrency and plug-in type according to claim 2, characterized in that: The step of traversing each tuple queue, taking out corresponding tuple data, and generating target data and returning it to the target client specifically includes: Traverse each tuple queue and read a tuple data from each tuple queue; Determine whether the tuple data is a preset null value; If the tuple data is not a preset null value, the tuple data is written into a data file of the database, and the next tuple data is continuously extracted for judgment until all tuple data are written into the data file to generate target data; If the tuple data is a preset null value, the corresponding tuple queue is marked as a processed queue, and the tuple data of the corresponding tuple queue is no longer written into the data file of the database; After all tuple queues are marked as processed queues, the obtained data files are determined as target data, and the target data are returned to the target client.

7. A device for loading data into a database based on multi-threaded concurrency and plug-in, characterized in that: include: A first acquisition module is used to acquire database data loading request information, wherein the database data loading request information includes a data loading function for loading database data, wherein the data loading function is loaded into a target database after being connected to a preset data loading tool through a preset database data loading plug-in; A second acquisition module, used for acquiring a configuration file according to the data loading function; A parsing module, used to start multiple parsing threads to concurrently parse the source database data according to the configuration file, and to group the corresponding parsing results into tuples and put them into corresponding tuple queues; The generation module is used to traverse each tuple queue, take out the corresponding tuple data, and generate target data and return it to the target client.

8. The device for loading data into a database based on multi-threaded concurrency and plug-in type according to claim 7, characterized in that: The device for loading data into the database based on multi-threaded concurrency and plug-in mode also includes: An initialization module, used for acquiring a shared memory storing a loading status and initializing loading status information of the shared memory; The updating module is used to update the loading status information in the shared memory after each parsing thread completes parsing.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the method for loading data into a database based on multi-threaded concurrency and plug-in type as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for loading data into a database based on multi-threaded concurrency and plug-in mode as claimed in any one of claims 1 to 6.

Images