Data real-time entry method and device, electronic equipment and storage medium

Abstract

The application relates to a data storage technology and discloses a data real-time lake entry method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a CDC event file in a database log by using an analysis tool, writing the CDC event file into a message queue to obtain CDC log data, subscribing to the CDC log data by using a log processing application to obtain real-time queue data, structurally processing the real-time queue data to obtain structured data, determining a SQL language rule corresponding to the structured data by using a blank decision tree, converting the structured data into a SQL statement by using the SQL language rule, determining an interface type corresponding to the SQL statement by using a neural network model, calculating an API interface corresponding to a maximum matching degree between the interface type and a preset data lake as an optimal transmission interface, and transmitting the SQL statement into an interactive application layer of the preset data lake through the API interface. The application can solve the problems of data update errors and large data update delays during data lake entry.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for real-time data import into a lake. Background Technology

[0002] With the arrival of the big data era, massive amounts of data are waiting to be processed. In order to improve the real-time performance and efficiency of data storage, a method is needed to automatically transmit and store data in real time. By parsing the database logs, the parsed message data is transferred to a real-time queue, realizing the real-time entry of data into the lake.

[0003] Existing traditional data import methods use offline approaches, typically supporting T+1 or intraday batch synchronization. In practice, this data import method is time-consuming, and the use of JDBC query interfaces during import places a burden on the source system, easily leading to data update errors and significant data update delays. Summary of the Invention

[0004] This invention provides a method, apparatus, and computer-readable storage medium for real-time data import into a lake, the main purpose of which is to solve the problems of data update errors and large data update delays during data import into the lake.

[0005] To achieve the above objectives, the present invention provides a method for real-time data import into a lake, comprising: Optionally, a preset database log is obtained, and a preset parsing tool is used to obtain the CDC event file in the database log; The CDC event file is written to a message queue to obtain CDC log data. The CDC log data is then subscribed to using a preset log processing application to obtain real-time queue data. The real-time queue data is processed into structured data, and the SQL language rules corresponding to the structured data are determined using a preset blank decision tree. The structured data is converted into SQL statements using SQL language rules, and the interface type corresponding to the SQL statement is determined by a preset neural network model. Calculate the matching degree between the interface type and the API interface of the preset data lake, select the API interface with the highest matching degree as the best transmission interface, and pass the SQL statement to the interactive application layer of the preset data lake through the API interface.

[0006] Optionally, the step of using a preset parsing tool to obtain the CDC event file from the database log includes: Connect the parsing tool to the database; The aforementioned parsing tool is used to parse the database logs in the database to obtain the parsed logs; The CDC event files in the parsed logs are transferred to the storage terminal using a parsing tool.

[0007] Optionally, determining the SQL language rules corresponding to the structured data using a preset blank decision tree includes: Add rules to a blank decision tree to obtain a rule-based decision tree; The rule decision tree is optimized and trained by acquiring preset input data and corresponding training rules to obtain a standard decision tree; The standard decision tree is used to determine the SQL language rules corresponding to the structured data.

[0008] Optionally, the step of adding rules to the blank decision tree to obtain a rule-based decision tree includes: Adding a decision tree to a blank decision tree results in an added decision tree. The pre-acquired dataset to be processed is input into the added decision tree to obtain a set of predicted values; Calculate the objective function value of the added decision tree based on the set of predicted values; When the objective function value is greater than or equal to the preset optimization threshold, the decision tree addition operation is performed until the objective function value is less than the optimization threshold, at which point the current added decision tree is output as a rule-based decision tree.

[0009] Optionally, the step of obtaining preset input data and corresponding training rules to optimize and train the rule decision tree to obtain a standard decision tree includes: Use rule-based decision trees to obtain the prediction rules corresponding to the input data; Calculate the error value between the prediction rule and the training rule; The parameters of the rule-based decision tree are adjusted based on the error value until the error value is within a preset error range, thus obtaining a standard decision tree.

[0010] Optionally, determining the interface type corresponding to the SQL statement through a preset neural network model includes: The SQL statements are encoded to obtain the SQL matrix; The SQL matrix is ​​processed by convolution and pooling to obtain low-dimensional feature information of the SQL matrix; Low-dimensional feature information is mapped to a pre-constructed high-dimensional space to obtain high-dimensional feature information; The interface type is obtained by filtering high-dimensional feature information using a preset activation function.

[0011] Optionally, calculating the matching degree between the interface type and the preset data lake API interface includes: The interface type and API interface are table-encoded to obtain the type coordinates and interface coordinates; The similarity between type coordinates and interface coordinates is calculated using the following matching formula: in, To calculate the similarity between the type coordinates and the interface coordinates, The x-coordinate of the type coordinates. The ordinate of the type coordinates. The x-coordinate of the interface coordinates. The ordinate of the interface coordinates; Similarity is defined as the degree of matching between the interface type and the API interface of the preset data lake.

[0012] To address the aforementioned problems, the present invention also provides a real-time data import device for a lake, the device comprising: Data acquisition module: used to acquire preset database logs, use preset parsing tools to acquire CDC event files in the database logs, write the CDC event files into a message queue to obtain CDC log data, and use a preset log processing application to subscribe to the CDC log data to obtain real-time queue data; Data processing module: used to perform structured processing on the real-time queue data to obtain structured data, use a preset blank decision tree to determine the SQL language rules corresponding to the structured data, use the SQL language rules to convert the structured data into SQL statements, and use a preset neural network model to determine the interface type corresponding to the SQL statements. Data Inbound Module: This module calculates the matching degree between the interface type and the preset data lake's API interface, selects the API interface with the highest matching degree as the best transmission interface, and transmits the SQL statement to the preset data lake's interactive application layer through the API interface.

[0013] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the real-time data import method described above.

[0014] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned real-time data import method.

[0015] This invention transmits SQL statements to the interactive application layer of a preset data lake via an API interface, thus ingesting data into the lake. This completes the entire real-time data ingestion architecture, ensuring the accuracy and efficiency of data transportation and making real-time data ingestion feasible. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a real-time data import method for a lake according to an embodiment of the present invention. Figure 2 This is a detailed flowchart illustrating one step of a real-time data import method for a lake according to an embodiment of the present invention. Figure 3 This is a detailed flowchart illustrating another step in the real-time data import method for a lake according to an embodiment of the present invention. Figure 4 This is a functional block diagram of a real-time data entry device for a lake according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device for implementing the real-time data import method in a lake according to an embodiment of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for real-time data import into a blockchain lake. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the real-time data import method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a real-time data import method for a lake according to an embodiment of the present invention. In this embodiment, the real-time data import method includes: S1. Obtain the preset database logs and use the preset parsing tool to obtain the CDC event files in the database logs; In this embodiment of the invention, the database log is a record of changes made to the database. It can record any operation of the database and save the record results in a separate file. The log has a very comprehensive record for each transaction process.

[0021] In this embodiment of the invention, the CDC event file actually provides historical change information to the user table by capturing the fact that the data manipulation language has been changed and the changed data itself. The CDC event file can be extracted in the form of a copy from the upstream data system.

[0022] In this embodiment of the invention, reference is made to Figure 2 As shown, the CDC event file in the database log is obtained using a preset parsing tool, including: S21. Connect the parsing tool to the database; S22. Use a parsing tool to parse the database logs in the database to obtain the parsed logs; S23. Transfer the CDC event file in the parsing log to the storage end using the parsing tool.

[0023] In this embodiment of the invention, the parsing tool includes, but is not limited to, a Python-based non-validation parser that can be used to parse SQL files.

[0024] In detail, obtaining CDC event files can capture all data changes, making it possible to obtain a more complete database file. At the same time, CDC has a low event latency rate, which can avoid increasing the CPU load, and can capture old record states and more metadata.

[0025] S2. Write the CDC event file into a message queue to obtain CDC log data, and use a preset log processing application to subscribe to the CDC log data to obtain real-time queue data. In this embodiment of the invention, the CDC event file is written to a message queue to obtain CDC log data, and the CDC event file is sent to the message queue via email.

[0026] In detail, writing CDC event files to a message queue can speed up data transmission, reduce waiting time, and keep data transmission efficient.

[0027] In this embodiment of the invention, real-time queue data is obtained by subscribing to CDC log data using a preset log processing application. The queue channel corresponding to the preset message queue is obtained, and the CDC log data and the queue channel are searched using the log processing application. When a queue channel matching CDC log data is found, the queue channel is subscribed to.

[0028] In detail, using log processing applications to retrieve CDC log data and queue channels is actually finding the subscription conditions corresponding to the queue channels. Only when CDC log data and queue channels are successfully matched can the corresponding data subscription be completed.

[0029] S3. The real-time queue data is processed into structured data to obtain structured data, and the SQL language rules corresponding to the structured data are determined using a preset blank decision tree. In this embodiment of the invention, structured data, also known as row data, is data that is logically expressed and implemented using a two-dimensional table structure. It strictly follows data format and length specifications and is mainly stored and managed through a relational database.

[0030] Because real-time queue data is complex and difficult to compute, it needs to be structured to make the data clearer and easier to use for subsequent calculations, thereby improving data processing efficiency.

[0031] In this embodiment of the invention, real-time queue data is processed into structured data, real-time queue data is segmented into words to obtain queue data words, queue data words are filled into a preset blank two-dimensional table, and the two-dimensional table is determined as structured data.

[0032] In this embodiment of the invention, reference is made to Figure 3 As shown, a pre-defined blank decision tree is used to determine the SQL language rules corresponding to structured data, including: S31. Add rules to the blank decision tree to obtain a rule-based decision tree; S32. Obtain the preset input data and corresponding training rules to optimize and train the rule decision tree, and obtain the standard decision tree; S33. Use standard decision trees to determine the SQL language rules corresponding to structured data.

[0033] In detail, the process of adding rules to a blank decision tree to obtain a rule-based decision tree includes: Adding a decision tree to a blank decision tree results in an added decision tree. The pre-acquired dataset to be processed is input into the added decision tree to obtain a set of predicted values; Calculate the objective function value of the added decision tree based on the set of predicted values; When the objective function value is greater than or equal to the preset optimization threshold, the decision tree addition operation is performed until the objective function value is less than the optimization threshold, at which point the current added decision tree is output as a rule-based decision tree.

[0034] In detail, the embodiments of the present invention compare the objective function value with the optimization threshold. When the objective function value is greater than or equal to the preset optimization threshold, it indicates that the predictive ability of the added decision tree is not strong enough. Therefore, it is necessary to perform the decision tree addition operation again until the objective function value is less than the optimization threshold. This indicates that the current added decision tree has a good predictive effect, and the current added decision tree is output as a rule decision tree.

[0035] Specifically, the step of acquiring preset input data and corresponding training rules to optimize and train the rule decision tree to obtain a standard decision tree includes: Use rule-based decision trees to obtain the prediction rules corresponding to the input data; Calculate the error value between the prediction rule and the training rule; The parameters of the rule-based decision tree are adjusted based on the error value until the error value is within a preset error range, thus obtaining a standard decision tree.

[0036] Furthermore, in this embodiment of the invention, the objective function value of the added decision tree is calculated using the following method based on the predicted value set: in, To add the objective function value to the decision tree, For the decision tree, The error value between the preset set of true values ​​and the set of predicted values. For the set of true values, For the set of predicted values, For the penalty term function, To add weights to the leaf nodes in the decision tree, To increase the number of leaf nodes in the decision tree, , and For fixed parameters, i refers to adding the i-th non-leaf node in the decision tree, n refers to the number of non-leaf nodes added in the decision tree, and j refers to adding the j-th leaf node in the decision tree.

[0037] In detail, the error value between the prediction rule and the training rule is calculated, including: The error between the prediction rule and the training rule is calculated using the following error calculation formula: in, This is the error value. In order to obtain the prediction rule, For training rules, The number of decision trees.

[0038] In detail, optimizing and training rule-based decision trees can improve their judgment ability, enhance their classification level, make the optimized standard decision trees more accurate and efficient, reduce data processing time, and enable real-time data transmission.

[0039] S4. Use SQL language rules to convert the structured data into SQL statements, and determine the interface type corresponding to the SQL statements through a preset neural network model; Because SQL statements require fixed language rules, it is essential to transform structured data according to these rules. Therefore, once the SQL language rules are determined, structured data can be transformed into the corresponding interface type using those rules.

[0040] In this embodiment of the invention, determining the interface type corresponding to the SQL statement through a preset neural network model includes: The SQL statements are encoded to obtain the SQL matrix; The SQL matrix is ​​processed by convolution and pooling to obtain low-dimensional feature information of the SQL matrix; Low-dimensional feature information is mapped to a pre-constructed high-dimensional space to obtain high-dimensional feature information; The interface type is obtained by filtering high-dimensional feature information using a preset activation function.

[0041] Specifically, low-dimensional feature information can be mapped to a pre-constructed high-dimensional space using preset mapping functions, including Gaussian Radial Basis Function and Gaussian function in the MATLAB library.

[0042] For example, if the low-dimensional feature information is a point in a two-dimensional plane, the two-dimensional coordinates of the point in the two-dimensional plane can be calculated using a mapping function to convert the two-dimensional coordinates into three-dimensional coordinates. Then, the calculated three-dimensional coordinates can be used to map the point to a pre-constructed three-dimensional space to obtain the high-dimensional feature information of the low-dimensional feature information.

[0043] In detail, convolution and pooling are performed on the SQL matrix to obtain its low-dimensional feature information, including: By pre-setting convolution kernels of different sizes, the convolution kernels are used to convolve the log matrix one by one to obtain the convolution matrix; Pooling is performed on the convolution matrix using the maximum or minimum value method based on the preset pooling window to obtain low-dimensional feature information of the log data.

[0044] In detail, there is a preset size difference between convolutional kernel pairs of different sizes. For example, a certain convolutional kernel is positioned at 3... Given a matrix of size 3, with a preset kernel size difference of 3, the sizes of the convolution kernel pairs are 3. 3 and 6 6. Setting a certain size difference can ensure that the resulting convolution matrix performs dimensionality reduction to the greatest extent while preserving the maximum features.

[0045] In this embodiment of the invention, the convolution matrix is ​​pooled using either a maximum or minimum value method according to a preset pooling window. Since the numerical characteristics of the convolution matrix are unpredictable, it is also impossible to determine whether to use the maximum or minimum value method for pooling. For example, in the convolution matrix, the preset pooling window is 5. For a matrix window of 5, if the number of larger values ​​in the convolution matrix window is much greater than the number of smaller values, then the minimum pooling method is selected. All values ​​in the pooling window are replaced with the minimum value in the pooling window, and then the next pooling window is analyzed until all values ​​in the convolution matrix pooling have completed the pooling operation.

[0046] S5. Calculate the matching degree between the interface type and the API interface of the preset data lake, select the API interface corresponding to the maximum matching degree as the best transmission interface, and transmit the SQL statement to the interactive application layer of the preset data lake through the API interface.

[0047] In this embodiment of the invention, the interactive application layer, also known as the application layer, is the highest layer of the open system. The application layer is directly connected to applications and provides common network application services.

[0048] Furthermore, APIs are essentially predefined functions that enable communication between computer software. Their primary function is to provide a general set of functionalities. In this embodiment of the invention, APIs are used to establish the interactive connection between the application layer and the application layer.

[0049] In this embodiment of the invention, calculating the matching degree between the interface type and the preset data lake API interface includes: The interface type and API interface are table-encoded to obtain the type coordinates and interface coordinates; The similarity between type coordinates and interface coordinates is calculated using the following matching formula: in, To calculate the similarity between the type coordinates and the interface coordinates, The x-coordinate of the type coordinates. The ordinate of the type coordinates. The x-coordinate of the interface coordinates. The ordinate of the interface coordinates; Similarity is defined as the degree of matching between the interface type and the API interface of the preset data lake.

[0050] Specifically, different interface types correspond to different API interfaces. Traditional API interfaces fall into four main categories: remote procedure calls, standard query languages, file transfer, and information delivery. To filter different API interfaces based on interface type, it is necessary to calculate the matching degree between the interface type and the API interface to achieve efficient and accurate data transmission.

[0051] In detail, passing SQL statements to the interactive application layer of the preset data lake through the API interface is the process of data ingestion into the lake. This completes the entire real-time data ingestion architecture, ensuring the accuracy and efficiency of data transportation and making real-time data ingestion into the lake feasible.

[0052] like Figure 4 The diagram shown is a functional block diagram of a real-time data entry device for a lake provided in an embodiment of the present invention.

[0053] The real-time data import device 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the real-time data import device 100 may include a data acquisition module 101, a data processing module 102, and a data import module 103. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0054] In this embodiment, the functions of each module / unit are as follows: The data acquisition module 101 is used to acquire preset database logs, use preset parsing tools to acquire CDC event files in the database logs, write the CDC event files into a message queue to obtain CDC log data, and use a preset log processing application to subscribe to the CDC log data to obtain real-time queue data. The data processing module 102 is used to perform structured processing on the real-time queue data to obtain structured data, determine the SQL language rules corresponding to the structured data using a preset blank decision tree, convert the structured data into SQL statements using the SQL language rules, and determine the interface type corresponding to the SQL statements through a preset neural network model. The data entry module 103 is used to calculate the matching degree between the interface type and the API interface of the preset data lake, select the API interface corresponding to the maximum matching degree as the best transmission interface, and transmit the SQL statement to the interactive application layer of the preset data lake through the API interface.

[0055] In detail, the modules in the real-time data entry device 100 described in this embodiment of the invention employ the same methods as described above during use. Figures 1 to 3 The method used is the same as the real-time data import method described in the previous section, and it can produce the same technical effect, so it will not be repeated here.

[0056] like Figure 5 The diagram shown is a schematic representation of an electronic device for implementing a real-time data import method for a lake, according to an embodiment of the present invention.

[0057] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a real-time data entry program.

[0058] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing real-time data import programs) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0059] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a real-time data import program, but also to temporarily store data that has been output or will be output.

[0060] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0061] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0062] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0063] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0064] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0065] The real-time data import program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 10, can achieve the following: Obtain the preset database logs, and use the preset parsing tool to obtain the CDC event files in the database logs; The CDC event file is written to a message queue to obtain CDC log data. The CDC log data is then subscribed to using a preset log processing application to obtain real-time queue data. The real-time queue data is processed into structured data, and the SQL language rules corresponding to the structured data are determined using a preset blank decision tree. The structured data is converted into SQL statements using SQL language rules, and the interface type corresponding to the SQL statement is determined by a preset neural network model. Calculate the matching degree between the interface type and the API interface of the preset data lake, select the API interface with the highest matching degree as the best transmission interface, and pass the SQL statement to the interactive application layer of the preset data lake through the API interface.

[0066] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0067] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0068] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: Obtain the preset database logs, and use the preset parsing tool to obtain the CDC event files in the database logs; The CDC event file is written to a message queue to obtain CDC log data. The CDC log data is then subscribed to using a preset log processing application to obtain real-time queue data. The real-time queue data is processed into structured data, and the SQL language rules corresponding to the structured data are determined using a preset blank decision tree. The structured data is converted into SQL statements using SQL language rules, and the interface type corresponding to the SQL statement is determined by a preset neural network model. Calculate the matching degree between the interface type and the API interface of the preset data lake, select the API interface with the highest matching degree as the best transmission interface, and pass the SQL statement to the interactive application layer of the preset data lake through the API interface.

[0069] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0070] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0072] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0073] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0074] The blockchain referred to in this invention is an application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0075] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0076] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for real-time data import into a lake, characterized in that, The method includes: Obtain the preset database logs, and use the preset parsing tool to obtain the CDC event files in the database logs; The CDC event file is written to a message queue to obtain CDC log data. The CDC log data is then subscribed to using a preset log processing application to obtain real-time queue data. The real-time queue data is processed into structured data, and the SQL language rules corresponding to the structured data are determined using a preset blank decision tree. The structured data is converted into SQL statements using SQL language rules, and the interface type corresponding to the SQL statement is determined by a preset neural network model. Calculate the matching degree between the interface type and the API interface of the preset data lake, select the API interface with the maximum matching degree as the best transmission interface, and pass the SQL statement to the interactive application layer of the preset data lake through the API interface; The step of determining the interface type corresponding to the SQL statement through a preset neural network model includes: encoding the SQL statement to obtain an SQL matrix; performing convolution and pooling processing on the SQL matrix to obtain low-dimensional feature information of the SQL matrix; mapping the low-dimensional feature information to a pre-constructed high-dimensional space to obtain high-dimensional feature information; and using a preset activation function to filter the high-dimensional feature information to obtain the interface type. The calculation of the matching degree between the interface type and the preset data lake API interface includes: The interface type and API interface are table-encoded to obtain the type coordinates and interface coordinates; The similarity between type coordinates and interface coordinates is calculated using the following matching formula: in, To calculate the similarity between the type coordinates and the interface coordinates, The x-coordinate of the type coordinates. The ordinate of the type coordinates. The x-coordinate of the interface coordinates. The ordinate of the interface coordinates; Similarity is defined as the degree of matching between the interface type and the API interface of the preset data lake.

2. The real-time data import method for the lake as described in claim 1, characterized in that, The step of obtaining the CDC event file from the database log using a preset parsing tool includes: Connect the parsing tool to the database; The aforementioned parsing tool is used to parse the database logs in the database to obtain the parsed logs; The CDC event files in the parsed logs are transferred to the storage terminal using a parsing tool.

3. The real-time data import method for the lake as described in claim 1, characterized in that, The step of determining the SQL language rules corresponding to the structured data using a preset blank decision tree includes: Add rules to a blank decision tree to obtain a rule-based decision tree; The rule decision tree is optimized and trained by acquiring preset input data and corresponding training rules to obtain a standard decision tree; The standard decision tree is used to determine the SQL language rules corresponding to the structured data.

4. The real-time data import method for the lake as described in claim 3, characterized in that, The process of adding rules to a blank decision tree to obtain a rule-based decision tree includes: Adding a decision tree to a blank decision tree results in an added decision tree. The pre-acquired dataset to be processed is input into the added decision tree to obtain a set of predicted values; Calculate the objective function value of the added decision tree based on the set of predicted values; When the objective function value is greater than or equal to the preset optimization threshold, the decision tree addition operation is performed until the objective function value is less than the optimization threshold, at which point the current added decision tree is output as a rule-based decision tree.

5. The real-time data import method for the lake as described in claim 3, characterized in that, The step of acquiring preset input data and corresponding training rules to optimize and train the rule decision tree to obtain a standard decision tree includes: Use rule-based decision trees to obtain the prediction rules corresponding to the input data; Calculate the error value between the prediction rule and the training rule; The parameters of the rule-based decision tree are adjusted based on the error value until the error value is within a preset error range, thus obtaining a standard decision tree.

6. A real-time data import device for implementing the real-time data import method as described in any one of claims 1 to 5, characterized in that, The device includes: Data acquisition module: used to acquire preset database logs, use preset parsing tools to acquire CDC event files in the database logs, write the CDC event files into a message queue to obtain CDC log data, and use a preset log processing application to subscribe to the CDC log data to obtain real-time queue data; Data processing module: used to perform structured processing on the real-time queue data to obtain structured data, use a preset blank decision tree to determine the SQL language rules corresponding to the structured data, use the SQL language rules to convert the structured data into SQL statements, and use a preset neural network model to determine the interface type corresponding to the SQL statements. Data Inbound Module: This module calculates the matching degree between the interface type and the preset data lake's API interface, selects the API interface with the highest matching degree as the best transmission interface, and transmits the SQL statement to the preset data lake's interactive application layer through the API interface.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the real-time data import method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the real-time data import method for the lake as described in any one of claims 1 to 5.

Images