Big data volume data export method and device based on data flow

Abstract

The application discloses a large-data-volume data export method and device based on data flow, and the method comprises the following steps: determining database calling instructions corresponding to data export requirements according to the data export requirements; connecting a database according to the database calling instructions to obtain to-be-called data; and obtaining data files corresponding to the to-be-called data through data flow. The corresponding database calling instructions are determined through the data export requirements, the database is operated, and the required data is exported in the form of data files through the data flow, so that the data export efficiency in the large-data-volume scenario is improved.

Description

Technical Field

[0001] This invention relates to the field of data stream technology, and in particular to a method and apparatus for exporting large amounts of data based on data streams. Background Technology

[0002] Exporting data from a database typically involves first temporarily storing the data in virtual machine memory, then using a backend language to write the data from the virtual machine memory to a generated data file, and finally exporting the data file via a data stream. However, this method is inefficient for large-scale data applications. Therefore, improving the efficiency of data stream-based export in large-scale data scenarios is crucial. Summary of the Invention

[0003] The technical problem that this invention aims to solve is that existing data export methods based on data streams require transition through virtual machine memory, resulting in low efficiency in application scenarios with large data volumes.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a method for exporting large volumes of data based on data streams, comprising:

[0005] Based on the data export requirements, determine the database call instructions corresponding to those requirements;

[0006] Connect to the database according to the database call instruction and obtain the data to be called;

[0007] The data file corresponding to the data to be invoked is obtained through the data stream.

[0008] As an optional implementation, before connecting to the database according to the database call instruction and obtaining the data to be called, the method further includes:

[0009] The database attribute file is determined based on the database type and the database call instruction;

[0010] Call the database property file;

[0011] The step of connecting to the database according to the database call instruction and obtaining the data to be called includes:

[0012] Based on the database property file, a first connection driver is established, and the database is connected using the first connection driver to obtain the data to be called.

[0013] As an optional implementation, the data export requirements include database type, and at least one of the following: data type, data volume, data file type, and data storage location;

[0014] The database call instruction includes a second connection driver and underlying processing commands. Determining the database call instruction corresponding to the data export requirement based on the data export requirement includes:

[0015] Based on the database type, determine the second connection driver and the underlying processing command;

[0016] The step of connecting to the database according to the database call instruction and obtaining the data to be called includes:

[0017] Connect to the database according to the second connection driver;

[0018] According to the underlying processing command, process the data to be invoked and obtain the processed data to be invoked.

[0019] As an optional implementation, after determining the second connection driver and underlying processing commands based on the database type, the method further includes:

[0020] Based on the second connection driver, a first data call encapsulation package is generated;

[0021] Based on the underlying processing commands, a second data call package is generated.

[0022] As an optional implementation, before obtaining the data file corresponding to the data to be invoked via a data stream, the method further includes:

[0023] Based on the data to be invoked, determine the data processing flow;

[0024] According to the data processing flow, the data to be called is processed to obtain the target data;

[0025] Based on the target data, the data to be invoked is updated to obtain the updated data to be invoked; wherein, the updated data to be invoked includes: the data to be invoked before the update, and the target data.

[0026] As an optional implementation, the method further includes:

[0027] Determine at least one preset data processing flow;

[0028] According to the preset data processing flow, the data status and data flow direction are determined;

[0029] Based on the data status and data flow direction, a panoramic dashboard is established; the panoramic dashboard is used to display a panoramic view corresponding to each of the data processing flows.

[0030] Based on the panoramic view, perform panoramic visual analysis.

[0031] Secondly, the present invention provides a large-volume data export system based on data streams, for implementing the large-volume data export method based on data streams disclosed in the first aspect, the system comprising:

[0032] The application layer module is used to initiate data export requests;

[0033] A service layer module is used to connect to the backend layer of the system according to the data export requirements; the backend layer includes at least: a tool layer and / or a data layer;

[0034] The tool layer module is used to determine the database call instruction corresponding to the data export requirement based on the data export requirement;

[0035] And / or, connect to the data layer according to the database call instruction to obtain the data to be called;

[0036] And / or, based on the data to be invoked, obtain the data file corresponding to the data to be invoked through a data stream;

[0037] The data layer module is used to store the data to be invoked.

[0038] Thirdly, the present invention provides a large-volume data export device based on data streams, the device comprising:

[0039] The instruction determination module is used to determine the database call instruction corresponding to the data export requirement based on the data export requirement.

[0040] The data acquisition module is used to connect to the database according to the database call instruction and obtain the data to be called;

[0041] The file acquisition module is used to acquire the data file corresponding to the data to be invoked through a data stream.

[0042] As an optional implementation, the device further includes an attribute file determination module, used before the data acquisition module connects to the database according to the database call instruction and obtains the data to be called.

[0043] The database attribute file is determined based on the database type and the database call instruction;

[0044] Call the database property file;

[0045] The data acquisition module connects to the database according to the database call instruction, and the specific methods for obtaining the data to be called include:

[0046] Based on the database property file, a first connection driver is established, and the database is connected using the first connection driver to obtain the data to be called.

[0047] As an optional implementation, the data export requirements include database type, and at least one of the following: data type, data volume, data file type, and data storage location;

[0048] The database call instruction includes a second connection driver and underlying processing commands. The instruction determination module determines the specific method of the database call instruction corresponding to the data export requirement based on the data export requirement, including:

[0049] Based on the database type, determine the second connection driver and the underlying processing command;

[0050] The data acquisition module connects to the database according to the database call instruction, and the specific methods for obtaining the data to be called include:

[0051] Connect to the database according to the second connection driver;

[0052] According to the underlying processing command, process the data to be invoked and obtain the processed data to be invoked.

[0053] As an optional implementation, the device further includes an encapsulation module, used after the instruction determination module determines the second connection driver and underlying processing commands based on the database type,

[0054] Based on the second connection driver, a first data call encapsulation package is generated;

[0055] Based on the underlying processing commands, a second data call package is generated.

[0056] As an optional implementation, the apparatus further includes a data processing module, configured to, before the file acquisition module acquires the data file corresponding to the data to be invoked via a data stream,

[0057] Based on the data to be invoked, determine the data processing flow;

[0058] According to the data processing flow, the data to be called is processed to obtain the target data;

[0059] Based on the target data, the data to be invoked is updated to obtain the updated data to be invoked; wherein, the updated data to be invoked includes: the data to be invoked before the update, and the target data.

[0060] As an optional implementation, the device further includes a visualization module for:

[0061] Determine at least one preset data processing flow;

[0062] According to the preset data processing flow, the data status and data flow direction are determined;

[0063] Based on the data status and data flow direction, a panoramic dashboard is established; the panoramic dashboard is used to display a panoramic view corresponding to each of the data processing flows.

[0064] Based on the panoramic view, perform panoramic visual analysis.

[0065] A fourth aspect of the present invention discloses another device for exporting large volumes of data based on data streams, the device comprising:

[0066] Memory containing executable program code;

[0067] A processor coupled to the memory;

[0068] The processor calls the executable program code stored in the memory to execute the large-volume data export method based on data stream disclosed in the first aspect of the present invention.

[0069] The fifth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the large-scale data export method based on data stream disclosed in the first aspect of the present invention.

[0070] Compared with existing technologies, the embodiments of the present invention have the following beneficial effects: Data files corresponding to the data to be called are obtained through data streams. Corresponding database call instructions are determined based on data export requirements, the database is operated on, and the required data is exported as data files through data streams, thus improving data export efficiency in scenarios with large data volumes. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a flowchart illustrating a method for exporting large amounts of data based on data streams, as disclosed in Embodiment 1 of the present invention.

[0073] Figure 2 This is a flowchart illustrating a method for exporting large amounts of data based on data streams, as disclosed in Embodiment 2 of the present invention.

[0074] Figure 3This is a schematic diagram of the structure of a large-volume data export system based on data stream disclosed in Embodiment 3 of the present invention;

[0075] Figure 4 This is a schematic diagram of a large-volume data export device based on data stream disclosed in Embodiment 4 of the present invention;

[0076] Figure 5 This is a schematic diagram of another large-scale data export device based on data stream disclosed in Embodiment 4 of the present invention;

[0077] Figure 6 This is a schematic diagram of a large-scale data export device based on data stream disclosed in Embodiment 5 of the present invention. Detailed Implementation

[0078] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0080] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0081] Exporting data from a database typically involves first temporarily storing the data in virtual machine memory, then using a backend language like Java to write the data from the virtual machine memory to a generated data file, and finally exporting the data file via a data stream. However, in applications with large datasets, this method is inefficient, can strain the application server, and results in poor performance, leading to low system efficiency. Therefore, improving the efficiency of data stream-based export in scenarios with large datasets is crucial.

[0082] This invention discloses a method and apparatus for exporting large volumes of data based on data streams. The method acquires data files corresponding to the data to be retrieved through data streams. It determines the corresponding database call instructions based on the data export requirements, operates on the database, and exports the required data as data files through data streams, thus improving the data export efficiency in scenarios with large volumes of data.

[0083] Example 1

[0084] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for exporting large volumes of data based on data streams, as disclosed in Embodiment 1 of the present invention. Figure 1 As shown, the method for exporting large amounts of data based on data streams may include the following operations:

[0085] S101. Based on the data export requirements, determine the database call instruction corresponding to the data export requirements;

[0086] In practical applications, database call commands can be underlying database command-line commands that correspond to data export requirements or database types.

[0087] S102. Connect to the database according to the database call instruction and obtain the data to be called;

[0088] S103. Obtain the data file corresponding to the data to be called through the data stream.

[0089] The data file can be in the form of an Excel spreadsheet or other file formats. In the application scenario of Example 1, the data occupies the server's storage resources but not the virtual machine's memory, and the corresponding file can be directly output to the user's terminal.

[0090] As an optional implementation, before connecting to the database according to the database call instruction and obtaining the data to be called, the method further includes:

[0091] The database attribute file is determined based on the database type and the database call instruction;

[0092] Database types can include MySQL, Oracle, DM, PostgreSQL, etc., and the corresponding database connection driver can be determined based on each database type. The database property file indicates the semantics for connecting to the database. At the appropriate system level, the semantics indicated by the database property file can be translated to generate corresponding database processing instructions and connection drivers.

[0093] Call the database property file;

[0094] The step of connecting to the database according to the database call instruction and obtaining the data to be called includes:

[0095] Based on the database property file, a first connection driver is established, and the database is connected using the first connection driver to obtain the data to be called.

[0096] By using database type and calling instructions, a database attribute file is generated to connect to the corresponding database and obtain the data to be called, which improves the flexibility of data calling in big data scenarios and thus improves the efficiency of data calling.

[0097] This embodiment provides a method for exporting large volumes of data based on data streams. The method includes: determining a database call instruction corresponding to the data export requirements; connecting to a database according to the database call instruction and obtaining the data to be retrieved; and obtaining a data file corresponding to the data to be retrieved through a data stream. By determining the corresponding database call instruction based on the data export requirements, operating the database, and exporting the required data as a data file through a data stream, the efficiency of data export in scenarios with large volumes of data is improved.

[0098] Example 2

[0099] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for exporting large volumes of data based on data streams, as disclosed in Embodiment 2 of the present invention. The data export requirements include database type, and at least one of the following: data type, data volume, data file type, and data storage location; the database call instructions include a second connection driver and underlying processing commands.

[0100] like Figure 2 As shown, based on any other embodiment, the method includes:

[0101] S201. Determine the second connection driver and the underlying processing command according to the database type;

[0102] The underlying processing command is used to instruct the data processing and data retrieval methods corresponding to the database type, and the second connection driver is used to connect to and activate the database of the corresponding type to realize data processing and retrieval.

[0103] As an optional implementation, after determining the second connection driver and underlying processing commands based on the database type, the method further includes:

[0104] Based on the second connection driver, a first data call encapsulation package is generated;

[0105] Based on the underlying processing commands, a second data call package is generated.

[0106] Data call wrappers are used to encapsulate corresponding methods or commands. In scenarios where methods are used repeatedly, calling them in the form of wrappers can improve the convenience and flexibility of the application. In data export application scenarios with large amounts of data and based on data streams, a feasible wrapper format is the JAR wrapper.

[0107] By encapsulating the corresponding functional modules into JAR files, the flexibility of data retrieval in scenarios with large data volumes is improved, thereby increasing the efficiency of data retrieval.

[0108] S202. Connect to the database according to the second connection driver;

[0109] S203. Process the data to be invoked according to the underlying processing command, and obtain the processed data to be invoked.

[0110] S204. Obtain the data file corresponding to the data to be called through the data stream.

[0111] The method provided in Embodiment 2 determines the corresponding database connection driver and underlying processing commands based on the database type, connects to the database according to the second connection driver, and processes the data to be called according to the underlying processing commands. This realizes data interaction corresponding to specific database types, improves the flexibility of data calling in large data scenarios, and thus improves the efficiency of data calling.

[0112] As an optional implementation, before obtaining the data file corresponding to the data to be invoked via a data stream, the method further includes:

[0113] Based on the data to be invoked, determine the data processing flow;

[0114] The data processing flow can be determined based on the characteristics of the data to be retrieved and the purpose of data retrieval, so as to improve the efficiency of data retrieval.

[0115] According to the data processing flow, the data to be called is processed to obtain the target data;

[0116] Based on the target data, the data to be invoked is updated to obtain the updated data to be invoked; wherein, the updated data to be invoked includes: the data to be invoked before the update, and the target data.

[0117] The target data can be statistical indicators or some intermediate states of data processing, depending on the specific data processing flow. Based on the target data, the content of the data to be called can be replaced or added.

[0118] By defining the data processing flow corresponding to the data to be called, preprocessing the data to be called and updating the corresponding data, the data preprocessing is completed before the call, thereby improving the efficiency of data calling.

[0119] As an optional implementation, the method further includes:

[0120] Determine at least one preset data processing flow;

[0121] According to the preset data processing flow, the data status and data flow direction are determined;

[0122] Based on the data status and data flow direction, a panoramic dashboard is established; the panoramic dashboard is used to display a panoramic view corresponding to each of the data processing flows.

[0123] A panoramic dashboard is a template for data visualization, used to indicate the data status and data flow corresponding to each preset data processing flow. Through the panoramic dashboard, the analysis of statistical indicators or changes in data status can be performed.

[0124] Based on the panoramic view, perform panoramic visual analysis.

[0125] By defining the underlying principles through a pre-defined data processing workflow and establishing corresponding data dashboards, the efficiency of data retrieval is improved.

[0126] The large-volume data export method based on data stream provided in this embodiment determines the corresponding database connection driver and underlying processing commands by database type, connects to the database according to the second connection driver, and processes the data to be called according to the underlying processing commands. This realizes data interaction corresponding to specific database types, improves the flexibility of data calling in large-volume scenarios, and thus improves the efficiency of data calling.

[0127] Example 3

[0128] Embodiment 3 of the present invention also provides a large-volume data export system based on data streams to implement the aforementioned method. Please refer to [link to documentation]. Figure 3 , Figure 3 This is a schematic diagram of the structure of a large-volume data export system based on data streams, as disclosed in Embodiment 3 of the present invention. Figure 3 As shown, based on any other embodiment, the system includes:

[0129] Application layer module 31 is used to initiate data export requests;

[0130] The application layer is actually the front-end layer of the system. In specific applications, it can be the HTML layer, which allows users to interact. Users initiate data export requests through the application layer in the form of actions or instructions, and the data export requests are converted into machine instructions that the corresponding data export system can recognize.

[0131] Service layer module 32 is used to connect to the backend layer of the system according to the data export requirements; the backend layer includes at least: a tool layer and / or a data layer; in specific applications, the service layer module may be an API gateway layer.

[0132] Tool layer module 33 is used to determine the database call instruction corresponding to the data export requirement based on the data export requirement;

[0133] And / or, connect to the data layer according to the database call instruction to obtain the data to be called;

[0134] And / or, based on the data to be invoked, obtain the data file corresponding to the data to be invoked through a data stream;

[0135] After the service layer module establishes a backend connection, the tool layer module can be used to implement the methods described in Example 1. Method modules can be invoked by calling pre-packaged packages, or the methods to be applied can be determined based on the corresponding data export requirements. When implementing corresponding methods by calling the packaged package, pre-imported methods can be used instead of waiting for instructions from the service layer module. Pre-calling the packaged package allows the corresponding methods to be executed directly when the service layer module and tool layer module connect, thereby improving the efficiency of data retrieval.

[0136] The data layer module 34 is used to store the data to be invoked.

[0137] The data layer module can be used for data storage, and can store large amounts of data through various types of databases. The data to be called can be part or all of the data stored in the data layer module.

[0138] The application layer module initiates data export requests, the service layer module establishes the connection between the front-end and back-end, and the tool layer module determines the corresponding database call instructions based on the data export requests to operate the data layer module. The data in the data layer module is exported as a data file through a data stream, which improves the data export efficiency in scenarios with large amounts of data.

[0139] Example 4

[0140] Embodiment 4 of the present invention also provides a large-volume data export device based on data streams to implement the aforementioned method. Please refer to [link to related documentation]. Figure 4 , Figure 4 This is a schematic diagram of a large-volume data export device based on data streams, as disclosed in Embodiment 4 of the present invention. Figure 4 As shown, based on any other embodiment, the apparatus includes:

[0141] The instruction determination module 41 is used to determine the database call instruction corresponding to the data export requirement based on the data export requirement.

[0142] Data acquisition module 42 is used to connect to the database according to the database call instruction and obtain the data to be called;

[0143] The file acquisition module 43 is used to acquire the data file corresponding to the data to be invoked through a data stream.

[0144] By determining the corresponding database call instructions based on data export requirements, the database is operated on, and the required data is exported in the form of data files through data streams, thereby improving the data export efficiency in scenarios with large amounts of data.

[0145] Please see Figure 5 , Figure 5 This is a schematic diagram of another large-volume data export device based on data stream disclosed in Embodiment 4 of the present invention. As an optional implementation, the device further includes an attribute file determination module 44, used before the data acquisition module 42 connects to the database according to the database call instruction and obtains the data to be called.

[0146] The database attribute file is determined based on the database type and the database call instruction;

[0147] Call the database property file;

[0148] The data acquisition module connects to the database according to the database call instruction, and the specific methods for obtaining the data to be called include:

[0149] Based on the database property file, a first connection driver is established, and the database is connected using the first connection driver to obtain the data to be called.

[0150] By using database type and calling instructions, a database attribute file is generated to connect to the corresponding database and obtain the data to be called, which improves the flexibility of data calling in big data scenarios and thus improves the efficiency of data calling.

[0151] As an optional implementation, the data export requirements include database type, and at least one of the following: data type, data volume, data file type, and data storage location;

[0152] The database call instruction includes a second connection driver and underlying processing commands. The instruction determination module 41 determines the specific method of the database call instruction corresponding to the data export requirement based on the data export requirement, including:

[0153] Based on the database type, determine the second connection driver and the underlying processing command;

[0154] The data acquisition module 42 connects to the database according to the database call instruction and obtains the specific method of the data to be called, including:

[0155] Connect to the database according to the second connection driver;

[0156] According to the underlying processing command, process the data to be invoked and obtain the processed data to be invoked.

[0157] The system determines the corresponding database connection driver and underlying processing commands based on the database type, connects to the database according to the second connection driver, and processes the data to be called according to the underlying processing commands. This enables data interaction corresponding to specific database types, improves the flexibility of data calling in large data volume scenarios, and thus improves the efficiency of data calling.

[0158] As an optional implementation, the device further includes an encapsulation module 45, used after the instruction determination module 41 determines the second connection driver and underlying processing commands based on the database type,

[0159] Based on the second connection driver, a first data call encapsulation package is generated;

[0160] Based on the underlying processing commands, a second data call package is generated.

[0161] By encapsulating the corresponding functional modules into JAR files, the flexibility of data retrieval in scenarios with large data volumes is improved, thereby increasing the efficiency of data retrieval.

[0162] As an optional implementation, the device further includes a data processing module 46, used to process data before the file acquisition module 43 acquires the data file corresponding to the data to be invoked via a data stream.

[0163] Based on the data to be invoked, determine the data processing flow;

[0164] According to the data processing flow, the data to be called is processed to obtain the target data;

[0165] Based on the target data, the data to be invoked is updated to obtain the updated data to be invoked; wherein, the updated data to be invoked includes: the data to be invoked before the update, and the target data.

[0166] By defining the data processing flow corresponding to the data to be called, preprocessing the data to be called and updating the corresponding data, the data preprocessing is completed before the call, thereby improving the efficiency of data calling.

[0167] As an optional implementation, the device further includes a visualization module 47, used for:

[0168] Determine at least one preset data processing flow;

[0169] According to the preset data processing flow, the data status and data flow direction are determined;

[0170] Based on the data status and data flow direction, a panoramic dashboard is established; the panoramic dashboard is used to display a panoramic view corresponding to each of the data processing flows.

[0171] Based on the panoramic view, perform panoramic visual analysis.

[0172] By defining the underlying principles through a pre-defined data processing workflow and establishing corresponding data dashboards, the efficiency of data retrieval is improved.

[0173] Example 5

[0174] Please see Figure 6 , Figure 6 This is a schematic diagram of a large-volume data export device based on data streams, as disclosed in Embodiment 5 of the present invention. Figure 6 As shown, the data stream-based large-volume data export device may include:

[0175] The device includes a processor 291 and a memory 292 storing executable program code; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 is coupled to the memory 292, and the processor 291 can call logical instructions (executable program code) in the memory 292 to execute the large-scale data export method based on data flow described in any of the above embodiments.

[0176] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0177] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0178] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0179] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when invoked, are used to implement the method described in any of the embodiments.

[0180] This invention also discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the data stream-based large-volume data export method described in any embodiment.

[0181] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0182] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0183] Finally, it should be noted that the data export method and apparatus based on data streams disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for exporting large volumes of data based on data streams, characterized in that, The method is applied to a large-volume data export system based on data streams. The system includes an application layer module, a service layer module, a tool layer module, and a data layer module. The system executes the method when the service layer module and the tool layer module are connected by pre-importing a packaged component. The method includes: The tool layer module determines the database call instruction corresponding to the data export requirement based on the data export requirement. The data export requirement is initiated by the application layer module, and the service layer module connects the tool layer module and / or the data layer module of the system according to the data export requirement. Connect to the database according to the database call instruction and obtain the data to be called; The data file corresponding to the data to be called is obtained through the data stream, and the data in the data layer module is exported in the form of the data file through the data stream; The data export requirements include database type, and at least one of the following: data type, data volume, data file type, and data storage location; The database call instruction includes a second connection driver and underlying processing commands. Determining the database call instruction corresponding to the data export requirement based on the data export requirement includes: Based on the database type, determine the second connection driver and the underlying processing command; The step of connecting to the database according to the database call instruction and obtaining the data to be called includes: Connect to the database according to the second connection driver; According to the underlying processing command, process the data to be invoked, and obtain the processed data to be invoked; Furthermore, after determining the second connection driver and underlying processing commands based on the database type, the method further includes: Based on the second connection driver, a first data call encapsulation package is generated; Based on the underlying processing command, a second data call package is generated; each package in the first and second data call packages is applied in data export application scenarios with large data volumes and based on data streams.

2. The method according to claim 1, characterized in that, Before connecting to the database according to the database call instruction and obtaining the data to be called, the method further includes: The database attribute file is determined based on the database type and the database call instruction; Call the database property file; The step of connecting to the database according to the database call instruction and obtaining the data to be called includes: Based on the database property file, a first connection driver is established, and the database is connected using the first connection driver to obtain the data to be called.

3. The method according to claim 1 or 2, characterized in that, Before obtaining the data file corresponding to the data to be invoked via the data stream, the method further includes: Based on the data to be invoked, determine the data processing flow; According to the data processing flow, the data to be called is processed to obtain the target data; Based on the target data, the data to be invoked is updated to obtain the updated data to be invoked; wherein, the updated data to be invoked includes: the data to be invoked before the update, and the target data.

4. The method according to claim 3, characterized in that, The method further includes: Determine at least one preset data processing flow; According to the preset data processing flow, the data status and data flow direction are determined; Based on the data status and data flow direction, a panoramic dashboard is established; the panoramic dashboard is used to display a panoramic view corresponding to each of the data processing flows. Based on the panoramic view, perform panoramic visual analysis.

5. A data stream-based system for exporting large volumes of data, used to implement the data stream-based method for exporting large volumes of data according to any one of claims 1-4, characterized in that, The system includes: The application layer module is used to initiate data export requests; A service layer module is used to connect to the backend layer of the system according to the data export requirements; the backend layer includes at least: a tool layer and / or a data layer; The tool layer module is used to determine the database call instruction corresponding to the data export requirement based on the data export requirement; And / or, connect to the data layer according to the database call instruction to obtain the data to be called; And / or, based on the data to be invoked, obtain the data file corresponding to the data to be invoked through a data stream; The data layer module is used to store the data to be invoked.

6. A device for exporting large volumes of data based on data streams, characterized in that, The apparatus is used to perform the large-volume data export method based on data streams as described in any one of claims 1-4, the apparatus comprising: The instruction determination module is used to determine the database call instruction corresponding to the data export requirement based on the data export requirement. The data acquisition module is used to connect to the database according to the database call instruction and obtain the data to be called; The file acquisition module is used to acquire the data file corresponding to the data to be invoked through a data stream.

7. A device for exporting large volumes of data based on data streams, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the large-volume data export method based on data stream as described in any one of claims 1-4.

8. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the large-scale data export method based on data streams as described in any one of claims 1-4.

Images