Method and device for quickly generating stream processing synchronization task and storage medium

Abstract

The invention discloses a method and equipment for quickly generating a stream processing synchronization task and a storage medium, and relates to the technical field of real-time data bins. The method comprises the steps of obtaining corresponding database link information based on preset database demand data to obtain database link data; obtaining to-be-synchronized data according to a preset streaming task configuration requirement; determining a configuration file storage position based on a preset configuration requirement; and obtaining a streaming task configuration file according to the to-be-synchronized data and the database link data, and storing the streaming task configuration file to the configuration file storage position. According to the technical scheme, the technical problems that the real-time warehouse counting construction difficulty is large, the consumed time is long, and wide application is difficult are solved.

Description

[0001] This application claims priority based on the invention patent application titled "A Method, Device, and Storage Medium for Rapidly Generating Stream Processing Synchronization Tasks" with the application number 202311872289.1, which was filed with the China National Intellectual Property Administration on December 29, 2023. This application incorporates the entire text of the above-mentioned Chinese patent application by reference. Technical Field

[0002] This application relates to the field of real-time data warehouse technology, and particularly to a method, device, and storage medium for rapidly generating stream processing synchronization tasks. Background Art

[0003] In the construction of a real-time data warehouse, data needed is first obtained from various data sources, which are diverse, including business data, buried point logs, etc. The most important data is that in the business database. However, due to the wide variety of business databases and the extremely large number of data tables, the construction is difficult, time-consuming, and it is hard to be widely used. Summary of the Invention

[0004] Embodiments of this application provide a method, device, and storage medium for rapidly generating stream processing synchronization tasks, which solve the technical problems of difficult construction, long time consumption, and hard wide use in the construction of a real-time data warehouse.

[0005] In a first aspect, embodiments of this application provide a method for rapidly generating stream processing synchronization tasks. The method includes: obtaining corresponding database connection information based on preset database requirement data to obtain database connection data; obtaining data to be synchronized according to preset streaming task configuration requirements; determining the storage location of the configuration file based on preset configuration requirements; obtaining a streaming task configuration file according to the data to be synchronized and the database connection data, and saving the streaming task configuration file to the storage location of the configuration file.

[0006] In an implementation manner of this application, obtaining corresponding database connection information based on preset database requirement data to obtain database connection data specifically includes: determining a server address based on preset database requirement data to obtain server address data; determining a corresponding port number according to the server address data; determining a database name based on the port number to obtain database information; verifying whether the database information is obtained according to the username and password, and obtaining database connection data according to the verification result; where the database connection data includes at least one of a server address, a port number, a database name, a username, and a password.

[0007] In this application, by determining key information data, obtaining database open permissions, and obtaining database connection data, the data basis for synchronization tasks is realized.

[0008] In an implementation manner of the present application, according to the database connection data, by determining the configuration requirements of the streaming task, the data to be synchronized is obtained, specifically including: according to the database connection data, determining that the configuration requirements of the streaming task are to synchronize the database, selecting all tables in the database to obtain the data of the database to be synchronized; determining that the configuration requirements of the streaming task are to synchronize specific tables, selecting the corresponding tables in the database to obtain the data of the tables to be synchronized; based on the data of the database to be synchronized or the data of the tables to be synchronized, obtaining the data to be synchronized; wherein, the data to be synchronized includes: the data of the tables to be synchronized and / or the data of the database to be synchronized.

[0009] In the present application, by determining the object to be synchronized according to the requirements, the scope and target of data synchronization can be clarified, ensuring that only necessary data is synchronized; it helps to improve the efficiency and accuracy of data synchronization and reduce unnecessary data transmission and processing.

[0010] In an implementation manner of the present application, based on the preset configuration requirements, by security and accessibility screening, the storage location of the configuration file is determined, specifically including: based on the preset configuration requirements, determining the data of the storage locations to be screened; according to the data of the storage locations to be screened, determining the storage locations that meet the preset requirements in the data of the storage locations to be screened; based on the storage locations, determining the storage location of the configuration file.

[0011] In the present application, by determining the storage location of the configuration file, the usage requirements and working environment are met.

[0012] In an implementation manner of the present application, according to the data to be synchronized and the storage location of the configuration file, by starting the synchronization task, the streaming task configuration file is obtained, specifically including: according to the data to be synchronized, performing the synchronization task to obtain the data to be stored in the configuration file; based on the storage location of the configuration file, storing the data to be stored in the configuration file to obtain the streaming task configuration file.

[0013] In an implementation manner of the present application, before obtaining the database connection data by obtaining the corresponding database connection information based on the preset database requirement data, the method further includes: determining the database type according to the preset database type requirement; based on the database type, determining the corresponding synchronization tool and settings.

[0014] In an implementation manner of the present application, after obtaining the streaming task configuration file, the method further includes: generating the corresponding download link based on the streaming task configuration file to obtain the download data; directly obtaining the streaming task configuration file according to the download data.

[0015] In an implementation manner of the present application, before generating the corresponding download link, the method further includes: determining the storage location of the configuration file to obtain the streaming task configuration file; sending the streaming task configuration file to the preset address.

[0016] In this application, through the subsequent acquisition method of the configured task profile, it is convenient for data docking and the extraction of the configuration file.

[0017] In a second aspect, an embodiment of the present application further provides a device for quickly generating a stream processing synchronization task. The device includes: a link data module, configured to obtain corresponding database link information based on preset database requirement data to obtain database link data; a synchronization data module, configured to obtain data to be synchronized according to preset streaming task configuration requirements; a storage location module, configured to determine the storage location of the configuration file based on preset configuration requirements; and a configuration file module, configured to obtain a streaming task configuration file according to the data to be synchronized and the database link data, and save the streaming task configuration file to the storage location of the configuration file.

[0018] In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method for quickly generating a stream processing synchronization task as described in any one of the above.

[0019] In a fourth aspect, an embodiment of the present application further provides a non-volatile computer storage medium, storing computer-executable instructions, which when executed by a processor implement the method for quickly generating a stream processing synchronization task as described in any one of the above.

[0020] An embodiment of the present application provides a method, a device, and a storage medium for quickly generating a stream processing synchronization task. Through the setting of the configuration file, it solves the technical problems of difficult construction, long time consumption, and difficult widespread use of the real-time data warehouse; and has the technical effects of fast synchronization speed, easy query of the configuration file, and reduction of the difficulty of building the real-time data warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings:

[0022] Figure 1 is a flowchart of a method for quickly generating a stream processing synchronization task provided by an embodiment of the present application;

[0023] Figure 2 is a specific flowchart of a method for quickly generating a stream processing synchronization task provided by an embodiment of the present application;

[0024] Figure 3 is a schematic internal structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments of this application and the corresponding drawings. Apparently, the described embodiments are only a part rather than all of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope of protection of this application.

[0026] An embodiment of this application provides a method, device, and storage medium for quickly generating a stream processing synchronization task. By setting a configuration file, it solves the technical problems of high difficulty, long time consumption, and difficult widespread use in the construction of a real-time data warehouse; it has the technical effects of fast synchronization speed, easy query of the configuration file, and reduced difficulty in building a real-time data warehouse.

[0027] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the drawings.

[0028] Figure 1 It is a flowchart of a method for quickly generating a stream processing synchronization task provided by an embodiment of this application. As Figure 1 shown, a method for quickly generating a stream processing synchronization task provided by an embodiment of this application specifically includes the following steps:

[0029] Step 101: Based on preset database requirement data, obtain the corresponding database connection information to obtain database connection data.

[0030] Among them, the database connection data includes: server address, port number, database name, username, and password.

[0031] Specifically, it includes: based on the preset database requirement data, determine the server address to obtain server address data; according to the server address data, determine the corresponding port number; based on the port number, determine the database name to obtain database information; according to the username and password, verify whether the database information is obtained, and obtain the database connection data according to the verification result.

[0032] Before obtaining the corresponding database connection information based on the preset database requirement data to obtain the database connection data, the method further includes: determining the database type according to the preset database type requirement; based on the database type, determining the corresponding synchronization tool and settings.

[0033] In this application, by determining the key information data, obtaining the database open permission, and obtaining the database connection data, the data basis for the synchronization task is realized.

[0034] In the embodiment of this application, it is explained in detail by the following Example 1:

[0035] Example 1: When the user selects the database type, a selection interface is provided on the front-end interface, which contains various required database type options.

[0036] The user currently needs to select the MySQL database. Select the option of "MySQL" on the selection interface, indicating that the user chooses to synchronize to the MySQL database.

[0037] Then, the system will perform corresponding synchronization tool configuration and settings according to the characteristics of the MySQL database. The synchronization tools include but are not limited to MySQL Workbench, phpMyAdmin, and Navicat for MySQL.

[0038] According to the actual requirements, determine the server address. An input interface is provided on the front-end interface, and the address can be input. According to the server address data, determine the corresponding port number. Similarly, an input interface is provided on the front-end interface (the input process will not be described in detail later), determine the database name, and obtain the database information. Finally, verify whether the database information can be obtained according to the username and password to ensure information security.

[0039] Under the condition that the username and password are correct, obtain the database connection data.

[0040] Step 102: Obtain the data to be synchronized according to the preset streaming task configuration requirements.

[0041] The data to be synchronized includes: data of tables to be synchronized, data of libraries to be synchronized.

[0042] Specifically, it includes: according to the database connection data, determine that the streaming task configuration requirement is to synchronize the database, select all tables in the database, and obtain the data of the library to be synchronized; determine that the streaming task configuration requirement is to synchronize specific tables, select the corresponding tables in the database, and obtain the data of the tables to be synchronized; based on the data of the library to be synchronized or the data of the tables to be synchronized, obtain the data to be synchronized.

[0043] In this application, according to the requirements, determine the object to be synchronized, which can clarify the scope and target of data synchronization and ensure that only necessary data is synchronized. This helps to improve the efficiency and accuracy of data synchronization and reduce unnecessary data transmission and processing.

[0044] In the embodiment of this application, it is explained in detail through the following Example 2:

[0045] Example 2: When the user selects to synchronize the data of interest, query the corresponding data information, as follows.

[0046] 1. Synchronize all data in the entire database. The synchronization task will cover all tables in the database and transfer all data records, which is applicable to the situation of quickly setting up and transferring the entire database.

[0047] 2. If the user is only interested in certain tables in the database, these specific tables can be selected for synchronization. The user can select the corresponding table names or use checkboxes to select multiple tables. The synchronization task will only involve the selected tables and transfer the specified data records; this is more accurate than synchronizing the entire database and can reduce unnecessary data transfer and synchronization time.

[0048] When selecting the tables or databases to be synchronized, the importance and real-time requirements of the data need to be considered to ensure that the synchronization task meets the actual needs.

[0049] Step 103: Determine the storage location of the configuration file based on the preset configuration requirements;

[0050] Specifically, it includes: determining the data to be screened for storage location based on the preset configuration requirements; through security and accessibility screening according to the data to be screened for storage location, determining the storage locations that meet the preset requirements in the data to be screened for storage location; and determining the storage location of the configuration file based on the storage location.

[0051] In this application, by determining the storage location of the configuration file, the usage requirements and working environment are met.

[0052] In the embodiments of this application, it is explained in detail through the following Example 3:

[0053] Example 3: When selecting the storage location of the configuration file, the following properties need to be considered:

[0054] Security, accessibility, and subsequent backup situations.

[0055] Security is to consider that the configuration file may contain sensitive information, such as database connection credentials, etc.; therefore, it is necessary to ensure that the configuration file is stored in a secure location to prevent unauthorized access.

[0056] Accessibility ensures that users can easily access the configuration file. If the configuration file is stored in a remote location, the user needs to ensure a stable network connection and sufficient permissions to access the file.

[0057] Step 104: Obtain a streaming task configuration file based on the data to be synchronized and the database link data, and save the streaming task configuration file to the storage location of the configuration file.

[0058] Specifically, it includes: performing a synchronization task based on the data to be synchronized to obtain the file to be stored in the configuration file; based on the storage location of the configuration file, storing the file to be stored in the configuration file to obtain a streaming task configuration file.

[0059] After obtaining the streaming task configuration file, the method further includes: generating a corresponding download link based on the streaming task configuration file to obtain download data; directly obtaining the streaming task configuration file according to the download data.

[0060] Before generating the corresponding download link, the method further includes: determining the storage location of the configuration file to obtain the streaming task configuration file; sending the streaming task configuration file to a preset address.

[0061] In this application, by setting the subsequent acquisition method of the streaming task configuration file, it is convenient for data docking and extraction of the configuration file.

[0062] The system will automatically download the generated streaming task configuration file to the location specified by the user or directly send it to a predetermined address. The streaming task configuration file contains all the settings and information required for synchronization tasks, including database connection information, table structure, data conversion rules, etc.

[0063] In future use, the synchronization task can be directly started or managed through the streaming task configuration file. The streaming task configuration file is crucial for ensuring the consistency and accuracy of data synchronization and needs to be properly stored and updated regularly.

[0064] The above is the method embodiment proposed in this application. Based on the same inventive concept, the embodiment of this application also provides a device for quickly generating a stream processing synchronization task, and its structure is as Figure 2 shown.

[0065] Figure 3 It is a schematic internal structure diagram of an electronic device provided by the embodiment of this application. As Figure 2 shown, the device includes:

[0066] At least one processor 301;

[0067] And a memory 302 communicatively connected to the at least one processor;

[0068] Wherein, the memory 302 stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can:

[0069] Based on the preset database requirement data, obtain the corresponding database link information to obtain database link data; obtain the data to be synchronized according to the preset streaming task configuration requirements; determine the storage location of the configuration file based on the preset configuration requirements; obtain the streaming task configuration file according to the data to be synchronized and the database link data, and save the streaming task configuration file to the storage location of the configuration file.

[0070] In other embodiments of the present application, when the instructions are executed by at least one processor 201, the steps of the method for quickly generating a stream processing synchronization task in any of the above embodiments may also be implemented.

[0071] Some embodiments of the present application provide a non-volatile computer storage medium corresponding to Figure 1 , storing computer-executable instructions, and the computer-executable instructions are set as follows:

[0072] Based on the preset database requirement data, obtain the corresponding database link information to obtain database link data; obtain the data to be synchronized according to the preset streaming task configuration requirements; determine the storage location of the configuration file based on the preset configuration requirements; obtain the streaming task configuration file according to the data to be synchronized and the database link data, and save the streaming task configuration file to the storage location of the configuration file.

[0073] In other embodiments of the present application, the executable instructions may also implement the steps of the method for quickly generating a stream processing synchronization task in any of the above embodiments.

[0074] Each embodiment in the present application is described in a progressive manner. The same or similar parts among the embodiments may be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of the Internet of Things devices and media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiments.

[0075] The systems and media provided by the embodiments of the present application correspond one-to-one with the methods. Therefore, the systems and media also have beneficial technical effects similar to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be elaborated here.

[0076] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and combinations of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0078] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0080] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0081] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0082] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0083] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0084] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for quickly generating stream processing synchronization tasks, characterized in that, The method includes: Based on the preset database requirement data, obtain the corresponding database link information to get database link data; According to the preset streaming task configuration requirements, obtain the data to be synchronized; Based on the preset configuration requirements, determine the storage location of the configuration file; According to the data to be synchronized and the database link data, obtain a streaming task configuration file, and save the streaming task configuration file to the storage location of the configuration file.

2. The method for quickly generating a stream processing synchronization task according to claim 1, wherein The database link data includes at least one of: server address, port number, database name, username, and password; Based on the preset database requirement data, obtaining the corresponding database link information to get database link data specifically includes: Based on the preset database requirement data, determine the server address to get server address data; According to the server address data, determine the corresponding port number; Based on the port number, determine the database name to obtain database information; According to the username and the password, verify whether the database information is obtained, and obtain the database link data according to the verification result.

3. A method for quickly generating a stream processing synchronization task according to claim 1, characterized in that, According to the database link data, by determining the streaming task configuration requirements, obtaining the data to be synchronized specifically includes: According to the database link data, determine that the streaming task configuration requirement is to synchronize the database, select all tables in the database to obtain the data of the database to be synchronized; Determine that the streaming task configuration requirement is to synchronize specific tables, select the corresponding tables in the database to obtain the data of the tables to be synchronized; Based on the data of the database to be synchronized or the data of the tables to be synchronized, obtain the data to be synchronized; wherein, the data to be synchronized includes: data of the tables to be synchronized and / or data of the database to be synchronized.

4. A method for quickly generating a stream processing synchronization task according to claim 1, characterized in that, Based on the preset configuration requirements, determining the storage location of the configuration file specifically includes: Based on the preset configuration requirements, determine the data of the storage location to be filtered; According to the data of the storage location to be filtered, through security and accessibility filtering, determine the storage location that meets the preset requirements in the data of the storage location to be filtered; Based on the storage location, determine the storage location of the configuration file.

5. A method for quickly generating a stream processing synchronization task according to claim 1, characterized in that, According to the data to be synchronized and the storage location of the configuration file, by starting the synchronization task, obtaining the streaming task configuration file specifically includes: According to the data to be synchronized, perform the synchronization task to obtain the data to be stored in the configuration file; Based on the storage location of the configuration file, store the data to be stored in the configuration file to obtain the streaming task configuration file.

6. A method for quickly generating a stream processing synchronization task according to claim 1, characterized in that Before obtaining the corresponding database link information based on the preset database requirement data to get database link data, the method further includes: According to the preset database type requirement, determine the database type; Based on the database type, determine the corresponding synchronization tool and settings.

7. A method for quickly generating a stream processing synchronization task according to claim 1, characterized in that After obtaining the streaming task configuration file, the method further includes: Based on the streaming task configuration file, generate a corresponding download link to get download data; According to the download data, directly obtain the streaming task configuration file.

8. A method for quickly generating a stream processing synchronization task according to claim 7, characterized in that Before generating the corresponding download link, the method further includes: Determine the storage location of the configuration file to obtain the streaming task configuration file; Send the streaming task profile to a preset address.

9. A method for quickly generating a stream processing synchronization task according to claim 1, characterized in that After saving the streaming task profile to the profile storage location, the method further includes: Execute the streaming task corresponding to the streaming task profile based on the streaming task profile.

10. A device for quickly generating stream processing synchronization tasks, characterized in that, The device includes: A link data module, configured to obtain corresponding database link information based on preset database requirement data to obtain database link data; A synchronization data module, configured to obtain data to be synchronized according to preset streaming task configuration requirements; A storage location module, configured to determine a profile storage location based on preset configuration requirements; A profile module, configured to obtain a streaming task profile according to the data to be synchronized and the database link data, and save the streaming task profile to the profile storage location.

11. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for quickly generating a stream processing synchronization task according to any one of claims 1 to 9.

12. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the method for quickly generating a stream processing synchronization task according to any one of claims 1 to 9.

Images