Data assembly method and related equipment

Abstract

The present application provides a data assembly method and related equipment, which are applied to a data assembly system. The data assembly system includes a SQL Server database with a configuration library, multiple SQL Server servers, a HBase cluster system, and an HBase database set in the HBase cluster system. The HBase cluster system configures a data assembly program and a configuration file. The method includes: reading the configuration file through the data assembly program to obtain data range information; performing data query through the configuration library according to the data range information to obtain row storage data query information; processing the row storage data query information using the data assembly program to obtain column storage data information; and loading the column storage data information through the HBase database. The required data is queried and assembled at one time, and the program only needs to be run once, which can better realize the loading of the specified range of data from the SQL Server database into the HBase database.

Description

Technical Field

[0001] The present application relates to the technical field of data assembly, and in particular to a data assembly method and related equipment. Background Art

[0002] Relational Database Management System stores and manages data through a data model consisting of data, relationships and constraints on data. In a non-shared database cluster, each node in a massively parallel processing system has an independent disk storage system and memory system. Business data is divided into various nodes according to the database model and application characteristics. Each data node is connected to each other through a dedicated network or a commercial general network, and they work together to provide database services as a whole. SQL Server (Structured Query Language Server) is a relational database management system launched by a network development company. It stores data in rows, while HBase is a distributed, column-oriented open source database. It is a massively parallel processing system suitable for databases that store unstructured data and stores data based on a column-based model. Currently, there is no better way to load a specified range of data directly from SQL Server into HBase.

[0003] Based on the above situation, in the prior art, the SQL Server is executed through the SSIS package (Microsoft SQL Server Integration Services) to retrieve the corresponding data, write the file in a fixed format, transfer the file to the HBase cluster server through SFTP (SSH File Transfer Protocol, secure file transfer protocol), and write the file content into HBase through the corresponding loading program. Multiple tables need to run the above steps multiple times, which makes the existing small amount of specified data extraction process long and has many steps, and the data of multiple tables need to be loaded multiple times, which is time-consuming and labor-intensive. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a data assembly method and related equipment to solve the above technical problems.

[0005] Based on the above purpose, the first aspect of the present application provides a data assembly method, which is applied to a data assembly system, wherein the data assembly system includes a SQL Server database for setting a configuration library, multiple SQL Server servers, a HBase cluster system, and an HBase database set in the HBase cluster system, and the HBase cluster system is configured with a data assembly program and a configuration file;

[0006] The method comprises:

[0007] The configuration file is read by the data assembly program to obtain data range information;

[0008] Performing data query through the configuration library according to the data range information to obtain row storage data query information;

[0009] Using the data assembly program to process the row storage data query information to obtain column storage data information;

[0010] The column storage data information is loaded through the HBase database.

[0011] A second aspect of the present application provides a data assembly system, the data assembly system comprising a SQL Server database for setting a configuration library, multiple SQL Server servers, a HBase cluster system and an HBase database set in the HBase cluster system, the HBase cluster system configuring a data assembly program and a configuration file;

[0012] A reading module is configured to read the configuration file through the data assembly program to obtain data range information;

[0013] A query module, configured to perform data query through the configuration library according to the data range information to obtain row storage data query information;

[0014] A conversion processing module is configured to process the row storage data query information using the data assembly program to obtain column storage data information;

[0015] The loading module is configured to load the column storage data information through the HBase database.

[0016] A third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0017] A fourth aspect of the present application provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0018] From the above, it can be seen that the data assembly method and related equipment provided by the present application read the configuration file through the data assembly program, read the information of the data to be assembled, specify the range of the data to be assembled, obtain the data range information, and then query the data based on the data range information through the configuration library to obtain the row storage data query information. Through concurrent query, the required data is queried and spliced ​​at one time, and only one run is required, which simplifies the extraction process and steps of the data to be assembled. Then, the row storage data query information is processed according to the data assembly program to obtain the column storage data information, and the column storage data information is loaded by the HBase database, so as to better realize the loading of the specified range of data to be assembled from the SQL Server database into the HBase database. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flowchart of a data assembly method according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a data assembly method according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the structure of a data assembly system according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0025] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0026] In the related art, SQL Server is generally executed through SSIS packages to retrieve corresponding data, write files in a fixed format, transfer the files to the HBase cluster server through SFTP, and write the file contents into HBase through the corresponding loading program. For example, due to the mechanism of province and database division, in order to extract partial data of 36 provinces, 3 systems, and 2 tables in each system, it is necessary to run the SSIS package 36*3*2=216 times, generate 216 compressed data files for step-by-step loading, so the extraction process for the specified data is long and has many steps, and the data of multiple tables need to be loaded multiple times, which is time-consuming and laborious.

[0027] An embodiment of the present application provides a data assembly method, which reads a configuration file through a data assembly program, reads information about data that needs to be assembled, specifies the range of data that needs to be assembled, obtains data range information, uses the configuration file to store the association between various data tables in the corresponding storage area, and all requirements can be adaptively and automatically expanded through the configuration file, and then performs data query based on the data range information through the configuration library to obtain row storage data query information, and through concurrent query, the required data is queried and assembled at one time, and only needs to run the program once, simplifying the extraction process and steps of the required data, and then processes the row storage data query information according to the data assembly program to obtain column storage data information, and the HBase database loads the column storage data information, thereby better realizing the loading of the specified range of data that needs to be assembled from the SQL Server database into the HBase database.

[0028] like Figure 1As shown, the method of this embodiment is applied to a data assembly system, the data assembly system includes a SQL Server database for setting a configuration library, multiple SQL Server servers, a HBase cluster system and an HBase database set in the HBase cluster system, and the HBase cluster system is configured with a data assembly program and a configuration file; the method includes:

[0029] Step 101, reading the configuration file through the data assembly program to obtain data range information.

[0030] In this step, the data assembly program and configuration file are pre-set, and the set data assembly program and configuration file are set in the Hadoop cluster system (Hadoop cluster), so that the configuration file can be read by the data assembly program, and the information of the data to be assembled is read to specify the range of the data to be assembled, and the data range information is obtained. Based on the data range information, the data assembly program is used to identify the needs, wherein the configuration file can be adaptively and automatically expanded during use to meet the different needs of data assembly. The configuration file can be modified to set the data range and related logic required for data assembly each time, thereby realizing fully configured use. For newly added information, the modified information can be adapted by changing the configuration file.

[0031] Step 102: perform data query through the configuration library according to the data range information to obtain row storage data query information.

[0032] In this step, the configuration library retrieves the server address of the SQL Server database to be queried according to the data range information, runs once according to the server address of the SQL Server database to be queried, initiates a concurrent query, and executes a query splicing according to the data range information through concurrent query to obtain row-stored data query information, wherein the row-stored data query information refers to all row-stored data information retrieved from the SQL Server database, which greatly improves the efficiency of extracting batch data within the specified data range information and improves the timeliness to seconds.

[0033] Step 103: Use the data assembly program to process the row-stored data query information to obtain column-stored data information.

[0034] In this step, the data assembly program is used to process the row-stored data query information, and the parameters passed into the data assembly program when the data assembly program is executed are obtained based on the row-stored data query information. The parameters passed into the data assembly program are matched with the parameter names obtained from the configuration file to realize data query and assembly requirements, and the row-stored data query information is converted into column-stored data information, where column-stored data information refers to data information stored by column.

[0035] Step 104: Load the column storage data information through the HBase database.

[0036] In this step, the generated write instruction for storing the column storage data information is controlled to load the column storage data information into the HBase database or not to load it temporarily.

[0037] Among them, the HBase database adopts column storage, and the row key (rowkey) represents a unique row. HBase performs retrieval based on the row key, finds the region (Region) where a row key (or a row key range) is located, and then routes the query data request to the region to obtain the data. It provides the row key and the required table name, and can directly generate the write instruction of the storage column storage data information of the corresponding data. By controlling the write instruction, the function of importing the data stored in the row of the corresponding SQL Server database into the column storage of the HBase database is realized, which improves the loading efficiency of the data assembled in the specified batch.

[0038] For example: the SQL Server database is divided into multiple SQL Server sub-databases and storage areas of multiple SQL Server sub-databases. The row key of the HBase database is designed to be the unique number of the insurance policy plus the organization number of the SQL Server sub-database and the code of the storage system. The configuration file is used to store the association relationship between the corresponding storage areas. The HBase database and the Haidup cluster system use the unique number of the insurance policy as the association condition to find out all row storage data information and convert it into column storage data information and store it in the write instruction.

[0039] Through the above scheme, the configuration file is read by the data assembly program to obtain the data range information, and the configuration file is used to store the association relationship of each data table in the corresponding storage area. All requirements can be adaptively and automatically expanded through the configuration file, and then the data query is performed based on the data range information through the configuration library to obtain the row storage data query information, and then the row storage data query information is processed according to the data assembly program to obtain the column storage data information, and a run is performed according to the server address of the SQL Server database to be queried, and a concurrent query is initiated. Through the concurrent query method, a query splicing is performed according to the data range information to obtain the row storage data query information, which greatly improves the efficiency of extracting batch data within the specified data range information and improves the timeliness to the second level. The column storage data information is loaded by the HBase database, and the parameters passed into the data assembly program when the data assembly program is executed are obtained based on the row storage data query information. The parameters passed into the data assembly program are matched with the parameter names obtained from the configuration file to realize data query and assembly requirements, and the row storage data query information is converted into column storage data information, so as to better realize the loading of the specified range of data from the SQL Server database into the HBase database.

[0040] In some embodiments, step 101 includes:

[0041] Step 1011, setting data information of target data for data assembly in the configuration file.

[0042] Step 1012, read each of the data information through the data assembly program to obtain data range information.

[0043] In the above scheme, the target data refers to the data that needs to be loaded, the configuration file sets the parameter name of the target data, and the data assembly program reads the parameter name of each target data to obtain the data range information to specify the range of data that needs to be assembled. The data range information refers to the information of all data to be retrieved for data assembly.

[0044] In some embodiments, the SQL Server database includes a plurality of SQL Server sub-databases;

[0045] Step 102 includes:

[0046] Step 1021, matching is performed through the configuration library according to the data range information to obtain the name of the SQL Server sub-database to be queried.

[0047] Step 1022: retrieve the stored SQL Server sub-database server address based on the name of the SQL Server sub-database through the configuration library, and query the corresponding SQL Server sub-database according to the SQL Server sub-database server address to obtain the data query information.

[0048] In the above scheme, the SQL Server database is divided into multiple SQL Server sub-databases, each SQL Server sub-database corresponds to a server, the configuration library stores the server addresses of each SQL Server sub-database, and the name of the corresponding SQL Server sub-database is obtained according to the data range information. Then, the server address of the corresponding SQL Server sub-database is retrieved by the configuration library through the name of the SQL Server sub-database, and the query is retrieved according to the retrieved server address of the SQL Server sub-database. Concurrent query is used to realize that only one SQL Server database name link is opened once for one operation to obtain data query information. The data query information refers to the row storage data query information obtained by querying and retrieval from the SQL Server sub-database, which realizes the automatic splicing of batch data of multiple SQL Server sub-databases in one operation, without the need for multiple scheduling of SQL Server sub-databases and the need for cross-server transmission through files.

[0049] In some embodiments, step 103 includes:

[0050] The data assembly program is used to perform data assembly processing on each row-stored data query information, and the row-stored data query information is converted into column-stored data information.

[0051] In the above scheme, a data assembling program is used to perform data assembling processing on the row-stored data query information. The parameters passed into the data assembling program when the data assembling program is executed are obtained based on the row-stored data query information. The parameters passed into the data assembling program are matched with the parameter names obtained from the configuration file to realize data query and assembly requirements, and the row-stored data query information is converted into column-stored data information.

[0052] In some embodiments, the column storage data information is stored as a write instruction file;

[0053] Step 104 includes:

[0054] In response to determining to execute the write instruction file, the write instruction file is read, the column storage data information is read, and the column storage data information is written into the HBase database.

[0055] In the above scheme, whether to write to the HBase database is controlled by writing an instruction file. When the writing instruction file is executed, the column storage data information in the writing instruction file is loaded and written into the HBase database. When the writing instruction file is not executed, the loading and writing of the column storage data information in the HBase database is not executed.

[0056] In some embodiments, the configuration repository stores a data table, and the data table stores the server address of the SQL Server sub-database.

[0057] In the above solution, by configuring the server address of the SQL Server sub-database stored in the library, concurrent queries can be performed, so that batch data of multiple SQL Server sub-databases can be automatically spliced ​​into a single operation, without the need to perform multiple scheduling and cross-server transmission through intermediate files.

[0058] In some embodiments, the SQL Server sub-database is stored in multiple storage areas, each of which corresponds to a different data table.

[0059] In the above solution, the configuration library contains SQL Server data tables, which store the server addresses of SQL Server sub-databases. The structures of data tables in different storage areas are designed differently. Through concurrent query of the configuration library, only one SQL Server database link needs to be opened for one run to obtain complex data associated with multiple data tables. There is no need to perform multiple scheduling on the SQL Server sub-databases or to transfer files across servers.

[0060] In some embodiments, Figure 2As shown, the data assembly program and the configuration file are configured in the Hadoop cluster system (Hadoop cluster), the data assembly program reads the data range (i.e., data range information) from the configuration file, the required link library name (i.e., the name of the SQL Server sub-database) is passed in based on the data range, the corresponding database address (i.e., the SQL Server sub-database server address) is retrieved through the configuration library according to the required link library name, and data is queried for server1, server2, server36, etc. according to the database address to obtain data query information, wherein server1, server2, and server36 all refer to SQL Server sub-databases, the data query information is processed by the data assembly program to obtain column storage data information, the column storage data information is written into the hbase shell command (i.e., the write instruction file), and the hbase shell command is controlled to realize whether the HBase database loads the column storage data information. When the hbase shell command is executed, the column storage data information is loaded and written into the HBase database, and when the hbase shell command is not executed, the column storage data information is not loaded and written into the HBase database.

[0061] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0062] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0063] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a data assembly system.

[0064] refer to Figure 3The data assembly system includes a SQL Server database for setting a configuration library, multiple SQL Server servers, a Haidup cluster system, and an HBase database set in the Haidup cluster system. The Haidup cluster system is configured with a data assembly program and a configuration file;

[0065] The reading module 301 is configured to read the configuration file through the data assembly program to obtain data range information;

[0066] A query module 302 is configured to perform data query through the configuration library according to the data range information to obtain row storage data query information;

[0067] The conversion processing module 303 is configured to process the row storage data query information using the data assembly program to obtain column storage data information;

[0068] The loading module 304 is configured to load the column storage data information through the HBase database.

[0069] In some embodiments, the reading module 301 is specifically configured as follows:

[0070] Data information of target data for data assembly set in the configuration file;

[0071] Each piece of data information is read through the data assembly program to obtain data range information.

[0072] In some embodiments, the SQL Server database includes a plurality of SQL Server sub-databases;

[0073] The query module 302 is specifically configured to:

[0074] According to the data range information, a match is performed through the configuration library to obtain the name of the SQL Server sub-database to be queried;

[0075] The configuration library retrieves the stored SQL Server sub-database server address based on the name of the SQL Server sub-database, and queries the corresponding SQL Server sub-database according to the SQL Server sub-database server address to obtain the data query information.

[0076] In some embodiments, the conversion processing module 303 is specifically configured to:

[0077] The data assembly program is used to perform data assembly processing on each row-stored data query information, and the row-stored data query information is converted into column-stored data information.

[0078] In some embodiments, the column storage data information is stored as a write instruction file;

[0079] The loading module 304 is specifically configured as follows:

[0080] In response to determining to execute the write instruction file, the write instruction file is read, the column storage data information is read, and the column storage data information is written into the HBase database.

[0081] In some embodiments, the configuration repository stores a data table, and the data table stores the server address of the SQL Server sub-database.

[0082] In some embodiments, the SQL Server sub-database is stored in multiple storage areas, each of which corresponds to a different data table.

[0083] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0084] The device of the above embodiment is used to implement the corresponding data assembly method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0085] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data assembly method described in any of the above embodiments is implemented.

[0086] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 401, a memory 402, an input / output interface 403, a communication interface 404, and a bus 405. The processor 401, the memory 402, the input / output interface 1030, and the communication interface 404 are connected to each other in communication within the device through the bus 405.

[0087] The processor 401 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0088] The memory 402 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 402 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 402 and called and executed by the processor 401.

[0089] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0090] The communication interface 404 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0091] The bus 405 comprises a pathway for transmitting information between the various components of the device (eg, the processor 401 , the memory 402 , the input / output interface 1030 , and the communication interface 404 ).

[0092] It should be noted that, although the above device only shows the processor 401, the memory 402, the input / output interface 403, the communication interface 404 and the bus 405, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0093] The electronic device of the above-mentioned embodiment is used to implement the corresponding data assembly method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0094] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the data assembly method described in any of the above embodiments.

[0095] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0096] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the data assembly method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0097] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0098] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0099] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0100] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A data assembly method, characterized in that: Applied to a data assembly system, the data assembly system includes a SQL Server database for setting a configuration library, multiple SQL Server servers, a HBase cluster system and an HBase database set in the HBase cluster system, and the HBase cluster system is configured with a data assembly program and a configuration file; The method comprises: The configuration file is read by the data assembly program to obtain data range information; Performing data query through the configuration library according to the data range information to obtain row storage data query information; Using the data assembly program to process the row storage data query information to obtain column storage data information; Loading the column storage data information through the HBase database; The configuration file is read by the data assembly program to obtain data range information, including: Data information of target data for data assembly set in the configuration file; Each piece of data information is read through the data assembly program to obtain data range information; The SQL Server database includes multiple SQL Server sub-databases; The step of performing data query through the configuration library according to the data range information to obtain row storage data query information includes: According to the data range information, a match is performed through the configuration library to obtain the name of the SQL Server sub-database to be queried; The configuration library retrieves the stored SQL Server sub-database server address based on the name of the SQL Server sub-database, and queries the corresponding SQL Server sub-database according to the SQL Server sub-database server address to obtain the row storage data query information.

2. The method according to claim 1, characterized in that The step of processing the row-stored data query information using the data assembly program to obtain column-stored data information includes: The data assembly program is used to perform data assembly processing on each row-stored data query information, and the row-stored data query information is converted into column-stored data information.

3. The method according to claim 2, characterized in that The column storage data information is stored as a write instruction file; The loading of the column storage data information through the HBase database includes: In response to determining to execute the write instruction file, the write instruction file is read, the column storage data information is read, and the column storage data information is written into the HBase database.

4. The method according to claim 1, characterized in that: The configuration library stores a data table, and the data table stores the server address of the SQL Server sub-database.

5. The method according to claim 4, characterized in that The SQL Server sub-database is stored in multiple storage areas, and each storage area corresponds to a different data table.

6. A data assembly system, characterized in that: The data assembly system includes a SQL Server database for setting a configuration library, multiple SQL Server servers, a Haidup cluster system, and an HBase database set in the Haidup cluster system. The Haidup cluster system is configured with a data assembly program and a configuration file; The reading module is configured to read the configuration file through the data assembly program to obtain data range information; the reading of the configuration file through the data assembly program to obtain data range information is specifically used to set the data information of the target data for data assembly in the configuration file; each of the data information is read through the data assembly program to obtain data range information; A query module, configured to perform data query through the configuration library according to the data range information to obtain row storage data query information; The SQL Server database includes a plurality of SQL Server sub-databases; the data query is performed through the configuration library according to the data range information to obtain row storage data query information, specifically for performing matching through the configuration library according to the data range information to obtain the name of the queried SQL Server sub-database; the stored SQL Server sub-database server address is retrieved through the configuration library based on the name of the SQL Server sub-database, and the corresponding SQL Server sub-database is queried according to the SQL Server sub-database server address to obtain the row storage data query information; A conversion processing module is configured to process the row storage data query information using the data assembly program to obtain column storage data information; The loading module is configured to load the column storage data information through the HBase database.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images