Wide table generation method, device and system

Abstract

The invention discloses a wide table generation method, device and system, which can flexibly configure a wide table and improve the query efficiency. The wide table generation method comprises the following steps: acquiring change data of an original table; generating a database object according to the change data and a preset rule of the original table; generating a structured query language statement based on the database object; and storing the structured query language statement into a database to generate a wide table.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a wide table generation method, apparatus, and system. Background Art

[0002] In a large microservices system, when a user needs to view a report, the report query may involve complex queries across multiple databases or data sources, and it is impossible to perform an associated query. Or when the data volume is large, it is necessary to coordinate data exchange and processing between multiple databases, and the query speed is relatively slow. Currently, the commonly used query method is that business code forms a wide table by redundant fields or updates fields of other business services through a scheduled task. However, such a method greatly invades the business code, and sometimes the current service does not need the fields of other services, resulting in redundancy, increasing the consumption of hardware resources, and thus increasing the operating cost and maintenance cost of the system. Summary of the Invention

[0003] To solve the above technical problems, this application is proposed. Embodiments of this application provide a wide table generation method, apparatus, and system, which can flexibly configure a wide table and improve query efficiency.

[0004] According to a first aspect of this application, a wide table generation method is provided, including: obtaining change data of an original table; generating a database object according to the change data and a preset rule of the original table; generating a Structured Query Language (SQL) statement based on the database object; and storing the SQL statement in a database to generate a wide table.

[0005] As a possible implementation, the wide table generation method includes: reading the preset rule of the original table; wherein, generating a database object according to the change data and the preset rule of the original table includes: sequentially processing the change data according to the preset rule to generate the database object.

[0006] As a possible implementation, generating a database object according to the change data and the preset rule of the original table includes: extracting fields from the change data based on the preset rule; or reading fields from the change data based on the preset rule; or calculating new field values for the change data based on the preset rule; or filtering and storing fields for the change data based on the preset rule; or filtering and deleting fields for the change data based on the preset rule.

[0007] As a possible implementation manner, based on the preset rule, calculate the changed data to generate a new field value, including: based on the preset rule, perform at least one of function calculation, arithmetic calculation, and logical calculation on the changed data to generate a new field value.

[0008] As a possible implementation manner, the wide table generation method includes: listening to a target message queue of a distributed stream processing platform; wherein, the data transmitted by the target message queue includes a database name and a table name; based on the database name and the table name in the target message queue, determine the database corresponding to the database object; obtain the type of the database, the database instance, the preset rule, and the type of operation statement; wherein, based on the database object, generate a structured query language statement, including: based on the type of the database, the database instance, and the type of operation statement, generate a structured query language statement for the corresponding database from the database object.

[0009] As a possible implementation manner, store the structured query language statement into a database to generate a wide table, including: obtain a database connection according to the database instance; after the connection is successful, store the structured query language statement into the database to generate a wide table.

[0010] As a possible implementation manner, before obtaining the changed data of the original table, the wide table generation method includes: listening to the archive log of a relational database management system; when the original table changes, the archive log records the change information; wherein, obtaining the changed data of the original table includes: based on the change information recorded in the archive log, obtain the changed data of the original table.

[0011] As a possible implementation manner, when the original table changes, the archive log records the change information, including: when the original table is added, modified, or deleted, the archive log records the change information; wherein, the change information includes addition information, modification information, or deletion information.

[0012] According to the second aspect of the present application, there is provided a wide table generation device, including: an acquisition module, configured to acquire the changed data of the original table; a first generation module, configured to generate a database object according to the changed data and the preset rule of the original table; a second generation module, configured to generate a structured query language statement based on the database object; and a storage module, configured to store the structured query language statement into a database to generate a wide table.

[0013] According to a third aspect of the present application, a wide table generation system is provided, including: a backend component for interacting with a database; a listening component for listening to a relational database management system and a distributed stream processing platform; and a wide table generation device as described in the second aspect above, where the wide table generation device is communicatively connected to the backend component and the listening component respectively.

[0014] The wide table generation method, device and system provided by the present application can process data specifically according to rules based on the settings of database tables and events to generate a target wide table, which not only improves the flexibility of wide table generation, but also can reduce the intrusion of business services, improve query efficiency, and reduce database redundancy. Brief Description of the Drawings

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other objects, features and advantages of the present application will become more obvious. The accompanying drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application, and do not constitute a limitation to the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a flowchart of a wide table generation method provided by an exemplary embodiment of the present application.

[0017] Figure 2 It is a structural diagram of a wide table generation device provided by an exemplary embodiment of the present application.

[0018] Figure 3 It is a structural diagram of a wide table generation system provided by an exemplary embodiment of the present application.

[0019] Figure 4 It is a working flowchart of a wide table generation system provided by an exemplary embodiment of the present application.

[0020] Figure 5 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. Detailed Description of the Embodiments

[0021] Next, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the exemplary embodiments described herein.

[0022] In a large-scale microservices system, when a user wants to view a report, this report may involve multiple databases. For example, a report query may involve complex queries across multiple databases or data sources, which usually leads to a slow query speed because it is necessary to coordinate data exchange and processing between multiple databases. Moreover, as the data volume increases, the query performance may further decline, thus affecting the user experience and system response time. Or, in a microservices architecture, each service usually has its own database and data model, which may result in data redundancy and inconsistency. When a report needs to query across multiple databases, problems such as inconsistent data versions, data conflicts, or data loss may be encountered, thus affecting the accuracy and reliability of the report.

[0023] To solve the problem of low efficiency in viewing reports, Figure 1 is a schematic flowchart of a wide table generation method provided by an exemplary embodiment of the present application. As Figure 1 shown, the wide table generation method includes: obtaining the change data of the original table (see Figure 1 S100). When an operation of adding, deleting, or modifying the original table is performed, obtain the change data generated by the change operation of the original table. Then, according to the change data and the preset rules of the original table, generate a database object (see Figure 1 S200). The preset rules are bound to the table and the event. When the original table needs to be updated to the target table due to changes, process the change data in accordance with the rules of the target table to form a database object corresponding to the database. Based on the database object, generate a Structured Query Language statement (see Figure 1 S300). The Structured Query Language (SQL) statement is a basic tool for database management. Through SQL statements, the structure and configuration of the database can be defined, such as creating databases, tables, indexes, etc. Therefore, generating SQL statements from database objects can facilitate database management and ensure the correct storage and access of data. Store the Structured Query Language statement in the database to generate a wide table (see Figure 1 S400). Store the SQL statement in the database (such as MySQL, ElasticSearch database, etc.) to generate a wide table. A wide table is a database table that contains many fields (i.e., columns). By integrating data related to multiple business themes together, the wide table makes queries simpler and more efficient. Generating a wide table can improve query efficiency and configure dynamic report generation without development, which can reduce the intrusion of business services.

[0024] The following combines Figure 1 to introduce the wide table generation method provided by the embodiments of the present application in more detail.

[0025] In S100, obtain the changed data of the original table. When there are addition, modification, or deletion changes to the original table, obtain the changed data of the original table.

[0026] In some embodiments, the archived logs of a relational database management system can be monitored; when there are changes to the original table, the archived logs record the change information; therefore, the method for obtaining the changed data of the original table can be: based on the change information recorded in the archived logs, obtain the changed data of the original table. For example, when there are additions, modifications, or deletions to the original table, the archived logs record the change information; among them, the change information includes addition information, modification information, or deletion information.

[0027] As a possible implementation of monitoring the archived logs of a relational database management system, the Canal component can be deployed to monitor the binlog logs of MySQL in real time and push them to Kafka. The backend service connects to Kafka, receives the changed data, and parses the data into a DynamicDataDto object. The Canal component is a framework for developing real-time data synchronization, used to capture database log data, parse it, and send it to the target end (such as a kafka message queue). Mysql is a relational database management system used to store and manage data. The binlog log is the binary log of MySQL, and the binlog log records all change operations in the database, such as insertions, updates, and deletions. These logs are very important for data recovery and synchronization. Kafka is a distributed stream processing platform that allows the publishing and subscribing of data streams, similar to a message queue. For example, when an operation of adding a data is performed on a table, the monitoring kafka object will receive the information of adding a new data and know the content of the data, that is, the changed data.

[0028] When operations such as addition, deletion, or modification occur in the original table, the binlog will record these changes and generate corresponding variations. As an example, when inserting a new record into the original table, the binlog will record this insertion operation, such as the timestamp of the insertion operation, the inserted SQL statement, the inserted data content, and relevant transaction information. This information will be written into the binlog for subsequent data recovery, master-slave replication, and other operations. When deleting a record from the original table, the binlog will similarly record this deletion operation, such as the timestamp of the deletion operation, the deleted SQL statement, the unique identifier (such as the primary key) of the deleted data, and relevant transaction information. When data needs to be restored, the deleted record can be found through the binlog and corresponding restoration measures can be taken. When a record in the original table is modified, the binlog will record this modification operation, such as the timestamp of the modification operation, the modified SQL statement, the data content before and after the modification, and relevant transaction information. For modification operations, the binlog will also record the specific modification type, such as whether a certain field value is updated or multiple field values are modified, etc.

[0029] In S200, database objects are generated according to the changed data and the preset rules of the original table. Based on the rule configuration, wide tables can be generated, which can reduce the intrusion of business services.

[0030] In some embodiments, the preset rules of the original table are read, and the changed data is processed in sequence according to the preset rules to generate database objects. That is to say, there can be multiple rules for processing data, and the types and orders of the rule configurations corresponding to each change event are not necessarily the same. The rules for each change event are configured based on their own requirements. For example, the preset type can be LISTEN, which is used to extract fields from the received data according to the rules. The preset rule type can be READ, which is used to read fields from the database according to the rule statement. The preset rule type can be RULE, which calculates according to the rules to generate new field values. The calculations include: function calculations: uuid(), md5(), cal(), int(); arithmetic calculations: addition (+), subtraction (-), multiplication (X), division (%); logical calculations: true, false. The preset rule type can be FILTER, which is used to filter and store fields according to the rule fields. The preset rule type can be TEMP, which filters and deletes fields according to the rule fields. Therefore, fields can be extracted from the changed data based on the preset rules; or fields can be read from the changed data based on the preset rules; or calculations can be performed on the changed data based on the preset rules to generate new field values; or the changed data can be filtered and stored based on the preset rules; or the changed data can be filtered and deleted based on the preset rules.

[0031] During the processing, the above-mentioned rule types can be supported, and the order of the rules is configured according to the association order of the wide table fields. It is possible that the read operation is at the end and the filtering is at the front, or the read operation is at the front and the filtering is at the end. For example, if you need to find the user department name by the user ID, and the original user table is associated with the user ID, then it may be necessary to first find the department ID according to the user ID, and then find the department name through the department ID. It is not possible to directly find the department name in one step through the user ID. The rule order can be configured such that the first step is to find the department ID according to the user ID, and the second step is to find the department name through the department ID. Moreover, the type of the rule is also set according to the processing requirements. For example, if reading is required, the read rule is set; if reading is not required, the read rule is not set.

[0032] As a possible implementation method for calculation according to rules, based on preset rules, at least one of function calculation, arithmetic calculation, and logical calculation can be performed on the changed data to generate a new field value. For example, there are two data in the original table, namely unit price and quantity. If the wide table wants to redundantly store the total price, then a calculation layer needs to be created, and a processing process is added in the middle. Through arithmetic calculation, the multiplication of the unit price and the quantity is performed to generate a new field value for the total price.

[0033] In S300, based on database objects, Structured Query Language (SQL) statements are generated. Since the changed data finally needs to be saved in the database, it is necessary to generate the corresponding SQL statements for the changed data. SQL statements are the basic tools for database management. Through SQL statements, the structure and configuration of the database can be defined, such as creating databases, tables, indexes, etc. SQL statements are used to define the structure of database tables, including columns, data types, and constraint conditions, to ensure the correct storage and access of data. Through SQL statements, indexes can be created to accelerate the data retrieval speed and improve the performance of the system. SQL statements can simplify data access and operations, and SQL statements define various constraint conditions, such as primary key constraints, foreign key constraints, unique constraints, and check constraints, to ensure data consistency and integrity. Therefore, generating SQL statements from database objects is the basis and key for multiple aspects such as database management, data storage, data access, data security, data consistency and integrity, performance optimization, data backup and recovery, data migration and integration, and data analysis and mining. By reasonably using SQL statements, the performance, reliability, and security of the database system can be significantly improved.

[0034] In some embodiments, the preset rules of the original table can be obtained by listening. A possible implementation of the listening method can be: listening to the target message queue of the distributed stream processing platform; wherein, the data transmitted in the target message queue includes the library name and the table name; based on the library name and the table name in the target message queue, determine the database corresponding to the database object; obtain the type of the database, the database instance, the preset rules, and the type of the operation statement. The data transmitted in the message queue may contain detailed information about database operations, such as the library name, the table name, and the operations to be executed, etc. For example, the wide table service listens to the cdl_gsp_* (Kafka topics prefixed with cdl_gsp_) of the kafka queue, finds the rule set by obtaining the library name and the table name in the data, and processes the changed data through the rule set. The rule set includes field conversion, database reading, function operation, arithmetic operation, logical operation, etc. Correspondingly, when generating the structured query language statement, based on the type of the database, the database instance, and the type of the operation statement, generate the structured query language statement of the corresponding database for the database object.

[0035] As a possible implementation of generating Structured Query Language (SQL) statements for database objects, it is necessary to first determine the type of database for the SQL, which database instance the SQL belongs to, and which type of operation statement the SQL is. Then, generate the executable statement corresponding to the database for the database object. Determining the type of database for the SQL means determining the type of the target database system. Different database systems (such as MySQL, PostgreSQL, Oracle, SQL Server, etc.) have different SQL grammars and features. Therefore, before generating SQL statements, it is necessary to clarify for which database system the code is written. A database instance is a specific running entity of a database system, which contains all the data and metadata of the database. Multiple database instances may be running on a server, and each instance manages a set of independent databases. Therefore, determining which database instance the SQL statement is for means clarifying in which specific database environment the statement should be executed, which is usually specified by the instance name or database name in the database connection string. There are various types of SQL statements, including Data Definition Language (DDL) statements (such as CREATE, ALTER, DROP, etc.), Data Manipulation Language (DML) statements (such as INSERT, UPDATE, DELETE, SELECT, etc.), Data Control Language (DCL) statements (such as GRANT, REVOKE, etc.), and Transaction Control Language (TCL) statements (such as COMMIT, ROLLBACK, etc.). Determining the type of SQL statement is crucial for ensuring the correct execution of the statement. Different types of statements have different functions and impacts in the database. For example, DDL statements are used to define the database structure, DML statements are used to operate on the data in the database, and DCL statements are used to control database access permissions.

[0036] In S400, store the Structured Query Language statement in the database to generate a wide table. After judging the SQL, the database instance corresponding to the Structured Query Language statement can be found, the database connection can be obtained, and after the connection is successful, the Structured Query Language statement is stored in the database to generate a wide table.

[0037] Multiple SQLs can form a wide table. A wide table is a database table with more fields (columns). It assembles multiple data tables related to business topics by associating fields into a large table to achieve the unified storage of different-dimensional attribute information of business entities.

[0038] Figure 2 It is a schematic structural diagram of a wide table generation device provided by an exemplary embodiment of the present application, as Figure 2As shown in the figure, the wide table generation device 2 includes: an acquisition module 21, configured to acquire the changed data of the original table; a first generation module 22, configured to generate a database object according to the changed data and the preset rules of the original table; a second generation module 23, configured to generate a structured query language statement based on the database object; and a storage module 24, configured to store the structured query language statement in the database to generate a wide table.

[0039] The wide table generation device provided in this application can, based on the rule settings of the database table and events, process data specifically according to the rules to generate a target wide table, which not only improves the flexibility of wide table generation, but also reduces the intrusion of business services, improves query efficiency, and reduces database redundancy.

[0040] As a possible implementation, the wide table generation device 2 can be configured to: read the preset rules of the original table; correspondingly, the first generation module 22 can be configured to: process the changed data in sequence according to the preset rules to generate a database object.

[0041] As a possible implementation, the first generation module 22 can be configured to: extract fields from the changed data based on the preset rules; or read fields from the changed data based on the preset rules; or calculate the changed data based on the preset rules to generate new field values; or filter and store fields from the changed data based on the preset rules; or filter and delete fields from the changed data based on the preset rules.

[0042] As a possible implementation, the first generation module 22 can also be configured to: perform at least one of function calculation, arithmetic calculation, and logical calculation on the changed data based on the preset rules to generate new field values.

[0043] As a possible implementation, the wide table generation device 2 can be configured to: monitor the target message queue of the distributed stream processing platform; where the data transmitted by the target message queue includes the database name and table name; determine the database corresponding to the database object based on the database name and table name in the target message queue; obtain the type of the database, the database instance, the preset rules, and the type of operation statement; correspondingly, the second generation module 23 can be configured to: generate a structured query language statement for the corresponding database based on the type of the database, the database instance, and the type of operation statement.

[0044] As a possible implementation, the storage module 24 can be configured to: obtain a database connection according to the database instance; after the connection is successful, store the structured query language statement in the database to generate a wide table.

[0045] As a possible implementation, the wide table generation device 2 can be configured to: monitor the archived logs of the relational database management system; when the original table changes, the archived logs record the change information; wherein, obtaining the change data of the original table includes: obtaining the change data of the original table based on the change information recorded in the archived logs.

[0046] As a possible implementation, the wide table generation device 2 can also be configured to: when there are additions, modifications, or deletions to the original table, the archived logs record the change information; wherein, the change information includes addition information, modification information, or deletion information.

[0047] To solve the problem of low efficiency in viewing reports, Figure 3 is a schematic structural diagram of a wide table generation system provided by an exemplary embodiment of the present application. As Figure 3 shown, the wide table generation system 3 is built to include: a backend component 31, the backend component 31 is used to interact with the database; a monitoring component 32, the monitoring component 32 is used to monitor the relational database management system and the distributed stream processing platform; the wide table generation device 2 provided above, the wide table generation device 2 is communicatively connected to the backend component 31 and the monitoring component 32 respectively.

[0048] The backend components implement the Controller layer to provide Http services through the Spring Cloud microservices architecture and the Spring MVC technology stack. The Controller and Service layers are connected through the Spring injection method. Finally, Mybatis is introduced as the technology stack for the interaction between the code and the database. The listening component can be a canal component. The Spring Cloud microservices architecture is a modern software architecture design that decomposes traditional monolithic applications into a series of small, independent services. These services are usually built around specific business functions and can be deployed and scaled independently. Spring Cloud provides a complete set of solutions, including service discovery, configuration management, message bus, load balancing, circuit breakers, etc., to support the rapid development and deployment of microservices architecture. Spring MVC is part of the Spring Framework and is a lightweight Web framework based on the Java objective MVC. Spring MVC is like a toolbox with many tools that can help quickly build a user-friendly interface. The Controller layer in Spring MVC is responsible for receiving user requests, processing these requests, and returning the processing results to the user. The Service layer is configured for the Controller layer through the Spring injection method. The Service layer is responsible for handling some complex business logics. Mybatis is responsible for processing the query requests from the Service layer into SQL statements, obtaining query results from the database, and then processing the query results into content that the Service layer can read. That is to say, Mybatis helps the code and the database to interact to obtain the information required by the user. The backend service can also be packaged into a Docker image in one click through the Devops platform and pushed to the Docker repository. It is containerized and deployed through the K8S platform. The deployed listening component is used to monitor the binlog logs of Mysql in real time and push them to Kafka. Therefore, the backend service is used to connect to kafka, receive information from the message queue, parse the data of kafka into change data, and transmit it to the wide table generation device. The wide table generation device processes the change data into SQL statements based on preset rules, stores them in the database, and finally generates the required wide table based on user queries.

[0049] Figure 4 It is a schematic diagram of the workflow of the wide table generation system provided by an exemplary embodiment of the present application, in order to Figure 4For example, the canal component can be used to monitor data sources, such as monitoring MySQL and ORACLE databases, and then pushing the data to Kafka. The wide table generation device obtains the changed data from Kafka and then processes the wide table, that is, obtains the changed data of the original table; generates database objects according to the changed data and the preset rules of the original table; generates Structured Query Language (SQL) statements based on the database objects; stores the SQL statements in a database (such as MySQL, ORACLE database or ES database) to generate a wide table. The wide table generation system supports multiple data sources, supports storing in Mysql, ES, etc., improves query efficiency, dynamically generates reports according to the configuration, eliminates the need for development, and can reduce the intrusion of business services.

[0050] An electronic device, comprising: a processor; a memory for storing processor-executable instructions; the processor for executing the wide table generation method described in the embodiments provided in the present application.

[0051] Next, with reference to Figure 5 to describe the electronic device according to an embodiment of the present application. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the input signals collected from them.

[0052] Figure 5 The block diagram of the electronic device according to an embodiment of the present application is illustrated.

[0053] As Figure 5 shown, the electronic device 10 includes one or more processors 11 and a memory 12.

[0054] The processor 11 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0055] The memory 12 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the wide table generation method of the various embodiments of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0056] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0057] When the electronic device is a stand-alone device, the input device 13 may be a communication network connector for receiving the collected input signals from the first device and the second device.

[0058] In addition, the input device 13 may further include, for example, a keyboard, a mouse, and the like.

[0059] The output device 14 may output various information to the outside, including the determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and the like.

[0060] Of course, for simplicity, Figure 5 only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, and the like are omitted. In addition, according to specific application scenarios, the electronic device 10 may further include any other appropriate components.

[0061] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0062] A computer-readable storage medium stores a computer program, and the computer program is used to execute the wide table generation method described in the embodiments provided by the present application.

[0063] The computer-readable storage medium may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0064] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize some of their variations, modifications, alterations, additions, and sub-combinations.

Claims

1. A method for generating a wide table, characterized in that: include: Get the changed data of the original table; Generate a database object according to the changed data and preset rules of the original table; Based on the database object, generate a structured query language statement; The structured query language statement is stored in a database to generate a wide table.

2. The method for generating a wide table according to claim 1, characterized in that: The wide table generation method comprises: Reading the preset rules of the original table; The step of generating a database object according to the changed data and the preset rules of the original table includes: The changed data is processed in sequence according to the preset rules to generate the database object.

3. The method for generating a wide table according to claim 1, characterized in that: Generating a database object according to the changed data and the preset rules of the original table includes: extracting fields from the change data based on the preset rule; or Based on the preset rule, read the field from the changed data; or Based on the preset rule, the changed data is calculated to generate a new field value; or Based on the preset rules, the changed data is filtered and the fields are stored; or Based on the preset rules, the changed data is filtered and fields are deleted.

4. The method for generating a wide table according to claim 3, characterized in that: Based on the preset rule, the changed data is calculated to generate a new field value, including: Based on the preset rule, at least one of function calculation, arithmetic calculation and logic calculation is performed on the changed data to generate a new field value.

5. The method for generating a wide table according to claim 1, characterized in that: The wide table generation method comprises: Monitor the target message queue of the distributed stream processing platform; wherein the data transmitted by the target message queue includes a library name and a table name; Determine the database corresponding to the database object based on the library name and table name in the target message queue; Obtaining the type of the database, database instance, preset rules, and operation statement type; Wherein, generating a structured query language statement based on the database object includes: Based on the type of the database, the database instance and the type of operation statement, the database object generates a structured query language statement corresponding to the database.

6. The method for generating a wide table according to claim 5, characterized in that: The structured query language statement is stored in a database to generate a wide table, including: According to the database instance, obtain a database connection; After the connection is successful, the structured query language statement is stored in the database to generate a wide table.

7. The method for generating a wide table according to claim 1, characterized in that: Before obtaining the changed data of the original table, the wide table generation method includes: Monitor the archive logs of relational database management systems; When the original table is changed, the archive log records the change information; The change data of the original table is obtained, including: Based on the change information recorded in the archive log, the change data of the original table is obtained.

8. The method for generating a wide table according to claim 7, characterized in that: When the original table is changed, the archive log records the change information, including: When the original table is added, modified or deleted, the archive log records the change information; wherein the change information includes addition information, modification information or deletion information.

9. A wide table generating device, characterized in that: include: The acquisition module is used to obtain the change data of the original table; A first generating module, used for generating a database object according to the changed data and preset rules of the original table; A second generating module, used for generating a structured query language statement based on the database object; The storage module is used to store the structured query language statement in a database to generate a wide table.

10. A wide table generation system, characterized in that: include: A backend component, wherein the backend component is used to interact with a database; A monitoring component, wherein the monitoring component is used to monitor a relational database management system and a distributed stream processing platform; As described in claim 9, the wide table generating device is communicatively connected with the backend component and the monitoring component respectively.

Images