Data conversion method and system and related equipment

Abstract

The invention provides a data conversion method and system and related equipment, and the method comprises the following steps: the data conversion system obtains a workflow corresponding to a data conversion request input by a user, executes the workflow, reads source table data from a first database based on an extraction node, converts the source table data into target table data based on a conversion node, and sends the target table data to a server; the target table data is stored in the second database based on the loading node, the whole data conversion process is realized by the data conversion system and does not depend on the data conversion capability of the database, the conversion process does not need to write customized codes by developers, and a user only needs to configure a workflow, so that the user experience is improved. The problem that data conversion between two-dimensional table data and multi-dimensional table data depends on the capacity of a database, so that data conversion cannot be achieved in many scenes and can only be achieved through code writing by developers is solved.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a data transformation method, system and related equipment. Background Technology

[0002] Currently, commonly used databases include relational database management systems (RDBMS), multidimensional databases (MDD), and hybrid databases. Relational databases store data in two-dimensional tables, which are easy to manage and operate. However, if users need to analyze data across multiple dimensions, they can only perform cross-table joins, and the more dimensions there are, the worse the query performance becomes. Multidimensional databases store data by combining multiple dimensions. This type of database is equivalent to pre-storing data after it has been partitioned according to various dimensions, enabling flexible and fast retrieval of various dimensions without the need for join queries, making it very suitable for big data analysis. Hybrid databases support the functions of both types of databases. They can leverage the advantages of each and, during business processing, transform data dimensions as needed, such as converting two-dimensional table data in a relational database into multidimensional table data in a multidimensional database, or vice versa.

[0003] Typically, database engines for hybrid databases have the aforementioned data dimension transformation capabilities. However, these engines can only transform two-dimensional and multi-dimensional tables within a hybrid database, such as converting two-dimensional table A in database A into a multi-dimensional table B stored in database A. They cannot perform data transformations between different databases, such as converting two-dimensional table A1 in database A into a multi-dimensional table B1 in database B. In such cases, developers are required to write custom code to implement this process, increasing development and operational costs. Summary of the Invention

[0004] This application provides a data conversion method, system, and related equipment to solve the problem that data conversion between two-dimensional table data and multi-dimensional table data depends on the database's own capabilities, resulting in many scenarios where data conversion cannot be achieved and can only be implemented by developers writing code.

[0005] Firstly, a data transformation method is provided, which is applied to a data transformation system. The data transformation system establishes communication connections with a first database and a second database. The method includes the following steps: the data transformation system obtains a data transformation request input by a first user. The data transformation request includes a workflow identifier. The data transformation system obtains the code of a first workflow from a workflow space based on the workflow identifier. The workflow space includes one or more pre-configured workflows. The first workflow includes an extraction node, a transformation node, and a loading node. The extraction node includes the read address of a source table in the first database. The transformation node includes data transformation rules. The loading node includes the storage address of a target table in the second database. The source table and the target table are tables with different dimensions. The transformation system executes the code of the first workflow, reads source table data from the first database based on the extraction node, transforms the source table data into target table data based on the transformation node, and stores the target table data in the second database based on the loading node.

[0006] The method described in the first aspect involves reading data from a first database, transforming the data, and storing it in a second database. The entire data transformation process is implemented by the data transformation system, independent of the database's own transformation capabilities. This solves the problem that data transformation between two-dimensional and multi-dimensional tables relies on the database's own capabilities, making data transformation impossible in many scenarios and requiring developers to write code. Furthermore, the data transformation process is automatically generated based on user-configured workflows, allowing users to flexibly configure the transformation process according to their business needs, rather than simply performing dimensional transformations from two-dimensional to multi-dimensional tables. Moreover, the absence of developer coding significantly reduces development and maintenance costs.

[0007] In one possible implementation, before the data conversion system obtains the data conversion request input by the user, the method further includes the following steps: the data conversion system obtains configuration information input by the second user, the configuration information including configuration information of the first workflow and the execution engine types supported by the first database and the second database; the data conversion system determines the first code template used by the first workflow based on the execution engine type, wherein the data conversion system includes code templates for multiple execution engine types; the data conversion system determines the code of the first workflow based on the first code template and stores it in the workflow space.

[0008] The above implementation method, by pre-configuring code templates that support various execution engine types, ensures that the code generated by the user's workflow based on the code templates is compatible with the engines supported by the first and second databases, thus improving the accuracy and reliability of the workflow code. Simultaneously, pre-configuring code templates for various engines reduces the workload for developers in writing and debugging code for different types of databases. Developers do not need to design and develop data transformation logic for each database; they can directly obtain engine-compatible workflow code by configuring the workflow, thereby reducing development costs.

[0009] In one possible implementation, the first code template includes one or more code sub-templates for each node. When the data conversion system determines the code of the first workflow based on the first code template and stores it in the workflow space, the data conversion system determines the multiple nodes included in the first workflow based on the configuration information of the first workflow, determines the code fragment of each node based on the first code template, determines the code of the first workflow based on the code fragment of each node, and stores it in the workflow space.

[0010] The above implementation uses reusable code sub-templates, allowing similar data transformation needs to share the same code snippets, improving code reuse efficiency. When the workflow needs to be modified or new nodes added, only the corresponding code sub-template needs to be updated, without the need for significant modifications to existing code sub-templates, enabling the system to quickly adapt to changing requirements. Furthermore, by configuring code sub-templates, user-configured workflows can automatically generate code, eliminating the need for developers to write code and greatly reducing development and maintenance manpower costs.

[0011] In one possible implementation, the configuration request input by the user may include table data configuration information. This information may include the address information of the source and target tables, the database information of the databases containing the source and target tables, and their location information within the databases. For example, the address information may include database links and table names. The table data configuration information may also include field information of the source table and dimension information of the target table. The field information of the source table includes information about the relevant fields involved in the data transformation, such as field names, data types, and lengths. The dimension information of the target table includes information about the relevant dimensions involved in the data transformation, such as dimension names, member names, and the number of dimensions. Table data configuration information is the foundation of data transformation and may include more content; this application does not impose specific limitations on it.

[0012] The above implementation configures the target table, source table, first database, and second database information through configuration table data. This allows subsequent configuration of mapping relationships, nodes, and workflows to directly reuse the table data configuration information, improving configuration efficiency. For example, after configuring table data configuration information M1, including the source table name, target source table name, database types of the first and second databases, and database connections, the second user can directly use data M1 to refer to the above content when configuring nodes, eliminating the need for repeated configuration and improving the user experience. Furthermore, if the second user needs to modify the relevant information of the source or target table, they can directly modify the table configuration information. All nodes and workflows using this table configuration information can be modified uniformly, rather than individually, further improving the second user's experience.

[0013] In one possible implementation, the configuration information may include mapping configuration information, which includes dimension mapping relationships and member mapping relationships. The dimension mapping relationship includes the mapping relationship between the source table data and the dimensions of the target table, and the member mapping relationship includes the mapping relationship between the source table data and the members of the target table.

[0014] In specific implementation, dimension mapping relationships can include the mapping relationship between fields in the source table and dimensions in the target table. For example, a two-dimensional table includes fields such as "Date," "Product Type," "Sales Amount," and "Region." The Date field can be mapped to the Time dimension, the Product Type field to the Product dimension, the Region to the Geographic dimension, and the Sales Amount to the Sales Amount metric. Dimension mapping relationships can also include the mapping relationship between field values ​​in the source table and dimensions in the target table. For example, the field values ​​under the Region field in a two-dimensional table can include Asia and Europe. Therefore, the Asia field value can be mapped to the Asia region dimension, and the Europe field value can be mapped to the Europe region dimension. It should be understood that the division of dimensions is determined based on the analysis needs of the second and first users. Therefore, dimensions can be determined based on the fields in the source table, or the field values ​​under some fields in the source table can be generalized to generate a new dimension. These can all be configured according to the actual usage needs of the second and first users, and this application does not impose specific limitations.

[0015] In specific implementation, member mapping relationships can include the mapping relationship between fields in the source table and members in the target table. For example, the field values ​​under the region field of a two-dimensional table can include Asia and Europe. Therefore, the region field can be mapped to the geographical dimension, and the values ​​of the Asia and Europe fields can be mapped to Asian and European members under the geographical dimension. Member mapping relationships can also include the mapping relationship between field values ​​in the source table and members in the target table. For example, a two-dimensional table includes fields for China, Japan, and India, and a multidimensional table includes an Asian dimension, which includes Chinese, Japanese, and Indian members. In this case, the China field in the two-dimensional table can be mapped to the Chinese members under the Asian dimension in the multidimensional table. It should be understood that the division of members is also determined based on the analysis needs of the second and first users. Therefore, members can be fields in the source table, merged or decomposed values ​​of fields in the source table, field values ​​in the source table, and merged or decomposed values ​​of field values. These can all be configured according to the actual usage needs of the second and first users, and this application does not impose specific limitations.

[0016] The above implementation ensures the correct mapping between the dimensions of the source table data and the target table by configuring dimension mapping relationships and member mapping relationships. This enables the data conversion system to convert two-dimensional table data to multi-dimensional table data. Regardless of the structure of the source table data, as long as dimension mapping relationships and member mapping relationships are configured, the system can correctly map the data to the target table and achieve unified data management.

[0017] In one possible implementation, the member mapping relationship includes condition mapping, and the mapping configuration information includes preset conditions. The condition mapping is used to indicate that there is a member mapping relationship between the source table data and the members of the multidimensional table when the source table data meets the preset conditions. The preset conditions include one or more of numerical range, set, and fuzzy matching.

[0018] The above implementation allows users to configure various preset conditions for condition mapping according to actual business needs. After data transformation, the target table required by the user can be obtained directly, instead of transforming the source table and then making various adjustments to the transformed multidimensional table based on business requirements, thus improving the user experience for secondary users. Furthermore, this process only requires configuring condition mappings and does not require developers to participate in code development, reducing labor costs.

[0019] In one possible implementation, the configuration information includes node configuration information. The configuration information of the transformation node includes mapping configuration information and data transformation rules. The data transformation rules include: performing data transformation on the source table data based on the transformation value table to obtain the target table data. The transformation value table includes transformation values ​​corresponding to different source table data.

[0020] Optionally, nodes may include extraction nodes, loading nodes, and transformation nodes, and may also include extension nodes and verification nodes. Extension nodes are used to extend and supplement the data transformation process, such as calling interfaces of other systems, using external services of other systems to enrich the data, or triggering operations of external systems. Verification nodes are used to verify the accuracy and integrity of the data after transformation. Verification nodes can define verification rules and can automatically check whether the data meets the expected format, range, integrity and other standards to ensure data quality. The above examples of node types are for illustration. This application may also include more node types, and the corresponding node configuration information may also include more content, which will not be illustrated here.

[0021] In the above implementation, the transformation node includes mapping configuration information and data transformation rules. The mapping configuration information determines which data in the source table has its transformation value written to position A in the target table, while the data transformation rules determine how to generate the transformation value and fill it into position A in the target table. This separate configuration method allows different transformation nodes to reuse the mapping configuration information and data transformation rules, improving the efficiency of node configuration. Furthermore, when updating the mapping relationship, only the mapping configuration data needs to be modified; the configuration information of each transformation node does not need to be changed, making the system easier to maintain and update.

[0022] In one possible implementation, the data transformation rules include: transforming source table data based on transformation labels to obtain target table data, wherein the transformation labels include fixed value labels and text value labels.

[0023] The above implementation method allows users to flexibly configure transformation rules, enabling not only the transformation of source table data to target table data, but also personalized transformations based on user needs. For example, the transformed value could be a fixed label or a fixed value when the source table data meets specific conditions. This ensures the resulting target table matches the user's requirements, rather than transforming the source table and then adjusting the multidimensional table based on business needs. Furthermore, this personalized transformation process is simple and quick, requiring no custom code development by developers, thus reducing labor costs.

[0024] In one possible implementation, a first database is deployed on a first computing device, and a second database is deployed on a second computing device. When the target table data is stored in the second database, the data conversion system obtains the communication message format supported by the second database, generates a message of the target table data based on the communication message format, and sends the message of the target table data to the second database.

[0025] The above implementation method achieves communication between the data conversion system and the first and second databases through message communication, enabling the data conversion system provided in this application to realize data conversion between databases on different computing devices. The data conversion process is automatically generated code based on the user-configured workflow, allowing users to flexibly configure the data conversion process according to business needs, rather than simply realizing the dimensional conversion of two-dimensional table data to multi-dimensional table data. Moreover, it does not require developers to write code, greatly reducing the manpower costs of development and maintenance.

[0026] Secondly, a data conversion system is provided. The data conversion system establishes communication connections with a first database and a second database. The system includes: an acquisition unit for acquiring a data conversion request input by a first user, the data conversion request including a workflow identifier; an acquisition unit for acquiring the code of a first workflow from a workflow space based on the workflow identifier, the workflow space including one or more pre-configured workflows, the first workflow including an extraction node, a transformation node, and a loading node, the extraction node including the read address of a source table in the first database, the transformation node including data conversion rules, and the loading node including the storage address of a target table in the second database, wherein the source table and the target table are tables with different dimensions; and an execution unit for the data conversion system to execute the code of the first workflow, reading source table data from the first database based on the extraction node, converting source table data into target table data based on the transformation node, and storing target table data in the second database based on the loading node.

[0027] The system described in the second aspect reads data from a first database, transforms it, and stores it in a second database. The entire data transformation process is implemented by the data transformation system itself, independent of the database's own transformation capabilities. This solves the problem that data transformation between two-dimensional and multi-dimensional tables relies on the database's own capabilities, making data transformation impossible in many scenarios and requiring developers to write code. Furthermore, the data transformation process is automatically generated based on user-configured workflows, allowing users to flexibly configure the transformation process according to their business needs, rather than simply performing dimensional transformations from two-dimensional to multi-dimensional tables. Moreover, since no code needs to be written by developers, this significantly reduces development and maintenance costs.

[0028] In one possible implementation, the system further includes a configuration unit for acquiring configuration information input by a second user, the configuration information including configuration information of the first workflow and execution engine types supported by the first and second databases; a code generation unit for determining a first code template used by the first workflow based on the execution engine type, wherein the data conversion system includes code templates for multiple execution engine types; and a code generation unit for determining the code of the first workflow based on the first code template and storing it in the workflow space.

[0029] In one possible implementation, the first code template includes one or more code sub-templates for each node, a code generation unit for determining multiple nodes included in the first workflow based on the configuration information of the first workflow, a code generation unit for determining code fragments for each node based on the first code template, and a code generation unit for determining the code of the first workflow based on the code fragments of each node and storing it in the workflow space.

[0030] In one possible implementation, the configuration information includes mapping configuration information, which includes dimension mapping relationships and member mapping relationships. The dimension mapping relationship includes the mapping relationship between the source table data and the dimensions of the multidimensional table, and the member mapping relationship includes the mapping relationship between the source table data and the members of the multidimensional table.

[0031] In one possible implementation, the member mapping relationship includes condition mapping, and the mapping configuration information includes preset conditions. The condition mapping is used to indicate that there is a member mapping relationship between the source table data and the members of the multidimensional table when the source table data meets the preset conditions. The preset conditions include one or more of numerical range, set, and fuzzy matching.

[0032] In one possible implementation, the configuration information includes node configuration information, and the configuration information of the transformation node includes mapping configuration information and data transformation rules. The data transformation rules include: performing data transformation on the source table data based on the transformation value table to obtain the target table data, wherein the transformation value table includes transformation values ​​corresponding to different source table data.

[0033] In one possible implementation, the data transformation rules include: transforming source table data based on transformation labels to obtain target table data, wherein the transformation labels include fixed value labels and text value labels.

[0034] In one possible implementation, a first database is deployed on a first computing device, a second database is deployed on a second computing device, and an execution unit is used to obtain the communication message format supported by the second database, generate a message of target table data based on the communication message format, and send the message of target table data to the second database.

[0035] Thirdly, a computing device is provided, the computing device including a processor and a memory, the memory for storing instructions and the processor for executing the instructions, such that the computing device implements the method described in the first aspect.

[0036] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0037] Fifthly, a computing device cluster is provided, the computing device cluster including at least one computing device, each computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster implements the method described in the first aspect.

[0038] In a sixth aspect, a computer program product comprising instructions is provided, the computer program product including instructions capable of running on a computing device or stored in any available medium, and when the computer program product is run on a computing device or a cluster of computing devices, causing the computing device or cluster of computing devices to perform the method described in the first aspect. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a multidimensional database data model;

[0040] Figure 2 This is an architecture diagram of a data conversion system provided in this application;

[0041] Figure 3 This is an example diagram of a data transformation system provided in this application deployed in a cloud environment;

[0042] Figure 4 This is a flowchart of the steps in the configuration phase of a data conversion method provided in this application;

[0043] Figure 5 This is an example diagram of a table data configuration interface provided in this application;

[0044] Figure 6 This is an example diagram of a mapping configuration interface provided in this application;

[0045] Figure 7 This is an example diagram of a node configuration interface provided in this application;

[0046] Figure 8 This is an example diagram of a workflow configuration interface provided in this application;

[0047] Figure 9This is a flowchart of the steps in the application stage of a data conversion method provided in this application;

[0048] Figure 10 This is a schematic diagram of the structure of a data conversion system provided in this application;

[0049] Figure 11 This is a schematic diagram of the structure of a computing device provided in this application;

[0050] Figure 12 This is an example diagram of a computing device cluster provided in this application;

[0051] Figure 13 This is another example diagram of a computing device cluster provided in this application. Detailed Implementation

[0052] First, the application scenarios involved in this application will be explained.

[0053] Relational databases store data in tabular form, with rows and columns representing records and columns representing attributes or fields. Different tables can be related, hence the term "two-dimensional table." While this type of database is easy to manage and operate, if users need to analyze data across multiple dimensions, cross-table joins are required. The more dimensions involved, the worse the query performance becomes, consuming significant computational resources and time. To address this issue, multidimensional databases emerged. Multidimensional databases store data by combining multiple dimensions, where each dimension represents an attribute or feature of the data. This storage method decouples the data, forming multidimensional tables through dimensional combinations. Users can flexibly slice and read data based on the dimensions they want to observe, making data retrieval more flexible, efficient, and intuitive, while also increasing computational efficiency, making it ideal for big data analysis.

[0054] For example, when developing an information management system for a clothing chain store, if a relational database (such as MySQL) is used, many tables need to be created to record clothing styles, sales dates, colors, regions, suppliers, price changes, distributors, etc. As the amount of data increases, the database becomes increasingly cumbersome. Every time a transaction is completed, all these tables need to be modified, which not only affects the performance of the database, but also makes data analysis more difficult. Different user roles have different analytical needs. For example, sales managers are concerned about sales in different regions, product directors are concerned about recent sales comparisons of various styles, and general managers are concerned about the overall sales in each quarter. In this case, cross-table joins need to be performed according to different business needs. The more tables there are, the worse the query performance becomes, which cannot meet the user's needs.

[0055] If a multidimensional database is used, multiple dimensions that can be used to describe the data can be determined based on the data's metadata. The data is then stored in a data cube by combining multiple dimensions. Each cube includes multiple dimensions, and each dimension corresponds to a different feature of the data. One form corresponds to one cube, and each cell in the form is represented by a multidimensional array. Each element in the multidimensional array is the value of each dimension in the cube. By representing the data with a multidimensional array, users can flexibly query data based on dimensions when querying data, thus improving the efficiency of data query.

[0056] For example, Figure 1 This is a schematic diagram of a multidimensional database data model. This cube has multiple "faces," with each face corresponding to a dimension, for example... Figure 1 In the example shown, one face includes various style dimensions, one face includes various date dimensions, and one face includes various region dimensions. Furthermore, each dimension includes multiple members (or values). For example, the style dimension includes styles A to E, the date dimension includes January 2022 to May 2022, and the region dimension includes cities F to H.

[0057] This way, when storing data, Figure 1 Each small cube in the table can correspond to a cell value in the form, and each cell value can be represented by a multidimensional array composed of values ​​from the aforementioned multiple dimensions, for example... Figure 1 The small, dark cube in the middle corresponds to the multidimensional array (May 2022, City F, Style A, Sales Revenue). Sales revenue can also be considered a metric corresponding to the multidimensional array (May 2022, City F, Style A). This way, when a user queries data, each dimension can act as an index. Users can flexibly retrieve the multidimensional array according to their needs, theoretically obtaining sales revenue for any style, time, and region, thus meeting diverse user requirements. It should be understood that... Figure 1 In the example, the metric is sales revenue. In reality, more metrics can be set according to business needs, such as (May 2022, City F, Style A, cost, sales revenue). The specific metric depends on the business needs, and this application does not impose any specific limitations.

[0058] It should be understood that in a multidimensional database, a dimension refers to a specific aspect or attribute used for analyzing and segmenting data. For example, time, region, and style can all be dimensions. A member is a specific instance or value of a dimension. For instance, in the time dimension, May 2022 is a member, and in the region dimension, City F is a member. A measure is used to represent the numerical value or indicator to be analyzed. Multidimensional databases also have many other types of fields, such as hierarchical fields, which represent the hierarchical structure of dimensions; attribute fields, which supplement the dimensions; and calculated fields, which represent the results calculated from other fields. These will not be listed here.

[0059] It should be understood that multidimensional databases store data in cubes, so modifications to the dimensions corresponding to one "face" will not affect other "faces," allowing data updates without impacting the index. Furthermore, direct queries to a single cube provide a fast response because the cube is pre-computed and stored. Unlike relational databases, data queries do not require joining various tables, resulting in very fast data analysis performance. Figure 1 In the example, users can flexibly read the sales figures for style A in City F in May 2022. In fact, based on this data structure, users can flexibly slice and read data based on the dimensions they want to observe, which not only makes data reading more flexible, efficient and intuitive, but also increases the computational efficiency of the data, making it very suitable for big data analysis.

[0060] It should be noted that, Figure 1 The example of a cube is used to illustrate the multidimensional nature of a cube. In actual processing, a data cube is not a physical cube, but an abstract concept. The data model of a multidimensional database can have any number of dimensions or faces. The actual data cube is similar to an n-dimensional hypercube, so it can be sliced, diced, rotated, and drilled from different dimensions to perform multidimensional analysis.

[0061] Given that both databases have their own advantages, business processes may require data to be converted back and forth between the two databases. For example, converting two-dimensional table data in a relational database into multidimensional table data in a multidimensional database, or converting multidimensional table data into two-dimensional table data. This process can be called data conversion or data integration.

[0062] For example, raw data (such as accounting tables and log tables) is typically stored in two-dimensional tables. When performing multidimensional analysis, the raw two-dimensional table data is converted into multidimensional table data. After obtaining the analysis results, these results may also need to be converted back into two-dimensional table data for integration with other systems. Therefore, the need for data conversion between the two databases is very frequent. To achieve this functionality, hybrid databases that support both database functions have emerged. The database engine of a hybrid database is usually equipped with the aforementioned data conversion capabilities, allowing the hybrid database to perform back-and-forth conversion between two-dimensional and multidimensional tables according to business needs.

[0063] However, hybrid databases can only convert two-dimensional and multi-dimensional tables within the same hybrid database, such as converting two-dimensional table A in hybrid database A into multi-dimensional table B in hybrid database A. They cannot convert data between different databases, such as converting two-dimensional table A1 in hybrid database A into multi-dimensional table B1 in hybrid database B. In such cases, developers are required to write custom code to achieve this process, which increases development and operation costs.

[0064] Meanwhile, much data is stored in non-hybrid databases. Since non-hybrid databases lack data transformation capabilities, data transformation can only be achieved by manually writing code. For example, if a user needs to transform table A1 in relational database A into multidimensional table B1 in multidimensional database B, since relational database A and multidimensional database B are non-hybrid databases, their database engines do not have data transformation capabilities. In this case, developers also need to write customized code to achieve this process, which also increases development and operation costs.

[0065] In summary, when converting data between two-dimensional and multi-dimensional tables, it is only possible to rely on the database engine of a hybrid database. This data conversion method, which depends on the database's own capabilities, has significant limitations and cannot meet the actual business needs of users in many scenarios. For example, non-hybrid databases cannot perform data conversion, and hybrid databases cannot perform cross-database conversion. These scenarios require developers to write code to implement data conversion, increasing development and operation costs and reducing the user experience.

[0066] To address the issue that data conversion between two-dimensional and multidimensional tables relies on the database's own capabilities, making data conversion impossible in many scenarios and requiring developers to write code, this application provides a data conversion solution. This solution reads data from a first database, converts it, and stores it in a second database. The entire data conversion process is implemented by the data conversion system, independent of the database's own conversion capabilities. Data conversion can be achieved regardless of whether the first and second databases are hybrid databases or the same database. The conversion process requires no custom code from developers; only some configuration information needs to be entered. This solution resolves the problem of data conversion between two-dimensional and multidimensional tables depending on the database's own capabilities, which prevents data conversion in many scenarios and necessitates developers writing code.

[0067] Figure 2 This is an architecture diagram of a data transformation system provided in this application, such as... Figure 2As shown, the architecture includes a client 100, a data conversion system 200, a first database 300, and a second database 400. The number of clients 100, the first database 300, and the second database 400 in this architecture can be one or more; this application does not impose a specific limitation. Clients 100, the data conversion system 200, the first database 300, and the second database 400 establish a communication connection via a network. This communication connection can be a wired connection or a wireless connection. The network can be the public internet, an internal local area network (LAN), a virtual private network (VPN), a dedicated line such as fiber optic lines, copper wires, or satellite connections, or a wireless network such as wireless LAN (Wi-Fi) or a cellular network; this application does not impose a specific limitation.

[0068] Client 100 is deployed on a terminal device to enable human-computer interaction. Client 100 sends user-input data, request commands, and other information to data conversion system 200 for processing. Terminal devices include personal computers, smartphones, wearable devices, handheld processing devices, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, smart conferencing devices, etc., without specific limitations here. The description of computing devices can be found above and will not be repeated here. Client 100 can also be deployed on a physical server, such as an ARM server or an x86 server; this application does not impose specific limitations on this.

[0069] The data conversion system 200 can be deployed on computing devices, or on a cluster of computing devices composed of multiple computing devices. The computing devices can be bare metal servers (BMS), virtual machines, containers, or storage devices. A BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has an independent basic input / output system (BIOS), hard drive, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process. In specific implementations, the computing device cluster can be a cloud data center, an enterprise private cluster, or a hybrid cloud environment, i.e., a deployment mode that uses both public and private clouds simultaneously; this application does not impose specific limitations.

[0070] The first database 300 and the second database 400 can be deployed on computing devices or clusters of computing devices, or on storage devices or storage arrays. The descriptions of computing devices and clusters of computing devices are as described above and will not be repeated here. Storage devices can be hard disk drives (HDDs), solid-state drives (SSDs), mechanical hard disks (HDDs), USB flash drives (universal serial buses), flash memory, SD cards (secure digital memory cards), Memory Sticks, etc., and this application does not impose specific limitations. Storage arrays can be redundant arrays of independent disks (RAID), network attached storage (NAS), storage area networks (SANs), etc., and this application does not impose specific limitations.

[0071] It should be noted that the first database 300 and the second database 400 are different types of databases, including relational databases, multidimensional databases, and hybrid databases as mentioned above. In other words, this application can convert two-dimensional table data to multidimensional table data, and vice versa; it can perform cross-database conversion within hybrid databases, and also data conversion within non-hybrid databases.

[0072] Optionally, the data transformation system 200, the first database 300, and the second database 400 are deployed on different computing devices or computing device clusters, for example... Figure 1 As shown. Alternatively, the data conversion system 200 and the first database 300 may be deployed on the same computing device or computing device cluster, or the data conversion system 200 and the second database 400 may be deployed on the same computing device or computing device cluster.

[0073] In specific implementation, client 100 can be software or an application running on a terminal device or computing device controlled by the user, such as a personal computer (PC) client, a World Wide Web (WWW) client accessed via a browser, an application (APP) client running on a mobile terminal, or a cloud platform console; this application does not impose specific limitations. The user holding client 100 is a person responsible for managing transaction business within the enterprise, such as a company's finance staff or information technology (IT) staff; this application does not impose specific limitations.

[0074] Optionally, client 100 can be a dedicated client for data transformation, such as a data transformation tool. When a user needs to convert a two-dimensional table to a multi-dimensional table, or vice versa, this tool can be used to perform the data transformation. Specifically, client 100 can perform back-and-forth conversion between two-dimensional and multi-dimensional tables within the same database, as well as between two-dimensional and multi-dimensional tables in different databases.

[0075] Optionally, client 100 may also be a comprehensive client that includes the above-mentioned data conversion functions, such as a database client, or a client for a certain application based on a database, such as a big data analysis client, a financial management client, etc. These comprehensive clients may include not only data conversion functions, but also other functions. For example, a big data analysis client may also include data analysis functions, and a financial management client may also include financial analysis functions, etc. The above examples are for illustration only, and this application does not make any specific limitations.

[0076] Optionally, client 100 can also be a client of a cloud platform, used for users to purchase and rent various cloud services. The data conversion solution provided in this application can be one of the cloud services, and users can purchase the cloud service separately to realize data conversion between two-dimensional tables and multi-dimensional tables; or, the cloud platform provides users with a comprehensive service, and the above-mentioned data conversion function can be a sub-service of the comprehensive cloud service. For example, if a user purchases a database cloud service, the above-mentioned data conversion function can be a sub-service of the cloud service. This application does not make specific limitations.

[0077] The preceding text described in detail the possible deployment methods for client 100, data conversion system 200, first database 300, and second database 400. In actual deployment, flexible deployment can be carried out based on specific application scenarios and business requirements. The following section provides examples of actual deployment methods for client 100, data conversion system 200, first database 300, and second database 400 in specific application scenarios.

[0078] In one application scenario, the client 100, the data conversion system 200, the first database 300, and the second database 400 can be deployed on office equipment within an enterprise. For example, the data conversion system 200 can be deployed on a server or server cluster purchased by the enterprise, the client 100 can be deployed on the enterprise's office computer, the data conversion system 200 can be deployed on the enterprise's server, and the first database 300 and the second database 400 can be deployed on the enterprise's storage device. The first database 300 and the second database 400 can be different databases on the same storage device or different databases on different storage devices. This application does not make any specific limitations.

[0079] In this application scenario, the company's finance staff can use their office computers to run client 100 and send a configuration request to data conversion system 200 through client 100. After configuring the workflow to implement data conversion, data conversion system 200 generates the workflow code corresponding to the workflow. When the user sends a data conversion request to data conversion system 200 through client 100, data conversion system 200 can execute the above workflow code, read source table data from the first database 300, convert it into target table data, and send it to the second database 400 for storage. When the source table is a two-dimensional table, the target table is a multi-dimensional table; when the source table is a multi-dimensional table, the target table is a two-dimensional table.

[0080] In another application scenario, client 100, data transformation system 200, first database 300, and second database 400 can be deployed in a cloud environment. Client 100 serves as the console of the cloud platform. For example, Figure 3 This is an example diagram of a data transformation system deployed in a cloud environment according to this application, such as... Figure 3 As shown, a user can initiate a purchase request for the data conversion cloud service through client 100. After client 100 sends the purchase request to the cloud platform, the cloud platform can grant the user access to the data conversion system 200 to client 100, enabling the user to use the data conversion system 200 to convert back and forth between two-dimensional tables and multidimensional tables through client 100.

[0081] In its implementation, the cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, and security resources, to meet the computing needs of the data conversion system 200 under different scales and loads. These computing resources can be dynamically scaled according to the usage requirements of the data conversion system 200 to ensure its stable operation. The first database 300 and the second database 400 can also be cloud services provided by the data center, such as elastic cloud services or cloud storage services. After purchasing cloud services, users configure the first database 300 and the second database 400, storing data in the databases. When users have data conversion needs, they can use the purchased data conversion cloud service to convert two-dimensional tables in the first database 300 into multi-dimensional tables and store them in the second database 400, or vice versa. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0082] It should be understood that the above application scenarios are for illustrative purposes only. The client 100, data conversion system 200, first database 300 and second database 400 can be flexibly deployed according to actual business needs. They will not be listed here.

[0083] Furthermore, the data conversion system 200 may include a configuration layer 210, an engine layer 220, and a runtime layer 230. The configuration layer 210, engine layer 220, and runtime layer 230 are an exemplary configuration; in specific implementations, the data conversion system 200 may not follow this configuration. Figure 2 The data conversion system 200 is divided into layers as shown. For example, the configuration layer 210 and the engine layer 220 can be merged, or the engine layer 220 can be merged with the running layer 230, or the data conversion system 200 is not divided into layers. This application does not make any specific limitations.

[0084] In practice, the configuration layer 210, engine layer 220, and runtime layer 230 can be implemented in software or hardware. For example, the implementation of the configuration layer 210 will be described below. Similarly, the implementation of the engine layer 220 and runtime layer 230 can refer to the implementation of the configuration layer 210.

[0085] Configuration layer 210, as an example of a software functional unit, may include code running on compute instances. These compute instances may include at least one of a physical host (compute device), a virtual machine, or a container. Furthermore, the compute instances may be one or more. For example, configuration layer 210 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0086] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0087] Configuration layer 210, as an example of a hardware functional unit, may include at least one computing device, such as a server. Alternatively, configuration layer 210 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-a-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0088] The multiple computing devices included in configuration layer 210 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in configuration layer 210 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in configuration layer 210 can be distributed in the same VPC or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0089] It should be noted that, in other embodiments, the configuration layer 210 can be used to execute any step in the data conversion method provided in this application, and the engine layer 220 and the running layer 230 can be used to execute any step in the data conversion method provided in this application. The steps implemented by the configuration layer 210, the engine layer 220 and the running layer 230 can be specified as needed. The configuration layer 210, the engine layer 220 and the running layer 230 respectively implement different steps in the data conversion method provided in this application to realize all the functions of the data conversion system 200.

[0090] The functions of configuration layer 210, engine layer 220 and runtime layer 230 are explained below.

[0091] Configuration layer 210 is used to obtain configuration information input by the second user and generate a workflow.

[0092] Optionally, the first user is the user who uses the workflow to complete the data transformation, and the second user is the user who configures the workflow. For example, the second user could be an IT operations and maintenance personnel or a developer, and the first user could be a finance personnel. In some cases, the second user and the first user can also be the same user. For example, after configuring the workflow, the finance personnel can choose to run the workflow they configured. This application does not impose any specific limitations.

[0093] Optionally, the configuration information input by the second user may include table data configuration information. Table data configuration information includes relevant information about the source and target tables. The source and target tables are tables of different dimensions; for example, the source table may be a two-dimensional table, and the target table a multi-dimensional table. Table data configuration information may include address information for the source and target tables. Address information may include database information of the databases where the source and target tables reside, as well as their location information within the databases. For example, address information may include database links, table names, etc. It may also include field information for the source table and dimension information for the target table. The field information for the source table includes information about the fields involved in the data transformation, such as field names, data types, and lengths. The dimension information for the target table includes information about the dimensions involved in the data transformation, such as dimension names, member names, and the number of dimensions. Table data configuration information is the foundation of data transformation and may include more content; this application does not impose specific limitations.

[0094] It should be understood that the technical solution provided in this application can realize the conversion between two-dimensional tables and multidimensional tables. That is, it can realize the conversion from two-dimensional tables to multidimensional tables and vice versa. Therefore, the source table can be a two-dimensional table or a multidimensional table, and the target table can be a multidimensional table or a two-dimensional table. For ease of explanation, the following will use a two-dimensional table as the source table and a multidimensional table as the target table for illustration, that is, the data conversion from a two-dimensional table to a multidimensional table will be used as an example for illustration.

[0095] Optionally, the configuration information may also include mapping configuration information. Mapping configuration information includes the mapping relationship between the source table and the target table, which includes dimension mapping relationships and member mapping relationships. Dimension mapping relationships include the mapping relationship between data in the source table and dimensions in the target table, while member mapping relationships include the mapping relationship between data in the source table and members in the target table. Member mapping relationships can include various member mapping types, such as conditional mapping, which maps data in the original table that meets a certain condition to a member in the target table. For example, if fields A and B in a two-dimensional table are greater than 0, then member X in the multi-dimensional table is "AB". Another example is exact mapping, which maps each piece of data in the source table to a member in the target table. For example, field A in a two-dimensional table maps to member A in a multi-dimensional table. It should be understood that the above member mapping types are for illustrative purposes only, and the mapping configuration information in this application may include more types, which are not illustrated here.

[0096] It should be understood that this application, through the configuration of member mapping relationships, enables data conversion not only to convert two-dimensional table data to multi-dimensional table data, but also to achieve conditional mapping based on the member mapping type configured by the second user. Currently, database engines of hybrid databases can only convert data formats when converting between two-dimensional and multi-dimensional table data. If conditional mapping is to be implemented, programmers need to write code to achieve it. This application only requires simple configuration by the second user, resulting in low learning costs, low development and maintenance costs for the second user, and improved user experience.

[0097] Optionally, the configuration information may also include node configuration information. Here, a node refers to a node in a workflow. A workflow is a series of ordered tasks or steps arranged in the order configured by the second user. Nodes are the individual tasks or steps that make up the workflow, and nodes can have different types and functions. The second user can use the pre-configured table data configuration information and mapping configuration information to complete the node configuration. In a specific implementation, a node may include at least an extraction node, a transformation node, and a loading node. The extraction node is used to extract data from the source table in the first database 300. The transformation node is used to convert the extracted source table data into target table data. The transformation node can be configured with transformation rules, such as performing the transformation according to conditional mapping or precise mapping. The loading node is used to store the target table data into the target table in the second database 400. Of course, nodes can include more types. For example, nodes can also include extension nodes and verification nodes. Extension nodes are used to extend and supplement the data transformation process. For example, they can call interfaces of other systems, use external services of other systems to enrich the data, or trigger operations of external systems. Verification nodes are used to verify the accuracy and integrity of the data after transformation. Verification nodes can define verification rules and can automatically check whether the data meets the expected format, range, integrity and other standards to ensure data quality. The above examples of node types are for illustration. This application can also include more node types, and the corresponding node configuration information can also include more content, which will not be illustrated here.

[0098] Optionally, the configuration information may also include workflow configuration information, which includes multiple nodes and the execution order between them. Specifically, each workflow includes at least an extraction node, a transformation node, and a loading node. It may also include one or more extension nodes and verification nodes depending on business needs. The number of transformation nodes can also be multiple; for example, transformation node 1 can be used to transform a certain range of two-dimensional table data, while transformation node 2 can be used to transform another range of two-dimensional table data. The second user can configure the workflow according to business needs using the pre-configured table data configuration information, mapping configuration information, and node configuration information.

[0099] Engine layer 220 is used to generate workflow code corresponding to the workflow based on the workflow configured by the second user.

[0100] Optionally, the engine layer 220 may store code templates. The code templates may include multiple predefined code snippets, which may include some configurable parameters. When generating code using the code templates, these parameters are replaced with specific values ​​or logic. In this way, based on the workflow and the code templates, workflow code corresponding to the workflow can be generated.

[0101] In specific implementations, code templates may include node code templates and process code templates. Node code templates generate node code for each step corresponding to a node, while process code templates generate process code corresponding to the execution order between nodes. Workflow code can be generated based on the node code and process code. Node code templates may include one or more code snippets corresponding to each type of node. For example, a transformation node's code template may include a precise mapping code template, used to precisely map fields in a source table to members in a target table. Configurable parameters in this code snippet include the source table name, source field name, target table name, and target member name. A transformation node's code template may also include a conditional mapping code template, used to map fields in a source table to a specific member in a target table under certain conditions. Configurable parameters in this code snippet also include conditions and transformation logic. It should be understood that the above examples are for illustrative purposes only and are not intended to limit the scope of the application. The process code template may include code snippets with multiple execution orders between nodes. For example, sequential execution means that node B will be executed after node A has finished executing. For example, strategy branch execution means that node B will be executed if the output of node A meets a certain condition, and node C will be executed if another condition is met. The above examples are for illustration only and are not specifically limited in this application.

[0102] Furthermore, the aforementioned workflow code needs to be executed by the execution engine before data transformation can be achieved. An execution engine is a component in a programming language or software, responsible for interpreting and executing program code. There are various types of execution engines. To ensure that the generated workflow code can be executed by the engine, the code template can include code templates for multiple engines, with each engine's code template containing code snippets that that engine can execute.

[0103] For example, Figure 2 The database engine shown is the database's own engine. If a database engine is used to perform data transformation, the generated workflow code should be code that the database can execute; for example... Figure 2 The extraction tool engine shown is a software tool for data extraction; therefore, the generated workflow code should be code that this software can recognize and execute. For example... Figure 2The application engine shown is software that processes specific application logic. It typically uses a database to perform business logic, such as big data analysis or financial report analysis. The generated workflow code should be code that the application software can recognize and execute. It should be understood that... Figure 2 The engine shown is for illustrative purposes only and is not intended to be a specific limitation in this application.

[0104] The runtime layer 230 is used to execute workflow code, transform the data in the source table in the first database, generate the target table, and send it to the second database 400 for storage.

[0105] In a specific implementation, the runtime layer 230 may include a workflow space containing multiple workflow codes. After the second user configures multiple workflows based on the configuration layer 210, the data conversion system 200 generates the workflow code corresponding to each workflow based on the engine layer 220 and stores it in the workflow space. The first user can select and execute a workflow code from the workflow space according to business needs. For example, workflow 1 is used to convert a two-dimensional table to a multi-dimensional table, and workflow 2 is used to convert a multi-dimensional table to a two-dimensional table. The first user can choose to execute workflow 1 or workflow 2 in different business scenarios. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0106] In summary, the data conversion system provided in this application reads data from the first database 300, converts the data, and stores it in the second database. The entire data conversion process is implemented by the data conversion system and does not depend on the data conversion capabilities of the database itself. Regardless of whether the first database 300 and the second database 400 are hybrid databases or whether they are the same database, data conversion can be achieved. The conversion process does not require developers to write customized code; only some configuration information needs to be input. This solves the problem that data conversion between two-dimensional table data and multi-dimensional table data depends on the database's own capabilities, which makes data conversion impossible in many scenarios and requires developers to write code.

[0107] Meanwhile, the data transformation process is automatically generated code based on the workflow configured by the second user. This allows the second user to flexibly configure the data transformation process according to their business needs, rather than simply converting two-dimensional table data to multidimensional table data. For example, conditional mapping can be implemented: if a field in the source table meets a certain condition, the corresponding member in the multidimensional table will have a value of 1; otherwise, the member will have a value of 0. Alternatively, a range of fields in the source table can be written to the multidimensional table according to transformation rule X, and another range of fields can be written to the multidimensional table according to transformation rule Y. Regardless of how the second user configures the workflow, the system can automatically generate the workflow code, eliminating the need for developers to write code and significantly reducing development and maintenance costs.

[0108] The data conversion system provided in this application has been described in detail above. The following section will combine... Figures 4-9 The data conversion method provided in this application is explained and described. This method can be applied to, for example... Figure 2 and Figure 3 The data conversion system shown. Among them, Figures 4-8 This refers to the steps and procedures during the configuration phase. Figure 9 It refers to the steps and procedures during the operational phase.

[0109] Figure 4 This is a flowchart illustrating the steps of a data transformation method provided in this application during the configuration phase, as follows: Figure 3 As shown, the method may include the following steps:

[0110] S410: Client 100 sends a table data configuration request to data transformation system 200. This step is based on... Figure 2 The configuration layer 210 shown is implemented.

[0111] In a specific implementation, the table data configuration request may include table data configuration information input by the second user. The table data configuration information includes relevant information about the source table and the target table. The source table and the target table are tables of different dimensions. Here, different dimensions refer to different data dimensions. For example, the source table is a two-dimensional table and the target table is a multi-dimensional table, or the source table is a multi-dimensional table and the target table is a two-dimensional table, or both the source table and the target table are multi-dimensional tables, but the number of dimensions is different. For example, the source table is a three-dimensional table and the target table is a four-dimensional table. This application does not make specific limitations.

[0112] In specific implementation, table data configuration information may include the address information of the source table and the target table. The address information may include the database information of the databases where the source and target tables reside, as well as their location information within the databases. For example, the address information may include database links, table names, etc. Table data configuration information may also include field information of the source table and dimension information of the target table. The field information of the source table includes information about the fields involved in the data transformation, such as field names, data types, and lengths. The dimension information of the target table includes information about the dimensions involved in the data transformation, such as dimension names, member names, and the number of dimensions. Table data configuration information is the foundation of data transformation and may include more content; this application does not impose specific limitations on it.

[0113] For example, Figure 5 This is an example diagram of a table data configuration interface provided in this application. Figure 5 This is an exemplary interface, and this application does not limit it. For example... Figure 5 As shown, the configuration interface may include a database configuration area 510 and a table configuration area 520.

[0114] The database configuration area 510 is used to configure database-related information, including a database name text box, a database type drop-down box, a purpose drop-down box, and a link address text box.

[0115] Database name: Enter the name of the database to be configured;

[0116] Database types: Includes various database types, such as MySQL, Oracle, PostgreSQL, multidimensional databases, etc.

[0117] Purpose: Configures whether this database is used as the source table or the target table. The purpose can include source table, target table, or either source table or target table.

[0118] Link address: Configure the link address of the database. For example, the link address of the MySQL database can be: "jdbc:mysql: / / host:port / dbname", where host is the hostname, port is the port number, and dbname is the database name.

[0119] Table configuration area 520 is used to configure relevant information about the table, including the model name text box, purpose drop-down box, database type drop-down box, database name text box, table name selection box, table header-description text box, table body-dimensional text box, and table body-description text box.

[0120] Model Name: Used to enter the name of the model to be configured. Here, the model can be defined as the data to be transformed. Since the data to be transformed may be a complete two-dimensional table or a part of the fields in the two-dimensional table, the model is used here to represent it.

[0121] Purpose: This indicates whether the input model (i.e., the data to be transformed) is source table data or target table data. The purpose can include source table, target table, or either source table or target table. Figure 5 Let's take the source table as an example for illustration;

[0122] Database types: Includes various database types, such as MySQL, Oracle, PostgreSQL, multidimensional databases, etc.

[0123] Database Name: The name of the database where the model resides. In the specific implementation, since the second user has configured the database name, database type and purpose in the database configuration area 510, after the second user selects the purpose and database type, the database name here can display the names of some databases that have been configured and linked, from which the second user can select.

[0124] Table name: The name of the table containing the model;

[0125] Header - Description: Descriptive information about the model;

[0126] Table Body - Dimension: Field names or dimension-related information. It should be understood that the model refers to the data to be transformed. If it is a two-dimensional table to a multidimensional table, then the model may be a part of the fields in the two-dimensional table. In this case, you can enter the field names of these fields. If it is a multidimensional table to two-dimensional table data, then the model may be a part of the dimensions in the multidimensional table. In this case, you can enter the dimension names of these dimensions.

[0127] Table Body - Description: This section provides descriptive information about the fields, allowing users to provide explanations and descriptions of the fields in the model.

[0128] In simple terms, users first configure the connection address for each database through database configuration area 510. Table configuration area 520 configures databases whose connection addresses have already been configured. This ensures that the models configured in table configuration area 520 have actual read addresses. For example... Figure 5 After configuring the link address of database XXX1, users can use model M1 to refer to some fields in table A1 of database XXX1, which facilitates the subsequent configuration of mapping relationships, nodes and workflows.

[0129] It should be understood that Figure 5 For illustrative purposes only, the table data configuration information may include more information, and the user configuration methods may include even more methods, which are not specifically limited in this application.

[0130] S420: Data conversion system 200 stores table data configuration information.

[0131] It should be understood that after the table data configuration information is stored, users can configure mapping relationships, nodes, and workflows based on the table data configuration information.

[0132] S430: Client 100 sends a mapping configuration request to data conversion system 200.

[0133] In its implementation, the mapping configuration request includes mapping configuration information input by the second user. This information includes the mapping relationship between the source table and the target table, encompassing both dimension mapping and member mapping. Dimension mapping involves the mapping between data in the source table and dimensions in the target table, while member mapping involves the mapping between data in the source table and members in the target table. It should be understood that this explanation uses a two-dimensional source table as an example; therefore, the dimension mapping is the mapping between dimensions of a two-dimensional table and dimensions of a multi-dimensional table. If the source table is a multi-dimensional table, then the dimension mapping is the mapping between data in the target table and dimensions of the source table.

[0134] It should be understood that a two-dimensional table includes rows and columns, with each column corresponding to a field and each row including the field value under each field. A multidimensional table includes multiple dimensions, each dimension can include multiple members, and each combination of dimensions (dimension, member) corresponds to a measure value. The fields in a two-dimensional table are determined based on the fields when the data is stored, while the dimensions and members in a multidimensional table are determined based on the user's actual analysis needs. Therefore, fields and dimensions, field values ​​and dimensions, fields and members, and field values ​​and members can all be flexibly mapped. Secondly, users can flexibly configure them according to their actual business scenarios.

[0135] In specific implementation, dimension mapping relationships can include the mapping relationship between fields in the source table and dimensions in the target table. For example, a two-dimensional table includes fields such as "Date," "Product Type," "Sales Amount," and "Region." The Date field can be mapped to the Time dimension, the Product Type field to the Product dimension, the Region to the Geographic dimension, and the Sales Amount to the Sales Amount metric. Dimension mapping relationships can also include the mapping relationship between field values ​​in the source table and dimensions in the target table. For example, the field values ​​under the Region field in a two-dimensional table can include Asia and Europe. Therefore, the Asia field value can be mapped to the Asia region dimension, and the Europe field value can be mapped to the Europe region dimension. It should be understood that the division of dimensions is determined based on the analysis needs of the second and first users. Therefore, dimensions can be determined based on the fields in the source table, or the field values ​​under some fields in the source table can be generalized to generate a new dimension. These can all be configured according to the actual usage needs of the second and first users, and this application does not impose specific limitations.

[0136] Similarly, member mapping relationships can include the mapping relationship between fields in the source table and members in the target table. For example, the field values ​​under the region field of a two-dimensional table can include Asia and Europe. The region field can be mapped to the geographical dimension, and the values ​​of the Asia and Europe fields can be mapped to Asian and European members under the geographical dimension. Member mapping relationships can also include the mapping relationship between field values ​​in the source table and members in the target table. For example, a two-dimensional table includes fields for China, Japan, and India, and a multidimensional table includes an Asian dimension, which includes Chinese, Japanese, and Indian members. In this case, the China field in the two-dimensional table can be mapped to the Chinese members under the Asian dimension in the multidimensional table. It should be understood that the division of members is also determined based on the analysis needs of the second and first users. Therefore, members can be fields in the source table, merged or decomposed values ​​of fields in the source table, field values ​​in the source table, and merged or decomposed values ​​of field values. These can all be configured according to the actual usage needs of the second and first users, and this application does not impose specific limitations.

[0137] Furthermore, member mapping relationships can include basic mapping and conditional mapping. Basic mapping refers to mapping source table data to a multidimensional table, while conditional mapping refers to mapping source table data to a multidimensional table when the source table data meets preset conditions. Specifically, when the source table data meets the preset conditions, the source table data has a member mapping relationship with the first member in the multidimensional database; when the preset conditions are not met, the source table data has a member mapping relationship with the second member in the multidimensional database.

[0138] For example, if fields A and B in a two-dimensional table are greater than 0, member X in the multidimensional table is "X"; otherwise, member X is "0". It should be understood that when converting two-dimensional table data to multidimensional table data, or vice versa, the first user may not only want to convert the dimensions of the data but also need to customize conversion rules to meet their specific output processing requirements. By configuring member mapping relationships, data conversion can not only achieve the conversion from two-dimensional table data to multidimensional table data but also achieve conditional mapping based on the member mapping type configured by the second user. Currently, hybrid database engines can only achieve the above basic mapping when converting between two-dimensional and multidimensional table data. To achieve conditional mapping, programmers need to write code. This application only requires simple configuration by the second user, resulting in low learning and development / maintenance costs and improving the user experience.

[0139] For example, Figure 6 This is an example diagram of a mapping configuration interface provided in this application. Figure 6 This is an exemplary approach, and this application does not impose specific limitations. For example... Figure 6 As shown, the configuration interface may include a mapping configuration area 610. The mapping configuration area 610 may include a mapping name text box, a source table model selection box, a target table model selection box, a dimension mapping configuration area, a member mapping configuration area, etc.

[0140] Mapping Name: Used to enter the name of the configured mapping relationship.

[0141] Source table model: Enter the name of the source table model. The models that can be entered in this area are... Figure 5 The configuration interface shown shows the configured model.

[0142] Target Table Model: Enter the name of the target table model. The models that can be entered in this area are... Figure 5 The configuration interface shown shows the configured model.

[0143] Dimension Mapping: Input dimension mapping relationship. The second user can select source table data and target table dimensions to establish the dimension mapping relationship, for example... Figure 6 The source table data shown is a region field, and the target table dimension is a geographic dimension.

[0144] Member Mapping: The second user can input the member mapping relationship, select source table data and target table dimensions to establish the member mapping relationship, choose the mapping type for that member mapping relationship, and then input the parameters required for that mapping type. For example... Figure 6 As shown, the source table data is a "China" field value, the target table member is "China", and the mapping type is exact mapping; the source table data is a date field, and the target table member is "first quarter". It should be understood that the source table data can be selected by the user, while the target table dimensions and members can be automatically recommended by the system. Examples of several member mapping types are given below for illustration.

[0145] Exact mapping means that the data in the source table is accurately mapped to the target table, which can also be understood as the basic mapping mentioned above.

[0146] Range mapping refers to mapping data within a range in the source table to members in the target table, rather than matching individual values. For example, a two-dimensional table may include a score field where the field value is a specific score value, and a multidimensional table may include a grade dimension with members of grade A, grade B, and grade C. Range mapping would be as follows: when the score value is 90-100, it is mapped to grade A members in the multidimensional table; when the score is 80-90, it is mapped to grade B members in the multidimensional table; and when the score is below 80, it is mapped to grade C members in the multidimensional table.

[0147] Set mapping indicates that when data in a source table belongs to a certain set, the source table data can be mapped to members of the target table. It should be understood that range mapping primarily applies to numeric source table data, while range mapping primarily applies to non-numeric source table data. For example, if the source table data includes a region field, and the multidimensional table data includes a region dimension with the member "Asia," and a second user configures a set A: "China, Japan, India, South Korea," then the field values ​​in the source table data belonging to set A can be mapped to the dimension member "Asia" in the multidimensional table.

[0148] Multidimensional mapping refers to mapping data from multiple source tables to multiple members of a multidimensional table.

[0149] Conditional mapping refers to mapping data from a source table to members of a target table when the data in the source table meets a certain condition.

[0150] Fuzzy mapping represents a certain degree of imprecise matching. For example, the data in the source table and the values ​​of the members in the target table are not exactly the same, but they have a certain similarity, and mapping can still be performed.

[0151] exist Figure 6 In the example shown, the second user configured an M1-M2 mapping, which refers to the data transformation where the source table is M1 and the target table is M2. Tables M1 and M2 are configured by the second user through... Figure 5 The interface shown already includes tables with linked and configured tables. The dimension mappings include mappings between the region field and the geographic dimension, as well as mappings between the date field and the time dimension. Specifically, under the region-geographic dimension mapping, the China field maps to Asian members, with a mapping type of exact mapping. Since exact mapping means the source table data is accurately mapped to the target dimension, this parameter does not need to be filled in. Under the date-time dimension mapping, the date field maps to members for the first quarter, with a mapping type of range mapping. The parameter is the range corresponding to this range mapping, that is, the source table data from January to March is mapped to the members for the first quarter of the target table. Figure 6 For illustrative purposes only, if the second user selects another mapping type, they can also enter the parameters required for that mapping type in the corresponding parameter area. For example, conditional mapping requires the input of conditions, and set mapping requires the input of sets. These will not be explained one by one here.

[0152] It should be understood that Figure 6 An example was given using two-dimensional mapping and two-member mapping. If the source table M1 has more data that needs to be mapped, the second user can also... Figure 6 The addition and deletion of controls, as well as the addition of dimension and member mappings, are shown in the examples; further examples will not be provided here.

[0153] It should be understood that Figure 6For illustrative purposes, when performing member mapping, the second user can select various mapping types according to actual business needs to meet the various mapping requirements of the second user. Through the data transformation method provided in this application, the second user can directly obtain the desired multidimensional table, instead of transforming the source table and then making various adjustments to the transformed multidimensional table based on business needs, thereby improving the user experience of the second user.

[0154] S440: Data conversion system 200 storage mapping configuration information.

[0155] It should be understood that after the mapping configuration information is stored, the second user can complete the configuration of nodes and workflows based on the table data configuration information and the mapping configuration information.

[0156] S450: Client 100 sends a node configuration request to data conversion system 200.

[0157] In the specific implementation, the node configuration request includes node configuration information configured by the second user. A node refers to a node in a workflow. A workflow is a series of ordered tasks or steps arranged in the order configured by the second user. Nodes are the individual tasks or steps that make up the workflow, and nodes can have different types and functions. The second user can complete the node configuration using the pre-configured table data configuration information and mapping configuration information.

[0158] In this embodiment, the nodes may include extraction nodes, transformation nodes, and loading nodes. The extraction node extracts data from a source table in the first database 300. The transformation node converts the extracted source table data into target table data. The transformation node can be configured with transformation rules, such as performing transformations using conditional mapping or exact mapping. The loading node stores the target table data in a target table in the second database 400. The extraction and loading nodes can be configured based on table data configuration information, and the transformation node can be configured based on mapping configuration information.

[0159] In one possible implementation, the node configuration information of the extraction node may include the table data configuration information of the source table and the extraction information. The table data configuration information of the source table is used to obtain the data to be extracted and transformed, and the extraction information is used to determine the extraction method, such as the extraction frequency, including one-time reading, periodic reading, etc., and authentication information, including the username and password of the database, etc. The specific content included in the node configuration information can be determined based on the actual business scenario, and will not be illustrated one by one here.

[0160] It should be understood that the second user has already configured the address and field information of multiple source and target tables on the S410, which is equivalent to configuring the extraction address and extraction range. Therefore, when configuring extraction nodes, the second user can directly select the configured table name and then configure some custom configurations required to perform the extraction action, such as extraction frequency and extraction conditions. This simplifies the configuration process and avoids repetitive configuration of source table addresses. Furthermore, if the second user needs to modify the extraction address and extraction range, they can directly modify the table configuration information. All extraction nodes using this table configuration information can be modified uniformly without needing to modify each extraction node individually, thus improving the user experience for the second user.

[0161] In one possible implementation, the node configuration information of the transformation node may include mapping configuration information and transformation rules. The mapping configuration information includes dimension mapping relationships and member mapping relationships. Based on the mapping configuration information, it can be determined which dimension or member of the target table corresponds to the source table data. The transformation rules include the generation method of dimension or member values. Based on the transformation rules, it can be determined how the source table data generates the transformed value and writes it to the corresponding dimension or member. For example, a fixed value is written to the corresponding dimension of the source table, or a function value is generated based on the source table data and written to the corresponding member of the source table. This application does not impose specific limitations. In short, the mapping configuration information determines the storage location of the transformed value, and the transformation rules determine how the transformed value is generated. Therefore, when configuring the transformation node, it is only necessary to select the already configured mapping configuration information and then configure the transformation rules.

[0162] Optionally, the transformation rules may include basic transformation rules. Basic transformation rules refer to obtaining the transformation values ​​of the source table data by querying a transformation value table. The transformation value table includes transformation values ​​corresponding to different source table data. Under basic transformation rules, the transformation values ​​can be obtained by querying the table, and then written to the corresponding dimensions, members, or measures of the source table data according to the mapping configuration information. It should be understood that by configuring a transformation value table to determine the transformation values, the system can directly complete the transformation by querying the table without additional calculations or processing, thus improving transformation efficiency.

[0163] For example, the source table includes a region code field, and the target table includes a region name dimension. There is a dimension mapping relationship between the two. If the basic transformation rules are used, a second user can configure a transformation value table. This transformation value table can be a region code table, which includes the relationship between region codes and region names. By querying this region code table, multiple field values ​​under the region code field in the source table can be converted into region names. The converted region names can be used as members under the region name dimension in the target table. The entire transformation process does not require calculation and can be completed with a simple table lookup, thus improving the efficiency of data transformation.

[0164] Optionally, the transformation rules may include fixed-value transformation rules, which refer to the transformation values ​​of the source table data into label values, which can be numerical or text. Specifically, fixed-value transformation rules include a label value table, which records the label values ​​corresponding to the source table data. For example, if the source table data is temperature values ​​and the corresponding target table dimension members are high temperature and low temperature, the label value table may include the temperature ranges under which the label value is high temperature and the temperature ranges under which the label value is low temperature. In this example, the member mapping is a range mapping, and the transformation rule is a fixed-value transformation rule. The above example is for illustration only and is not specifically limited in this application.

[0165] Optionally, the transformation rules may include combined transformation rules, which can be a combination of multiple transformation rules to achieve mapping. For example, the source table data corresponding to this dimension may use basic transformation rules under certain conditions and fixed value transformation rules under other conditions. For instance, a region code table may be used to complete the transformation in the Asian region, while a fixed text label "Americas" may be used uniformly to complete the transformation in the Americas region.

[0166] It should be understood that the above conversion rules are some examples provided in this application. This application can also provide more types of conversion rules for the second user to choose from, such as calculation conversion rules, which include calculation formulas for the source table data, and the conversion values ​​obtained based on the calculation formulas and written into the corresponding dimensions or members of the source table data; or slicing rules, which include slicing methods for slicing the source table data, such as slicing dates by month. The above examples are for illustration. Based on the actual business scenarios of the second user, multiple conversion rules can be provided to the second user. This application does not make any specific limitations.

[0167] For example, Figure 7 This is an example diagram of a node configuration interface provided in this application. Figure 7 This is an exemplary approach, and this application does not impose specific limitations, such as Figure 7 As shown, the configuration interface may include a node configuration area 710. The node configuration area 710 may include a conversion node name text box, a mapping name drop-down box, a description text box, and a conversion rule configuration area.

[0168] Transformation Node Name: Used to configure the name of the transformation node. Figure 7 Let's take the Z1 as an example.

[0169] Mapping Name: Used to enter the mapping name, which is the name of the mapping configuration information that the second user has already configured, for example... Figure 6 The mapping configuration information shown for M1-M2 includes both dimension mapping relationships and member mapping relationships.

[0170] Description: Used to enter explanatory information.

[0171] Transformation rule configuration: Used to configure the transformation rules corresponding to each dimension. For example, Figure 7 In the middle, the second user can view the dimensions and members with configured mapping relationships according to the M-M2 mapping configuration information. The second user can configure transformation rules according to the dimension. For example, the geographic dimension can be configured with general transformation rules, using the mapping set of the regional value table, and selecting to slice. For the time dimension, fixed value transformation rules can be configured, using the mapping set of the label value table, and selecting not to slice.

[0172] It should be understood that the second user configured the dimension and member mapping relationships of the data model on S420, which is equivalent to determining the source of the values ​​for each dimension and member. Therefore, when configuring transformation nodes, the second user can directly select the configured mapping relationships and then add transformation rules, that is, how to generate transformation values ​​based on the source data and write them to each dimension and member. Simply put, the mapping configuration input on S420 determines which data's transformation value from the source table is written to which position in the target table, while the transformation node configuration input on S440 determines how the transformation values ​​are calculated. This separate configuration method simplifies node configuration; only the transformation rules need to be configured. If the second user needs to reuse the same mapping relationship on different transformation nodes, they do not need to repeatedly configure the mapping relationship. Furthermore, when updating the mapping relationship, only the mapping configuration data needs to be modified; the configuration information of each transformation node does not need to be changed, making the system easier to maintain and update.

[0173] In one possible implementation, the configuration information of the loading node may include table data configuration information of the target table and loading information. The table data configuration information of the target table is used to determine the storage location of the target table, and the loading information is used to determine the loading method, such as full loading (loading all target table data generated by the conversion node into the target table), incremental loading (loading newly added target table data into the target table), merging (merging the target table data generated by the conversion node with existing data in the target table), or replacement (replacing data in the target table with data generated by the conversion node). The examples of loading information above are for illustrative purposes only and are not intended to limit the scope of this application.

[0174] It should be noted that the configuration information of the loading node may also include the temporary storage address of the target table data. If the second user and the first user need to write the converted table data directly to the target table, then only the table data configuration information and loading information of the target table need to be configured. If the second user and the first user need to temporarily store the converted target table data in a certain place and wait for a period of time before storing it in the second database 400 where the target table is located, then the configuration information of the loading node may also include the temporary storage address. The loading information may be configured with the storage time in the target table. This application does not make specific limitations.

[0175] It should be understood that the second user has configured the address and field information of multiple source and target tables on the S410, which is equivalent to configuring the location where the target table data needs to be loaded. Therefore, when configuring the extraction node, the second user can directly select the configured table name and then configure some loading methods to complete the configuration of the loading node. This configuration method simplifies the configuration process of the loading node and avoids repetitive configuration of the target table address. Furthermore, if the second user needs to modify the loading address, they can directly modify the table configuration information. All loading nodes using this table configuration information can be modified uniformly without having to modify each loading node individually, thus improving the user experience for the second user.

[0176] Optionally, nodes may also include extension nodes and verification nodes. Extension nodes are used to extend and supplement the data transformation process, such as calling interfaces of other systems, using external services of other systems to enrich the data, or triggering operations of external systems. Verification nodes are used to verify the accuracy and integrity of the data after transformation. Verification nodes can be defined with verification rules and can automatically check whether the data meets the expected format, range, integrity and other standards to ensure data quality. The above examples of node types are for illustration. This application may also include more node types, and the corresponding node configuration information may also include more content, which will not be illustrated here.

[0177] It needs to be explained that, Figure 7 An example configuration interface for a conversion node is provided; configuration interfaces for other nodes can be found in the example. Figure 7 For example, the extraction node can include the transformation node name, data link name or type, mapping name, extraction method, etc.; the loading node can include the loading node name, data link name or type, mapping name, loading method, etc. Similarly, the configuration interface for the extension node and the verification node can be obtained. Here, we will not give examples of the configuration interface for each type of node.

[0178] S460: Configuration information for the 200 storage nodes of the data conversion system.

[0179] It should be understood that after the node configuration information is stored, a second user can complete the workflow configuration based on the table data configuration information, mapping configuration information, and node configuration information.

[0180] S470: Client 100 sends a workflow configuration request to data conversion system 200.

[0181] In the specific implementation, the workflow configuration request includes workflow configuration information input by the second user. Specifically, client 100 can receive the second user's operation information, including user drag-and-drop and node connection operations. Based on this operation information, workflow configuration information is generated. This workflow configuration information includes multiple nodes and their execution order. Each workflow includes at least an extraction node, a transformation node, and a loading node. It can also be configured with one or more extension nodes and verification nodes according to business needs. The number of transformation nodes can also be multiple; for example, transformation node 1 can be used to transform a certain range of two-dimensional table data, while transformation node 2 can be used to transform another range of two-dimensional table data. The second user can complete the workflow configuration using the pre-configured table data configuration information, mapping configuration information, and node configuration information, according to business needs.

[0182] For example, Figure 8 This is an example diagram of a workflow configuration interface provided in this application. The workflow configuration interface may include a name configuration area 810, a selection area 820, a toolbar 830, and an arrangement area 840.

[0183] The name configuration area 810 includes a workflow name text box, a mapping name drop-down box, and an effective option drop-down box. The workflow name is used to enter the name of the workflow, for example... Figure 8 The G1 shown is used to select the mapping configuration information that the second user has already configured, such as... Figure 6 The M1-M2 mapping is shown. The activation option is used to select whether to activate this workflow.

[0184] Selection area 820 is used to display multiple selectable nodes to the second user, including extraction nodes, transformation nodes, loading nodes, expansion nodes, and verification nodes. It should be understood that after the second user selects a mapping name in the name configuration area 810, selection area 820 can display multiple nodes created using that mapping configuration information.

[0185] Toolbar 830 may include multiple arrangement tools for a second user to use in arranging workflows, such as... Figure 8 The selection tool shown is used to select nodes, the move tool is used to move nodes, the settings tool is used to modify the configuration information of nodes, the delete tool is used to delete nodes, the add tool is used to add nodes, the connect tool is used to connect nodes, and the sort tool is used to sort nodes.

[0186] It should be understood that by orchestrating nodes to generate workflow configuration information, a second user can flexibly orchestrate multiple workflows to meet the data transformation needs of different application scenarios. Moreover, the orchestration process is simple and fast, does not require developers to write program code, and has a low learning cost for the second user.

[0187] S480: Data conversion system 200 stores workflow configuration information and generates workflow code.

[0188] In practice, the generated workflow code can be stored in the workflow space, which includes one or more workflow codes configured by the second user. Each workflow code can be distinguished by a workflow identifier, so that when the first user inputs a data conversion request, the workflow code that the first user needs to use can be determined based on the workflow identifier carried in the data conversion request.

[0189] Optionally, since the workflow code needs to be executed by the execution engine before data transformation can be achieved, the execution engine is a component in a programming language or software responsible for interpreting and executing program code. There are various types of execution engines. To ensure that the generated workflow code can be executed by the engine, the code template can include code templates for multiple engines, with each engine's code template containing code snippets that that engine can execute.

[0190] For example, Figure 2 The database engine shown is the database's own engine. If a database engine is used to perform data transformation, the generated workflow code should be code that the database can execute; for example... Figure 2 The extraction tool engine shown is a software tool for data extraction; therefore, the generated workflow code should be code that this software can recognize and execute. For example... Figure 2 The application engine shown is software that processes specific application logic. It typically uses a database to perform business logic, such as big data analysis or financial report analysis. The generated workflow code should be code that the application software can recognize and execute. It should be understood that... Figure 2 The engine shown is for illustrative purposes only and is not intended to be a specific limitation in this application.

[0191] In practice, the system can pre-obtain the first database and the engine types supported by the second database from the second user, and then use the code template corresponding to that engine type to generate workflow code. The system can pre-store code templates corresponding to multiple engine types. If the second user modifies the engine type, the system can also switch to the corresponding code template to generate workflow code that the engine can execute.

[0192] Optionally, the data conversion system 200 may store code templates. The code templates may include multiple predefined code snippets, which may include some configurable parameters. When generating code using the code templates, these parameters are replaced with specific values ​​or logic. In this way, based on the workflow and the code templates, workflow code corresponding to the workflow can be generated.

[0193] In specific implementation, the code template may include node code templates and process code templates. The node code template may include one or more code sub-templates for each node. After determining the multiple nodes included in the workflow based on the workflow configuration information, code snippets corresponding to each node can be generated based on the code sub-templates of each node. The process code template is used to generate process code snippets corresponding to the execution order between nodes. Workflow code can be generated based on the code snippets corresponding to each node and the process code snippets.

[0194] Optionally, each node can correspond to one or more code sub-templates. For example, an extraction node can correspond to multiple extraction code sub-templates based on different extraction methods; a transformation node can determine multiple transformation code sub-templates based on different transformation rules and different mapping types; a loading node can determine multiple loading code sub-templates based on different loading methods; a verification node can determine multiple verification code sub-templates based on different verification rules; and an extension node can determine multiple extension code sub-templates based on the extended content. This application does not impose specific limitations. The system can pre-define the code sub-templates for each node by combining various application scenarios and analyzing historical data, conducting surveys, etc.

[0195] In practice, each code sub-template can include multiple atomic code templates. Based on the function that each code sub-template needs to implement, multiple atomic steps required to implement the function are determined. Each atomic step can correspond to an atomic code template. These atomic steps are the steps that the node may use when processing the task. In this way, after the second user configures the node, the steps can be decomposed according to the processing flow corresponding to the node configured by the second user. Each step can obtain the corresponding atomic code template and then generate the code fragment corresponding to the node.

[0196] For example, when an extraction node performs a data extraction task, atomic steps may include establishing a connection with the source database, executing an SQL query, or calling an API to read data from the source table. Establishing a connection with the source database can correspond to an atomic code template, in which the configurable parameter is the link address of the source database. Executing an SQL query can correspond to an atomic code template, in which the configurable parameters are the field information of the source table data to be extracted, as well as the extraction time, extraction frequency, etc.

[0197] For example, when a transformation node performs a data transformation task, atomic steps may include data cleaning, data transformation, and data calculation. Each atomic step can be configured with one or more atomic code templates. For instance, a data transformation step may include multiple atomic code templates because the data transformation process involves multiple mapping types, such as conditional mapping, exact mapping, range mapping, and fuzzy mapping. Conversion rules also include multiple types, such as general conversion rules, fixed-value conversion rules, combined conversion rules, and computational conversion rules. Each mapping type and each conversion rule can be configured with a corresponding atomic code template. The configurable parameters in this code template are determined based on the parameters of the corresponding mapping type or conversion rule. For example, in the atomic code template for range mapping, the range configured by the second user is a configurable parameter; similarly, in the fixed-value conversion rule, the fixed-value table input by the second user is a configurable parameter. Likewise, data cleaning, data calculation, and other steps can be broken down into atomic steps to obtain the code template corresponding to each atomic step. The code template examples for the transformation node described above are for illustrative purposes only and are not intended to limit the scope of the application.

[0198] Optionally, the process code template may include process code fragments representing multiple possible execution orders between nodes. For example, sequential execution means that node B will be executed after node A has finished executing. Alternatively, strategy branch execution means that node B will be executed if the output of node A meets a certain condition, and node C will be executed if another condition is met. The above examples are for illustration only and are not specifically limited in this application.

[0199] It should be understood that when determining the workflow code, each node in the workflow is broken down into atomic steps to obtain multiple atomic steps for each node. Then, the atomic code template corresponding to each atomic step is obtained to obtain the node code fragment corresponding to each node. Based on the execution order between each node in the workflow, the process code fragment is obtained. The workflow code is obtained based on the node code fragment and the process code fragment.

[0200] The preceding text describes in detail the steps of the data transformation method provided in this application during the configuration phase. The following section will combine this with... Figure 9 The steps and procedures of the data conversion method provided in this application during the application stage are explained.

[0201] Figure 9 This is a flowchart illustrating the steps of a data transformation method provided in this application during the application phase, as shown below. Figure 9 The method may include the following steps:

[0202] S910: Client 100 sends a data conversion request to data conversion system 200. This step is based on... Figure 2 The runtime layer 230 is shown.

[0203] In practice, the data transformation request may include a workflow identifier, which instructs the data transformation system 200 to execute the code of the corresponding workflow to complete the data transformation process. It should be understood that after the second user configures multiple workflows, the data transformation system 200 generates workflow code for each workflow. The first user can select the workflow code to execute based on business needs. The second user and the first user can also be the same user. For example, workflow 1 is used to transform a two-dimensional table into a multi-dimensional table, and workflow 2 is used to transform a multi-dimensional table into a two-dimensional table. The first user can choose to execute workflow 1 or workflow 2 in different business scenarios. When the first user selects to execute workflow 1, the client 100 can send a data transformation request carrying the identifier of workflow 1 to the data transformation system 200.

[0204] S920: Data conversion system 200 executes workflow code. This step is based on... Figure 2 The runtime layer 230 is shown.

[0205] In its implementation, the data conversion system 200 can obtain and execute the workflow code corresponding to the identifier in the data conversion request. The workflow code is code that the execution engine can recognize. The data conversion system 200 can call the execution engine to execute the workflow code, such as through a database engine, a data conversion tool engine, or an application engine. Data conversion no longer depends on the database engine; regardless of the type of database (first or second), or the engine it supports, data conversion can be achieved. Furthermore, the configuration and application processes of the data conversion do not require developer involvement, reducing the learning cost for the second user and lowering the system's development and maintenance costs.

[0206] S930: Data transformation system 200 obtains source table data from first database 300. This step is based on... Figure 2 The runtime layer 230 is shown.

[0207] In specific implementation, the workflow is configured with at least an extraction node, a transformation node, and a loading node. Executing the workflow code can realize the steps configured in the extraction node, obtain source table data from the first database 300, and store the extracted source table data in the storage device of the data transformation system 200 for data transformation. The storage device can be an internal storage device of the data transformation system 200, an external storage device, or a cloud storage device. This application does not make any specific limitations.

[0208] S940: Data transformation system 200 transforms the source table data to generate the target table data. This step is based on... Figure 2 The runtime layer 230 is shown.

[0209] In the specific implementation, the workflow code includes the code for the transformation nodes. The S940 can execute the code for the transformation nodes to realize data transformation. The description of the transformation nodes can be found in the previous content, and will not be repeated here.

[0210] S950: Data conversion system 200 sends target table data to second database 400. This step is based on... Figure 2 The runtime layer 230 is shown.

[0211] In specific implementation, the data conversion system 200 can directly send the generated target table data to the second database 400, or it can first store it in the aforementioned storage device of the data conversion system 200. The timing of S950 is determined according to the loading node configured by the second user. This application does not make any specific limitations.

[0212] In its implementation, the data conversion system 200 can display the running status of the workflow to the first user, such as initialization, submission, running, completion, termination, and failure. The first user can monitor the progress of the data conversion based on the running status. It can also display more relevant information about the workflow's operation, which is not specifically limited in this application.

[0213] In specific implementation, when the data conversion system 200 communicates with the first database 300 and the second database 400, it can transmit source table data and target table data in the form of messages. The message may include a certain message format, which defines the structure and rules of data transmission, enabling both communicating parties to understand and parse the data, and ensuring correct data transmission. For example, the message may include a message header, address information, and payload. The message header includes the message's metadata information, including message type, length, version number, etc. The address information includes the address information of the first database 300 and the second database 400, which is the relevant information of the source table address and the target table address. The payload refers to the data to be transmitted. For example, when S930 extracts source table data, the payload includes the source table data; when S950 sends target table data, the payload includes the target table data. Before data communication begins, the data conversion system 200 can convert source table data or target packet data into data conforming to the message format and transmit it to the first database 300 or the second database 400 via the network. This makes the data conversion independent of the database engine's capabilities, and data conversion can be achieved regardless of what kind of database the first database 300 or the second database 400 is.

[0214] It should be understood that the above explanation uses the conversion of a two-dimensional table to a multidimensional table as an example. In fact, after configuring the table data, mapping relationship and nodes through the data conversion method provided in this application, the workflow for converting a multidimensional table to a two-dimensional table can also be configured. The configuration methods of the two are similar, and will not be elaborated here.

[0215] Furthermore, due to the existence of configuration data, after the two-dimensional table data is converted into multidimensional table data, the mapping relationship between the two-dimensional table and the multidimensional table has already been established because the configuration data is equivalent to the multidimensional table data already being converted back into two-dimensional table data. At this time, it is not necessary to execute the workflow again. It can also be achieved directly by modifying the multidimensional table data format, which is more convenient for the first user and further improves the user experience.

[0216] In summary, the data conversion method provided in this application reads data from the first database 300, converts the data, and stores it in the second database. The entire data conversion process is implemented by the data conversion system and does not depend on the data conversion capabilities of the database itself. Regardless of whether the first database 300 and the second database 400 are hybrid databases or whether they are the same database, data conversion can be achieved. The conversion process does not require developers to write customized code; only some configuration information needs to be input. This solves the problem that data conversion between two-dimensional table data and multi-dimensional table data depends on the database's own capabilities, which makes data conversion impossible in many scenarios and requires developers to write code.

[0217] Meanwhile, the data transformation process is automatically generated code based on the workflow configured by the second user. This allows users to flexibly configure the data transformation process according to their business needs, rather than simply converting two-dimensional table data to multidimensional table data. For example, conditional mapping can be implemented: if a field in the source table meets a certain condition, the corresponding member in the multidimensional table will have a value of 1; otherwise, the member will have a value of 0. Alternatively, a range of fields in the source table can be written to the multidimensional table according to transformation rule X, and another range of fields can be written to the multidimensional table according to transformation rule Y. Regardless of how the user configures the workflow, the system can automatically generate the workflow code, eliminating the need for developers to write code and greatly reducing development and maintenance manpower costs.

[0218] The above text combined Figures 4-9 The data conversion method provided in this application has been explained and described below, in conjunction with... Figure 10 The internal unit module division of the data conversion system provided in this application is illustrated by way of example.

[0219] Figure 10 This is a schematic diagram of the structure of a data conversion system provided in this application. The data conversion system can be... Figures 2-9 The data conversion system 200 in the middle should be understood. Figure 10 This is an exemplary division method for the data conversion system 200. The data conversion system 200 may also include more or fewer modules, such as... Figure 10 Merge some modules in the middle, or... Figure 10 Some modules within the framework are further subdivided, but examples will not be provided here.

[0220] like Figure 10 As shown, the data conversion system 200 may include a configuration unit 211, a code generation unit 221, an acquisition unit 231, and an execution unit 232.

[0221] The acquisition unit 231 is used to acquire a data transformation request input by the user. The data transformation request includes a workflow identifier and is specifically used to implement... Figure 9 S910 and its optional steps in the embodiments.

[0222] Acquisition unit 231 is used to acquire the workflow corresponding to the workflow identifier. The workflow includes extraction nodes, transformation nodes, and loading nodes. The extraction node includes the read address of the source table in the first database, the transformation node includes data transformation rules, and the loading node includes the storage address of the target table in the second database. The source table is a two-dimensional table, and the target table is a multi-dimensional table, or the source table is a multi-dimensional table, and the target table is a two-dimensional table. Specifically, it is used to implement... Figure 9 S910 and its optional steps in the embodiments.

[0223] Execution unit 232 is used to execute the workflow. Based on the extraction node, it reads source table data from the first database; based on the transformation node, it converts the source table data into target table data; and based on the loading node, it stores the target table data in the second database. Specifically, it is used to implement… Figure 9 S920 to S950 and their optional steps in the embodiments.

[0224] Configuration unit 211 is used to obtain configuration information input by the user. The configuration information includes workflow configuration information and the execution engine types supported by the first and second databases. Specifically, it is used to implement... Figure 4 S410 to S470 and their optional steps in the embodiments.

[0225] Code generation unit 221 is used to determine code templates based on execution engine type. The data transformation system includes code templates for various execution engine types, and each node includes one or more code templates specifically used for implementation. Figure 4 S480 and its optional steps in the embodiments.

[0226] Code generation unit 221 is used to determine the workflow code based on the code template, specifically for implementing... Figure 4 S480 and its optional steps in the embodiments.

[0227] Execution unit 232 is used to call the execution engine to execute the workflow code, specifically for implementing... Figure 9 S920 to S950 and their optional steps in the embodiments.

[0228] In one possible implementation, the configuration information includes mapping configuration information, which includes dimension mapping relationships and member mapping relationships. The dimension mapping relationship includes the mapping relationship between the source table data and the dimensions of the multidimensional table, and the member mapping relationship includes the mapping relationship between the source table data and the members of the multidimensional table.

[0229] In one possible implementation, the member mapping relationship includes condition mapping, and the mapping configuration information includes preset conditions. The condition mapping is used to indicate that there is a member mapping relationship between the source table data and the members of the multidimensional table when the source table data meets the preset conditions. The preset conditions include one or more of numerical range, set, and fuzzy matching.

[0230] In one possible implementation, the configuration information includes node configuration information, and the configuration information of the transformation node includes mapping configuration information and data transformation rules; the data transformation rules include: performing data transformation on the source table data based on the transformation value table to obtain the target table data, wherein the transformation value table includes transformation values ​​corresponding to different source table data.

[0231] In one possible implementation, the data transformation rules include: transforming source table data based on transformation labels to obtain target table data, wherein the transformation labels include fixed value labels and text value labels.

[0232] In one possible implementation, the first database is deployed on the first computing device, the second database is deployed on the second computing device, and the execution unit 232 is used to obtain the communication message format supported by the second database, generate a message of target table data based on the communication message format, and send the message of target table data to the second database.

[0233] In summary, the data transformation system provided in this application reads data from the first database 300, transforms the data, and stores it in the second database. The entire data transformation process is implemented by the data transformation system itself, without relying on the database's own data transformation capabilities. This solves the problem that data transformation between two-dimensional and multi-dimensional tables depends on the database's own capabilities, making data transformation impossible in many scenarios and requiring developers to write code. Furthermore, the data transformation process is automatically generated based on the user-configured workflow, allowing users to flexibly configure the data transformation process according to their business needs, rather than simply performing a dimensional conversion from two-dimensional to multi-dimensional data. Moreover, it eliminates the need for developers to write code, significantly reducing development and maintenance costs.

[0234] The data conversion method and data conversion system provided in this application have been described in detail above. The following section will combine... Figures 11-13 The computing device provided in this application will be explained.

[0235] Figure 11 This is a schematic diagram of the structure of a computing device provided in this application, such as... Figure 11 As shown, the computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, memory 1106, and communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100. The computing device 1100 can be any of the aforementioned... Figures 1-12 The data conversion system 200 in this embodiment.

[0236] Bus 1102 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus 1102 is represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1102 may include a path for transmitting information between various components of the computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108). The unified bus may also be referred to as the Lingqu bus.

[0237] The processor 1104 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0238] Memory 1106 may include volatile memory, such as random access memory (RAM). Processor 1104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 1106 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0239] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 1106, or two or more types of storage media can be configured to realize the function of memory 1106. This application does not limit this.

[0240] The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to implement the functions of the aforementioned data conversion system 200, including... Figure 11 The functions of the acquisition unit and execution unit shown are used to implement the data conversion method provided in this application. That is, the memory 1106 stores instructions for executing the data conversion method.

[0241] Alternatively, the memory 1106 stores executable code, and the processor 1104 executes the executable code to implement the functions of the configuration unit, code generation unit, acquisition unit, and execution unit, thereby implementing the data conversion method provided in this application. That is, the memory 1106 stores instructions for executing the data conversion method.

[0242] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0243] As one possible implementation, the computing device 1100 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the data conversion method. For simplicity, further details are omitted here. The processor can be implemented using a CPU, or it can be implemented using computing devices or AI chips such as GPUs, DPUs, NPUs, XPUs, SoCs, offloading cards, or accelerator cards.

[0244] As one possible implementation, the computing device 1100 may include multiple types of processors 1104, meaning the computing device 1100 is a heterogeneous device. For example, the computing device 1100 may include a CPU and a GPU, and the operation steps corresponding to the data conversion method can be executed by at least one of the processors 1104. For the sake of brevity, further details will not be provided here.

[0245] This application also provides a computing device cluster. For example... Figure 12 As shown, Figure 12 This is an example diagram of a computing device cluster provided in this application, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0246] like Figure 12 As shown, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for performing data transformation methods.

[0247] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the data conversion method. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for executing the data conversion method.

[0248] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data conversion system 200. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more modules in the data conversion system 200.

[0249] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 One possible implementation is shown. For example... Figure 13 As shown, Figure 13This is a schematic diagram of another computing device cluster structure provided in this application. Two computing devices, 1100A and 1100B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of the code generation unit and the configuration unit. Simultaneously, the memory 1106 in computing device 1100B stores instructions for executing the functions of the acquisition unit and the execution unit.

[0250] Figure 13 The connection method between the computing device clusters shown can be based on the fact that the data conversion method provided in this application has a configuration stage and an application stage. Therefore, the unit modules are distributed on different computing devices. The functions of the configuration unit and code generation unit in the configuration stage are handed over to the computing device 1100A for execution, and the functions of the acquisition unit and execution unit in the matching stage are handed over to the computing device 1100B for execution.

[0251] It should be understood that Figure 13 The functions of computing device 1100A shown can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0252] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the data conversion method. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for executing the data conversion method.

[0253] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data conversion method.

[0254] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data conversion method, or instruct the computing device to perform a data conversion method.

[0255] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data conversion method, characterized in that, The method is applied to a data conversion system, which establishes communication connections with a first database and a second database. The method includes: The data conversion system acquires a data conversion request input by a first user, and the data conversion request includes a workflow identifier; The data transformation system obtains the code of the first workflow from the workflow space according to the workflow identifier. The workflow space includes one or more pre-configured workflows. The first workflow includes an extraction node, a transformation node, and a loading node. The extraction node includes the read address of the source table in the first database. The transformation node includes data transformation rules. The loading node includes the storage address of the target table in the second database. The source table and the target table are tables with different dimensions. The data transformation system executes the code of the first workflow, reads source table data from the first database based on the extraction node, transforms the source table data into target table data based on the transformation node, and stores the target table data in the second database based on the loading node.

2. The method according to claim 1, characterized in that, Before the data conversion system obtains the data conversion request input by the user, the method further includes: The data conversion system obtains configuration information input by the second user, the configuration information including the configuration information of the first workflow and the execution engine types supported by the first database and the second database; The data conversion system determines the first code template used by the first workflow based on the execution engine type, wherein the data conversion system includes code templates for multiple execution engine types; The data conversion system determines the code for the first workflow based on the first code template and stores it in the workflow space.

3. The method according to claim 2, characterized in that, The first code template includes one or more code sub-templates for each node. The data transformation system, based on the first code template, determines the code for the first workflow and stores it in the workflow space, including: The data conversion system determines the multiple nodes included in the first workflow based on the configuration information of the first workflow; Based on the first code template, determine the code snippet for each node; The code for the first workflow is determined based on the code snippets of each node and stored in the workflow space.

4. The method according to claim 2 or 3, characterized in that, The configuration information includes mapping configuration information, which includes dimension mapping relationships and member mapping relationships. The dimension mapping relationship includes the mapping relationship between the source table data and the dimensions of the target table, and the member mapping relationship includes the mapping relationship between the source table data and the members of the target table.

5. The method according to claim 4, characterized in that, The member mapping relationship includes condition mapping, and the mapping configuration information includes preset conditions. The condition mapping is used to indicate that when the source table data meets the preset conditions, there is a member mapping relationship between the source table data and the members of the target table. The preset conditions include one or more of numerical range, set, and fuzzy matching.

6. The method according to any one of claims 2 to 5, characterized in that, The configuration information includes node configuration information, and the configuration information of the conversion node includes the mapping configuration information and the data conversion rules; The data transformation rules include: transforming the source table data based on the transformation value table to obtain the target table data, wherein the transformation value table includes transformation values ​​corresponding to different source table data.

7. The method according to claim 6, characterized in that, The data transformation rules include: transforming the source table data based on transformation tags to obtain the target table data, wherein the transformation tags include fixed value tags and text value tags.

8. The method according to any one of claims 1 to 7, characterized in that, The first database is deployed on a first computing device, and the second database is deployed on a second computing device; The step of storing the target table data in the second database includes: The data conversion system obtains the communication message format supported by the second database and generates the message of the target table data based on the communication message format; The data conversion system sends a message containing the target table data to the second database.

9. A data conversion system, characterized in that, The data conversion system establishes communication connections with the first database and the second database. The system includes: An acquisition unit is used to acquire a data conversion request input by a first user, wherein the data conversion request includes a workflow identifier; The acquisition unit is used to acquire the code of the first workflow from the workflow space according to the workflow identifier. The workflow space includes one or more pre-configured workflows. The first workflow includes an extraction node, a transformation node, and a loading node. The extraction node includes the read address of the source table in the first database. The transformation node includes data transformation rules. The loading node includes the storage address of the target table in the second database. The source table and the target table are tables with different dimensions. An execution unit is used by the data conversion system to execute the code of the first workflow, read source table data from the first database based on the extraction node, convert the source table data into target table data based on the conversion node, and store the target table data in the second database based on the loading node.

10. The system according to claim 8, characterized in that, The system also includes a configuration unit. The configuration unit is used to obtain configuration information input by the second user, the configuration information including the configuration information of the first workflow and the execution engine types supported by the first database and the second database; The code generation unit is used to determine the first code template used by the first workflow based on the execution engine type, wherein the data conversion system includes code templates for multiple execution engine types; The code generation unit is used to determine the code of the first workflow from the first code template and store it in the workflow space.

11. The system according to claim 10, characterized in that, The first code template includes one or more code sub-templates for each node, and the code generation unit is used to determine the multiple nodes included in the first workflow based on the configuration information of the first workflow; The code generation unit is used to determine the code fragment for each node based on the first code template; The code generation unit is used to determine the code of the first workflow based on the code fragments of each node and store it in the workflow space.

12. The system according to claim 10 or 11, characterized in that, The configuration information includes mapping configuration information, which includes dimension mapping relationships and member mapping relationships. The dimension mapping relationship includes the mapping relationship between the source table data and the dimensions of the target table, and the member mapping relationship includes the mapping relationship between the source table data and the members of the target table.

13. The system according to claim 12, characterized in that, The member mapping relationship includes condition mapping, and the mapping configuration information includes preset conditions. The condition mapping is used to indicate that when the source table data meets the preset conditions, there is a member mapping relationship between the source table data and the members of the target table. The preset conditions include one or more of numerical range, set, and fuzzy matching.

14. The system according to any one of claims 9 to 13, characterized in that, The configuration information includes node configuration information, and the configuration information of the conversion node includes the mapping configuration information and the data conversion rules; The data transformation rules include: transforming the source table data based on the transformation value table to obtain the target table data, wherein the transformation value table includes transformation values ​​corresponding to different source table data.

15. The system according to claim 14, characterized in that, The data transformation rules include: transforming the source table data based on transformation tags to obtain the target table data, wherein the transformation tags include fixed value tags and text value tags.

16. The system according to any one of claims 9 to 15, characterized in that, The first database is deployed on a first computing device, and the second database is deployed on a second computing device; The execution unit is configured to obtain the communication message format supported by the second database, generate a message of the target table data based on the communication message format, and send the message of the target table data to the second database.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the operational steps of the method as described in any one of claims 1 to 8.

18. A computing device, characterized in that, The computing device includes a processor and a memory, the memory being used to store instructions and the processor being used to execute the instructions such that the computing device implements the method as claimed in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computing device or a cluster of computing devices, implement the method as claimed in any one of claims 1 to 8.

Images