Data source synchronization method and device

Abstract

The invention provides a data source synchronization method and device, and relates to the technical field of data processing, and the method comprises the steps: creating a first target table in response to the selection operation of a first data source on a target table creation page, the configuration of target table parameters and the configuration of target table fields; in response to a selection operation on the first data source and the source table thereof and a selection operation on the first target table in a mapping relationship creation page, loading a source table field of the source table and a target table field of the first target table, and correspondingly generating a mapping relationship between the source table field and the target table field with the same field name; and performing data synchronization based on the mapping relationship between the source table field and the target table field. Therefore, the complexity of the development process of data synchronization is greatly reduced, and the learning cost of personnel is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data source synchronization method and device. Background Art

[0002] While data stored on storage media (such as HDFS) is beneficial for storage, its query timeliness often fails to meet the query timeliness requirements of application systems. Therefore, in query scenarios, data needs to be transferred from storage media to query media, such as from HIVE to HBASE, to improve query speed. However, the development process for data synchronization between various data sources is complex and requires a high learning curve. Summary of the Invention

[0003] In view of this, the present application provides a data source synchronization method and device, which aims to simplify the development process of data synchronization between data sources and facilitate personnel's learning and application.

[0004] In a first aspect, the present application provides a data source synchronization method, comprising:

[0005] In response to the selection operation of the first data source, the configuration of the target table parameters and the configuration of the target table fields on the target table creation page, a first target table is created;

[0006] In response to the selection of the first data source and its source table and the selection of the first target table on the mapping relationship creation page, source table fields of the source table and target table fields of the first target table are loaded, and mapping relationships are generated between source table fields and target table fields having the same field names;

[0007] Data synchronization is performed based on the mapping relationship between the source table fields and the target table fields.

[0008] Optionally, the configuration of the target table parameters in response to the selection of the first data source includes:

[0009] In response to a triggering operation of a configuration bar control in a target table creation page, a target table configuration window is displayed;

[0010] In response to a selection operation of at least one data source among a plurality of data sources provided by a data source item in a target table configuration window, determining a data source corresponding to the first target table;

[0011] In response to the first parameter configured in the parameter configuration table in the target table configuration window, the parameters for creating the first target table are determined, where the first parameter includes the number of shards and the number of replicas, and the number of shards and the number of replicas are set according to the cluster parameters of the first data source.

[0012] Optionally, in response to the configuration of the target table field, the following steps are included:

[0013] In response to a triggering operation of a field bar control in a target table creation page, a target table field configuration window is displayed, wherein the target table field configuration window includes a plurality of field rows of the first target table, and each field row includes field information of a corresponding field;

[0014] In response to configuration of field information in the plurality of field rows, a target table field of the first target table is determined, wherein the field information includes field name, field type, field length, whether it is a primary key, and field storage format.

[0015] Optionally, the performing data synchronization includes: performing periodic data synchronization.

[0016] Optionally, when there is a first field in the field names of the source table that is different from the field names included in the target table fields, after generating the mapping relationship, the method further includes:

[0017] Based on the mapping relationship, display the source table fields and the target table fields corresponding to the source table fields on the mapping relationship creation page;

[0018] The mapping relationship is adjusted and updated according to the field name of the source table field and the field information of the target table field.

[0019] Optionally, adjusting and updating the mapping relationship according to the field name of the source table field and the field information of the target table field includes:

[0020] In response to each first field in the source table fields, a second field corresponding to the first field is selected from the target table fields, a mapping between the first field and the second field is established, and the mapping is added to the mapping relationship.

[0021] Optionally, when multiple first fields correspond to one second field, the response to configuring the field information in the multiple field rows includes:

[0022] The field row of the second field includes field value bits of multiple dimensions, and the field value bit of each dimension is used to fill in the field value of the first field of the dimension.

[0023] Optionally, the response to configuring the field information in the multiple field rows includes:

[0024] According to the field type of the target field in the source table field, at least one field row corresponding to the target field is configured in the target table field, wherein when multiple field rows are configured for the target field, the field types of the multiple field rows are different from each other.

[0025] Optionally, before responding to the selection of the first data source, the method further includes:

[0026] In response to a trigger operation on a newly added data source control, a data source information filling page is displayed; the data source connection configuration page includes a plurality of data source parameter name bits, a data source parameter value bit corresponding to each of the data source parameter name bits, a data source parameter description bit corresponding to each of the data source parameter name bits, an import control, and a test connection control;

[0027] In response to a triggering operation on the test connection control, it is configured to perform a data source connection test based on the data source information filled in the data source information filling page and feed back a test result;

[0028] In response to a triggering operation on the import option, the data source information entered in the data source information filling page is stored.

[0029] In a second aspect, the present application provides a data source synchronization device, the device comprising:

[0030] A creating unit, configured to create a first target table in response to a selection operation on a first data source, configuration of target table parameters, and configuration of target table fields on a target table creation page;

[0031] a mapping unit configured to, in response to a selection operation on the first data source and its source table and a selection operation on the first target table on a mapping relationship creation page, load source table fields of the source table and target table fields of the first target table, and generate a mapping relationship between source table fields and target table fields having the same field name;

[0032] The synchronization unit is used to synchronize data based on the mapping relationship between the source table fields and the target table fields.

[0033] In a third aspect, the present application provides a device comprising a memory and a processor, wherein the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes a data source synchronization method as described in any one of the first aspects above.

[0034] In a fourth aspect, the present application provides a computer storage medium storing a code. When the code is executed, the device executing the code implements a data source synchronization method as described in any one of the first aspects above.

[0035] The present application provides a data source synchronization method and device. When executing the method, a first target table is first generated based on the selected data source, the configuration of the target table parameters, and the configuration of the target table fields. Then, the source table fields of the first data source and the target table fields of the first target table are loaded, and a mapping relationship between the source table fields and the target table fields is automatically generated through the consistency of the field names. Finally, based on the mapping relationship, data synchronization is performed between the source table and the first target table. In this way, the first data source to be synchronized, the target table parameters, and the target table fields are visually selected, which facilitates user visual operations and allows for intuitive and convenient creation of the first target table. The source table and target table are visually selected, and the mapping is automatically generated through the field names. This simplifies the steps and enables visual operations, greatly reducing the complexity of the data synchronization development process and the learning cost of personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the following briefly introduces the drawings required for use in the embodiment or the prior art description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A flowchart of a data source synchronization method provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a target table creation page including a target table configuration window provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of a target table creation page of another target table configuration window provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a target table creation page in another target table configuration window provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of the correspondence between source table fields and target field names provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of a data source synchronization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0044] In the description and claims of the embodiments of the present disclosure, as well as in the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate to describe the embodiments of the present disclosure herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0045] Unless otherwise specified, the term "plurality" means two or more. In the disclosed embodiments, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B. The term "and / or" describes an association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0047] See also Figure 1 , Figure 1 A schematic diagram of a data source synchronization method provided in an embodiment of the present application, wherein the data source synchronization method includes:

[0048] S101: In response to the selection operation of a first data source, the configuration of target table parameters, and the configuration of target table fields on a target table creation page, create a first target table.

[0049] Generate a target table by selecting a data source, configuring target table parameters for establishing the target table, and configuring the field information of the target table.

[0050] The above-mentioned first data source is pre-configured in the data system based on the project progress and production resource availability. You can directly select the first data source from the corresponding data source item on the target table creation page to call it. Of course, if the data system does not pre-configure the above-mentioned first data source, you will need to set the connection information of the first data source required by the business project in the data system to facilitate its use.

[0051] S102. In response to the selection operation of the first data source and its source table and the selection operation of the first target table in the mapping relationship creation page, the source table fields of the source table and the target table fields of the first target table are loaded, and a mapping relationship is generated between the source table fields and the target table fields having the same field names.

[0052] The operation of selecting the first data source and its source table mentioned above can be a user selecting the first data source and the source table of the first data source displayed in the form of a drop-down box on the mapping relationship creation page. Similarly, the operation of selecting the first target table mentioned above can be a user selecting the first target table from multiple target tables displayed in the form of a drop-down box on the mapping relationship creation page.

[0053] The source table includes multiple field rows of the source table, each corresponding to a field, and each field row includes field information (field name, field value, field length, field type, field description, etc.) for the corresponding field. The field information in the multiple field rows of the source table forms the source table fields. Similarly, the first target table also includes multiple field rows of the target table, each corresponding to a field, and the field information in the multiple field rows of the target table forms the target table fields.

[0054] Select the source table of the first data source to be synchronized and the first target table created corresponding to the first data source to load the source table fields of the source table and the target table fields of the first target table, and automatically establish a mapping relationship between the source table fields and the target table fields with the same field names to improve processing efficiency.

[0055] S103: Perform data synchronization based on the mapping relationship between the source table fields and the target table fields.

[0056] Exemplarily, the data synchronization may be performed using MapReduce technology. Therefore, the mapping relationship refers to the corresponding relationship between the source table and the target table when data synchronization is performed using MapReduce technology.

[0057] The above-mentioned data synchronization may be periodic data synchronization, and the period of data synchronization may be set according to human needs or the speed of data changes, such as data synchronization once an hour or once a day.

[0058] Based on the above steps S101-S103, it can be seen that the present application visually selects the first data source to be synchronized, configures the target table parameters, and configures the target table fields, which facilitates user visual operations and creates the first target table intuitively and conveniently. It also visually selects the source table and the target table, automatically generates mappings through field names, simplifies steps, and combines visual operations to greatly reduce the complexity of the data synchronization development process and reduce personnel learning costs.

[0059] In the embodiment of the present application, the above Figure 1 There are multiple possible implementations of step S101, which are described below. It should be noted that the implementations described below are only for illustrative purposes and do not represent all implementations of the embodiments of the present application.

[0060] See also Figure 2 ,like Figure 2 The target table creation page diagram shown includes a target table configuration window. In response to the selection of the first data source and the configuration of the target table parameters in step S101, the following steps may be included:

[0061] First, in response to a triggering operation of a configuration bar control in a target table creation page, a target table configuration window is displayed.

[0062] Optional, see Figure 2 The left side of the target table creation page has fixed settings for the configuration bar control and field bar control, etc. Of course, you can also refer to Figure 3 Another target table creation page diagram including a target table configuration window is shown, where the configuration bar control and field bar control are fixedly set on the upper side of the target table creation page. Figure 2 In the example shown, in response to the user triggering the operation on the configuration bar control, the right side of the target table creation page (such as Figure 2 ) will display the target table configuration window. Of course, you can also Figure 3 In the example shown, in response to the user triggering an operation on the field bar control on the upper side, the target table field configuration window will be displayed on the lower side of the target table creation page to facilitate the user to visually adjust the field information in multiple field rows in the field configuration window.

[0063] Secondly, in response to a selection operation on at least one data source from a plurality of data sources provided by the data source item in the target table configuration window, the data source corresponding to the first target table is determined.

[0064] Illustratively, the above-mentioned data source item may correspond to a drop-down list including multiple data sources for selection. In response to the user selecting the first data source in the drop-down list, the first data source corresponding to the first target table may be determined.

[0065] Finally, in response to the first parameter configured in the parameter configuration table in the target table configuration window, the parameters for creating the first target table are determined, where the first parameter includes the number of shards and the number of replicas, which are set according to the cluster parameters of the first data source.

[0066] For example, for the first data source determined from the drop-down list corresponding to the above data source item, the first target table can be optimized according to the number of shards stored corresponding to the cluster parameters of the first data source, the storage nodes of the primary and secondary copies, and other parameters (different cluster parameters will correspond to different numbers of shards and storage node parameters of the primary and secondary copies, and thus, corresponding adjustments can also be made when configuring the target table parameters such as the number of shards and the number of copies for the target table), rather than automatically creating the target table directly based on the source table, so that the target table can better adapt to the corresponding data source.

[0067] See also Figure 2 , the parameter configuration table can include multiple parameter names (such as Figure 2 solr.indexName, solr.shards, solr.replicas, etc.), each parameter name corresponds to a parameter value and parameter description (for example Figure 2 The parameter value of solr.shards shown is 2, and the corresponding parameter name is the Solr table name (index name). The parameters of the target table may include the table name, number of shards, number of replicas, etc.

[0068] Optionally, a parameter in the above parameter configuration table (such as table name) can also be set individually, see Figure 4 Another target table creation page diagram shown includes a target table configuration window, which determines the table name of the first target table in response to the first table name filled in the blank corresponding to the table name item in the target table configuration window.

[0069] This allows users to directly and visually select data sources and configure target table parameters, making it easier for them to learn and use. Furthermore, the target table's parameters are configured in conjunction with the data source cluster parameters. For example, the number of shards and replicas for the target table matches the number of shards and replicas for the primary data source. This helps ensure data consistency and facilitates data exception handling and recovery. It also enables parallel processing to improve synchronization efficiency and facilitates the monitoring and maintenance of the corresponding stored and queried data sources.

[0070] Further, for the above step S101, see Figure 3 , in response to the configuration of the target table field, may include:

[0071] In response to a triggering operation of a field bar control in a target table creation page, a target table field configuration window is displayed, wherein the target table field configuration window includes a plurality of field rows of the first target table, and each field row includes field information of a corresponding field.

[0072] Optionally, you can sort the field rows and set a corresponding sorting sequence number for each field row.

[0073] In response to configuration of field information in the plurality of field rows, a target table field of the first target table is determined, wherein the field information includes field name, field type, field length, whether it is a primary key, and field storage format.

[0074] The aforementioned field names can be provided by the business party (the customer who needs to query information through the first target table), or they can also be the corresponding field names in the source table of the first data source. The primary key in the first target table must be uniquely verified. The aforementioned field storage format includes whether the corresponding field requires redundant storage. In addition, the aforementioned field information may also include a field description.

[0075] Optionally, the above field types can be set according to the field types in the source table, or according to query requirements. For example, for word segmentation query, you can choose the text field type, and for exact query, you can choose the keyword field type.

[0076] Optionally, based on the field type of the target field in the source table field, at least one field row corresponding to the target field is configured in the target table field, wherein when multiple field rows are configured for the target field in the target table field, the field types of the multiple field rows are different from each other, and the field names are also different from each other.

[0077] Optionally, when configuring field information in a field row, you can manually fill in and modify each field information (such as field name, field description, etc.), or configure a drop-down box containing multiple options for selection (such as field type, etc.), or configure a check box (for example, distinguishing whether it is a primary key by checking or unchecking the check box).

[0078] Therefore, a drop-down box containing multiple field types can be configured for field types, allowing users to directly select configurations and reduce input and filling operations. For example, for a field in the source table, multiple fields can be configured in the target table, and these multiple fields can be configured with different field types. For example, if the source table is of type varchar, you can choose types such as keyword / text when syncing to the target table. When selecting numeric types in the source table, you can choose different precisions to meet query requirements (long / double / integer) when selecting them in the target table. In this way, when establishing a mapping, a corresponding field in the source table can be mapped to corresponding fields of different field types, rather than a single mapping, which facilitates querying.

[0079] In the embodiment of the present application, the above Figure 1 There are multiple possible implementations of step S102, which are described below. It should be noted that the implementations described below are only for illustrative purposes and do not represent all implementations of the embodiments of the present application.

[0080] Optional, see Figure 5 The diagram below shows the correspondence between source table fields and target field names. Based on the mapping relationship, the first area (e.g., the middle area) of the mapping relationship creation page is divided into two columns: the left column displays the source table fields, and the right column displays the target table fields corresponding to the source table fields. This makes it easier to view the correspondence between the source and target fields.

[0081] Optional, see Figure 5 A filter box for filtering field information can be set in the second area (for example, the lower area) of the mapping relationship creation page, wherein the filter box can provide filter items for various field information such as field name, field description, type, etc. In response to the filter information set by the user for one or more filter items (for example, field name, field description, field type, etc.) in the filter box, the fields that meet the filter information are searched from the source table and the target table, and are displayed in the source table fields and target table fields in the above two columns, so as to facilitate the user to adjust the field information.

[0082] See also Figure 5 The diagram shows the correspondence between the field names of a source table and a target. In step S102, a mapping relationship is generated between the source table fields and the target table fields with the same field names. However, there may be cases where, for example, the fields of the target table are provided by the business party and do not correspond to the field names of the source table.

[0083] Therefore, when there is a first field in the field name of the source table that is different from the field names included in the target table fields, after generating the mapping relationship, the mapping relationship can also be adjusted and updated according to the field name of the source table field and the field information of the target table field.

[0084] Specifically, in response to each first field in the source table field, a second field corresponding to the first field is selected from the target table field, a mapping between the first field and the second field is established, and the mapping is added to the mapping relationship.

[0085] For example, for a field named filed_c in the source table, if there is no field named filed_c in the target table, the field named solr_filed_c in the target table is selected. The field named solr_filed_c can be a field created corresponding to the field named filed_c in the source table when the target table field is configured.

[0086] Furthermore, source fields with different dimensions can be mapped to a field in the target table, that is, the above-mentioned multiple first fields correspond to one second field. Specifically, the field row where the second field in the target table is located can also include field value bits of multiple dimensions, and the field value bit of each dimension is used to fill in the field value of the first field of the dimension.

[0087] In this way, when configuring a mapping relationship, you can map multiple fields in the source table to a single field in the target table. For example, the longitude and latitude fields in the source table are mapped to a field in the target table with the field attribute set to geo_point. The field with the field attribute set to geo_point has a longitude field value bit corresponding to the longitude field, and a latitude field value bit corresponding to the latitude field.

[0088] In addition, when establishing a one-to-many mapping between the same source table field and multiple target fields of different field types, the field rows corresponding to the multiple target fields of different field types can be used to configure the same source table field in the field rows corresponding to the source table side, for example, in the form of a drop-down box.

[0089] Based on the above embodiment, the connection information of the first data source can be pre-set in the data system, and the first data source can be directly selected in the corresponding data source item on the target table creation page for calling. Of course, if the data system does not pre-set the first data source, the connection information of the first data source required by the business project needs to be set in the data system for calling. The specific operation process can be as follows:

[0090] First, in response to the triggering operation of the newly added data source control, the data source information filling page is displayed; the data source connection configuration page includes multiple data source parameter name bits, data source parameter value bits corresponding to each data source parameter name bit, data source parameter description bits corresponding to each data source parameter name bit, import controls and test connection controls.

[0091] The data source parameter name field is used to enter the name of the data source parameter that implements the data source link or to modify an already entered data source parameter name. The data source parameter value field is used to enter or modify the parameter value corresponding to the data source parameter name. The data source parameter description field is used to enter the description information of the corresponding data source parameter. For example, see the table below:

[0092] Parameter name Parameter value Parameter Description driver.class org.apache.hive.jdbc.HiveDriver Connect Driver jdbc.url jdbc:hive2: / / 12.103.113.69:10000 Connection URL username hive Login Name password 123456 password …… …… ……

[0093] Secondly, in response to the triggering operation of the test connection control, it is used to perform a data source connection test based on the data source information filled in the data source information filling page and feed back the test result.

[0094] Finally, in response to the triggering operation of the import option, the data source information filled in the data source information filling page is stored in the data system.

[0095] After the data source information has been filled in and stored in the data source information filling page, the newly configured data source will also be added to the corresponding option of selecting a data source, such as the data source corresponding to the data source item in the target table configuration window.

[0096] The above are some specific implementations of a data source synchronization method provided by the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided by the embodiment of the present application will be introduced from the perspective of functional modularization.

[0097] See also Figure 6 The structure diagram of a data source synchronization device shown is as follows, wherein the data source synchronization device comprises:

[0098] The creation unit 601 is configured to create a first target table in response to the selection of the first data source, the configuration of target table parameters, and the configuration of target table fields on the target table creation page;

[0099] A mapping unit 602 is configured to, in response to a selection operation on the first data source and its source table and a selection operation on the first target table on the mapping relationship creation page, load source table fields of the source table and target table fields of the first target table, and generate a mapping relationship between source table fields and target table fields having the same field name;

[0100] The synchronization unit 603 is configured to perform data synchronization based on the mapping relationship between the source table fields and the target table fields.

[0101] Based on the above device, it can be seen that the creation unit 601 of this application visually selects the first data source to be synchronized, configures the target table parameters, and configures the target table fields, which facilitates user visual operations and creates the first target table intuitively and conveniently. The source table and the target table are visually selected through the mapping unit 602, and the mapping is automatically generated through the field name. The synchronization unit 603 automatically synchronizes, simplifies the steps, and combines visual operations to greatly reduce the complexity of the development process of data synchronization and reduce the learning cost of personnel.

[0102] In one possible implementation, the creation unit 601 is specifically configured to display a target table configuration window in response to a triggering operation of a configuration bar control in a target table creation page; determine the data source corresponding to the first target table in response to a selection operation of at least one data source from a plurality of data sources provided by a data source item in the target table configuration window; and determine parameters for creating the first target table in response to a first parameter configured in a parameter configuration table in the target table configuration window, where the first parameter includes the number of shards and the number of replicas, and the number of shards and the number of replicas are set according to cluster parameters of the first data source.

[0103] Optionally, the creation unit 601 is specifically used to display a target table field configuration window in response to a trigger operation of a field bar control in a target table creation page, wherein the target table field configuration window includes multiple field rows of the first target table, and each field row includes field information of a corresponding field; in response to the configuration of the field information in the multiple field rows, the target table field of the first target table is determined, and the field information includes field name, field type, field length, whether it is a primary key, and field storage format.

[0104] Optionally, the synchronization unit 603 is specifically used to perform periodic data synchronization.

[0105] When there is a first field in the field name of the source table that is different from the field names included in the target table fields, the mapping unit 602 is further used to display the source table field and the target table field corresponding to the source table field on the mapping relationship creation page based on the mapping relationship; and adjust and update the mapping relationship according to the field name of the source table field and the field information of the target table field.

[0106] Optionally, the mapping unit 602 is further used to select a second field corresponding to each first field in the source table field from the target table field, establish a mapping between the first field and the second field, and add the mapping to the mapping relationship.

[0107] Optionally, when multiple first fields correspond to one second field, the creation unit 601, specifically for the field row of the second field, includes field value bits of multiple dimensions, and the field value bit of each dimension is used to fill in the field value of the first field of the dimension.

[0108] Optionally, the creation unit 601 is specifically used to configure at least one field row corresponding to the target field in the target table field according to the field type of the target field in the source table field, wherein when multiple field rows are configured for the target field, the field types of the multiple field rows are different from each other.

[0109] Optionally, the device further includes a data source unit, which is specifically used to display a data source information filling page in response to a trigger operation on a newly added data source control; the data source connection configuration page includes multiple data source parameter name bits, data source parameter value bits corresponding to each of the data source parameter name bits, data source parameter description bits corresponding to each of the data source parameter name bits, an import control and a test connection control; in response to a trigger operation on the test connection control, it is used to perform a data source connection test based on the data source information filled in the data source information filling page and feedback the test result; in response to a trigger operation on the import option, it is used to store the data source information filled in the data source information filling page.

[0110] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0111] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes, so that the device executes a data source synchronization method described in any embodiment of the present application.

[0112] The computer storage medium stores code, and when the code is executed, the device executing the code implements a data source synchronization method described in any embodiment of the present application.

[0113] The "first" and "second" in the names such as "first" and "second" (if any) mentioned in the embodiments of this application are only used as name identifiers and do not represent the first or second in order.

[0114] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment or certain parts of the embodiments of the present application.

[0115] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0116] The above description is merely an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application.

Claims

1. A data source synchronization method, characterized in that: include: In response to the selection operation of the first data source, the configuration of the target table parameters and the configuration of the target table fields on the target table creation page, a first target table is created; In response to the selection of the first data source and its source table and the selection of the first target table on the mapping relationship creation page, source table fields of the source table and target table fields of the first target table are loaded, and mapping relationships are generated between source table fields and target table fields having the same field names; Data synchronization is performed based on the mapping relationship between the source table fields and the target table fields.

2. The method according to claim 1, characterized in that The configuration of target table parameters in response to the selection of the first data source on the target table creation page includes: In response to a triggering operation of a configuration bar control in a target table creation page, a target table configuration window is displayed; In response to a selection operation of at least one data source among a plurality of data sources provided by a data source item in a target table configuration window, determining a data source corresponding to the first target table; In response to the first parameter configured in the parameter configuration table in the target table configuration window, the parameters for creating the first target table are determined, where the first parameter includes the number of shards and the number of replicas, and the number of shards and the number of replicas are set according to the cluster parameters of the first data source.

3. The method according to claim 2, characterized in that In response to the configuration of the target table field, the method includes: In response to a triggering operation of a field bar control in a target table creation page, a target table field configuration window is displayed, wherein the target table field configuration window includes a plurality of field rows of the first target table, and each field row includes field information of a corresponding field; In response to configuration of field information in the plurality of field rows, a target table field of the first target table is determined, wherein the field information includes field name, field type, field length, whether it is a primary key, and field storage format.

4. The method according to claim 3, characterized in that The data synchronization includes: performing periodic data synchronization.

5. The method according to claim 3, characterized in that When there is a first field in the field names of the source table that is different from the field names included in the target table fields, after generating the mapping relationship, the method further includes: Based on the mapping relationship, display the source table fields and the target table fields corresponding to the source table fields on the mapping relationship creation page; The mapping relationship is adjusted and updated according to the field name of the source table field and the field information of the target table field.

6. The method according to claim 5, characterized in that The adjusting and updating the mapping relationship according to the field name of the source table field and the field information of the target table field includes: In response to each first field in the source table fields, a second field corresponding to the first field is selected from the target table fields, a mapping between the first field and the second field is established, and the mapping is added to the mapping relationship.

7. The method according to claim 6, characterized in that When multiple first fields correspond to one second field, the responding to the configuration of the field information in the multiple field rows includes: The field row of the second field includes field value bits of multiple dimensions, and the field value bit of each dimension is used to fill in the field value of the first field of the dimension.

8. The method according to any one of claims 3 to 7, characterized in that: The responding to the configuration of the field information in the plurality of field rows includes: According to the field type of the target field in the source table field, at least one field row corresponding to the target field is configured in the target table field, wherein when multiple field rows are configured for the target field, the field types of the multiple field rows are different from each other.

9. The method according to any one of claims 1 to 6, characterized in that: Before responding to the selection of the first data source, the method further includes: In response to a trigger operation on a newly added data source control, a data source information filling page is displayed; the data source connection configuration page includes a plurality of data source parameter name bits, a data source parameter value bit corresponding to each of the data source parameter name bits, a data source parameter description bit corresponding to each of the data source parameter name bits, an import control, and a test connection control; In response to a triggering operation on the test connection control, it is configured to perform a data source connection test based on the data source information filled in the data source information filling page and feed back a test result; In response to a triggering operation on the import option, the data source information entered in the data source information filling page is stored.

10. A data source synchronization device, characterized in that: The device comprises: A creating unit, configured to create a first target table in response to a selection operation on a first data source, configuration of target table parameters, and configuration of target table fields on a target table creation page; a mapping unit configured to, in response to a selection operation on the first data source and its source table and a selection operation on the first target table on a mapping relationship creation page, load source table fields of the source table and target table fields of the first target table, and generate a mapping relationship between source table fields and target table fields having the same field name; The synchronization unit is used to synchronize data based on the mapping relationship between the source table fields and the target table fields.

Images