Data synchronization method, apparatus and electronic device

Abstract

The application discloses a data synchronization method and device and electronic equipment. The method is applied to a first electronic device and includes the following steps: generating K first query statements according to first configuration information, wherein K is an integer greater than 1; synchronizing K source data tables located in at least one first database to P target data tables in a second database according to the K first query statements, wherein P is an integer greater than 1 and not more than K; obtaining second configuration information, wherein the second configuration information includes mapping information, a filtering condition, a connection relationship of the P target data tables and storage of the P target data tables in the second database; generating a second query statement according to the second configuration information; performing joint query on the P target data tables according to the second query statement to obtain target data; and synchronizing the target data to an index. The method can improve the data synchronization efficiency between a search engine and a database and avoid affecting the business of a business party.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to a data synchronization method and device and electronic equipment. BACKGROUND

[0002] When searching data in a database storing a large amount of data, a search engine tool with strong search analysis capability needs to be used to improve search efficiency. In the above data retrieval scenario, since the data to be queried is stored in a relational database, the relational database has a low retrieval efficiency when processing the retrieval of a large amount of data.

[0003] Compared with the traditional relational database, the search engine has a faster data query speed, such as Elasticsearch (ES), and therefore, in order to ensure the efficiency of the data query result, data synchronization between the search engine and the database needs to be implemented. In the traditional technology, the data synchronization between the search engine and the database cannot be efficiently implemented.

[0004] Therefore, how to improve the data synchronization efficiency between the search engine and the database becomes a problem to be solved. SUMMARY

[0005] The present application provides a data synchronization method, device and electronic equipment, which can improve the data synchronization efficiency between the search engine and the database and avoid affecting the business of the business party.

[0006] In a first aspect, a data synchronization method is provided, applied to a first electronic device, and includes: generating K first query statements according to first configuration information, the first configuration information including table names of K source data tables and storage of the K source data tables in at least one first database, the K first query statements and the K source data tables corresponding to each other, each first query statement being used to query a corresponding source data table, K being an integer greater than 1; synchronizing the K source data tables located in the at least one first database to a second database according to the K first query statements, so that P target data tables corresponding to the K source data tables are stored in the second database, data in the K source data tables being generated by an application of an electronic device where the at least one first database is located, and the electronic device where the at least one first database is located and an electronic device where the second database is located being different, P being an integer greater than 1 and not more than K; obtaining second configuration information, the second configuration information including mapping information, a filtering condition, a connection relationship of the P target data tables, and storage of the P target data tables in the second database, the mapping information including a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, the filtering condition being used to filter field values of the to-be-queried fields corresponding to at least one target data table satisfying a preset condition; generating a second query statement according to the second configuration information, the second query statement being used to query field values of the plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields and satisfying the preset condition, the associated data table being a data table obtained by associating the P target data tables according to the connection relationship; performing joint query on the P target data tables in the second database according to the second query statement to obtain target data, the target data including the plurality of fields and the field values of the plurality of to-be-queried fields in the associated data table and satisfying the preset condition; and synchronizing the target data to the index.

[0007] The K source data tables are located in the at least one first database, that is, the K source data tables can be located in one first database or a plurality of first databases.

[0008] The P target data tables are a plurality of tables to be queried, for example, data in the P target data tables can be data generated by a business server. For example, the P target data tables can include a main table, and data tables other than the main table in the P target data tables can be sub-tables.

[0009] The at least one target data table can be a main table, or can also be a sub-table, which is not specifically limited.

[0010] The mapping relationship between the plurality of fields in the index of the search engine and the plurality of to-be-queried fields corresponding to the P target data tables should be understood that the field values of the fields in the index that have the mapping relationship should be consistent with the field values of the to-be-queried fields in the corresponding target data tables. It can be understood that in the embodiments of the present application, the ultimate purpose of data synchronization is to synchronize the field values of the plurality of to-be-queried fields in the P target data tables that satisfy the preset condition to the field values of the plurality of fields in the corresponding index.

[0011] The P target data tables correspond to a plurality of to-be-queried fields, wherein each target data table corresponds to at least one to-be-queried field in the plurality of to-be-queried fields.

[0012] The one query statement generated according to the second configuration information is used to query the field values of the plurality of to-be-queried fields in the associated data table corresponding to the plurality of fields that satisfy the preset condition, wherein the associated data table is a data table obtained by associating the P target data tables according to a connection relationship. It can be understood that the one second query statement has the effect of associating the P target data tables according to the connection relationship, the effect of mapping the plurality of fields in the index and the plurality of to-be-queried fields corresponding to the P target data tables, and the effect of querying the field values of the plurality of to-be-queried fields in the associated data table corresponding to the plurality of fields that satisfy the preset condition.

[0013] In the technical solution, on the one hand, in the process of synchronizing K source data tables in at least one first database to a source data table in a second database according to K first query statements, since the data in the K source data tables is generated by an application of an electronic device where the at least one first database is located, and the electronic device where the at least one first database is located is different from an electronic device where the second database is located, the business of the business party (i.e., the electronic device where the at least one first database is located) can be avoided. On the other hand, since the second configuration information includes a filtering condition, a connection relationship of P target data tables, and information that the P target data tables are stored in the second database, a query statement (i.e., a second query statement) generated according to the second configuration information can simultaneously achieve the purpose of extracting data (i.e., the P target data tables) from a data source (i.e., the second database) and filtering the data extracted from the data source according to a query requirement (i.e., querying field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition), so that the method can improve the efficiency of data query, thereby improving the data synchronization efficiency between the search engine and the database. In addition, since the second configuration information further includes a mapping relationship (i.e., mapping information) between a plurality of fields in an index and a plurality of to-be-queried fields corresponding to the P target data tables, the second query statement generated according to the second configuration information is used to query field values of the plurality of to-be-queried fields in the associated data table (i.e., a data table obtained by associating the P target data tables according to the connection relationship) that meet the preset condition, so that the target data obtained by jointly querying the P target data tables using the query statement includes the plurality of fields in the index and the field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition. Thereafter, the obtained target data does not need to be subjected to field mapping (e.g., mapping of fields in the database to fields in the index) and the like, and can be directly synchronized to the index, so that the method can improve the data synchronization efficiency between the search engine and the database. In summary, based on the technical solution of the present application, the data synchronization efficiency between the search engine and the database can be improved while avoiding affecting the business of the business party.

[0014] In a possible implementation, according to the first configuration information, K first query statements are generated, including: according to the first configuration information, selecting a first generator from a plurality of candidate generators, wherein the plurality of candidate generators correspond to a plurality of types of statements, each candidate generator is used to generate a statement of a corresponding type, and the first generator is used to generate a statement for querying each source data table; and processing the first configuration information by using the first generator to obtain the K first query statements.

[0015] In the technical solution, the first generator needed can be selected from the plurality of candidate generators according to the first configuration information, and then the first generator is used to process the first configuration information to obtain the K first query statements matched with the first configuration information. In this implementation, when the query requirement changes (for example, the filter condition in the configuration information changes), the target generator actually needed can be selected more flexibly from the plurality of candidate generators, so that the reuse of the generator is facilitated.

[0016] In another possible implementation, P is equal to K, and the P target data tables are the K source data tables, and the fields of any two source data tables in the K source data tables are not completely same.

[0017] In the technical solution, the P target data tables are the K source data tables, that is, the backup (that is, synchronization) of the K source data tables by the first electronic device in the implementation process can not involve the operation step of merging the K source data tables by the first electronic device. In the scenario where the processing performance of the electronic device where the K source data tables are located is good, the K source data tables do not need to be backed up in this implementation, so that the efficiency of the data synchronization between the search engine and the database is further improved.

[0018] In another possible implementation, P is less than K, the P target data tables include a data table obtained by merging part of the K source data tables and the remaining source data tables except the part of the K source data tables, any two source data tables in the part of the K source data tables include the same fields, any one source data table in the remaining source data tables includes fields that are not completely same as the fields included in any one source data table in the part of the K source data tables, and the K source data tables located in the at least one first database are synchronized to the second database according to the K first query statements, so that the second database stores the P target data tables corresponding to the K source data tables. The method comprises the following steps: according to part of the first query statements in the K first query statements, the part of the source data tables located in the at least one first database are synchronized to the same data table in the second database, and according to the first query statements except the part of the first query statements, the remaining source data tables located in the at least one first database are synchronized to the second database, so that the second database stores the P target data tables.

[0019] In the implementation, since P is an integer greater than 1 and less than K, and K is an integer greater than 1, K in the implementation should be an integer greater than or equal to 3.

[0020] Part of the K source data tables includes the same fields, that is, the part of the source data tables is a plurality of sub-data tables obtained by splitting a certain business table.

[0021] Any one of the remaining source data tables is not a plurality of sub-data tables obtained by splitting a certain business table, so in the case that the number of source data tables included in the remaining source data tables is greater than or equal to 2, the fields included in any two of the remaining source data tables are not completely the same.

[0022] In the above technical solution, part of the K source data tables is a plurality of data tables obtained by splitting a certain data table, so in the process of source data backup, the part of the source data tables can be merged into a data table stored in the second database, so as to facilitate improving the efficiency of subsequent query operation on the P target data tables located in the second database.

[0023] In another possible implementation, according to the second configuration information, the second query statement is generated, including: according to the second configuration information, selecting a plurality of target generators from a plurality of candidate generators, wherein the plurality of candidate generators and the plurality of types of statements correspond, each candidate generator is used to generate a statement of a corresponding type, the plurality of target generators includes a first generator, a second generator and a third generator, the first generator is used to generate a statement of querying the field value of a plurality of fields having a mapping relationship with the plurality of fields, the second generator is used to generate a statement representing a connection relationship, and the third generator is used to generate a statement representing a filtering condition; processing the second configuration information by using the plurality of target generators to obtain a plurality of statements, and splicing the plurality of statements to obtain the second query statement.

[0024] In the above technical solution, according to the second configuration information, the plurality of target generators needed can be selected from the plurality of candidate generators, and then the second configuration information is processed by using the plurality of target generators to obtain the second query statement matched with the second configuration information. In this implementation, when the query requirement changes (for example, the filtering condition in the configuration information changes), the target generator actually needed can be more flexibly selected from the plurality of candidate generators, so as to facilitate the reuse of the generator.

[0025] In another possible implementation, the target data is synchronized to the index, including: sending the target data to the message queue, so as to synchronize the target data located in the message queue to the index.

[0026] For example, the above message queue can be but is not limited to message queue kafka.

[0027] In the technical solution, the first electronic device sends the obtained target data to the message queue first, and then the downstream search engine can read the target data from the message queue according to its own rhythm, so that the first electronic device synchronizes the target data in the message queue to the index, thereby avoiding the phenomenon of reading blockage of the search engine caused by too large data volume of the target data, improving the reading efficiency of the search engine, and improving the data synchronization efficiency between the search engine and the database.

[0028] In another possible implementation, the search engine is located in a second electronic device different from the first electronic device, and the step of synchronizing the target data to the index comprises: sending the target data to the search engine in the second electronic device, so that the search engine synchronizes the obtained target data to the index.

[0029] In the technical solution, when the second electronic device where the search engine is located and the first electronic device are two different electronic devices, the interaction between the second electronic device and the first electronic device is further involved in the process of synchronizing the target data to the index.

[0030] In another possible implementation, the P target data tables include a first data table and a second data table, wherein the to-be-queried field corresponding to the first data table is a first field, the to-be-queried field corresponding to the second data table includes a second field and a third field, the plurality of fields include a fourth field, a fifth field and a sixth field, and

[0031] The mapping relationship includes an unconditional mapping relationship and a conditional mapping relationship, wherein the unconditional mapping relationship is a mapping relationship between the fourth field and the first field, the conditional mapping relationship includes a mapping relationship between the fifth field and the third field when the field value of the second field is a first value, and a mapping relationship between the sixth field and the third field when the field value of the second field is a second value, and the first value and the second value are different.

[0032] For example, the first data table can be a master table, and the second data table can be a slave table.

[0033] The mapping relationship can be set according to the to-be-queried field corresponding to each data table in the P data tables, and no specific limitation is made in this regard.

[0034] In another possible implementation, the preset condition includes that the field value of the to-be-queried field corresponding to at least one target data table exceeds a preset value; or, the preset condition includes that the field value of the to-be-queried field corresponding to at least one target data table does not exceed a preset value; or, in the case that at least one target data table further includes a non-to-be-queried field in addition to the to-be-queried field corresponding to at least one target data table, the preset condition includes that the field value of the non-to-be-queried field is equal to a preset value.

[0035] The non-query field in the at least one target data table refers to a field included in the at least one target data table and not queried in the data synchronization process.

[0036] For example, the at least one target data table includes a field A and a field B, and the at least one target data table corresponds to the field A as the query field, and the field B can be referred to as the non-query field of the at least one target data table.

[0037] It should be understood that the preset condition in the above implementation manner is only illustrative, that is, the preset condition is not specifically limited in the present application, and can be set according to actual business query requirements.

[0038] In another possible implementation manner, the first electronic device is provided with a configuration page, and the method further includes: obtaining the first configuration information and the second configuration information through the configuration page.

[0039] In the above technical solution, the first electronic device obtains the configuration information (that is, the first configuration information and the second configuration information) in the manner of page configuration, so that the system use threshold is reduced. Compared with the manner of manually writing query statements by engineers in the traditional technology, the method can reduce the probability of errors in the written query statements, improve the accuracy of the data query result, and thus improve the accuracy of the data synchronization result between the search engine and the database.

[0040] In another possible implementation manner, the search engine is a distributed full-text search engine.

[0041] For example, the distributed full-text search engine can be, but is not limited to, an ES search engine or a RediSearch search engine.

[0042] In a second aspect, a data synchronization apparatus is provided, and the data synchronization apparatus is applied to a first electronic device, and the data synchronization apparatus includes a processing unit, and the processing unit is configured to execute any one of the methods in the first aspect.

[0043] In a third aspect, an electronic device is provided, and the electronic device includes a unit for executing any one of the methods in the first aspect. The device can be a terminal device or a chip in the terminal device. The device can include an input unit and a processing unit.

[0044] When the device is a terminal device, the processing unit can be a processor, and the input unit can be a communication interface. The terminal device can further include a memory for storing computer program code, and when the processor executes the computer program code stored in the memory, the terminal device executes any one of the methods in the first aspect.

[0045] When the device is a chip in a terminal device, the processing unit can be a processing unit inside the chip, and the input unit can be an output interface, a pin, a circuit, or the like; the chip can further include a memory, which can be a memory (for example, a register, a cache, or the like) inside the chip, or a memory (for example, a read-only memory, a random access memory, or the like) outside the chip; the memory is configured to store computer program code, and when the processor executes the computer program code stored in the memory, the chip is caused to execute any one of the methods in the first aspect.

[0046] In a possible implementation, the memory is configured to store computer program code; and the processor is configured to execute the computer program code stored in the memory, and when the computer program code stored in the memory is executed, the processor is configured to execute any one of the methods in the first aspect.

[0047] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer program code, and when the computer program code is run by a data synchronization apparatus, the data synchronization apparatus is caused to execute any one of the data synchronization methods in the first aspect.

[0048] In a fifth aspect, a computer program product is provided, and the computer program product includes computer program code, and when the computer program code is run by a data synchronization apparatus, the data synchronization apparatus is caused to execute any one of the data synchronization methods in the first aspect.

[0049] It can be understood that beneficial effects of the second aspect to the fifth aspect described above can be referred to the related description in the first aspect, and will not be described here again.

[0050] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it can be understood that the description of a feature or a beneficial effect means that the specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and beneficial effects described in the embodiments can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1AThis is a schematic diagram illustrating an application scenario of the data synchronization method provided in the embodiments of this application.

[0052] Figure 1B This is a schematic diagram illustrating another application scenario of the data synchronization method provided in the embodiments of this application.

[0053] Figure 2 This is a schematic diagram of the system architecture applicable to the data synchronization methods provided by traditional technologies.

[0054] Figure 3 This is a schematic diagram of a system architecture applicable to the data synchronization method provided in the embodiments of this application.

[0055] Figure 4 The above Figure 3 The diagram shown illustrates a specific application scenario corresponding to the system architecture.

[0056] Figure 5 The above Figure 3 The diagram illustrates another specific application scenario corresponding to the system architecture shown.

[0057] Figure 6 The above Figure 3 The diagram shown illustrates another specific application scenario corresponding to the system architecture.

[0058] Figure 7 The above Figure 3 The diagram shown illustrates another specific application scenario corresponding to the system architecture.

[0059] Figure 8 This is a schematic diagram of the software system of an electronic device that executes the embodiments of this application.

[0060] Figure 9 This is a schematic diagram of a data synchronization method provided in an embodiment of this application.

[0061] Figure 10 The above Figure 9 This is a diagram illustrating a configuration page corresponding to the configuration information provided in the data synchronization method.

[0062] Figure 11 The above Figure 9 This is a diagram illustrating another configuration page corresponding to the configuration information provided in the data synchronization method.

[0063] Figure 12 The above Figure 10 The provided configuration page shows a schematic diagram of a configuration interface corresponding to the data source.

[0064] Figure 13 The above Figure 9A schematic diagram of a data synchronization method provided in the present application is shown in FIG. 1.

[0065] Figure 14 A schematic diagram of another data synchronization method provided in the present application is shown in FIG. 2.

[0066] Figure 15 A schematic diagram of yet another data synchronization method provided in the present application is shown in FIG. 3.

[0067] Figure 16 A schematic diagram of yet another data synchronization method provided in the present application is shown in FIG. 4.

[0068] Figure 17 A schematic diagram of yet another data synchronization method provided in the present application is shown in FIG. 5.

[0069] Figure 18 A schematic diagram of a data synchronization device provided in the present application is shown in FIG. 6.

[0070] Figure 19 A schematic diagram of a structure of an electronic device provided in the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0072] In order to better understand the data synchronization method provided in the present application, first, the professional terms involved in the embodiments of the present application are briefly introduced.

[0073] 1. Full-text search engine

[0074] The full-text search engine is a mainstream search engine widely used at present. The working principle of the full-text search engine is that a computer indexing program establishes an index for each word in an article by scanning each word in the article, indicating the number of times and the position of the word appearing in the article. When a user queries, the retrieval program searches according to the previously established index and feeds back the search results to the user's retrieval method. This process is similar to the process of searching for a word in a dictionary through a retrieval word table.

[0075] For example, the full-text search engine can be but is not limited to an ES search engine or a RediSearch search engine.

[0076] Next, taking the full-text search engine as an example of ES, the differences and connections between the full-text search engine and the traditional relational database are introduced.

[0077] For example, Table A shows the comparison between the concepts in ES and the concepts in traditional relational databases.

[0078] Table A

[0079] Relational database MySQL ES search engine database Index Table Type row Document Column Field

[0080] As can be seen from Table A, the index in ES is similar to the database concept in the MySQL database. In simple terms, the index is a container for documents, a collection of a class of documents, a classification in logical space, and a sharding in physical space. The type in ES is similar to the table concept in the MySQL database. The type in ES is a logical classification or partition on the index in ES. The document in ES is similar to the row concept in the MySQL database. The field in ES is similar to the column concept in the MySQL database.

[0081] There can be multiple indexes in ES, each index can contain multiple types, each type can contain multiple documents, and each document can contain multiple fields.

[0082] ES uses the data structure of inverted index to accelerate retrieval, thereby improving data retrieval efficiency.

[0083] 2, Indexing method

[0084] The indexing method includes forward indexing and inverted indexing.

[0085] Forward indexing: It is a structure that uses the unique identifier ID of the document object as the index and uses the document content as the record.

[0086] Inverted indexing: It is a structure that uses the words in the document content as the index and uses the document ID containing the word as the record.

[0087] The indexing method in the traditional relational database (for example, the MySQL database) is forward indexing, and the indexing method in ES is inverted indexing.

[0088] The retrieval efficiency of inverted indexing is higher than that of forward indexing.

[0089] For example, when the MySQL database queries "mobile phone" based on the forward indexing method, the MySQL database will match whether the two characters "mobile phone" appear in each record (i.e. the row in the table). In fact, not all records contain "mobile phone", so a lot of useless work is done, and the retrieval efficiency is low. In addition, as the amount of data in the database increases, the resources and time consumed will increase linearly.

[0090] When the ES queries "mobile phone" based on the inverted index, since the ES has established the mapping relationship between the word "mobile phone" and the document in advance, the ES can directly query the corresponding document, thereby avoiding the query for the document without the word "mobile phone", and thus the search efficiency can be improved.

[0091] 3. Relational database

[0092] The relational database refers to a database using a relational model to organize data. In simple terms, the relational model is a two-dimensional table model. For example, the relational database can be, but is not limited to, any one of the following databases: MySQL, Oracle, DB2, Sybase or PostgreSQL database.

[0093] 4. Full-text search

[0094] Part of the information in the unstructured data is extracted, reorganized, and made to have a certain structure, and then the data with a certain structure is searched, so as to achieve the purpose of relatively fast search. This part of the information extracted from the unstructured data and then reorganized is called index. This process of first establishing an index and then searching the index is called full-text search.

[0095] 5. Data table has sub-tables

[0096] In the present application, a data table can be split to obtain multiple sub-data tables of the data table, wherein the fields included in the data table and the fields included in any one of the multiple sub-data tables are completely same, and the multiple sub-data tables collectively record the data in the data table, so that the data table has sub-tables.

[0097] For example, taking a user table as an example, when the number of users recorded in the user table is large, single table storage encounters a bottleneck, and the user information recorded in the user table needs to be split and stored in multiple data tables corresponding to multiple databases, such as splitting the user table into user table 1 and user table 2, and storing the user table 1 in database 1 and the user table 2 in database 2, wherein the database 2 and the database 1 are two different databases.

[0098] 6. Data table has no sub-tables

[0099] In the present application, a data table is not split, i.e., the data table has no sub-tables.

[0100] 7. Master table

[0101] A primary table is a table that contains a primary key and a unique identifier. The primary key of a table is a column in the table that represents uniqueness. That is, each row in the primary table must have a unique identifier. The primary key is a restriction condition that does not allow two rows to have the same identifier. A common primary table controls data input and guarantees data integrity, and a trigger can be written to maintain it.

[0102] In general, a primary table contains a directory of an entity. For example, a table named "student" contains information of a student, such as a student number, a name, a gender, a date of birth, and the like. The purpose of the primary table is to determine the location of data by a unique identifier and to provide a method of connecting to other tables.

[0103] 8. Secondary table

[0104] A secondary table is also called a slave table or an attached table. That is, the secondary table in the present application can be replaced by the description of the slave table or the attached table.

[0105] The secondary table exists based on the primary table and is associated by connecting common fields. The primary table and the secondary table can be connected in a left join or an inner join. The secondary table usually stores a lot of information related to the primary table and needs to acquire the information through the primary table. The secondary table can be associated with the primary table through a plurality of association fields, but the association fields must have uniqueness.

[0106] For example, a secondary table named "course" contains information of a course selected by a student, such as a course number, a student number, a score, and the like, and thus, the associated data needs to be acquired through the unique identifier "student number" of the primary table "student" to acquire the score information of the corresponding student.

[0107] 9. Ambiguous field

[0108] An ambiguous field refers to a field that has different values under different conditions. For example, the content field in Table B below is an example. Referring to Table B, the value of the content field corresponding to the value of the label identifier field is different under the condition that the course identifier is the same, that is, under the condition that the course identifier is 1, if the value of the label identifier field is "2", the value of the content field is "main title"; under the condition that the course identifier is 1, if the value of the label identifier field is "3", the value of the content field is "sub title"; based on this, the content field can be called an ambiguous field.

[0109] Table B

[0110] Course ID (course_id) Label identifier (label_id) Content 1 2 Main title 1 3 subtitle

[0111] 10. Binary log (Binlog)

[0112] The binary log can also be referred to as an update log. The binary log records all data definition language (DDL) and data manipulation language (DML), but does not record data query statements (for example, SELECT statements). For example, the language for creating a database and creating a data table are DDL. For example, INSERT, UPDATE and DELETE are DML.

[0113] Next, the application scenarios to which the data synchronization method provided by the embodiments of the present application is applied are introduced in combination with the accompanying drawings.

[0114] Figure 1A is a schematic diagram of an application scenario to which the data synchronization method provided by the embodiments of the present application is applied.

[0115] For example, referring to Figure 1A the application scenario shown in FIG. 1, the application scenario includes a first electronic device 110 and a second electronic device 120. The first electronic device 110 and the second electronic device 120 can communicate through an end-to-end communication mode. The communication connection between the first electronic device 110 and the second electronic device 120 can be a short-distance communication connection, for example, a Wi-Fi point-to-point connection or a Bluetooth connection, etc. The communication connection between the first electronic device 110 and the second electronic device 120 can also be a long-distance communication connection, for example, an Internet connection, etc. The specific type of the end-to-end communication mode is not limited herein.

[0116] In the embodiments of the present application, the device types of the first electronic device 110 and the second electronic device 120 are not limited. For example, the first electronic device 110 and the second electronic device 120 can be non-terminal devices such as servers, or any one of the electronic devices can be a terminal device such as a mobile phone, a tablet computer and a notebook computer, etc.

[0117] As shown in Figure 1A , the data table #1, …, the data table #N are all located in the local database 121 of the second electronic device 120, where N is a positive integer; and the search engine is located in the first electronic device 110. It should be understood that the local database 121 of the second electronic device 120 is a relational database, for example, the relational database can be but is not limited to any one of the following databases: Mysql, Oracle, DB2, Sybase or PostgreSQL.

[0118] In the above Figure 1AIn the shown application scenario, in order to improve the efficiency of data retrieval, the data (e.g., part of the data or all of the data) in the data tables (e.g., N data tables) in the local database 121 of the second electronic device 120 can be synchronized to the index of the search engine of the first electronic device 110 according to business needs. Thereafter, the purpose of searching the data in the N data tables in the local database 121 of the second electronic device 120 in the search engine can be achieved.

[0119] It should be noted that the above Figure 1A In the shown scenario, the N data tables are all data tables located in the same database (i.e., the database 121). Alternatively, Figure 1A The N data tables shown can also be located in multiple different databases.

[0120] For example, taking N equal to 2 as an example, the above Figure 1A The shown application scenario can also be replaced by Figure 1B The shown application scenario. As Figure 1B shown, the data table #1 is located in the database 121 of the second electronic device 120, and the data table #2 is located in the database 131 of the third electronic device 130, that is, the database 121 and the database 131 are two different databases.

[0121] In some application scenarios, such as vertical search scenarios, it is usually necessary to jointly query the data in multiple business data tables to obtain query results that meet business query requirements, wherein the fields included in the multiple business data tables can not be completely the same. Next, in combination with Figure 2 the implementation process of synchronizing the data in the multiple data tables located in the multiple databases to the index of the search engine is introduced.

[0122] Figure 2 is a schematic diagram of a system architecture to which the data synchronization method provided in the prior art is applicable. For example, referring to Figure 2 , the system architecture includes a business system database 210, an electronic device 220, and an electronic device 230.

[0123] The business system database 210 includes 2 databases (i.e., database 1 and database 2), wherein the 2 databases correspond to 2 data tables, and each database is used to store the corresponding data table. The multiple databases in the business system database 210 can be databases in the same server or databases in multiple servers, and no specific limitation is made thereto.

[0124] The electronic device 220 includes an Extract-Transform-Load (ETL) architecture 221, where the ETL architecture 221 includes data extraction (Extract) 2211, data transformation (Transform) 2212, and data loading (Load) 2213.

[0125] The data extraction 2211 is configured to extract data from various data sources. As shown, the data extraction 2211 is configured to extract source data from the two databases included in the business system database 210. Figure 2

[0126] The data transformation 2212 is configured to clean, transform, and arrange the data extracted by the data extraction 2211 to meet the requirements of the target system (e.g., the requirements of the data format stored in the search engine 2311). For example, the data transformation can include, but is not limited to, data cleaning, format conversion, data merging (i.e., merging multiple data tables, field merging), and the like.

[0127] The data loading 2213 is configured to load the data processed by the data transformation 2212 into the index of the search engine 2311 included in the data warehouse 231. The manner of data loading by the data loading 2213 is not specifically limited and can be selected according to the actual situation. For example, the data loading can be performed in a batch loading, incremental loading, or real-time loading manner.

[0128] The electronic device 230 includes a data warehouse 231, where the data warehouse 231 includes a search engine 2311. The search engine 2311 is configured to provide retrieval services to users and store the data synchronized from the two data tables corresponding to the two databases in the business system database 210.

[0129] Based on the above Figure 2 ​The principle of synchronizing the data in the two data tables corresponding to the two databases in the business system database 210 to the index field of the search engine is as follows: first, an engineer manually writes two data query statements (for example, select statements in a MySQL database) corresponding to a data synchronization request based on an interface provided by the data extraction 2211, wherein the two data query statements correspond to the two data tables one by one, and each data query statement is used to query the data in the corresponding data table. Then, the data extraction 2211 performs a query operation on the two data tables according to the plurality of data query statements to obtain two query results corresponding to the two data query statements, wherein each query result is obtained by performing a query operation on the data table in the corresponding database by using the corresponding data query statement. Next, the data conversion 2212 obtains the two query results from the data extraction 2211 and performs a data conversion (including data filtering and data merging) operation on the two query results to obtain target data meeting the requirements of the search engine. Finally, the data loading 2213 loads the target data obtained from the data conversion 2212 to the index of the search engine 2311 located in the data warehouse 231.

[0130] It can be known that, in the process of implementing data synchronization in the prior art, on the one hand, the query statement used to perform a data query is manually written by an engineer, and there is a phenomenon of errors in the written query statement, which leads to problems of being unable to query a result or the queried result being untrustworthy, thereby leading to inaccurate data synchronization results. On the other hand, after the data extraction 2211 obtains a plurality of query results (for example, the two query statements), the data conversion 2212 needs to perform a data conversion operation on the plurality of query results, and the execution steps of the data conversion operation are relatively complex, which leads to a problem of low query efficiency when data synchronization is performed based on the prior art. In conclusion, when data in a database is synchronized to a search engine based on the prior art, there are problems of inaccurate data synchronization results and low data synchronization efficiency.

[0131] In order to solve the above problems in the prior art, the present application provides a data synchronization method, device and electronic equipment. Next, a schematic diagram of a system architecture suitable for the data synchronization method provided by the embodiments of the present application is introduced in combination with the drawings.

[0132] Figure 3 is a schematic diagram of a system architecture suitable for the data synchronization method provided by the embodiments of the present application.

[0133] As Figure 3 shown, the system architecture includes at least one data table (for example, the data table 310), a data synchronization service 320, a mirror database 330, a meta database 340 and a search engine 350.

[0134] The at least one data table is used to store the data generated by the business party in the form of a two-dimensional data table. The data stored in the at least one data table is not specifically limited, and can be set according to actual business needs. For example, the data recorded in the at least one data table can be data generated by a personnel management application, or data generated by a project management application, and the like.

[0135] The data synchronization service 320 is used to generate a data query statement, and perform a query operation on the at least one data table according to the data query statement, to obtain the to-be-synchronized data corresponding to the data query statement, and synchronize the to-be-synchronized data to the index of the search engine 350.

[0136] For example, in some implementations, the data synchronization service 320 can provide a configuration page service to the business party server storing the at least one data table, the business party server can set the data query information based on the configuration information provided by the configuration page service, and after the business party server successfully sets the data query information through the interface, the data synchronization service can generate a corresponding data query statement based on the data query information.

[0137] The mirror database 330 is used to back up the at least one data table, that is, the mirror database 330 stores the at least one data table. In the system architecture provided in the present application, by introducing the mirror database 330, frequent query operations on the at least one data table of the business party can be avoided, so that the influence on other businesses in the business party where the at least one data table is located can be avoided.

[0138] Optionally, the above Figure 3 The system architecture shown can also not include the mirror database 330.

[0139] In some implementations, if Figure 3 The architecture shown includes one or more data tables, which are located in one database. In this implementation, Figure 3 The architecture shown can include the mirror database 330, or can also not include the mirror database 330. It can be understood that, in Figure 3 In the case where the architecture shown does not include the mirror database 330, the data synchronization service 320 can directly query the at least one data table by using the generated data query statement to obtain the to-be-synchronized data, and synchronize the to-be-synchronized data obtained by the query to the index of the search engine 350.

[0140] In other implementations, Figure 3 The architecture shown includes a plurality of data tables, which can be located in a plurality of different databases. In this implementation, Figure 3 The architecture shown should include the mirror database 330.

[0141] Metadatabase 340 is used to store metadata used by data synchronization service 320 when executing data synchronization methods, and to store data query statements generated by data synchronization service 320. For example, the above metadata includes, but is not limited to, the following information: database information (e.g., database name and the server where the database is located), data table information (e.g., data table name and the relationship between data table and database), linking methods, and the relationship between data tables (e.g., the course table and the tag table are associated through course identifiers; or, the course table is the main table of the tag table, and the tag table is the secondary table of the course table), etc.

[0142] Optionally, in other implementations, where the data synchronization service 320 is capable of storing metadata, the above... Figure 3 The system architecture shown may also exclude the metadata database 340.

[0143] The search engine 350 is used to store the data to be synchronized sent by the data synchronization service 320 and to provide users with retrieval and query services. In this embodiment, the search engine is not specifically limited and can be selected according to actual needs. For example, the search engine can be, but is not limited to, Elasticsearch or RediSearch.

[0144] Optionally, in some implementations, a message queue can also be set up between the data synchronization service 320 and the search engine 350. Figure 3 (Not shown in the image), the message queue is used to cache the data to be synchronized sent by the data synchronization service 320. That is, the data synchronization service 320 first temporarily stores the data to be synchronized obtained from the query into the message queue. The downstream search engine 350 can read the data to be synchronized from the message queue at its own pace. In this way, the phenomenon of the search engine 350 being blocked due to the large amount of data to be synchronized can be avoided. For example, the message queue can be a message queue like Kafka.

[0145] In the embodiments of this application, for Figure 3 The servers where the various modules shown are located are not specifically limited and can be configured according to actual business needs. In other words, the above... Figure 3 The modules shown can be logical or physical divisions, without any specific limitations.

[0146] For example, Figure 3 The multiple data tables shown can be located on the same server 1; the data synchronization service 320, the mirror database 330, and the metadata database 340 can be located on the same server 2; the search engine 350 can be located on server 3; wherein, server 3, server 2, and server 1 are three different servers.

[0147] For example, Figure 3 The plurality of data tables shown can be located in a plurality of different servers; the data synchronization service 320, the mirror database 330, the meta database 340 and the search engine 350 can be located in the same server other than the plurality of different servers.

[0148] It should be understood that the above Figure 3 The system architecture shown is only illustrative and does not constitute any limitation on the system architecture to which the data synchronization method provided by the embodiments of the present application is applicable. For example, the above Figure 3 The system architecture shown can further include a larger number (e.g., 3 or 5, etc.) of data tables. For another example, the above Figure 3 The system architecture shown can further include a message queue for buffering data sent by the data synchronization service 320 to the mirror database 330. For another example, the above Figure 3 The mirror database 330 and the meta database 340 in the system architecture shown can be the same database.

[0149] The above describes Figure 3 The roles of the modules in the system architecture shown. In the above Figure 3 When the system architecture shown includes a plurality of data tables, the plurality of data tables can be located in the same database or a plurality of different databases; and some of the plurality of data tables are data tables obtained by splitting one data table, or some of the plurality of data tables are not data tables obtained by splitting one data table. It can be understood that the fields included in one data table before splitting and the fields included in any one of the plurality of sub-data tables obtained by splitting the one data table are exactly the same, and the plurality of sub-data tables collectively record the data in the one data table.

[0150] Next, based on the above Figure 3 The system architecture shown includes a plurality of data tables, and in combination with the case where the plurality of data tables are located in a database and whether some of the plurality of data tables are tables obtained by splitting one data table, the above Figure 3 The system architecture shown corresponds to a plurality of specific application scenarios.

[0151] For ease of description, the following describes Figure 3 The plurality of data tables shown are taken as an example of M data tables, where M is an integer greater than 1.

[0152] In some application scenarios, the above Figure 3The M data tables are located in at least one database, and no part of the M data tables is a data table obtained by splitting a same business data table. In this implementation manner, the M data tables have a connection relationship, the data tables having the connection relationship have a same column name, and the fields of any two data tables in the M data tables are not completely same.

[0153] For ease of description, in the following of the present application, the scenario that the M data tables are located in a same database and no part of the M data tables is a data table obtained by splitting is briefly denoted as scenario one, that is, the scenario one is a single-database and data table non-splitting scenario.

[0154] Exemplarily, Figure 4 A schematic diagram of a specific application scenario of the scenario one is shown. As shown in the diagram, Figure 4 The data table 1 and the data table 2 in the scenario one are located in a local database A of the server 410, the server 420 includes a mirror database and a data synchronization service (not shown in the figure), and the server 430 includes a search engine and a message queue. Figure 4

[0155] The above Figure 4 The two data tables shown can be a user table and a user item table, wherein the user table includes an identification field and a username field, and the user item table includes a username field and an item field, that is, the user table and the user item table include a same username field, that is, the connection relationship between the user table and the user item table is represented by the username field, and the user table and the user item table are not split tables. Optionally, the above Figure 4 The database A shown can also include only one data table (for example, the data table 1 or the data table 2).

[0156] For ease of description, in the following of the present application, the scenario that the M data tables are located in multiple databases and no part of the M data tables is a data table obtained by splitting is briefly denoted as scenario two, that is, the scenario two is a multi-database and data table non-splitting scenario.

[0157] Exemplarily, Figure 5 A schematic diagram of a specific application scenario of the scenario two is shown. As shown in the diagram, Figure 5 The data table 1 in the scenario two is located in a local database A of the server 510, and the data table 2 is located in a local database B of the server 520, wherein the database A and the database B are two different databases; the server 530 includes a mirror database and a data synchronization service (not shown in the figure); and the server 540 includes a search engine and a message queue. Figure 5

[0158] The above Figure 5 The examples of the data table 1 and the data table 2 shown can be referred to the examples of the data table 1 and the data table 2 in the scenario one.​​ Figure 4 The description of Data Table 1 and Data Table 2 in the foregoing is not repeated here in detail.

[0159] In some other application scenarios, the M data tables shown above Figure 3 The M data tables shown above are located in the same database, and some of the M data tables are data tables obtained by splitting a same data table. In this implementation manner, the fields of the some of the M data tables are completely same, and the fields of the remaining data tables are not completely same. In addition, the remaining data tables have a connection relationship, and the data tables having the connection relationship have same column names. It can be understood that, in this implementation manner, any one of the some of the M data tables is a split table, and any one of the remaining data tables is not a split table.

[0160] For ease of description, in the following of the present application, the scenario that the M data tables shown above are located in the same database and some of the M data tables are split tables is briefly denoted as scenario three, i.e., the scenario three is a single-database split-table scenario.

[0161] Exemplarily, Figure 6 A schematic diagram of a specific application scenario of the scenario three is shown. As shown in the diagram, Figure 6 The scenario three includes three data tables, i.e., Data Table 1 Split_0, Data Table 1 Split_1 and Data Table 2. The Data Table 1 Split_0, the Data Table 1 Split_1 and the Data Table 2 are located in a local database A of a server 610. The server 620 includes a mirror database and a data synchronization service (not shown in the diagram), where the mirror database includes the Data Table 2 and the Data Table 1, and the Data Table 1 is obtained by merging the Data Table 1 Split_0 and the Data Table 1 Split_1. The server 630 includes a search engine and a message queue. Figure 6

[0162] It should be understood that the foregoing Figure 6 The Data Table 1 Split_0 and the Data Table 1 Split_1 shown above can be two tables obtained by splitting a user table including 100 rows of data. In a case where the user table includes an identifier field and a username field, the two data tables can be the Data Table 1 Split_0 including the identifier field and the username field, and the Data Table 1 Split_1 including the identifier field and the username field. The Data Table 1 Split_0 can be used to record the first 50 rows of data in the user table, and the Data Table 1 Split_1 can be used to record the last 50 rows of data in the user table, i.e., the Data Table 1 Split_0 and the Data Table 1 Split_1 are both split tables of the Data Table 1.

[0163] Optionally, the database A shown above Figure 6 may not include the Data Table 2.

[0164] ​For ease of description, the above M data tables are located in multiple databases, and the M data tables exist in a scenario of table splitting in the following of the present application, which is referred to as scenario four, i.e., scenario four is multiple databases, and the data tables have the scenario of table splitting.

[0165] Exemplarily, Figure 7 A schematic diagram of one specific application scenario of scenario four is shown. As shown in the figure, Figure 7 The scenario four includes three data tables, i.e., data table 1_split_0, data table 1_split_1 and data table 2. The data table 1_split_0 and the data table 2 are located in the local database A of the server 710, the data table 1_split_1 is located in the local database B of the server 720, and the database B and the database A are two different databases. The server 730 includes a mirror database and a data synchronization service (not shown in the figure), wherein the mirror database includes the data table 2 and the data table 1, and the data table 1 is obtained by merging the data table 1_split_0 and the data table 1_split_1. The server 740 includes a search engine and a message queue. Figure 7

[0166] The above Figure 7 The examples of the data table 1_split_0, the data table 1_split_1 and the data table 2 shown in the figure can refer to the description of the data table 1_split_0, the data table 1_split_1 and the data table 2 in the above Figure 6 The examples of the data table 1_split_0, the data table 1_split_1 and the data table 2 shown in the figure can refer to the description of the data table 1_split_0, the data table 1_split_1 and the data table 2 in the above

[0167] Optionally, the database A shown in the figure can not include the data table 2. Figure 7 The above

[0168] The roles of the mirror database, the message queue and the search engine shown in the figure can refer to the description of the corresponding modules in the above Figures 4 to 7 The roles of the mirror database, the message queue and the search engine shown in the figure can refer to the description of the corresponding modules in the above Figure 3

[0169] It should be understood that the application scenarios shown in the above Figures 4 to 7 The application scenarios shown in the above Figures 4 to 7 The application scenarios shown in the above Figures 4 to 7 The application scenarios shown in the above Figure 3 The application scenarios shown in the above

[0170] As described above, the data synchronization method provided by the embodiments of the present application can be applied to two electronic devices. Next, the software structure of the electronic device is described in detail with reference to the accompanying drawings. It can be understood that the following Figure 8 ​​The illustrated electronic device software architecture can be the software architecture of each of at least two electronic devices that perform the data synchronization method provided in the embodiments of this application.

[0171] For example, Figure 8 A schematic diagram of a software system for an electronic device implementing embodiments of this application is shown. See also Figure 8 The software system adopts a layered architecture. This layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer 810, the application framework layer 820, the Android Runtime and core library layer 830, the hardware abstraction layer (HAL) 840, and the kernel layer 850.

[0172] Application layer 810 may include a series of application packages. For example, application packages may include applications such as camera, gallery, chat, call, map, navigation, calendar, Bluetooth, music, video, and data synchronization services. It is understood that the data synchronization service located in application layer 810 can perform the data synchronization methods provided in this application.

[0173] The aforementioned applications may include more specific functional modules; for example, a gallery may contain business modules and notification modules.

[0174] The application data generated by the aforementioned applications can be stored in a data table in the local database of the electronic device. For example, data generated by a table application can be stored in a data table in the local database of the electronic device.

[0175] The application framework layer 820 provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer 820 includes some predefined functions.

[0176] like Figure 8 As shown, the application framework layer 820 may include a window manager, notification manager, activity manager, input manager, view system, content provider, resource manager, etc.

[0177] The window manager provides a window management service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for the input system.

[0178] A content provider stores and provides data to be accessed by applications. The data can include videos, images, audios, dialed and received phone numbers, browsing history and bookmarks, phone book, etc.

[0179] A view system includes visual controls, such as a control for displaying text, a control for displaying an image, etc. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying an image.

[0180] A resource manager provides various resources for an application, such as localized strings, icons, images, layout files, video files, etc.

[0181] A notification manager enables an application to display notification information in a status bar, which can be used to convey a message of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify a download completion, a message reminder, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application running in the background, and can also be a notification in the form of a dialog window appearing on the screen. For example, a text information is prompted in the status bar, a prompt sound is emitted, the electronic device is vibrated, a light flashes, etc.

[0182] An activity manager can provide an activity manager service (AMS), which can be used for the start, switching, scheduling of system components (such as activities, services, content providers, broadcast receivers), and the management and scheduling of application processes.

[0183] An input manager can provide an input manager service (IMS), which can be used to manage the input of the system, such as touch screen input, key input, sensor input, etc. The IMS takes events from input device nodes and distributes the events to appropriate windows through interaction with the WMS.

[0184] An Android runtime includes a core library and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0185] The core library includes two parts: one part is a function function that a programming language (such as the java language) needs to call, and the other part is the core library of Android.

[0186] The application program layer 810 and the application framework layer 820 run in a virtual machine. The virtual machine executes the programming files (for example, java files) of the application program layer 810 and the application framework layer 820 into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, management of security and exceptions, and garbage collection.

[0187] The core library layer 830 can include a plurality of functional modules. For example: a surface manager, a media framework, libc, SQLite, OpenGL ES, Webkit, and the like.

[0188] It can be understood that the database involved in the present application can be but is not limited to being stored in the core library layer, and the binary log used to implement the data incremental synchronization can also be but is not limited to being stored in the core library layer.

[0189] The surface manager is used to manage the display subsystem, and provides fusion of two-dimensional (2-Dimensional, 2D) and three-dimensional (3-Dimensional, 3D) layers for a plurality of application programs.

[0190] The media framework supports playback and recording of a plurality of commonly used audio, video formats, and static image files. The media library can support a plurality of audio and video coding formats, for example: MPEG6, H.266, MP3, AAC, AMR, JPG, PNG, and the like.

[0191] The hardware abstraction layer (HAL) 840 is an interface layer between the operating system kernel and the upper layer software, and the purpose is to abstract the hardware. The hardware abstraction layer is an abstract interface of the device kernel driver, and is used to implement an application programming interface for providing access to the underlying device to a higher level Java API framework. The HAL includes a plurality of library modules, for example: a camera HAL, a Vendor repository, a display screen, Bluetooth, audio, and the like. Each library module implements an interface for a specific type of hardware component. It can be understood that the camera HAL can provide an interface for the camera FWK to access hardware components such as a camera. The Vendor repository can provide an interface for the media FWK to access hardware components such as an encoder. When the system framework layer API requires access to the hardware of the portable device, the Android operating system will load the library module for the hardware component.

[0192] The kernel layer 850 is the basis of the Android operating system, and the ultimate functions of the Android operating system are all completed through the kernel layer. The kernel layer can include a display driver, a camera driver, an audio driver, and a sensor driver.

[0193] It should be noted that the Figure 8 The software structure diagram of the electronic device shown is only an example, and does not limit the specific module division in different layers of the Android operating system. For details, refer to the introduction of the software structure of the Android operating system in the conventional technology. In addition, the photographing method provided in the present application can also be implemented based on other operating systems (for example, IOS or Harmony, etc.), and the present application will not be illustrated one by one.

[0194] Next, the data synchronization method provided by the embodiment of the present application will be described in detail. Figures 9 to 17 The data synchronization method provided by the embodiment of the present application will be described in detail.

[0195] Figure 9 is a schematic diagram of a data synchronization method provided by the embodiment of the present application. The data synchronization method provided by the embodiment of the present application can be executed by an electronic device. It can be understood that the electronic device can be implemented as software, or a combination of software and hardware. For example, the electronic device executing the data synchronization method of the embodiment of the present application can be, but is not limited to, a server including the above-mentioned Figure 3 The server of the data synchronization service 320 shown.

[0196] For example, as Figure 9 The data synchronization method provided by the embodiment of the present application includes steps S910 to S960. Next, S910 to S960 will be described in detail.

[0197] S910, the first electronic device generates K first query statements according to the first configuration information, wherein the first configuration information includes the table names of K source data tables and the K source data tables are stored in at least one first database, the K first query statements correspond to the K source data tables one by one, each first query statement is used to query the corresponding source data table, and K is an integer greater than 1.

[0198] In the above S910 step, the K source data tables are located in at least one first database, that is, the K source data tables can be located in one first database or multiple first databases, and the number of first databases corresponding to the K source data tables is not specifically limited.

[0199] For example, taking the K source data tables including source data table 1 and source data table 2 as an example, the source data table 1 and the source data table 2 can be located in the database A.

[0200] For example, taking the K source data tables including source data table 1 and source data table 2 as an example, the source data table 1 can be located in the database A, and the source data table 2 can be located in the database B, and the database A and the database B are servers located in different electronic devices.

[0201] In the embodiments of the present application, the implementation of the first electronic device generating K first query statements according to the first configuration information in S910 is not specifically limited.

[0202] In some implementations, the first electronic device generates K first query statements according to the first configuration information, including selecting a first generator from a plurality of candidate generators according to the first configuration information, wherein the plurality of candidate generators correspond to the plurality of types of statements, each candidate generator is used to generate a statement of a corresponding type, and the first generator is used to generate a statement for querying each source data table; and the first electronic device processes the first configuration information by using the first generator to obtain K first query statements. It can be understood that in this implementation, the principle of the first electronic device generating K first query statements is the same as the principle of the second electronic device generating second query statements in S940 below, and the content not described in detail here can be referred to the related description in S940 below.

[0203] It should be understood that in S910, the first configuration information includes the table names of K source data tables and the K source data tables are stored in at least one first database as an example.

[0204] Optionally, the first configuration information in the above implementation can further include path information of the K source data tables, for example, the path information of the K source data tables can include a path for indicating a database storing the K source data tables, a username and a password for accessing the database storing the K source data tables. For another example, the path information of the K source data tables can include a path for indicating a database storing the K source data tables.

[0205] S920, the first electronic device synchronizes K source data tables located in at least one first database to a second database according to K first query statements, so that P target data tables corresponding to the K source data tables are stored in the second database, wherein the data in the K source data tables is generated by an application of an electronic device where the at least one first database is located, and the electronic device where the at least one first database is located and the electronic device where the second database is located are different, and P is an integer greater than 1 and not more than K.

[0206] In S920, P is an integer greater than 1 and not more than K.

[0207] In some implementations, P is equal to K, and the P target data tables are the K source data tables, and the fields of any two source data tables in the K source data tables are not completely the same.

[0208] In the above implementation, the P target data tables are the K source data tables, that is, the backup (that is, synchronization) of the K source data tables by the first electronic device is involved in the above implementation process, but the operation step of merging the K source data tables by the first electronic device is not involved.

[0209] For example, when K is equal to 2, the K source data tables (that is, the P target data tables) can be Table 1 and Table 2 as follows.

[0210] Table 1

[0211] Course identifier (id) Course Name (course_name) Status 1 math 1 2 Chinese 0

[0212] The value of the state field in Table 1 above is used to indicate whether the row data where the state field is located is in an available state. For example, the value of the state field is equal to "1", indicating that the row data where the state field is located is available state data, and the value of the state field is equal to "0", indicating that the row data where the state field is located is unavailable state data.

[0213] Table 2

[0214] Course identifier (id) Label identifier (label_id) Content 1 49 Trigonometric functions 1 50 sine function 2 51 article 2 52 prose

[0215] It can be seen that Table 1 and Table 2 above have an id field with the same name, and Table 1 and Table 2 are connected through the id field.

[0216] In some other implementations, P is less than K, the P target data tables include a data table obtained by merging part of the K source data tables, and the remaining source data tables in the K source data tables except for the part of the source data tables, wherein any two source data tables in the part of the source data tables include the same fields, any one source data table in the remaining source data tables includes fields that are not completely the same as the fields included in any one source data table in the part of the source data tables, and the K source data tables located in the at least one first database are synchronized to the second database according to the K first query statements, so that the second database stores the P target data tables corresponding to the K source data tables, including: synchronizing the part of the source data tables located in the at least one first database to the same data table in the second database according to part of the K first query statements, and synchronizing the remaining source data tables located in the at least one first database to the second database according to the statements except for the part of the K first query statements, so that the second database stores the P target data tables.

[0217] It can be understood that in the above implementation, since P is an integer greater than 1 and less than K, and K is an integer greater than 1, K in the above implementation should be an integer greater than or equal to 3.

[0218] The part of the K source data tables includes the same fields, i.e., the part of the K source data tables is a plurality of sub-data tables obtained by splitting a business table.

[0219] For example, the part of the K source data tables includes two source data tables, which can include Table 1.0 and Table 1.1 in the following.

[0220] Table 1.0

[0221] Course identifier (id) Course Name (course_name) Status 1 math 1

[0222] Table 1.1

[0223] Course identifier (id) Course Name (course_name) Status 2 Chinese 0

[0224] It can be understood that the Table 1.0 and Table 1.1 are data tables obtained by splitting the Table 1 in the above, i.e., the Table 1.0 and Table 1.1 are sub-tables of the Table 1 in the above. The Table 1.0 and Table 1.1 can be stored in the same database of the same server, different databases of the same server, or different databases of different servers, which is not limited specifically.

[0225] Any one of the remaining source data tables is not a plurality of sub-data tables obtained by splitting a business table, so that in the case that the number of the source data tables included in the remaining source data tables is greater than or equal to two, the fields included in any two of the remaining source data tables are not completely the same.

[0226] For example, the remaining source data tables of the K source data tables include two source data tables, which can be the data tables shown in the Table 1 and Table 2 in the above.

[0227] The fields included in any one of the remaining source data tables and the fields included in any one of the part of the source data tables are not completely the same, which can mean that the fields included in any one of the remaining source data tables and the fields included in any one of the part of the source data tables are completely different, or that part of the fields included in any one of the remaining source data tables and part of the fields included in any one of the part of the source data tables are the same, which is not limited specifically.

[0228] For example, the remaining source data tables of the K source data tables can be the Table 2 in the above, and the part of the source data tables of the K source data tables include the Table 1.0 and Table 1.1 in the above.

[0229] In the implementation manner, part of the K source data tables are multiple data tables obtained by splitting a data table, and thus, the part of the source data tables can be merged into one data table stored in the second database in the data backup process, so that the efficiency of subsequent query operations on the P target data tables is improved.

[0230] In the S920, before the first electronic device synchronizes the K source data tables in the at least one first database to the second database according to the K first query statements, if none of the K source data tables is stored in the at least one first database, the synchronization process of the first electronic device synchronizing the K source data tables in the at least one first database to the second database according to the K first query statements is a full data synchronization process.

[0231] Optionally, after the first electronic device synchronizes the K source data tables in the at least one first database to the second database, if part of the data in the K source data tables in the at least one first database changes, the first electronic device can further perform a step of synchronizing the changed part of the data in the at least one first database to the second database, so that the data in the second database after the re-synchronization is consistent with the data in the at least one first database, and thus, the reliability of the data in the data synchronization between the search engine and the database is ensured. It can be understood that, in this implementation manner, the process of synchronizing the changed part of the data in the at least one first database to the second database is an incremental data synchronization process, and the incremental data synchronization process can be implemented according to a binary log, wherein the binary log records an operation statement causing the change of the part of the data in the K source data tables, for example, the operation statement can be but is not limited to an insertion statement, a deletion statement or an update statement.

[0232] S930, the first electronic device obtains second configuration information, wherein the second configuration information includes mapping information, a filtering condition, a connection relationship of the P target data tables and storage of the P target data tables in the second database, the mapping information includes a mapping relationship between multiple fields in an index of the search engine and multiple to-be-queried fields corresponding to the P target data tables, the filtering condition is used to filter field values of the to-be-queried fields corresponding to at least one target data table satisfying a preset condition, and P is an integer greater than 1.

[0233] In the S930, the second configuration information is taken as an example and is introduced, and the second configuration information can further include other information, and the other information can be configured according to a specific business scenario.

[0234] For example, the other information can include information indicating whether one of the plurality of fields in the index is a participating field in the retrieval. In the case where one of the fields in the index is a participating field in the retrieval, the one field can be used as a retrieval field. For another example, the other information can also be information indicating that the target data table is split into sub-tables. For another example, in the case where one target data table exists in sub-tables, the other information can also include a name of a group, where the name of the group is a name of the target data table that exists in sub-tables. For another example, the other information can also include path information of a second database storing the P target data tables, where the path information of the second database can include a url of the second database, a username and a password for accessing the second database.

[0235] Optionally, in the case where the second configuration information in the S930 further includes path information of the second database, before the S930, i.e., before the first electronic device performs the joint query on the P target data tables in the second database according to the query statement to obtain the target data, the following step can be further performed: the first electronic device acquires the P target data tables according to the path information of the second database.

[0236] The mapping information includes a mapping relationship between the plurality of fields in the index of the search engine and the plurality of to-be-queried fields corresponding to the P target data tables, where the plurality of fields are different, and the plurality of to-be-queried fields are different. It should be understood that the field values of the fields in the index that exist in the mapping relationship should be consistent with the field values of the to-be-queried fields in the corresponding target data table. That is, in the embodiments of the present application, the purpose of the data synchronization is to synchronize the field values of the plurality of to-be-queried fields in the P target data tables that satisfy the preset condition to the field values of the plurality of fields in the corresponding index.

[0237] The mapping information includes a mapping relationship between the plurality of fields in the index of the search engine and the plurality of to-be-queried fields corresponding to the P target data tables, where the plurality of fields are different, and the plurality of to-be-queried fields are different. It should be understood that the field values of the fields in the index that exist in the mapping relationship should be consistent with the field values of the to-be-queried fields in the corresponding target data table. That is, in the embodiments of the present application, the purpose of the data synchronization is to synchronize the field values of the plurality of to-be-queried fields in the P target data tables that satisfy the preset condition to the field values of the plurality of fields in the corresponding index.

[0238] The P target data tables correspond to a plurality of fields to be queried, wherein each of the P target data tables corresponds to at least one of the plurality of fields to be queried. It should be understood that the field to be queried corresponding to each target data table is a field to be queried in the target data table.

[0239] The number of the P target data tables, the number of the fields to be queried corresponding to each target data table, the fields to be queried corresponding to each target data table, the number of the plurality of fields in the index, and whether the data amount of the plurality of fields to be queried corresponding to the P target data tables are equal are not specifically limited and can be selected according to actual conditions.

[0240] The mapping relationship between the plurality of fields in the index and the plurality of fields to be queried corresponding to the P target data tables described by the mapping information is described below by way of example. The mapping relationship between the plurality of fields and the plurality of fields to be queried can be a non-conditional mapping relationship or a conditional mapping relationship, which can be set according to the conditions of the plurality of fields to be queried and is not specifically limited. It should be understood that the mapping relationship between the plurality of fields and the plurality of fields to be queried described by way of example below is only illustrative and does not constitute any limitation on the mapping relationship between the plurality of fields and the plurality of fields to be queried indicated by the mapping information in the embodiments of the present application.

[0241] In some implementations, the P target data tables include a first data table and a second data table, wherein the field to be queried corresponding to the first data table is a first field, the fields to be queried corresponding to the second data table include a second field and a third field, the plurality of fields include a fourth field, a fifth field, and a sixth field, and the mapping relationship includes a non-conditional mapping relationship and a conditional mapping relationship, wherein the non-conditional mapping relationship is a mapping relationship between the fourth field and the first field, the conditional mapping relationship includes a mapping relationship between the fifth field and the third field when the field value of the second field is a first value and a mapping relationship between the sixth field and the third field when the field value of the second field is a second value, and the first value and the second value are different.

[0242] In the above implementations, the plurality of fields to be queried corresponding to the P target data tables include a first field, a second field, and a third field.

[0243] The first data table in the above implementations can be a primary table, and the second data table can be a secondary table. For example, the first data table can be Table 1 described above, and the second data table can be Table 2 described above.

[0244] In the case that the first data table is Table 1, the second data table is Table 2 in the above implementation manners, the first field can be the id field in Table 1, the second field can be the label_id field in Table 2, and the third field can be the content field in Table 2. Based on this, for example, the fourth field can be the id field in the index, the fifth field can be the title field in the index, and the sixth field can be the sub_title field in the index. It can be understood that the content field in Table 2 is an ambiguous field.

[0245] Optionally, in some other implementation manners, the P target data tables include a first data table and a second data table, the first data table corresponds to a first field as the to-be-queried field, the second data table corresponds to a second field as the to-be-queried field, the plurality of fields include a third field and a fourth field, and the mapping relationship includes a mapping relationship between the third field and the first field and a mapping relationship between the fourth field and the second field.

[0246] In the above implementation manners, the plurality of to-be-queried fields corresponding to the P target data tables include a first field and a second field.

[0247] In the above implementation manners, the mapping relationship between the third field and the first field and the mapping relationship between the fourth field and the second field are both non-conditional mapping relationships, that is, the mapping relationship between the third field and the first field exists without needing to meet a certain condition, and the mapping relationship between the fourth field and the second field exists without needing to meet a certain condition.

[0248] The filtering condition in the above S930 is used to filter the field values of the to-be-queried fields corresponding to at least one target data table that meets the preset condition, that is, the filtering condition is used to filter the field values of the to-be-queried fields corresponding to one or more target data tables in the plurality of target data tables that meet the preset condition, wherein the to-be-queried field corresponding to the at least one target data table is at least one to-be-queried field in the plurality of to-be-queried fields corresponding to the P target data tables.

[0249] For example, taking the above plurality of target data tables including Table 1 and Table 2 as an example, the to-be-queried field corresponding to Table 1 is the id field, and therefore the filtering condition can be to filter the field values of the id field in Table 1 when the field value of the status field in Table 1 is equal to “1”.

[0250] In the embodiments of the present application, the preset condition in the above filtering condition is not specifically limited and can be set according to specific business query requirements.

[0251] In some embodiments, the preset condition comprises that a field value of the to-be-queried field corresponding to the at least one target data table exceeds a preset value; or, the preset condition comprises that the field value of the to-be-queried field corresponding to the at least one target data table does not exceed the preset value; or, in a case where the at least one target data table further comprises a non-to-be-queried field other than the to-be-queried field corresponding to the at least one target data table, the preset condition comprises that a field value of the non-to-be-queried field is equal to a preset value. It should be understood that the non-to-be-queried field in the at least one target data table refers to a field that is not queried in the data synchronization process.

[0252] For example, in a case where the to-be-queried field corresponding to the at least one target data table is an id field and the preset value is 5, the preset condition comprises that the field value of the id field exceeds 5.

[0253] For example, in a case where the to-be-queried field corresponding to the at least one target data table is an id field, the preset value is 1, and the non-to-be-queried field in the at least one target data table is a status field, the preset condition comprises that the field value of the status field is equal to 1.

[0254] The P target data tables have a connection relationship, where the connection relationship between a data table and another data table refers to that the two data tables can be connected through a same field, and the connection manner in which the two data tables are connected through the same field can be, but is not limited to, an inner join, a left join, a right join, or a full join.

[0255] In the embodiments of the present application, the connection relationship of the P target data tables is not specifically limited, and can be determined according to the fields included in each of the P target data tables.

[0256] In some embodiments, there is a connection relationship between any two data tables in the P target data tables, where the any two target data tables have a same field. In this embodiment, the connection relationship between the P target data tables comprises the connection relationship between any two data tables in the P target data tables.

[0257] For example, the P target data tables comprise a data table A, a data table B, and a data table C, where the data table A and the data table B have a same field 1, the data table A and the data table C have a same field 2, and the data table B and the data table C have a same field 3, and the field 1, the field 2, and the field 3 are all different. In this embodiment, the data table A and the data table B have a connection relationship 1, the data table A and the data table C have a connection relationship 2, and the data table B and the data table C have a connection relationship 3, and the connection relationship between the three data tables comprises the connection relationship 1, the connection relationship 2, and the connection relationship 3.

[0258] In some implementations, the P target data tables include data table A, data table B, and data table C, wherein data table A and data table B have a same field 1, data table B and data table C have a same field 2, and the field 1 and the field 2 are different. In such implementations, data table A and data table B have a connection relationship 1, and data table B and data table C have a connection relationship 2. The connection relationship between the three data tables includes the connection relationship 1 and the connection relationship 2.

[0259] In the embodiments of the present application, the manner in which the first electronic device obtains the second configuration information described in the S930 step is not specifically limited.

[0260] In some implementations, the S930 step is performed, i.e., the first electronic device obtains the second configuration information, including that the first electronic device receives the second configuration information sent from the second electronic device, so that the first electronic device obtains the second configuration information.

[0261] In some other implementations, the first electronic device is provided with a configuration page, and the S930 step is performed, i.e., the first electronic device obtains the second configuration information, including that the first electronic device obtains the second configuration information through the configuration page.

[0262] In the above implementations, in the step of the first electronic device obtaining the second configuration information through the configuration page, the first electronic device provides a configuration interface to the second electronic device, wherein the configuration interface is used for the second electronic device to configure the mapping information, the filtering condition, the connection relationship of the P target data tables, and the storage of the P target data tables in the second database; and the first electronic device obtains the second configuration information according to the configuration interface successfully configured by the second electronic device. The second electronic device can be an electronic device different from the first electronic device, and is not specifically limited, for example, the second electronic device can be an electronic device that generates data of records in the P target data tables.

[0263] In the embodiments of the present application, the first configuration information involved in the S910 step and the second configuration information involved in the S930 step can be set based on a configuration page.

[0264] Next, the configuration interface corresponding to the configuration information (i.e., the first configuration information and the second configuration information) described above will be introduced in combination with the accompanying drawings. It should be understood that the configuration interface described below is only illustrative and does not constitute any limitation on the embodiments of the present application.

[0265] For example, referring to Figure 10 a schematic view of a configuration page corresponding to the configuration information is shown.

[0266] In some implementations, the first configuration information in the S910 can be configured through the table information in the configuration page shown in Figure 10 Optionally, in this implementation, the table information in the configuration page shown in Figure 10 Optionally, the table information in the configuration page shown in

[0267] In some other implementations, the second configuration information in the S930 can be configured through the configuration page shown in Figure 10 The configuration page includes the mapping relationship between the fields to be queried in the target data table and the fields in the index of the search engine, the data source of the target data table (i.e., the location of the database storing the target data table), the connection relationship between the plurality of target data tables, and the filtering condition. It should be understood that Figure 10 Only the table configuration information of one of the P target data tables is shown, and the table configuration information of a new target data table can be flexibly added through the add control shown in Figure 10

[0268] Optionally, the configuration page shown in Figure 10 The configuration page shown in

[0269] Optionally, the configuration information shown in Figure 10 The configuration information shown in

[0270] In the technical solution, the first electronic device can automatically generate a corresponding query statement according to the configuration information (e.g., the first configuration information or the second configuration information) configured in the configuration interface shown in Figure 10 The configuration interface shown in

[0271] For example, Figure 11 is a specific configuration page of the configuration page provided in the above Figure 10 It should be understood that Figure 11 The configuration information set in the configuration page shown in Figure 11 ​The above P target data tables are taken as two target data tables (i.e., the course table as a main table and the course_ext table as a sub table) for description.

[0272] As shown in Figure 11 The configuration page includes a group, a mapping relationship between a field in each of the two to-be-queried data tables and a field in an index of a search engine, a connection relationship between the two to-be-queried data tables, a filter condition, and table splitting configuration information.

[0273] Figure 11 The name of the group shown is the course table, i.e., the course table is subjected to table splitting.

[0274] Figure 11 The mapping relationship between the to-be-queried field in each of the two to-be-queried data tables and the field in the index of the search engine includes: a non-conditional mapping relationship between the id field in the index of the search engine (the index name of the index is "10001_course_1.0") and the id field in the course table; a non-conditional mapping relationship between the id field in the index and the id field in the course_ext table; a conditional mapping relationship between the title field in the index and the content field in the course_ext table when the field value of the label_id field in the course_ext table is 49; and a conditional mapping relationship between the sub_title field in the index and the content field in the course_ext table when the field value of the label_id field in the course_ext table is 50.

[0275] The table splitting configuration information indicates that the course table is split into two data tables, wherein the two data tables correspond to two data sources (i.e., a data source corresponding to 0 encoding and a data source corresponding to 1 encoding). For example, the two split tables corresponding to the course table can be represented as Table 1.0 and Table 1.1 in the above.

[0276] Figure 11 The connection relationship between the two to-be-queried data tables shown indicates that the course table and the course_ext table are connected through the id field in the course table and the id field in the course_ext table, i.e., the id field in the course table and the id field in the course_ext table are equivalent.

[0277] Figure 11The filter condition represented by the data filter shown is used to filter out the field value of the queried field (i.e., the id field) in the course table that satisfies the field value of the status field being equal to 1.

[0278] For example, in the case that the course table in the configuration interface shown is not a sub-table (i.e., the sub-table configuration option in the configuration page shown is not selected, and the sub-table configuration content is empty), the course table can be the data table shown in Table 1 above, and the above-mentioned Figure 11 Figure 11 For example, in the case that the course table in the configuration interface shown is not a sub-table (i.e., the sub-table configuration option in the configuration page shown is not selected, and the sub-table configuration content is empty), the course table can be the data table shown in Table 1 above, and the above-mentioned Figure 11 The course_ext table in the configuration page shown can be represented as the data table shown in Table 2 above.

[0279] It should be understood that the above-mentioned Figure 11 For example, in the case that the queried data table (i.e., the course table) in the configuration page shown is split into multiple data tables, the above-mentioned Figure 11 For example, in the case that the queried data table (i.e., the course table) in the configuration page shown is split into multiple data tables, the above-mentioned

[0280] In the above-mentioned Figure 10 For example, in the case that the queried data table (i.e., the course table) in the configuration page shown is split into multiple data tables, the above-mentioned Figure 11 The data source of the data table in the configuration page shown involves the data table, and therefore, the present application embodiment further provides a step of configuring the data source of the data table.

[0281] In the present application embodiment, the data source of the data table can include a path for representing a database storing the data table, a username and a password for accessing the database storing the data table. Optionally, the data source of the data table can further include other information, for example, the other information can but not exist be an identification of the database storing the data table, a Chinese name of the database storing the data table, or a type of the database storing the data table, etc.

[0282] For example, in the case that the queried data table (i.e., the course table) in the configuration page shown is split into multiple data tables, the above-mentioned Figure 12 is a schematic diagram of a configuration interface corresponding to a data source provided by the present application embodiment. As shown in Figure 12 The configuration page of the data source includes a data source code, a database Chinese name, a database type, a database url, a username, a password, and a connection pool parameter.

[0283] The data source code is used to identify a database, and different databases correspond to different data source codes. For example, the data source code can but not exist be a number or a letter.

[0284] The database Chinese name is the Chinese name of the database identified by the data source code.

[0285] The database type is the type of the database identified by the data source code. For example,​ Figure 12 The data source type shown is a database of the MySQL type.

[0286] The database url refers to the address in the network where the database identified by the data source code is located.

[0287] The username is an account for accessing the database identified by the data source code, and the password is a password for accessing the database.

[0288] The connection pool parameter refers to other configuration information required for connecting the database identified by the data source code, which can be set according to actual needs, and is not specifically limited. The connection pool parameter is a non-mandatory option.

[0289] It should be understood that the above Figure 12 The content shown only includes information of one data source. Optionally, according to actual needs, a new data source information can be added by the adding control shown. Figure 12

[0290] In the embodiments of the present application, the search engine and the second database (i.e., the relational database) in the above S930 step are not specifically limited, and can be selected according to actual needs.

[0291] In some implementations, the search engine is a distributed full-text search engine. For example, the distributed full-text search engine can be, but is not limited to, ES.

[0292] In some implementations, the second database is a relational database. For example, the second database can be, but is not limited to, any one of the following databases: MySQL, Oracle, DB2, Sybase, or PostgreSQL database.

[0293] In the embodiments of the present application, whether the electronic device where the search engine is located and the first electronic device are the same electronic device is not specifically limited, that is, the electronic device where the search engine is located can be the first electronic device, or can be another electronic device other than the first electronic device.

[0294] S940, the first electronic device generates a second query statement according to the second configuration information, wherein the second query statement is used to query field values of a plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields and satisfying a preset condition, and the associated data table is a data table obtained by associating the P target data tables according to a connection relationship.

[0295] ​In the step S940, the first electronic device generates a second query statement according to the second configuration information, the second query statement being used to query field values of a plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields and satisfying a preset condition, the associated data table being a data table obtained by associating the P target data tables according to the connection relationship. It can be understood that the query statement (i.e., the second query statement) has the effect of associating the P target data tables according to the connection relationship, the effect of mapping the plurality of fields in the index and the plurality of to-be-queried fields corresponding to the P target data tables, and the effect of querying the field values of the plurality of to-be-queried fields in the associated data table corresponding to the plurality of fields and satisfying the preset condition. That is, in the embodiment of the present application, the first electronic device generates a query statement (i.e., the second query statement) according to the second configuration information, which realizes the functions of data extraction, data mapping and data filtering, and avoids the complex data conversion process based on the ETL architecture in the prior art, that is, the data synchronization method provided by the present application can reduce the development and maintenance costs of the system and improve the maintainability of the system.

[0296] For example, the P target data tables include the table 1 and the table 2 in the above, the associated data table obtained by associating the table 1 and the table 2 in the step S940 can be represented as the table 3 in the following.

[0297] Table 3

[0298]

[0299] In the embodiment of the present application, the method for the first electronic device to generate the second query statement according to the second configuration information in the step S940 is not limited.

[0300] In some implementations, the step S940 is performed, that is, the first electronic device generates the second query statement according to the second configuration information, including: the first electronic device directly generates the corresponding second query statement according to the second configuration information.

[0301] In some implementations, the step S940 of generating the second query statement according to the second configuration information is performed by the first electronic device, including: selecting, by the first electronic device, a plurality of target generators from a plurality of candidate generators according to the second configuration information, wherein the plurality of candidate generators correspond to the plurality of types of statements, each candidate generator is configured to generate a statement of a corresponding type, the plurality of target generators include a first generator, a second generator, and a third generator, the first generator is configured to generate a statement of field values of a plurality of fields to be queried that have a mapping relationship with the plurality of fields, the second generator is configured to generate a statement representing a connection relationship, and the third generator is configured to generate a statement representing a filter condition; processing, by the first electronic device, the second configuration information using the plurality of target generators to obtain a plurality of statements, and splicing the plurality of statements to obtain the second query statement.

[0302] In the step of processing, by the first electronic device, the second configuration information using the plurality of target generators to obtain a plurality of statements, and splicing the plurality of statements to obtain the second query statement, the first electronic device processes, using the first generator, the second configuration information including that P target data tables are stored in the second database and the join information to obtain a first statement, processes, using the second generator, the connection relationship in the second configuration information to obtain a second statement, processes, using the third generator, the filter condition in the second configuration information to obtain a third statement, and splices the first statement, the second statement, and the third statement in sequence to obtain the second query statement.

[0303] The plurality of candidate generators in the above implementations are not limited, and can be set or extended according to specific businesses. For example, in some implementations, the plurality of candidate generators in the above implementations can include an abstract generator and a specific generator.

[0304] The abstract generator includes a select fragment (i.e., statement) generator, a join on fragment generator, and a where fragment generator, wherein the select fragment generator is configured to generate a select fragment, the join on fragment generator is configured to generate a join on fragment, and the where fragment generator is configured to generate a where fragment.

[0305] The specific generator includes a simple generator and an alias generator.

[0306] The simple generator is a subclass of the select fragment generator, and is used to perform generation of a select fragment without an alias (as), i.e., the select fragment generated by the simple generator does not include an alias as. For example, the format of the select fragment generated by the simple generator can be: select column name from table name, select case when, or select column name 1, column name 2… from table name.

[0307] The alias generator is a subclass of the select fragment generator, and is used to perform generation of a select fragment with an alias (as), i.e., the select fragment generated by the alias generator includes an alias as. For example, the format of the select fragment generated by the alias generator can be: select column name (expression) as alias from table name.

[0308] It can be understood that the order of each generator included in the plurality of target generators corresponds to the order of splicing the plurality of statements generated by the plurality of target generators. That is, when splicing the statements generated by the plurality of target generators, the statement generated by the second generator needs to be spliced after the statement generated by the first generator, the statement generated by the third generator needs to be spliced after the statement generated by the second generator, and the statement generated by the fourth generator needs to be spliced after the statement generated by the third generator.

[0309] In the embodiments of the present application, the step of generating the first query statement according to the first configuration information by the first electronic device in the S910 step, and the step of generating the second query statement according to the second configuration information by the first electronic device in the S940 step. Next, in combination with the drawings, a flow of generating a query statement according to configuration information provided by the embodiments of the present application is introduced.

[0310] An exemplary, Figure 13 The first electronic device generates a query statement according to configuration information in the above-mentioned implementation manner is shown in the schematic diagram. It can be understood that, Figure 13 The SQL generation module shown can be a module in the first electronic device, for example, it can be a module in the data synchronization service of the first electronic device, and the data synchronization service is an application in the application layer of the first electronic device.

[0311] As Figure 13As shown, the SQL generation module can include a plurality of candidate generators, a SQL generator chain builder, a SQL generator chain (i.e., the plurality of target generators in the above), and a SQL verifier. Optionally, Figure 13 The SQL verifier can also be excluded. It should be understood that, Figure 13 The SQL generation module shown includes various modules only for illustration and does not constitute any limitation.

[0312] In the following, based on the above Figure 13 As shown in the SQL generation module, the SQL generation module in this application generates a query statement according to the configuration information as follows: first, the configuration information obtained is subjected to data parsing and verification and other processing to obtain a SQL generation context corresponding to the configuration information, wherein the SQL generation context contains all information for generating SQL (i.e., a query statement), as well as intermediate results and final results of the generation. Then, the SQL generator chain builder extracts n target SQL generators from the plurality of candidate generators according to the SQL generation context, and combines the n target SQL generators to obtain an ordered arrangement of n segment generators (i.e., segment generator 1, …, segment generator n) included in the SQL generator chain, n being a positive integer. Next, each segment generator in the SQL generator chain generates a statement corresponding to the segment generator according to the SQL context, wherein each segment generator can generate a corresponding statement using string concatenation. Thereafter, the n statements generated by the n segment generators are simply concatenated to obtain the final SQL (i.e., a query statement), wherein the order of the n segment generators included in the SQL generator chain corresponds to the order of the statements generated by the n segment generators. Finally, the SQL verifier performs syntax verification on the SQL obtained by the SQL generator chain. In the case where the SQL verifier determines that the SQL is grammatically correct, the SQL that passes the verification is stored in the database for subsequent use. It can be understood that, in the case where the SQL generation module is a module included in the data synchronization service, Figure 13 The database shown can be a meta-database associated with the data synchronization service.

[0313] It can be understood that any one of the generators described above can be regarded as a function in the code, and the SQL generation context can be regarded as a parameter of the function, i.e., the SQL generation context can be passed through a function method call.

[0314] It should be noted that the types of the plurality of candidate generators shown in Figure 13 in the above are only for illustration and do not constitute any limitation. Optionally, the plurality of candidate generators shown in Figure 13 can also be extended according to business needs. For example, Figure 13 The multiple candidate generators can also include a table fragment generator for creating new data tables, such as a table fragment used to generate the following statement: create table tab_new like tab_old.

[0315] Based on the above Figure 13 The illustrated SQL generation architecture allows for more flexible selection of the target SQL generator or the addition of new SQL generators when business requirements change, thus facilitating generator reuse. For example, based on the above... Figure 13 The SQL architecture shown, in the case of generating K first query statements based on the first configuration information in step S910 above, may also include only a select fragment generator, wherein the select fragment generator is used for each of the K first query statements.

[0316] Below, examples are given to describe the first configuration information and K first query statements in step S910 above.

[0317] For example, taking the first electronic device generating K first query statements based on the first configuration information in step S910 above as an example, the process includes: when the first configuration page information includes the table name of the data table as Table 1 and the data source of Table 1, and the data table is split into two data tables (Table 1.0 and Table 1.1), the SQL generation module included in the data synchronization service can parse the configuration page information to obtain the following information:

[0318] SQL template: select id from[table];

[0319] List of table names: Table 1.0, Table 1.1.

[0320] The SQL template above represents a query for all records in the `id` column of tables (i.e., Tables 1.0 and 1.1). Specifically, the SQL template and the list of table names correspond to two SQL statements (SQL#1 and SQL#2). SQL#1 queries the value of the `id` field in Table 1.1, and SQL#2 queries the value of the `id` field in Table 1.0. In this implementation, the fragment generator in the SQL generation module can include only a `select` generator.

[0321] The following example illustrates the second configuration information and the second query statement in step S940 above.

[0322] For example, as described above Figure 11 Taking the second configuration information set in the configuration page shown as an example, the first electronic device is based on... Figure 11The second configuration information set in the configuration page is shown, and a second query statement is generated. The first electronic device determines, according to the second configuration information, that the plurality of target generators include a select case when segment generator, a join on segment generator, and a where segment generator. The first electronic device processes the second configuration information by using the select segment generator to generate a select case when segment. The first electronic device processes the second configuration information by using the join on segment generator to generate a join on segment. The first electronic device processes the second configuration information by using the where segment generator to generate a where segment. The first electronic device concatenates the select case when segment, the join on segment, and the where segment in sequence to obtain a final query statement, where the query statement can be as follows:

[0323]

[0324] In the above query statement, the alias of course is c, and the alias of course_ext is ce. In the above statement, select course.id,…case when…from course c is a select case when segment, course c left join course_ext ce on c.id=ce.id is a join on segment, and where c.status=1 is a where segment.

[0325] S950, the first electronic device performs a joint query on the P target data tables in the second database according to the second query statement to obtain target data, where the target data includes a plurality of fields and field values of a plurality of to-be-queried fields in the associated data table that satisfy a preset condition.

[0326] The above S950 is executed, that is, the first electronic device implements a joint query on a plurality of data tables (i.e., the P target data tables) in the second database by using only one query statement (i.e., the second query statement) to obtain target data including a plurality of fields and field values of a plurality of to-be-queried fields in the associated data table that satisfy a preset condition.

[0327] For example, the P target data tables in step S950 above can be Table 1 and Table 2 mentioned above. Table 1 has an id field as the field to be queried, and Table 2 has label_id and content fields to be queried. Multiple fields from the search engine can include id, title, and sub_title fields. If the preset condition is to filter out the id field value when the status field in Table 1 has a value of 1, the resulting target data can be represented as: {"id":1,"title":trigonometric function,"sub_title":sine function}. Here, the id, title, and sub_title fields in this target data are multiple fields from the index, and "1", "trigonometric function", and "sine function" are the field values ​​of multiple fields to be queried in the related data tables of Tables 1 and 2 that satisfy the preset condition.

[0328] S960, the first electronic device synchronizes the target data to the index.

[0329] The target data in step S960 above includes multiple fields in the search engine and multiple query fields in the associated data table that meet preset conditions. Therefore, synchronizing the target data to the index means synchronizing the field values ​​of multiple query fields in the associated data table that meet preset conditions to multiple fields in the search engine's index.

[0330] For example, taking the target data in step S960 above as {"id":1,"title":trigonometric function,"sub_title":sine function}, the id field, title field, and sub_title field are multiple fields in the index, and "1", "trigonometric function", and "sine function" are the field values ​​of multiple fields to be queried in the related data tables corresponding to Table 1 and Table 2 that meet the preset conditions. Therefore, executing step S960 above includes: the first electronic device synchronizing the field value of the id field in the target data to the field value of the id field in the index (i.e., "1"), synchronizing the field value of the title field in the target data (i.e., "trigonometric function") to the field value of the id field in the index, and synchronizing the field value of the sub_title field in the target data (i.e., "sine function") to the field value of the id field in the index.

[0331] In this embodiment of the application, the method for synchronizing target data to the index as described in step S960 above is not specifically limited to the first electronic device.

[0332] In some implementations, the step S960 of synchronizing the target data to the index by the first electronic device comprises: sending the target data to a message queue, so that the target data in the message queue is synchronized to the index.

[0333] For example, when the message queue and the search engine are located in the first electronic device, the first electronic device sends the target data to the message queue, so that the target data in the message queue is synchronized to the index, which comprises: a data synchronization service of the first electronic device sends the target data to the message queue; and the search engine synchronizes the target data in the message queue to the index. It can be understood that the data synchronization service of the first electronic device can execute the data synchronization method provided in the embodiments of the present application, and the data synchronization service is an application in the application layer of the first electronic device.

[0334] For example, when the message queue and the search engine are located in the second electronic device different from the first electronic device, the first electronic device sends the target data to the message queue, so that the target data in the message queue is synchronized to the index, which comprises: the first electronic device sends the target data to the message queue of the second electronic device, so that the search engine synchronizes the target data in the message queue to the index.

[0335] The message queue in the above implementations can be, but is not limited to, a message queue kafka.

[0336] In the above implementations, the first electronic device sends the obtained target data to the message queue first, and then the search engine downstream can read the target data from the message queue at its own pace, so that the first electronic device synchronizes the target data in the message queue to the index, which can avoid the phenomenon of reading blockage of the search engine due to too large data volume of the target data, and can improve the reading efficiency of the search engine, thereby improving the data synchronization efficiency between the search engine and the database.

[0337] In some other implementations, the search engine is located in the second electronic device different from the first electronic device, and the step S960 of synchronizing the target data to the index by the first electronic device comprises: the first electronic device sends the target data to the search engine in the second electronic device, so that the search engine synchronizes the obtained target data to the index. In this implementation, when the search engine in the second electronic device receives the target data from the first electronic device, the search engine in the second electronic device can actively synchronize the obtained target data to the index corresponding to the search engine.

[0338] It should be understood that the above Figure 9 The data synchronization method shown is only illustrative and does not constitute any limitation on the data synchronization method provided by the present application. For example, the electronic device in which the at least one first database is located and the electronic device in which the second database is located in the above S920 step can also be the same electronic device. In this implementation, the at least one first database is the second database, the P target data tables are the K source data tables (i.e., P equals K), or the P target data tables include one data table obtained by merging part of the K source data tables and the remaining source data tables of the K source data tables except for the part of the K source data tables (i.e., P is less than K). Therefore, in this implementation, the first electronic device can not perform the above S910 step and S920 step, i.e., the first electronic device can directly perform the above S930 step to S960 step to implement the data synchronization method provided by the present application.

[0339] In the embodiment of the present application, on the one hand, in the process of synchronizing K source data tables located in at least one first database to source data in a second database according to K first query statements, since the data in the K source data tables is generated by an application of an electronic device where the at least one first database is located, and the electronic device where the at least one first database is located is different from the electronic device where the second database is located, the influence on the business of the business party (i.e., the electronic device where the at least one first database is located) can be avoided. On the other hand, since the second configuration information includes the filtering condition, the connection relationship of the P target data tables, and the information that the P target data tables are stored in the second database, the purpose of extracting data (i.e., the P target data tables) from the data source (i.e., the second database) and filtering the data extracted from the data source according to the query requirement (i.e., querying the field values of the multiple to-be-queried fields in the associated data table that meet the preset condition) can be simultaneously achieved by using one query statement (i.e., the second query statement) generated according to the second configuration information, and therefore, the efficiency of data query can be improved, and the efficiency of data synchronization between the search engine and the database can be improved. In addition, since the second configuration information further includes the mapping relationship (i.e., mapping information) between the multiple fields in the index and the multiple to-be-queried fields corresponding to the P target data tables, the second query statement generated according to the second configuration information is used to query the field values of the multiple to-be-queried fields in the associated data table (i.e., the data table obtained by associating the P target data tables according to the connection relationship) that meet the preset condition, and therefore, the target data obtained by jointly querying the P target data tables by using the one query statement includes the multiple fields in the index and the field values of the multiple to-be-queried fields in the associated data table that meet the preset condition, and after that, the obtained target data can be directly synchronized to the index without the need for field mapping (e.g., mapping the fields in the database to the fields in the index) and other operations, and therefore, the efficiency of data synchronization between the search engine and the database can be improved. In addition, since the query statements (e.g., the K first query statements and the second query statement) are automatically generated by the first electronic device according to the configuration information, the process of generating the query statements avoids manual programming by engineers, and therefore, the accuracy of the query statements can be ensured, and the accuracy of the data synchronization result can be improved. In summary, based on the technical solution of the present application, the accuracy and the efficiency of the data synchronization result between the search engine and the database can be improved, and the influence on the business of the business party can be avoided.

[0340] In the following, another data synchronization method provided by the embodiment of the present application is described by taking the application scenario (i.e., single database, non-table splitting scenario) shown in the above application scenario (i.e., single database, non-table splitting scenario) as an example, which does not include the application scenario of the mirror database. Figure 4

[0341] Figure 14 ​is a schematic diagram of a data synchronization method provided by an embodiment of the present application. It can be understood that Figure 14 The data synchronization method provided by the embodiment is the data synchronization method provided by the above Figure 9 The data synchronization method provided by the embodiment is a specific example of the data synchronization method provided by the above Figure 14 The data synchronization method provided by the embodiment is the data synchronization method provided by the above Figure 9 The second database for storing P target data tables in the data synchronization method provided by the above Figure 14 The data synchronization method provided by the embodiment is the data synchronization method provided by the above Figure 9 The steps S910 and S920 shown in the embodiment are not involved in the above

[0342] For example, as shown in the embodiment, the data synchronization method includes steps S1410 to S1460. In the following, S1410 to S1460 are described in detail. Figure 14

[0343] S1410, the data synchronization service in the server B provides a configuration page to the server A, and the server A configures in the configuration page to make the data synchronization service obtain configuration information 1 (i.e. an example of the second configuration information in the data synchronization method provided by the above Figure 9

[0344] As shown in the embodiment, the local database A of the server A (i.e. an example of the second database in the data synchronization method provided by the above Figure 14 Figure 9 As shown in the embodiment, the local database A of the server A (i.e. an example of the second database in the data synchronization method provided by the above Figure 9 The plurality of data tables include table #1 and table #2, wherein table #1 can be table 1 shown in the above, and table #2 can be table 2 shown in the above, i.e. table #1 and table #2 are connected through the course identifier id field.

[0345] The configuration information 1 can be the content shown in the configuration page of the above Figure 11 Since table #1 and table #2 in the embodiment of the present application are not split into multiple data tables, the split table configuration information content in the configuration page shown in the above Figure 11 Figure 11 ​​​​The table partitioning configuration shown should be unselected, and the corresponding content should be empty. The `course` table has only one data source. Therefore, the configuration information 1 in step S1410 above includes mapping relationships between fields in the ES index and fields in the data table to be queried, including mapping relationship 1, mapping relationship 2, and mapping relationship 3. Mapping relationship 1 is the mapping between the `id` field in the ES index and the `id` field in table #1; mapping relationship 2 is the mapping between the `title` field in the ES index and the `content` field in table #2 when `label_id` equals 49; mapping relationship 3 is the mapping between the `sub_title` field in the ES index and the `content` field in table #2 when `label_id` equals 50.

[0346] It is understood that the id field, title field, and sub_title field in the embodiments of this application are as described above. Figure 9 An example of multiple fields in the index of the provided data synchronization method, and the aforementioned id field, label_id field, and content field are as described above. Figure 9 An example of multiple fields to be queried corresponding to P target data tables in the provided data synchronization method.

[0347] It is understandable that the above configuration information 1 can be the above Figure 11 In the case of the content shown on the configuration page, Table #1 in this embodiment of the application is Figure 11 The course table shown, Table #2 in this embodiment of the application is Figure 11 The course_ext table is shown.

[0348] S1420, the data synchronization service in server B parses configuration information 1 and generates query statement 1 corresponding to configuration information 1 (i.e., the above). Figure 9 (An example of the second query statement in the provided data synchronization method).

[0349] The data synchronization service may include an SQL generation module that executes step S1420 above. Specifically, the SQL generation module within the data synchronization service parses configuration information 1 in the configuration page and generates query statement 1 corresponding to configuration information 1. It can be understood that the working principle of the SQL generation module in the data synchronization service generating query statement 1 based on configuration information 1 is similar to that described above. Figure 13 The SQL generation module works in the same way as the configuration page information; for details not elaborated here, please refer to the relevant descriptions above.

[0350] Based on the configuration page corresponding to configuration information 1 described in step S1410 above, the query statement 1 obtained by the data synchronization service after executing step S1420 above can be as follows:

[0351]

[0352]

[0353] As can be seen, the above query statement 1 combines data extraction and data filtering into one, that is, it realizes data extraction and data filtering operations through a single query statement. This can improve data query efficiency, thereby improving data synchronization efficiency.

[0354] Optionally, server B's local database may also include a metadata database, and query statement 1 obtained by the data synchronization service in step S1620 above may also be stored in the metadata database. Subsequently, when the data synchronization service needs to perform a query operation using query statement 1, the data synchronization service can retrieve query statement 1 from the metadata database.

[0355] S1430, the data synchronization service in server B performs a join query on tables #1 and #2 in database A based on query statement 1 to obtain the data to be synchronized (i.e., the data mentioned above). Figure 9 An example of the target data in the provided data synchronization method.

[0356] Based on the above query statement 1, a join query is performed on tables #1 and #2. The resulting data to be synchronized can be represented as follows:

[0357] {"id":1,"title":Trigonometric functions,"sub_title":Sine function}.

[0358] The data to be synchronized above indicates that when the value of the id field in the index of the search engine is 1, the value of the corresponding title field is a trigonometric function, and the value of the corresponding sub_title field is a sine function.

[0359] S1440, the data synchronization service in server B sends the data to be synchronized to the message queue in server C (i.e., the above). Figure 9 (An example of a message queue in the provided data synchronization method).

[0360] The message queues mentioned above can be associated with indexes in Elasticsearch (ES), meaning that the message queues are used to store data from associated ES indexes.

[0361] The data synchronization service executes the above S1440 step, that is, the data synchronization service can package the data to be synchronized obtained in S1430 and send it to the message queue.

[0362] The message queue in server C can reside in server C's memory or disk, without any specific restrictions.

[0363] S1450, the message queue in server C is caching data to be synchronized.

[0364] After executing S1450 above, the message queue stores data to be synchronized in the following format:

[0365] {"id":1,"title":Trigonometric functions,"sub_title":Sine function}.

[0366] S1460, Server C in ES (i.e., the above) Figure 9 (An example of a search engine in the provided data synchronization method) retrieves the data to be synchronized from the message queue and stores the data to be synchronized in the Elasticsearch index (i.e., the one mentioned above). Figure 9 (An example of an index in the provided data synchronization method).

[0367] The configuration information 1 in S1410 above includes mapping relationship 1 between fields in the ES index and fields in table #1 of database A, and mapping relationship 2 between fields in the ES index and table #2. Specifically, mapping relationship 1 includes the mapping relationship between the id field in the ES index and the id field in table #1; mapping relationship 2 includes the mapping relationship between the title field in the ES index and the content field in table #2 when label_id equals 49, and the mapping relationship between the sub_title field in the ES index and the content field in table #2 when label_id equals 50.

[0368] Therefore, executing S1460 above, where Elasticsearch stores the data to be synchronized into its index, includes: synchronizing the value of the id field in the data to be synchronized to the value of the id field in the Elasticsearch index; synchronizing the value of the title field in the data to be synchronized to the value of the title field in the Elasticsearch index; and synchronizing the value of the sub_title field in the data to be synchronized to the value of the sub_title field in the Elasticsearch index. In other words, after executing S1460, the value of the id field in the Elasticsearch index is 1, the value of the title field in the index is a trigonometric function, and the value of the sub_title field in the index is a sine function.

[0369] Optionally, after ES executes the above S1460 step, the message queue can also delete the data to be synchronized, which can improve the storage utilization of the message queue.

[0370] It should be understood that the above Figure 14The data synchronization method shown is only illustrative, and does not constitute any limitation on the data synchronization method provided in the present application. For example, the ES described above can also be located in server B. For example, the message queue described above can also be a queue in server D other than server A, server B and server C.

[0371] In the embodiment of the present application, first, the server as a service party (i.e., server A) can achieve the purpose of obtaining configuration information 1 according to the configuration page provided by the data synchronization service of server B. Next, the data synchronization service generates a data query statement (i.e., query statement 1) according to the configuration information 1, wherein the data query statement can simultaneously achieve the purposes of data extraction (i.e., reading data from the mirror database) and data filtering and conversion (i.e., filtering the data read from the mirror database according to the filtering condition in the configuration information 1), i.e., combining data extraction and data filtering and conversion into one, so that the data query efficiency can be improved. Thereafter, the to-be-synchronized data obtained by querying is synchronized to the fields of the index of the corresponding ES, so that the data synchronization efficiency can be improved. In addition, in the above data synchronization process, the data synchronization service and the ES directly interact to achieve sending the to-be-synchronized data queried by the data synchronization service to the ES, and in the case that the data amount of the to-be-synchronized data is small, this implementation manner can improve the data transmission efficiency, thereby further improving the data synchronization efficiency.

[0372] Next, the above Figure 5 The application scenario shown (i.e., the scenario of multiple databases and no table splitting) is taken as an example to describe another data synchronization method provided in the embodiment of the present application.

[0373] Figure 15 is a schematic diagram of a data synchronization method provided in the embodiment of the present application. It can be understood that, Figure 15 The data synchronization method provided in the embodiment of the present application is a specific example of the above Figure 9 The data synchronization method provided in the embodiment of the present application is a specific example of the above Figure 15 The data synchronization method shown is the data synchronization method provided in the above Figure 9 The P target data tables located in the second database in the data synchronization method provided in the above embodiment are obtained by data backup on the K source data tables located in at least one first database, wherein P is equal to K, i.e., the P target data tables are the K source data tables, the at least one first database can be a database located in a service party electronic device, and the data in the K source data tables can be data generated by an application of the service party electronic device.

[0374] For example, as Figure 15 shown, the method includes steps S1510 to S1591. Next, S1510 to S1591 are specifically described.

[0375] S1510, the data synchronization service in server B provides a configuration page to server Ai. Server Ai configures itself on the configuration page so that the data synchronization service receives the configuration information #i (i.e., the above). Figure 9 (An example of the first configuration information in the provided data synchronization method), i = 1, 2.

[0376] The data table #1 to be queried in database A of server A1 can be table 1 mentioned above, and the data table #2 to be queried in database B of server A2 can be table 2 mentioned above. That is, the data table #1 to be queried and the data table #2 to be queried are connected through the course identifier field.

[0377] Configuration information #i includes configuration information #1 and configuration information #2. Configuration information #1 includes the table name and data source of table #1 to be queried, and configuration information #2 includes the table name and data source of table #2 to be queried. The data source of table #1 includes the URL for accessing database A on server A1, and the username and password for accessing database A on server A1. The data source of table #2 includes the URL for accessing database B on server A2, and the username and password for accessing database B on server A2.

[0378] The above configuration information #i can be obtained by referring to the above text. Figure 10 The table information shown on the configuration page is configured to obtain the desired information. Figure 10 The data source in the configuration page shown can be accessed via the above. Figure 11 The interface shown is configured as follows; it is optional. Figure 10 The configuration page shown may also exclude the main table option.

[0379] S1520, the data synchronization service in server B parses the configuration information #i and generates the query statement #i corresponding to the configuration information #i (i.e., the above). Figure 9 (An example of the K first query statements in the provided data synchronization method).

[0380] Executing step S1520 above, namely, the data synchronization service in server B parses the configuration information #i and generates the query statement #i corresponding to the configuration information #i, including: the SQL generation module included in the data synchronization service parses the configuration information #1 and generates the query statement #1 corresponding to the configuration information #1, wherein the query statement #1 is used to query table #1 in database A located on server A1; the SQL generation module included in the data synchronization service parses the configuration information #2 and generates the query statement #2 corresponding to the configuration information #2, wherein the query statement #2 is used to query table #2 in database B located on server A2.

[0381] It can be understood that the SQL generation module included in the data synchronization service generates the query statement #i according to the configuration information #i, and the working principle is the same as that of the SQL generation module in the above Figure 13 configuration page information, and the details are not described herein and can be referred to the related description in the above.

[0382] S1530, the data synchronization service in the server B queries the database in the server Ai according to the query statement #i to obtain the table #i (i.e. an example of the K source data tables in the data synchronization method provided above). Figure 9

[0383] The step S1530, i.e. the data synchronization service in the server B queries the database in the server Ai according to the query statement #i to obtain the table #i, includes: the data synchronization service queries the database A in the server A1 according to the query statement #1 to obtain the table 1; and the data synchronization service queries the database B in the server A2 according to the query statement #2 to obtain the table 2.

[0384] S1540, the data synchronization service in the server B stores the queried table #i into the mirror database in the server B (i.e. an example of the second database in the data synchronization method provided above). Figure 9

[0385] After the steps S1510 to S1540, the mirror database in the server B stores the business table (i.e. the table #i) in the business server (i.e. the server Ai).

[0386] S1550, the data synchronization service provides a configuration page to the server A2, and the server A2 configures in the configuration page so that the data synchronization service obtains the configuration information 1 (i.e. an example of the second configuration information in the data synchronization method provided above). Figure 9

[0387] The difference between the configuration information 1 in the step S1550 and the configuration information 1 in the step S1410 is that the data source of the table #i in the step S1550 is the mirror database in the server B, and the user and the password in the data source are the user and the password for accessing the mirror database; the data source of the table #i in the step S1410 is the database A in the server A, and the user and the password in the data source are the user and the password for accessing the database A.

[0388] ​​​In the embodiment of the present application, the server A2 can also store the mapping relationship between the fields in the table #1 in the database A of the server A1 to be queried and the fields in the index in the ES. Based on this, after the server 2 configures in the configuration page, the data synchronization service can obtain the configuration information 1.

[0389] The above S1550 step is described by taking the server A2 configuring in the configuration page as an example. Alternatively, the above S1550 step can also be replaced by the following steps: the data synchronization service provides a configuration page to the server A1, and the server A1 configures in the configuration page, so that the data synchronization service obtains the configuration information 1. In this implementation manner, the server A1 can also store the mapping relationship between the fields in the table #2 in the database B of the server A2 to be queried and the fields in the index in the ES.

[0390] S1560, the data synchronization service in the server B parses the configuration information 1 to generate a query statement 1 (i.e., an example of the second query statement in the data synchronization method provided by the present application). Figure 9

[0391] The difference between the query statement 1 in the above S1550 and the query statement 1 in the above S1420 is that the query statement 1 in the above S1550 is used to query the table #1 and the table #2 in the mirror database in the server B; and the query statement 1 in the above S1410 is used to query the table #1 and the table #2 in the database A of the server A.

[0392] S1570, the data synchronization service in the server B performs a joint query on the table #1 and the table #2 in the mirror database according to the query statement 1 to obtain to-be-synchronized data (i.e., an example of the target data in the data synchronization method provided by the present application). Figure 9

[0393] The to-be-synchronized data in the above S1570 step is the same as the to-be-synchronized data in the above S1430 step, and the working principle of the above S1570 step and the above S1430 step is the same. The content not described in detail here can be referred to the related description in the S1430 step above.

[0394] S1580, the data synchronization service in the server B sends the to-be-synchronized data to a message queue in the server C (i.e., an example of the message queue in the data synchronization method provided by the present application). Figure 9

[0395] The working principle of the above S1580 and the above S1440 is the same. The content not described in detail here can be referred to the related description in the S1440 above.

[0396] ​​​S1590, the message queue in server C is caching data to be synchronized.

[0397] The working principle of steps S1590 and S1450 is the same. For details not elaborated here, please refer to the relevant description in step S1450 above.

[0398] S1591, Server C's Elasticsearch (ES) retrieves the data to be synchronized from the message queue and stores the retrieved data to be synchronized in ES (i.e., the aforementioned...). Figure 9 The index (i.e., the one mentioned above) of the search engine in the provided data synchronization method) Figure 9 (An example of an index in the provided data synchronization method).

[0399] The working principle of step S1591 above is the same as that of step S1460 above. For details not described in detail here, please refer to the relevant description in step S1460 above.

[0400] It should be understood that the above Figure 15 The data synchronization method shown is for illustrative purposes only and does not constitute any limitation on the data synchronization method provided in this application. For example, the above steps may not include a message queue. Based on this, after executing steps S1510 to S1570, the data synchronization service directly sends the data to be synchronized to the ES, and then the ES synchronizes the data to be synchronized to its index. For example, the ES may also be located on server B. For example, the message queue may also be a queue on server D, other than servers A, B, and C.

[0401] In the embodiment of the present application, on the one hand, the query statement #i generated based on the configuration information #i obtained from the configuration page provided by the data synchronization service of the server B can achieve the purpose of source data synchronization, that is, synchronizing the plurality of business data tables in the plurality of databases of the plurality of business servers (that is, table #1 and table #2 in the database A of the server A, and table #1 and table #2 are not split tables obtained by splitting a certain business data table, that is, the fields included in table #1 and the fields included in table #2 are not completely the same) to the mirror database of the server B, so that the frequent access of the server B to the business server (that is, the server A) can be avoided, and the influence on the business of the business server can be avoided, thereby facilitating to improve the processing performance of the business server. On the other hand, based on the configuration page provided by the data synchronization service of the server B, the data synchronization service can obtain the configuration information 1. Next, the data synchronization service generates a data query statement (that is, query statement 1) according to the configuration information 1, wherein the data query statement can simultaneously achieve the purposes of data extraction (that is, reading data from the mirror database) and data filtering and conversion (that is, filtering the data read from the mirror database according to the filtering condition in the configuration information 1), that is, combining data extraction and data filtering and conversion into one, so that the data query efficiency can be improved. Thereafter, the to-be-synchronized data obtained by querying is synchronized to the fields of the index of the corresponding ES, so that the data synchronization efficiency can be improved. In addition, in the above data synchronization process, the message queue is introduced to realize the caching of the to-be-synchronized data, and in the scene where the data amount of the to-be-synchronized data is large or the processing capability of the ES is poor, the processing pressure of the ES reading data can be relieved.

[0402] Next, taking the application scenario (that is, a single database with split tables) shown in the above Figure 6 application scenario (that is, a single database with split tables) shown in the above

[0403] Figure 16 is a schematic diagram of a data synchronization method provided by an embodiment of the present application. It can be understood that Figure 16 the data synchronization method provided by the above Figure 9 is a specific example of the data synchronization method provided by the above Figure 16 the data synchronization method shown in the above Figure 9 is a specific example of the data synchronization method provided by the above

[0404] As shown in Figure 16 , the method comprises steps S1610 to S1690. Details of S1610 to S1690 are described below.

[0405] S1610, the data synchronization service in server B provides a configuration page to server A, and server A configures in the configuration page to make the data synchronization service obtain configuration information #i (i.e. the first configuration information in the data synchronization method provided above), i = 1.0, 1.1, 2. Figure 9 An example of the first configuration information in the data synchronization method provided above), i = 1.0, 1.1, 2.

[0406] As shown in Figure 16 , in the embodiment of the present application, table #1.0 and table #1.1 in database A of server A are two sub-data tables obtained by splitting table #1, wherein the fields included in table #1.0, the fields included in table #1.1 and the fields included in table #1 are completely same, and the data in table #1 is recorded in table #1.0 and table #1.1. Figure 16 As shown in , database A of server A further includes table #2. Table #1 can be table 1 in the above, table #1.0 can be table 1.0 in the above, table #1.1 can be table 1.1 in the above, and table #2 can be table 2 in the above.

[0407] Configuration information #i includes configuration information #1.0, configuration information #1.1 and configuration information #2, wherein configuration information #1.0 includes the table name of table #1.0 to be queried and the data source of table #1.0 to be queried, configuration information #1.1 includes the table name of table #1.1 to be queried and the data source of table #1.1 to be queried, and configuration information #2 includes the table name of table #2 to be queried and the data source of table #2 to be queried. Since table #1.0 to be queried, table #1.1 to be queried and table #2 to be queried are all different two data tables in the same database of the same server, the data source of table #1.0 to be queried, the data source of table #1.1 to be queried and the data source of table #2 to be queried are same, i.e. the data source of table #1.0 to be queried, the data source of table #1.1 to be queried and the data source of table #2 to be queried all include the path information of accessing database A of server A, the user and password of accessing database A of server A.

[0408] The above configuration information #i can be obtained by configuring the table information in the configuration page shown in Figure 10 , and Figure 10 the data source in the configuration page shown in Figure 11 can be obtained by configuring the interface shown in the above, and Figure 10 optionally, the configuration page shown in may further not include the main table option.

[0409] S1620, the data synchronization service in server B parses the configuration information #i and generates the query statement #i corresponding to the configuration information #i (i.e., the above). Figure 9 (An example of the K first query statements in the provided data synchronization method).

[0410] Executing S1620 above, that is, the data synchronization service in server B parses the configuration information #i and generates the query statement #i corresponding to the configuration information #i, including: the SQL generation module included in the data synchronization service parses the configuration information #1.0 and generates the query statement #1.0 corresponding to the configuration information #1.0, wherein the query statement #1.0 is used to query table #1.0 in database A located on server A; the SQL generation module included in the data synchronization service parses the configuration information #1.1 and generates the query statement #1.1 corresponding to the configuration information #1.1, wherein the query statement #1.1 is used to query table #1.1 in database A located on server A; the SQL generation module included in the data synchronization service parses the configuration information #2 and generates the query statement #2 corresponding to the configuration information #2, wherein the query statement #2 is used to query table #2 in database A located on server A.

[0411] It is understandable that the SQL generation module included in the data synchronization service generates query statements based on configuration information #i, and its working principle is similar to that described above. Figure 13 The SQL generation module works in the same way as the configuration page information; for details not elaborated here, please refer to the relevant descriptions above.

[0412] Optionally, server B's local database may also include a metadata database, and the query statement #i obtained by the data synchronization service in step S1620 above can also be stored in the metadata database. It is understood that the metadata database and the mirror database are two independent databases. Subsequently, when the data synchronization service needs to perform a query operation using query statement #i, the data synchronization service can retrieve query statement #i from the metadata database.

[0413] S1630, the data synchronization service on server B synchronizes database A (i.e., the aforementioned database) on server A according to the query statement #i. Figure 9 (An example of at least one first database in the provided data synchronization method) queries to obtain tables #1.0, #1.1, and #2.

[0414] The S1630 is executed, that is, the data synchronization service in the server B queries the database in the server Ai according to the query statement #i to obtain the table #i, including: the data synchronization service queries the database A in the server Ai according to the query statement #1.0 to obtain the table #1.0; the data synchronization service queries the database A in the server Ai according to the query statement #1.1 to obtain the table #1.1; and the data synchronization service queries the database A in the server Ai according to the query statement #2 to obtain the table #2.

[0415] It can be understood that the table #1.0, the table #1.1 and the table #2 in the S1630 are examples of the K source data tables in the data synchronization method provided by the application. Figure 9 The table #1.0 and the table #1.1 are examples of part of the K source data tables in the data synchronization method provided by the application. Figure 9 The table #2 is an example of the source data table in the K source data tables in the data synchronization method provided by the application, except for the part of the source data tables. Figure 9 The table #2 is an example of the source data table in the K source data tables in the data synchronization method provided by the application, except for the part of the source data tables.

[0416] The S1640 is executed, that is, the data synchronization service in the server B writes the table #1.0 and the table #1.1 into the table #1 in the mirror database in the server B, and writes the table #2 into the mirror database (that is, an example of the second database in the data synchronization method provided by the application). Figure 9 The S1640 is executed, that is, the data synchronization service in the server B writes the table #1.0 and the table #1.1 into the table #1 in the mirror database in the server B, and writes the table #2 into the mirror database (that is, an example of the second database in the data synchronization method provided by the application).

[0417] It can be understood that the table #1 and the table #2 in the S1640 are examples of the P target data tables in the data synchronization method provided by the application. Figure 9 The table #1 and the table #2 in the S1640 are examples of the P target data tables in the data synchronization method provided by the application.

[0418] After the S1610 to the S1640 are executed, the mirror database in the server B stores the business tables (that is, the table #1 and the table #2, wherein the table #1 is a table obtained by merging the table #1.0 and the table #1.1) in the business server (that is, the server Ai).

[0419] The S1650 is executed, that is, the data synchronization service in the server B provides a configuration page to the server Ai, and the server Ai configures on the configuration page to make the data synchronization service obtain the configuration information 1 (that is, an example of the second configuration information in the data synchronization method provided by the application). Figure 9 The S1650 is executed, that is, the data synchronization service in the server B provides a configuration page to the server Ai, and the server Ai configures on the configuration page to make the data synchronization service obtain the configuration information 1 (that is, an example of the second configuration information in the data synchronization method provided by the application).

[0420] The configuration information 1 in the step S1650 is different from the configuration information 1 in the step S1410 in that the data source of the table #i in the step S1660 is the mirror database in the server B, and the user and the password in the data source are the user and the password for accessing the mirror database; the data source of the table #i in the step S1410 is the database A of the server A, and the user and the password in the data source are the user and the password for accessing the database A.

[0421] In the step S1660, the data synchronization service in the server B parses the configuration information 1 to generate a query statement 1 (i.e., an example of the second query statement in the data synchronization method provided). Figure 9

[0422] The query statement 1 in the step S1660 is different from the query statement 1 in the step S1420 in that the query statement 1 in the step S1660 is used for querying the table #1 and the table #2 in the mirror database in the server B; the table #i in the step S1410 is used for querying the table #1 and the table #2 in the database A of the server A.

[0423] The step S1660 and the step S1420 have the same working principle, and the content not described in detail here can be referred to the related description in the step S1420.

[0424] Optionally, the local database of the server B can further include a meta database, and the query statement 1 obtained by the data synchronization service in the step S1660 can be stored in the meta database. It can be understood that the meta database and the mirror database are two independent databases. Thereafter, when the data synchronization service needs to perform a query operation by using the query statement 1, the data synchronization service can obtain the query statement 1 from the meta database.

[0425] In the step S1670, the data synchronization service in the server B performs a joint query on the table #1 and the table #2 in the mirror database in the server B according to the query statement 1 to obtain to-be-synchronized data (i.e., an example of the target data in the data synchronization method provided). Figure 9

[0426] The to-be-synchronized data in the step S1670 is the same as the to-be-synchronized data in the step S1430, and the step S1670 and the step S1430 have the same working principle, and the content not described in detail here can be referred to the related description in the step S1430.

[0427] In the step S1680, the data synchronization service in the server B sends the to-be-synchronized data to the ES.

[0428] In the step S1690, the ES stores the to-be-synchronized data in the ES (i.e., an example of the data synchronization method provided).​​ Figure 9 The index (i.e. the index mentioned above in the search engine of the data synchronization method provided by the present application) of the search engine in the data synchronization method provided by the present application. Figure 9 The index in the data synchronization method provided by the present application.

[0429] The working principle of the data synchronization service described in the above S1690 step and the above S1460 step for storing the data to be synchronized into the index of the ES is the same, and the content not described in detail here can be referred to the relevant description in the S1460 step.

[0430] It should be understood that the above Figure 16 The data synchronization method shown is only illustrative and does not constitute any limitation on the data synchronization method provided by the present application. For example, the above Figure 16 The method shown can further include a message queue, based on which, after the above S1610 step to S1670 step is executed, the data synchronization service first sends the data to be synchronized to the message queue in the server C for caching, and then the ES reads the data to be synchronized from the message queue and stores the obtained data to be synchronized into the index of the ES.

[0431] In the embodiment of the present application, on the one hand, the query statement #i generated based on the configuration information #i obtained from the configuration page provided by the data synchronization service of the server B can achieve the purpose of source data synchronization, i.e. synchronizing multiple business data tables (i.e. table #1.0, table #1.1 and table #2 in the database A of the server A, and table #1.0 and table #1.1 are obtained by splitting the business data table 1, i.e. the fields included in table #1.0 and table #1.1 are exactly the same) in the local database of the same business server to the mirror database of the server B, so that the frequent access of the server B to the business server (i.e. the server A) can be avoided, the influence on the business of the business server can be avoided, and thus the processing performance of the business server can be improved. On the other hand, based on the configuration page provided by the data synchronization service of the server B, the data synchronization service can obtain the configuration information 1. Next, the data synchronization service generates a data query statement (i.e. query statement 1) according to the configuration information 1, wherein the data query statement can simultaneously achieve the purposes of data extraction (i.e. reading data from the mirror database) and data filtering and conversion (i.e. filtering the data read from the mirror database according to the filtering condition in the configuration information 1), i.e. combining the data extraction and data filtering and conversion processes into one, so that the data query efficiency can be improved. Then, the data to be synchronized obtained by querying is synchronized to the fields of the index of the corresponding ES, so that the data synchronization efficiency can be improved.

[0432] Next, taking the above Figure 7 application scenario (i.e. the scenario of multiple databases and table splitting) as an example, another data synchronization method provided by the present application is described.

[0433] Figure 17 This is a schematic diagram illustrating a data synchronization method provided in an embodiment of this application. It can be understood that... Figure 17 The data synchronization method provided is as described above. Figure 9 A specific example of the provided data synchronization method, and Figure 17 The data synchronization method shown is as described above. Figure 9 The data synchronization method provided in the example is to use the P target data tables in the second database as an example of data backup of K source data tables in at least one first database. Here, K is an integer greater than P, that is, the P target data tables are obtained by merging some data tables in the K source data tables. The at least one first database can be a database located in the electronic device of the business party, and the data in the K source data tables can be data generated by the application of the electronic device of the business party.

[0434] For example, such as Figure 17 As shown, the method includes steps S1710 to S1791. Steps S1710 to S1791 will be described in detail below.

[0435] S1710, the data synchronization service in server B provides a configuration page to the business server. The business server configures itself on the configuration page so that the data synchronization service receives the configuration information #i (i.e., the above). Figure 9 An example of the first configuration information in the provided data synchronization method), wherein the servers include server A0 and server A1, i = 1.0, 1.1, 2.

[0436] like Figure 17 As shown in this embodiment, table #1.0 in database A on server A0 and table #1.1 in database B on server A1 are two sub-tables obtained by splitting table #1. That is, the fields included in table #1.0, the fields included in table #1.1, and the fields included in table #1 are exactly the same. The data in table #1 is recorded in tables #1.0 and #1.1. Figure 17 As shown, database A of server A also includes table 2. Table #1 can be table 1 mentioned above, table #1.0 can be table 1.0 mentioned above, table #1.1 can be table 1.1 mentioned above, and table #2 can be table 2 mentioned above.

[0437] The configuration information #i includes configuration information #1.0, configuration information #1.1 and configuration information #2. The configuration information #1.0 includes the table name of the table #1.0 to be queried and the data source of the table #1.0 to be queried. The configuration information #1.1 includes the table name of the table #1.1 to be queried and the data source of the table #1.1 to be queried. The configuration information #2 includes the table name of the table #2 to be queried and the data source of the table #2 to be queried. Since the table #1.0 to be queried and the table #2 to be queried are two different data tables in the same database of the same server, the data source of the table #1.0 to be queried and the data source of the table #2 to be queried are the same, that is, the data source of the table #1.0 to be queried and the data source of the table #2 to be queried both include the path information of accessing the database A of the server A0, the user and the password of accessing the database A of the server A0. The data source of the table #1.1 to be queried includes the path information of accessing the database B of the server A1, the user and the password of accessing the database B of the server A1.

[0438] The configuration information #i can be obtained by configuring the table information in the configuration page shown in the above Figure 10 , and Figure 10 The data source in the configuration page shown in the above Figure 11 can be obtained by configuring the interface shown in the above Figure 10 , optionally, The configuration page shown in the above

[0439] In S1720, the data synchronization service in the server B parses the configuration information #i to generate a query statement #i (that is, one of the K first query statements in the data synchronization method provided in the above Figure 9 ).

[0440] In the above S1720, the data synchronization service in the server B parses the configuration information #i to generate the query statement #i corresponding to the configuration information #i, including: the SQL generation module included in the data synchronization service parses the configuration information #1.0 to generate the query statement #1.0 corresponding to the configuration information #1.0, wherein the query statement #1.0 is used to query the table #1.0 in the database A of the server A0; the SQL generation module included in the data synchronization service parses the configuration information #1.1 to generate the query statement #1.1 corresponding to the configuration information #1.1, wherein the query statement #1.1 is used to query the table #1.1 in the database B of the server A1; the SQL generation module included in the data synchronization service parses the configuration information #2 to generate the query statement #2 corresponding to the configuration information #2, wherein the query statement #2 is used to query the table #2 in the database A of the server A0.

[0441] It can be understood that the working principle of the SQL generation module included in the data synchronization service to generate the query statement #i according to the configuration information #i is the same as the working principle of the SQL generation module to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description. Figure 13 Figure 13 The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description.

[0442] S1730, the data synchronization service in the server B queries the database of the service server (i.e., one example of the at least one first database in the data synchronization method provided in the foregoing description) according to the query statement #i, and obtains the table #1.0, the table #1.1 and the table #2. Figure 9 Figure 9 The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description. The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description.

[0443]

[0443] It can be understood that the table #1.0, the table #1.1 and the table #2 in the foregoing S1730 are one example of the K source data tables in the data synchronization method provided in the foregoing description. The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description.

[0444]

[0444] The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 Figure 9 The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 Figure 9 The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 Figure 9 The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description.

[0445]

[0445] The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 Figure 9 The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description.

[0446]

[0446] The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 Figure 9 The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description.

[0447]

[0447] The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description, and the table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description.

[0448] S1750, the data synchronization service in server B provides a configuration page to server A1, and server A configures in the configuration page to make the data synchronization service get configuration information 1 (i.e. the above Figure 9 An example of the second configuration information in the data synchronization method provided by the application).

[0449] The difference between the configuration information 1 in the above S1750 and the configuration information 1 in the above S1410 is that the data source of table #i in the above S1760 is the mirror database in server B, and the user and password in the data source are the user and password for accessing the mirror database; the data source of table #i in the above S1410 is the database A in server A, and the user and password in the data source are the user and password for accessing the database A.

[0450] S1760, the data synchronization service in server B parses the configuration information 1 to generate a query statement 1 (i.e. the above Figure 9 An example of the second query statement in the data synchronization method provided by the application).

[0451] The difference between the query statement 1 in the above S1760 and the query statement 1 in the above S1420 is that the query statement 1 in the above S1760 is used to query table #1 and table #2 in the mirror database in server B; the table #i in the above S1410 is used to query table #1 and table #2 in the database A in server A.

[0452] The working principle of the above S1760 and the above S1420 is the same, and the contents not described in detail here can be referred to the related description in the above S1420.

[0453] S1770, the data synchronization service in server B performs a joint query on table #1 and table #2 in the mirror database according to the query statement 1 to get the to-be-synchronized data (i.e. the above Figure 9 An example of the target data in the data synchronization method provided by the application).

[0454] The to-be-synchronized data in the above S1770 is the same as the to-be-synchronized data in the above S1430, and the working principle of the above S1770 and the above S1430 is the same, and the contents not described in detail here can be referred to the related description in the above S1430.

[0455] S1780, the data synchronization service in server B sends the to-be-synchronized data to the message queue in server C (i.e. the above Figure 9 An example of the message queue in the data synchronization method provided by the application).

[0456] S1790, the message queue in the server C caches the data to be synchronized.

[0457] S1791, the ES in the server C (i.e. the above-mentioned Figure 9 search engine in the data synchronization method provided by the present application) obtains the data to be synchronized from the message queue, and stores the obtained data to be synchronized into the index (i.e. the above-mentioned Figure 9 index in the data synchronization method provided by the present application).

[0458] The working principle of the data synchronization service described in the above-mentioned S1791 step and the above-mentioned S1460 step for storing the data to be synchronized into the index of the ES is the same, and the content not described in detail herein can be referred to the relevant description in the above-mentioned S1460 step.

[0459] It should be understood that the above-mentioned Figure 17 data synchronization method and the above-mentioned Figure 16 data synchronization method are different in that the above-mentioned Figure 17 Q in the data synchronization method shown in the above-mentioned Figure 16 Q in the data synchronization method shown in the above-mentioned

[0460] It should be understood that the above-mentioned Figure 17 data synchronization method is only illustrative, and does not constitute any limitation on the data synchronization method provided by the present application. For example, the above-mentioned Figure 17 method can further not include the message queue, based on which, after the above-mentioned S1710 step to S1770 step is executed, the data synchronization service directly sends the data to be synchronized to the ES, and thereafter, the ES synchronizes the data to be synchronized into the index of the ES.

[0461] In the embodiment of the present application, on the one hand, the query statement #i generated based on the configuration information #i obtained from the configuration page provided by the data synchronization service of the server B can achieve the purpose of source data synchronization, that is, synchronizing the plurality of business data tables in the plurality of databases of the plurality of business servers (that is, the table #1.0 and the table #2 in the database A of the server A0, and the table #1.1 in the database B of the server A1, and the table #1.0 and the table #1.1 are obtained after splitting the business data table 1, that is, the fields included in the table #1.0 and the table #1.1 are completely the same) to the mirror database of the server B, so that the frequent access of the server B to the business servers (that is, the server A0 and the server A1) can be avoided, and the influence on the business of the business servers can be avoided, thereby facilitating to improve the processing performance of the business servers. On the other hand, based on the configuration page provided by the data synchronization service of the server B, the data synchronization service can obtain the configuration information 1. Next, the data synchronization service generates a data query statement (that is, the query statement 1) according to the configuration information 1, wherein the data query statement can simultaneously achieve the purposes of data extraction (that is, reading data from the mirror database) and data filtering and conversion (that is, filtering the data read from the mirror database according to the filtering condition in the configuration information 1), that is, combining the data extraction and the data filtering and conversion into one, so that the data query efficiency can be improved. Subsequently, the to-be-synchronized data obtained by querying is synchronized to the fields of the index of the corresponding ES, so that the data synchronization efficiency can be improved.

[0462] The data synchronization method of the embodiment of the present application is described in detail above, Figures 9 to 17 The device embodiment of the present application will be described in detail below in combination with Figure 18 and Figure 19 It should be understood that the data synchronization device in the embodiment of the present application can perform various data synchronization methods of the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

[0463] Figure 18 FIG. 1 is a schematic diagram of a data synchronization device provided by the embodiment of the present application.

[0464] Exemplarily, Figure 18 The data synchronization device shown in the figure is applied to a first electronic device, and the data synchronization device 1800 includes a processing unit 1810, wherein the processing unit 1810 is configured to execute any one of the data synchronization methods provided by the embodiments of the present application.

[0465] It should be noted that the data synchronization device 1800 described above is in the form of a functional unit. The term "unit" herein can be implemented in the form of software and / or hardware, and no specific limitation is made thereto.

[0466] For example, the "unit" can be a software program, a hardware circuit, or a combination of both, which implements the above functions. The hardware circuit can include an application specific integrated circuit (ASIC), an electronic circuit, a processor (for example, a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combination logic circuit, and / or other suitable components that support the described functions.

[0467] Therefore, the units of each example described in the embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0468] Figure 19 FIG. 1 is a schematic diagram of a structure of an electronic device provided by the present application. Figure 19 The dashed line in FIG. 19 indicates that the unit or the module is optional. The electronic device 1900 can be used to implement the data synchronization method described in the above method embodiments.

[0469] The electronic device 1900 includes one or more processors 1901, which can support the electronic device 1900 to implement the data synchronization method in the method embodiments. The processor 1901 can be a general-purpose processor or a special-purpose processor. For example, the processor 1901 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates or transistor logic devices, or discrete hardware components.

[0470] The processor 1901 can be used to control the electronic device 1900, execute software programs, and process data of the software programs. The electronic device 1900 can further include a communication unit 1905 to implement input (reception) and output (transmission) of signals.

[0471] For example, the electronic device 1900 can be a chip, the communication unit 1905 can be an input and / or output circuit of the chip, or the communication unit 1905 can be a communication interface of the chip, and the chip can be a component of a terminal device or other electronic device.

[0472] For another example, the electronic device 1900 can be a terminal device, the communication unit 1905 can be a transceiver of the terminal device, or the communication unit 1905 can be a transceiver circuit of the terminal device.

[0473] The electronic device 1900 can include one or more memories 1902, which store programs 1904 that can be run by the processor 1901 to generate instructions 1903, so that the processor 1901 executes the data synchronization method described in the above method embodiments according to the instructions 1903.

[0474] Optionally, the memory 1902 can also store data. Optionally, the processor 1901 can also read the data stored in the memory 1902, which can be stored in the same storage address as the program 1904, or can be stored in a different storage address from the program 1904.

[0475] The processor 1901 and the memory 1902 can be separately arranged, or can be integrated together; for example, integrated on a system on chip (SOC) of a terminal device.

[0476] For example, the memory 1902 can be used to store the related program 1904 of the data synchronization method provided in the embodiments of the present application, and the processor 1901 can be used to call the related program 1904 of the data synchronization method stored in the memory 1902 to execute the data synchronization method of the embodiments of the present application.

[0477] The present application also provides a computer program product, which, when executed by a processor, implements the data synchronization method described in any of the method embodiments of the present application.

[0478] The computer program product can be stored in a memory, for example, a program, which is finally converted into an executable object file that can be executed by a processor through preprocessing, compiling, assembling, and linking processes.

[0479] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a computer, implements the data synchronization method described in any of the method embodiments of the present application. The computer program can be a high-level language program or an executable object program.

[0480] In this application, "at least one" means one or more, "multiple" means two or more. "At least one of the following (one)" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0481] It should be understood that the size of the sequence of the above-mentioned processes does not mean the order of execution in various embodiments of the present application. The execution order of the processes should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0482] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0483] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0484] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic; for example, the division of the units is only a logical function division, and actual implementation can have another division manner; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other form.

[0485] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0486] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each of the units can exist physically, or two or more units can be integrated in one unit.

[0487] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of data synchronization, the method comprising: Applied to the first electronic device, the method comprises: According to the first configuration information, K first query statements are generated, wherein the first configuration information comprises the table names of K source data tables and the K source data tables are stored in at least one first database, the K first query statements correspond to the K source data tables one by one, each first query statement is used to query the corresponding source data table, K is an integer greater than 1; According to the K first query statements, the K source data tables located in the at least one first database are synchronized to a second database, so that P target data tables corresponding to the K source data tables are stored in the second database, wherein the data in the K source data tables is generated by an application of an electronic device where the at least one first database is located, and the electronic device where the at least one first database is located and the second database are different, P is an integer greater than 1 and not more than K, the P target data tables include a first data table and a second data table, wherein the first field is the first data table corresponding to the to-be-queried field, and the second data table corresponding to the to-be-queried field includes a second field and a third field; Obtain second configuration information, wherein the second configuration information comprises mapping information, filtering conditions, connection relationships of the P target data tables, and the P target data tables stored in the second database, the mapping information comprises a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, and the filtering conditions are used to filter field values of to-be-queried fields corresponding to at least one target data table that meets a preset condition; the mapping relationship includes a non-conditional mapping relationship and a conditional mapping relationship, wherein the plurality of fields include a fourth field, a fifth field and a sixth field, the non-conditional mapping relationship is a mapping relationship between the fourth field and the first field, the conditional mapping relationship includes a mapping relationship between the fifth field and the third field when the field value of the second field is a first value, and a mapping relationship between the sixth field and the third field when the field value of the second field is a second value, the first value and the second value are different; According to the second configuration information, a second query statement is generated, wherein the second query statement is used to query field values of the plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields that meet the preset condition, and the associated data table is a data table obtained by associating the P target data tables according to the connection relationship; According to the second query statement, the P target data tables in the second database are jointly queried to obtain target data, wherein the target data comprises the plurality of fields and the field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition; Synchronize the target data to the index.

2. The method of claim 1, wherein, The K first query statements are generated according to the first configuration information, comprising: According to the first configuration information, a first generator is selected from a plurality of candidate generators, wherein the plurality of candidate generators correspond to a plurality of types of statements, each candidate generator is configured to generate a statement of a corresponding type, and the first generator is configured to generate a statement for querying each source data table; The first configuration information is processed by using the first generator, and K first query statements are obtained.

3. The method according to claim 1 or 2, characterized in that, P is equal to K, and The P target data tables are the K source data tables, and fields of any two source data tables in the K source data tables are not completely same.

4. The method according to claim 1 or 2, characterized in that, P is less than K, the P target data tables include one data table obtained by merging part of the K source data tables and remaining source data tables except the part of the K source data tables, any two source data tables in the part of the K source data tables include same fields, and fields of any one source data table in the remaining source data tables are not completely same with fields of any one source data table in the part of the K source data tables, and According to the K first query statements, the K source data tables located in the at least one first database are synchronized to a second database, so that the second database stores the P target data tables corresponding to the K source data tables, including: According to part of the K first query statements, the part of the source data tables located in the at least one first database are synchronized to a same data table in the second database, and according to statements except the part of the K first query statements, the remaining source data tables located in the at least one first database are synchronized to the second database, so that the second database stores the P target data tables.

5. The method according to claim 1 or 2, characterized in that, According to the second configuration information, a second query statement is generated, including: According to the second configuration information, a plurality of target generators are selected from a plurality of candidate generators, wherein the plurality of candidate generators correspond to a plurality of types of statements, each candidate generator is configured to generate a statement of a corresponding type, the plurality of target generators include a first generator, a second generator and a third generator, the first generator is configured to generate a statement for querying field values of the plurality of to-be-queried fields having the mapping relationship with the plurality of fields, the second generator is configured to generate a statement representing the connection relationship, and the third generator is configured to generate a statement representing the filtering condition; The second configuration information is processed by using the plurality of target generators, a plurality of statements are obtained, and the plurality of statements are spliced to obtain the second query statement.

6. The method of claim 1 or 2, wherein, The target data is synchronized to the index, including: The target data is sent to a message queue, so that the target data located in the message queue is synchronized to the index.

7. The method according to claim 1 or 2, characterized in that, The search engine is located in a second electronic device different from the first electronic device, and The target data is synchronized to the index, including: The target data is synchronized to the index. sending the target data to the search engine in the second electronic device, so that the search engine synchronizes the target data obtained into the index.

8. The method of claim 1 or 2, wherein the preset condition comprises that a field value of a to-be-queried field corresponding to the at least one target data table exceeds a preset value; or, the preset condition comprises that a field value of a to-be-queried field corresponding to the at least one target data table does not exceed the preset value; or, in a case where the at least one target data table further comprises a non-to-be-queried field other than the to-be-queried field corresponding to the at least one target data table, the preset condition comprises that a field value of the non-to-be-queried field is equal to the preset value. The first electronic device is provided with a configuration page, and the method further comprises:

9. The method of claim 1 or 2, wherein, obtaining the first configuration information and the second configuration information through the configuration page. The search engine is a distributed full-text search engine.

10. The method of claim 1 or 2, wherein, The electronic device comprises a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the processor executes the data synchronization method in any one of claims 1 to 10.

11. An electronic device, comprising: The computer readable storage medium stores a computer program, when the computer program is executed by the processor, the processor executes the data synchronization method in any one of claims 1 to 10.

12. A chip, characterized by The computer readable storage medium stores a computer program, when the computer program is executed by the processor, the processor executes the data synchronization method in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, ​

Images