Data synchronization method, apparatus and electronic device

Abstract

The application discloses a data synchronization method and device and electronic equipment. The method is applied to a first electronic device and includes: obtaining configuration information, the configuration information including mapping information, filtering conditions, connection relationships of P target data tables, and storage of the P target data tables in a target database, the mapping information including a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, the filtering conditions being used for screening field values of the to-be-queried fields corresponding to at least one target data table satisfying a preset condition, and P being an integer greater than 1; generating a query statement according to the configuration information; performing joint query on the P target data tables in the target database according to the query statement to obtain target data; and synchronizing the target data to the index. Based on the technical scheme of the application, the data synchronization efficiency between the search engine and the database can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and more particularly, to a data synchronization method and device and electronic equipment. BACKGROUND

[0002] When searching data in a database storing a large amount of data, a search engine tool with strong search analysis capability needs to be used to improve search efficiency. In the above data retrieval scenario, since the data to be queried is stored in a relational database, the relational database has the problem of low retrieval efficiency when processing retrieval of a large amount of data.

[0003] Compared with a traditional relational database, a search engine has faster data query speed, such as Elasticsearch (ES), and therefore, in order to ensure the efficiency of data query results, data synchronization between the search engine and the database needs to be implemented. In the traditional technology, data synchronization between the search engine and the database cannot be efficiently implemented.

[0004] Therefore, how to improve the data synchronization efficiency between the search engine and the database becomes a problem to be solved. SUMMARY

[0005] The present application provides a data synchronization method, device and electronic equipment, which can improve the data synchronization efficiency between the search engine and the database.

[0006] In a first aspect, a data synchronization method is provided, applied to a first electronic device, including: obtaining configuration information, wherein the configuration information includes mapping information, filtering conditions, a connection relationship of P target data tables, and the P target data tables stored in a target database, the mapping information includes a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, the filtering conditions are used to filter field values of the to-be-queried fields corresponding to at least one target data table satisfying a preset condition, and P is an integer greater than 1; generating a query statement according to the configuration information, wherein the query statement is used to query field values of the plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields and satisfying the preset condition, and the associated data table is a data table obtained by associating the P target data tables according to the connection relationship; performing a joint query on the P target data tables in the target database according to the query statement to obtain target data, wherein the target data includes the plurality of fields and the field values of the plurality of to-be-queried fields in the associated data table and satisfying the preset condition; and synchronizing the target data to the index.

[0007] The P target data tables are a plurality of tables to be queried, and the data in the P target data tables can be data generated by a business server, for example. The P target data tables can include a main table, and the data tables other than the main table in the P target data tables can be sub-tables.

[0008] The at least one target data table can be a main table or a sub-table, and no specific limitation is imposed thereon.

[0009] The mapping relationship between the plurality of fields in the index of the search engine and the plurality of to-be-queried fields corresponding to the P target data tables should be understood as follows: the field values of the fields in the index that have the mapping relationship should be consistent with the field values of the to-be-queried fields in the corresponding target data table. It can be understood that, in the embodiments of the present application, the ultimate purpose of data synchronization is to synchronize the field values of the plurality of to-be-queried fields in the P target data tables that satisfy the preset condition to the field values of the plurality of fields in the corresponding index.

[0010] The P target data tables correspond to a plurality of to-be-queried fields, and each target data table corresponds to at least one to-be-queried field in the plurality of to-be-queried fields.

[0011] The one query statement generated according to the configuration information is used to query the field values of the plurality of to-be-queried fields in the associated data table corresponding to the plurality of fields and satisfying the preset condition, and the associated data table is a data table obtained by associating the P target data tables according to a connection relationship. It can be understood that the one query statement has the following functions: associating the P target data tables according to a connection relationship, mapping the plurality of fields in the index and the plurality of to-be-queried fields corresponding to the P target data tables, and querying the field values of the plurality of to-be-queried fields in the associated data table corresponding to the plurality of fields and satisfying the preset condition.

[0012] In the technical solution, the configuration information includes the filtering condition, the connection relationship of the P target data tables, and the information that the P target data tables are stored in the target database. Therefore, by using the query statement generated according to the configuration information, the data (that is, the data in the P target data tables) can be extracted from the data source (that is, the target database), and the data extracted from the data source can be filtered (that is, the field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition are queried) according to the query requirement (that is, the preset condition). Therefore, the method can improve the data query efficiency, thereby improving the data synchronization efficiency between the search engine and the database. In addition, the configuration information further includes the mapping relationship (that is, the mapping information) between the plurality of fields in the index of the search engine and the plurality of to-be-queried fields corresponding to the P target data tables. Therefore, the query statement generated according to the configuration information is used to query the field values of the plurality of to-be-queried fields in the associated data table (that is, the data table obtained by associating the P target data tables according to the connection relationship) that meet the preset condition. Therefore, the target data obtained by jointly querying the P target data tables by using the query statement includes the field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition and the plurality of fields in the index. Thereafter, the target data does not need to be subjected to field mapping (for example, mapping the fields in the database to the fields in the index) and the like, and can be directly synchronized to the index. Therefore, the method can improve the data synchronization efficiency between the search engine and the database. In summary, the data synchronization method provided in the present application can improve the data synchronization efficiency between the search engine and the database.

[0013] In a possible implementation, the configuration information further includes path information of the target database, and before the P target data tables in the target database are jointly queried according to the query statement to obtain target data, the method further includes: acquiring the P target data tables according to the path information of the target database.

[0014] The path information of the target database is used to indicate the path of the target database connected to the P target data tables. For example, the path information of the target database includes the uniform resource locator (URL) of the target database, the username and password for accessing the target database.

[0015] In the technical solution, the first electronic device can acquire the P target data tables indicated by the path information of the target database carried in the configuration information, and the implementation is relatively simple.

[0016] In another possible implementation, the generating the query statement according to the configuration information comprises: selecting a plurality of target generators from a plurality of candidate generators according to the configuration information, wherein the plurality of candidate generators correspond to a plurality of types of statements, each candidate generator is used to generate a statement of a corresponding type, the plurality of target generators comprise a first generator, a second generator and a third generator, the first generator is used to generate a statement of field values of the plurality of fields to be queried which have the mapping relationship with the plurality of fields, the second generator is used to generate a statement representing the connection relationship, and the third generator is used to generate a statement representing the filter condition; processing the configuration information by using the plurality of target generators to obtain the plurality of statements, and splicing the plurality of statements to obtain the query statement.

[0017] In the technical solution, the configuration information can be used to select the plurality of target generators from the plurality of candidate generators, and then the configuration information is processed by using the plurality of target generators to obtain the query statement matching the configuration information. In this implementation, when the query requirement changes (for example, the filter condition in the configuration information changes), the target generator actually needed can be selected from the plurality of candidate generators more flexibly, so that the reuse of the generator is facilitated.

[0018] In another possible implementation, the synchronizing the target data to the index comprises: sending the target data to a message queue, so that the target data located in the message queue is synchronized to the index.

[0019] For example, when the message queue and the search engine are located in a first electronic device, the first electronic device sends the target data to the message queue, so that the target data located in the message queue is synchronized to the index, which comprises: a data synchronization service of the first electronic device sends the target data to the message queue; and the search engine synchronizes the target data located in the message queue to the index. It can be understood that the data synchronization service of the first electronic device can execute the data synchronization method provided in the application, and the data synchronization service is an application located in an application layer of the first electronic device.

[0020] For example, when the message queue and the search engine are located in a second electronic device different from the first electronic device, the first electronic device sends the target data to the message queue, so that the target data located in the message queue is synchronized to the index, which comprises: the first electronic device sends the target data to the message queue of the second electronic device, so that the search engine synchronizes the target data located in the message queue to the index.

[0021] For example, the message queue can be, but is not limited to, a message queue Kafka.

[0022] In the technical solution, the first electronic device sends the obtained target data to the message queue first, and then the search engine downstream can read the target data from the message queue at its own pace, so that the first electronic device synchronizes the target data in the message queue to the index, which can avoid the phenomenon of read blocking of the search engine due to too large data volume of the target data, improve the reading efficiency of the search engine, and thus improve the data synchronization efficiency between the search engine and the database.

[0023] In another possible implementation, the search engine is located in a second electronic device different from the first electronic device, and the synchronizing the target data to the index includes sending the target data to the search engine in the second electronic device, so that the search engine synchronizes the obtained target data to the index.

[0024] In the technical solution, when the second electronic device where the search engine is located and the first electronic device are two different electronic devices, the interaction between the second electronic device and the first electronic device is also involved in the process of synchronizing the target data to the index.

[0025] In another possible implementation, the P target data tables include a first data table and a second data table, the first data table corresponds to a first field as the to-be-queried field, the second data table corresponds to a second field and a third field as the to-be-queried fields, the multiple fields include a fourth field, a fifth field, and a sixth field, and the mapping relationship includes a non-conditional mapping relationship and a conditional mapping relationship, the non-conditional mapping relationship is a mapping relationship between the fourth field and the first field, the conditional mapping relationship includes a mapping relationship between the fifth field and the third field when the field value of the second field is a first value, and a mapping relationship between the sixth field and the third field when the field value of the second field is a second value, and the first value and the second value are different.

[0026] For example, the first data table can be a master table, and the second data table can be a slave table.

[0027] The mapping relationship can be set according to the to-be-queried field corresponding to each data table in the P data tables, and no specific limitation is made in this regard.

[0028] In another possible implementation manner, the preset condition comprises that a field value of the to-be-queried field corresponding to the at least one target data table exceeds a preset value; or, the preset condition comprises that the field value of the to-be-queried field corresponding to the at least one target data table does not exceed the preset value; or, in a case where the at least one target data table further comprises a non-to-be-queried field in addition to the to-be-queried field corresponding to the at least one target data table, the preset condition comprises that a field value of the non-to-be-queried field is equal to the preset value.

[0029] The non-to-be-queried field in the at least one target data table refers to a field that is not queried in the at least one target data table in the data synchronization process.

[0030] For example, the at least one target data table comprises a field A and a field B, the to-be-queried field corresponding to the at least one target data table is the field A, and the field B can be referred to as a non-to-be-queried field of the at least one target data table.

[0031] It should be understood that the preset condition in the above implementation manner is merely illustrative, that is, the preset condition is not specifically limited in the present application, and can be set according to actual business query requirements.

[0032] In another possible implementation manner, the first electronic device is provided with a configuration page, and the obtaining of the configuration information comprises: obtaining the configuration information through the configuration page.

[0033] In the above technical solution, the first electronic device obtains the configuration information in a page configuration manner, which reduces the system use threshold. Compared with the manner of manually writing a query statement by an engineer in the prior art, the method can reduce the probability of errors in the written query statement, improve the accuracy of the data query result, and thus improve the accuracy of the data synchronization result between the search engine and the database.

[0034] In another possible implementation manner, the search engine is a distributed full-text search engine.

[0035] For example, the distributed full-text search engine can be, but is not limited to, an ES search engine or a RediSearch search engine.

[0036] In a second aspect, a data synchronization apparatus is provided, which is applied to a first electronic device, and includes a processing unit, configured to: obtain configuration information, wherein the configuration information includes mapping information, a filtering condition, a connection relationship of P target data tables, and storage of the P target data tables in a target database, the mapping information includes a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, the filtering condition is used to filter a field value of a to-be-queried field corresponding to at least one target data table satisfying a preset condition, and P is an integer greater than 1; generate a query statement according to the configuration information, wherein the query statement is used to query field values of the plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields and satisfying the preset condition, and the associated data table is a data table obtained by associating the P target data tables according to the connection relationship; perform a joint query on the P target data tables in the target database according to the query statement, to obtain target data, wherein the target data includes the plurality of fields and the field values of the plurality of to-be-queried fields in the associated data table and satisfying the preset condition; and synchronize the target data to the index.

[0037] In a third aspect, an electronic device is provided, which includes units for performing any of the methods in the first aspect. The device can be a terminal device, or a chip in a terminal device. The device can include an input unit and a processing unit.

[0038] When the device is a terminal device, the processing unit can be a processor, and the input unit can be a communication interface; the terminal device can further include a memory, which is configured to store computer program code, and when the processor executes the computer program code stored in the memory, the terminal device is caused to perform any of the methods in the first aspect.

[0039] When the device is a chip in a terminal device, the processing unit can be a processing unit inside the chip, and the input unit can be an output interface, a pin, or a circuit, etc.; the chip can further include a memory, which can be a memory (e.g., a register, a cache, etc.) inside the chip, or a memory (e.g., a read-only memory, a random access memory, etc.) outside the chip; the memory is configured to store computer program code, and when the processor executes the computer program code stored in the memory, the chip is caused to perform any of the methods in the first aspect.

[0040] In a possible implementation, the memory is configured to store computer program code; and the processor is configured to execute the computer program code stored in the memory, and when the computer program code stored in the memory is executed, the processor is configured to perform any of the methods in the first aspect.

[0041] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program codes, when the computer program codes are run by a data synchronization apparatus, the data synchronization apparatus is caused to perform any of the data synchronization methods in the first aspect.

[0042] In a fifth aspect, a computer program product is provided, which comprises computer program codes, when the computer program codes are run by a data synchronization apparatus, the data synchronization apparatus is caused to perform any of the data synchronization methods in the first aspect.

[0043] It can be understood that the beneficial effects of the second aspect to the fifth aspect described above can be referred to the related description in the first aspect described above, and will not be repeated here.

[0044] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it can be understood that the description of a feature or a beneficial effect means that the specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and beneficial effects described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1A is a schematic diagram of an application scenario suitable for the data synchronization method provided by the embodiments of the present application.

[0046] Figure 1B is a schematic diagram of another application scenario suitable for the data synchronization method provided by the embodiments of the present application.

[0047] Figure 2 is a schematic diagram of a system architecture suitable for the data synchronization method provided by the embodiments of the present application.

[0048] Figure 3 is a schematic diagram of a system architecture suitable for the data synchronization method provided by the embodiments of the present application.

[0049] Figure 4 is a schematic diagram of a specific application scenario corresponding to the system architecture shown above. Figure 3 is a schematic diagram of a specific application scenario corresponding to the system architecture shown above.

[0050] Figure 5 is the above Figure 3 is a schematic diagram of another specific application scenario corresponding to the system architecture shown.

[0051] Figure 6 is the above Figure 3 is a schematic diagram of another specific application scenario corresponding to the system architecture shown.

[0052] Figure 7 is the above Figure 3 is a schematic diagram of another specific application scenario corresponding to the system architecture shown.

[0053] Figure 8 is a schematic diagram of a software system of an electronic device executing an embodiment of the present application.

[0054] Figure 9 is a schematic diagram of a data synchronization method provided by an embodiment of the present application.

[0055] Figure 10 is the above Figure 9 is a schematic diagram of a configuration page corresponding to configuration information in the data synchronization method provided.

[0056] Figure 11 is the above Figure 9 is a schematic diagram of another configuration page corresponding to configuration information in the data synchronization method provided.

[0057] Figure 12 is the above Figure 10 is a schematic diagram of a configuration interface corresponding to a data source in the configuration page provided.

[0058] Figure 13 is the above Figure 9 is a schematic diagram of a query statement generated by a first electronic device according to configuration information in the data synchronization method provided.

[0059] Figure 14 is a schematic diagram of another data synchronization method provided by an embodiment of the present application.

[0060] Figure 15 is a schematic diagram of another data synchronization method provided by an embodiment of the present application.

[0061] Figure 16 is a schematic diagram of another data synchronization method provided by an embodiment of the present application.

[0062] Figure 17 is a schematic diagram of another data synchronization method provided by an embodiment of the present application.

[0063] Figure 18 is a schematic diagram of a data synchronization device provided by an embodiment of the present application.

[0064] Figure 19 is a schematic diagram of a structure of an electronic device provided in the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0066] In order to better understand the data synchronization method provided in the present application, first, the professional terms involved in the embodiments of the present application are briefly introduced.

[0067] 1. Full-text search engine

[0068] The full-text search engine is a mainstream search engine widely used at present. The working principle of the full-text search engine is that a computer indexing program establishes an index for each word in an article by scanning each word in the article, indicating the number of times and the position of the word appearing in the article, when a user queries, the retrieval program searches according to the previously established index, and feeds back the search result to the user. This process is similar to the process of searching for a word in a dictionary through a retrieval word table.

[0069] For example, the full-text search engine can be but is not limited to an ES search engine or a RediSearch search engine.

[0070] Next, taking the full-text search engine as an example of ES, the differences and connections between the full-text search engine and the traditional relational database are introduced.

[0071] For example, Table A shows the comparison between the concepts in ES and the concepts in the traditional relational database.

[0072] Table A

[0073] Relational database MySQL ES search engine database Index Table Type row Document Column Field

[0074] As can be seen from Table A, the index in ES is similar to the database concept in the MySQL database. Simply put, the index is a container of documents, a collection of a class of documents, a classification in logical space, and a sharding in physical space. The type in ES is similar to the table concept in the MySQL database. The type in ES is a logical classification or partition on the index in ES. The document in ES is similar to the row concept in the MySQL database. The field in ES is similar to the column concept in the MySQL database.

[0075] There can be multiple indexes in an ES, each index can contain multiple types, each type can contain multiple documents, and each document can contain multiple fields.

[0076] ES uses the data structure of inverted index to accelerate retrieval, thereby improving data retrieval efficiency.

[0077] 2. Indexing method

[0078] The indexing method includes a forward index and an inverted index.

[0079] Forward index: a structure that uses the unique identifier ID of the document object as the index and the document content as the record.

[0080] Inverted index: a structure that uses the word in the document content as the index and the document ID containing the word as the record.

[0081] The indexing method in the traditional relational database (e.g., MySQL database) is forward index, and the indexing method in ES is inverted index.

[0082] The retrieval efficiency of inverted index is higher than that of forward index.

[0083] For example, when the MySQL database queries "mobile phone" based on the forward index method, the MySQL database will match whether the two characters "mobile phone" appear in each record (i.e., row in the table). In fact, not all records contain "mobile phone", so a lot of useless work is done, and the retrieval efficiency is low. In addition, as the amount of data in the database increases, the resources and time consumed will increase linearly.

[0084] When ES queries "mobile phone" based on inverted index, since ES has established the mapping relationship between the word "mobile phone" and the document in advance, ES can directly query the corresponding document, thereby avoiding querying the document that does not contain the word "mobile phone", which can improve the retrieval efficiency.

[0085] 3. Relational database

[0086] A relational database refers to a database that uses a relational model to organize data. Simply put, the relational model is a two-dimensional table model. For example, the relational database can be, but is not limited to, any of the following databases: MySQL, Oracle, DB2, Sybase, or PostgreSQL database.

[0087] 4. Full-text search

[0088] The part of information in the unstructured data is extracted, reorganized, and structured, and then the structured data is searched, so as to achieve the purpose of relatively fast search. The part of information extracted from the unstructured data and then reorganized is called index. The process of establishing index first and then searching the index is called full-text search.

[0089] 5, data table has sub-table

[0090] In the present application, a data table can be split to obtain multiple sub-data tables of the data table, wherein the fields included in the data table and the fields included in any one of the multiple sub-data tables are completely same, and the multiple sub-data tables collectively record the data in the data table, so that the data table has sub-tables.

[0091] For example, taking a user table as an example, when the number of users recorded in the user table is large, single table storage encounters a bottleneck, and the user information recorded in the user table needs to be split into multiple data tables corresponding to multiple databases for storage, such as splitting the user table into user table 1 and user table 2, and storing user table 1 in database 1 and user table 2 in database 2, wherein database 2 and database 1 are two different databases.

[0092] 6, data table has no sub-table

[0093] In the present application, a data table is not split, i.e., the data table has no sub-table.

[0094] 7, master table

[0095] The master table is a table containing a primary key and a unique identifier. The primary key of a table is a column in the table that represents uniqueness. That is, each row in the master table must have a unique identifier. The primary key is a restriction condition, which does not allow two rows to have the same identifier. Common master tables control data input and ensure data integrity, and a trigger can be written to maintain it.

[0096] Generally, the master table contains the directory of an entity. For example, a table named "student" contains the information of a student, such as student number, name, gender, and birth date. The purpose of the master table is to determine the location of the data through the unique identifier and provide a method for connecting to other tables.

[0097] 8, secondary table

[0098] The secondary table is also called the slave table or the attached table, that is, the secondary table in the present application can be replaced by the description of the slave table or the attached table.

[0099] A secondary table exists based on a primary table, and is associated with the primary table by connecting common fields. The primary table and the secondary table can be connected in a left-join or an inner-join manner. The secondary table usually stores a lot of information related to the primary table, and the information needs to be obtained through the primary table. The secondary table can be associated with the primary table through multiple association fields, but these association fields must be unique.

[0100] For example, a secondary table named "Course" contains information of courses selected by students, such as course number, student number, and score, and therefore, the score information of a corresponding student can be obtained by obtaining the associated data through the unique identifier "student number" of the primary table "Student".

[0101] 9, ambiguous field

[0102] An ambiguous field refers to a field having different values under different conditions. For example, the content field in Table B below is taken as an example. Referring to Table B, under the condition that the course identifier is the same, the value of the content field corresponding to the value of the label identifier field is different, that is, under the condition that the course identifier is 1, if the value of the label identifier field is "2", the value of the content field is "main title"; under the condition that the course identifier is 1, if the value of the label identifier field is "3", the value of the content field is "sub title". Based on this, the content field can be referred to as an ambiguous field.

[0103] Table B

[0104] Course ID (course_id) Label identifier (label_id) Content 1 2 Main title 1 3 subtitle

[0105] 10, binary log (Binlog)

[0106] The binary log can also be referred to as an update log. The binary log records all data definition language (DDL) and data manipulation language (DML), but does not record data query statements (for example, SELECT statements) in the binary log. For example, languages such as creating a database and creating a data table belong to DDL. For example, INSERT, UPDATE, and DELETE belong to DML.

[0107] Next, an application scenario to which the data synchronization method provided in the embodiments of the present application is applied is introduced in combination with the accompanying drawings.

[0108] Figure 1A is a schematic diagram of an application scenario to which the data synchronization method provided in the embodiments of the present application is applied.

[0109] For example, referring to Figure 1AAn application scenario is shown, which includes a first electronic device 110 and a second electronic device 120. The first electronic device 110 and the second electronic device 120 can communicate with each other through an end-to-end communication mode. The communication connection between the first electronic device 110 and the second electronic device 120 can be a short-distance communication connection, for example, a Wi-Fi point-to-point connection or a Bluetooth connection, etc.; the communication connection between the first electronic device 110 and the second electronic device 120 can also be a long-distance communication connection, for example, an Internet connection, etc., and the specific type of the end-to-end communication mode is not limited herein.

[0110] In the embodiments of the present application, the device types of the first electronic device 110 and the second electronic device 120 are not limited. For example, the first electronic device 110 and the second electronic device 120 can be non-terminal devices such as servers, or either of the electronic devices can be a terminal device such as a mobile phone, a tablet computer, and a notebook computer, etc.

[0111] As shown in Figure 1A , the data table #1, …, the data table #N are all located in the local database 121 of the second electronic device 120, where N is a positive integer; and the search engine is located in the first electronic device 110. It should be understood that the local database 121 of the second electronic device 120 is a relational database, for example, the relational database can be but is not limited to any one of the following databases: Mysql, Oracle, DB2, Sybase, or PostgreSQL.

[0112] In the above Figure 1A application scenario, in order to improve the efficiency of data retrieval, the data (for example, part of the data or all the data) in the data tables (for example, N data tables) located in the local database 121 of the second electronic device 120 can be synchronized to the index of the search engine of the first electronic device 110 according to business needs. Thereafter, the purpose of searching the data in the N data tables in the local database 121 of the second electronic device 120 in the search engine can be achieved.

[0113] It should be noted that the above Figure 1A application scenario is exemplified by taking the N data tables as data tables located in the same database (i.e., the database 121). Alternatively, Figure 1A the N data tables shown in may also be located in multiple different databases.

[0114] Exemplarily, taking N equal to 2 as an example, the above Figure 1A application scenario can be replaced by the application scenario shown in Figure 1B . As shown in Figure 1BAs shown, data table #1 is located in database 121 of the second electronic device 120, and data table #2 is located in database 131 of the third electronic device 130. Databases 121 and 131 are two different databases.

[0115] In some application scenarios, such as vertical search, it is often necessary to perform a joint query on data from multiple business data tables to obtain query results that meet the business query requirements. These multiple business data tables may contain different fields. Below, we will combine... Figure 2 This paper introduces the implementation process of synchronizing data from multiple tables in multiple databases to the index of a search engine, which is provided by traditional technology.

[0116] Figure 2 This is a schematic diagram of a system architecture applicable to data synchronization methods provided in traditional technologies. For an example, see [link to example]. Figure 2 The system architecture includes a business system database 210, electronic devices 220 and electronic devices 230.

[0117] The business system database 210 includes two databases (database 1 and database 2), each corresponding to a data table, and each database is used to store the corresponding data table. The databases in the business system database 210 can be databases on the same server or databases on multiple servers; there is no specific limitation on this.

[0118] Electronic device 220 includes an Extract-Transform-Load (ETL) architecture 221, wherein the ETL architecture 221 includes data extraction 2211, data transformation 2212 and data loading 2213.

[0119] Data extraction (2211) is used to extract data from various data sources. For example... Figure 2 As shown, data extraction 2211 is used to extract source data from the two databases included in the business system database 210.

[0120] Data transformation 2212 is used to clean, transform, and organize the data extracted by data extraction 2211 to meet the requirements of the target system (e.g., the data format requirements stored in the search engine 2311). For example, data transformation may include, but is not limited to, operations such as data cleaning, format conversion, and data merging (i.e., merging multiple data tables and merging fields).

[0121] The data loading 2213 is configured to load the data processed by the data conversion 2212 into the index of the search engine 2311 included in the data warehouse 231. The manner of data loading 2213 is not limited in particular, and can be selected according to actual conditions. For example, the data loading can be performed in a batch loading, incremental loading or real-time loading manner.

[0122] The electronic device 230 includes the data warehouse 231, where the data warehouse 231 includes the search engine 2311. The search engine 2311 is configured to provide a retrieval service to a user, and store data after synchronization of two data tables corresponding to two databases in the business system database 210.

[0123] Based on the above Figure 2 According to the system architecture shown, the principle of synchronizing data in the two data tables corresponding to the two databases in the business system database 210 to the index field of the search engine is as follows: first, an engineer manually writes two data query statements (for example, select statements in a MySQL database) corresponding to a data synchronization request based on an interface provided by the data extraction 2211, where the two data query statements correspond one-to-one to the two data tables, and each data query statement is configured to query data in the corresponding data table. Then, the data extraction 2211 performs a query operation on the two data tables according to the plurality of data query statements to obtain two query results corresponding to the two data query statements, where each query result is obtained by performing a query operation on a data table in the corresponding database by the corresponding data query statement. Next, the data conversion 2212 obtains the two query results from the data extraction 2211, and performs a data conversion (including data filtering and data merging) operation on the two query results to obtain target data meeting the requirements of the search engine. Finally, the data loading 2213 loads the target data obtained from the data conversion 2212 into the index of the search engine 2311 in the data warehouse 231.

[0124] It can be seen that, in the process of implementing data synchronization in the conventional technology, on the one hand, the query statement used to perform data query is manually written by an engineer, and there may be an error in the written query statement, resulting in an issue that no result can be queried or the queried result is not reliable, thereby causing the data synchronization result to be inaccurate. On the other hand, after the data extraction 2211 obtains a plurality of query results (for example, the two query statements described above), the data conversion 2212 needs to perform a data conversion operation on the plurality of query results, and the execution steps of the data conversion operation are relatively complex, which causes an issue that the query efficiency is low when data synchronization is performed based on the conventional technology. In summary, when data in a database is synchronized to a search engine based on the conventional technology, there are issues that the data synchronization result is inaccurate and the data synchronization efficiency is low.

[0125] To solve the above problems in the prior art, the present application provides a data synchronization method, device and electronic equipment. Next, a schematic diagram of a system architecture suitable for the data synchronization method provided by the embodiments of the present application is introduced in conjunction with the drawings.

[0126] Figure 3 is a schematic diagram of a system architecture suitable for the data synchronization method provided by the embodiments of the present application.

[0127] As shown in Figure 3 , the system architecture includes at least one data table (for example, data table 310), a data synchronization service 320, a mirror database 330, a meta database 340 and a search engine 350.

[0128] The at least one data table is used to store data generated by a business party in the form of a two-dimensional data table. The data stored in the at least one data table is not specifically limited and can be set according to actual business needs. For example, the data recorded in the at least one data table can be data generated by a personnel management application or data generated by a project management application, etc.

[0129] The data synchronization service 320 is used to generate a data query statement and perform a query operation on the at least one data table according to the data query statement to obtain the to-be-synchronized data corresponding to the data query statement and synchronize the to-be-synchronized data to the index of the search engine 350.

[0130] For example, in some implementations, the data synchronization service 320 can provide a configuration page service to a business party server storing the at least one data table, the business party server can set data query information based on the configuration information provided by the configuration page service, and after the business party server successfully sets the data query information through the interface, the data synchronization service can generate a corresponding data query statement based on the data query information.

[0131] The mirror database 330 is used to back up the at least one data table, that is, the mirror database 330 stores the at least one data table. In the system architecture provided by the present application, by introducing the mirror database 330, frequent query operations on the at least one data table of the business party can be avoided, so that the impact on other businesses in the business party where the at least one data table is located can be avoided.

[0132] Optionally, the system architecture shown above Figure 3 may not include the mirror database 330.

[0133] In some implementations, if Figure 3 the architecture shown above includes one or more data tables, the one or more data tables are located in a database. In this implementation, Figure 3The architecture shown may or may not include mirror database 330. It is understood that... Figure 3 In the case where the architecture shown does not include the mirror database 330, the data synchronization service 320 can directly query at least one data table using the generated data query statement to obtain the data to be synchronized, and synchronize the queried data to be synchronized to the index of the search engine 350.

[0134] In some other implementations, Figure 3 The illustrated architecture includes multiple data tables, which can reside in multiple different databases. In this implementation, Figure 3 The architecture shown should include mirror database 330.

[0135] Metadatabase 340 is used to store metadata used by data synchronization service 320 when executing data synchronization methods, and to store data query statements generated by data synchronization service 320. For example, the above metadata includes, but is not limited to, the following information: database information (e.g., database name and the server where the database is located), data table information (e.g., data table name and the relationship between data table and database), linking methods, and the relationship between data tables (e.g., the course table and the tag table are associated through course identifiers; or, the course table is the main table of the tag table, and the tag table is the secondary table of the course table), etc.

[0136] Optionally, in other implementations, where the data synchronization service 320 is capable of storing metadata, the above... Figure 3 The system architecture shown may also exclude the metadata database 340.

[0137] The search engine 350 is used to store the data to be synchronized sent by the data synchronization service 320 and to provide users with retrieval and query services. In this embodiment, the search engine is not specifically limited and can be selected according to actual needs. For example, the search engine can be, but is not limited to, Elasticsearch or RediSearch.

[0138] Optionally, in some implementations, a message queue can also be set up between the data synchronization service 320 and the search engine 350. Figure 3 (Not shown in the image), the message queue is used to cache the data to be synchronized sent by the data synchronization service 320. That is, the data synchronization service 320 first temporarily stores the data to be synchronized obtained from the query into the message queue. The downstream search engine 350 can read the data to be synchronized from the message queue at its own pace. In this way, the phenomenon of the search engine 350 being blocked due to the large amount of data to be synchronized can be avoided. For example, the message queue can be a message queue like Kafka.

[0139] In the embodiments of this application, for Figure 3 The servers where the various modules shown are located are not specifically limited and can be configured according to actual business needs. In other words, the above... Figure 3 The modules shown can be logical or physical divisions, without any specific limitations.

[0140] For example, Figure 3 The multiple data tables shown can be located on the same server 1; the data synchronization service 320, the mirror database 330, and the metadata database 340 can be located on the same server 2; the search engine 350 can be located on server 3; wherein, server 3, server 2, and server 1 are three different servers.

[0141] For example, Figure 3 The multiple data tables shown can be located on multiple different servers; the data synchronization service 320, mirror database 330, metadata database 340, and search engine 350 can be located on the same server other than the multiple different servers.

[0142] It should be understood that the above Figure 3 The system architecture shown is for illustrative purposes only and does not constitute any limitation on the system architecture to which the data synchronization method provided in this application applies. For example, the above... Figure 3 The system architecture shown can also include more data tables (e.g., 3 or 5). For example, as mentioned above... Figure 3 The system architecture shown may also include a message queue, which is used to cache data sent by the data synchronization service 320 to the mirror database 330. For example, as described above... Figure 3 The mirror database 330 and the meta database 340 in the system architecture shown can be the same database.

[0143] The above text introduced Figure 3 The diagram illustrates the role of each module in the system architecture. (The above...) Figure 3 When the illustrated system architecture includes multiple data tables, these multiple data tables can reside in the same database or multiple different databases; and, some of these multiple data tables are derived from the same data table through splitting, or none of these multiple data tables are derived from the same data table through splitting. It is understood that before splitting a data table, the fields included in that data table are exactly the same as the fields included in any one of the multiple sub-data tables derived from splitting that data table, and the multiple sub-data tables jointly record the data from that original data table.

[0144] Below, based on the above Figure 3The system architecture shown includes multiple data tables. Taking the case where multiple data tables reside in a database, and whether some of these data tables are derived from splitting a single data table, this paper introduces the system architecture based on the above. Figure 3 Examples of several specific application scenarios corresponding to the system architecture are shown.

[0145] For ease of description, the following text will use the following terms: Figure 3 The multiple data tables shown are described using M data tables as an example, where M is an integer greater than 1.

[0146] In some application scenarios, the above Figure 3 The M data tables shown are located in at least one database, and none of the M data tables are derived from splitting the same business data table. In this implementation, there are join relationships between the M data tables, the join relationships between the data tables have the same column names, and the fields of any two data tables are not completely identical.

[0147] For ease of description, the scenario in this application where all M data tables are located in the same database and there are no table partitions in the M data tables is referred to as Scenario 1, that is, Scenario 1 is a single database scenario where the data tables are not partitioned.

[0148] For example, Figure 4 A schematic diagram illustrating a specific application scenario of Scenario 1 is shown. For example... Figure 4 As shown, in this scenario, both data table 1 and data table 2 are located in local database A on server 410. Server 420 includes a mirror database and a data synchronization service. Figure 4 (Not shown in the image), server 430 includes a search engine and a message queue.

[0149] The above Figure 4 The two data tables shown can be a user table and a user item table. The user table includes an identifier field and a username field, while the user item table includes a username field and an item field. That is, the user table and the user item table share the same username field. In other words, the join relationship between the user table and the user item table is represented by the username field, and the user table and the user item table are not separate tables. Optionally, the above... Figure 4 The database A shown may also include only one data table (e.g., data table 1 or data table 2).

[0150] For ease of description, the scenario in this application where the above M data tables are located in multiple databases and there are no table partitions in the M data tables is referred to as Scenario 2. That is, Scenario 2 is a scenario with multiple databases and no table partitions in the data tables.

[0151] For example, Figure 5A schematic diagram of one specific application scenario of scenario two is shown. As Figure 5 shown, data table 1 in scenario two is located in a local database A of server 510, and data table 2 is located in a local database B of server 520, where database A and database B are two different databases; server 530 includes a mirror database and a data synchronization service (not shown in the figure); and server 540 includes a search engine and a message queue. Figure 5

[0152] The above Figure 5 The examples of data table 1 and data table 2 shown can refer to the description of data table 1 and data table 2 in the above Figure 4 , and will not be described in detail here.

[0153] In some other application scenarios, the above Figure 3 The M data tables shown are located in the same database, and some of the M data tables are data tables obtained by splitting the same data table. In this implementation manner, the fields of the some of the M data tables are completely same, and the fields of the remaining data tables are not completely same, where the remaining data tables have a connection relationship, and the data tables having the connection relationship have the same column name. It can be understood that in this implementation manner, any one of the some of the M data tables is a split table, and any one of the remaining data tables is not a split table.

[0154] For ease of description, in the following of the present application, the scenario that the above M data tables are located in the same database and some of the M data tables are split tables is simply denoted as scenario three, i.e., scenario three is a single-database and split-tables scenario.

[0155] An exemplary Figure 6 A schematic diagram of one specific application scenario of scenario three is shown. As Figure 6 shown, scenario three includes three data tables, i.e., data table 1_split_0, data table 1_split_1 and data table 2, where data table 1_split_0, data table 1_split_1 and data table 2 are all located in a local database A of server 610; server 620 includes a mirror database and a data synchronization service (not shown in the figure), where the mirror database includes data table 2 and data table 1, and data table 1 is obtained by merging data table 1_split_0 and data table 1_split_1; and server 630 includes a search engine and a message queue. Figure 6

[0156] It should be understood that the above Figure 6 ​​The shown data table 1 sub-table_0 and data table 1 sub-table_1 can be two tables obtained by splitting a user table including 100 rows of data, and in the case that the user table includes an identification field and a user name field, the two data tables can be data table 1 sub-table_0 including the identification field and the user name field and data table 1 sub-table_1 including the identification field and the user name field, wherein data table 1 sub-table_0 is used to record the first 50 rows of data in the user table, and data table 1 sub-table_1 is used to record the last 50 rows of data in the user table, and data table 1 sub-table_0 and data table 1 sub-table_1 are both sub-tables.

[0157] Optionally, the above-mentioned Figure 6 The shown database A can also not include data table 2.

[0158] For ease of description, in the following of the present application, the above-mentioned scenario that M data tables are located in multiple databases and the M data tables exist sub-tables is briefly referred to as scenario four, i.e., scenario four is a multiple-database, data-table-with-sub-table scenario.

[0159] Exemplarily, Figure 7 A schematic diagram of one specific application scenario of scenario four is shown. As Figure 7 shown, the scenario four includes three data tables, i.e., data table 1 sub-table_0, data table 1 sub-table_1 and data table 2, wherein data table 1 sub-table_0 and data table 2 are both located in the local database A of server 710, data table 1 sub-table_1 is located in the local database B of server 720, and database B and database A are two different databases; server 730 includes a mirror database and a data synchronization service (not shown in the figure), wherein the mirror database includes data table 2 and data table 1, and data table 1 is obtained by merging data table 1 sub-table_0 and data table 1 sub-table_1; server 740 includes a search engine and a message queue. Figure 7

[0160] The above-mentioned Figure 7 The examples of data table 1 sub-table_0, data table 1 sub-table_1 and data table 2 shown can be referred to the description of data table 1 sub-table_0, data table 1 sub-table_1 and data table 2 in the above-mentioned Figure 6 , and will not be described in detail here.

[0161] Optionally, the above-mentioned Figure 7 The shown database A can also not include data table 2.

[0162] The above-mentioned Figures 4 to 7 The roles of the shown mirror database, message queue and search engine can be referred to the description of the corresponding modules in the above-mentioned Figure 3 , and will not be described in detail here.

[0163] It should be understood that the above-mentioned Figures 4 to 7 ​The application scenarios shown are for illustrative purposes only and do not constitute any limitation on the application scenarios to which the data synchronization method provided in this application is applicable. For example, the above Figures 4 to 7 The application scenarios shown may not include message queues. For example, the above... Figures 4 to 7 The application scenarios shown may also include the above. Figure 3 The metadata database shown.

[0164] As mentioned above, the data synchronization method provided in this application can be applied to two electronic devices. The software structure of the electronic devices will now be described in detail with reference to the accompanying drawings. It is understood that the following... Figure 8 The illustrated electronic device software architecture can be the software architecture of each of at least two electronic devices that perform the data synchronization method provided in the embodiments of this application.

[0165] For example, Figure 8 A schematic diagram of a software system for an electronic device implementing embodiments of this application is shown. See also Figure 8 The software system adopts a layered architecture. This layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer 810, the application framework layer 820, the Android Runtime and core library layer 830, the hardware abstraction layer (HAL) 840, and the kernel layer 850.

[0166] Application layer 810 may include a series of application packages. For example, application packages may include applications such as camera, gallery, chat, call, map, navigation, calendar, Bluetooth, music, video, and data synchronization services. It is understood that the data synchronization service located in application layer 810 can perform the data synchronization methods provided in this application.

[0167] The aforementioned applications may include more specific functional modules; for example, a gallery may contain business modules and notification modules.

[0168] The application data generated by the aforementioned applications can be stored in a data table in the local database of the electronic device. For example, data generated by a table application can be stored in a data table in the local database of the electronic device.

[0169] The application framework layer 820 provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer 820 includes some predefined functions.

[0170] As Figure 8 shown, the application framework layer 820 can include a window manager, a notification manager, an activity manager, an input manager, a view system, a content provider, a resource manager, etc.

[0171] The window manager provides a window manager service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for an input system.

[0172] The content provider is used to store and retrieve data, and make the data accessible to applications. The data can include videos, images, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc.

[0173] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, etc. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0174] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0175] The notification manager enables an application to display notification information in a status bar, which can be used to convey a type of message that can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify a download completion, a message reminder, etc. The notification manager can also be a notification that appears in the form of a chart or a scroll bar text in the top status bar of the system, such as a notification of an application running in the background, and can also be a notification that appears in the form of a dialog window on the screen. For example, a text information is prompted in the status bar, a prompt sound is emitted, the electronic device vibrates, the indicator light flashes, etc.

[0176] The activity manager can provide an activity manager service (AMS), which can be used for the startup, switching, scheduling of system components (such as activities, services, content providers, broadcast receivers), and the management and scheduling of application processes.

[0177] The input manager can provide an input manager service (IMS), which can be used to manage the input of the system, such as touch screen input, key input, sensor input, etc. The IMS takes events from input device nodes, and through interaction with the WMS, distributes the events to appropriate windows.

[0178] The Android Runtime includes a core library and a virtual machine. The Android Runtime is responsible for scheduling and managing the Android system.

[0179] The core library includes two parts: one part is a function function that a programming language (for example, java language) needs to call, and the other part is the core library of Android.

[0180] The application layer 810 and the application framework layer 820 run in the virtual machine. The virtual machine executes the programming files (for example, java files) of the application layer 810 and the application framework layer 820 into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, management of security and exceptions, and garbage collection.

[0181] The core library layer 830 can include a plurality of function modules. For example: surface manager, media framework, libc, SQLite, OpenGL ES, Webkit, etc.

[0182] It can be understood that the database involved in the present application can be but is not limited to being stored in the core library layer, and the binary log used to implement the data incremental synchronization can also be but is not limited to being stored in the core library layer.

[0183] The surface manager is used to manage the display subsystem, and provides a plurality of applications with fusion of two-dimensional (2-Dimensional, 2D) and three-dimensional (3-Dimensional, 3D) layers.

[0184] The media framework supports playback and recording of a plurality of commonly used audio, video formats, and static image files. The media library can support a plurality of audio and video coding formats, for example: MPEG6, H.266, MP3, AAC, AMR, JPG, PNG, etc.

[0185] The Hardware Abstraction Layer (HAL) 840 is an interface layer located between the operating system kernel and upper-level software, its purpose being to abstract hardware. The HAL is an abstract interface for device kernel drivers, providing application programming interfaces (APIs) that allow access to the underlying device to higher-level Java API frameworks. The HAL contains multiple library modules, such as the Camera HAL, Vendor repository, display, Bluetooth, and audio. Each library module implements an interface for a specific type of hardware component. For example, the Camera HAL provides the camera firmware (FWK) with an interface to access hardware components such as the camera lens. The Vendor repository provides the media firmware (FWK) with an interface to access hardware components such as the encoder. When the system framework layer API requires access to the portable device's hardware, the Android operating system loads the library module for that hardware component.

[0186] The kernel layer (850) is the foundation of the Android operating system; all the final functions of the Android operating system are implemented through the kernel layer. The kernel layer can contain display drivers, camera drivers, audio drivers, and sensor drivers.

[0187] It should be noted that the application provides Figure 8 The illustrated software architecture diagram of the electronic device is merely an example and does not limit the specific module divisions within different layers of the Android operating system. For details, please refer to the descriptions of the Android operating system software architecture in conventional technologies. Furthermore, the shooting method provided in this application can also be implemented based on other operating systems (e.g., iOS or HarmonyOS), which will not be listed here.

[0188] Next, combined Figures 9 to 17 The data synchronization method provided in the embodiments of this application will be described in detail.

[0189] Figure 9 This is a schematic diagram illustrating a data synchronization method provided in an embodiment of this application. The data synchronization method provided in this embodiment can be executed by an electronic device. It is understood that the electronic device can be implemented as software, or a combination of software and hardware. For example, the electronic device executing the data synchronization method of this embodiment may include, but is not limited to, the methods described above. Figure 3 The server shown is the data synchronization service 320.

[0190] For example, such as Figure 9 As shown, the data synchronization method provided in this application includes steps S910 to S940. Steps S910 to S940 will be described in detail below.

[0191] S910, the first electronic device obtains configuration information, wherein the configuration information includes mapping information, a filtering condition, a connection relationship of P target data tables, and that the P target data tables are stored in a target database, the mapping information includes a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, the filtering condition is used to filter a field value of a to-be-queried field corresponding to at least one target data table that meets a preset condition, and P is an integer greater than 1.

[0192] The configuration information in the above S910 step includes mapping information, a filtering condition, a connection relationship of P target data tables, and information that the P target data tables are stored in a target database. Optionally, the configuration information can also include other information, which can be configured according to a specific business scenario.

[0193] For example, the other information can include information indicating whether one of the plurality of fields in the index is a field participating in retrieval. In the case where one of the fields in the index is a field participating in retrieval, the one field can be used as a retrieval field. For another example, the other information can also be information that the target data table is split into a sub-table. For another example, in the case where a target data table exists in a sub-table, the other information can also include a name of a group, wherein the name of the group is the name of the target data table that exists in the sub-table. For another example, the other information can also include path information of the target database storing the P target data tables, wherein the path information of the target database can include a url of the target database, a username and a password for accessing the target database.

[0194] Optionally, in the case where the configuration information in the above S910 step also includes path information of the target database, before performing the above S910 step, that is, before the first electronic device performs joint querying on the P target data tables in the target database according to the query statement to obtain target data, the following step can also be performed: the first electronic device obtains the P target data tables according to the path information of the target database.

[0195] The mapping information includes a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, wherein the plurality of fields are different, and the plurality of to-be-queried fields are different. It should be understood that the field value of the field in the index that has the mapping relationship should be consistent with the field value of the to-be-queried field in the corresponding target data table. That is, in the embodiment of the present application, the purpose of data synchronization is to synchronize the field value of the plurality of to-be-queried fields in the P target data tables that meet the preset condition to the field value of the plurality of fields in the corresponding index.

[0196] The mapping information includes a mapping relationship between the plurality of fields in the index of the search engine and the plurality of to-be-queried fields corresponding to the P target data tables, and therefore, the mapping information can include an identifier of the index, an identifier of each of the P target data tables, an identifier of each of the plurality of fields in the index, an identifier of each of the plurality of to-be-queried fields, and a mapping between the plurality of fields and the plurality of to-be-queried fields. It should be understood that the identifier of the index is used to represent the index, the identifier of each of the target data tables is used to identify each of the target data tables, the identifier of each of the fields is used to identify each of the fields, and the identifier of each of the to-be-queried fields is used to identify each of the to-be-queried fields. Any of the above identifiers is not specifically limited, for example, the identifier of the index can be an index name of the index, the identifier of the target data table can be a table name of the target data table, the identifier of each of the fields can be a field name of each of the fields, and the identifier of each of the to-be-queried fields can be a field name of each of the to-be-queried fields.

[0197] The P target data tables correspond to the plurality of to-be-queried fields, where each of the P target data tables corresponds to at least one of the plurality of to-be-queried fields, and it should be understood that the to-be-queried field corresponding to each of the target data tables is a field to be queried in the target data table.

[0198] The number of the P target data tables, the number of the to-be-queried fields corresponding to each of the target data tables, the to-be-queried field corresponding to each of the target data tables, and whether the number of the plurality of fields in the index and the data amount of the plurality of to-be-queried fields corresponding to the P target data tables are equal are not specifically limited, and can be selected according to actual conditions.

[0199] In the following, the mapping relationship between the plurality of fields in the index and the plurality of to-be-queried fields corresponding to the P target data tables described by the mapping information is described by way of example, where the mapping relationship between the plurality of fields and the plurality of to-be-queried fields can be a non-conditional mapping relationship or a conditional mapping relationship, and can be set according to the conditions of the plurality of to-be-queried fields, and this is not specifically limited. It should be understood that the mapping relationship between the plurality of fields and the plurality of to-be-queried fields described by way of example in the following is only illustrative, and does not constitute any limitation on the mapping relationship between the plurality of fields and the plurality of to-be-queried fields indicated by the mapping information in the embodiments of the present application.

[0200] In some implementations, the P target data tables include a first data table and a second data table, the first data table corresponds to a first field as the to-be-queried field, the second data table corresponds to a second field and a third field as the to-be-queried fields, the plurality of fields include a fourth field, a fifth field and a sixth field, and the mapping relationship includes an unconditional mapping relationship and a conditional mapping relationship, the unconditional mapping relationship is a mapping relationship between the fourth field and the first field, the conditional mapping relationship includes a mapping relationship between the fifth field and the third field when the field value of the second field is a first value, and a mapping relationship between the sixth field and the third field when the field value of the second field is a second value, and the first value and the second value are different.

[0201] In the above implementations, the plurality of to-be-queried fields corresponding to the P target data tables include a first field, a second field and a third field.

[0202] The first data table in the above implementations can be a master table, and the second data table can be a slave table. For example, the first data table can be Table 1 below, and the second data table can be Table 2 below.

[0203] Table 1

[0204]

[0205]

[0206] The value of the status field in Table 1 above is used to indicate whether the row data where the status field is located is in an available state. For example, the value of the status field is equal to "1", indicating that the row data where the status field is located is available data, and the value of the status field is equal to "0", indicating that the row data where the status field is located is unavailable data.

[0207] Table 2

[0208] Course identifier (id) Label identifier (label_id) Content 1 49 Trigonometric functions 1 50 sine function 2 51 article 2 52 prose

[0209] In the case where the first data table in the above implementations is Table 1 above and the second data table is Table 2 above, the first field can be the id field in Table 1, the second field can be the label_id field in Table 2, and the third field can be the content field in Table 2. Based on this, for example, the fourth field above can be the id field in the index, the fifth field can be the title field in the index, and the sixth field can be the sub_title field in the index. It can be understood that the content field in Table 2 above is an ambiguous field.

[0210] Optionally, in some other implementations, the P target data tables include a first data table and a second data table, wherein the first data table corresponds to a first field as the to-be-queried field, the second data table corresponds to a second field as the to-be-queried field, the plurality of fields include a third field and a fourth field, and the mapping relationship includes a mapping relationship between the third field and the first field and a mapping relationship between the fourth field and the second field.

[0211] In the above implementations, the plurality of to-be-queried fields corresponding to the P target data tables include a first field and a second field.

[0212] In the above implementations, the mapping relationship between the third field and the first field and the mapping relationship between the fourth field and the second field are both unconditional mapping relationships, i.e., the mapping relationship between the third field and the first field exists without needing to meet a certain condition, and the mapping relationship between the fourth field and the second field exists without needing to meet a certain condition.

[0213] The filtering condition in the above S910 is used to filter the field value of the to-be-queried field corresponding to at least one target data table that meets the preset condition, i.e., the filtering condition is used to filter the field value of the to-be-queried field corresponding to one or more target data tables in the plurality of target data tables that meet the preset condition, wherein the to-be-queried field corresponding to the at least one target data table is at least one to-be-queried field in the plurality of to-be-queried fields corresponding to the P target data tables.

[0214] For example, taking the above plurality of target data tables including the above table 1 and table 2 as an example, the to-be-queried field corresponding to the table 1 is the id field, and therefore, the filtering condition can be to filter the field value of the id field in the table 1 when the field value of the status field meets the condition of being equal to "1".

[0215] In the embodiments of the present application, the preset condition in the above filtering condition is not specifically limited and can be set according to specific business query requirements.

[0216] In some implementations, the preset condition includes that the field value of the to-be-queried field corresponding to the at least one target data table exceeds a preset value, or the preset condition includes that the field value of the to-be-queried field corresponding to the at least one target data table does not exceed a preset value, or in the case that the at least one target data table further includes a non-to-be-queried field in addition to the to-be-queried field corresponding to the at least one target data table, the preset condition includes that the field value of the non-to-be-queried field is equal to a preset value. It should be understood that the non-to-be-queried field in the at least one target data table refers to a field included in the at least one target data table that is not queried in this data synchronization process.

[0217] For example, in a case where the to-be-queried field corresponding to the at least one target data table is an id field, and the preset value is 5, the preset condition includes that the field value of the id field is greater than 5.

[0218] For example, in a case where the to-be-queried field corresponding to the at least one target data table is an id field, the preset value is 1, and the non-to-be-queried field in the at least one target data table is a status field, the preset condition includes that the field value of the status field is equal to 1.

[0219] The P target data tables have a connection relationship, where the connection relationship between a data table and another data table means that the two data tables can be connected through one same field, and the connection manner in which the two data tables are connected through the one same field can be, but is not limited to, inner connection, left connection, right connection, or full connection.

[0220] In an embodiment of the present application, the connection relationship of the P target data tables is not specifically limited, and can be determined according to the fields included in each of the P target data tables.

[0221] In some implementations, there is a connection relationship between any two data tables in the P target data tables, where the any two target data tables have one same field. In this implementation, the connection relationship between the P target data tables includes the connection relationship between any two data tables in the P target data tables.

[0222] For example, the P target data tables include data table A, data table B, and data table C, where data table A and data table B have one same field 1, data table A and data table C have one same field 2, and data table B and data table C have one same field 3, and the field 1, the field 2, and the field 3 are all different. In this implementation, data table A and data table B have a connection relationship 1, data table A and data table C have a connection relationship 2, and data table B and data table C have a connection relationship 3, and the connection relationship between the three data tables includes the connection relationship 1, the connection relationship 2, and the connection relationship 3.

[0223] In another implementation, the P target data tables include data table A, data table B, and data table C, where data table A and data table B have one same field 1, data table B and data table C have one same field 2, and the field 1 and the field 2 are different. In this implementation, data table A and data table B have a connection relationship 1, and data table B and data table C have a connection relationship 2, and the connection relationship between the three data tables includes the connection relationship 1 and the connection relationship 2.

[0224] In an embodiment of the present application, the source of the P target data tables in the target database is not specifically limited.

[0225] In some implementations, the P target data tables in the target database are backed up from the K source data tables in Q databases, wherein the Q databases are different from the target database, each of the Q databases stores at least one of the K source data tables, Q is a positive integer, and K is equal to P. In this implementation, before performing the above S910, the first electronic device can further perform the following steps: generating, according to the first configuration information, K first query statements, wherein the first configuration information includes the table names of the K source data tables and the storage of the K source data tables in the Q databases; and synchronizing, according to the K first query statements, the K source data tables in the Q databases to the target database, so that the target database stores the P target data tables.

[0226] In the above implementation, the P target data tables are the K source data tables, that is, the above implementation process involves the backup (synchronization) of the K source data tables by the first electronic device, but does not involve the operation step of merging the K source data tables by the first electronic device.

[0227] In other implementations, the P target data tables include one data table obtained by merging part of the K source data tables in the Q databases, and the remaining source data tables in the K source data tables excluding the part of the source data tables, wherein any two of the part of the source data tables include the same fields, any one of the remaining source data tables includes fields that are not completely the same as any one of the part of the source data tables, K is an integer greater than P, and Q is a positive integer. In this implementation, before performing the above S910, the first electronic device can further perform the following steps: synchronizing, according to part of the K first query statements, the part of the source data tables in the at least one first database to the same data table in the second database, and synchronizing, according to the K first query statements excluding the part of the first query statements, the remaining source data tables in the at least one first database to the second database, so that the second database stores the P target data tables.

[0228] The fields included in any one of the remaining source data tables and the fields included in any one of the part of the source data tables are not completely the same, which can mean that the fields included in any one of the remaining source data tables and the fields included in any one of the part of the source data tables are completely different, or that part of the fields included in any one of the remaining source data tables and part of the fields included in any one of the part of the source data tables are the same, which is not limited in particular.

[0229] Exemplarily, the remaining source data tables in the K source data tables can be Table 2 in the above, and the partial source data tables in the K source data tables include Table 1.0 and Table 1.1 in the below.

[0230] Table 1.0

[0231] Course identifier (id) Course Name (course_name) Status 1 math 1

[0232] Table 1.1

[0233] Course identifier (id) Course Name (course_name) Status 2 Chinese 0

[0234] It can be understood that Table 1.0 and Table 1.1 in the above are data tables obtained by splitting Table 1 in the above, that is, Table 1.0 and Table 1.1 are sub-tables of Table 1 in the above. Table 1.0 and Table 1.1 in the above can be stored in the same database of the same server, different databases of the same server, or different databases of different servers, and no specific limitation is made to this.

[0235] In the above implementation manner, the partial source data tables in the K source data tables are a plurality of data tables obtained by splitting a data table, and therefore, in the process of data backup, the partial source data tables can be merged into one data table stored in the target database, so that the efficiency of subsequent query operations on the P target data tables can be improved.

[0236] Optionally, the first configuration information in the above implementation manner can further include path information of the K source data tables, for example, the path information of the K source data tables can include a path for indicating a database storing the K source data tables, a username and a password for accessing the database storing the K source data tables. For another example, the path information of the K source data tables can include a path for indicating a database storing the K source data tables.

[0237] In the above implementation manner, before the first electronic device synchronizes K (for example, K is equal to P, or K is greater than P) source data tables located in Q databases to the target database according to the K first query statements, if any data in the K source data tables is not stored in the Q databases, the synchronization process of the first electronic device for synchronizing the K source data tables located in the Q databases to the target database according to the K first query statements is a full data synchronization process.

[0238] Optionally, after the first electronic device synchronizes the K source data tables in the Q databases to the target database, if part of the data in the K source data tables in the Q databases changes, the first electronic device can further synchronize the changed part of the data in the Q databases to the target database, so that the data in the target database after re-synchronization is consistent with the data in the Q databases, thereby ensuring the reliability of the data in the data synchronization between the search engine and the databases. It can be understood that in this implementation, the process of synchronizing the changed part of the data in the Q databases to the target database is an incremental data synchronization process, and the incremental data synchronization process can be implemented according to a binary log, wherein the binary log records operation statements that cause the part of the data in the K source data tables to change, for example, the operation statements can be, but are not limited to, an insert statement, a delete statement, or an update statement.

[0239] In the embodiments of the present application, the manner in which the first electronic device obtains the configuration information described in the above S910 step is not specifically limited.

[0240] In some implementations, the above S910 step, i.e., the first electronic device obtaining the configuration information, includes that the first electronic device receives the configuration information sent by the second electronic device, so that the first electronic device obtains the configuration information.

[0241] In other implementations, the first electronic device is provided with a configuration page, and the above S910 step, i.e., the first electronic device obtaining the configuration information, includes that the first electronic device obtains the configuration information through the configuration page.

[0242] In the above implementation, in the step of the first electronic device obtaining the configuration information through the configuration page, an example includes that the first electronic device provides a configuration interface to the second electronic device, wherein the configuration interface is used by the second electronic device to configure the mapping information, the filtering condition, the connection relationship of the P target data tables, and the storage of the P target data tables in the target database; and the first electronic device obtains the configuration information according to the configuration interface successfully configured by the second electronic device. The second electronic device can be an electronic device different from the first electronic device, and the second electronic device is not specifically limited, for example, the second electronic device can be an electronic device that generates the data in the P target data tables.

[0243] Next, the configuration interface corresponding to the configuration information in the above implementation is described with reference to the accompanying drawings. It should be understood that the configuration interface described below is only illustrative and does not constitute any limitation on the embodiments of the present application.

[0244] For example, referring to Figure 10 a schematic view of the configuration page corresponding to the configuration information is shown.

[0245] In some implementations, the configuration information in step S910 above can be obtained through... Figure 10 The configuration page shown includes the mapping between the fields to be queried in the target data table and the fields in the search engine's index, the data source of the target data table (i.e., the location of the database where the target data table is stored), the join relationships between multiple target data tables, and filtering conditions. It should be understood that... Figure 10 Only the table configuration information for one of the P target data tables is shown. This can be achieved through... Figure 10 The table configuration information shown demonstrates how to flexibly add new target data using the added controls.

[0246] Optionally, the above Figure 10 The configuration page shown can also include groups, which are the names of target data tables that have sub-tables.

[0247] Optionally, the above Figure 10 The configuration information shown may also include other information, including information indicating whether a field in the search engine's index is a field to be retrieved, wherein if the field is a field to be retrieved, the field can be used as a retrieval field, and the other information also includes information that the target data table is split into sub-tables.

[0248] In the above technical solution, the first electronic device is based on the above... Figure 10 The configuration information shown in the configuration interface can automatically generate corresponding query statements. That is, the first electronic device can flexibly define data query statements by using the configuration page in combination with the automatically generated query statements. This method does not require code modification and can save development costs. In addition, since the configuration information in the configuration interface takes effect in real time, it can support business needs in a timely manner.

[0249] In some other implementations, the first configuration information in step S910 above can also be obtained through the above... Figure 10 Configure the table information shown on the configuration page. Optionally, in this implementation, Figure 10 The table information shown on the configuration page may also exclude options for the main table control.

[0250] For example, Figure 11 The above Figure 10 This is a diagram illustrating a specific configuration page. It is understood that... Figure 11 The above example uses two target data tables (the course table as the main table and the course_ext table as the secondary table) instead of the P target data tables mentioned above.

[0251] like Figure 11As shown, the configuration page includes a group, a mapping relationship between a field in each of two to-be-queried data tables and a field in an index of a search engine, a connection relationship between the two to-be-queried data tables, a filter condition, and table splitting configuration information.

[0252] Figure 11 As shown, the name of the group is course, that is, the course table is split into tables.

[0253] Figure 11 As shown, the mapping relationship between the field in each of the two to-be-queried data tables and the field in the index of the search engine includes: a non-conditional mapping relationship between an id field in an index of the search engine (the index name of the index is “10001_course_1.0”) and an id field in the course table; a non-conditional mapping relationship between the id field in the index and an id field in the course_ext table; a conditional mapping relationship between a title field in the index and a content field in the course_ext table when a field value of a label_id field in the course_ext table is 49; and a conditional mapping relationship between a sub_title field in the index and the content field in the course_ext table when the field value of the label_id field in the course_ext table is 50.

[0254] The table splitting configuration information indicates that the course table is split into two data tables, and the two data tables correspond to two data sources (that is, a data source corresponding to 0 encoding and a data source corresponding to 1 encoding). For example, the two split tables corresponding to the course table can be represented as Table 1.0 and Table 1.1 in the foregoing.

[0255] Figure 11 As shown, the connection relationship between the two to-be-queried data tables indicates that the course table and the course_ext table are connected through an id field in the course table and an id field in the course_ext table, that is, the id field in the course table and the id field in the course_ext table are equivalent.

[0256] Figure 11 As shown, the filter condition represented by the data filter is used to filter the field value of the to-be-queried field (that is, the id field) in the course table, which satisfies that the field value of the status field is equal to 1.

[0257] For example, in the configuration interface shown in FIG. 6, the course table is not split into tables (that is, the course table is not split into tables). Figure 11 As shown, the configuration interface shown in FIG. 6 includes a group, a mapping relationship between a field in each of two to-be-queried data tables and a field in an index of a search engine, a connection relationship between the two to-be-queried data tables, a filter condition, and table splitting configuration information. Figure 11In the case that the whether-to-split configuration option in the configuration page is in the unselected state and the split configuration content is empty, the course table can be the data table shown in Table 1 above, and the above Figure 11 The course_ext table in the configuration page shown can be represented as the data table shown in Table 2 above.

[0258] It should be understood that the above Figure 11 The configuration page shown in the configuration page is described by taking the case that the data table to be queried (i.e., the course table) is split into multiple data tables as an example. Alternatively, the above Figure 11 The course table shown can also be the data table shown in Table 1 above, i.e., not split into multiple data tables.

[0259] In the above Figure 10 and Figure 11 The configuration page shown in the configuration page involves the data source of the data table, and therefore, the embodiment of the present application further provides a step of configuring the data source of the data table.

[0260] In the embodiment of the present application, the data source of the data table can include a path for representing a database storing the data table, a username and a password for accessing the database storing the data table. Alternatively, the data source of the data table can further include other information, for example, the other information can but not exist be an identifier of the database storing the data table, a Chinese name of the database storing the data table, or a type of the database storing the data table.

[0261] Exemplarily, Figure 12 is a schematic diagram of a configuration interface corresponding to a data source provided by the embodiment of the present application. As Figure 12 shown, the configuration page of the data source includes a data source code, a database Chinese name, a database type, a database url, a username, a password, and a connection pool parameter.

[0262] The data source code is used to identify a database, and the data source codes corresponding to different databases are different. For example, the data source code can but not limited to be a number or a letter.

[0263] The database Chinese name is the Chinese name of the database identified by the data source code.

[0264] The database type is the type of the database identified by the data source code. For example, Figure 12 The data source type shown is a database of the MySQL type.

[0265] The database url refers to the address of the database identified by the data source code in the network.

[0266] The username is an account for accessing the database identified by the data source code, and the password is a password for accessing the database.

[0267] The connection pool parameters refer to other configuration information required for connecting the database identified by the data source code, which can be set according to actual needs, and is not specifically limited. The connection pool parameter is a non-mandatory option.

[0268] It should be understood that the above Figure 12 The content shown only includes information of one data source. Alternatively, according to actual needs, a new data source information can be added by adding a control. Figure 12 The shown adding control adds new data source information.

[0269] In the embodiments of the present application, the search engine and the target database (i.e., the relational database) in the above S910 step are not specifically limited, and can be selected according to actual needs.

[0270] In some implementations, the search engine is a distributed full-text search engine. For example, the distributed full-text search engine can be, but is not limited to, ES.

[0271] In some implementations, the target database is a relational database. For example, the target database can be, but is not limited to, any one of the following databases: MySQL, Oracle, DB2, Sybase, or PostgreSQL database.

[0272] In the embodiments of the present application, whether the electronic device where the search engine is located and the first electronic device are the same electronic device is not specifically limited, i.e., the electronic device where the search engine is located can be the first electronic device, or can be another electronic device other than the first electronic device.

[0273] S920, the first electronic device generates a query statement according to the configuration information, wherein the query statement is used to query field values of a plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields and satisfying a preset condition, and the associated data table is a data table obtained by associating the P target data tables according to a connection relationship.

[0274] In the step S920, the first electronic device generates a query statement according to the configuration information, the query statement being used to query field values of the plurality of to-be-queried fields in the associated data table corresponding to the plurality of fields and satisfying the preset condition, wherein the associated data table is a data table obtained by associating the P target data tables according to the connection relationship. It can be understood that the query statement has the functions of associating the P target data tables according to the connection relationship, mapping the plurality of fields in the index and the plurality of to-be-queried fields corresponding to the P target data tables, and querying the field values of the plurality of to-be-queried fields in the associated data table corresponding to the plurality of fields and satisfying the preset condition. That is, in the embodiment of the present application, the first electronic device generates a query statement according to the configuration information, which realizes the functions of data extraction, data mapping and data filtering, avoids the complex data conversion process based on the ETL architecture in the prior art, and reduces the development cost and maintenance cost of the system and improves the maintainability of the system.

[0275] For example, the P target data tables include the table 1 and the table 2, and the associated data table obtained by associating the table 1 and the table 2 in the step S920 can be represented as the table 3.

[0276] Table 3

[0277]

[0278] In the embodiment of the present application, the method for the first electronic device to generate a query statement according to the configuration information in the step S920 is not limited.

[0279] In some implementations, the step S920 is performed, that is, the first electronic device generates a query statement according to the configuration information, including that the first electronic device directly generates a corresponding query statement according to the configuration information.

[0280] In another implementation, the step S920 is performed, that is, the first electronic device generates a query statement according to the configuration information, including that the first electronic device selects a plurality of target generators from a plurality of candidate generators according to the configuration information, wherein the plurality of candidate generators correspond to a plurality of types of statements, each candidate generator is used to generate a statement of a corresponding type, the plurality of target generators include a first generator, a second generator and a third generator, the first generator is used to generate a statement for querying field values of a plurality of to-be-queried fields corresponding to a plurality of fields, the second generator is used to generate a statement representing a connection relationship, and the third generator is used to generate a statement representing a filtering condition; the first electronic device processes the configuration information by using the plurality of target generators to obtain a plurality of statements, and splices the plurality of statements to obtain the query statement.

[0281] In the step of processing the configuration information by the first electronic device using the plurality of target generators to obtain a plurality of sentences, and splicing the plurality of sentences to obtain the query sentence, the example includes: the first electronic device processes the P target data tables stored in the target database and the join information included in the configuration information by using a first generator to obtain a first sentence; the first electronic device processes the connection relationship in the configuration information by using a second generator to obtain a second sentence; the first electronic device processes the filter condition in the configuration information by using a third generator to obtain a third sentence; and the first electronic device splices the first sentence, the second sentence and the third sentence in sequence to obtain the query sentence.

[0282] The plurality of candidate generators in the above implementation mode are not specifically limited, and can be set or extended according to specific businesses. For example, in some implementation modes, the plurality of candidate generators in the above implementation mode can include an abstract generator and a specific generator.

[0283] The abstract generator includes a select fragment (i.e., a sentence) generator, a join on fragment generator and a where fragment generator, wherein the select fragment generator is used to generate a select fragment; the join on fragment generator is used to generate a join on fragment; and the where fragment generator is used to generate a where fragment.

[0284] The specific generator includes a simple generator and an alias generator.

[0285] The simple generator is a subclass of the select fragment generator, and the simple generator is used to perform generation of a select fragment without an alias (as), i.e., the select fragment generated by the simple generator does not include the alias as. For example, the format of the select fragment generated by the simple generator can be: select column name from table name, select case when, or select column name 1, column name 2… from table name.

[0286] The alias generator is a subclass of the select fragment generator, and the alias generator is used to perform generation of a select fragment with an alias (as), i.e., the select fragment generated by the alias generator includes the alias as. For example, the format of the select fragment generated by the alias generator can be: select column name (expression) as alias from table name.

[0287] It can be understood that the order of each generator included in the plurality of target generators corresponds to the order of splicing the plurality of sentences generated by the plurality of target generators. That is, when splicing the sentences generated by the plurality of target generators, it is necessary to splice the sentence generated by the second generator after the sentence generated by the first generator, it is necessary to splice the sentence generated by the third generator after the sentence generated by the second generator, and it is necessary to splice the sentence generated by the fourth generator after the sentence generated by the third generator.

[0288] Exemplary, Figure 13 The first electronic device in the above implementation manner generates a query sentence according to configuration information. It can be understood that, Figure 13 The SQL generation module shown can be a module in the first electronic device, for example, can be a module in a data synchronization service of the first electronic device, which is an application in the application layer of the first electronic device.

[0289] As Figure 13 As shown, the SQL generation module can include a plurality of candidate generators, a SQL generation chain builder, a SQL generation chain (i.e. the plurality of target generators above) and a SQL verifier. Optionally, Figure 13 It can also not include a SQL verifier. It should be understood that, Figure 13 The SQL generation module shown includes various modules only for illustration and does not constitute any limitation.

[0290] Next, based on the above Figure 13The SQL generation module in the illustrated modules, the SQL generation module in this application generates a query statement according to the configuration information as follows: first, the obtained configuration information is executed data parsing and verification and other processing, and the SQL generation context corresponding to the configuration information is obtained, wherein the SQL generation context contains all the information of the generated SQL (i.e. query statement), as well as the intermediate result and the final result of the generation. Then, the SQL generation chain constructor extracts n target SQL generators from a plurality of candidate generators according to the SQL generation context, and combines the n target generators to obtain the ordered arrangement of n segment generators (i.e. segment generator 1,…, segment generator n) included in the SQL generation chain, n is a positive integer. Next, each segment generator in the SQL generation chain generates a statement corresponding to each segment generator according to the SQL context, wherein each segment generator can use string splicing to generate a corresponding statement. After that, n statements generated by n segment generators are simply spliced to obtain the final SQL (i.e. query statement), wherein the order of n segment generators included in the SQL generation chain corresponds to the order of splicing the statements generated by n segment generators. Finally, the SQL generated by the SQL generation chain is checked by the SQL checker. In the case where the SQL checker determines that the SQL syntax is correct, the SQL that passes the verification is stored in the database for subsequent use. It can be understood that, in the case where the SQL generation module is included in the data synchronization service, Figure 13 The illustrated database can be a metadata database associated with the data synchronization service.

[0291] It can be understood that any one of the generators described above can be regarded as a function in the code, and the SQL generation context can be regarded as a parameter of the function, that is, the SQL generation context can be passed through function method call.

[0292] It should be noted that the types of the plurality of candidate generators illustrated in Figure 13 are only illustrative and do not constitute any limitation. Optionally, the plurality of candidate generators illustrated in Figure 13 may also be extended according to business needs. For example, the plurality of candidate generators in Figure 13 may also include a table segment generator for creating a new data table, such as the table segment for generating the following statement: create table tab_new like tab_old.

[0293] Based on the above Figure 13 illustrated architecture for generating SQL, in the case where the business requirements change, the target SQL generator can be selected more flexibly or the new SQL generator can be extended, so as to facilitate the reuse of the generator. For example, in the case where the above Figure 13 The SQL architecture shown, in the case of generating K first query statements based on the first configuration information in step S910 above, may also include only a select fragment generator, wherein the select fragment generator is used for each of the K first query statements.

[0294] The following example illustrates the configuration information and query statements in step S920 above.

[0295] For example, as described above Figure 11 Taking the configuration information set in the configuration page shown as an example, the first electronic device, according to... Figure 11 The configuration information set in the configuration page shown generates a query statement, including: A first electronic device, based on the configuration information, determines multiple target generators, including a select case when fragment generator, a join on fragment generator, and a where fragment generator; the first electronic device uses the select fragment generator to process the configuration information and generate a select case when fragment; the first electronic device uses the join on fragment generator to process the configuration information and generate a join on fragment; the first electronic device uses the where fragment generator to process the configuration information and generate a where fragment; the first electronic device sequentially concatenates the select case when fragment, join on fragment, and where fragment to obtain the final query statement, which can be as follows:

[0296]

[0297] In the above query, course is aliased as c, and course_ext is aliased as ce. In the above statement, select course.id,…case when…from course c is a select case when segment, course cleft join course_ext ce on c.id=ce.id is a join on segment, and where c.status=1 is a where segment.

[0298] For example, in some other implementations, taking the first electronic device generating K first query statements based on the first configuration information in step S910 above as an example, the following is included: when the first configuration page information includes the table name of the data table as Table 1 and the data source of Table 1, and the data table is split into two data tables (Table 1.0 and Table 1.1), the SQL generation module included in the data synchronization service can parse the configuration page information to obtain the following information:

[0299] SQL template: select id from [table];

[0300] Table name list: table 1.0, table 1.1.

[0301] The above SQL template indicates querying all records of the id column in the tables (i.e., table 1.0 and table 1.1), that is, the above SQL template and the table name list correspond to two SQL statements (i.e., SQL#1 and SQL#2), wherein SQL#1 is used to query the field value of the id field in table 1.1, and SQL#2 is used to query the field value of the id field in table 1.0. In this implementation manner, the snippet generator in the SQL generation module can only include the select generator.

[0302] S930, the first electronic device performs a joint query on P target data tables in the target database according to the query statement to obtain target data, wherein the target data includes a plurality of fields and field values of a plurality of to-be-queried fields in the associated data table that meet the preset condition.

[0303] The above S930 is executed, that is, the first electronic device only uses one query statement to implement a joint query on a plurality of data tables (i.e., P target data tables) in the target database to obtain target data including a plurality of fields in the index and field values of a plurality of to-be-queried fields in the associated data table that meet the preset condition.

[0304] For example, the P target data tables in the above S930 can be table 1 and table 2 in the above, wherein the to-be-queried field corresponding to table 1 is the id field, the to-be-queried field corresponding to table 2 includes the label_id field and the content field, and the plurality of fields in the search engine can include the id field, the title field and the sub_title field. In the case where the above preset condition is to filter the field value of the to-be-queried id field when the field value of the status field in table 1 is 1, the obtained target data can be represented as: {"id": 1, "title": trigonometric function, "sub_title": sine function}. Among them, the id field, the title field and the sub_title field in the target data are a plurality of fields in the index, and "1", "trigonometric function" and "sine function" are field values of a plurality of to-be-queried fields in the associated data table corresponding to table 1 and table 2 that meet the preset condition.

[0305] S940, the first electronic device synchronizes the target data to the index.

[0306] The target data in the step S940 includes field values of the plurality of to-be-searched fields in the association data table that satisfy the preset condition, and thus, synchronizing the target data to the index means synchronizing the field values of the plurality of to-be-searched fields in the association data table that satisfy the preset condition to the plurality of fields in the index of the search engine.

[0307] For example, the target data in the step S940 is {“id”: 1, “title”: trigonometric function, “sub_title”: sine function}, where the id field, the title field and the sub_title field are the plurality of fields in the index, and “1”, “trigonometric function” and “sine function” are the field values of the plurality of to-be-searched fields in the association data table that satisfy the preset condition. Thus, the step S940 includes: the first electronic device synchronizing the field value of the id field in the target data to the field value of the id field in the index (i.e., “1”), synchronizing the field value of the title field in the target data to the field value of the id field in the index (i.e., “trigonometric function”), and synchronizing the field value of the sub_title field in the target data to the field value of the id field in the index (i.e., “sine function”).

[0308] In the embodiments of the present application, the method of synchronizing the target data to the index described in the step S940 is not limited to the first electronic device.

[0309] In some implementations, the step S940 includes: the first electronic device sending the target data to a message queue, so as to synchronize the target data in the message queue to the index.

[0310] For example, when the message queue and the search engine are located in the first electronic device, the first electronic device sends the target data to the message queue, so as to synchronize the target data in the message queue to the index, including: a data synchronization service of the first electronic device sends the target data to the message queue; and the search engine synchronizes the target data in the message queue to the index. It can be understood that the data synchronization service of the first electronic device can perform the data synchronization method provided in the embodiments of the present application, and the data synchronization service is an application in the application layer of the first electronic device.

[0311] Exemplarily, in a case where the message queue and the search engine are located in a second electronic device different from the first electronic device, the first electronic device sends the target data to the message queue, so as to synchronize the target data located in the message queue into the index, including: the first electronic device sends the target data to the message queue of the second electronic device, so as to synchronize the target data located in the message queue into the index by the search engine.

[0312] The message queue in the above implementation manner can be, but is not limited to, a message queue kafka.

[0313] In the above implementation manner, the search engine downstream can read the target data from the message queue at its own pace after the first electronic device sends the obtained target data to the message queue first, so as to synchronize the target data located in the message queue into the index by the first electronic device, which can avoid the phenomenon of reading block of the search engine caused by too large data volume of the target data, and can improve the reading efficiency of the search engine, thereby improving the data synchronization efficiency between the search engine and the database.

[0314] In some other implementation manners, the search engine is located in a second electronic device different from the first electronic device, and the step S940 is executed, i.e., the first electronic device synchronizes the target data into the index, including: the first electronic device sends the target data to the search engine in the second electronic device, so as to synchronize the obtained target data into the index by the search engine. In this implementation manner, in a case where the search engine in the second electronic device receives the target data from the first electronic device, the search engine in the second electronic device can actively synchronize the obtained target data into the index corresponding to the search engine.

[0315] It should be understood that the above Figure 9 The data synchronization method shown is only illustrative, and does not constitute any limitation on the data synchronization method provided by the present application. For example, before the steps S910 to S940 are executed, the first electronic device can further perform the source data synchronization process described above, i.e., synchronizing K source data tables located in Q databases into the target database, so as to make the target database include P target data tables.

[0316] In the embodiment of the present application, since the configuration information includes the filtering condition, the connection relationship of the P target data tables, and the information that the P target data tables are stored in the target database, the purpose of extracting data (i.e., the P target data tables) from the data source (i.e., the target database) and filtering the data extracted from the data source according to the query requirement (i.e., querying the field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition) can be simultaneously achieved by using one query statement generated according to the configuration information. Therefore, the method can improve the efficiency of data query, thereby improving the data synchronization efficiency between the search engine and the database. In addition, since the configuration information further includes the mapping relationship (i.e., mapping information) between the plurality of fields in the index and the plurality of to-be-queried fields corresponding to the P target data tables, the query statement generated according to the configuration information is used to query the field values of the plurality of to-be-queried fields in the associated data table (i.e., the data table obtained by associating the P target data tables according to the connection relationship) that meet the preset condition. Therefore, the target data obtained by jointly querying the P target data tables by using the query statement includes the field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition and the plurality of fields in the index. Thereafter, the obtained target data can be directly synchronized to the index without the need for field mapping (e.g., mapping the fields in the database to the fields in the index) and other operations. Therefore, the method can improve the data synchronization efficiency between the search engine and the database. In addition, since the query statement is automatically generated by the first electronic device according to the configuration information, the process of generating the query statement avoids manual programming by engineers, thereby ensuring the accuracy of the query statement and improving the accuracy of the data synchronization result. In summary, based on the technical solution of the present application, the accuracy and the data synchronization efficiency of the data synchronization result between the search engine and the database can be improved.

[0317] In the following, another data synchronization method provided by the present application is described by taking the application scenario (i.e., a single database, a non-table-splitting scenario) shown in the above Figure 4 application scenario (i.e., a single database, a non-table-splitting scenario) shown in the above

[0318] Figure 14 is a schematic diagram of a data synchronization method provided by an embodiment of the present application. It can be understood that Figure 14 the data synchronization method provided by the present application is one specific example of the data synchronization method provided in the above Figure 9 application scenario (i.e., a single database, a non-table-splitting scenario) shown in the above Figure 14 application scenario (i.e., a single database, a non-table-splitting scenario) shown in the above Figure 9 application scenario (i.e., a single database, a non-table-splitting scenario) shown in the above

[0319] For example, such as Figure 14 As shown, the data synchronization method includes steps S1410 to S1460. Steps S1410 to S1460 will be described in detail below.

[0320] S1410, the data synchronization service in server B provides a configuration page to server A. Server A configures itself on the configuration page so that the data synchronization service receives configuration information 1 (i.e., the above). Figure 9 (An example of configuration information in the provided data synchronization method).

[0321] like Figure 14 As shown, server A's local database A (i.e., the one mentioned above) Figure 9 An example of the target database in the provided data synchronization method) includes multiple data tables (i.e., those mentioned above). Figure 9 An example of P target data tables in the provided data synchronization method, wherein the plurality of data tables includes table #1 and table #2, wherein table #1 may be table 1 shown above and table #2 may be table 2 shown above, that is, table #1 and table #2 are connected by the course identifier id field.

[0322] The above configuration information 1 can be the above Figure 11 The configuration page shows the content, since Table #1 and Table #2 in this embodiment of the application are not split into multiple data tables, therefore the above... Figure 11 The table partitioning configuration information shown on the configuration page should be empty (i.e., the above). Figure 11 The table partitioning configuration shown should be unselected, and the corresponding content should be empty. The `course` table has only one data source. Therefore, the configuration information 1 in step S1410 above includes mapping relationships between fields in the ES index and fields in the data table to be queried, including mapping relationship 1, mapping relationship 2, and mapping relationship 3. Mapping relationship 1 is the mapping between the `id` field in the ES index and the `id` field in table #1; mapping relationship 2 is the mapping between the `title` field in the ES index and the `content` field in table #2 when `label_id` equals 49; mapping relationship 3 is the mapping between the `sub_title` field in the ES index and the `content` field in table #2 when `label_id` equals 50.

[0323] It is understood that the id field, title field, and sub_title field in the embodiments of this application are as described above. Figure 9 An example of multiple fields in the index of the provided data synchronization method, and the aforementioned id field, label_id field, and content field are as described above. Figure 9An example of the P target data tables corresponding to the plurality of fields to be queried in the provided data synchronization method.

[0324] It can be understood that, in the case that the configuration information 1 described above can be the content shown in the configuration page described above Figure 11 , the table #1 in the embodiment of the application is the course table shown in the configuration page, and the table #2 in the embodiment of the application is the course_ext table shown in the configuration page. Figure 11 Figure 11

[0325] S1420, the data synchronization service in the server B parses the configuration information 1, and generates a query statement 1 corresponding to the configuration information 1 (i.e., an example of the query statement in the provided data synchronization method described above). Figure 9

[0326] The data synchronization service can include a SQL generation module, and the step S1420 described above is performed, i.e., the SQL generation module included in the data synchronization service parses the configuration information 1 in the configuration page, and generates the query statement 1 corresponding to the configuration information 1. It can be understood that, the working principle of the SQL generation module included in the data synchronization service for generating the query statement 1 according to the configuration information 1 is the same as the working principle of the SQL generation module in the Figure 13 above for generating the SQL according to the configuration page information, and the content not described in detail herein can be referred to the related description in the above.

[0327] Based on the configuration page corresponding to the configuration information 1 described in the step S1410 above, the query statement 1 obtained after the step S1420 described above is performed by the data synchronization service can be as follows:

[0328]

[0329]

[0330] It can be known that the query statement 1 described above combines data extraction and data filtering into one, i.e., data extraction and data filtering operations are implemented through one query statement, so that the data query efficiency can be improved, and thus the data synchronization efficiency can be improved.

[0331] Optionally, the local database of the server B can further include a meta database, and the query statement 1 obtained after the step S1620 described above is performed by the data synchronization service can be stored in the meta database. Thereafter, when the data synchronization service needs to perform a query operation by using the query statement 1, the data synchronization service can obtain the query statement 1 from the meta database.

[0332] S1430, the data synchronization service in the server B performs a joint query on the table #1 and the table #2 in the database A according to the query statement 1, and obtains the data to be synchronized (i.e., the data to be synchronized described above).​​​ Figure 9 An example of target data in the provided data synchronization method.

[0333] Based on the above query statement 1, a joint query is performed on the above table #1 and table #2, and the obtained to-be-synchronized data can be represented as:

[0334] {"id": 1, "title": "trigonometric function", "sub_title": "sine function"}.

[0335] The above to-be-synchronized data indicates that when the value of the id field in the index of the search engine obtained by the query is 1, the value of the corresponding title field is trigonometric function, and the value of the corresponding sub_title is sine function.

[0336] In S1440, the data synchronization service in server B sends the to-be-synchronized data to the message queue in server C (i.e., the above Figure 9 An example of the message queue in the provided data synchronization method.

[0337] The above message queue can be associated with the index in the ES, that is, the message queue is used to store the data in the associated index of the ES.

[0338] The data synchronization service performs the above S1440 step, that is, the data synchronization service can send the to-be-synchronized data obtained in the above S1430 after packaging to the message queue.

[0339] The message queue in server C can be located in the memory or the disk of server C, and no specific limitation is made thereto.

[0340] In S1450, the message queue in server C caches the to-be-synchronized data.

[0341] After the message queue performs the above S1450, the to-be-synchronized data in the following format is stored in the message queue:

[0342] {"id": 1, "title": "trigonometric function", "sub_title": "sine function"}.

[0343] In S1460, the ES (i.e., the above Figure 9 An example of the search engine in the provided data synchronization method) in server C obtains the to-be-synchronized data from the message queue, and stores the to-be-synchronized data into the index (i.e., the above Figure 9 An example of the index in the provided data synchronization method) of the ES.

[0344] The mapping relationship 1 between the fields in the index of the ES and the fields in the table #1 in the database A, and the mapping relationship 2 between the fields in the index of the ES and the table #2 in the configuration information 1 in the above S1410. Specifically, the mapping relationship 1 includes the mapping relationship between the id field in the index of the ES and the id field in the table #1; the mapping relationship 2 includes the mapping relationship between the title field in the index of the ES and the content field in the table #2 when the label_id is equal to 49, and the mapping relationship between the sub_title field in the index of the ES and the content field in the table #2 when the label_id is equal to 50.

[0345] Therefore, the ES stores the to-be-synchronized data into the index of the ES in the above S1460, including: the ES synchronizes the field value of the id field in the to-be-synchronized data into the field value of the id field in the index of the ES, synchronizes the field value of the title field in the to-be-synchronized data into the field value of the title field in the index of the ES, and synchronizes the field value of the sub_title field in the to-be-synchronized data into the field value of the sub_title field in the index of the ES. That is, after the above S1460 is executed, the field value of the id field in the index of the ES is 1, the field value of the title field in the index is the trigonometric function, and the field value of the sub_title field in the index is the sine function.

[0346] Optionally, after the above S1460 is executed, the message queue can also delete the to-be-synchronized data, so that the storage utilization of the message queue can be improved.

[0347] It should be understood that the above Figure 14 The data synchronization method shown in the above is only illustrative, and does not constitute any limitation on the data synchronization method provided in the present application. For example, the above ES can also be located in the server B. For example, the above message queue can also be a queue in a server D other than the server A, the server B and the server C.

[0348] In this embodiment, the server (server A), acting as the business party, first obtains configuration information 1 from the configuration page provided by the data synchronization service of server B. Next, the data synchronization service generates a data query statement (query statement 1) based on configuration information 1. This query statement simultaneously performs data extraction (reading data from the mirror database) and data filtering transformation (filtering the data read from the mirror database according to the filtering conditions in configuration information 1), combining the data extraction and filtering processes into one, thus improving data query efficiency. Subsequently, the queried data to be synchronized is synchronized to the corresponding index field of Elasticsearch, further improving data synchronization efficiency. Furthermore, during the data synchronization process, the data synchronization service directly interacts with Elasticsearch to send the data to be synchronized queried by the data synchronization service to Elasticsearch. When the amount of data to be synchronized is small, this implementation method improves data transmission efficiency, thereby further enhancing data synchronization efficiency.

[0349] Below, based on the above Figure 5 Taking the illustrated application scenario (i.e., a scenario with multiple databases and no table partitioning) as an example, this application describes another data synchronization method provided by the embodiments of this application.

[0350] Figure 15 This is a schematic diagram illustrating a data synchronization method provided in an embodiment of this application. It can be understood that... Figure 15 The data synchronization method provided is as described above. Figure 9 A specific example of the provided data synchronization method, and Figure 15 The data synchronization method shown is as described above. Figure 9 The data synchronization method provided in the example uses P target data tables in the target database, which are obtained by backing up K source data tables in Q databases. Here, P equals K, meaning that P target data tables are K source data tables. The Q databases can be databases located in the business party's electronic devices, and the data in the K source data tables can be data generated by applications on the business party's electronic devices.

[0351] For example, such as Figure 15 As shown, the method includes steps S1510 to S1591. Steps S1510 to S1591 will be described in detail below.

[0352] S1510, the data synchronization service in server B provides a configuration page to server Ai. Server Ai configures itself on the configuration page so that the data synchronization service receives the configuration information #i (i.e., the above). Figure 9 (An example of the first configuration information in the provided data synchronization method), i = 1, 2.

[0353] The to-be-queried data table #1 in the database A of the server A1 can be the table 1 in the above, and the to-be-queried data table #2 in the database B of the server A2 can be the table 2 in the above, that is, the to-be-queried data table #1 and the to-be-queried data table #2 are connected through the course identifier field.

[0354] The configuration information #i includes the configuration information #1 and the configuration information #2, wherein the configuration information #1 includes the table name of the to-be-queried table #1 and the data source of the to-be-queried table #1, and the configuration information #2 includes the table name of the to-be-queried table #2 and the data source of the to-be-queried table #2. The data source of the to-be-queried table #1 includes the url for accessing the database A of the server A1, the user and the password for accessing the database A of the server A1. The data source of the to-be-queried table #2 includes the url for accessing the database B of the server A2, the user and the password for accessing the database B of the server A2.

[0355] The above configuration information #i can be obtained by configuring the table information in the configuration page shown in the above, and Figure 10 The data source in the configuration page shown in the above can be obtained by configuring the interface shown in the above, and Figure 10 Optionally, the configuration page shown in the above can not include the main table option. Figure 11 Figure 10

[0356] In S1520, the data synchronization service in the server B parses the configuration information #i to generate the query statement #i corresponding to the configuration information #i (that is, one of the K first query statements in the data synchronization method provided in the above). Figure 9 In S1520, the data synchronization service in the server B parses the configuration information #i to generate the query statement #i corresponding to the configuration information #i (that is, one of the K first query statements in the data synchronization method provided in the above).

[0357] In S1520, the data synchronization service in the server B parses the configuration information #i to generate the query statement #i corresponding to the configuration information #i (that is, one of the K first query statements in the data synchronization method provided in the above).

[0358] It can be understood that the working principle of the SQL generation module included in the data synchronization service for generating the query statement #i according to the configuration information #i is the same as the working principle of the SQL generation module for generating the SQL according to the configuration page information in the above, and the contents not described in detail herein can be referred to the related description in the above. Figure 13

[0359] ​​​S1530, the data synchronization service in server B queries the database in server Ai according to the query statement #i to obtain table #i (i.e., the above Figure 9 An example of the K source data tables in the data synchronization method provided.

[0360] The execution of the above S1530 step, i.e., the data synchronization service in server B queries the database in server Ai according to the query statement #i to obtain table #i, includes: the data synchronization service queries the database A in server A1 according to the query statement #1 to obtain table 1; and the data synchronization service queries the database B in server A2 according to the query statement #2 to obtain table 2.

[0361] S1540, the data synchronization service in server B stores the queried table #i into the mirror database in server B (i.e., the above Figure 9 An example of the target database in the data synchronization method provided.

[0362] After the execution of the above S1510 step to S1540 step, the mirror database in server B stores the business table (i.e., table #i) in the business server (i.e., server Ai).

[0363] S1550, the data synchronization service provides a configuration page to server A2, and server A2 configures in the configuration page so that the data synchronization service obtains configuration information 1 (i.e., the above Figure 9 An example of the configuration information in the data synchronization method provided.

[0364] The difference between the configuration information 1 in the above S1550 step and the configuration information 1 in the above S1410 step is that the data source of table #i in the above S1550 is the mirror database in server B, and the user and password in the data source are the user and password for accessing the mirror database; the data source of table #i in the above S1410 step is the database A in server A, and the user and password in the data source are the user and password for accessing the database A.

[0365] In the embodiment of the present application, the server A2 can also store the mapping relationship between the to-be-queried field in table #1 in the database A in server A1 and the field in the index in ES. Based on this, after the configuration in the configuration page, the data synchronization service can obtain the configuration information 1.

[0366] The step S1550 is described by taking the server A2 as an example. Alternatively, the step S1550 can be replaced by the following steps: the data synchronization service provides a configuration page to the server A1, and the server A1 configures the configuration page to obtain the configuration information 1. In this implementation, the server A1 can also store the mapping relationship between the fields to be queried in the table #2 in the database B of the server A2 and the fields in the index in the ES.

[0367] In the step S1560, the data synchronization service in the server B parses the configuration information 1 to generate a query statement 1 (i.e., an example of the query statement in the data synchronization method provided). Figure 9

[0368] The query statement 1 in the step S1550 is used to query the table #1 and the table #2 in the mirror database in the server B, while the query statement 1 in the step S1410 is used to query the table #1 and the table #2 in the database A of the server A.

[0369] In the step S1570, the data synchronization service in the server B performs a joint query on the table #1 and the table #2 in the mirror database according to the query statement 1 to obtain the data to be synchronized (i.e., an example of the target data in the data synchronization method provided). Figure 9

[0370] The data to be synchronized in the step S1570 is the same as the data to be synchronized in the step S1430, and the working principle of the step S1570 is the same as that of the step S1430. The details of the step S1570 can be found in the description of the step S1430.

[0371] In the step S1580, the data synchronization service in the server B sends the data to be synchronized to a message queue in the server C (i.e., an example of the message queue in the data synchronization method provided). Figure 9

[0372] The working principle of the step S1580 is the same as that of the step S1440. The details of the step S1580 can be found in the description of the step S1440.

[0373] In the step S1590, the message queue in the server C buffers the data to be synchronized.

[0374] The working principle of the step S1590 is the same as that of the step S1450. The details of the step S1590 can be found in the description of the step S1450.

[0375] ​​​S1591, the ES in the server C acquires the to-be-synchronized data from the message queue, and stores the obtained to-be-synchronized data into the ES (i.e., the index of the search engine in the data synchronization method provided by the present application, as described above Figure 9 S1591, the ES in the server C acquires the to-be-synchronized data from the message queue, and stores the obtained to-be-synchronized data into the ES (i.e., the index of the search engine in the data synchronization method provided by the present application, as described above Figure 9 S1591, the ES in the server C acquires the to-be-synchronized data from the message queue, and stores the obtained to-be-synchronized data into the ES (i.e., the index of the search engine in the data synchronization method provided by the present application, as described above

[0376] The working principle of the above S1591 step is the same as that of the above S1460 step, and the content not described in detail herein can be referred to the relevant description in the above S1460 step.

[0377] It should be understood that the above Figure 15 It should be understood that the above

[0378] In the embodiment of the present application, on the one hand, the query statement #i generated based on the configuration information #i obtained from the configuration page provided by the data synchronization service of the server B can achieve the purpose of source data synchronization, that is, synchronizing the plurality of business data tables in the plurality of databases of the plurality of business servers (that is, table #1 and table #2 in the database A of the server A, and table #1 and table #2 are not split tables obtained by splitting a certain business data table, that is, the fields included in table #1 and the fields included in table #2 are not completely the same) to the mirror database of the server B, so that the frequent access of the server B to the business server (that is, the server A) can be avoided, and the influence on the business of the business server can be avoided, thereby facilitating to improve the processing performance of the business server. On the other hand, based on the configuration page provided by the data synchronization service of the server B, the data synchronization service can obtain the configuration information 1. Next, the data synchronization service generates a data query statement (that is, query statement 1) according to the configuration information 1, wherein the data query statement can simultaneously achieve the purposes of data extraction (that is, reading data from the mirror database) and data filtering and conversion (that is, filtering the data read from the mirror database according to the filtering condition in the configuration information 1), that is, combining the data extraction and data filtering and conversion processes into one, so that the data query efficiency can be improved. Thereafter, the to-be-synchronized data obtained by querying is synchronized to the fields of the index of the corresponding ES, so that the data synchronization efficiency can be improved. In addition, in the above data synchronization process, the message queue is introduced to realize the caching of the to-be-synchronized data, and in the scenario where the data amount of the to-be-synchronized data is large or the processing capability of the ES is poor, the processing pressure of the ES reading data can be relieved.

[0379] Next, taking the application scenario (that is, a single database with split tables) shown in the above Figure 6 application scenario (that is, a single database with split tables) shown in the above

[0380] Figure 16 is a schematic diagram of a data synchronization method provided by an embodiment of the present application. It can be understood that Figure 16 the data synchronization method provided by the above Figure 9 application scenario (that is, a single database with split tables) shown in the above Figure 16 application scenario (that is, a single database with split tables) shown in the above Figure 9 application scenario (that is, a single database with split tables) shown in the above

[0381] As shown in Figure 16 The method includes steps S1610 to S1690. Details of S1610 to S1690 are described below.

[0382] S1610, the data synchronization service in server B provides a configuration page to server A, and server A configures in the configuration page to make the data synchronization service obtain configuration information #i (i.e. the first configuration information in the data synchronization method provided above, i = 1.0, 1.1, 2). Figure 9 An example of the first configuration information in the data synchronization method provided above), i = 1.0, 1.1, 2.

[0383] As shown in Figure 16 In the embodiment of the present application, table #1.0 and table #1.1 in database A of server A are two sub-data tables obtained by splitting table #1, wherein the fields included in table #1.0, the fields included in table #1.1, and the fields included in table #1 are completely same, and the data in table #1 is recorded in table #1.0 and table #1.1. Figure 16 As shown in

[0384] The configuration information #i includes configuration information #1.0, configuration information #1.1, and configuration information #2, wherein the configuration information #1.0 includes the table name of the table #1.0 to be queried and the data source of the table #1.0 to be queried, the configuration information #1.1 includes the table name of the table #1.1 to be queried and the data source of the table #1.1 to be queried, and the configuration information #2 includes the table name of the table #2 to be queried and the data source of the table #2 to be queried. Since the table #1.0 to be queried, the table #1.1 to be queried, and the table #2 to be queried are all different data tables in the same database of the same server, the data source of the table #1.0 to be queried, the data source of the table #1.1 to be queried, and the data source of the table #2 to be queried are same, i.e. the data source of the table #1.0 to be queried, the data source of the table #1.1 to be queried, and the data source of the table #2 to be queried all include the path information for accessing the database A of server A, the user for accessing the database A of server A, and the password for accessing the database A of server A.

[0385] The configuration information #i above can be obtained by configuring the table information in the configuration page shown above, and Figure 10 The data source in the configuration page shown above can be obtained by configuring the interface shown above, and Figure 10 The configuration page shown above can not include the main table option. Figure 11 Figure 10

[0386] ​​S1620, the data synchronization service in server B parses the configuration information #i and generates the query statement #i corresponding to the configuration information #i (i.e., the above). Figure 9 (An example of the K first query statements in the provided data synchronization method).

[0387] Executing S1620 above, that is, the data synchronization service in server B parses the configuration information #i and generates the query statement #i corresponding to the configuration information #i, including: the SQL generation module included in the data synchronization service parses the configuration information #1.0 and generates the query statement #1.0 corresponding to the configuration information #1.0, wherein the query statement #1.0 is used to query table #1.0 in database A located on server A; the SQL generation module included in the data synchronization service parses the configuration information #1.1 and generates the query statement #1.1 corresponding to the configuration information #1.1, wherein the query statement #1.1 is used to query table #1.1 in database A located on server A; the SQL generation module included in the data synchronization service parses the configuration information #2 and generates the query statement #2 corresponding to the configuration information #2, wherein the query statement #2 is used to query table #2 in database A located on server A.

[0388] It is understandable that the SQL generation module included in the data synchronization service generates query statements based on configuration information #i, and its working principle is similar to that described above. Figure 13 The SQL generation module works in the same way as the configuration page information; for details not elaborated here, please refer to the relevant descriptions above.

[0389] Optionally, server B's local database may also include a metadata database, and the query statement #i obtained by the data synchronization service in step S1620 above can also be stored in the metadata database. It is understood that the metadata database and the mirror database are two independent databases. Subsequently, when the data synchronization service needs to perform a query operation using query statement #i, the data synchronization service can retrieve query statement #i from the metadata database.

[0390] S1630, the data synchronization service on server B synchronizes database A (i.e., the aforementioned database) on server A according to the query statement #i. Figure 9 (An example of Q databases in the provided data synchronization method) The query retrieves tables #1.0, #1.1, and #2.

[0391] The S1630 is executed, that is, the data synchronization service in the server B queries the database in the server Ai according to the query statement #i to obtain the table #i, including: the data synchronization service queries the database A in the server Ai according to the query statement #1.0 to obtain the table #1.0; the data synchronization service queries the database A in the server Ai according to the query statement #1.1 to obtain the table #1.1; and the data synchronization service queries the database A in the server Ai according to the query statement #2 to obtain the table #2.

[0392] It can be understood that the table #1.0, the table #1.1 and the table #2 in the S1630 are examples of the K source data tables in the data synchronization method provided by the application. Figure 9 The table #1.0 and the table #1.1 are examples of part of the K source data tables in the data synchronization method provided by the application. Figure 9 The table #2 is an example of the source data table in the K source data tables in the data synchronization method provided by the application, except for the part of the source data tables. Figure 9 The table #2 is an example of the source data table in the K source data tables in the data synchronization method provided by the application, except for the part of the source data tables.

[0393] The S1640 is executed, that is, the data synchronization service in the server B writes the table #1.0 and the table #1.1 into the table #1 in the mirror database in the server B, and writes the table #2 into the mirror database (that is, an example of the target database in the data synchronization method provided by the application). Figure 9 The S1640 is executed, that is, the data synchronization service in the server B writes the table #1.0 and the table #1.1 into the table #1 in the mirror database in the server B, and writes the table #2 into the mirror database (that is, an example of the target database in the data synchronization method provided by the application).

[0394] It can be understood that the table #1 and the table #2 in the S1640 are examples of the P target data tables in the data synchronization method provided by the application. Figure 9 The table #1 and the table #2 in the S1640 are examples of the P target data tables in the data synchronization method provided by the application.

[0395] After the S1610 to the S1640 are executed, the mirror database in the server B stores the business tables (that is, the table #1 and the table #2, wherein the table #1 is a table obtained by merging the table #1.0 and the table #1.1) in the business server (that is, the server Ai).

[0396] The S1650 is executed, that is, the data synchronization service in the server B provides a configuration page to the server Ai, and the server Ai configures on the configuration page to make the data synchronization service obtain the configuration information 1 (that is, an example of the configuration information in the data synchronization method provided by the application). Figure 9 The S1650 is executed, that is, the data synchronization service in the server B provides a configuration page to the server Ai, and the server Ai configures on the configuration page to make the data synchronization service obtain the configuration information 1 (that is, an example of the configuration information in the data synchronization method provided by the application).

[0397] The configuration information 1 in the step S1650 and the configuration information 1 in the step S1410 are different in that the data source of the table #i in the step S1660 is the mirror database in the server B, and the user and the password in the data source are the user and the password for accessing the mirror database; the data source of the table #i in the step S1410 is the database A of the server A, and the user and the password in the data source are the user and the password for accessing the database A.

[0398] In the step S1660, the data synchronization service in the server B parses the configuration information 1 to generate a query statement 1 (i.e., an example of the query statement in the data synchronization method provided). Figure 9

[0399] The query statement 1 in the step S1660 and the query statement 1 in the step S1420 are different in that the query statement 1 in the step S1660 is used for querying the table #1 and the table #2 in the mirror database in the server B; the table #i in the step S1410 is used for querying the table #1 and the table #2 in the database A of the server A.

[0400] The step S1660 and the step S1420 have the same working principle, and the content not described in detail here can be referred to the related description in the step S1420.

[0401] Optionally, the local database of the server B can further include a meta database, and the query statement 1 obtained by the data synchronization service in the step S1660 can be stored in the meta database. It can be understood that the meta database and the mirror database are two independent databases. Thereafter, when the data synchronization service needs to perform a query operation by using the query statement 1, the data synchronization service can obtain the query statement 1 from the meta database.

[0402] In the step S1670, the data synchronization service in the server B performs a joint query on the table #1 and the table #2 in the mirror database in the server B according to the query statement 1 to obtain target data (i.e., an example of the target data in the data synchronization method provided). Figure 9

[0403] The target data in the step S1670 and the target data in the step S1430 are the same, and the step S1670 and the step S1430 have the same working principle, and the content not described in detail here can be referred to the related description in the step S1430.

[0404] In the step S1680, the data synchronization service in the server B sends the target data to the ES.

[0405] In the step S1690, the ES stores the target data in the ES (i.e., the step S1430 in the data synchronization method provided). Figure 9 ​​The index of the search engine in the data synchronization method provided by the present application (i.e. the index of the search engine in the data synchronization method provided by the present application Figure 9 The index of the data synchronization method provided by the present application

[0406] The working principle of the data synchronization service described in the above S1690 step and the above S1460 step for storing the data to be synchronized into the index of the ES is the same, and the content not described in detail here can be referred to the related description in the S1460 step.

[0407] It should be understood that the above Figure 16 The data synchronization method shown is only illustrative and does not constitute any limitation on the data synchronization method provided by the present application. For example, the above Figure 16 The method shown can further include a message queue, based on which, after the above S1610 step to the S1670 step is executed, the data synchronization service first sends the data to be synchronized to the message queue in the server C for caching, and then the ES reads the data to be synchronized from the message queue and stores the obtained data to be synchronized into the index of the ES.

[0408] In the embodiment of the present application, on the one hand, the query statement #i generated based on the configuration information #i obtained from the configuration page provided by the data synchronization service of the server B can achieve the purpose of source data synchronization, i.e. synchronizing multiple business data tables in the local database of the same business server (i.e. table #1.0, table #1.1 and table #2 in the database A of the server A, and table #1.0 and table #1.1 are obtained by splitting the business data table 1, i.e. the fields included in table #1.0 and table #1.1 are exactly the same) to the mirror database of the server B, so that the frequent access of the server B to the business server (i.e. the server A) can be avoided, and the influence on the business of the business server can be avoided, thereby facilitating the improvement of the processing performance of the business server. On the other hand, based on the configuration page provided by the data synchronization service of the server B, the data synchronization service can obtain the configuration information 1. Next, the data synchronization service generates a data query statement (i.e. query statement 1) according to the configuration information 1, wherein the data query statement can simultaneously achieve the purposes of data extraction (i.e. reading data from the mirror database) and data filtering and conversion (i.e. filtering the data read from the mirror database according to the filtering condition in the configuration information 1), i.e. combining the data extraction and data filtering and conversion processes into one, so that the data query efficiency can be improved. Then, the data to be synchronized obtained by querying is synchronized to the fields of the index of the corresponding ES, so that the data synchronization efficiency can be improved.

[0409] Next, the above Figure 7 Taking the application scenario shown (i.e. the scenario of multiple databases and table splitting) as an example, another data synchronization method provided by the present application is described.

[0410] Figure 17 is a schematic diagram of a data synchronization method provided by an embodiment of the present application. It can be understood that, Figure 17 the data synchronization method provided by the embodiment is the data synchronization method provided by the above Figure 9 The data synchronization method provided by the embodiment is a specific example of the data synchronization method provided by the above Figure 17 The data synchronization method provided by the embodiment is the data synchronization method provided by the above Figure 9 The P target data tables in the target database in the data synchronization method provided by the above

[0411] For example, as shown in Figure 17 The method includes steps S1710 to S1791. Hereinafter, S1710 to S1791 are described in detail.

[0412] S1710, the data synchronization service in the server B provides a configuration page to the business server, and the business server configures on the configuration page to make the data synchronization service obtain configuration information #i (i.e., an example of the first configuration information in the data synchronization method provided by the above Figure 9 The server includes server A0 and server A1, and i=1.0, 1.1, 2.

[0413] As shown in Figure 17 In the embodiment of the present application, table #1.0 in the database A of server A0 and table #1.1 in the database B of server A1 are two sub-data tables obtained by splitting table #1, i.e., the fields included in table #1.0, the fields included in table #1.1 and the fields included in table #1 are exactly the same, and the data in table #1 is recorded in table #1.0 and table #1.1. Figure 17 As shown in

[0414] The configuration information #i includes configuration information #1.0, configuration information #1.1 and configuration information #2. The configuration information #1.0 includes the table name of the table #1.0 to be queried and the data source of the table #1.0 to be queried. The configuration information #1.1 includes the table name of the table #1.1 to be queried and the data source of the table #1.1 to be queried. The configuration information #2 includes the table name of the table #2 to be queried and the data source of the table #2 to be queried. Since the table #1.0 to be queried and the table #2 to be queried are two different data tables in the same database of the same server, the data source of the table #1.0 to be queried and the data source of the table #2 to be queried are the same, that is, the data source of the table #1.0 to be queried and the data source of the table #2 to be queried both include the path information of accessing the database A of the server A0, the user and the password of accessing the database A of the server A0. The data source of the table #1.1 to be queried includes the path information of accessing the database B of the server A1, the user and the password of accessing the database B of the server A1.

[0415] The configuration information #i can be obtained by configuring the table information in the configuration page shown in the above Figure 10 , and Figure 10 The data source in the configuration page shown in the above Figure 11 can be obtained by configuring the interface shown in the above Figure 10 , optionally, The configuration page shown in the above

[0416] In S1720, the data synchronization service in the server B parses the configuration information #i to generate a query statement #i (that is, one of the K first query statements in the data synchronization method provided in the above Figure 9 ).

[0417] In the above S1720, the data synchronization service in the server B parses the configuration information #i to generate the query statement #i corresponding to the configuration information #i, including: the SQL generation module included in the data synchronization service parses the configuration information #1.0 to generate the query statement #1.0 corresponding to the configuration information #1.0, wherein the query statement #1.0 is used to query the table #1.0 in the database A of the server A0; the SQL generation module included in the data synchronization service parses the configuration information #1.1 to generate the query statement #1.1 corresponding to the configuration information #1.1, wherein the query statement #1.1 is used to query the table #1.1 in the database B of the server A1; the SQL generation module included in the data synchronization service parses the configuration information #2 to generate the query statement #2 corresponding to the configuration information #2, wherein the query statement #2 is used to query the table #2 in the database A of the server A0.

[0418] It can be understood that the working principle of the SQL generation module included in the data synchronization service to generate the query statement #i according to the configuration information #i is the same as the working principle of the SQL generation module to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description. Figure 13 The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description.

[0419] S1730, the data synchronization service in the server B queries the database (i.e., one example of the Q databases in the data synchronization method provided in the foregoing description) of the service server according to the query statement #i, to obtain the table #1.0, the table #1.1 and the table #2. Figure 9 The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description.

[0420] The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description.

[0421] It can be understood that the table #1.0, the table #1.1 and the table #2 in the foregoing S1730 are one example of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 The table #1.0 and the table #1.1 are one example of the part of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 The table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description. Figure 9 The table #2 is one example of the source data table other than the part of the K source data tables in the data synchronization method provided in the foregoing description.

[0422] S1740, the data synchronization service in the server B writes the table #1.0 and the table #1.1 into the table #1 in the mirror database, and writes the table #2 into the mirror database (i.e., one example of the target database in the data synchronization method provided in the foregoing description). Figure 9 The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description.

[0423] It can be understood that the table #1 and the table #2 in the foregoing S1740 are one example of the P target data tables in the data synchronization method provided in the foregoing description. Figure 9 The working principle of the SQL generation module included in the data synchronization service to generate the SQL according to the configuration page information in the foregoing description, and the content not described in detail herein can be referred to the related description in the foregoing description.

[0424] After the foregoing S1710 to S1740 are performed, the mirror database in the server B stores the business tables (i.e., the table #1 and the table #2, wherein the table #1 is a table obtained by merging the table #1.0 and the table #1.1) in the service server.

[0425] S1750, the data synchronization service in server B provides a configuration page to server A1, and server A configures in the configuration page to make the data synchronization service get configuration information 1 (i.e. the above Figure 9 An example of the configuration information in the data synchronization method provided by the application).

[0426] The difference between the configuration information 1 in the above S1750 step and the configuration information 1 in the above S1410 step is that the data source of table #i in the above S1760 step is the mirror database in server B, and the user and password in the data source are the user and password for accessing the mirror database; the data source of table #i in the above S1410 step is the database A of server A, and the user and password in the data source are the user and password for accessing the database A.

[0427] S1760, the data synchronization service in server B parses the configuration information 1 to generate a query statement 1 (i.e. the above Figure 9 An example of the query statement in the data synchronization method provided by the application).

[0428] The difference between the query statement 1 in the above S1760 step and the query statement 1 in the above S1420 step is that the query statement 1 in the above S1760 step is used to query table #1 and table #2 in the mirror database in server B; the table #i in the above S1410 step is used to query table #1 and table #2 in the database A of server A.

[0429] The working principle of the above S1760 step and the above S1420 step is the same, and the contents not described in detail here can be referred to the related description in the above S1420 step.

[0430] S1770, the data synchronization service in server B performs a joint query on table #1 and table #2 in the mirror database according to the query statement 1 to get the to-be-synchronized data (i.e. the above Figure 9 An example of the target data in the data synchronization method provided by the application).

[0431] The to-be-synchronized data in the above S1770 step is the same as the to-be-synchronized data in the above S1430 step, and the working principle of the above S1770 step and the above S1430 step is the same, and the contents not described in detail here can be referred to the related description in the above S1430 step.

[0432] S1780, the data synchronization service in server B sends the to-be-synchronized data to a message queue (i.e. the above Figure 9 An example of the message queue in the data synchronization method provided by the application).

[0433] S1790, the message queue in the server C caches the data to be synchronized.

[0434] S1791, the ES in the server C (i.e. the search engine in the data synchronization method provided by the present application) obtains the data to be synchronized from the message queue, and stores the obtained data to be synchronized into the index (i.e. the index in the data synchronization method provided by the present application) of the ES. Figure 9 S1791, the ES in the server C (i.e. the search engine in the data synchronization method provided by the present application) obtains the data to be synchronized from the message queue, and stores the obtained data to be synchronized into the index (i.e. the index in the data synchronization method provided by the present application) of the ES. Figure 9 S1791, the ES in the server C (i.e. the search engine in the data synchronization method provided by the present application) obtains the data to be synchronized from the message queue, and stores the obtained data to be synchronized into the index (i.e. the index in the data synchronization method provided by the present application) of the ES.

[0435] The working principle of the data synchronization service described in the above S1791 step and the above S1460 step for storing the data to be synchronized into the index of the ES is the same, and the content not described in detail herein can be referred to the related description in the above S1460 step.

[0436] It should be understood that the data synchronization method shown in the above Figure 17 The difference between the data synchronization method shown in the above Figure 16 The difference between the data synchronization method shown in the above Figure 17 Q in the data synchronization method shown in the above Figure 16 Q in the data synchronization method shown in the above

[0437] It should be understood that the data synchronization method shown in the above Figure 17 The data synchronization method shown in the above Figure 17 The method shown in the above

[0438] In the embodiment of the present application, on the one hand, the query statement #i generated based on the configuration information #i obtained from the configuration page provided by the data synchronization service of the server B can achieve the purpose of source data synchronization, that is, synchronizing the plurality of business data tables in the plurality of databases of the plurality of business servers (that is, the table #1.0 and the table #2 in the database A of the server A0, and the table #1.1 in the database B of the server A1, and the table #1.0 and the table #1.1 are obtained after splitting the business data table 1, that is, the fields included in the table #1.0 and the table #1.1 are completely the same) to the mirror database of the server B, so that the frequent access of the server B to the business servers (that is, the server A0 and the server A1) can be avoided, and the influence on the business of the business servers can be avoided, thereby facilitating to improve the processing performance of the business servers. On the other hand, based on the configuration page provided by the data synchronization service of the server B, the data synchronization service can obtain the configuration information 1. Next, the data synchronization service generates a data query statement (that is, the query statement 1) according to the configuration information 1, wherein the data query statement can simultaneously achieve the purposes of data extraction (that is, reading data from the mirror database) and data filtering and conversion (that is, filtering the data read from the mirror database according to the filtering condition in the configuration information 1), that is, combining the data extraction and the data filtering and conversion into one, so that the data query efficiency can be improved. Subsequently, the to-be-synchronized data obtained by querying is synchronized to the fields of the index of the corresponding ES, so that the data synchronization efficiency can be improved.

[0439] The data synchronization method of the embodiment of the present application is described in detail above, Figures 9 to 17 The device embodiment of the present application will be described in detail below. Figure 18 and Figure 19 It should be understood that the data synchronization device in the embodiment of the present application can execute the various data synchronization methods of the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

[0440] Figure 18 FIG. 1 is a schematic diagram of a data synchronization device provided by the embodiment of the present application.

[0441] Exemplarily, Figure 18 The data synchronization device shown in the figure is applied to a first electronic device, and the data synchronization device 1800 includes a processing unit 1810, wherein the processing unit 1810 is configured to execute any one of the data synchronization methods provided by the embodiments of the present application.

[0442] It should be noted that the data synchronization device 1800 described above is in the form of a functional unit. The term “unit” herein can be implemented in the form of software and / or hardware, and no specific limitation is made thereto.

[0443] For example, the "unit" can be a software program, a hardware circuit, or a combination of both, which implements the above functions. The hardware circuit can include an application specific integrated circuit (ASIC), an electronic circuit, a processor (for example, a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combination logic circuit, and / or other suitable components that support the described functions.

[0444] Therefore, the units of each example described in the embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0445] Figure 19 FIG. 1 is a schematic diagram of a structure of an electronic device provided by the present application. Figure 19 The dashed line in FIG. 19 indicates that the unit or the module is optional. The electronic device 1900 can be used to implement the data synchronization method described in the above method embodiments.

[0446] The electronic device 1900 includes one or more processors 1901, which can support the electronic device 1900 to implement the data synchronization method in the method embodiments. The processor 1901 can be a general-purpose processor or a special-purpose processor. For example, the processor 1901 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates or transistor logic devices, or discrete hardware components.

[0447] The processor 1901 can be used to control the electronic device 1900, execute software programs, and process data of the software programs. The electronic device 1900 can further include a communication unit 1905 to implement input (reception) and output (transmission) of signals.

[0448] For example, the electronic device 1900 can be a chip, the communication unit 1905 can be an input and / or output circuit of the chip, or the communication unit 1905 can be a communication interface of the chip, and the chip can be a component of a terminal device or other electronic device.

[0449] For another example, the electronic device 1900 can be a terminal device, the communication unit 1905 can be a transceiver of the terminal device, or the communication unit 1905 can be a transceiver circuit of the terminal device.

[0450] The electronic device 1900 can include one or more memories 1902, which store programs 1904 that can be run by the processor 1901 to generate instructions 1903, so that the processor 1901 executes the data synchronization method described in the above method embodiments according to the instructions 1903.

[0451] Optionally, the memory 1902 can also store data. Optionally, the processor 1901 can also read the data stored in the memory 1902, which can be stored in the same storage address as the program 1904, or can be stored in a different storage address from the program 1904.

[0452] The processor 1901 and the memory 1902 can be separately arranged, or can be integrated together; for example, integrated on a system on chip (SOC) of a terminal device.

[0453] For example, the memory 1902 can be used to store the related program 1904 of the data synchronization method provided in the embodiments of the present application, and the processor 1901 can be used to call the related program 1904 of the data synchronization method stored in the memory 1902 to execute the data synchronization method of the embodiments of the present application.

[0454] The present application also provides a computer program product, which, when executed by a processor, implements the data synchronization method described in any of the method embodiments of the present application.

[0455] The computer program product can be stored in a memory, for example, a program, which is finally converted into an executable object file that can be executed by a processor through preprocessing, compiling, assembling, and linking processes.

[0456] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a computer, implements the data synchronization method described in any of the method embodiments of the present application. The computer program can be a high-level language program or an executable object program.

[0457] In this application, "at least one" means one or more, "multiple" means two or more. "At least one of the following (one)" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0458] It should be understood that the size of the sequence of the above processes in various embodiments of the present application does not mean the order of execution, and the execution order of the processes should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0459] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0460] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0461] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic; for example, the division of the units is only a logical function division, and actual implementation can have another division manner; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be through some interface, device or unit indirect coupling or communication connection, which can be electrical, mechanical or other forms.

[0462] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0463] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each of the units can exist physically, or two or more units can be integrated in one unit.

[0464] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data synchronization method, characterized by, Applied to the first electronic device, the method comprises: obtaining configuration information, wherein the configuration information comprises mapping information, filtering conditions, connection relationships of P target data tables, and information that the P target data tables are stored in a target database, the mapping information comprises a mapping relationship between a plurality of fields in an index of a search engine and a plurality of to-be-queried fields corresponding to the P target data tables, the filtering conditions are used to filter field values of the to-be-queried fields corresponding to at least one target data table that meets a preset condition, and P is an integer greater than 1; generating a query statement according to the configuration information, wherein the query statement is used to query field values of the plurality of to-be-queried fields in an associated data table corresponding to the plurality of fields that meet the preset condition, and the associated data table is a data table obtained by associating the P target data tables according to the connection relationships; performing a joint query on the P target data tables in the target database according to the query statement to obtain target data, wherein the target data comprises the plurality of fields and the field values of the plurality of to-be-queried fields in the associated data table that meet the preset condition; synchronizing the target data to the index; wherein the P target data tables comprise a first data table and a second data table, wherein the to-be-queried field corresponding to the first data table is a first field, the to-be-queried field corresponding to the second data table comprises a second field and a third field, the plurality of fields comprise a fourth field, a fifth field and a sixth field, and the mapping relationship comprises a non-conditional mapping relationship and a conditional mapping relationship, wherein the non-conditional mapping relationship is a mapping relationship between the fourth field and the first field, the conditional mapping relationship comprises a mapping relationship between the fifth field and the third field when the field value of the second field is a first value, and a mapping relationship between the sixth field and the third field when the field value of the second field is a second value, and the first value and the second value are different.

2. The method of claim 1, wherein, The configuration information further comprises path information of the target database, and before the joint query on the P target data tables in the target database according to the query statement to obtain target data, the method further comprises: obtaining the P target data tables according to the path information of the target database.

3. The method according to claim 1 or 2, characterized in that, The generating of the query statement according to the configuration information comprises: selecting a plurality of target generators from a plurality of candidate generators according to the configuration information, wherein the plurality of candidate generators correspond to a plurality of types of statements, each candidate generator is used to generate a statement of a corresponding type, the plurality of target generators comprise a first generator, a second generator and a third generator, the first generator is used to generate a statement that queries field values of the plurality of to-be-queried fields that have the mapping relationship with the plurality of fields, the second generator is used to generate a statement that represents the connection relationship, and the third generator is used to generate a statement that represents the filtering conditions. The configuration information is processed by the plurality of target generators to obtain a plurality of sentences, and the plurality of sentences are spliced to obtain the query sentence.

4. The method according to claim 1 or 2, characterized in that, The synchronizing of the target data to the index comprises: The target data is sent to a message queue, so that the target data in the message queue is synchronized to the index.

5. The method according to claim 1 or 2, characterized in that, The search engine is located in a second electronic device different from the first electronic device, and The synchronizing of the target data to the index comprises: The target data is sent to the search engine in the second electronic device, so that the search engine synchronizes the obtained target data to the index.

6. The method of claim 1 or 2, wherein the preset condition comprises that a field value of a to-be-queried field corresponding to the at least one target data table exceeds a preset value; or The preset condition comprises that a field value of a to-be-queried field corresponding to the at least one target data table does not exceed the preset value; or In a case where the at least one target data table further comprises a non-to-be-queried field other than the to-be-queried field corresponding to the at least one target data table, the preset condition comprises that a field value of the non-to-be-queried field is equal to the preset value. The first electronic device is provided with a configuration page, and 7. The method according to claim 1 or 2, characterized in that, The obtaining of the configuration information comprises: The configuration information is obtained through the configuration page. The search engine is a distributed full-text search engine.

8. The method of claim 1 or 2, wherein, The electronic device comprises a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the processor executes the data synchronization method in any one of claims 1 to 8.

9. An electronic device, comprising: The processor executes the data synchronization method in any one of claims 1 to 8 when executing instructions.

10. A chip, characterized by The computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the processor executes the data synchronization method in any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, ​

Images