Search engine updating method and apparatus, device, medium, and product

Abstract

The application discloses a search engine updating method and device, equipment, medium and product, and the method comprises the following steps: determining the updated data record in the slave table referenced by the master table according to the log file of the relational database, and obtaining the incremental data corresponding to the data record from the slave table; storing the incremental data to the storage space corresponding to the distributed database, so as to convert the incremental data into the key-value pair storage form, trigger the data updating event, and the key name of each key-value pair of the same incremental data comprises the pointer identifier pointing to the slave table data record; in response to the data updating event, extracting the key-value pair corresponding to the pointer identifier from the distributed database, and converting the key-value pair into the incremental data expression in the preset protocol format; and outputting the incremental data expression to the data processing interface of the search engine. The application flattens the updating data of the relational database and then stores the updating data into the distributed database, so that the data competition problem existing in the multi-table data updating of the search system is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database dump, in particular to a search engine updating method, and further relates to a device, equipment, nonvolatile storage medium and computer program product corresponding to the method. BACKGROUND

[0002] In large e-commerce businesses, many complex search scenarios are often faced, and special search engines such as Elasticsearch, solr, etc. are needed. Business data exists in a relational database. Therefore, it is necessary to realize real-time synchronization of business data from the relational database to the search engine. Due to the complexity of search business, the search scenario often needs to join a main table and several sub-tables to form a wide table to provide search online services for the search engine. The association relationship of the main table and the sub-tables is one-to-one, one-to-many and many-to-one respectively. In order to ensure the real-time nature of the data, the log files of the relational database can be parsed to obtain the newly added, deleted or updated data of the main table and the sub-tables, and then the incremental data is joined, cleaned, aggregated and calculated to flow down to the search engine. Due to the complexity of the join relationship and the need to use full data in many calculation processes, the existing technology generally selects HBase to store full historical data and assist in join operation, but the existing technology has the following problems:

[0003] Firstly, when the main table and the sub-tables are in one-to-many relationship, one record in the main table corresponds to multiple records in the sub-tables. The method is to store the records of the sub-tables in the main table after merging. Therefore, when processing the incremental data of the sub-tables, the historical data needs to be read first, and then updated and inserted. This is prone to data competition problems between incremental data updated at different times, such as A data arrives first, reads the historical record to prepare to write, and then the system starts to process B data. After B data is merged with the historical data and written, A data is written. At this time, B data may be overwritten by A data, so that new data cannot be updated to the search engine.

[0004] Secondly, since HBase itself does not have a binlog similar to a relational database, the downstream search engine cannot perceive the real-time message changes in HBase. The search engine needs to automatically detect whether the data stored in the distributed database is updated, which affects the data synchronization and update efficiency of the search system.

[0005] In view of the above problems, the present application makes corresponding exploration to solve the problems. SUMMARY

[0006] The application aims to meet the user needs and provide a search engine updating method, and further relates to the corresponding device, equipment, nonvolatile storage medium and computer program product.

[0007] To achieve the purpose of the application, the following technical solutions are adopted:

[0008] A search engine updating method proposed to achieve the purpose of the application includes the following steps:

[0009] According to the log file of the relational database, determine the updated data record in the slave table referenced by the master table, and obtain the incremental data corresponding to the updated data record from the slave table;

[0010] Store the incremental data to the storage space corresponding to the distributed database, thereby converting the incremental data into a key-value pair storage form, triggering a data update event, and the key name of each key-value pair of the same incremental data contains a pointer identifier pointing to the data record of the slave table;

[0011] In response to the data update event, extract the key-value pair corresponding to the pointer identifier from the distributed database, and convert it into an incremental data expression in a preset protocol format;

[0012] Output the incremental data expression to the data processing interface of the search engine for the data processing interface to parse and read the incremental data expression.

[0013] In further embodiments, storing the incremental data to the storage space corresponding to the distributed database, thereby converting the incremental data into a key-value pair storage form, triggering a data update event, includes the following steps:

[0014] Convert each field in the incremental data into a key-value pair, wherein the key name of each key-value pair contains the corresponding field name and the pointer identifier, the pointer identifier contains the table name of the slave table and the unique index value of the data record, and the value domain of each key-value pair is used to store the data of the corresponding field;

[0015] Store all key-value pairs corresponding to the incremental data to the storage space preset for the relational database by the distributed database;

[0016] Trigger a data update event after completing the storage, and the unique index value is contained in the message body of the event.

[0017] In further embodiments, in response to the data update event, extract the key-value pair corresponding to the pointer identifier from the distributed database, and convert it into an incremental data expression in a preset protocol format, including the following steps:

[0018] extracting a unique index value in a message body of the event in response to the data update event;

[0019] reading all key-value pairs corresponding to the unique index value from the storage space of the distributed database;

[0020] converting all the read key-value pairs into an incremental data expression in a preset protocol format, wherein the incremental data expression includes the unique index value, and field names and value domain data corresponding to each key-value pair.

[0021] In a further embodiment, the incremental data expression is output to a data processing interface of a search engine, so that the data processing interface parses and reads the incremental data expression, including the following steps:

[0022] The search engine acquires the incremental data expression through the data processing interface, and parses and determines that the data expression includes the unique index value;

[0023] The search engine inserts the value domain data into a search database thereof after processing according to the unique index value and the field names corresponding to each key-value pair in the data expression.

[0024] In a preferred embodiment, the method includes the following pre-step:

[0025] detecting a conversion switch of a relational database, and when the conversion switch is in an open state, performing conversion storage processing on incremental data of the relational database.

[0026] In a preferred embodiment, the distributed database is constructed based on an HBase database.

[0027] A search engine updating device is provided for the purpose of the present application, and includes:

[0028] an incremental data acquisition module, configured to determine data records updated in a slave table referenced by a master table according to a log file of a relational database, and acquire incremental data corresponding to the updated data records from the slave table;

[0029] an incremental data conversion storage module, configured to convert the incremental data to a storage space corresponding to a distributed database, so as to convert the incremental data into a key-value pair storage form, trigger a data update event, and the key names of each key-value pair of the same incremental data all include a pointer identifier pointing to a data record of the slave table;

[0030] an update event response module, configured to extract key-value pairs corresponding to the pointer identifier from the distributed database in response to the data update event, and convert the key-value pairs into an incremental data expression in a preset protocol format;

[0031] The expression output module is used to output the incremental data expression to the data processing interface of the search engine so that the data processing interface can parse and read the incremental data expression.

[0032] In a further embodiment, the incremental data transfer module includes:

[0033] A field conversion submodule is configured to convert each field in the incremental data into a key-value pair, wherein the key name of each key-value pair includes the corresponding field name and the pointer identifier, the pointer identifier includes the table name of the slave table and the unique index value of the data record, and the value domain of each key-value pair is used to store the data of the corresponding field;

[0034] A key-value pair storage submodule, configured to store all key-value pairs corresponding to the incremental data into a storage space preset by the distributed database for the relational database;

[0035] The update event triggering submodule is used to trigger a data update event after the storage is completed, and the message body of the event includes the unique index value.

[0036] In a further embodiment, the update event response module includes:

[0037] An index value extraction submodule, configured to respond to the data update event and extract a unique index value from the message body of the event;

[0038] A key-value pair reading submodule, configured to read all key-value pairs corresponding to the unique index value from the storage space of the distributed database;

[0039] Key-value pair conversion submodule. It is used to convert all the key-value pairs read out into incremental data expressions in a preset protocol format. The incremental data expression includes the unique index value, and the field name and value domain data corresponding to each key-value pair.

[0040] In a further embodiment, the expression output module includes:

[0041] An index value determination submodule, configured for the search engine to obtain a search increment data expression through a data processing interface, and to parse and determine whether the data expression contains the unique index value;

[0042] The key-value pair insertion submodule is used for the search engine to process the value domain data according to the unique index value and the field name corresponding to each key-value pair in the data expression and then insert it into its own search database.

[0043] To solve the above technical problems, the embodiment of the present application further provides a computer device, comprising a memory and a processor, the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to make the processor execute the steps of the search engine updating method.

[0044] To solve the above technical problems, the embodiment of the present application further provides a storage medium storing computer readable instructions, and the computer readable instructions are executed by one or more processors to make the one or more processors execute the steps of the search engine updating method.

[0045] To solve the above technical problems, the embodiment of the present application further provides a computer program product, comprising a computer program and computer instructions, and the computer program and computer instructions are executed by a processor to make the processor execute the steps of the search engine updating method.

[0046] Compared with the prior art, the advantages of the present application are as follows:

[0047] The present application solves the data competition problem of storing global historical data in the relational database into the distributed database by performing flat processing on the incremental data of the slave table in the relational database and storing it into the distributed database for search engine data synchronization update. Specifically, the incremental data of the slave table is converted into a key-value pair, and the key-value pair is stored in the storage space corresponding to the primary key of the slave table in the distributed database by referring to the master-slave key relationship between the master table and the slave table in the relational database, so as to realize data flattening by utilizing the columnar storage feature of the distributed database, and avoid the data competition problem existing in data update.

[0048] In addition, after the incremental data in the relational database is stored into the distributed database, the present application triggers a data update event to make the distributed database have the function of notifying the downstream search engine to perform data synchronization update, thereby saving the computing power spent by the search engine for detecting the distributed database, and effectively improving the efficiency of data synchronization update of the whole search system. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0050] Figure 1 A typical network deployment architecture diagram for implementing the technical solutions of the present application;

[0051] Figure 2 A flowchart of a typical embodiment of the search engine updating method of the present application;

[0052] Figure 3 A flowchart diagram formed for the specific embodiment of the present application about incremental data dump to distributed database;

[0053] Figure 4 A flowchart diagram formed for the specific embodiment of the present application about converting key-value pair to incremental data expression in preset protocol format;

[0054] Figure 5 A flowchart diagram formed for the specific embodiment of the present application about data processing interface of search engine to parse and read the incremental data expression;

[0055] Figure 6 A principle block diagram of a typical embodiment of the search engine updating device of the present application;

[0056] Figure 7 A basic structure block diagram of a computer device of an embodiment of the present application. DETAILED DESCRIPTION

[0057] Embodiments of the present application are described in detail below with reference to the attached drawing figures, wherein the embodiments of the application are shown by way of examples. The embodiments of the application described below are exemplary, and are not to be construed as limiting the present application.

[0058] Those skilled in the art can understand that the singular forms "a," "an," and "the" used herein include plural references unless expressly stated to the contrary. It should be further understood that the word "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements, and / or components listed are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intervening elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any one of the associated listed items, and all combinations thereof.

[0059] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with meanings in the context of the present technology, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0060] Those skilled in the art will appreciate that the term "client", "terminal", "terminal device" as used herein encompasses both devices that are solely wireless signal receivers, devices that are solely wireless signal receivers without transmission capability, and devices that have both receiving and transmitting hardware enabling two-way communications over a two-way communications link. Such devices can include cellular or other communication devices with or without a single-line or multiple-line display capabilities, PCS (Personal Communications Service) devices that can combine a voice, data processing, facsimile, and / or data communications capabilities, PDA (Personal Digital Assistant) devices that can include a radio frequency receiver, a pager, Internet / Intranet access, a Web browser, a calendar, a to-do list, and / or a GPS (Global Positioning System) receiver, a conventional laptop and / or palmtop computer or other devices that have a radio frequency receiver, and / or that have and / or include a radio frequency receiver. The term "client", "terminal", "terminal device" as used herein can be portable, transportable, installed in a vehicle (aeronautical, maritime, and / or land-based), or adapted and / or configured for local and / or distributed operation on Earth and / or in any other location in space. The term "client", "terminal", "terminal device" as used herein can also be a communication terminal, an Internet terminal, a music / video playing terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playing capabilities, a smart television, a set-top box, and / or the like.

[0061] The term "server", "client", "service node", and the like as used herein refers to hardware that essentially has the equivalent capability of a personal computer, i.e., a hardware device having a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device, and the like necessary components disclosed by the von Neumann principle, a computer program is stored in the memory, the central processing unit calls the program stored in the external memory into the memory for running, executes the instructions in the program, and interacts with the input and output devices, thereby completing a specific function.

[0062] It should be noted that the concept of "server" in the present application can also be extended to the case of a server cluster. According to the principle of network deployment understood by those skilled in the art, the servers should be logically divided, and in physical space, these servers can be independent but can be called through an interface, or can be integrated into a physical computer or a computer cluster. Those skilled in the art should understand this variation and should not be restricted by the implementation of the network deployment of the present application.

[0063] Referring to Figure 1 The hardware foundation required for the implementation of the technical solutions related to the present application can be deployed according to the architecture shown in the figure. The server 80 in the present application is deployed in the cloud and serves as a business server, which can be responsible for further connecting related data servers and other servers that provide related support, etc., to form a logically associated service cluster to provide services for related terminal devices such as the smart phone 81 and the personal computer 82 shown in the figure or a third-party server (not shown). The smart phone and the personal computer can access the Internet through a known network access method and establish a data communication link with the server 80 in the cloud to run terminal application programs related to the services provided by the server.

[0064] For the server, the application program is usually built as a service process to open a corresponding program interface for remote calling by application programs running on various terminal devices. The technical solutions related to the present application suitable for running on the server can be implemented in the server in this way.

[0065] The application program refers to an application program running on a server or a terminal device. This application program implements the technical solutions related to the present application in a programming manner, and its program code can be saved in a computer-recognizable non-volatile storage medium in the form of computer executable instructions and be called into the memory by the central processing unit for running. The related device of the present application is constructed through the running of the application program on the computer.

[0066] For the server, the application program is usually built as a service process to open a corresponding program interface for remote calling by application programs running on various terminal devices. The technical solutions related to the present application suitable for running on the server can be implemented in the server in this way.

[0067] Those skilled in the art should know that the various methods of the present application, although based on the same concept and common to each other, are independently executable unless otherwise specified. Similarly, each embodiment disclosed in the present application is based on the same inventive concept, and therefore, the same concept and appropriately modified concepts for convenience, although the concepts are different, should be understood as equivalent.

[0068] Referring to Figure 2 A search engine updating method of the present application, in a typical embodiment, includes the following steps:

[0069] In step S11, the data record updated in the slave table referenced by the master table is determined according to the log file of the relational database, and the incremental data corresponding to the updated data record is obtained from the slave table:

[0070] The server queries the slave table referenced by the master table in which the data is updated, according to the data record updated in the log file of the relational database, to obtain the incremental data corresponding to the data record from the slave table.

[0071] The relational database refers to a database using a relational model to organize data, which generally stores a plurality of master tables and slave tables referenced by the master tables, i.e., the master tables and the slave tables have a reference and referenced relationship, and the reference relationship between the master tables and the slave tables is organized by a relational module. The relational database can be constructed using Oracle, DB2, MySQL, Microsoft SQL Server, Microsoft Access, and other relational databases.

[0072] When the data stored in the slave table in the relational database is updated, the data record representing the updated data of the slave table is generated, and the data record is stored in the log file, so that the server obtains the incremental data of the slave table through the log file.

[0073] The server queries the slave table pointed by the latest data record in the log file from the relational database to determine the update data pointed by the data record from the slave table, generates the incremental data containing the update data and the field of the column name in the slave table, and it is noted that the incremental data in this application is the data refined to a row of data in the slave table, as shown in the following table. When the commodity slave table A has data update, the purchase price of the commodity ID 124 is modified from 42 to 52, and the selling price is modified from 70 to 80, the incremental data is the purchase price 52 of the commodity ID 124 in the commodity slave table A and the selling price 80 of the commodity ID 124 in the commodity slave table A:

[0074] Item ID Purchase price Selling price 124 52 80 127 32 50

[0075] In step S12, the incremental data is stored in the storage space corresponding to the distributed database, so as to convert the incremental data into a key-value pair storage form, trigger a data update event, and the key name of each key-value pair of the same incremental data contains a pointer identifier pointing to the data record in the slave table:

[0076] After the server obtains the incremental data in the slave table, the incremental data is stored in the corresponding storage control in the distributed database, so as to convert the incremental data into a key-value pair data format for storage.

[0077] The distributed database refers to a database constructed based on a distributed file system, and generally refers to a database constructed based on HBase, which has distributed and column-oriented characteristics, and is applied to a system requiring real-time read and write and random access to a large-scale data set, such as the search engine described in this application. The incremental data is stored in the distributed database, so as to subsequently and efficiently insert the incremental data into the search engine, and facilitate the search engine to provide online services for corresponding businesses.

[0078] Specifically, when the server stores the incremental data in the distributed database, the incremental data is converted in storage form. Each field contained in the incremental data is converted into a key-value pair, the key name of each key-value pair contains the field name of the corresponding field and the pointer identifier, the pointer identifier contains the table name of the slave table and the unique index value of the data record, and the value domain of each key-value pair is used to store the data of the corresponding field. The unique index value refers to the index value of the primary key-slave key of the incremental data in the master table and the slave table.

[0079] Further, after the server converts the incremental data into a plurality of key-value pairs, the server stores all the key-value pairs into a corresponding storage space of the distributed database. The storage space is generally a storage space preset for a master table of a slave table according to the relational database in the distributed database. That is, the storage space of the relational database in the distributed database stores data of all slave tables referenced by the relational database and data stored in the relational database. The server determines a master table of a slave table to which the incremental data belongs through the unique index value corresponding to the incremental data, so as to determine a row storage space (i.e., the storage space) of the key-value pairs of the incremental data in the HBase database. It should be noted that the unique index value is not a row key in the HBase database. The server determines a row key (rowkey) of the master table associated with the unique index value in the HBase database. The master table is a master table of a slave table to which the unique index value belongs. When the fields of the incremental data are converted into the key-value pairs, the key values of the key-value pairs are column names of the row storage space in the HBase database, and the value domains of the key-value pairs are stored in column storage spaces of the column names. The column names and the data stored in the column storage spaces form the fields of the incremental data converted into the key-value pairs.

[0080] Specifically, when the distributed database is a database constructed based on an HBase database, the server determines a row key (rowkey) of the incremental data in the HBase database through the unique index value corresponding to the incremental data, so as to determine a row storage space (i.e., the storage space) of the key-value pairs of the incremental data in the HBase database. It should be noted that the unique index value is not a row key in the HBase database. The server determines a row key (rowkey) of the master table associated with the unique index value in the HBase database. The master table is a master table of a slave table to which the unique index value belongs. When the fields of the incremental data are converted into the key-value pairs, the key values of the key-value pairs are column names of the row storage space in the HBase database, and the value domains of the key-value pairs are stored in column storage spaces of the column names. The column names and the data stored in the column storage spaces form the fields of the incremental data converted into the key-value pairs.

[0081] It can be understood that, by converting the incremental data into the key-value pairs, the fields of the incremental data are refined to the updated data under a certain data referenced by the master table in the slave table, which can exclude the problem of old data covering new data caused by data competition when all the updated data of the slave table is combined and stored into the relational database.

[0082] When the server stores the key-value pairs of the incremental data into the corresponding storage space of the distributed database, the data update event containing the unique index value is triggered, so that the key-value pairs corresponding to the pointer identifier are extracted from the distributed database, and the incremental data expression in the preset protocol format is converted.

[0083] Step S13, in response to the data update event, extracting the pointer-identified corresponding key-value pair from the distributed database, and converting it into an incremental data expression in a preset protocol format:

[0084] In response to the data update event triggered by the server storing the key-value pair into the corresponding storage space in the distributed database, the server determines the storage space in the distributed database for incremental data conversion according to the unique index value contained in the data update event, extracts all key-value pairs in this storage space, i.e. extracts all key-value pairs in the master table of the slave table to which the incremental data belongs in the distributed database, and these key-value pairs contain all key-value pairs converted by the incremental data, i.e. the pointer-identified corresponding key-value pairs, to convert these key-value pairs into the incremental data expression in the preset protocol format.

[0085] The preset protocol format is preset for converting the key-value pair into the incremental data expression called by the search engine, specifically, the server extracts all key-value pairs corresponding to the pointer identifier from the distributed database, and generates the incremental data expression containing the unique index value and the field name and value domain data corresponding to these key-value pairs according to the preset protocol format, for example, when the distributed database is an HBase database, the server determines the row storage space corresponding to the row key from the HBase database, extracts all key-value pairs in the row storage space, and determines the field name in the incremental data through the column name of these key-value pairs, which is the column name of the key-value pair in the slave table to which the value domain data belongs, and further generates the incremental data expression containing the unique index value and the field name and value domain data of each key-value pair; further, when the server extracts the key-value pair with a unique index value of A from the row storage space in the HBase database, i.e. "table1_field1_A=value1" and "table1_field2_A=value2", the incremental data expression generated according to the preset protocol format is [{A, feild1:value1, feild2:value2}].

[0086] Step S14, outputting the incremental data expression to the data processing interface of the search engine for the data processing interface to parse and read the incremental data expression:

[0087] The server converts the extracted key-value pairs in the distributed database into the incremental data expression in response to the data update event, and outputs the incremental data expression to a data processing interface of the search engine to drive the data processing interface to parse and read the incremental data expression, so as to ensure the synchronization update of data in the search engine.

[0088] The business data formed by the relational database and the distributed database is decoupled from the business logic of the online service of the search engine. The server outputs the incremental data expression to the data processing interface of the search engine to drive the data processing interface to parse and read the incremental data expression, generates search data required by the search engine to provide corresponding online services, and stores the search data into a corresponding search database, so as to complete the synchronization update of data in the search engine. For example, when the incremental data expression contains the price and name of a product, the server outputs the incremental data expression to the data processing interface to drive the data processing interface to determine the corresponding product in the search database of the search engine through the unique index value in the expression, and update the price and name in the search database to the price and name of the product contained in the incremental data expression, thereby synchronously updating the online service of the price statistical data of the product type to which the product belongs in the search engine.

[0089] It can be understood that, by splitting and storing the incremental data of the slave table in the relational database into the distributed database, and generating and transmitting the data expression in response to the trigger of the data update event, the downstream search engine is notified to perform data synchronization update, which eliminates the data competition problem caused by storing all data of the master table and its associated slave table into the distributed database. Through the trigger response of the data update event, the business data storage system decoupled from the online service system of the search engine has the function of actively notifying the search engine to perform data synchronization update, and the search engine online service system does not need to monitor the incremental data event of the business data storage system at regular intervals or in real time, which effectively improves the data synchronization update efficiency of the overall system.

[0090] The above typical embodiments and their variant embodiments fully disclose the implementation of the search engine update method of the present application, but various variant embodiments of the method can be deduced by transforming and expanding some technical means, as summarized below:

[0091] In one embodiment, please refer to Figure 3 The server stores the incremental data into the corresponding storage space of the distributed database, thereby converting the incremental data into a key-value pair storage form, and triggers the data update event, and the specific implementation includes the following specific steps:

[0092] Step S121, converting each field in the incremental data into a key-value pair respectively, wherein the key name of each key-value pair contains the corresponding field name and the pointer identifier, and the value domain of each key-value pair is used to store the data of the corresponding field.

[0093] After the server obtains the incremental data, it converts each field contained in the incremental data into a corresponding key-value pair respectively, wherein the key name of the key-value pair contains the field name of the field and the pointer identifier, and the key domain of the key-value pair is the data corresponding to the field. For example, when the field of the incremental data represents that the price of trousers A in the slave table A is updated to 50, the key-value pair of the field is the key name slave table A_price_trousers A and the key domain is 50, wherein the pointer identifier contained in the key name of the key-value pair is the slave table A_trousers A.

[0094] Step S122, storing all key-value pairs corresponding to the incremental data into the storage space preset for the relational database in the distributed database:

[0095] After the server converts all fields contained in the incremental data into the key-value pairs respectively, it stores these key-value pairs into the storage space preset for the relational database in the distributed database. The preset storage space is generally the storage space preset for the master table in the relational database. The server will query the storage space of all key-value pairs of the incremental data in the distributed database according to the master key-foreign key association rule between the master table referring to the incremental data and the slave table to which the incremental data belongs. For example, when the distributed database is a database constructed based on HBase, the server determines the row key of the incremental data in the distributed database according to the master table referring to the slave table to which the incremental data belongs, i.e. the master key-foreign key association rule between the master table and the slave table, so as to store all key-value pairs corresponding to the incremental data into the row storage space corresponding to the row key in the distributed database.

[0096] Further, when the key-value pair is the key name table1_field1_A and the value domain is value1, and the row key of the row storage space of the master table referring to table1 in the distributed database is 102, the schematic table of the distributed database is as follows:

[0097]

[0098] The key-value pair filedA=A and filedB=B in the row key 102 of the row storage space in the table is the data stored by the master table corresponding to the row key 102 in the relational database itself, that is, if the data of the master table in the relational database is updated, the updated data is directly inserted into the storage in the distributed database.

[0099] After the server determines the storage space of the incremental data in the distributed database, all key-value pairs of the incremental data are inserted into the storage space for storage.

[0100] In step S123, a data update event is triggered after the storage is completed, and the unique index value is included in the message body of the event.

[0101] After the server stores all key-value pairs of the incremental data in the storage space in the distributed database, the data update event is triggered, and the unique index value and the location identifier (i.e., the row key in HBase) of the storage space are included in the message body of the event. In response to the data update event, all key-value pairs in the storage space are extracted for data expression conversion.

[0102] In this embodiment, all fields of the incremental data from the table are converted into key-value pairs and stored in the distributed database, so as to refine the incremental data from the table, and data from the table is flattened, the processing logic of the relational database for converting data to the distributed database is optimized, and the data competition problem caused by the traditional storage of the incremental data from the table in the relational database into the distributed database of the master table is excluded.

[0103] In one embodiment, referring to Figure 4 In response to the data update event, the server extracts the key-value pair corresponding to the pointer identifier from the distributed database, and converts it into an incremental data expression in a preset protocol format. The specific implementation includes the following specific steps.

[0104] In step S131, in response to the data update event, the unique index value in the message body of the event is extracted.

[0105] In response to the data update event, the server extracts the unique index value in the message body of the event and the location identifier of the storage space included in the message body, so as to determine the key-value pair corresponding to the unique index value in the location identifier pointed by the location identifier in the distributed database, and construct an incremental event expression.

[0106] In step S132, all key-value pairs corresponding to the unique index value are read from the storage space of the distributed database.

[0107] After the server determines the storage space of the distributed database, it reads all key-value pairs in the storage space, which include the key-value pairs of the primary table corresponding to the storage space and all key-value pairs of the incremental data corresponding to the unique index value.

[0108] In step S133, all read key-value pairs are converted into an incremental data expression in a preset protocol format, which includes the unique index value, and the field name and value domain data corresponding to each key-value pair.

[0109] The server converts all key-value pairs of the incremental data corresponding to the unique index value read from the storage space into corresponding incremental data expressions according to the preset protocol format, and restores the key-value pairs of the primary table corresponding to the storage space. For example, when the storage structure of the storage space is as shown in the following table:

[0110]

[0111] The key-value pairs of the storage space are extracted as follows:

[0112] [fieldA:A,fieldB:B,table1_field1_A:value1,table1_field2_A:Value2]

[0113] Wherein, the unique index value of the key-value pairs "table1_field1_A:value1" and "table1_field2_A:Value" is A, the field names are field1 and field2 respectively, and the value domain data are vlaue1 and vlu2 respectively. The incremental data expression converted according to the preset protocol format is:

[0114] [{102:A,filed1:value1,table1_field2_A:value2}]

[0115] In this embodiment, the distributed database has the function of automatically sensing data update by responding to the data update event, so as to convert the key-value pairs of the incremental data into an incremental data expression in advance, thereby improving the synchronization update efficiency of the search engine.

[0116] In one embodiment, please refer to Figure 5 , the data processing interface of the server outputting the incremental data expression to the search engine, and the specific implementation of the search engine reading the incremental data expression, which includes the following specific steps:

[0117] Step S141, the search engine acquires the search incremental data expression through the data processing interface, and determines the unique index value contained in the data expression:

[0118] After the server outputs the incremental data expression to the data processing interface of the search engine, the search engine parses the incremental data expression to determine the unique index value contained in the expression.

[0119] Step S142, the search engine inserts the value domain data into the search database according to the unique index value and the field name corresponding to each key-value pair in the data expression:

[0120] The search engine determines the search data associated with the incremental data expression according to the unique index value, and inserts the processing result into the search database after data processing according to the field name and value domain data of each key-value pair contained in the data expression. For example, when the unique index value in the incremental data expression is product A, the search engine determines the search data of product A, such as the price of the product type to which product A belongs, the size of the product type to which product A belongs, and the value domain data of the field name product price (i.e. the price of product A in the slave table of the relational database changes), and re-counts the price of the product type to which product A belongs to complete the processing, and inserts the processing result (i.e. the price of the product type to which product A belongs) into the search database.

[0121] In this embodiment, the transmission of the incremental data expression through the data processing interface makes the business data formed by the relational database and the distributed database and the business logic of the online service of the search engine decoupled.

[0122] In one embodiment, before the server performs the incremental data conversion processing, the following pre-step is further included:

[0123] Step S10, detecting the conversion switch of the relational database, when it is in the open state, the incremental data of the relational database will be converted:

[0124] When the server performs the incremental data conversion of the relational database, it will detect the state of the conversion switch of the relational database. When it is in the open state, the data records of the log file in the relational database will be converted from the slave table to the distributed database. When it is in the closed state, the above conversion processing will not be performed.

[0125] In the embodiment, the conversion switch is set for the relational database to control the conversion processing of the incremental data of the relational database of the server, so that the conversion processing of the incremental data is not triggered when the data debugging is performed on the relational database, the data in the distributed database and the search engine is affected, and the data of the whole system is chaotic.

[0126] Further, each step in the method disclosed in the above embodiments can be functionalized to construct a search engine updating device of the present application. According to this idea, please refer to Figure 6 In a typical embodiment, the device includes: an incremental data acquisition module 11, configured to determine the updated data record in the slave table referenced by the master table according to the log file of the relational database, and acquire the incremental data corresponding to the updated data record from the slave table; an incremental data conversion module 12, configured to convert the incremental data to the storage space corresponding to the distributed database, so as to convert the incremental data into a key-value pair storage form, trigger a data update event, and the key name of each key-value pair of the same incremental data contains a pointer identifier pointing to the data record of the slave table; an update event response module 13, configured to respond to the data update event, extract the key-value pair corresponding to the pointer identifier from the distributed database, and convert it into an incremental data expression in a preset protocol format; and an expression output module 14, configured to output the incremental data expression to a data processing interface of a search engine for the data processing interface to parse and read the incremental data expression.

[0127] In an embodiment, the incremental data conversion module 12 includes: a field conversion submodule, configured to convert each field in the incremental data into a key-value pair, wherein the key name of each key-value pair contains a corresponding field name and the pointer identifier, the pointer identifier contains the table name of the slave table and the unique index value of the data record, and the value domain of each key-value pair is used to store the data of the corresponding field; a key-value pair storage submodule, configured to store all key-value pairs corresponding to the incremental data to the storage space preset for the relational database by the distributed database; and an update event triggering submodule, configured to trigger a data update event after the storage is completed, and the unique index value is contained in the message body of the event.

[0128] In one embodiment, the update event response module 13 includes: an index value extraction submodule, used to respond to the data update event and extract the unique index value in the message body of the event; a key-value pair reading submodule, used to read all key-value pairs corresponding to the unique index value from the storage space of the distributed database; a key-value pair conversion submodule. It is used to convert all the read key-value pairs into incremental data expressions in a preset protocol format, and the incremental data expression includes the unique index value, as well as the field name and value domain data corresponding to each key-value pair.

[0129] In one embodiment, the expression output module 14 includes: an index value determination submodule, which is used by the search engine to obtain the search incremental data expression through the data processing interface, and parse to determine whether the data expression contains the unique index value; a key-value pair insertion submodule, which is used by the search engine to process the value domain data according to the unique index value and the field name corresponding to each key-value pair in the data expression, and then insert it into its own search database.

[0130] In order to solve the above technical problems, the embodiment of the present application further provides a computer device for running a computer program implemented according to the search engine updating method. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0131] like Figure 7 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a search engine update method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a search engine update method. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] The processor in this embodiment is configured to execute the specific functions of each module / submodule in the search engine updating apparatus of the present application, and the memory stores the program codes and various data required for executing the above modules. The network interface is configured to transmit data between the user terminal and the server. The memory in this embodiment stores the program codes and data required for executing all the modules / submodules in the search engine updating apparatus, and the server can call the program codes and data of the server to execute the functions of all the submodules.

[0133] The present application also provides a non-volatile storage medium, in which the search engine updating method is written as a computer program and stored in the form of computer readable instructions. When the computer readable instructions are executed by one or more processors, it means that the program is running in the computer, thereby causing the one or more processors to execute the steps of the search engine updating method of any of the above embodiments.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0135] In summary, the present application solves the data competition problem of storing global historical data in a relational database into a distributed database by converting the incremental data of the table in the relational database into flat data and storing it into the distributed database for search engine data synchronization update.

[0136] It should be understood that although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0137] Those skilled in the art can understand that the steps, measures and schemes in various operations, methods and processes discussed in the present application can be alternated, changed, combined or deleted. Further, other steps, measures and schemes in various operations, methods and processes discussed in the present application can also be alternated, changed, rearranged, decomposed, combined or deleted. Further, the steps, measures and schemes in various operations, methods and processes in the prior art can also be alternated, changed, rearranged, decomposed, combined or deleted.

[0138] The above only describes some embodiments of the present application. It should be pointed out that those skilled in the art can make some improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A search engine updating method, characterized in that: The steps include: Detecting a switch of a relational database, and when the switch is in an on state, transferring incremental data of the relational database; Determining updated data records in a slave table referenced by a master table according to a log file of the relational database, and obtaining incremental data corresponding to the updated data records from the slave table; The incremental data is transferred to the storage space corresponding to the distributed database, thereby converting the incremental data into a key-value pair storage form, triggering a data update event, and the key name of each key-value pair of the same incremental data contains a pointer identifier pointing to the slave table data record, including: converting each field in the incremental data into a key-value pair, wherein the key name of each key-value pair contains the corresponding field name and the pointer identifier, the pointer identifier contains the table name of the slave table and the unique index value of the data record, and the value domain of each key-value pair is used to store the data of the corresponding field; storing all key-value pairs corresponding to the incremental data into the storage space preset by the distributed database for the relational database; triggering a data update event after the storage is completed, and including the unique index value in the message body of the event; In response to the data update event, extract the key-value pair corresponding to the pointer identifier from the distributed database and convert it into an incremental data expression in a preset protocol format; The incremental data expression is output to the data processing interface of the search engine so that the data processing interface can parse and read the incremental data expression.

2. The method according to claim 1, characterized in that In response to the data update event, extracting the key-value pair corresponding to the pointer identifier from the distributed database and converting it into an incremental data expression in a preset protocol format includes the following steps: In response to the data update event, extract a unique index value in the message body of the event; Reading all key-value pairs corresponding to the unique index value from the storage space of the distributed database; All key-value pairs read out are converted into incremental data expressions in a preset protocol format, wherein the incremental data expressions include the unique index value, and the field name and value range data corresponding to each key-value pair.

3. The method according to claim 1, characterized in that Outputting the incremental data expression to a data processing interface of a search engine so that the data processing interface can parse and read the incremental data expression includes the following steps: The search engine obtains the search increment data expression through the data processing interface, and parses and determines that the data expression contains the unique index value; The search engine processes the value domain data according to the unique index value and the field name corresponding to each key-value pair in the data expression, and then inserts the value domain data into its own search database accordingly.

4. The method according to claim 1, wherein The distributed database is built based on the HBase database.

5. A search engine updating device, characterized in that: It includes: The incremental data acquisition module is used to detect the conversion switch of the relational database, and when it is in the on state, the incremental data of the relational database will be transferred and processed; Determining updated data records in a slave table referenced by a master table according to a log file of the relational database, and obtaining incremental data corresponding to the updated data records from the slave table; An incremental data transfer module is used to transfer the incremental data to the storage space corresponding to the distributed database, thereby converting the incremental data into a key-value pair storage format and triggering a data update event. The key name of each key-value pair of the same incremental data contains a pointer identifier pointing to the slave table data record, including: converting each field in the incremental data into a key-value pair, wherein the key name of each key-value pair contains the corresponding field name and the pointer identifier, the pointer identifier contains the table name of the slave table and the unique index value of the data record, and the value domain of each key-value pair is used to store the data of the corresponding field; storing all key-value pairs corresponding to the incremental data into the storage space preset by the distributed database for the relational database; triggering a data update event after the storage is completed, and including the unique index value in the message body of the event; An update event response module, configured to respond to the data update event, extract the key-value pair corresponding to the pointer identifier from the distributed database, and convert it into an incremental data expression in a preset protocol format; The expression output module is used to output the incremental data expression to the data processing interface of the search engine so that the data processing interface can parse and read the incremental data expression.

6. The device according to claim 5, characterized in that The update event response module includes: An index value extraction submodule, configured to respond to the data update event and extract a unique index value from the message body of the event; A key-value pair reading submodule, configured to read all key-value pairs corresponding to the unique index value from the storage space of the distributed database; Key-value pair conversion submodule. It is used to convert all the key-value pairs read out into incremental data expressions in a preset protocol format. The incremental data expression includes the unique index value, and the field name and value domain data corresponding to each key-value pair.

7. The device according to claim 5, characterized in that The expression output module includes: An index value determination submodule, configured for the search engine to obtain a search increment data expression through a data processing interface, and to parse and determine whether the data expression contains the unique index value; The key-value pair insertion submodule is used for the search engine to process the value domain data according to the unique index value and the field name corresponding to each key-value pair in the data expression and then insert it into its own search database.

8. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 4.

9. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 4 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images