Method and system for updating data of multiple indexes, electronic device, storage medium

Abstract

The application provides a method and system for updating multiple indexes, an electronic device and a storage medium, and the method comprises the following steps: an index updating node of an index updating system submits a constructed target data document to a search engine node of the index updating system; wherein the target data document comprises an index identifier, and the index identifier comprises a document identifier and a global identifier; the search engine node searches a local data document according to the document identifier, and if the local data document is found, it is judged whether the global identifier of the target data document is greater than the global identifier of the local data document, and if yes, the local data document is updated with the target data document. According to the scheme, the global identifier is constructed, the phenomenon that the data is covered and repeatedly updated in the updating process is effectively avoided, and resource waste is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and system for multi-path index updating data, electronic equipment and computer readable storage medium. BACKGROUND

[0002] A search system generally extracts and arranges business data and stores the data into a search engine to provide data retrieval services. The search system generally uses multi-path indexes to synchronize data to the search engine, wherein the multi-path indexes include an incremental index line, a full index line and a real-time index line. When there are multiple business scenarios for an index library, each scenario needs to include the full, incremental and real-time index lines.

[0003] In the scenario of multi-path indexes, when multiple lines update the same document at the same time, due to the inconsistency in the way of querying data, constructing documents and indexing data and the time consumed by each line, the constructed documents will have old and new data. The existing technical solutions to avoid the old data from covering the new data mainly include: using the timestamp of data change as a version number, repeatedly updating data, and transferring the change ID extracted by the real-time index line to the incremental line to update again.

[0004] However, the above three technical solutions have certain defects, which will cause inaccurate data, repeated update of changed data, excessive service pressure, double pressure on the search engine, and serious resource waste problems. SUMMARY

[0005] Embodiments of the present application provide a method for updating data of multi-path indexes to solve the problems of old data covering new data, repeated update of data, high pressure on the search engine and resource waste.

[0006] Embodiments of the present application provide a method for updating data of multi-path indexes, applied to an index updating system, the index updating system including a plurality of index updating nodes and a search engine node, comprising:

[0007] The index updating node submits a constructed target data document to the search engine node; wherein the target data document includes an index identifier, and the index identifier includes a document identifier and a global identifier;

[0008] The search engine node searches a local data document according to the document identifier, and if the local data document is found, judges whether the global identifier of the target data document is greater than the global identifier of the local data document, and if yes, updates the local data document with the target data document.

[0009] In an embodiment, the method further comprises:

[0010] If the local data document corresponding to the document identifier is not found, the search engine node stores the target data document.

[0011] In an embodiment, the method further comprises:

[0012] If the global identifier of the target data document is not greater than the global identifier of the local data document, the local data document is kept.

[0013] In an embodiment, the index updating system comprises an extraction node corresponding to each index updating node;

[0014] Before the built target data document is submitted to the search engine node, the method further comprises:

[0015] When the changed data exists in the target data source, the extraction node acquires the document identifier and the global identifier of the changed data, and constructs an index identifier based on the document identifier and the global identifier, and sends the index identifier to the corresponding index updating node;

[0016] The index updating node builds a target data document according to the index identifier.

[0017] In an embodiment, the index updating system comprises an identifier allocation node;

[0018] Before the document identifier and the global identifier of the changed data are acquired, the method further comprises:

[0019] The identifier allocation node allocates a unique global identifier to the changed data; wherein the global identifier is incremented one by one.

[0020] In an embodiment, the extraction node determines that the data in the target data source is changed according to the incremental detection or real-time detection.

[0021] In an embodiment, before the built target data document is submitted to the search engine node, the method further comprises:

[0022] The extraction node acquires the document identifier and the global identifier of all data in the target data source, and constructs an index identifier based on the document identifier and the global identifier, and sends the index identifier to the corresponding index updating node.

[0023] Embodiments of the present application also provide a system for updating multiple indexes, comprising a plurality of index updating nodes and a search engine node, comprising:

[0024] A document submission module is configured to submit the built target data document to the search engine node by the index updating node, wherein the target data document comprises an index identifier, and the index identifier comprises a document identifier and a global identifier.

[0025] a searching module, configured to search the local data document according to the document identifier of the target data document;

[0026] a judging module, configured to, if the searching is successful, judge whether the global identifier of the target data document is greater than the global identifier of the local data document, and if yes, update the local data document with the target data document.

[0027] The embodiment of the present application further provides an electronic device, which comprises:

[0028] a processor;

[0029] a memory for storing processor-executable instructions;

[0030] The processor is configured to execute any one of the methods for updating data with multiple indexes.

[0031] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executable by a processor to execute any one of the methods for updating data with multiple indexes.

[0032] The technical scheme provided by the above-mentioned embodiment of the present application is that the index updating node of the index updating system submits the constructed target data document to the searching engine node of the index updating system, the searching engine node searches the local data document according to the document identifier of the target data document, and determines whether the local data document is updated according to the global identifier size of the target data document and the local data document, thereby effectively avoiding the phenomenon that new data is covered by old data and data is repeatedly updated, and reducing resource waste. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical scheme of the embodiment of the present application, the drawings needed to be used in the embodiment of the present application will be briefly introduced.

[0034] Figure 1 The application scenario diagram of the method for updating data with multiple indexes provided by the embodiment of the present application;

[0035] Figure 2 The structural schematic diagram of the electronic device provided by the embodiment of the present application;

[0036] Figure 3 The flow schematic diagram of the method for updating data with multiple indexes provided by the embodiment of the present application;

[0037] Figure 4 The flow schematic diagram of the method for constructing a target data document provided by the embodiment of the present application;

[0038] Figure 5 This is a block diagram of a system for updating data via a multi-way index, as provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0040] Similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0041] Figure 1 This diagram illustrates an application scenario for the multi-way index update data method provided in this application embodiment. For example... Figure 1 As shown, the target data source obtains data through the application layer and is detected by multiple extraction nodes, including incremental extraction nodes and real-time extraction nodes. When an extraction node detects a data change in the target data source, the changed data is assigned a unique global identifier. The extraction node obtains the document identifier and global identifier of the changed data, constructs an index identifier, and sends it to the corresponding index update node, which is a node in the index cluster. The index update node constructs and submits the target data document to the search engine node based on the index identifier.

[0042] The search engine node locates the local data document based on the document identifier of the target data document, and determines whether the local data document has been updated based on the global identifier size of the target data document and the local data document.

[0043] Figure 2 This application provides an embodiment of an electronic device 2, which includes at least one processor 21 and a memory 22. Figure 2 Taking a processor as an example, processor 21 and memory 22 are connected via bus 20. Memory 22 stores instructions that can be executed by processor 21, and the instructions are executed by processor 21. Processor 21 is configured to execute the method for updating data via multiplexing provided in this embodiment.

[0044] The processor 21 may be a device that includes a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities. It can process data from other components in the electronic device 2 and control other components in the electronic device 2 to perform desired functions.

[0045] The memory 22 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache memory, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage medium, and the processor 21 can execute the program instructions to implement the method of updating multi-path index data described below. Various application programs and various data, such as various data used and / or generated by the application programs, and the like, can also be stored in the computer-readable storage medium.

[0046] Figure 2 The components and structures of the electronic device 2 shown are only exemplary and not limiting, and the electronic device 2 can also have other components and structures according to needs in an embodiment.

[0047] In an embodiment, an example electronic device 2 for implementing the method of updating multi-path index data of the embodiments of the present application can be implemented as a smart phone, a tablet computer, a desktop computer, a notebook computer, a vehicle terminal, and the like.

[0048] Figure 3 A flowchart of a method of updating multi-path index data according to an embodiment of the present application is shown. As shown, the method can be executed by Figure 3 the electronic device 2 shown, applied to an index updating system to implement the method of updating multi-path index data, and the method includes the following steps S310-S320. Figure 2

[0049] Step S310: The index updating node submits the constructed target data document to the search engine node; wherein the target data document includes an index identifier, and the index identifier includes a document identifier and a global identifier.

[0050] In this step, the index updating node of the index updating system submits the constructed target data document to the search engine node of the index updating system. The target data document includes an index identifier, and the index identifier includes a document identifier and a global identifier. The document identifier is the identity information of the data document, which can be a document ID; the global identifier is the update version information of the data document, which can be a global ID; and the index identifier is the identity information and update version information of the data document, which can be an index ID.

[0051] Step S320: The search engine node searches for a local data document according to the document identifier, and if found, judges whether the global identifier of the target data document is greater than the global identifier of the local data document, and if so, updates the local data document with the target data document.​

[0052] In this step, the search engine node finds the corresponding local data document according to the document identification in the target data document, that is, finds the local data document containing the same document identification as the target data document. For example, if the document ID of the target data document is 1, the search engine node finds the local data document with the document ID of 1.

[0053] If the local data document containing the same document identification as the target data document is found, the global identification of the target data document and the local data document is compared to determine whether the global identification of the target data document is greater than the global identification of the local data document. If the global identification of the target data document is greater than the global identification of the local data document, the local data document is updated with the target data document. For example, if the global ID of the target data document is 104 and the global ID of the local data document is 103, the local data document is replaced with the target data document.

[0054] If the local data document containing the same document identification as the target data document is not found, the search engine node stores the target data document.

[0055] In another embodiment, if the global identification of the target data document is not greater than the global identification of the local data document, the search engine node retains the local data document. For example, if the global ID of the target data document is 105 and the global ID of the local data document is 106, the local data document is retained.

[0056] In an embodiment, before step S310, the method provided by the embodiment of the application further includes steps S410-S440, as shown in the following table. Figure 4

[0057] Step S410: The extraction node determines that there is data change in the target data source according to the incremental detection or real-time detection.

[0058] In this step, the extraction node detects the data of the target data source, and the detection method includes incremental detection and real-time detection. When the incremental detection or real-time detection detects the data change of the target data source, the extraction node can determine that there is data change in the target data source. The incremental detection method is to obtain the changed data every time unit, and the real-time detection method is to obtain the changed data in real time through the log.

[0059] Step S420: When there is data change in the target data source, the identification allocation node allocates a unique global identification to the changed data; wherein the global identification is incremented one by one.

[0060] ​In this step, when there is data change in the target data source, the identification allocation node of the index updating system allocates a unique global identification to the updated data, and the global identification is incremented one by one. As a unique global ID is allocated to the updated data, the global ID needs to be incremented one by one in the subsequent data updating process according to 101, 102, 103.

[0061] Step S430: The extraction node acquires the document identification and the global identification of the changed data, and constructs an index identification based on the document identification and the global identification, and sends to the corresponding index updating node.

[0062] In this step, after the identification allocation node of the index updating system allocates a unique global identification to the updated data, the extraction node acquires the document identification and the global identification of the changed data. Then the document identification and the global identification of the changed data are combined to construct an index identification, and the index identification is sent to the corresponding index updating node of the index updating system. When the document identification and the global identification are document ID and global ID respectively, the extraction node acquires the document ID and the global ID of the changed data, and combines the document ID and the global ID into an index ID, and sends the index ID to the corresponding index updating node of the index updating system.

[0063] In another embodiment, the extraction node acquires the document identification and the global identification of all data in the target data source, and then combines the document identification and the global identification of all data to construct an index identification respectively, and sends the index identification to the corresponding index updating node of the index updating system.

[0064] Step S440: The index updating node constructs the target data document according to the index identification.

[0065] In this step, the index updating node constructs the target data document according to the document identification and the global identification in the index identification. The index updating node can construct the target data document according to the document ID and the global ID in the index ID.

[0066] The following is an embodiment of the system of the present application, which can be used to execute the method embodiment of the above multi-path index updating data of the present application. For details not disclosed in the system embodiment of the present application, please refer to the method embodiment of the multi-path index updating data of the present application.

[0067] Figure 5 A block diagram of a multi-path index updating data system according to an embodiment of the present application is shown in FIG. 5. As shown in FIG. 5, the system includes a document submission module 510, a search module 520, and a judgment module 530. Figure 5

[0068] ​The document submitting module 510 is configured to submit the target data document built by the index updating node to the search engine node, wherein the target data document comprises an index identifier, and the index identifier comprises a document identifier and a global identifier.

[0069] The searching module 520 is configured to search the local data document according to the document identifier.

[0070] The judging module 530 is configured to, if the target data document is found, judge whether the global identifier of the target data document is greater than the global identifier of the local data document, and if yes, update the local data document with the target data document.

[0071] The functions and effects of the modules in the system are described in detail in the implementation process of the corresponding steps in the method for updating multiple indexes, which will not be repeated here.

[0072] In the several embodiments provided in the present application, the disclosed system and method can also be implemented in other ways. The system embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operation of the system, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts and block diagrams can represent a module, a segment or a portion of code, which contains one or more executable instructions for implementing the specified logic function. In some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0073] In addition, the functional modules in each embodiment of the present application can be integrated together to form a separate part, or each module can exist independently, or two or more modules can be integrated to form a separate part.

[0074] If the functions are implemented in the form of software function modules and sold or used as independent products, the functions can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A method for multi-path index update data, applied to an index update system, the index update system comprising a plurality of index update nodes and a search engine node, each index update node corresponding to an extraction node and an identity allocation node, characterized in that, The method comprises the following steps: The index updating node submits the built target data document to the search engine node, wherein the target data document comprises an index identifier, and the index identifier comprises a document identifier and a global identifier; The search engine node searches for a local data document according to the document identifier, and if the local data document is found, judges whether the global identifier of the target data document is greater than the global identifier of the local data document, and if yes, updates the local data document with the target data document; Before the step of submitting the built target data document to the search engine node, the method further comprises the following steps: The extraction node acquires the document identifier and the global identifier of the changed data when the target data source has changed data, Before the step of acquiring the document identifier and the global identifier of the changed data, the method further comprises the following step: The identifier allocation node allocates a unique global identifier to the changed data; wherein the global identifier is incremented one by one; The document identifier is identity information of the data document, which is a document ID; the global identifier is update version information of the data document, which is a global ID; and the index identifier is the identity information and the update version information of the data document, which is an index ID.

2. The method of claim 1, wherein, The method further comprises the following step: If the local data document corresponding to the document identifier is not found, the search engine node stores the target data document.

3. The method of claim 1, wherein, The method further comprises the following step: If the global identifier of the target data document is not greater than the global identifier of the local data document, the local data document is kept.

4. The method of claim 1, wherein, An index identifier is constructed based on the document identifier and the global identifier, and is sent to a corresponding index updating node; The index updating node builds a target data document according to the index identifier.

5. The method of claim 4, wherein, The extraction node determines that the target data source has changed data in a way of incremental detection or real-time detection.

6. The method of claim 4, wherein, Before the step of submitting the built target data document to the search engine node, the method further comprises the following steps: The extraction node acquires the document identifier and the global identifier of all data in the target data source, and constructs an index identifier based on the document identifier and the global identifier, and sends the index identifier to a corresponding index updating node.

7. A system for multipath index update data, the system comprising a plurality of index update nodes, a search engine node, an extraction node corresponding to each index update node and an identification assignment node, characterized in that, The method comprises the following steps: A document submission module is configured to submit the built target data document to the search engine node by the index updating node, wherein the target data document comprises an index identifier, and the index identifier comprises a document identifier and a global identifier; A searching module is configured to search for a local data document according to the document identifier by the search engine node; A judging module is configured to judge whether the global identifier of the target data document is greater than the global identifier of the local data document if the local data document is found, and update the local data document with the target data document if yes; Before the step of submitting the built target data document to the search engine node, the extraction node acquires the document identifier and the global identifier of the changed data when the target data source has changed data; Before the step of acquiring the document identifier and the global identifier of the changed data, the identifier allocation node allocates a unique global identifier to the changed data; wherein the global identifier is incremented one by one; The document identifier is identity information of the data document, which is a document ID; the global identifier is update version information of the data document, which is a global ID; and the index identifier is identity information and update version information of the data document, which is an index ID.

8. An electronic device, comprising: The electronic device comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the method for updating data of multiple indexes according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executable by the processor to complete the method for updating data of multiple indexes according to any one of claims 1-6.

Images