Data indexing method, medium, and system

Abstract

The application discloses a data indexing method, medium and system, wherein the method comprises the following steps: a first indexer acquires indexing data, splits the indexing data into view list data and file content data, sends the view list data to a second indexer, and sends the file content data to a third indexer; the second indexer sends the view list data to a view list indexing application server, so that the view list data is stored by the view list indexing application server; and the third indexer asynchronously sends the file content data to a file content indexing application server, so that the file content data is stored by the file content indexing application server; the application can effectively reduce the frequency of occurrence of the problems of slow system response speed and service suspension in the access pressure peak period, and improve user experience.

Description

Technical Field

[0001] This application relates to the field of data management, specifically to a data indexing method, a storage medium, and a data indexing system. Background Technology

[0002] In related technologies, data indexing often involves storing data uniformly through a single application server. In this way, the system can run normally when the amount of data is small and the access pressure is low. However, when the access pressure is high, the system will experience slow response speed and service hangs, which will seriously affect the customer experience. Summary of the Invention

[0003] In view of the above problems, this application provides a data indexing method that can effectively reduce the frequency of slow system response and service hangs during peak access periods, thereby improving user experience.

[0004] To achieve the above objectives, the inventors provide a data indexing method, comprising: a first indexer acquiring index data and splitting the index data into view list data and file content data; sending the view list data to a second indexer and sending the file content data to a third indexer; the second indexer sending the view list data to a view list indexing application server for storage; and the third indexer asynchronously sending the file content data to a file content indexing application server for storage.

[0005] Unlike existing technologies, the above technical solution stores view list data and file content data separately through different index application servers; this separates the processing of view list data from the processing of file content data, thereby effectively reducing the frequency of slow system response and service hangs during peak access periods and improving user experience.

[0006] In some embodiments, the view list index application server and the file content index application server are Solr clusters.

[0007] In some embodiments, the method further includes: acquiring a data query request through a first viewer, and determining the request type corresponding to the data query request, wherein the request type includes view columns.

[0008] Table data query request and file content data query request.

[0009] In some embodiments, if the request type of the data query request is a view list data query request, then the second viewer is used to invoke the view list index application server to obtain the view list data corresponding to the view list data query request.

[0010] 35 In some embodiments, if the request type of the data query request is a file content data query...

[0011] If a query request is received, a third-party viewer is used to invoke the file content indexing application server to obtain the file content data corresponding to the file content data query request.

[0012] To achieve the above objectives, the inventors have provided a storage medium storing a computer.

[0013] The program, when executed by a processor, implements the steps of the data indexing method described above. 40. To achieve the above objective, the inventors provide a data indexing system, including a first indexer, a second indexer, a third indexer, a view list indexing application server, and a file content indexing application server; the first indexer is used to acquire index data, split the index data into view list data and file content data, and send the view list data to the second indexer, and...

[0014] The file content data is sent to the third indexer; the second indexer is used to send the view list 45 data to the view list indexing application server; the third indexer is used to send the file content...

[0015] Data is asynchronously sent to the file content indexing application server; the view list indexing application server is used to store the received view list data; the file content indexing application server is used to store the received file content data.

[0016] In some embodiments, the view list indexing application server and the file content indexing application server are Solr clusters.

[0017] In some embodiments, a first viewer is further included, which is used to acquire a data query request and determine the request type corresponding to the data query request, wherein the request type includes a view list data query request and a file content data query request.

[0018] In some embodiments, a second viewer and a third viewer are further included; the second viewer is used to invoke the view list index application server to obtain the view list data corresponding to the view list data query request when the request type of the data query request is a view list data query request; the third viewer is used to invoke the file content index application server to obtain the file content data corresponding to the file content data query request when the request type of the data query request is a file content data query request.

[0019] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0020] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of this application and other related content, and should not be considered as limitations on this application.

[0021] In the accompanying drawings of the instruction manual:

[0022] Figure 1 This is a flowchart illustrating a data indexing method according to an embodiment of the present invention;

[0023] Figure 2 This is a block diagram of a data indexing system according to an embodiment of the present invention. Detailed Implementation

[0024] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0025] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0026] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0027] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0028] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0029] Unless otherwise specified, the use of terms such as “comprising,” “including,” “having,” or other similar expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0030] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0031] In the description of the embodiments of this application, spatially related terms such as "center," "vertical direction," "lateral direction," "length," "width," "thickness," "up," "down," "front," "rear," "left," "right," and "vertical" are used.

[0032] "Horizontal", "Vertical", "Top", "Bottom", "Inner", "Outer", "Clockwise", "Counterclockwise", "Axial", "Radial"

[0033] The directions or positional relationships indicated by "circumferential direction" are based on the directions or positional relationships shown in the specific embodiments or drawings, and are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and are not intended to indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore should not be construed as a limitation on the embodiments of this application.

[0034] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0035] As with the data indexing methods described in the background technology, the system can operate normally when the data volume is small and the access pressure is low; however, when the access pressure is high, the system will experience slow response speed and service 120 hangs, seriously affecting the customer experience. Based on this problem, the inventors analyzed the system code and system logs and found that the problem was mainly caused by the large amount of file content data in the index data (including view list data and file content data), which led to excessive cache space being occupied.

[0036] Therefore, the system needs to spend a considerable amount of time indexing the file content data, consuming I / O during the indexing process.

[0037] The inability to process other business requests properly results in slow page response times and a poor user experience.

[0038] 125 According to the data indexing method proposed in this application, the view list data and file content data are stored separately through multiple indexing application servers to separate the view list data and file content data and avoid mutual interference between the two; in this way, when querying data in the future, the processing of file content data is asynchronous, and the normal operation of the business system will not be affected by the excessive time consumed by indexing file content data.

[0039] Please refer to 130 Figure 1 , Figure 1 This is a flowchart illustrating the data indexing method 100 according to an embodiment of the present invention; as follows: Figure 1 As shown, the data indexing method 100 includes the following steps:

[0040] S101, the first indexer obtains index data, splits the index data into view list data and file content data, sends the view list data to the second indexer, and sends the file content data to the third indexer.

[0041] In other words, after the first indexer obtains the index data, it splits the index data into view list data and file content data. The view list data is the data used in the project to query list data and has high real-time requirements. The file content data is used for full-text search and has lower real-time requirements. Then, the split view list data is sent to the second indexer and the file content data is sent to the third indexer.

[0042] In some embodiments, the view list indexing application server and the file content indexing application server are Solr clusters.

[0043] S102, the second indexer sends the view list data to the view list indexing application server for storage.

[0044] S103, the third indexer asynchronously sends the file content data to the file content indexing application server so that the file content data can be stored by the file content indexing application server.

[0045] In other words, after the first indexer splits the index data, it selects and calls the appropriate indexer (second or third indexer) to process the index data based on the data type; then, there are multiple index application servers, so that the view list data and file content data are stored through multiple index application servers respectively.

[0046] In some embodiments, the method further includes: obtaining a data query request through a first viewer and determining the request type corresponding to the data query request, wherein the request type includes a view list data query request and a file content data query request.

[0047] In some embodiments, if the request type of the data query request is a view list data query request, then the second viewer is used to invoke the view list index application server to obtain the view list data corresponding to the view list data query request.

[0048] In some embodiments, if the request type of the data query request is a file content data query request, a third-party viewer is used to invoke the file content indexing application server to obtain the file content data corresponding to the file content data query request.

[0049] As an example, when a user needs to perform a data query, they first send a data query request to the business system, which then forwards the request to a first viewer. Next, the first viewer determines the request type. If the request type is a view list data query request, a second viewer is used to call the view list index application server to retrieve the corresponding view list data. If it is a file content data query request, a third viewer is used to call the file content index application server to retrieve the corresponding file content data. In this way, by using the appropriate viewer to process query requests of different types, the two types of data queries do not interfere with or affect each other, separating I / O. This effectively avoids system business malfunctions caused by excessively large file contents or excessive cache space usage.

[0050] In summary, the first indexer acquires index data, splits it into view list data and file content data, sends the view list data to the second indexer, and sends the file content data to the third indexer. The second indexer sends the view list data to a view list indexing application server for storage. The third indexer asynchronously sends the file content data to a file content indexing application server for storage. This effectively reduces the frequency of slow system response and service hangs during peak access periods, improving user experience.

[0051] To implement the above embodiments, this invention proposes a storage medium storing a computer program that, when executed by a processor, implements the steps of the data indexing method described above.

[0052] To implement the above embodiments, this invention proposes a data indexing system 200; such as Figure 2 As shown, the data indexing system 200 includes: a first indexer 10, a second indexer 20, a third indexer 30, a view list indexing application server 40, and a file content indexing application server 50.

[0053] The first indexer 10 is used to obtain index data, split the index data into view list data and file content data, send the view list data to the second indexer, and send the file content data to the third indexer.

[0054] The second indexer 20 is used to send view list data to the view list index application server;

[0055] The third indexer 30 is used to asynchronously send file content data to the file content indexing application server;

[0056] The view list index application server 40 is used to store the received view list data;

[0057] The file content indexing application server 50 is used to store the received file content data.

[0058] In some embodiments, the view list index application server and the file content index application server are Solr clusters.

[0059] In some embodiments, the system further includes a first viewer, which is used to acquire data query requests and determine the request type corresponding to the data query requests, wherein the request type includes view list data query requests and file content data query requests.

[0060] In some embodiments, a second viewer and a third viewer are also included; the second viewer is used to invoke the view list index application server to obtain the view list data corresponding to the view list data query request when the request type of the data query request is a view list data query request; the third viewer is used to invoke the file content index application server to obtain the file content data corresponding to the file content data query request when the request type of the data query request is a file content data query request.

[0061] In summary, by setting up a first indexer to retrieve index data, splitting the index data into view list data and file content data, and sending the view list data to a second indexer and the file content data to a third indexer; the second indexer sends the view list data to the view list indexing application server; the third indexer asynchronously sends the file content data to the file content indexing application server; the view list indexing application server stores the received view list data; and the file content indexing application server stores the received file content data. This effectively reduces the frequency of slow system response and service hangs during peak access periods, improving user experience.

[0062] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A data indexing method, characterized in that, include: The first indexer acquires index data, splits the index data into view list data and file content data, sends the view list data to the second indexer, and sends the file content data to the third indexer; The second indexer sends the view list data to the view list indexing application server for storage. The third indexer asynchronously sends the file content data to the file content indexing application server so that the file content data can be stored by the file content indexing application server; View list data is used in the project to query list data, and has high real-time requirements; while file content data is used for full-text search, and has lower real-time requirements.

2. The data indexing method as described in claim 1, characterized in that, The view list index application server and the file content index application server are Solr clusters.

3. The data indexing method as described in claim 1, characterized in that, Also includes: The first viewer obtains the data query request and determines the request type corresponding to the data query request. The request type includes view list data query request and file content data query request.

4. The data indexing method as described in claim 3, characterized in that, If the data query request type is a view list data query request, then the second viewer is used to call the view list index application server to obtain the view list data corresponding to the view list data query request.

5. The data indexing method as described in claim 3, characterized in that, If the request type of the data query request is a file content data query request, then the third viewer is used to call the file content indexing application server to obtain the file content data corresponding to the file content data query request.

6. A storage medium, characterized in that: The storage medium stores a computer program that, when executed by a processor, implements the steps of the data indexing method as described in any one of claims 1 to 5.

7. A data indexing system, characterized in that, This includes a first indexer, a second indexer, a third indexer, a view list indexing application server, and a file content indexing application server; The first indexer is used to acquire index data, split the index data into view list data and file content data, send the view list data to the second indexer, and send the file content data to the third indexer; The second indexer is used to send the view list data to the view list indexing application server; The third indexer is used to asynchronously send the file content data to the file content indexing application server; The view list index application server is used to store the received view list data; The file content indexing application server is used to store the received file content data; View list data is used in the project to query list data, and has high real-time requirements; while file content data is used for full-text search, and has lower real-time requirements.

8. The data indexing system as described in claim 7, characterized in that, The view list index application server and the file content index application server are Solr clusters.

9. The data indexing system as described in claim 7, characterized in that, It also includes a first viewer, which is used to obtain data query requests and determine the request type corresponding to the data query requests, wherein the request type includes view list data query requests and file content data query requests.

10. The data indexing system as described in claim 9, characterized in that, It also includes a second viewer and a third viewer; The second viewer is used to call the view list index application server to obtain the view list data corresponding to the view list data query request when the request type of the data query request is a view list data query request; The third viewer is used to invoke the file content indexing application server to obtain the file content data corresponding to the file content data query request when the request type of the data query request is a file content data query request.

Images