Method, device and system for dynamically generating query interface based on Elasticsearch search engine

Abstract

The invention relates to the technical field of data processing, and provides a method, device and system for dynamically generating a query interface based on an Elasticsearch engine, and the method comprises the steps: obtaining configuration information used for generating a target query interface of the Elasticsearch engine, and constructing target request mapping information of the target query interface; the target query interface is registered based on a preset program processing module and target request mapping information of the target query interface, and when an interface request instruction of the target query interface is received, the interface request instruction is analyzed and processed; the interface request instruction is sent to a server, so that the interface request instruction is converted into a target query interface of an Elasticsearch search engine. According to the method, the query efficiency is improved by utilizing an Elasticsearch search engine technology, the ES script field is analyzed based on the configuration information and the program processing module, the function of dynamically generating the target query interface is provided, and the functions of efficient query and simplified interface construction are provided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and system for dynamically generating a query interface based on an Elasticsearch search engine. Background Art

[0002] With the rapid development of information technology and the continuous changes in business needs, data query efficiency has become an important focus of enterprises and organizations. The efficiency of traditional coded static interfaces is difficult to meet the rapidly growing business needs, especially in the application of emerging fields such as big data and artificial intelligence, the demand for high-real-time data query is more urgent.

[0003] Data query is the core function of an application, which is directly related to the practicality, user experience and business operation of the application. With the rise of big data technology, the growth of data volume and the increase in data processing requirements make it particularly important to improve the efficiency of data interface query services. Improving the efficiency of data interface query can effectively improve the performance of the application and user experience.

[0004] The dynamic interface generation function can not only simplify the interface writing process, but also dynamically allocate resources and perform unified cache optimization, thereby improving query efficiency.

[0005] Currently, there are two main ways to improve interface query efficiency. One is to optimize the interface processing logic, reasonably configure database indexes and query SQL, and speed up interface access by improving hardware levels. The other is to use search engines to improve query performance through software expansion.

[0006] The current solutions for dynamically generating interfaces are mainly based on the first method of writing SQL scripts, which requires users to have high SQL script writing capabilities. As users, it is often difficult to master good SQL writing skills. Therefore, the query efficiency of the generated interface is low, and the query efficiency improvement is unsatisfactory. Summary of the invention

[0007] In view of the above problems, the present invention proposes a method, device and system for dynamically generating a query interface based on an Elasticsearch search engine in order to overcome the above problems.

[0008] One aspect of the present invention provides a method for dynamically generating a query interface based on an Elasticsearch search engine, the method comprising: Obtain configuration information for generating a target query interface of an Elasticsearch search engine, wherein the configuration information of the target query interface includes API interface information, API access path, request method, and target ES script field, wherein the target ES script field includes a query parameter flag placeholder; Generate target request mapping information of the target query interface according to the API interface information, API access path and request method; Registering the target query interface according to a preset program processing module and target request mapping information of the target query interface, wherein the program processing module is integrated with a parameter processing function and an ES script parsing and encoding function; When receiving an interface request instruction of the target query interface, parsing the interface request instruction according to the program processing module to convert the interface request instruction into a target query interface of an Elasticsearch search engine; Among them, parsing and processing the interface request instruction according to the program processing module includes: using the parameter processing function in the program processing module to extract the query parameters in the interface request instruction, using the ES script parsing and encoding function to replace the query parameter flag placeholders contained in the ES script field corresponding to the target query interface with the query parameters corresponding to each query parameter flag placeholder one by one, and parsing to generate an ES query template.

[0009] Furthermore, the program processing module is also integrated with a response data processing function; After generating the ES query template, the method further includes: Pushing the target ES query template to the Elasticsearch search engine according to the API access path, so that the Elasticsearch search engine returns the target data information; The target data information returned by the Elasticsearch search engine is obtained, the target data information is integrated into target response data with a unified data structure using the response data processing function, and the target response data is fed back to the response interface of the interface request instruction.

[0010] Furthermore, after converting the interface request instruction into a target query interface of an Elasticsearch search engine, the method further includes: Testing the target query interface and putting the target query interface online after passing the test; Creating an application ID for the target query interface after going online, and authorizing the target query interface to limit the access rights of the application ID; Configure a current limiting rule for the target query interface to control the traffic of the target query interface accessing the Elasticsearch search engine; The target query interface is verified to verify whether the target query interface can return an expected response result.

[0011] Furthermore, before pushing the target ES query template to the Elasticsearch search engine according to the API access path, the method further includes: Accessing the target query interface; Determine whether the target query interface has access rights; If the target query interface has access rights, determining whether the target query interface is configured with a current limiting rule; If the target query interface is configured with a current limiting rule, the current limiting rule is called, and the current limiting is performed on the target query interface according to the current limiting rule; When the target query interface passes the current limiting or the current limiting rule is not configured, the target ES query template is pushed to the Elasticsearch search engine according to the API access path.

[0012] Further, limiting the flow of the target query interface according to the flow limiting rule includes: Obtain the current limiting threshold type of the target query interface corresponding to the current current limiting rule, and determine whether the current data flow exceeds the current limiting threshold corresponding to the current current limiting threshold type; If it exceeds, the flow control effect identifier of the target query interface is obtained. When the flow control effect identifier is direct rejection, access rejection information is fed back; when the flow control effect identifier is uniform queuing, the target query interface is controlled to queue until the current flow limit is passed; After queuing to pass the current current limiting, determine whether there is a next current limiting rule. If so, limit the target query interface according to the next current limiting rule until the target query interface passes all current limiting rules.

[0013] Furthermore, the method for implementing the parameter processing function includes: For an interface request instruction whose request method is a POST request, a preset filter is used to cache the body content of the POST request to obtain a first query parameter, so that the first query parameter can be reused when generating a target query interface; For an interface request instruction whose request method is a GET request, the request parameter of the GET request is directly obtained to obtain the second query parameter.

[0014] Furthermore, before obtaining the configuration information of the target query interface, the method further includes: Capturing change information of target data in a target business system database, where the target data is large data volume or data associated with an interface that is slow to query in the target business system database; A data synchronization script language is generated based on the change information of the target data, and is sent to the Elasticsearch search engine, so that the Elasticsearch search engine performs a corresponding target data change operation according to the data synchronization script language.

[0015] Furthermore, the generating of the data synchronization script language based on the change information of the target data includes: Use the preset middleware tool to query the target data in the target business system database, and write the change information of the target data into the Kafka message queue, where the change information of the target data includes data addition, data deletion and data modification; Consume the target data change information in the Kafka message queue based on the customized Kafka consumer service to obtain the target data change operation method corresponding to the target data change information; An index of an Elasticsearch search engine corresponding to the target data is obtained, and a data synchronization script language that can be understood by the Elasticsearch search engine is generated according to the index and the change operation method of the target data.

[0016] Another aspect of the present invention provides a device for dynamically generating a query interface based on an Elasticsearch search engine, the device comprising: An acquisition module is used to acquire configuration information of a target query interface for generating an Elasticsearch search engine, wherein the configuration information of the target query interface includes API interface information, API access path, request method, and target ES script field, wherein the target ES script field includes a query parameter flag placeholder; A mapping generation module, used to generate target request mapping information of the target query interface according to the API interface information, API access path and request method; An interface registration module, used to register the target query interface according to a preset program processing module and target request mapping information of the target query interface, wherein the program processing module is integrated with a parameter processing function and an ES script parsing and encoding function; The interface generation module is used for, when receiving an interface request instruction of the target query interface, parsing the interface request instruction according to the program processing module to convert the interface request instruction into a target query interface of an Elasticsearch search engine; Among them, parsing and processing the interface request instruction according to the program processing module includes: using the parameter processing function in the program processing module to extract the query parameters in the interface request instruction, using the ES script parsing and encoding function to replace the query parameter flag placeholders contained in the ES script field corresponding to the target query interface with the query parameters corresponding to each query parameter flag placeholder one by one, and parsing to generate an ES query template.

[0017] Another aspect of the present invention provides a system for dynamically generating a query interface based on an Elasticsearch search engine, the system comprising multiple target business system databases, a computer device and an Elasticsearch search engine, wherein the computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the method for dynamically generating a query interface based on an Elasticsearch search engine as described in any of the above items are implemented.

[0018] The embodiment of the present application provides a method, device and system for dynamically generating a query interface based on an Elasticsearch search engine. The method constructs target request mapping information of the target query interface by acquiring configuration information for generating a target query interface of the Elasticsearch search engine; and registers the target query interface based on a preset program processing module and the target request mapping information of the target query interface, and when an interface request instruction of the target query interface is received, the interface request instruction is parsed and processed to convert the interface request instruction into a target query interface of the Elasticsearch search engine. The present invention improves query efficiency by using Elasticsearch search engine technology, and dynamically generates a target query interface by parsing ES script fields based on configuration information and program processing modules, providing functions of efficient query and simplified interface construction.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a method for dynamically generating a query interface based on an Elasticsearch search engine according to an embodiment of the present invention; Figure 2 A flowchart of a method for executing a current limiting rule on a target query interface according to an embodiment of the present invention; Figure 3 A principle block diagram of a data synchronization method according to an embodiment of the present invention; Figure 4 The present invention is a schematic diagram of the structure of a device for dynamically generating a query interface based on an Elasticsearch search engine according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0022] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0023] The following is a detailed introduction to the method, device and system for dynamically generating a query interface based on the Elasticsearch search engine provided by the present invention in conjunction with the accompanying drawings.

[0024] Embodiment 1 The embodiment of the present invention provides a method for dynamically generating a query interface based on an Elasticsearch search engine, such as Figure 1 As shown, the method for dynamically generating a query interface based on the Elasticsearch search engine includes the following steps: S1. Obtain configuration information for generating a target query interface of an Elasticsearch search engine, wherein the configuration information of the target query interface includes API interface information, API access path, request method, and target ES script field, wherein the target ES script field includes a query parameter flag placeholder; In the embodiment of the present invention, the API interface information includes the API interface information name and / or the API interface description. The API access path corresponds to the URL of the interface, and the request method can be selected from GET or POST.

[0025] Furthermore, the ES script field is implemented using ES templating technology. Unlike the conventional ES query language DSL, matching parameters need to be enclosed in a special marker "{{}}". For example, pagination in the script is represented by {"from":"{{parameter name}}","size":"{{parameter name}}"}.

[0026] S2. Generate target request mapping information of the target query interface according to the API interface information, API access path and request method; The embodiment of the present invention is developed using the Spring Boot framework of the Java language. After receiving the configuration information of the target query interface, the background system will perform verification according to the script content, including but not limited to SQL injection verification, XSS verification, and sensitive word verification. After the verification passes, the interface information will be persisted to the local database so that the system can reconstruct the interface every time it is restarted. The interface itself has two states: offline and online. These two states are used to distinguish the current stage of the interface, that is, the test stage or the production stage. The offline interface is only open to the system for internal testing, and external access will be denied.

[0027] Furthermore, in a specific embodiment of the present invention, the RequestMapping technology in the Spring Boot framework is used to first construct the request mapping information of the interface according to the API access path and request method filled in when configuring the above interface, and the request mapping ensures that the interface can be accessed.

[0028] S3, registering the target query interface according to a preset program processing module and target request mapping information of the target query interface, wherein the program processing module is integrated with a parameter processing function and an ES script parsing and encoding function; In an embodiment of the present invention, dynamic registration is performed when the system is running, an interface proxy is created, request mapping information and program processing module are registered in RequestMappingInfoHandlerMapping, and the information is injected into the SpringBoot framework to realize dynamic generation of the interface.

[0029] S4. When receiving an interface request instruction of the target query interface, the interface request instruction is parsed according to the program processing module to convert the interface request instruction into a target query interface of an Elasticsearch search engine; Among them, parsing and processing the interface request instruction according to the program processing module includes: using the parameter processing function in the program processing module to extract the query parameters in the interface request instruction, using the ES script parsing and encoding function to replace the query parameter flag placeholders contained in the ES script field corresponding to the target query interface with the query parameters corresponding to each query parameter flag placeholder one by one, and parsing to generate an ES query template.

[0030] In a specific embodiment of the present invention, the Spring Boot framework dynamically generates a target query interface of the Elasticsearch search engine based on request mapping information and ES query templates, providing the functions of efficient query and simplified interface construction.

[0031] Furthermore, the program processing module of the embodiment of the present invention also integrates a response data processing function. After generating the ES query template, the method for dynamically generating a query interface based on the Elasticsearch search engine of the embodiment of the present invention also includes: S5. Pushing the target ES query template to the Elasticsearch search engine according to the API access path, so that the Elasticsearch search engine returns the target data information; S6. Obtain the target data information returned by the Elasticsearch search engine, use the response data processing function to integrate the target data information into target response data with a unified data structure, and feed back the target response data to the response interface of the interface request instruction.

[0032] Furthermore, the implementation method of the parameter processing function of the program processing module of the embodiment of the present invention specifically includes: for an interface request instruction whose request method is a POST request, a preset filter is used to cache the body content of the POST request to obtain a first query parameter, so that the first query parameter can be reused when generating a target query interface; for an interface request instruction whose request method is a GET request, the request parameter of the GET request is directly obtained to obtain a second query parameter.

[0033] It should be noted that since the body content of the POST method is stored in the data stream and can only be read once, the body content of the POST method is cached through the Servlet filter, so it is reused in the Servlet filter chain and interception chain. Then the parameters in the body are merged with the GET method parameters, and the differences between the two parameters can be unified within the program.

[0034] The embodiment of the present invention is based on the Spring Boot framework, converts the interface information input by the user into the HTTP RESTful API query interface provided by ES, and initiates a request to ES, so that the user can quickly and accurately build a query interface and improve the search efficiency of the interface.

[0035] In addition, the present invention also provides a set of interface lists and a method for dynamically generating a query interface operation interface. The function of the interface list is to maintain the interfaces created in the system; and the function of dynamically generating interfaces allows users to configure interface generation parameters, so that users can automatically and dynamically generate a target query interface of the Elasticsearch search engine through simple interface configuration.

[0036] Furthermore, after converting the interface request instruction into a target query interface of the Elasticsearch search engine, the method of the embodiment of the present invention also includes: testing the target query interface and putting the target query interface online after passing the test; creating an application ID for the target query interface after going online, and authorizing the target query interface to limit the access rights of the application ID; configuring current limiting rules for the target query interface to control the traffic of the target query interface accessing the Elasticsearch search engine; and verifying the target query interface to verify whether the target query interface can return the expected response result.

[0037] Furthermore, the present invention provides a test interface program. The program can read the configuration information filled in when building the interface and automatically generate a test address. In addition, the test interface program can also parse the identifier parameters in the ES script and provide parameter value filling and modification functions. When performing an interface test, the system will respond and return relevant information, including return data content, test results, and response delays. During the interface testing phase, permission control is removed and only parameter values ​​are verified. This means that during the test process, permission verification is no longer required, but instead the focus is on checking the legality of parameter values.

[0038] Furthermore, after the test is passed, the interface status is updated to online. The online interface can be accessed in the external network, but it must be configured by the authorized application. Permission management is responsible for maintaining the application permissions required to access the interface. For the access rights of the interface, the system needs to establish an application configuration, automatically generate an application ID, and bind the interface with permission to the application ID. Each time the interface is called, the request parameters must include the application ID field. The system will verify whether the application ID carried in the request is associated with the interface. If there is a binding association, access is allowed; otherwise, the system will deny access and give a corresponding prompt.

[0039] Furthermore, the present invention also provides a method for configuring a flow limiting rule for the target query interface, wherein the flow limiting rule is composed of a resource name, a flow limiting threshold type, a flow limiting threshold, a flow control effect, and the like. The resource name corresponds to the dynamically generated interface described in this article, and the flow limiting threshold type is divided into two modes, the number of concurrent threads and QPS (query rate per second), and the size of the flow limiting threshold is set according to the type; there are two modes of flow control effect, direct rejection and uniform queuing, and the final result of direct rejection is to return an error message for the request, such as 403 access rejection, and uniform queuing will cause the response time of the request to be limited to increase with the order of the rules. The system matches the interface according to the resource name and configures the flow limiting rule for it. Once the flow limiting threshold is reached, the flow control mechanism is triggered immediately. Multiple flow limiting rules can be configured, and flow limiting is performed based on the principle of first arrival.

[0040] Furthermore, in the verification stage, in a specific embodiment of the present invention, by utilizing an API interface testing tool such as Postman or JMeter, the URL address of the interface is accessed and request parameters and application ID are filled in to verify whether the interface can correctly return the expected response result.

[0041] Furthermore, before pushing the target ES query template to the Elasticsearch search engine according to the API access path, the present invention authenticates and limits the flow of the target query interface based on the preset flow limiting rules, specifically as follows Figure 2 As shown, the method includes: S41, accessing the target query interface; S42, determining whether the target query interface has access rights; S43: If the target query interface has access rights, determine whether the target query interface is configured with a current limiting rule; S44, if the target query interface is configured with a current limiting rule, the current limiting rule is called, and the target query interface is current limited according to the current limiting rule; S45. When the target query interface passes the current limiting rule or the current limiting rule is not configured, the target ES query template is pushed to the Elasticsearch search engine according to the API access path.

[0042] Furthermore, limiting the flow of the target query interface according to the flow limiting rule includes: obtaining the flow limiting threshold type of the target query interface corresponding to the current flow limiting rule, and determining whether the current data traffic exceeds the flow limiting threshold corresponding to the current flow limiting threshold type; if it exceeds, obtaining the flow control effect identifier of the target query interface, and when the flow control effect identifier is direct rejection, feedback of access rejection information; when the flow control effect identifier is uniform queuing, controlling the target query interface to queue until the current flow limiting is passed; after queuing until the current flow limiting is passed, determining whether there is a next flow limiting rule, and if so, limiting the flow of the target query interface according to the next flow limiting rule until the target query interface passes all flow limiting rules.

[0043] The method for dynamically generating a query interface based on an Elasticsearch search engine provided in the embodiment of the present invention further includes a method for synchronizing target data to the Elasticsearch search engine. Specifically, before obtaining configuration information of the target query interface, the method further includes: S01. Capturing change information of target data in a target business system database, where the target data is large data volume or data associated with an interface that is slow to query in the target business system database; Specifically, the present invention uses Logstash middleware tool and Kafka to realize data synchronization. Specifically, the data synchronization script language generated based on the change information of the target data in the embodiment of the present invention specifically includes: S02. Generate a data synchronization script language based on the change information of the target data, and send it to the Elasticsearch search engine, so that the Elasticsearch search engine performs a corresponding target data change operation according to the data synchronization script language.

[0044] Furthermore, the generating of the data synchronization script language based on the change information of the target data includes: S011. Use a preset middleware tool to query the target data in the target business system database, and write the change information of the target data into the message queue of Kafka, wherein the change information of the target data includes data addition, data deletion and data modification; The business system databases in the embodiments of the present invention include Mysql, SQLServer, Oracle, and PostgreSQL. The present invention analyzes each business system database, determines the hot data involved in large data volumes or slow query interfaces as target data, and uses the business system database to which the target data belongs as the target business system database.

[0045] In one embodiment of the present invention, a Logstash middleware tool is used to capture target data in a target business system database, and write the target data into a Kafka message queue in real time.

[0046] Furthermore, before using the Logstash middleware tool to capture the target data in the target business system database, it is necessary to configure the config information of the Logstash synchronization file, wherein the config information specifically includes: database connection parameters, database name, user name, password, driver, filter rules, kafka address, kafka port number, kafka topic name, etc. The present invention can send data in different target business system databases to the port of kafka corresponding to the target business system database by configuring the config information of the Logstash synchronization file.

[0047] S012. Consuming the target data change information in the Kafka message queue based on the customized Kafka consumer service to obtain a target data change operation method corresponding to the target data change information; The embodiment of the present invention is also equipped with a customized Kafka consumer service. The Kafka consumer service consumes the data addition, deletion and modification messages with JSON structure in the Kafka message queue in real time, and parses the operation methods in JSON, such as INSERT, UPDATE and DELETE. That is, when there are addition, deletion and modification operations on the target data in any target business system database, they are all transmitted to the port of Kafka in the form of JSON structured messages. The Kafka consumer service can then obtain the change information of the target data by consuming the messages on the Kafka port. The Kafka ports corresponding to different target business system databases are different. The Kafka consumer service can determine which business system database the data changes come from based on the different ports consumed.

[0048] S013. Obtain an index of the Elasticsearch search engine corresponding to the target data, and generate a data synchronization script language that can be understood by the Elasticsearch search engine according to the index and the target data change operation method.

[0049] In the embodiment of the present invention, when performing data change operations, it is also necessary to create an index structure of the Elasticsearch search engine and convert it into a DSL (data synchronization script language) that the Elasticsearch search engine can understand, send it to the Elasticsearch service and realize data synchronization.

[0050] Furthermore, in order to optimize the storage efficiency of the Elasticsearch index, the system also builds a special query structure, adding a unique ROUTING field to each Elasticsearch operation request in the code to ensure the sharding routing processing of the same source data. That is, a navigation field is added to the target data so that the target data in the target business system database is saved to the slice corresponding to the Elasticsearch search engine.

[0051] At the same time, the association between data uses the parent-child relationship data type in the Elasticsearch search engine to improve the efficiency of association queries. That is, a parent-child association relationship between indexes is established for the target data, and the parent association field of the current index is marked in the association field of the current index.

[0052] Furthermore, to achieve this goal, the structure of the association relationship can be expressed as {"case_associated_field":{"parent":"","name":""}}, where parent is the parent table name and name is the current table name.

[0053] Figure 3 A system framework diagram is shown in which target data change information in a target business system database is captured, a data synchronization script language is generated based on the target data change information, and the data synchronization script language is sent to an Elasticsearch search engine so that the Elasticsearch search engine performs a corresponding target data change operation according to the data synchronization script language according to an embodiment of the present invention. First, the Logstash middleware tool synchronizes the target data change information to the Kafka message queue in real time. Different consumer systems consume the information in the Kafka message queue according to their own customized Kafka consumer services, generate data synchronization script language for the consumed information, and send it to the Elasticsearch search engine, so that the Elasticsearch search engine executes the corresponding target data change operation according to the data synchronization script language.

[0054] It should be noted that in addition to using the Logstash middleware system to synchronize data in the business database system to the Elasticsearch search engine, you can also use Flink CDC technology. This technology relies on the Aphche Flink distributed computing framework and uses query or log monitoring methods to capture changes in databases such as MySQL, Oracle, SQLServer, etc. in real time, and write the data directly to the downstream Elasticsearch search engine.

[0055] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0056] Embodiment 2 Figure 4 The schematic diagram of the structure of a device for dynamically generating a query interface based on an Elasticsearch search engine provided by an embodiment of the present invention is shown schematically. Figure 4 The device for dynamically generating a query interface based on an Elasticsearch search engine according to an embodiment of the present invention specifically comprises an acquisition module 401, a mapping generation module 402, an interface registration module 403 and an interface generation module 404, wherein: The acquisition module 401 is used to acquire configuration information of a target query interface for generating an Elasticsearch search engine, wherein the configuration information of the target query interface includes API interface information, API access path, request method, and target ES script field, wherein the target ES script field includes a query parameter flag placeholder; A mapping generation module 402 is used to generate target request mapping information of the target query interface according to the API interface information, API access path and request method; An interface registration module 403 is used to register the target query interface according to a preset program processing module and target request mapping information of the target query interface, wherein the program processing module is integrated with a parameter processing function and an ES script parsing and encoding function; The interface generation module 404 is used for, when receiving an interface request instruction of the target query interface, parsing the interface request instruction according to the program processing module to convert the interface request instruction into a target query interface of an Elasticsearch search engine; Among them, parsing and processing the interface request instruction according to the program processing module includes: using the parameter processing function in the program processing module to extract the query parameters in the interface request instruction, and using the ES script parsing and encoding function to replace the query parameter flag placeholders contained in the ES script field corresponding to the target query interface with the query parameters corresponding to each query parameter flag placeholder one by one, and parsing to generate an ES query template.

[0057] Furthermore, the program processing module is also integrated with a response data processing function; The device also includes: A push module, used to push the target ES query template to the Elasticsearch search engine according to the API access path, so that the Elasticsearch search engine returns the target data information; The data response module is used to obtain the target data information returned by the Elasticsearch search engine, integrate the target data information into target response data with a unified data structure using the response data processing function, and feed back the target response data to the response interface of the interface request instruction.

[0058] Furthermore, the embodiment of the present invention also includes an initial search interface configuration module including: A test submodule, used to test the target query interface and put the target query interface online after passing the test; The authorization submodule is used to create an application ID for the target query interface after going online, and authorize the target query interface to limit the access rights of the application ID; A traffic configuration submodule, used to configure a current limiting rule for the target query interface to control the traffic of the target query interface accessing the Elasticsearch search engine; The verification submodule is used to verify the target query interface to verify whether the target query interface can return an expected response result.

[0059] Furthermore, the embodiment of the present invention also includes a flow control submodule, which is used to perform flow control on the target query interface before pushing the target ES query template to the Elasticsearch search engine according to the API access path, specifically including: An access submodule, used for accessing the target query interface; An authentication submodule, used to determine whether the target query interface has access rights; A judgment submodule, used for judging whether the target query interface is configured with a current limiting rule if the target query interface has access rights; A current limiting submodule, configured to call the current limiting rule if the target query interface is configured with the current limiting rule, and to limit the current of the target query interface according to the current limiting rule; The push submodule is used to push the ES query template to the Elasticsearch search engine according to the API access path when the target query interface passes the current limiting rule or the current limiting rule is not configured.

[0060] Furthermore, the current limiting submodule specifically includes: The acquisition subunit is used to obtain the current limiting threshold type of the target query interface corresponding to the current current limiting rule, and determine whether the current data flow exceeds the current limiting threshold corresponding to the current current limiting threshold type; The current limiting response subunit is used to obtain the flow control effect identifier of the target query interface if it exceeds, and when the flow control effect identifier is direct rejection, feedback access rejection information; when the flow control effect identifier is uniform queuing, control the target query interface to queue until passing the current current limiting; The current limiting response subunit is also used to determine whether there is a next current limiting rule after queuing and passing the current current limiting. If so, limit the current of the target query interface according to the next current limiting rule until the target query interface passes all current limiting rules.

[0061] Furthermore, the embodiment of the present invention also includes: A data capture module is used to capture change information of target data in a target business system database, where the target data is large data volume or data associated with an interface that is slow to query in the target business system database; The data synchronization module is used to generate a data synchronization script language based on the change information of the target data, and send it to the Elasticsearch search engine, so that the Elasticsearch search engine performs the corresponding target data change operation according to the data synchronization script language.

[0062] Furthermore, the data synchronization module specifically includes: The information capture submodule is used to query the target data in the target business system database using a preset middleware tool, and write the change information of the target data into the message queue of Kafka, wherein the change information of the target data includes data addition, data deletion and data modification; A service consumption submodule, which is used to consume the change information of the target data in the message queue of the Kafka based on the customized Kafka consumer service, so as to obtain the change operation method of the target data corresponding to the change information of the target data; The script language generation submodule is used to obtain the index of the Elasticsearch search engine corresponding to the target data, and generate a data synchronization script language that can be understood by the Elasticsearch search engine according to the index and the change operation method of the target data.

[0063] The embodiment of the present application provides a method and device for dynamically generating a query interface based on an Elasticsearch search engine. The method constructs target request mapping information of the target query interface by acquiring configuration information for generating a target query interface of the Elasticsearch search engine; and registers the target query interface based on a preset program processing module and the target request mapping information of the target query interface. When an interface request instruction of the target query interface is received, the interface request instruction is parsed and processed to convert the interface request instruction into a target query interface of the Elasticsearch search engine. The present invention improves query efficiency by using Elasticsearch search engine technology, and dynamically generates a target query interface by parsing ES script fields based on configuration information and program processing modules, providing functions of efficient query and simplified interface construction.

[0064] Embodiment 3 An embodiment of the present invention provides a system for dynamically generating a query interface based on an Elasticsearch search engine. The system includes multiple target business system databases, a computer device, and an Elasticsearch search engine.

[0065] The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments for dynamically generating a query interface based on an Elasticsearch search engine are implemented, for example: Figure 1 Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device for dynamically generating a query interface based on the Elasticsearch search engine are realized, for example Figure 4 The obtained module 401, the mapping generation module 402, the interface registration module 403 and the interface generation module 404 are shown.

[0066] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamically generating a query interface based on an Elasticsearch search engine, characterized in that: The method comprises: Obtain configuration information for generating a target query interface of an Elasticsearch search engine, wherein the configuration information of the target query interface includes API interface information, API access path, request method, and target ES script field, wherein the target ES script field includes a query parameter flag placeholder; Generate target request mapping information of the target query interface according to the API interface information, API access path and request method; Registering the target query interface according to a preset program processing module and target request mapping information of the target query interface, wherein the program processing module is integrated with a parameter processing function and an ES script parsing and encoding function; When receiving an interface request instruction of the target query interface, parsing the interface request instruction according to the program processing module to convert the interface request instruction into a target query interface of an Elasticsearch search engine; Among them, parsing and processing the interface request instruction according to the program processing module includes: using the parameter processing function in the program processing module to extract the query parameters in the interface request instruction, using the ES script parsing and encoding function to replace the query parameter flag placeholders contained in the ES script field corresponding to the target query interface with the query parameters corresponding to each query parameter flag placeholder one by one, and parsing to generate an ES query template.

2. The method according to claim 1, characterized in that: The program processing module also integrates a response data processing function; After generating the ES query template, the method further includes: Pushing the target ES query template to the Elasticsearch search engine according to the API access path, so that the Elasticsearch search engine returns the target data information; The target data information returned by the Elasticsearch search engine is obtained, the target data information is integrated into target response data with a unified data structure using the response data processing function, and the target response data is fed back to the response interface of the interface request instruction.

3. The method according to claim 2, characterized in that After converting the interface request instruction into a target query interface of an Elasticsearch search engine, the method further includes: Testing the target query interface and putting the target query interface online after passing the test; Creating an application ID for the target query interface after going online, and authorizing the target query interface to limit the access rights of the application ID; Configure a current limiting rule for the target query interface to control the traffic of the target query interface accessing the Elasticsearch search engine; The target query interface is verified to verify whether the target query interface can return an expected response result.

4. The method according to claim 3, characterized in that Before pushing the target ES query template to the Elasticsearch search engine according to the API access path, the method further includes: Accessing the target query interface; Determine whether the target query interface has access rights; If the target query interface has access rights, determining whether the target query interface is configured with a current limiting rule; If the target query interface is configured with a current limiting rule, the current limiting rule is called, and the current limiting is performed on the target query interface according to the current limiting rule; When the target query interface passes the current limiting or the current limiting rule is not configured, the target ES query template is pushed to the Elasticsearch search engine according to the API access path.

5. The method according to claim 4, characterized in that The limiting the flow of the target query interface according to the flow limiting rule includes: Obtain the current limiting threshold type of the target query interface corresponding to the current current limiting rule, and determine whether the current data flow exceeds the current limiting threshold corresponding to the current current limiting threshold type; If it exceeds, the flow control effect identifier of the target query interface is obtained. When the flow control effect identifier is direct rejection, access rejection information is fed back; when the flow control effect identifier is uniform queuing, the target query interface is controlled to queue until the current flow limit is passed; After passing the current current limiting, it is determined whether there is a next current limiting rule. If so, the target query interface is current limited according to the next current limiting rule until the target query interface passes all current limiting rules.

6. The method according to claim 1, characterized in that The method for implementing the parameter processing function includes: For an interface request instruction whose request method is a POST request, a preset filter is used to cache the body content of the POST request to obtain a first query parameter, so that the first query parameter can be reused when generating a target query interface; For an interface request instruction whose request method is a GET request, the request parameter of the GET request is directly obtained to obtain the second query parameter.

7. The method according to claim 1, characterized in that Before obtaining the configuration information of the target query interface, the method further includes: Capturing change information of target data in a target business system database, where the target data is large data volume or data associated with an interface that is slow to query in the target business system database; A data synchronization script language is generated based on the change information of the target data, and is sent to the Elasticsearch search engine, so that the Elasticsearch search engine performs a corresponding target data change operation according to the data synchronization script language.

8. The method according to claim 7, characterized in that The generating of the data synchronization script language based on the change information of the target data comprises: Use the preset middleware tool to query the target data in the target business system database, and write the change information of the target data into the Kafka message queue, where the change information of the target data includes data addition, data deletion and data modification; Consume the target data change information in the Kafka message queue based on the customized Kafka consumer service to obtain the target data change operation method corresponding to the target data change information; An index of an Elasticsearch search engine corresponding to the target data is obtained, and a data synchronization script language that can be understood by the Elasticsearch search engine is generated according to the index and the change operation method of the target data.

9. A device for dynamically generating a query interface based on an Elasticsearch search engine, characterized in that: The device comprises: An acquisition module is used to acquire configuration information of a target query interface for generating an Elasticsearch search engine, wherein the configuration information of the target query interface includes API interface information, API access path, request method, and target ES script field, wherein the target ES script field includes a query parameter flag placeholder; A mapping generation module, used to generate target request mapping information of the target query interface according to the API interface information, API access path and request method; An interface registration module, used to register the target query interface according to a preset program processing module and target request mapping information of the target query interface, wherein the program processing module is integrated with a parameter processing function and an ES script parsing and encoding function; The interface generation module is used for, when receiving an interface request instruction of the target query interface, parsing the interface request instruction according to the program processing module to convert the interface request instruction into a target query interface of an Elasticsearch search engine; Among them, parsing and processing the interface request instruction according to the program processing module includes: using the parameter processing function in the program processing module to extract the query parameters in the interface request instruction, using the ES script parsing and encoding function to replace the query parameter flag placeholders contained in the ES script field corresponding to the target query interface with the query parameters corresponding to each query parameter flag placeholder one by one, and parsing to generate an ES query template.

10. A system for dynamically generating a query interface based on an Elasticsearch search engine, characterized in that: The system includes multiple target business system databases, computer devices and an Elasticsearch search engine, wherein the computer device includes a memory, a processor and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the method for dynamically generating a query interface based on an Elasticsearch search engine as described in any one of claims 1 to 8 are implemented.

Images