Data source management method
Through the preprocessor and converter combined with the velocity template engine, different versions of Elasticsearch syntax are generated, which solves the problems of difficulty in learning operation, inconvenient resource management and version compatibility when using Elasticsearch, and realizes cross-version compatible multi-data source management. Simplifies index management and data flow archiving, improving resource utilization and work efficiency.
Patent Information
- Application Number
- CN202411094436.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When using Elasticsearch, enterprises face difficulties in learning operations, inconvenient resource management and version compatibility issues, resulting in difficulties in index management, data flow archiving and cross-version data retrieval, low resource utilization, many repetitive work, and low efficiency.
Using preprocessor and converter combined with velocity template engine, different versions of Elasticsearch syntax are generated, Elasticsearch cluster is accessed through the httpclient module, Easyexcel and Opencsv plug-in are integrated, and hot deployment and snapshot recovery is achieved using Nginx+Ik word segmenter and NFS, and Codemirror verification is used. Format sql and json to simplify index management and data import.
Implement cross-version compatible multi-data source management, lowers the threshold for use of Elasticsearch, simplifies index management and data flow archiving, improves resource utilization, reduces duplicate work, improves work efficiency, supports one-click archiving and cross-clusters Data migration.
Smart Images

Figure CN119066247A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and in particular to a data source management method. Background Art
[0002] With the advent of the big data era, traditional relational databases are unable to meet the needs of large-scale data search and analysis. In traditional databases, when the amount of data is large, the query speed will slow down, and real-time search and analysis cannot be performed. Elasticsearch is a tool that can search and analyze large-scale data quickly and in real time, and is easy to use and deploy. It supports distributed architecture, real-time indexing, multi-data type support, aggregate analysis, etc., and is very powerful.
[0003] Existing enterprises have the following main problems when using Elasticsearch:
[0004] 1. Difficult to learn and operate: Generally, when managing data in Elasticsearch, enterprises need to archive data regularly. The collection of data sets managed in Elasticsearch is called an index. Enterprises need to archive indexes regularly during this process, which is time-consuming and labor-intensive. Indexes need to be created frequently. Creating indexes requires building data structures, the number of index shards, the number of replicas, and the type of tokenizer used, etc. For frequently created repetitive types, it is even more necessary to create index templates. The operation method provided by Elasticsearch is to operate the Elasticsearch cluster and manage data through a restful-style API. This method is very painful for people who don't know much about Elasticsearch to build the required index templates, data streams, indexes, etc. The learning cost is relatively high and the operation is prone to errors.
[0005] 2. Resource management: As mentioned above, enterprises will regularly archive and store Elasticsearch data that has not been accessed for a long time, in order to release resources and provide more available resources for hot data. On the other hand, it is for archiving queries and efficient retrieval.
[0006] Operation and maintenance personnel need to manually perform archiving or policy archiving, and the archived data needs to be managed in a unified manner so that it can be used for query.
[0007] For enterprises, there may be multiple Elasticsearch clusters, which makes it inconvenient to uniformly manage resources, cluster health, and so on.
[0008] 3. Version compatibility: As time goes by, ECMAScript standards are constantly updated and new features are introduced in the process of using Elasticsearch by existing enterprises. For enterprises, one set of code is not compatible with all Elasticsearch version clusters. For enterprise service upgrades, function expansion is limited by the Elasticsearch version.
[0009] In view of the above problems, a data source management method is proposed. Summary of the invention
[0010] The purpose of the present invention is to solve at least one of the technical problems existing in the prior art and to provide a data source management method that can solve the problems of the background technology.
[0011] To achieve the above object, the present invention provides the following technical solution: a data source management method, comprising the following steps:
[0012] Step 1: The preprocessor preprocesses the request, parses metadata information, including cluster name, connection method, and builds the Elasticsearch client;
[0013] Step 2: Import the velocity template engine, parse the request into the corresponding message body through the converter, and construct the request parameters;
[0014] Step 3: Import the httpclient module, access the corresponding Elasticsearch cluster through http requests, get the returned results, process the results uniformly through the parser, and return the data;
[0015] Based on the above functions:
[0016] (1) Integrate Easyexcel and Opencsv plug-ins to parse Csv, Excel and other format files. Use the data import module to put them into the Elasticsearch cluster through the API;
[0017] (2) Nginx+Ik word segmenter implements hot deployment of word library updates, which can be updated in minutes without restarting;
[0018] (3) Use NFS to achieve storage visibility between nodes, and combine with Elasticsearch API to achieve snapshot recovery within the cluster and between clusters;
[0019] (4) Use the front-end components Codemirror and bin-code-editor to verify and format SQL and JSON, and combine with the Elasticsearch API to implement standard SQL retrieval and RESTful interface access.
[0020] Preferably, the Elasticsearch is a distributed search engine based on Lucene, which has functions such as efficient full-text search, real-time data analysis and data visualization, etc. It uses an inverted index to store and search data, can quickly respond to user search requests, and supports horizontal expansion.
[0021] Preferably, the second step starts using the velocity template engine, creates a directory templates under src, and creates a file test.vm under the templates directory.
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] (1) This data source management method is compatible with multiple data source management systems for multiple versions of Elasticsearch clusters. Under the preprocessor and converter, the velocity template engine is combined to generate request message bodies corresponding to different versions, and then interact with the corresponding Elasticsearch through https / http, and the parser uniformly processes and returns. Based on this as the core, the operation difficulty of the Elasticsearch cluster is simplified, and functions such as index management, data stream archive query, data archiving, and popular word library update are realized.
[0024] (2) This data source management method generates different versions of Elasticsearch syntax based on different Elasticsearch sources based on velocity templates. If the business implements data management of different Elasticsearch sources, data retrieval across versions and data sources can be achieved without restarting the project. Compared with the prior art, the Elasticsearch connection parameters are configured when the project starts, and cannot be switched at will after startup, or are limited by the Elasticsearch version and need to switch the dependent version.
[0025] (3) This data source management method lowers the threshold for using Elasticsearch, reasonably creates indexes and data streams, and sets corresponding configuration information such as the number of shards, the number of replicas, mapping, index templates, etc. It maximizes the use of the Elasticsearch cluster, improves resource utilization, reduces duplication of work, and improves work efficiency.
[0026] (4) This data source management method can archive data with one click, migrate data across clusters, and implement strategic archiving. It can release resources occupied by non-hot data and enable targeted and precise search. This can save resources and improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The present invention is further described below in conjunction with the accompanying drawings and embodiments:
[0028] Figure 1 is a schematic diagram of the core components of the present invention;
[0029] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings.
[0032] See also Figure 1-2 The present invention provides a technical solution: a data source management method, comprising the following steps:
[0033] Step 1: The preprocessor preprocesses the request, parses metadata information, including cluster name, connection method, and builds the Elasticsearch client;
[0034] Elasticsearch is a distributed search engine based on Lucene, which has the functions of efficient full-text search, real-time data analysis and data visualization. It uses inverted index to store and search data, can quickly respond to user search requests, and supports horizontal expansion.
[0035] Step 2: Import the velocity template engine, parse the request into the corresponding message body through the converter, and construct the request parameters;
[0036] Start using the velocity template engine, create a directory templates under src, and create a file test.vm in the templates directory with the following content:
[0037] The code is as follows:
[0038]
[0039]
[0040] <meta http-equiv="Content-Type"content="text html;charset=utf-8">
[0041]
[0042]
[0043] #set($this = "Velocity")
[0044] $this is great!
[0045] $name
[0046] hi,i am letian
[0047] <h1> Hello< / h1>
[0048]
[0049]
[0050] Create a new Java file MyVelocity01.java under the src directory:
[0051] The code is as follows:
[0052] import java.io.*;
[0053] import javax.servlet.*;
[0054] import javax.servlet.http.*;
[0055] import org.apache.velocity.app.Velocity;
[0056] import org.apache.velocity.app.VelocityEngine;
[0057] import org.apache.velocity.VelocityContext;
[0058] import java.util.Properties;
[0059] public class MyVelocity01 extends HttpServlet{
[0060] @Override
[0061] public void doGet(HttpServletRequest request, HttpServletResponseresponse)
[0062] throwsIOException,ServletException{
[0063] request.setCharacterEncoding("UTF-8");
[0064] response.setContentType("text / html;charset=UTF-8");
[0065] PrintWriter out=response.getWriter();
[0066] Properties properties=new Properties();
[0067] properties.setProperty("resource.loader","class");
[0068] properties.setProperty("class.resource.loader.class","org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader");
[0069] / / properties.setProperty("input.encoding","UTF-8");
[0070]
[0071] Configure web.xml: The code is as follows:
[0072]
[0073] <servlet-mapping>
[0074] <servlet-name> ve< / servlet-name>
[0075] <url-pattern> / ve< / url-pattern>
[0076] < / servlet-mapping> .
[0077] Step 3: Import the httpclient module and access the corresponding Elasticsearch cluster through http requests. Get the returned results, process the results uniformly through the parser, and return the data;
[0078] Import the HttpClient module
[0079] / / app.module.ts
[0080] import{httpClientModule}from'@angular / common / http';
[0081] imports:[
[0082] httpClientModule ]
[0084] 2) Inject the HttpClient service instance object to send the request
[0085] / / app.component.ts
[0086] import{HttpClient}from'@angular / common / http';
[0087] export class AppComponent{
[0088] constructor(private http:HttpClient){}
[0089] }
[0090] 3) Send a request
[0091] import{HttpClient}from"@angular / common / http"
[0092] export class AppComponent implements OnInit{
[0093] constructor(private http:HttpClient){}
[0094] ngOnInit(){
[0095] this.getUsers().subscribe(console.log)
[0096] }
[0097] getUsers(){
[0098] return this.http.get("https: / / jsonplaceholder.typicode.com / users")
[0099] }
[0100] }
[0101] 2. Request method
[0102] this.http.get(url[, options]);
[0103] this.http.post(url, data[, options]);
[0104] this.http.delete(url[, options]);
[0105] this.http.put(url, data[, options]);
[0106] this.http.get<Post[]>(' / getAllPosts')
[0107] .subscribe(response => console.log(response))
[0108] 3. Request parameters
[0109] 1. HttpParams class
[0110] export declare class HttpParams {
[0111] constructor(options?: HttpParamsOptions);
[0112] has(param: string): boolean;
[0113] get(param: string): string | null;
[0114] getAll(param: string): string[] | null;
[0115] keys(): string[];
[0116] append(param: string, value: string): HttpParams;
[0117] set(param: string, value: string): HttpParams;
[0118]
[0119] Generate different versions of Elasticsearch syntax based on different Elasticsearch sources based on velocity templates. If the business implements data management of different Elasticsearch sources, data retrieval across versions and data sources can be achieved without restarting the project. Configure Elasticsearch connection parameters when the project starts, and you cannot switch at will after startup, or you need to switch the dependent version due to restrictions on the Elasticsearch version.
[0120] The usage process of Elasticsearch.
[0121] Index: An index is a logical concept, similar to a table in a database. It is a logical grouping of documents with similar characteristics. Each index has a unique name and is used to store, search, and aggregate data in Elasticsearch.
[0122] Document: A document is the basic unit of data in Elasticsearch. It is a structured data object represented in JSON format. A document can be any type of data, such as product information, user records, log entries, etc. Each document has a unique ID in the index to identify and retrieve it.
[0123] Field: A field is a specific data item in a document. It consists of a field name and a corresponding value. Fields can be of various types, such as strings, numbers, dates, Boolean values, etc. In Elasticsearch, fields are dynamically mapped to specific types, or you can specify the mapping manually.
[0124] Mapping: Mapping defines the structure of documents and the types of fields in the index. It defines information such as the name, data type, index settings, and analyzer of the field. Mapping allows Elasticsearch to index and search text data according to specified rules.
[0125] Shards and Replicas: Elasticsearch divides the index into multiple shards. Each shard is a subset of the index and contains a portion of the index's data. Sharding allows the index to be horizontally expanded and processed in parallel in the cluster. In addition, each shard can have one or more replicas to provide high availability and fault recovery capabilities.
[0126] Query: Query refers to the operation of searching and retrieving documents in the index. Elasticsearch provides a rich set of query functions, including full-text search, exact match, range query, aggregate query, etc. You can use query DSL (Domain Specific Language) to build complex queries.
[0127] Aggregation: Aggregation is the operation of grouping, filtering, and calculating documents. It can be used to generate statistical information, analyze data distribution, perform data analysis, etc. Aggregation can classify documents according to various conditions and generate summary results.
[0128] Cluster: A cluster is a distributed environment consisting of multiple Elasticsearch nodes. A node can be a master node, a data node, or a coordinator node. A cluster provides features such as high availability, load balancing, and data replication.
[0129] Tokenization: Tokenization is the process of splitting text data into meaningful words (terms). In Elasticsearch, when a document is indexed, the text field is tokenized by the analyzer into a series of terms for full-text search and query.
[0130] Analyzer: An analyzer is a processing chain consisting of character filters, tokenizers, and token filters. It defines how text is processed and tokenized during indexing and searching.
[0131] Inverted Index: Inverted Index is the core data structure used to achieve fast search in Elasticsearch. It achieves fast reverse search from term to document by mapping each term to the document containing the term.
[0132] Routing: Routing is the process of deciding which shard a document will be stored in. Elasticsearch uses the document ID and a routing algorithm to determine which shard the document should be assigned to.
[0133] Search Suggestion: Search suggestion is a feature that provides relevant suggestions and auto-completion when users enter search queries. Elasticsearch provides the search suggestion feature to improve user experience and the accuracy of search results.
[0134] Real-time Search: Real-time search refers to the ability to search and return the latest results immediately after the document is indexed. Elasticsearch supports real-time search, allowing you to monitor and retrieve the latest data in real time.
[0135] Client Libraries: Elasticsearch provides official client libraries in multiple programming languages, making it easy for developers to interact and integrate with Elasticsearch. These client libraries provide convenient ways to communicate with Elasticsearch's API.
[0136] On this basis, the functions are realized:
[0137] 1. Integrate Easyexcel and Opencsv plug-ins to parse Csv, Excel and other format files. Use the data import module to put them into the Elasticsearch cluster through the API.
[0138] 2. Nginx+Ik word segmenter realizes hot deployment of word library update, which can be updated in minutes without restarting.
[0139] 3. Use NFS to achieve storage visibility between nodes, and combine with Elasticsearch API to implement snapshot recovery within and between clusters.
[0140] 4. Use the front-end components Codemirror and bin-code-editor to verify and format sql and json, and combine with Elasticsearch API to implement standard sql retrieval and restful interface access.
[0141] For enterprises that have been using Elasticsearch for a long time, as time goes by, business needs change, and they need to introduce a new version of Elasticsearch for business expansion. In order to be compatible with previous businesses, not only the Elasticsearch data needs to be migrated, but also the code and business compatibility needs to be achieved. The official ElasticsearchClient on the market has a strong dependency on the Elasticsearch version, and the compatibility between major versions is poor. Compared with this solution, there is no need to rewrite the code, and it is seamlessly compatible.
[0142] Lower the threshold for using Elasticsearch, create indexes and data streams reasonably, and set corresponding configuration information such as the number of shards, the number of replicas, mapping, index templates, etc. Maximize the use of Elasticsearch clusters, improve resource utilization, reduce duplication of work, and improve work efficiency.
[0143] Monitor and manage Elasticsearch cluster index-related disk usage, document count, index health status, and other information. Provide alarms for sub-healthy and error-state indexes and data streams, and push them to internal and external sites.
[0144] One-click data archiving, cross-cluster data migration, and policy archiving. Free up resources occupied by non-hotspot data, and conduct targeted and precise searches. This saves resources and improves efficiency.
[0145] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A data source management method, characterized in that: The following steps are involved: Step 1: The preprocessor preprocesses the request, parses metadata information, including cluster name, connection method, and builds the Elasticsearch client; Step 2: Import the velocity template engine, parse the request into the corresponding message body through the converter, and construct the request parameters; Step 3: Import the httpclient module, access the corresponding Elasticsearch cluster through http requests, get the returned results, process the results uniformly through the parser, and return the data; Based on the above functions: (1) Integrate Easyexcel and Opencsv plug-ins to parse Csv, Excel and other format files. Use the data import module to put them into the Elasticsearch cluster through the API. (2) Nginx+Ik word segmenter implements hot deployment of word library updates, which can be updated in minutes without restarting. (3) Use NFS to achieve storage visibility between nodes, and combine with Elasticsearch API to implement snapshot recovery within and between clusters. (4) Use the front-end components Codemirror and bin-code-editor to verify and format SQL and JSON, and combine with the Elasticsearch API to implement standard SQL retrieval and RESTful interface access.
2. A data source management method according to claim 1, characterized in that: The Elasticsearch is a distributed search engine based on Lucene, which has the functions of efficient full-text search, real-time data analysis and data visualization, etc. It uses an inverted index to store and search data, can quickly respond to user search requests, and supports horizontal expansion.
3. A data source management method according to claim 2, characterized in that: The second step starts using the velocity template engine, creates a directory templates under src, and creates a file test.vm under the templates directory.
Citation Information
Cited By
Multi-cluster and multi-cluster index management method and system, electronic equipment and medium
CN120804392A
