Performance optimization system for real-time memory retrieval based on reverse index

Abstract

The technical scheme of the invention provides a performance optimization system for real-time memory retrieval based on an inverted index. After the technical scheme disclosed by the invention is adopted, the average response time of an interface is less than 10ms under the condition of multi-field combined query (OR and AND) under the condition of 1 billion data volume in a high-frequency write operation service. According to the query optimization system based on the memory inverted index technology disclosed by the invention, the efficient inverted index structure is constructed in the memory, and the AND and OR operations between the fields are accelerated by utilizing the structure, so that the complex query performance of a large table is remarkably improved. The query efficiency is remarkably improved; the expandability of the system is enhanced; and user experience is improved.

Description

Technical Field

[0001] The present invention relates to a multi-condition memory-based query system under large data volume, and in particular to a performance optimization system for real-time memory retrieval based on inverted index. Background Art

[0002] MySQL's large table multi-condition query can optimize query speed by establishing appropriate joint indexes, but in actual work, when the data volume of a single table reaches more than 10 million, even if the index is complete, the efficiency of multi-condition query is not ideal. For example: to query the news in the following news table, the industry is electricity and finance, the listing code is 6000101.SH, and the title contains electricity. The SQL statement is as follows: select ID from NEWS where (industry = electricity OR industry = finance) AND code = 600101.SH AND title like '%Electricity%'.

[0003] News ID News Headlines News Industry Listing Code 9300177 MSCI Asia Pacific Index rose 2% finance APEX 9300178 Mingxing Electric Power hits the daily limit, Xichang Electric Power hits the daily limit electricity 600101.SH 9300179 Japanese Yen, breaking news! finance UBS.N

[0004] When the news data volume is 10 million, we query through MySQL's in-memory database, and the performance is as follows:

[0005] Query conditions 1 concurrent 10 concurrent AND*1 5ms 15ms AND*50 1s 100s+ AND*100 5s 300s+

[0006] Such performance does not meet our requirements. In actual production, news attributes are much more than in the example, so the performance cannot meet the requirements. We need to achieve a query performance of 10ms under 100 concurrency and AND*100. Especially when performing IN operations on multiple tables through SQL statements, sorting the result set will be a nightmare. MySQL is very slow in sorting query results and cannot support massive data (>100 million). In the prior art, although inverted index technology has been widely used in search engines and other scenarios to improve text search efficiency, there is no mature and efficient solution in the field of database query optimization with vertical content and clear structure. Summary of the invention

[0007] The technical problem to be solved by the present invention is: to ensure that in a business with high-frequency write operations, a business multi-field combination query (OR and AND) is performed at a data level of 100 million, and the average response time of the interface is below 10ms, thereby achieving a fast response of the query interface.

[0008] In order to solve the above technical problems, the technical solution of the present invention provides a performance optimization system for real-time memory retrieval based on inverted index, characterized in that the overall service architecture of the performance optimization system includes:

[0009] The ORACLE database used to store news data can provide three operations of adding, modifying and deleting data in the ORACLE database through various storage programs; the performance optimization system monitors the log of adding, modifying and deleting news data tables, and monitors the data changes in the ORACLE database through the CDC system. When the title, content, author, data source, industry and listed company of the news change, these contents and the type of data change are sent to the corresponding Topic of KAFKA, and one Topic is used to store all types of change messages; when the data query cache service is started, the full amount of data is read from the ORACLE database to build a cache. During the normal operation of the service, the data query cache service performs data addition and update processing through the CDC message service, and accepts the CDC message in the Topic of KAFKA to update its own cache; the data query cache service provides cache query service to the outside through the TCP protocol, and queries the list of qualified news through JSON format commands; the data full-text retrieval service uses the KEY-VALUE form to store the entire news, obtains the qualified news ID through the data query cache service, and transmits the news ID to the data full-text retrieval service to obtain the corresponding news text.

[0010] Preferably, the news types of the news data include important news, cooperative news and Internet news, wherein:

[0011] The data of important news comes from the editing of editors. News editors use a special data publishing program to enter the important news into the ORACLE database;

[0012] Cooperative news is pushed by news media partners. News media partners push news data to the news FTP service, and the dedicated news synchronization storage service stores the cooperative news into the ORACLE database;

[0013] Internet news collects data through common Internet programs, and Internet news is stored in the ORACLE database through common Internet data collection services.

[0014] Preferably, the message retention period of the Topic is set to 7 days. Once the period is reached, the old messages will be deleted.

[0015] Preferably, during normal operation of the data query cache service, when the data query cache service updates the cache: for newly added messages, the inverted index items are determined and the document ID is added; for deleted messages, the index items are determined and the document ID is deleted; for updated messages, the index items of the original messages need to be determined, all document IDs are deleted, and then the newly added index items are calculated and the document ID is added.

[0016] Preferably, when the data query cache service builds a cache, a memory inverted index structure is used to cache data, wherein a memory cache instance is composed of dimensions and codes: dimensions are attributes of news data, including news type dimensions, title dimensions, source dimensions, industry dimensions, and listed company code dimensions; dimensions are composed of codes, and one dimension can contain N codes, which are N attribute values ​​of the attribute corresponding to the current dimension, and each code represents a collection of news IDs of all news data related to the current attribute value under the current dimension. Then, building an index structure includes the following steps:

[0017] Step 1: Establish a connection with the ORACLE database;

[0018] Step 2: Dimension dictionary construction: Extract source dimension, industry dimension, listed company code dimension, title dimension, text dimension and custom label classification dimension from the database dimension table, and assign a dimension ID to each unique dimension;

[0019] Step 3: Read news data: Use predefined SQL to extract news ID, news title, content, author, data source, industry, listed company and the text content of the corresponding news from the ORACLE database through streaming query;

[0020] Step 4: News document processing - word cleaning: For the read news data, remove stop words, perform preprocessing, and perform a standardization process on the words;

[0021] Step 5: News document processing - word segmentation: perform word segmentation and word stem extraction on the read news data, and assign a unique integer news ID to each news document;

[0022] Step 6: Constructing the code of vocabulary type: For the title dimension and the text dimension, traverse the news, segment the title and text of the news, extract all the news words, and use each news word as a key. The value corresponding to the key is a list, which is the ID of all news containing the word.

[0023] CODE construction of attribute type: listed company code dimension, source dimension, industry dimension and custom label classification dimension. The specific value of the dimension is used as the key, and the value corresponding to the key is a list, which is a list of all news IDs involving this key value;

[0024] Step 7: Summarize the attribute type CODE and the vocabulary type CODE, and assign a CODEID to each unique CODE;

[0025] Step 8, Index compression: compress the index, CODE uses integer encoding;

[0026] Step 9: Establish the parent-child relationship between dimension and CODE. One dimension contains N CODEs.

[0027] Step 10: Index update strategy: When new news is added or the content of news is updated, the inverted index is updated, and an incremental update strategy is adopted to update only the affected news;

[0028] Step 11: Index storage: store the constructed inverted index in memory;

[0029] Step 12: Index query optimization: Implement various optimization measures to improve query performance;

[0030] Step 13, Index maintenance: Regularly maintain the index, clean up unused CODE, optimize the storage structure, and monitor the size and performance of the index, and make adjustments as needed.

[0031] Preferably, the data query cache service receives query requests and processes them in the following order:

[0032] Parse the query conditions in JSON format and extract the fields involved from the corresponding query commands in JSON format;

[0033] Through lexical analysis: split the input logical expression string into tokens, including words, brackets, operators, and equal signs;

[0034] Through grammatical analysis: construct an expression tree based on the priority of operators and the nesting relationship of parentheses;

[0035] Traversing the expression tree: traverse the expression tree through depth-first search, calculate the value of the expression, and convert it into AND and OR logical expressions; use the inverted index in memory to quickly locate the news ID set that meets a single query condition, perform intersection or union operations on the news ID sets of multiple query conditions, and obtain the final query result:

[0036] For the union operation, a red-black tree algorithm is used to merge sets. At the same time, for title search, a max-heap algorithm is used for optimization. The max-heap algorithm uses a max-heap to maintain a fixed-size heap, where the top of the heap is always the most relevant news data. As the search algorithm proceeds, new news data is evaluated and may be compared with the news data in the heap: if the new news data is more relevant than the news data at the bottom of the heap, the new news data can be added to the heap and the documents at the bottom of the heap are removed;

[0037] For the intersection operation, a small column filtering algorithm and a hash algorithm are used. The Bloom filter algorithm is used to optimize the problem of the hash algorithm occupying more space. For the operation of A AND B AND C, the small column filtering algorithm is first used to filter the items on each CODE to reduce the search range. Then, the Bloom filter algorithm is used to determine whether an element may exist in a set. If it may exist in the set, the intersection is calculated.

[0038] Pre-sort the memory index and use the index sorting feature to reduce the number of comparisons. In addition, parallel processing technology is also used in the program to accelerate set operations.

[0039] The inverted index is converted to an integer value, effectively reducing memory usage.

[0040] After adopting the technical solution disclosed in the present invention, in the business of high-frequency write operations, the average response time of the interface for multi-field combination query (OR and AND) under the condition of 100 million data items is less than 10ms. The query optimization system based on the inverted index technology disclosed in the present invention significantly improves the performance of complex queries on large tables by constructing an efficient inverted index structure in memory and using the structure to accelerate AND and OR operations between fields.

[0041] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0042] 1) Significantly improve query efficiency: Through the inverted index structure in memory, fast positioning and retrieval of large table data is achieved, significantly improving the query efficiency of AND and OR operations between fields. Reduce system resource consumption: Reduce disk I / O operations, reduce CPU and memory usage, and improve the overall performance of the system;

[0043] 2) Enhance system scalability: Support parallel processing technology, which can easily cope with larger data volumes and more complex query requirements;

[0044] 3) Improve user experience: shorten query response time and improve user satisfaction and efficiency when using the database system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The overall service architecture of the present invention is illustrated;

[0046] Figure 2 It illustrates the in-memory inverted index structure. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0048] The overall service architecture of a performance optimization system based on real-time memory retrieval based on an inverted index disclosed in an embodiment of the present invention is as follows: Figure 1 As shown, including:

[0049] Relevant news data is stored in the ORACLE database. News types include important news, cooperative news and Internet news. Important news data comes from editors, cooperative news is pushed by news media partners, and Internet news is collected through Internet general programs. News editors use a dedicated data publishing program to enter important news releases into the ORACLE database. News media partners push news data to the news FTP service, and the dedicated news synchronization storage service stores cooperative news in the ORACLE database. Internet news is stored in the ORACLE database through the Internet general data collection service. The daily data volume of news from various channels is more than 100,000.

[0050] The warehousing procedures in various ways can provide three operations of adding, modifying and deleting data in the ORACLE database. The system monitors the addition, modification and deletion of the news data table through logs. Through the CDC system, the data changes in the ORACLE database are monitored. To capture changes of the insert, update and delete types, the CDC strategy is implemented through database log monitoring. The title, content, author, data source, industry and listed company of the news are the most important contents. When the data changes, these contents and the type of data change will be sent to the corresponding Topic of KAFKA. Change event format: Each change event will be represented in JSON format, which contains detailed information of the change. The event will contain the original data, new data, and value information before and after the change. Use a Topic to store all types of change messages. Data retention and expiration policy: Set the message retention time of the Topic to 7 days. Once reached, the old message will be deleted. Ensure that Kafka does not grow indefinitely, and at the same time control the expiration time of the data.

[0051] When the data query cache service starts, it reads data from the ORACLE database and builds a cache. When the service starts, it reads all the data from the ORACLE database (the program supports setting the time range for reading data). The data query cache service only loads the full data at startup. During the normal operation of the service, the data in the ORACLE database will change. These daily data changes are processed by the CDC message service for data addition and update. The program accepts CDC messages in KAFKA's Topic to update its own cache: for new messages, it determines the inverted index items and adds the document ID; for deleted messages, it determines the index items and deletes the document ID; for updated messages, it is necessary to determine the index items of the original message, delete all document IDs, and then calculate the newly added index items and add the document ID.

[0052] The data query cache service provides cache query services to the outside world through the TCP protocol. Use JSON format commands to query the list of news that meet the conditions. For example: the data query cache service inputs a combination of query conditions such as title, content, industry, company, etc., and returns a list of news that meet the conditions (including ID).

[0053] The data full-text retrieval service uses the KEY-VALUE format to store the entire news. The news text can be queried through the news ID. For example: the client program queries the news list, first uses the news cache query service to query and obtain the list of qualified news. If a specific news needs to be viewed, the data full-text retrieval service is called through the news ID to obtain the text content of the news.

[0054] When the data query cache service builds a cache, it uses a memory inverted index structure to cache data, such as Figure 2As shown. The memory cache instance consists of dimensions and codes. The detailed construction process is explained using the example of news data. News data includes title, source, industry, and listed company attributes. These attributes are dimensions. Dimensions are divided into news type dimension, title dimension, source dimension, industry dimension, and listed company code dimension. Dimensions are composed of codes. A dimension can contain N codes. Code represents a component of a dimension. For example, in the industry dimension, Code includes specific industries, such as finance, electricity, and medicine. Each code represents the set of IDs of all news involved in this industry. For example, in the listed company code dimension, the listed company dimension includes the set of codes of all listed companies. The code of the listed company dimension is the inverted index represented by the codes of all listed companies. For example, 000001.SZ is a code under the listed company dimension. This code is an inverted index. It is a deduplication set. The set is composed of the IDs of the news involved. For example, all news IDs related to the 000001.SZ company are all placed in the set named by this code. For example, in the title dimension, a code is a set represented by a word involved in the document. This set is composed of the IDs of all news containing this word. Through the inverted index, we can quickly and accurately locate the news list that matches the query term, thereby greatly improving the query performance.

[0055] Building an index structure is a critical step in data cache query services. It directly determines the efficiency and accuracy of queries. Building an index structure includes the following steps:

[0056] 1. Establish a connection with the ORACLE database: Use dynamic password technology to obtain the password of the ORACLE database, and use the ORACLE database driver to establish a connection and pass the security verification.

[0057] 2. Dimension dictionary construction: Extract source dimension, industry dimension, listed company code dimension, title dimension, text dimension and custom label classification dimension from the database dimension table. For each unique dimension, assign a dimension ID. Use a hash table to store this dimension information.

[0058] 3. Word segmentation tool selection: Choose the appropriate word segmentation tool according to the language of the news data. For Chinese, use the Jieba word segmentation tool; for English, use the NLTK library.

[0059] 4. News data reading: Use predefined SQL to extract news ID, news title, content, author, data source, industry, listed company and the text content of the corresponding news from the ORACLE database through streaming query.

[0060] 5. News document processing - word cleaning: For the read news data, remove stop words, perform preprocessing, and perform a standardization process on the words (such as lowercase conversion, punctuation removal, etc.) to enhance the relevance of search results.

[0061] 6. News document processing - word segmentation: perform word segmentation and word stem extraction on the read news. Assign a unique integer document ID to each news document.

[0062] 7. Code construction of vocabulary type: For the title dimension and the text dimension, traverse the news, segment the title and text of the news, extract all the news words, take each news word as the key, and the value corresponding to the key is a list, which is the ID of all news containing the word.

[0063] CODE construction of attribute type: listed company code dimension, source dimension, industry dimension and custom label classification dimension. The specific value of the dimension is used as the key, and the value corresponding to the key is a list, which is a list of all news IDs involving this key value. For example, in the listed company dimension, the key corresponds to the ID of the listed company, and the value is a list of IDs of all news involving this listed company. Create a document list, which records all document IDs containing the CODE and the location information of the CODE in the document. Use the Skip List data structure to store this information.

[0064] 8. Summarize the attribute type CODE and vocabulary type CODE, and assign a CODE ID to each unique CODE. Use a hash table to store CODE.

[0065] 9. Index compression: In order to save storage space and improve query efficiency, the index is compressed. CODE uses integer encoding. The purpose of encoding is to optimize the index structure, reduce memory usage, and improve query efficiency.

[0066] 10. Establish the parent-child relationship between dimension and CODE. One dimension contains N CODEs.

[0067] 11. Index update strategy: When new news is added or the content of news is updated, the inverted index needs to be updated. An incremental update strategy is used to update only the affected news.

[0068] 12. Index storage: Store the constructed inverted index in memory for fast access.

[0069] 13. Index query optimization: To improve query performance, various optimization measures can be implemented. Use index pruning to return only the most relevant document lists. Use cache to store frequently queried vocabulary and news lists.

[0070] 14. Index maintenance: Maintain the index regularly, clean up unused CODEs, and optimize the storage structure. Monitor the size and performance of the index and make adjustments as needed.

[0071] Through the above steps, the present invention constructs an efficient and accurate memory-based inverted index. This index structure will serve as the basis for query services and support fast and accurate text searches.

[0072] The data query cache service provides query services through the TCP protocol. Its interface uses the JSON format and is as follows: {industry:[Electricity, Finance]code=600101.SH title=Electricity}. The service query returns the IDs of all documents that meet the conditions. The data full-text retrieval service provides query services through the TCP protocol. The parameter of its interface is the document ID, and the result returned is the full text of the document. The ID of the document that meets the conditions is obtained through the data query cache service, and the document ID is passed to the data full-text retrieval service to obtain the content of the corresponding document.

[0073] The data query cache service receives query requests and processes them in the following order:

[0074] First, parse the query conditions in JSON format and extract the fields involved from the corresponding query commands in JSON format.

[0075] Second, use expression trees to perform lexical analysis: split the input logical expression string into tokens, including words, brackets, operators (AND, OR), equal signs, etc. Perform syntactic analysis: construct an expression tree based on the priority of operators (usually brackets are the highest, followed by AND, and finally OR) and the nesting relationship of brackets.

[0076] Third, traverse the expression tree (Traversal): traverse the expression tree through depth-first search, calculate the value of the expression, and convert it into AND and OR logical expressions. Use the inverted index in memory to quickly locate the record ID set that meets a single query condition. Perform intersection (AND operation) or union (OR operation) operations on the record ID sets of multiple query conditions to obtain the final query result.

[0077] Fourth, for OR operations, a red-black tree algorithm is used to merge sets. At the same time, for title searches, a max-heap algorithm is used for optimization. The specific scenario is when it comes to OR operations involving inverted indexes that require fast access and processing of a large amount of priority data, a max-heap algorithm is used for optimization. For example, if a user searches for "fund OR financial management", the search will return all documents containing "fund" or "financial management" or both. The max-heap can be used to optimize performance when processing search results, especially when it is necessary to sort search results or select the top N most relevant documents. After performing an OR operation, the system will return a large number of candidate documents. If it is necessary to sort by the relevance of the documents, the max-heap is used to maintain a fixed-size heap, in which the top of the heap is always the most relevant document at the moment. As the search algorithm proceeds, new documents are evaluated and may be compared with documents in the heap. If the new document is more relevant than the document at the bottom of the heap, the new document can be added to the heap and the document at the bottom of the heap can be removed.

[0078] For AND operations, a small column filtering algorithm and a hash algorithm are used, and the Bloom filter algorithm is used to optimize the problem of the high space usage of the hash algorithm. For the operation of A AND B AND C, the small column filtering algorithm is first used to filter the items on each CODE to reduce the search range. For example, for the operation of A AND B, first select the smaller set of A and B, apply the filtering conditions, and filter to further reduce the number of elements in the set. The obtained set is sorted and filtered, and then the Bloom filter algorithm is used to determine whether an element may exist in a set (but it cannot be determined that an element must exist in a set). If it may exist in the set, the program calculates the intersection. Although the same effect can be achieved by using a hash table, using a Bloom filter to replace the hash table can reduce the use of space.

[0079] Fifth, pre-sorting the memory index greatly improves the query speed, and uses the sorting characteristics of the index to reduce the number of comparisons. In addition, parallel processing technology is also used in the program to accelerate set operations.

[0080] Sixth, the inverted index is converted to an integer value, which effectively reduces the memory usage. After the system is launched, the actual running performance is shown in the following table:

[0081] Query conditions 1 concurrent 10 concurrent 100 concurrent AND*1 1ms 1ms 1ms AND*50 1ms 1ms 1ms AND*100 1ms 1ms 1ms AND*500 2ms 2ms 2ms

Claims

1. A performance optimization system for real-time memory retrieval based on inverted index, characterized in that: The overall service architecture of the performance optimization system includes: The ORACLE database used to store news data can provide three operations of adding, modifying and deleting data in the ORACLE database through various storage programs; the performance optimization system monitors the log of adding, modifying and deleting news data tables, and monitors the data changes in the ORACLE database through the CDC system. When the title, content, author, data source, industry and listed company of the news change, these contents and the type of data change are sent to the corresponding Topic of KAFKA, and one Topic is used to store all types of change messages; when the data query cache service is started, the full amount of data is read from the ORACLE database to build a cache. During the normal operation of the service, the data query cache service performs data addition and update processing through the CDC message service, and accepts the CDC message in the Topic of KAFKA to update its own cache; the data query cache service provides cache query service to the outside through the TCP protocol, and queries the list of qualified news through JSON format commands; the data full-text retrieval service uses the KEY-VALUE form to store the entire news, obtains the qualified news ID through the data query cache service, and transmits the news ID to the data full-text retrieval service to obtain the corresponding news text.

2. A performance optimization system for real-time memory retrieval based on inverted index as claimed in claim 1, characterized in that: The news types of the news data include important news, cooperative news and Internet news, among which: The data of important news comes from the editing of editors. News editors use a special data publishing program to enter the important news into the ORACLE database; Cooperative news is pushed by news media partners. News media partners push news data to the news FTP service, and the dedicated news synchronization storage service stores the cooperative news into the ORACLE database; Internet news collects data through common Internet programs, and Internet news is stored in the ORACLE database through common Internet data collection services.

3. A performance optimization system for real-time memory retrieval based on inverted index as claimed in claim 1, characterized in that: Set the Topic message retention period to 7 days. Once the period is reached, the old messages will be deleted.

4. A performance optimization system for real-time memory retrieval based on inverted index as claimed in claim 1, characterized in that: During normal operation of the data query cache service, when the data query cache service updates the cache: for newly added messages, the inverted index items are determined and the document ID is added; for deleted messages, the index items are determined and the document ID is deleted; for updated messages, the index items of the original messages need to be determined, all document IDs are deleted, and then the newly added index items are calculated and the document ID is added.

5. The performance optimization system for real-time memory retrieval based on inverted index according to claim 1, characterized in that: When the data query cache service builds a cache, it uses a memory inverted index structure to cache data, wherein the memory cache instance is composed of dimensions and codes: dimensions are attributes of news data, including news type dimensions, title dimensions, source dimensions, industry dimensions, and listed company code dimensions; dimensions are composed of codes, and one dimension can contain N codes, which are N attribute values ​​of the attribute corresponding to the current dimension, and each code represents the set of news IDs of all news data related to the current attribute value under the current dimension. Then, building the index structure includes the following steps: Step 1: Establish a connection with the ORACLE database; Step 2: Dimension dictionary construction: Extract source dimension, industry dimension, listed company code dimension, title dimension, text dimension and custom label classification dimension from the database dimension table, and assign a dimension ID to each unique dimension; Step 3: Read news data: Use predefined SQL to extract news ID, news title, content, author, data source, industry, listed company and the text content of the corresponding news from the ORACLE database through streaming query; Step 4: News document processing - word cleaning: For the read news data, remove stop words, perform preprocessing, and perform a standardization process on the words; Step 5: News document processing - word segmentation: perform word segmentation and word stem extraction on the read news data, and assign a unique integer news ID to each news document; Step 6: Constructing the code of vocabulary type: for the title dimension and the text dimension, traverse the news, segment the title and text of the news, extract all the news vocabulary, take each news vocabulary as the key, and the value corresponding to the key is a list, which is the ID of all news containing the vocabulary; CODE construction of attribute type: listed company code dimension, source dimension, industry dimension and custom label classification dimension. The specific value of the dimension is used as the key, and the value corresponding to the key is a list, which is a list of all news IDs involving this key value; Step 7: Summarize the attribute type CODE and the vocabulary type CODE, and assign a CODEID to each unique CODE; Step 8, Index compression: compress the index, CODE uses integer encoding; Step 9: Establish the parent-child relationship between dimension and CODE. One dimension contains N CODEs. Step 10: Index update strategy: When new news is added or the content of news is updated, the inverted index is updated, and an incremental update strategy is adopted to update only the affected news; Step 11: Index storage: store the constructed inverted index in memory; Step 12: Index query optimization: Implement various optimization measures to improve query performance; Step 13, Index maintenance: Regularly maintain the index, clean up unused CODE, optimize the storage structure, and monitor the size and performance of the index, and make adjustments as needed.

6. A performance optimization system for real-time memory retrieval based on inverted index as claimed in claim 1, characterized in that: The data query cache service receives query requests and processes them in the following order: Parse the query conditions in JSON format and extract the fields involved from the corresponding query commands in JSON format; Through lexical analysis: split the input logical expression string into tokens, including words, brackets, operators, and equal signs; Through grammatical analysis: construct an expression tree based on the priority of operators and the nesting relationship of parentheses; Traversing the expression tree: traverse the expression tree through depth-first search, calculate the value of the expression, and convert it into logical expressions of AND and OR; By using the inverted index in memory, we can quickly locate the news ID set that meets a single query condition, and perform intersection or union operations on the news ID sets of multiple query conditions to obtain the final query result: For the union operation, a red-black tree algorithm is used to merge sets. At the same time, for title search, a max-heap algorithm is used for optimization. The max-heap algorithm uses a max-heap to maintain a fixed-size heap, where the top of the heap is always the most relevant news data. As the search algorithm proceeds, new news data is evaluated and may be compared with the news data in the heap: if the new news data is more relevant than the news data at the bottom of the heap, the new news data can be added to the heap and the documents at the bottom of the heap are removed; For the intersection operation, a small column filtering algorithm and a hash algorithm are used. The Bloom filter algorithm is used to optimize the problem of the hash algorithm occupying more space. For the operation of A AND B AND C, the small column filtering algorithm is first used to filter the items on each CODE to reduce the search range. Then, the Bloom filter algorithm is used to determine whether an element may exist in a set. If it may exist in the set, the intersection is calculated. Pre-sort the memory index and use the index sorting feature to reduce the number of comparisons. In addition, parallel processing technology is also used in the program to accelerate set operations. The inverted index is converted to an integer value, effectively reducing memory usage.

Images