Real-time data stream processing and dynamic data retrieval platform
By analyzing the access frequency of user search results in the data retrieval service system, identifying and localizing backup and storage of hot data, the problems of long response time and high system load are solved, and the rapid acquisition and retrieval of high-frequency access data are achieved. Optimization of efficiency.
Patent Information
- Application Number
- CN202411423485.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When the data volume and request volume increase, existing data retrieval service systems lead to long response time, high system load, poor user experience, and inability to effectively identify and utilize high-frequency accessed data.
A real-time data flow processing and dynamic data retrieval platform was designed, including user interface module, search engine module, hot spot detection module, data storage module, data management module, data identification module, data synchronization module and monitoring module. Through analysis of search results to identify hotspot data and store locally, reduce the access load of the corresponding database of hotspot data, and optimize retrieval efficiency.
It realizes timely reading and retrieval results of hot spot data, reduces system load, improves user experience, and ensures rapid acquisition of high-frequency access data.
Smart Images

Figure CN118939673A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a real-time data stream processing and dynamic data retrieval platform, in particular to a real-time data stream processing and dynamic data retrieval platform applied in the field of data retrieval platforms. Background Art
[0002] In traditional data retrieval service systems, user retrieval requests are usually sent directly to the database for query, which can meet the needs when the amount of data is small. However, with the rapid growth of data volume, directly querying the database will lead to problems such as long response time, high system load, and poor user experience. In addition, traditional methods often ignore the analysis of user behavior data and cannot effectively identify and utilize hot data to optimize retrieval performance.
[0003] In order to solve the problem of real-time data retrieval by the retrieval platform, a certain retrieval platform in the market adopts the design of real-time access to the source database, which has a certain market share.
[0004] Chinese invention patent CN112435022B discloses a dynamic retrieval system and method based on real-time user data, which builds a user data model for user data in heterogeneous data sources and maps the user data to user data model parameters for storage; monitors and obtains existing or changed user data in heterogeneous databases; routes the obtained user data to specify data processing nodes, and stores messages for consumption by one or a group of downstream; deploys message processing tasks based on big data real-time processing technology to achieve large-scale concurrent processing of messages; and provides data persistence and retrieval services based on user data models. It can realize on-demand payment for tenants for this function, and the high real-time performance can realize rapid collection of data from different business dimensions, processing and integration, and rapid provision to customer-oriented staff such as sales, customer service, and marketing, and can achieve fast response speed for real-time user data.
[0005] In existing data retrieval service systems, user search requests are usually sent directly to the database for query. As the amount of data and the number of requests increase, directly querying the database is likely to lead to problems such as long response time, high system load, and poor user experience. Existing data retrieval service systems cannot effectively identify and utilize frequently accessed data in the search results. For frequently accessed data, it is difficult for some users to obtain its complete data in a timely manner after the search. Summary of the invention
[0006] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is that in the existing data retrieval service system, user retrieval requests are usually sent directly to the database for query. As the amount of data and the number of requests increase, directly querying the database is likely to lead to problems such as long response time, high system load, and poor user experience. The existing data retrieval service system cannot effectively identify and utilize the frequently accessed data in the retrieval results.
[0007] To solve the above problems, the present invention provides a real-time data stream processing and dynamic data retrieval platform, including: a user interface module for providing an interface for user search input and display of search results; The search engine module is used to process user search requests and provide search services for the database associated with the search platform; Hotspot detection module, used to record and analyze the access frequency of each search result and identify hotspot data; Hot data includes search result source data with access frequency higher than the set identification value; The data storage module includes a hotspot database and a warm database; the hotspot database includes a distributed database; The data management module is used to manage the data storage module. The specific functions include: clearing hot data and warm data and storing hot data in segments; A data identification module is used to identify hot data and warm data; the data identification module is provided with a warm data list and a hot data list; A data synchronization module is used to monitor the database update log associated with the retrieval platform and synchronously update the data in the data storage module; The monitoring module is used to monitor the system operation status and system performance indicators. The system performance indicators include system load and memory usage.
[0008] In the above-mentioned real-time data stream processing and dynamic data retrieval platform, the hot data in the retrieval results are analyzed and locally backed up and stored. The hot data in the retrieval results are provided with auxiliary access nodes through local backup to reduce the access load of the database corresponding to the hot data.
[0009] As a further improvement of the present application, after the hotspot detection module identifies the hotspot data, it updates the hotspot list, which records the identifier of the hotspot data source data; the warm data list records the identifier of the warm data source data.
[0010] As a further improvement of the present application, the data synchronization module uses database triggers to achieve data synchronization. The database triggers monitor the corresponding source data based on the hot data list and the warm data list. When the corresponding source data in the database is inserted, updated or deleted, the trigger is triggered and the data synchronization module performs the data synchronization operation.
[0011] As a further improvement of the present application, when the access frequency of hot spot data is lower than the standard value, the data identification module converts the hot spot data into warm data, and the data management module clears the shard backup of the data, and retrieves the source data backup from its data source and stores it in the warm database.
[0012] As another improvement of the present application, when the access frequency of warm data is lower than a set value, the data is converted into cold data and the backup is cleared.
[0013] As another improvement of the present application, a dynamic scheduling module is also included, and the dynamic scheduling module is used for preloading warm data when the system load is low; the warm data includes source data corresponding to the search results whose access frequency is lower than the standard frequency and greater than the set frequency B; a cache unit matching the dynamic scheduling module is set in the data storage module; When the monitoring module detects that the system load is lower than the set value, the dynamic scheduling module is started, the dynamic scheduling module preloads the warm data and backs it up to the cache unit, and the data synchronization module updates the data in the cache unit in real time.
[0014] As another improved supplement to the present application, after the retrieval engine module performs a search, it identifies the hot data, warm data and warm data in the retrieval results, and when setting the data source access node, sets the source data storage location backed up in the data storage module as the priority access node.
[0015] As another improvement of the present application, a plurality of storage nodes are provided in the hot database, and when the source data of the hot data is backed up in the hot database, it is fragmented and stored in the plurality of storage nodes.
[0016] To summarize, this solution analyzes the dynamic data and real-time data stream processing of users' access to search results to analyze the hot data in the search results and perform local backup storage. The hot data in the search results are provided with auxiliary access nodes through local backup to reduce the access load of the database corresponding to the hot data, ensure users' timely reading of hot data, and optimize the access efficiency of search results after retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 This is a block diagram of the search platform system of the first implementation mode of this application; Figure 2 This is a working logic diagram of the search platform of the first implementation mode of this application; Figure 3 This is a logic block diagram of the second implementation mode of the present application; Figure 4 This is a diagram of the working steps of the retrieval platform for the three implementation methods of this application. DETAILED DESCRIPTION
[0018] The following describes two implementation modes of the present application in detail with reference to the accompanying drawings.
[0019] The first implementation method: Figure 1-2 A real-time data stream processing and dynamic data retrieval platform is shown, comprising: a user interface module for providing an interface for user search input and displaying search results; the user interface module can sort and paginate search results and provide filtering and sorting options to help users quickly find the information they need The search engine module is used to process user search requests and provide search services for the database associated with the search platform; this module uses existing efficient search algorithms, can quickly process complex queries, and index and optimize the data; it is set by those skilled in the art using a suitable search engine in the prior art; The hotspot detection module is used to record and analyze the access frequency of each search result. The hotspot detection module counts the access frequency of each search result source data when each user searches within a continuous time period. Based on the analysis of the access frequency statistics, the hotspot data is identified. The hotspot detection module uses Redis for fast access frequency statistics. Hot data includes search result source data with access frequency higher than the set identification value; The data storage module includes a hotspot database and a warm database; the hotspot database includes a distributed database; a plurality of storage nodes are arranged in the hotspot database, and when the source data of the hotspot data is backed up in the hotspot database, it is stored in multiple storage nodes in pieces; the data storage module combines Redis and MySQL to realize the caching and persistence of the hotspot data and the warm data; The data management module is used to manage the data storage module. The specific functions include: clearing hot data and warm data and storing hot data in segments; A data identification module is used to identify hot data and warm data; the data identification module is provided with a warm data list and a hot data list; When the access frequency of hot data is lower than the standard value, the data identification module converts the hot data into warm data. At the same time, the data management module clears the shard backup of the data and retrieves the source data backup from its data source and stores it in the warm database. When the access frequency of warm data is lower than the set value, it is converted into cold data and the backup is cleared.
[0020] The data synchronization module is used to monitor the database update log associated with the retrieval platform and synchronously update the data in the data storage module; The monitoring module is used to monitor the system operation status and system performance indicators. The system performance indicators include system load and memory usage.
[0021] After the hotspot detection module identifies the hotspot data, it updates the hotspot list, which records the identifier of the hotspot data source data; the warm data list records the identifier of the warm data source data, which is used to locate the data source; when the search results are presented, the identifier is presented together with other features of the data (such as title, content summary, etc.); The data synchronization module uses database triggers to achieve data synchronization. The database triggers monitor the corresponding source data based on the hot data list and the warm data list. When the corresponding source data in the database is inserted, updated, or deleted, the trigger is triggered and the data synchronization module performs the data synchronization operation.
[0022] After the search engine module performs the search, it identifies the hot data, warm data and warm data in the search results, and when setting the data source access node, sets the source data storage location backed up in the data storage module as the priority access node.
[0023] The search platform user of this solution submits a query request through the user interface. After receiving the request, the search engine queries the external database and returns the result. During the operation of the search platform, the hotspot detection module analyzes the access frequency of each search result in real time, marks the frequently accessed data as hotspot data, and stores it in the local hotspot database. This solution uses a cache management module to maintain the cache of hotspot data to ensure fast access to hotspot data; the data synchronization module monitors changes in the external database and updates local hotspot and warm data in a timely manner.
[0024] When the access frequency of the identified hot data decreases, the system will downgrade it to warm data for observation. After its access frequency continues to decrease, it will eventually be backed up and cleared in the local database to ensure effective use of memory space. For cold data with low access frequency, the system directly obtains it from the external database to save local storage resources.
[0025] Second implementation method: Figure 3 It is shown that the solution also includes a dynamic scheduling module, which is used for preloading warm data when the system load is low; the warm data includes source data corresponding to the search results whose access frequency is lower than the standard frequency and greater than the set frequency B; a cache unit matching the dynamic scheduling module is set in the data storage module; When the monitoring module detects that the system load is lower than the set value, the dynamic scheduling module is started, the dynamic scheduling module preloads the warm data and backs it up to the cache unit, and the data synchronization module updates the data in the cache unit in real time.
[0026] When it is identified that the access frequency of warm data is higher than the set identification value, the data identification module converts the warm data into hot data, and then transfers the data in the cache unit to the hot database; When the access frequency of warm data drops below a set value, the corresponding data in the cache unit is cleared; In this embodiment, when the system load is low, the dynamic scheduling module will use the free storage space to preload the warm data with higher access frequency into the cache. On the one hand, it reduces the user waiting time. On the other hand, when the access frequency of warm data increases, since its source data is cached in the data storage module, it is convenient to quickly convert it into hot data, saving the source data backup time after the hot data is identified.
[0027] The third implementation method: Figure 4 As shown, the working steps of the retrieval platform of this scheme are as follows: A1, user input is submitted to the search engine through the user interface module; A2, the search engine queries the external database and returns the search results; A3, the hotspot detection module counts the access frequency of each user to each search result source data during a continuous time period; and identifies the hotspot data based on the analysis of the access frequency statistics results; During identification, when it is identified that the access frequency of hot data is reduced below the standard value, the hot data is converted into warm data; and the hot data list and the warm data list are updated according to the analysis results; A4, the data management module updates the data storage module according to the hot data list and the warm data list; A5, the data synchronization module monitors external database updates and synchronizes hot data and warm data; A6: When the user selects a search result and needs to obtain its complete data: When obtaining the source data of hotspot data and warm data, the data management module directly provides data from the local hotspot database; When obtaining non-hotspot data, the search engine retrieves data from the data source and provides it to the user.
[0028] To summarize, this solution analyzes the dynamic data and real-time data stream processing of users' access to search results to analyze the hot data in the search results and perform local backup storage. The hot data in the search results are provided with auxiliary access nodes through local backup to reduce the access load of the database corresponding to the hot data, ensure users' timely reading of hot data, and optimize the access efficiency of search results after retrieval.
[0029] In view of current practical needs, the above-mentioned implementation mode adopted in this application is not limited to the scope of protection. Various changes made within the knowledge scope of technical personnel in this field without departing from the concept of this application still fall within the scope of protection of the present invention.
Claims
1. A real-time data stream processing and dynamic data retrieval platform, characterized by: include: User interface module, used to provide an interface for user search input and display of search results; The search engine module is used to process user search requests and provide search services for the database associated with the search platform; Hotspot detection module, used to record and analyze the access frequency of each search result and identify hotspot data; The hotspot data includes search result source data with an access frequency higher than a set identification value; The data storage module includes a hotspot database and a warm database; the hotspot database includes a distributed database; The data management module is used to manage the data storage module. The specific functions include: clearing hot data and warm data and storing hot data in segments; A data identification module is used to identify hot data and warm data; the data identification module is provided with a warm data list and a hot data list; A data synchronization module is used to monitor the database update log associated with the retrieval platform and synchronously update the data in the data storage module; The monitoring module is used to monitor the system operation status and system performance indicators. The system performance indicators include system load and memory usage.
2. A real-time data stream processing and dynamic data retrieval platform according to claim 1, characterized in that: After the hotspot detection module identifies the hotspot data, it updates the hotspot list, in which the identifier of the hotspot data source data is recorded; the warm data list records the identifier of the warm data source data.
3. A real-time data stream processing and dynamic data retrieval platform according to claim 2, characterized in that: The data synchronization module uses a database trigger to achieve data synchronization. The database trigger monitors the corresponding source data based on the hot data list and the warm data list. When the corresponding source data in the database is inserted, updated or deleted, the trigger is triggered and the data synchronization module performs the data synchronization operation.
4. A real-time data stream processing and dynamic data retrieval platform according to claim 1, characterized in that: When the access frequency of the hot data is lower than the standard value, the data identification module converts the hot data into warm data, and the data management module clears the shard backup of the data and retrieves the source data backup from its data source and stores it in the warm database.
5. A real-time data stream processing and dynamic data retrieval platform according to claim 1, characterized in that: When the access frequency of the warm data is lower than a set value, the warm data is converted into cold data and the backup is cleared.
6. A real-time data stream processing and dynamic data retrieval platform according to claim 1, characterized in that: It also includes a dynamic scheduling module, which is used for preloading warm data when the system load is low; the warm data includes source data corresponding to the search results whose access frequency is lower than the standard frequency and greater than the set frequency B; the data storage module is provided with a cache unit matching the dynamic scheduling module; When the monitoring module detects that the system load is lower than the set value, the dynamic scheduling module is started, the dynamic scheduling module preloads the warm data and backs it up in the cache unit, and the data synchronization module updates the data in the cache unit in real time.
7. A real-time data stream processing and dynamic data retrieval platform according to claim 6, characterized in that: After the search engine module performs the search, it identifies the hot data, warm data and warm data in the search results, and when setting the data source access node, sets the source data storage location backed up in the data storage module as the priority access node.
8. A real-time data stream processing and dynamic data retrieval platform according to claim 6, characterized in that: The hot database is provided with a plurality of storage nodes. When the source data of the hot data is backed up in the hot database, it is fragmented and stored in the plurality of storage nodes.
Citation Information
Patent Citations
Data archiving system, method, device and equipment based on data division
CN116166691A
Monitoring data-oriented acquisition, retrieval analysis and fault prediction method and system
CN117931570A
Electric power big data security calling system
CN118295985A
Hot spot data management processing system
JP1991010343A
