Hotspot data query method and device, equipment and storage medium

Abstract

The application discloses a hotspot data query method and device, equipment and a storage medium, and belongs to the technical field of data processing technology. The hotspot data query method comprises the following steps: obtaining to-be-screened hotspot data; performing deep paging and rolling query on the to-be-screened hotspot data based on an ES database to obtain a first hotspot data set; performing query on the to-be-screened hotspot data based on a Redis database to obtain a second hotspot data set; and obtaining target hotspot data by taking the intersection of the first hotspot data set and the second hotspot data set. The hit rate of the hotspot data in the hotspot data screening process is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a hot data query method and device, equipment and a storage medium. BACKGROUND

[0002] In the related art, there are two hot data screening schemes: one is to use the query statistics function of an ES (Elasticsearch is an open source distributed search and analysis engine) database, and the other is to use the query statistics function of a Redis (Remote Dictionary Server is an open source in-memory data storage system) database.

[0003] However, the former has a slow query speed and cannot query comprehensive hot data, and the latter may have a situation that when the key value key is very large, too much memory is occupied or the processing time is too long, causing the system to be prone to errors, so both have the technical problem of low hot data screening hit rate. SUMMARY

[0004] The main purpose of the present application is to provide a hot data query method, device, equipment and storage medium, which aims to solve the technical problem of low hot data screening hit rate in the related art.

[0005] To achieve the above purpose, the present application provides a hot data query method, which comprises the following steps:

[0006] obtaining hot data to be screened;

[0007] performing deep paging and rolling query on the hot data to be screened based on an ES database to obtain a first hot data set;

[0008] performing query on the hot data to be screened based on a Redis database to obtain a second hot data set;

[0009] taking the intersection of the first hot data set and the second hot data set to obtain target hot data.

[0010] Optionally, the step of obtaining hot data to be screened comprises:

[0011] obtaining original data, the original data comprising the hot data to be screened and full information associated with the hot data to be screened;

[0012] The step of taking the intersection of the first hot data set and the second hot data set to obtain target hot data comprises:

[0013] taking the intersection of the first hot data set and the second hot data set to obtain a third hot data set;

[0014] query target full information associated with data in the third hot data set based on the Redis database and the raw data;

[0015] obtain target hot data based on the third hot data set and the target full information.

[0016] Optionally, the step of querying the to-be-screened hot data based on the ES database to obtain the first hot data set includes:

[0017] determine whether the number of data of the to-be-screened hot data in the ES database is greater than a first preset number; wherein the to-be-screened hot data is arranged in the ES database in sequence;

[0018] If the number of data is greater than the first preset number, extract the first preset number of data from all the to-be-screened hot data, obtain a to-be-queried data set, and update the to-be-screened hot data;

[0019] query the to-be-queried data set to obtain an initial query result;

[0020] return to the step of determining whether the number of data of the to-be-screened hot data in the ES database is greater than the first preset number until all the to-be-screened hot data is queried;

[0021] obtain the first hot data set based on all the initial query results.

[0022] Optionally, after the step of determining whether the number of data of the to-be-screened hot data in the ES database is greater than the first preset number, the method further includes:

[0023] If the number of data is less than or equal to the first preset number, extract all the to-be-screened hot data to obtain the to-be-queried data set.

[0024] Optionally, the to-be-screened hot data includes a field value of a preset field;

[0025] The step of querying the to-be-queried data set to obtain an initial query result includes:

[0026] based on the preset field, statistics of data in the to-be-queried data set are obtained;

[0027] sort the statistics according to the field value of the preset field to obtain a first ordered data set;

[0028] extract the first preset number of data with the largest field value from the first ordered data set to obtain the initial query result.

[0029] Optionally, the step of querying the to-be-screened hot data based on the Redis database to obtain the second hot data set includes:

[0030] The repeated data in the to-be-screened hotspot data is merged into merged data, and the merged to-be-screened hotspot data is obtained;

[0031] Based on the number of repeated data in the merged data, a storage weight is given to the merged data; the storage weight is positively correlated with the number of repeated data;

[0032] Based on the numerical value of the storage weight, the data in the merged to-be-screened hotspot data is sorted, and a second ordered data set is obtained;

[0033] The second preset number of data with the largest storage weight value in the second ordered data set is extracted to form a second hotspot data set.

[0034] Optionally, the step of obtaining the to-be-screened hotspot data further comprises:

[0035] If it is detected that the current time is the preset time query time, the to-be-screened hotspot data is obtained.

[0036] In addition, to achieve the above object, the present application also provides a hotspot data query device, the device comprises:

[0037] The data acquisition module is configured to obtain the to-be-screened hotspot data;

[0038] The first data query module is configured to perform deep paging and scrolling query on the to-be-screened hotspot data based on the ES database, and obtain a first hotspot data set;

[0039] The second data query module is configured to query the to-be-screened hotspot data based on the Redis database, and obtain a second hotspot data set;

[0040] The data analysis module is configured to take the intersection of the first hotspot data set and the second hotspot data set, and obtain target hotspot data.

[0041] In addition, to achieve the above object, the present application also provides a hotspot data query device, the device comprises: a memory, a processor and a hotspot data query program stored in the memory and executable on the processor, the hotspot data query program is configured to implement the steps of the above-mentioned hotspot data query method.

[0042] In addition, to achieve the above object, the present application also provides a computer readable storage medium, the computer readable storage medium stores a hotspot data query program, and the hotspot data query program implements the steps of the above-mentioned hotspot data query method when executed by a processor.

[0043] The hotspot data query method comprises the following steps: obtaining to-be-screened hotspot data; performing deep paging and rolling query on the to-be-screened hotspot data based on an ES database to obtain a first hotspot data set; performing query on the to-be-screened hotspot data based on a Redis database to obtain a second hotspot data set; and obtaining target hotspot data by taking the intersection of the first hotspot data set and the second hotspot data set.

[0044] Therefore, by combining the ES database and the Redis database, the ES database is used to query fast and the Redis database is used to store data without repetition, so that the hotspot data is quickly screened, the intersection of the query result of the ES database and the query result of the Redis database is taken as the hotspot data, the accuracy of the hotspot data is guaranteed to the maximum extent, and the hit rate of the hotspot data query is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a structural schematic diagram of a hotspot data query device of a hardware running environment related to an embodiment of the present application.

[0046] Figure 2 is a flowchart of the first embodiment of the hotspot data query method of the present application.

[0047] Figure 3 is a functional module schematic diagram of the hotspot data query device of the present application.

[0048] Figure 4 is a flowchart of example 1 of the hotspot data query method of the present application.

[0049] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0051] Reference Figure 1 , Figure 1 is a structural schematic diagram of a hotspot data query device of a hardware running environment related to an embodiment of the present application.

[0052] As Figure 1As shown, the hotspot data query device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the hotspot data query device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0054] As Figure 1 As shown, the memory 1005 as a storage medium can include an operating system, a data storage module, a network communication module, a user interface module, and a hotspot data query program.

[0055] In Figure 1 In the hotspot data query device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the hotspot data query device of the present application can be arranged in the hotspot data query device. The hotspot data query device calls the hotspot data query program stored in the memory 1005 through the processor 1001, and executes the hotspot data query method provided by the embodiment of the present application.

[0056] The embodiment of the present application provides a hotspot data query method, which refers to Figure 2 , Figure 2 The flowchart of the first embodiment of the hotspot data query method of the present application.

[0057] In this embodiment, the hotspot data query method can include:

[0058] Step S100: Obtain the hotspot data to be screened.

[0059] In a specific implementation, taking the banking field as an example, the to-be-screened hot data can be customer transaction data and / or basic business data available for relevant personnel to query in the data server. When step S100 is executed, the customer transaction data and / or the basic business data need to be read from the data server according to corresponding data reading permissions, and written into the ES database and the Redis database.

[0060] The ES database includes a distributed search and analysis engine based on open source, and is mainly used for storing, searching and analyzing large-scale real-time data. The Redis database includes an open source in-memory data storage system, which can be used as a database, cache and message middleware, and stores data in the form of key-value pairs and supports rich data structures.

[0061] In a specific implementation, step S100 specifically includes:

[0062] If it is detected that the current time is the preset time query time, the to-be-screened hot data is obtained.

[0063] It can be understood that the data in the data server is constantly changing every day, and the data query demand of the user also changes over time, so that the hot data obtained by the previous query may no longer have the demand of the user query. Therefore, a query time node can be set for the hot data query, that is, the preset time query time. If the current time coincides with the preset time query time, step S100 is executed.

[0064] Step S200: performing deep paging and scrolling query on the to-be-screened hot data based on the ES database to obtain a first hot data set.

[0065] Specifically, the ES database performs deep paging and scrolling query on the written data through a distributed search engine and an analysis engine, and queries the corresponding hot data set according to the preset query condition, that is, the first hot data set. The preset query condition can be a certain field or parameter contained in the to-be-screened hot data.

[0066] The deep paging and scrolling query is a query mechanism provided by the ES database for retrieving large amounts of data. When it is necessary to gradually obtain results from a search request containing a large number of results, the traditional paging mechanism based on cursor or offset may encounter performance problems, and the deep paging and scrolling query provides a more efficient way to handle such cases. When using the deep paging and scrolling query, the results of the query are cached internally in the ES database, rather than being recalculated at each request. This means that it can provide higher performance and efficiency when handling large-scale data sets.

[0067] In a specific embodiment, step S200 specifically comprises:

[0068] Step S210: judging whether the data quantity of the to-be-screened hot data in the ES database is greater than a first preset quantity.

[0069] The to-be-screened hot data are arranged in sequence in the ES database.

[0070] Step S230: if the data quantity is greater than the first preset quantity, extracting the first preset quantity of data from all the to-be-screened hot data, obtaining a to-be-queried data set and updating the to-be-screened hot data.

[0071] Step S250: querying the to-be-queried data set to obtain an initial query result.

[0072] Returning to step S210 until all the to-be-screened hot data are queried.

[0073] Step S270: obtaining a first hot data set based on all the initial query results.

[0074] It can be understood that in a specific data query process, a large amount of data is usually faced, and if all the data is queried at one time, a huge pressure will be brought to the query server. Therefore, in a specific implementation procedure, a large amount of data can be queried through a deep paging scroll query mechanism.

[0075] Specifically, before the paging scroll query is performed, the data quantity of each query, i.e., the first preset quantity, needs to be set. In a specific implementation process, the first preset quantity can be adjusted according to a specific application scenario, which is not limited here. First, it is judged whether the data quantity of the to-be-screened hot data written into the ES database is greater than the first preset quantity n, if the data quantity of the to-be-screened hot data is greater than the first preset quantity n, the to-be-screened hot data are arranged in sequence, the first preset quantity n of data are read from the arranged to-be-screened hot data as a to-be-queried data set, and the position of the last data in the read data in the to-be-screened hot data is marked. An initial result is obtained by querying the to-be-queried data set through the ES database. Then it is judged whether the to-be-screened hot data after the marked position is greater than the first preset quantity n, if it is less than or equal to the first preset quantity n, all the to-be-screened hot data after the marked position are read as the to-be-queried data set, if it is greater than the first preset quantity n, the first preset quantity n of data are read from the marked position as the to-be-queried data set, and the position mark is updated to the position of the last data in the read data in the to-be-screened hot data, until all the to-be-screened hot data are queried.

[0076] In an example, n = 100, and the 1000 pieces of hotspot data to be screened. At this time, the number of pieces of hotspot data to be screened is greater than 100, and therefore the 1000 pieces of hotspot data to be screened are arranged in sequence, the first 100 pieces of hotspot data to be screened are read as the set of data to be queried, and the position of the 100th piece of hotspot data to be screened is marked. The 100 pieces of hotspot data to be screened are queried through the ES database to obtain an initial result. Then the 101st to 200th pieces of hotspot data to be screened are read as the set of data to be queried from the 100th piece of hotspot data to be screened, and all the pieces of hotspot data to be screened are queried through 9 cycles.

[0077] Further, after step S210, the method can further include:

[0078] Step S220: If the number of pieces of data is less than or equal to the first preset number of pieces, all the pieces of hotspot data to be screened are extracted to obtain the set of data to be queried.

[0079] It can be understood that when the pieces of hotspot data to be screened are less than or equal to the first preset number n of pieces, the page-by-page scrolling query is not needed, and all the pieces of hotspot data to be screened are read at one time for query.

[0080] In the embodiment, the deep page-by-page scrolling query is implemented for a large amount of data, the pieces of hotspot data to be screened are queried in batches, the fast query of hotspot data is achieved, and the pressure on the query server is also reduced to a certain extent.

[0081] In a specific implementation, step S250 specifically includes:

[0082] Step S251: Based on a preset field, the data in the set of data to be queried are counted to obtain a counting result.

[0083] Step S252: The counting result is sorted according to the field value of the preset field to obtain a first ordered data set.

[0084] Step S253: The first preset number of pieces of data with the largest field value in the first ordered data set are extracted to obtain an initial query result.

[0085] It can be understood that the ES database has strong aggregation and counting capabilities, and can count and sort a large amount of data based on a specific field in the data. Specifically, in the embodiment, the pieces of hotspot data to be screened written into the ES database include multiple fields, and in the process of querying the hotspot data through the ES database, a field can be preset as a query condition according to a specific application scenario, that is, as a preset field. Then the preset field is counted and sorted to obtain a data set sorted based on the field value of the preset field, that is, a first ordered data set.

[0086] In a specific implementation process, the preset field includes all fields that can reflect the frequency of data being queried, and can be an account number or an account name in the to-be-screened hot data, which is not limited here.

[0087] In the embodiment, the to-be-screened hot data is statistically queried based on the preset field that can reflect the frequency of data being queried, so that the data with the highest frequency of being queried can be obtained, and the hit rate of hot data screening is improved.

[0088] In an example, the to-be-screened hot data is transaction data of a commodity, which includes a commodity name, a commodity category, a commodity description, a commodity price, and a commodity transaction quantity. In order to find specific information of a hot-selling product, the commodity transaction quantity can be set as a query condition, the transaction data is queried by the ES database, the commodities are sorted in descending order based on the commodity transaction quantity, and then the commodity descriptions of the top 5 commodities with the largest transaction quantity are extracted as the specific information of the hot-selling product.

[0089] Step S300: Query the to-be-screened hot data based on the Redis database to obtain a second hot data set.

[0090] It can be understood that the Redis database is not only a key-value storage system, but also provides multiple data structures including strings, lists, sets, and ordered sets. Therefore, in a specific implementation process, the Redis database can take the key value of the preset parameter as a query condition to query the corresponding ordered data set, and then extract the hot data meeting the expected requirements from the ordered data set to obtain the second hot data set.

[0091] The preset parameter can be the storage weight of data or the number of repeated data in a specific application scenario, so that the query result is more accurate and closer to the expected hot data.

[0092] In a specific implementation, step S300 specifically includes:

[0093] Step S310: Merge the repeated data in the to-be-screened hot data into merged data to obtain the merged to-be-screened hot data.

[0094] Step S320: Assign a storage weight to the merged data based on the number of repeated data in the merged data; the storage weight is positively correlated with the number of repeated data.

[0095] Step S330: Sort the data in the merged to-be-screened hot data based on the numerical value of the storage weight to obtain a second ordered data set.

[0096] Step S340: extracting a second preset number of data with the largest storage weight value in the second ordered data set to form a second hot data set.

[0097] Specifically, the Redis database will remove duplicate hot data for storage when data is written. According to the number of duplicate hot data removed, the remaining hot data is given a corresponding storage weight. The storage weight is positively correlated with the number of duplicate data, that is, the more duplicate data removed, the larger the value of the storage weight of the data retained.

[0098] For example, when writing the hot data to be filtered, there are 3 data A, and 2 data A need to be removed, and 1 data A is retained, and the data A retained is given a storage weight value of 3.

[0099] The finally retained hot data to be filtered is sorted according to the size of the storage weight value to obtain an ordered set based on the storage weight, that is, a second ordered data set, which can be sorted from large to small or from small to large. The second preset number of data with the largest storage weight value is extracted from the second ordered data set to form a second hot data set.

[0100] The specific value of the second preset number can be flexibly adjusted according to the specific application scenario, which is not limited here.

[0101] In this embodiment, the Redis database removes duplicate data in the hot data to be filtered for storage, reducing the workload of the Redis database data processing, greatly improving the efficiency of data processing. At the same time, based on the number of duplicate data, the data retained is given a storage weight, and the second hot data set is obtained based on the storage weight value, which can make the filtering result more in line with the expected effect, greatly improving the hit rate of hot data.

[0102] Step S400: taking the intersection of the first hot data set and the second hot data set to obtain the target hot data.

[0103] Specifically, after obtaining the first hot data set and the second hot data set, on the one hand, in order to make the query result of the hot data more accurate; on the other hand, the capacity of the cache is limited, too many hot data cannot be written into the cache. Therefore, the intersection of the first hot data set and the second hot data set is needed. That is, comparing the hot data in the first hot data set and the second hot data set, extracting the hot data that exists in both the first hot data set and the second hot data set as the final hot data, that is, the target hot data.

[0104] In the embodiment, the ES database and the Redis database are respectively queried to obtain a first hot data set and a second hot data set from the read data server, and the target hot data is obtained based on the intersection of the first hot data set and the second hot data set.

[0105] Therefore, in the hot data query process, the embodiment has the characteristics of fast ES database query and accurate Redis database query, realizes fast screening of hot data, maximally guarantees the accuracy of hot data, and improves the hit rate of hot data query.

[0106] Further, based on the above embodiment, a second embodiment of a hot data query method is provided. In the embodiment, step S100 can include:

[0107] The original data is obtained, and the original data includes the hot data to be screened and the full information associated with the hot data to be screened.

[0108] In a specific embodiment, step S400 specifically includes:

[0109] Step S410: The first hot data set and the second hot data set are intersected to obtain a third hot data set.

[0110] Step S420: Based on the Redis database and the original data, the target full information associated with the data in the third hot data set is queried.

[0111] Step S430: Based on the third hot data set and the target full information, the target hot data is obtained.

[0112] It can be understood that the original data obtained from the data server usually includes the hot data to be screened and the full information associated therewith. Therefore, in the process of intersecting the first hot data set and the second hot data set, in order to reduce the workload of data comparison, part of the associated information that does not affect the screening result can be ignored in the specific implementation process, and only the information related to this screening is compared, so as to obtain an intermediate result, that is, the third hot data set. Then, the data in the third hot data set is written into the Redis database, and the full information associated with the data in the third hot data set is queried from the original data through the Redis database, that is, the target full information. Finally, the data in the third hot data set and the corresponding target full information are combined to obtain the final hot data, that is, the target hot data.

[0113] In the embodiment, only the information related to the screening is compared by ignoring part of the information that does not affect the screening result, so that the workload of taking the intersection of the first hot data set and the second hot data set is reduced, and the efficiency of the entire hot data query is improved.

[0114] In order to enable those skilled in the art to better understand the protection scope of the claims of the present application, the technical solutions described in the claims of the present application are explained and described below through specific implementation examples in specific application scenarios. It can be understood that the following examples are only used to explain the present application, and are not used to limit the protection scope of the claims of the present application.

[0115] Example 1:

[0116] As shown in Figure 4 , the log system writes each completed transaction data into the corresponding ES database and Redis database respectively for deep paging scroll query and weight query, filters out the respective hot data, takes the intersection of the two screening results to obtain more accurate final hot data, and then queries the associated full information based on the final hot data through the Redis database. Finally, the final hot data and the corresponding full information are written into the cache.

[0117] As shown in Figure 4 , in the process of deep paging scroll query, the ES database stores disordered transaction data at the beginning. The data is queried by deep paging with 2 pieces of data per page. The first 2 pieces of data are found by using statistical query. At this time, the position is marked by a cursor and the mark information is returned. The second query brings the mark information returned before, and continues to query from the mark position. The cycle is repeated 3 times to find all the results.

[0118] As shown in Figure 4 , in the weight query process, the Redis database removes duplicate data after storing transaction data. A weight parameter is added each time data is stored. The value of the stored weight is accumulated with the number of data repetitions to obtain a disordered data set based on the stored weight query. Finally, the Redis database is queried by weight range, and the disordered data set is reordered to find the customer data with the largest weight.

[0119] The hot data hit situation before and after the optimization of the hot data double query mechanism is shown in Table 1.

[0120] Table 1

[0121]

[0122] As shown in Table 1, before the staging strategy is optimized, the hit rate is 16.71% using the ES database, and after optimization and using the hot data double-check mechanism, the hit rate is 25.93%, and the hot data hit rate improvement effect is 55.17%; before the credit strategy is optimized, the hit rate is 27.32% using the Redis database, and after optimization and using the hot data double-check mechanism, the hit rate is 49.80%, and the hot data hit rate improvement effect is 82.28%. As can be seen, after introducing the hot data double-check mechanism, the improvement of the hot data hit rate is very obvious.

[0123] Further, to achieve the above object, the application also provides a hot data query device, which can include:

[0124] a data acquisition module, configured to acquire hot data to be screened;

[0125] a first data query module, configured to perform deep paging and rolling query on the hot data to be screened based on an ES database to obtain a first hot data set;

[0126] a second data query module, configured to perform query on the hot data to be screened based on a Redis database to obtain a second hot data set;

[0127] a data analysis module, configured to take intersection of the first hot data set and the second hot data set to obtain target hot data.

[0128] It should be noted that the functions that can be achieved by each module in the hot data query device provided in the embodiment and the technical effects achieved thereby can be referred to the description of the specific implementation in each embodiment of the hot data query method of the application, and will not be described here again for the sake of brevity of the description.

[0129] In addition, the application also provides a computer readable storage medium, and the computer readable storage medium stores a hot data query program, and the hot data query program is executed by a processor to implement the steps of the cross-network comparison method as described above. Therefore, no further description will be given here. In addition, the beneficial effects of using the same method will not be described again. For technical details not disclosed in the computer readable storage medium embodiments of the application, please refer to the description of the method embodiments of the application. For example, the program instructions can be deployed to be executed on one computing device, or on multiple computing devices located in one place, or on multiple computing devices distributed in multiple places and interconnected through a communication network.

[0130] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the statement "comprises a hot spot data query" does not exclude the existence of additional identical elements in a process, method, article, or apparatus including the element.

[0131] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0132] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device) execute the methods of the various embodiments of the present application.

[0133] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for querying data of hotspots, characterized in that, The hotspot data query method comprises: acquiring to-be-screened hotspot data; performing deep paging and rolling query on the to-be-screened hotspot data based on an ES database to obtain a first hotspot data set; performing query on the to-be-screened hotspot data based on a Redis database to obtain a second hotspot data set; taking intersection of the first hotspot data set and the second hotspot data set to obtain target hotspot data; the step of performing deep paging and rolling query on the to-be-screened hotspot data based on the ES database to obtain the first hotspot data set comprises: judging whether the data quantity of the to-be-screened hotspot data in the ES database is greater than a first preset quantity; wherein the to-be-screened hotspot data is arranged in the ES database in sequence; if the data quantity is greater than the first preset quantity, extracting the first preset quantity of data from all the to-be-screened hotspot data to obtain a to-be-queried data set and update the to-be-screened hotspot data; querying the to-be-queried data set to obtain an initial query result; returning to the step of judging whether the data quantity of the to-be-screened hotspot data in the ES database is greater than the first preset quantity until all the to-be-screened hotspot data is queried; obtaining the first hotspot data set based on all the initial query results; the step of performing query on the to-be-screened hotspot data based on the Redis database to obtain the second hotspot data set comprises: merging repeated data in the to-be-screened hotspot data into merged data to obtain merged to-be-screened hotspot data; assigning a storage weight to the merged data based on the repeated data quantity in the merged data; the storage weight is positively correlated with the repeated data quantity; sorting data in the merged to-be-screened hotspot data based on the numerical value of the storage weight to obtain a second ordered data set; extracting the second preset quantity of data with the maximum storage weight value from the second ordered data set to form the second hotspot data set.

2. The method of claim 1, wherein, the step of acquiring to-be-screened hotspot data comprises: acquiring original data, wherein the original data comprises the to-be-screened hotspot data and full information associated with the to-be-screened hotspot data; the step of taking intersection of the first hotspot data set and the second hotspot data set to obtain target hotspot data comprises: taking intersection of the first hotspot data set and the second hotspot data set to obtain a third hotspot data set; querying target full information associated with data in the third hotspot data set based on the Redis database and the original data; obtaining the target hotspot data based on the third hotspot data set and the target full information.

3. The method of claim 1, wherein the hotspot data query is a query for a hotspot of a specific type of data. after the step of judging whether the data quantity of the to-be-screened hotspot data in the ES database is greater than the first preset quantity, the method further comprises: if the data quantity is less than or equal to the first preset quantity, extracting all the to-be-screened hotspot data to obtain the to-be-queried data set.

4. The method of claim 1, wherein the hotspot data query is a query for a hotspot of a specific type of data. the to-be-screened hotspot data comprises field values of preset fields; the step of obtaining an initial query result from the to-be-queried data set comprises: Based on the preset field, the data in the to-be-queried data set is counted to obtain a statistical result; The statistical result is sorted according to the field value size of the preset field to obtain a first ordered data set; The first preset number of data with the largest field value in the first ordered data set is extracted to obtain the initial query result.

5. The method of claim 1-4, wherein, The step of obtaining the to-be-screened hot data further includes: If it is detected that the current time is a preset time query time, the to-be-screened hot data is obtained.

6. A hot spot data query apparatus, characterized by comprising: The device includes: A data acquisition module for acquiring to-be-screened hot data; A first data query module for performing deep paging and scrolling query on the to-be-screened hot data based on an ES database to obtain a first hot data set; A second data query module for performing query on the to-be-screened hot data based on a Redis database to obtain a second hot data set; A data analysis module for taking the intersection of the first hot data set and the second hot data set to obtain target hot data; The first data query module is further configured to judge whether the data number of the to-be-screened hot data in the ES database is greater than a first preset number; wherein the to-be-screened hot data is arranged in the ES database in sequence; If the data number is greater than the first preset number, the first preset number of data from all the to-be-screened hot data is extracted to obtain a to-be-queried data set and update the to-be-screened hot data; The to-be-queried data set is queried to obtain an initial query result; The step of judging whether the data number of the to-be-screened hot data in the ES database is greater than the first preset number is returned to be executed until all the to-be-screened hot data is queried; Based on all the initial query results, a first hot data set is obtained; The second data query module is further configured to merge the repeated data in the to-be-screened hot data into merged data to obtain merged to-be-screened hot data; Based on the data repeated number in the merged data, a storage weight is given to the merged data; the storage weight is positively correlated with the data repeated number; Based on the numerical value of the storage weight, the data in the merged to-be-screened hot data is sorted to obtain a second ordered data set; The second preset number of data with the largest storage weight numerical value in the second ordered data set is extracted to form the second hot data set.

7. A hot spot data query device, characterized by, The device includes a memory, a processor, and a hot data query program stored on the memory and executable on the processor, the hot data query program being configured to implement the steps of the hot data query method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a hot data query program, and the hot data query program is executed by the processor to implement the steps of the hot data query method according to any one of claims 1 to 5.

Images