Data query method, device and storage medium
By indexing internet data in a data query platform and storing it in a database, and then using a database retrieval engine for querying, the problem of excessively long response times in massive amounts of internet data is solved, enabling fast data retrieval.
Patent Information
- Application Number
- CN202111654688.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-30
AI Technical Summary
The excessively long response time during the query and retrieval process from the massive amounts of data on the internet leads to a poor user experience.
Data is retrieved from the Internet using a pre-set data crawling program on the data query platform, indexed, and stored in a database. When responding to user query requests, the database retrieval engine retrieves relevant data from the database.
This significantly reduces the time for data retrieval from 10 seconds in existing technologies to 2 seconds, thus improving the user experience.
Smart Images

Figure CN114398536B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database technology, and in particular to a data query method, device and storage medium. Background Art
[0002] The financial data query platform can provide users with the query and analysis of financial data based on Internet big data. For example, a user sends a request to query the financial data of a certain company to the financial data query platform through a terminal device, and the financial data query platform can obtain the financial data related to the company from the massive data on the Internet according to the user's needs.
[0003] In many cases, the financial data query platform itself does not generate financial data. After receiving the user's query request, it also needs to obtain relevant financial data from various website platforms. For example, the financial data query platform may need to obtain relevant financial data from third-party financial institutions. However, in many cases, these website platforms do not provide an API interface for encapsulation and calling. Therefore, when the financial data query platform receives the user's query request, it can only obtain the financial data of various website platforms through Internet search, and cannot directly call the database of each website platform. Since this method of searching from the Internet lacks indexes for querying data, when the amount of queried data is relatively large (for example, more than 60 million pieces of data), the feedback time will be very long, resulting in a poor user experience.
[0004] With respect to the technical problem in the above-mentioned prior art that the feedback time of the query and retrieval process in the massive data on the Internet is too long, resulting in a poor user experience, no effective solution has been proposed so far. Summary of the invention
[0005] The embodiments of the present disclosure provide a data query method, device and storage medium to at least solve the technical problem in the prior art that the feedback time of the query and retrieval process in the massive data on the Internet is too long, resulting in a poor user experience.
[0006] According to one aspect of an embodiment of the present disclosure, there is provided a data query method for a data query platform, comprising: obtaining first Internet data from the Internet using a preset data crawling program, and storing the first Internet data in a database of the data query platform after adding a corresponding index to the first Internet data; receiving a data query request; and in response to the data query request, retrieving database return data related to the query request from the first Internet data in a database according to the index through a preset database retrieval engine in a database.
[0007] According to another aspect of the embodiments of the present disclosure, a storage medium is further provided. The storage medium includes a stored program, wherein the above method is executed by a processor when the program is running.
[0008] According to another aspect of the embodiments of the present disclosure, there is also provided a data query device for use in a data query platform, characterized in that it includes: a first Internet data acquisition module, used to acquire first Internet data from the Internet using a preset data crawling program, and store the first Internet data in a database of the data query platform after adding a corresponding index to the first Internet data; a query request receiving module, used to receive a data query request; and a database data return module, used to respond to the data query request, and retrieve database return data related to the query request from the first Internet data in the database according to the index through a preset database retrieval engine.
[0009] According to another aspect of the embodiments of the present disclosure, a data query device is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining first Internet data from the Internet using a preset data crawling program, and storing the first Internet data in a database of the data query platform after adding a corresponding index to the first Internet data; receiving a data query request; and in response to the data query request, retrieving database return data related to the query request from the first Internet data in the database according to the index through a preset database retrieval engine.
[0010] According to the technical solution disclosed in the present invention, the data query platform first obtains Internet data from the data source of the Internet, then adds an index to the obtained data, and saves the data after adding the index to the database of the data query platform. Thus, after the data query platform receives the query request of the user, it does not obtain Internet data by searching the Internet, but retrieves the data obtained from the Internet in the database according to the index, thereby obtaining data information related to the query request (i.e., the database returns data). Therefore, compared with the prior art, since the technical solution disclosed in the present invention updates the Internet data to the database and adds an index in advance, the data related to the query request can be retrieved in the database according to the added index, thereby greatly shortening the time for querying data. After testing and verification, it takes about 10 seconds to query more than 3 million pieces of data according to the prior art, but it only takes 2 seconds to complete through the technical solution disclosed in the present invention. Thus, compared with the prior art, the time for retrieving data is greatly reduced, and the technical problem that the feedback time for data query in the massive data of the Internet is too long, resulting in poor user experience, which is solved in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:
[0012] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;
[0013] Figure 2 is a schematic diagram of a data query system according to Embodiment 1 of the present disclosure;
[0014] Figure 3 is a schematic diagram of the module architecture of the data query platform according to Embodiment 1 of the present disclosure;
[0015] Figure 4 is a flowchart of a data query method according to the first aspect of Embodiment 1 of the present disclosure;
[0016] Figure 5 is a schematic diagram of a thread pool of a data query platform according to Embodiment 1 of the present disclosure;
[0017] Figure 6 It is a schematic diagram of a thread pool and a thread waiting queue of a data query platform according to Embodiment 1 of the present disclosure;
[0018] Figure 7 is a schematic diagram of a storage space managed by a data temporary storage unit of a data query platform according to Embodiment 1 of the present disclosure;
[0019] Figure 8 is a schematic diagram of a data query device according to Embodiment 2 of the present disclosure; and
[0020] Fig. 9 It is a schematic diagram of the data query device described in Example 3 of the present disclosure. DETAILED DESCRIPTION
[0021] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.
[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0023] Example 1
[0024] According to this embodiment, a method embodiment of a data query method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0025] The method embodiment provided in this embodiment can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device for implementing a data query method. Figure 1 As shown, the computing device may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0026] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As involved in the embodiments of the present disclosure, the data processing circuitry acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0027] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data query method in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the data query method of the above-mentioned application is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0028] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.
[0029] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the computing device.
[0030] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.
[0031] Figure 2is a schematic diagram of a data query system according to this embodiment. Figure 2 As shown, the system includes: terminal devices 101~103 and a data query platform 200 that communicates with the terminal devices 101~103 through a network. Among them, the terminal devices 101~103 can send a query request for query data to the data query platform 200. As an example, the data query platform 200 can be, for example, a financial data query platform, so that users can query the data query platform 200 for financial data through the terminal devices 101~103. In addition, the data query platform 200 can access various data sources 301~303 through the Internet, and obtain data information related to financial business from each data source 301~303. Among them, the data source 301~303 can be, for example, a third-party financial information publishing website, or an information publishing website of a listed company. No specific limitation is made here, as long as the data query platform 200 can obtain relevant financial data (i.e., Internet data) from the data source 301~303 through the Internet. In addition, Figure 2 Data sources 301-303 and terminal devices 101-103 are shown in the figure, but this is only an exemplary description for facilitating those skilled in the art to understand the technical solution of the present disclosure. Those skilled in the art should be aware that in reality, the number of data sources accessed by the data query platform 200 through the Internet is much larger than this, and the number of terminal devices that can query data from the data query platform 200 at the same time is also much larger than this.
[0032] Figure 3 FIG. 2 shows a schematic diagram of the module architecture of the data query platform 200. Figure 3 As shown, the data query platform 200 includes: a user interface module 210, a thread scheduling module 220 and a search module 230. Among them, the user interface module 210 is used to interact with the terminal devices 101~103, receive the user's query request from the terminal devices 101~103, and return the searched data to the terminal devices 101~103. The thread scheduling module 220 is used to schedule and manage the threads according to the received query request. The specific content of the thread scheduling module 220 will be described in detail below. The search module 230 is used to search for data information corresponding to the query request, and return the searched data information to the user interface module 210, and return it to the terminal devices 101~103 through the user interface module.
[0033] It should be noted that the terminal devices 101-103 and the data query platform 200 in the system can all be applicable to the hardware structure described above.
[0034] Under the above operating environment, according to the first aspect of this embodiment, a data query method is provided. Figure 2The data query platform 200 shown in FIG. Figure 4 A schematic diagram showing the process of the method is shown in FIG. Figure 3 As shown, the method includes:
[0035] S102: using a preset data crawling program to obtain first Internet data from the Internet, and storing the first Internet data in a database of a data query platform after adding a corresponding index to the first Internet data;
[0036] S104: receiving a query request; and
[0037] S106: In response to the query request, a preset database search engine is used to retrieve database return data related to the query request from the first Internet data in the database according to the index.
[0038] Specifically, see Figure 3 As shown, the search module 230 of the data query platform 200 includes a data capture unit 231. Thus, through the data capture unit 231, the data query platform 200 can periodically (i.e., at a first time point) obtain data information related to financial services (i.e., first Internet data) from various data sources 301 to 303 on the Internet. The data information related to the financial services can be, for example, data information about the stock market, listed companies, or financial institutions (i.e., target objects). Then the data capture unit 210 transmits the acquired data to the data search unit 232, and the data search unit 232 adds corresponding indexes to the acquired data, and then saves them to the database of the data search unit 232 to achieve synchronization of database data (S102).
[0039] Specifically, the time for the data query platform 200 to obtain the first Internet data from the Internet can be set to start at 1 o'clock every evening (at this time, the frequency of users accessing the data query platform 200 is relatively low, which is suitable for synchronizing the data in the database of the data query platform 200). The data capture unit 210 can, for example, obtain updated data from the data sources 301 to 303 once every predetermined period. In addition, the data query platform 200 can subdivide the interval period for obtaining updated data from the data sources 301 to 303 according to the operation duration of the data capture unit 231 to obtain data and the operation duration of the data search unit 232 to update the database. For example, the data query platform 200 can obtain updated data from the data sources 301 to 303 once every hour, and save the data to the database for synchronization after adding an index.
[0040] Then, the user interface module 210 of the data query platform 200 receives the user's query request from the terminal device 101 (S104). The query request may be, for example, a request to query the transaction data information of the stock market within a certain time period, or a request to query the stock information of a listed company. No specific limitation is made here.
[0041] Then, after receiving the query request through the user interface 210 module, the data query platform 200 will start the corresponding thread through the thread scheduling module 220 in response to the query request, and call the query interface unit 233 of the search module 230 through the thread. Then, the data query platform 200 retrieves the relevant data information (i.e., database return data) in the database according to the index added by the data search unit 232 through the database search engine of the data search unit 232, wherein the retrieval operation includes retrieving the data information related to the query request from the data information (i.e., the first Internet data) obtained from the data sources 301-303 as the database return data returned to the user (S106).
[0042] As described in the background technology, after receiving the query request from the user, the existing data query platform also needs to obtain relevant data information from each website platform. However, in many cases, these website platforms do not provide an API interface for encapsulation and calling. Therefore, after receiving the query request from the user, the data query platform can only obtain the data information of each website platform through Internet search, but cannot directly call the database of each website platform. Since this method of searching from the Internet lacks an index for querying data, when the amount of data queried is relatively large (for example, more than 60 million pieces of data), the feedback time will be very slow.
[0043] In view of this, according to the technical solution disclosed in the present invention, the data query platform first obtains Internet data from the data source of the Internet, then adds an index to the obtained data, and saves the indexed data to the database of the data query platform. Therefore, after the data query platform receives the query request of the user, it does not obtain Internet data by searching the Internet, but retrieves the data obtained from the Internet in the database according to the index, thereby obtaining data information related to the query request (i.e., the database returns the data). Therefore, compared with the prior art, since the technical solution disclosed in the present invention updates the Internet data to the database and adds an index in advance, the data related to the query request can be retrieved in the database according to the added index, thereby greatly shortening the time for querying data. After testing and verification, it takes about 10 seconds to query more than 3 million pieces of data according to the prior art, but it only takes 2 seconds to complete through the technical solution disclosed in the present invention. Therefore, compared with the prior art, the time for retrieving data is greatly reduced, and the technical problem that the feedback time for data query in the massive data on the Internet is too long, resulting in poor user experience, which is solved in the prior art. In addition, although the present embodiment is illustrated by taking financial data query as an example, data query in other fields is also applicable to the disclosed technical solution. As an example, the data search unit 232 may use a search tool based on Elasticsearch.
[0044] Optionally, the operation of obtaining the first Internet data from the Internet using a preset data capture program includes: obtaining historical information of related operations of storing the Internet data in a database; and determining a time interval for executing the operation of obtaining the first Internet data from the Internet based on the historical information.
[0045] Specifically, the data query platform 200 can collect historical operation information of synchronizing the database by acquiring Internet data from the Internet data sources 301-303 through the search module 230. For example, the search module 230 collects information such as the time points when the data capturing unit 231 acquires data from the data sources 301-303 and synchronizes the data to the database, the amount of data acquired, and the duration of each operation.
[0046] When the average amount of data obtained by each operation or the average operation duration of each operation counted by the search module 230 is greater than a predetermined threshold, it means that the period interval between each operation is too large, resulting in too much Internet data captured by each operation and too much data synchronized to the database. Therefore, in this case, the search module 230 will automatically reduce the time interval between each operation, thereby reducing the amount of data obtained by a single operation and the duration of the operation. For example, the previously set data capture from the Internet every 2 hours can be adjusted to capture data from the Internet every 1.5 hours.
[0047] On the contrary, when the average amount of data obtained by each operation or the average operation duration of each operation counted by the search module 230 is less than a predetermined threshold, it means that the period interval between each operation is too small, resulting in the data query platform 200 crawling Internet data from the Internet too frequently. Therefore, in this case, the search module 230 will automatically increase the time interval between each operation, thereby increasing the amount of data obtained by a single operation and the duration of the operation. For example, the previously set crawling data from the Internet every 1 hour can be adjusted to crawling data from the Internet every 1.5 hours.
[0048] In this way, the data query platform 200 can adjust the time interval and frequency of grabbing data from the Internet and synchronizing to the data according to the amount of data obtained from the Internet each time, thereby ensuring the normal and stable operation of the data query platform 200.
[0049] Optionally, the method also includes: in response to a query request, using a pre-set Internet search engine to control a data crawler program to obtain second Internet data related to the query request from the Internet, wherein the second Internet data is data that is not synchronized with the database; and combining the database retrieval data and the second Internet data to generate retrieval return data corresponding to the query request.
[0050] Specifically, refer to Figure 3 As shown, the data search unit 232 also includes an Internet search engine, so that in addition to searching the Internet data (i.e., the first Internet data) related to the query request from the database through the database search engine, the data search unit 232 can also call the data capture unit 231 through the Internet search engine to obtain the Internet data (i.e., the second Internet data) that has not been synchronized in the database from the Internet data source 301-303. Then, the data search unit 232 combines the data retrieved from the database through the database search engine and the data retrieved from the Internet data source 301-303 through the Internet search engine to form the search return data returned to the terminal device 101.
[0051] As described above, the operation of the data search unit 232 to capture Internet data from the Internet data sources 301-303 and synchronize the database is not performed in real time, but is performed every specified period. Therefore, if only data related to the query request is retrieved from the database, the latest updated data of the Internet data sources 301-303 may not be returned to the user.
[0052] In view of this, the technical solution of the present disclosure sets an Internet search engine in the data search unit 232, which is used to respond to the user's query request and capture the Internet data (i.e., the second Internet data) related to the query request that has not yet been synchronized to the database from the Internet data source 301-303. For example, the Internet search engine can obtain the Internet data related to the query request from the Internet data source 301-303 in the time period after the synchronization time according to the last time the Internet data was synchronized to the database. Then the data search unit 232 combines the data retrieved from the database by the database search engine and the data retrieved from the Internet data source 301-303 by the Internet search engine to form the search return data returned to the terminal device 101. Thus, the data query platform 200 transmits the search return data to the terminal device 101 through the user interface module 210. In this case, since the Internet search engine only performs a search related to the query request for the Internet data that is not synchronized with the database, the amount of data obtained on the Internet is greatly reduced. Compared with the prior art of obtaining all the Internet data related to the query request from the Internet data source 301-303, the time for searching data can be greatly reduced.
[0053] In this way, the data query platform 200 can return data to users in a timely and comprehensive manner while reducing the feedback time of query requests for Internet big data.
[0054] In addition, optionally, the method further includes adding a tag to the second Internet data and storing it in the database. Specifically, after the data search unit 232 retrieves the Internet data (i.e., the second Internet data) related to the query request through the Internet search engine, it will add a tag to the Internet data and synchronize it to the database. In this way, the database can be synchronized while retrieving the Internet data related to the query request. This allows the database to save the latest data, which is conducive to returning the latest data information to the user.
[0055] Optionally, in response to a query request, an operation of retrieving database return data related to the query request from the first Internet data in the database according to an index through a pre-set database retrieval engine includes: in response to the query request, creating a query thread corresponding to the query request; deploying the query thread in a pre-set thread pool, wherein the thread pool is deployed with multiple query threads running in parallel with the query thread; and calling the database retrieval engine to retrieve the database return data in the database through the query thread corresponding to the query request.
[0056] Specifically, refer to Figure 3As shown, the data query platform 200 is also provided with a thread scheduling module 220 for creating a query thread corresponding to the query request according to the query request received from the user interface module 210, and using the created query thread to call the query interface unit 233 in the search module 230 for data query retrieval.
[0057] Among them reference Figure 3 As shown, the thread scheduling module 220 is provided with a thread pool 221, so that after creating the query thread, the thread scheduling module 220 deploys the query thread in the thread pool 221. Figure 5 FIG. 2 shows a schematic diagram of the thread pool 221. Figure 3 and Figure 5 As shown, after creating a query thread corresponding to a query request, the thread scheduling module 220 defines a unique ID for the query thread, such as “cur”. The thread scheduling module 220 then deploys the query thread cur to the process pool 221 .
[0058] refer to Figure 5 As shown, the thread pool 221 can, for example, simultaneously support N query threads to concurrently call the search module 230. Thus, the query thread cur can concurrently call the query interface unit 231 together with other query threads abc, wad, ddc, etc. to perform queries.
[0059] In this way, the data query platform 200 can receive query requests from different users in parallel, and process these query requests in parallel through multiple query threads deployed in the thread pool, so as to quickly provide feedback to the query requests of multiple users.
[0060] In addition, reference Figure 3 As shown, the thread scheduling module 220 is also provided with a thread waiting queue 222. Thus, when the data query platform 200 receives a query request from the terminal device 101 and the threads deployed in the thread pool 221 are full and no more threads can be deployed in the thread pool 221, the thread scheduling module 220 can cache the thread in the thread waiting queue 222 first.
[0061] For example, after receiving the query request sent by the terminal device 101, the thread scheduling module 220 will create a query thread cur corresponding to the query request. Figure 6As shown, since the threads deployed by the thread pool 221 are full and no new threads can be deployed, the thread scheduling module 220 temporarily deploys the query thread cur in the thread waiting queue 222. For example, the thread waiting queue 222 can adopt a first-in-first-out method. When a query thread (such as the query thread ddc) in the thread pool 221 completes its task and terminates, and is cleared in the thread pool 221, the thread scheduling module 220 deploys the query thread fac ranked first in the thread waiting queue 222 to the thread pool 221. Therefore, when the query thread cur ranks first in the thread waiting queue 222, it can replace the completed query thread in the thread pool 221 and start running in the thread pool 221.
[0062] In this way, when a large number of users send query requests to the data query platform 200 at the same time, the pressure of the data query platform 200 can be effectively relieved through the thread waiting queue, thereby ensuring the stable operation of the data query platform 200.
[0063] In addition, reference Figure 3 As shown, the thread scheduling module 220 also includes a data temporary storage unit 224 and a task matching unit 223. Thus, in the query thread cur corresponding to the query request, Figure 6 When deployed in the thread waiting queue 222 as shown, the thread scheduling module 220 can obtain data information corresponding to the query thread cur through the data temporary storage unit 224 and the task matching unit 223.
[0064] Specifically, when the thread scheduling module 220 creates a corresponding query thread cur in response to a query request received from the terminal device 101, the data temporary storage unit 224 will further allocate a corresponding storage area for the query thread cur in the storage space (such as a cache or a virtual cache divided on the hard disk), and mark the storage area with a specific identifier (for example, it can be the same identifier cur as the query thread).
[0065] in Figure 7 FIG. 2 is a schematic diagram showing a storage space managed by the data temporary storage unit 224. Figure 7 As shown, the storage space managed by the data temporary storage unit 224 is divided into two parts. The storage space 1 is divided into multiple storage areas, which are respectively used to temporarily store the database return data retrieved from the database by each past search task, that is, the database return data retrieved from the database by the query task executed by the query thread created by the data query platform 200 in response to each past historical query request.
[0066] Preferably, the storage space 1 only stores the database return data retrieved from the database by each query task after the search module 230 synchronizes the database with the acquired Internet data for the most recent time. And each time the search module 230 synchronizes the database with the acquired Internet data, the data temporary storage unit 224 clears the data temporarily stored in the storage space 1. In this way, it can be ensured that the data in the database are the same when each query task retrieves the database.
[0067] The storage space 2 includes a plurality of storage areas, which are allocated to corresponding query threads by the data temporary storage unit 224 and are used to temporarily store database return data related to the corresponding query threads.
[0068] In addition, reference Figure 3 As shown, the thread scheduling module 220 also includes a task matching unit 223. When the thread scheduling module 220 creates a query thread cur according to a query request of the terminal device 101 and deploys it in the thread waiting queue 222, the task matching unit 223 parses the query request to determine the query task corresponding to the query request (or query thread cur).
[0069] Then the task matching unit 223 matches the parsed query task corresponding to the query request with the past query tasks corresponding to each storage area of the storage space 1, thereby determining the query task that is the same as the query task corresponding to the query request in the past query tasks.
[0070] For example, the task matching unit 223 parses the query request to find out that the database search engine needs to search for data related to the stock trading of listed company A in Table A, Table B, and Table C of the database. Therefore, the task matching unit 223 can parse out the query tasks corresponding to the query request as follows:
[0071] Query task 1, retrieve the data related to the stock transaction of listed company A in table A; query task 2, retrieve the data related to the stock transaction of listed company A in table B; and query task 3, retrieve the data related to the stock transaction of listed company A in table C.
[0072] In addition, the task matching unit 223 performs Figure 7 The query tasks corresponding to the storage areas in the storage space 1 shown are analyzed, and three query tasks are analyzed to retrieve data related to the stock trading of listed company A. They include: query task 1, retrieving data related to the stock trading of listed company A in table B of the database; query task 2, retrieving data related to the stock trading of listed company A in table D of the database; and query task 3, retrieving data related to the stock trading of listed company A in table E of the database.
[0073] Therefore, the task matching unit 223 can determine that the query task 2 of the query request corresponding to the query thread cur is the same query task as the previous query task 1.
[0074] Then the data temporary storage unit 224 copies the database return data stored in the storage area corresponding to the past query task 1 to the storage area corresponding to the query thread cur. That is, the database return data related to the stock trading of listed company A retrieved from table B of the database by the past query task 1 is copied from storage space 1 to the storage area cur corresponding to the query thread cur in storage space 2. In this way, for the query request received from the terminal device 101, before the thread scheduling module 220 deploys the corresponding thread cur to the thread pool 221, the database return data of the part of the query task corresponding to the query thread can be first stored in the storage area cur corresponding to the query thread.
[0075] Thus, when the query thread cur is deployed to the thread pool 221 and starts the retrieval corresponding to the query request. The database retrieval engine will only execute the query tasks of the query thread cur that are different from the previous query tasks, such as the query task of retrieving the database return data related to the stock trading of the listed company A in Table A and Table C of the database. The data temporary storage unit 224 will combine the database return data retrieved by the query thread cur (i.e., the database return data related to the stock trading of the listed company A in Table A and Table C of the database) with the database return data related to the stock trading of the listed company A in the previously stored Table B, and use the combined data (i.e., the database return data related to the stock trading of the listed company A in Table A to Table C) as the database return data corresponding to the query request.
[0076] Thus, in this way, when the query thread is deployed in the thread waiting queue 222, the data query platform 200 will use this time to determine the partial database return data corresponding to the query thread from the storage space managed by the data temporary storage unit 224. As a result, the query thread obtains the partial database return data corresponding to its retrieval task before it starts execution. Therefore, in this way, the speed of data query can be greatly accelerated. Especially when a large number of users send query requests to the data query platform 200 through terminal devices, this method can greatly reduce the time for the query thread to query the corresponding database return data in the database. Thus, the data query platform 200 can feedback the query request sent by the user at a faster speed. Moreover, in this way, the repeated execution of the same query task can also be reduced, thereby greatly improving the query efficiency of the data query platform 200.
[0077] In addition, reference Figure 1As shown, according to the second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.
[0078] According to the technical solution disclosed in the present invention, the data query platform first obtains Internet data from the data source of the Internet, then adds an index to the obtained data, and saves the data after adding the index to the database of the data query platform. Thus, after the data query platform receives the query request of the user, it does not obtain Internet data by searching the Internet, but retrieves the data obtained from the Internet in the database according to the index, thereby obtaining data information related to the query request (i.e., the database returns data). Therefore, compared with the prior art, since the technical solution disclosed in the present invention updates the Internet data to the database and adds an index in advance, the data related to the query request can be retrieved in the database according to the added index, thereby greatly shortening the time for querying data. After testing and verification, it takes about 10 seconds to query more than 3 million pieces of data according to the prior art, but it only takes 2 seconds to complete through the technical solution disclosed in the present invention. Thus, compared with the prior art, the time for retrieving data is greatly reduced, and the technical problem that the feedback time for data query in the massive data of the Internet is too long, resulting in poor user experience, which is solved in the prior art.
[0079] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0081] Example 2
[0082] Figure 8The data query device 800 according to this embodiment is shown. The device 800 corresponds to the method according to the first aspect of embodiment 1 and is used in a data query platform. Figure 8 As shown, the device 800 includes: a first Internet data acquisition module 810, which is used to acquire first Internet data from the Internet using a preset data crawling program, and store the first Internet data in a database of a data query platform after adding a corresponding index to the first Internet data; a query request receiving module 820, which is used to receive a data query request; and a database data return module 830, which is used to respond to the data query request and retrieve database return data related to the query request from the first Internet data in the database according to the index through a preset database retrieval engine in the database.
[0083] Optionally, the first Internet data acquisition module 810 includes: a historical information acquisition submodule, used to acquire historical information of related operations of storing Internet data in a database; and a time interval determination submodule, used to determine the time interval for performing the operation of acquiring the first Internet data from the Internet based on the historical information.
[0084] Optionally, the device also includes: a second Internet data acquisition module, which is used to respond to the query request and use a pre-set Internet search engine to control the data crawling program to obtain second Internet data related to the query request from the Internet, wherein the second Internet data is data that has not been updated in the database; and a retrieval data return module, which is used to combine the database retrieval data and the second Internet data to generate retrieval return data corresponding to the query request.
[0085] Optionally, the database data return module 830 includes: a thread creation submodule, which is used to create a data query thread corresponding to the data query request in response to the data query request; a thread deployment submodule, which is used to deploy the data query thread in a pre-set thread pool, wherein the thread pool is deployed with multiple data query threads running in parallel with the data query thread; and a database data return submodule, which is used to call the database retrieval engine to retrieve the database return data in the database through the data query thread corresponding to the data query request.
[0086] According to the technical solution disclosed in the present invention, the data query platform first obtains Internet data from the data source of the Internet, then adds an index to the obtained data, and saves the data after adding the index to the database of the data query platform. Thus, after the data query platform receives the query request of the user, it does not obtain Internet data by searching the Internet, but retrieves the data obtained from the Internet in the database according to the index, thereby obtaining data information related to the query request (i.e., the database returns data). Therefore, compared with the prior art, since the technical solution disclosed in the present invention updates the Internet data to the database and adds an index in advance, the data related to the query request can be retrieved in the database according to the added index, thereby greatly shortening the time for querying data. After testing and verification, it takes about 10 seconds to query more than 3 million pieces of data according to the prior art, but it only takes 2 seconds to complete through the technical solution disclosed in the present invention. Thus, compared with the prior art, the time for retrieving data is greatly reduced, and the technical problem that the feedback time for data query in the massive data of the Internet is too long, resulting in poor user experience, which is solved in the prior art.
[0087] Example 3
[0088] Fig. 9 The data query device 900 according to the first aspect of this embodiment is shown, and the device 900 corresponds to the method according to the first aspect of embodiment 1. Fig. 9 As shown, the device 900 includes: a processor 910; and a memory 920, which is connected to the processor 920 and is used to provide the processor 920 with instructions for processing the following processing steps: using a preset data crawling program to obtain first Internet data from the Internet, and storing the first Internet data in a database of a data query platform after adding corresponding indexes to the first Internet data; receiving a data query request; and responding to the data query request, retrieving database return data related to the query request from the first Internet data in the database according to the index through a preset database retrieval engine in the database.
[0089] Optionally, the operation of obtaining the first Internet data from the Internet using a preset data capture program includes: obtaining historical information of related operations of storing the Internet data in a database; and determining a time interval for executing the operation of obtaining the first Internet data from the Internet based on the historical information.
[0090] Optionally, the memory is also used to provide the processor with instructions for processing the following processing steps: in response to a query request, using a pre-set Internet retrieval engine to control a data crawler program to obtain second Internet data related to the query request from the Internet, wherein the second Internet data is data that has not been updated in the database; and combining the database retrieval data and the second Internet data to generate retrieval return data corresponding to the query request.
[0091] Optionally, in response to a data query request, an operation of retrieving database return data related to the query request from the first Internet data in the database according to an index through a pre-set database retrieval engine includes: in response to the data query request, creating a data query thread corresponding to the data query request; deploying the data query thread in a pre-set thread pool, wherein the thread pool is deployed with multiple data query threads running in parallel with the data query thread; and calling the database retrieval engine to retrieve the database return data in the database through the data query thread corresponding to the data query request.
[0092] According to the technical solution disclosed in the present invention, the data query platform first obtains Internet data from the data source of the Internet, then adds an index to the obtained data, and saves the data after adding the index to the database of the data query platform. Thus, after the data query platform receives the query request of the user, it does not obtain Internet data by searching the Internet, but retrieves the data obtained from the Internet in the database according to the index, thereby obtaining data information related to the query request (i.e., the database returns data). Therefore, compared with the prior art, since the technical solution disclosed in the present invention updates the Internet data to the database and adds an index in advance, the data related to the query request can be retrieved in the database according to the added index, thereby greatly shortening the time for querying data. After testing and verification, it takes about 10 seconds to query more than 3 million pieces of data according to the prior art, but it only takes 2 seconds to complete through the technical solution disclosed in the present invention. Thus, compared with the prior art, the time for retrieving data is greatly reduced, and the technical problem that the feedback time for data query in the massive data of the Internet is too long, resulting in poor user experience, which is solved in the prior art.
[0093] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0094] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0095] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0097] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0099] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A data query method, used in a data query platform, It is characterized in that include: Acquire first Internet data from the Internet using a preset data crawling program, and store the first Internet data in a database of the data query platform after adding corresponding indexes to the first Internet data; Receive data query requests; as well as In response to the data query request, a pre-set database search engine is used to retrieve database return data related to the query request from the first Internet data in the database according to the index, wherein The operation of acquiring the first Internet data from the Internet using a preset data crawling program includes: acquiring historical information of the operation of storing the Internet data in the database; and determining a time interval for executing the operation of acquiring the first Internet data from the Internet according to the historical information, and The method further includes: in response to the query request, using a preset Internet search engine to control the data crawling program to obtain second Internet data related to the query request from the Internet, wherein the second Internet data is data that is not synchronized with the database; and combining the database search data and the second Internet data to generate search return data corresponding to the query request, and The method further includes adding a tag to the second Internet data and storing it in the database, wherein In response to the data query request, the operation of retrieving database return data related to the query request from the first Internet data in the database according to the index through a preset database retrieval engine includes: in response to the data query request, creating a data query thread corresponding to the data query request; deploying the data query thread in a preset thread pool, wherein the thread pool is deployed with multiple data query threads running in parallel with the data query thread; and calling the database retrieval engine to retrieve the database return data in the database through the data query thread corresponding to the data query request, and When the query request is received and the threads deployed in the thread pool are full and no more threads can be deployed in the thread pool, the threads are cached in a thread waiting queue, wherein When a query thread corresponding to the query request is deployed in the thread waiting queue, data information corresponding to the query thread is obtained, and when a corresponding query thread is created in response to the query request, a corresponding storage area is allocated to the query thread in a storage space, and the storage area is marked with a specific identifier, wherein The first storage space is divided into a plurality of storage areas, each of which is used to temporarily store the database return data retrieved from the database by each query task in the past. The first storage space only stores the database return data retrieved from the database by each query task after the database is updated most recently by synchronizing the database with the acquired Internet data. After each synchronization of the database with the acquired Internet data, the temporarily stored data in the first storage space is cleared. The second storage space includes a plurality of storage areas, which are allocated to corresponding query threads and used to temporarily store the database return data related to the corresponding query threads, wherein When the query thread is created and deployed in the thread waiting queue according to the query request, the query request is parsed to determine the query task corresponding to the query request, the parsed query task corresponding to the query request is matched with the past query tasks corresponding to each storage area of the first storage space, so as to determine the query task that is the same as the query task corresponding to the query request in the past query tasks, and the database return data stored in the storage area corresponding to the past query task is copied to the storage area corresponding to the query thread, wherein When the query thread is deployed to the thread pool and starts to start the retrieval corresponding to the query request, the query task of the query thread that is different from the previous query task is executed, the database return data retrieved by the query thread is combined with the previously stored database return data, and the combined data is used as the database return data corresponding to the query request, wherein When the query thread is deployed in the thread waiting queue, part of the database return data corresponding to the query thread is determined from the storage space, so that the query thread obtains part of the database return data corresponding to the query task before starting execution.
2. A storage medium, It is characterized in that The storage medium includes a stored program, wherein when the program is executed, the method of claim 1 is executed by a processor.
3. A data query device, used for a data query platform, It is characterized in that include: A first Internet data acquisition module, used to acquire first Internet data from the Internet using a preset data crawling program, and store the first Internet data in a database of the data query platform after adding a corresponding index to the first Internet data; A query request receiving module, used for receiving a data query request; as well as A database data return module is used to respond to the data query request and retrieve database return data related to the query request from the first Internet data in the database according to the index through a preset database retrieval engine, wherein The first Internet data acquisition module includes: a historical information acquisition submodule, which is used to acquire historical information of the operation of storing the Internet data in the database; and a time interval determination submodule, which is used to determine the time interval for performing the operation of acquiring the first Internet data from the Internet according to the historical information, and The device further comprises: a second Internet data acquisition module, which is used to respond to the query request and use a preset Internet search engine to control the data crawling program to obtain second Internet data related to the query request from the Internet, wherein the second Internet data is data that has not been updated in the database; and a search data return module, which is used to combine the database search data and the second Internet data to generate search return data corresponding to the query request, and The device also includes adding a tag to the second Internet data and storing it in the database, wherein the database data return module includes: a thread creation submodule, which is used to respond to the data query request and create a data query thread corresponding to the data query request; a thread deployment submodule, which is used to deploy the data query thread in a pre-set thread pool, wherein the thread pool is deployed with multiple data query threads running in parallel with the data query thread; and a database data return submodule, which is used to call the database retrieval engine to retrieve the database return data in the database through the data query thread corresponding to the data query request, and When the query request is received and the threads deployed in the thread pool are full and no more threads can be deployed in the thread pool, the threads are cached in a thread waiting queue, wherein When a query thread corresponding to the query request is deployed in the thread waiting queue, data information corresponding to the query thread is obtained, and when a corresponding query thread is created in response to the query request, a corresponding storage area is allocated to the query thread in a storage space, and the storage area is marked with a specific identifier, wherein The first storage space is divided into a plurality of storage areas, each of which is used to temporarily store the database return data retrieved from the database by each query task in the past. The first storage space only stores the database return data retrieved from the database by each query task after the database is updated most recently by synchronizing the database with the acquired Internet data. After each synchronization of the database with the acquired Internet data, the temporarily stored data in the first storage space is cleared. The second storage space includes a plurality of storage areas, which are allocated to corresponding query threads and used to temporarily store the database return data related to the corresponding query threads, wherein When the query thread is created and deployed in the thread waiting queue according to the query request, the query request is parsed to determine the query task corresponding to the query request, the parsed query task corresponding to the query request is matched with the past query tasks corresponding to each storage area of the first storage space, so as to determine the query task that is the same as the query task corresponding to the query request in the past query tasks, and the database return data stored in the storage area corresponding to the past query task is copied to the storage area corresponding to the query thread, wherein When the query thread is deployed to the thread pool and starts to start the retrieval corresponding to the query request, the query task of the query thread that is different from the previous query task is executed, the database return data retrieved by the query thread is combined with the previously stored database return data, and the combined data is used as the database return data corresponding to the query request, wherein When the query thread is deployed in the thread waiting queue, part of the database return data corresponding to the query thread is determined from the storage space, so that the query thread obtains part of the database return data corresponding to the query task before starting execution.
4. A data query device, It is characterized in that include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Acquire first Internet data from the Internet using a preset data crawling program, and store the first Internet data in a database of the data query platform after adding corresponding indexes to the first Internet data; Receive data query requests; as well as In response to the data query request, a pre-set database search engine is used to retrieve database return data related to the query request from the first Internet data in the database according to the index, wherein The operation of acquiring the first Internet data from the Internet using a preset data crawling program includes: acquiring historical information of the operation of storing the Internet data in the database; and determining a time interval for executing the operation of acquiring the first Internet data from the Internet according to the historical information, and The method further includes: in response to the query request, using a preset Internet search engine to control the data crawling program to obtain second Internet data related to the query request from the Internet, wherein the second Internet data is data that is not synchronized with the database; and combining the database search data and the second Internet data to generate search return data corresponding to the query request, and The method further includes adding a tag to the second Internet data and storing it in the database, wherein In response to the data query request, the operation of retrieving database return data related to the query request from the first Internet data in the database according to the index through a preset database retrieval engine includes: in response to the data query request, creating a data query thread corresponding to the data query request; deploying the data query thread in a preset thread pool, wherein the thread pool is deployed with multiple data query threads running in parallel with the data query thread; and calling the database retrieval engine to retrieve the database return data in the database through the data query thread corresponding to the data query request, and When the query request is received and the threads deployed in the thread pool are full and no more threads can be deployed in the thread pool, the threads are cached in a thread waiting queue, wherein When a query thread corresponding to the query request is deployed in the thread waiting queue, data information corresponding to the query thread is obtained, and when a corresponding query thread is created in response to the query request, a corresponding storage area is allocated to the query thread in a storage space, and the storage area is marked with a specific identifier, wherein The first storage space is divided into a plurality of storage areas, each of which is used to temporarily store the database return data retrieved from the database by each query task in the past. The first storage space only stores the database return data retrieved from the database by each query task after the database is updated most recently by synchronizing the database with the acquired Internet data. After each synchronization of the database with the acquired Internet data, the temporarily stored data in the first storage space is cleared. The second storage space includes a plurality of storage areas, which are allocated to corresponding query threads and used to temporarily store the database return data related to the corresponding query threads, wherein When the query thread is created and deployed in the thread waiting queue according to the query request, the query request is parsed to determine the query task corresponding to the query request, the parsed query task corresponding to the query request is matched with the past query tasks corresponding to each storage area of the first storage space, so as to determine the query task that is the same as the query task corresponding to the query request in the past query tasks, and the database return data stored in the storage area corresponding to the past query task is copied to the storage area corresponding to the query thread, wherein When the query thread is deployed to the thread pool and starts to start the retrieval corresponding to the query request, the query task of the query thread that is different from the previous query task is executed, the database return data retrieved by the query thread is combined with the previously stored database return data, and the combined data is used as the database return data corresponding to the query request, wherein When the query thread is deployed in the thread waiting queue, part of the database return data corresponding to the query thread is determined from the storage space, so that the query thread obtains part of the database return data corresponding to the query task before starting execution.
