Data acquisition method and device, electronic equipment and storage medium
By setting up a multi-layer cache architecture in the data acquisition system, using the first cache and the second cache to improve data acquisition efficiency, solving the problem of low data acquisition efficiency in the prior art, and achieving faster and more efficient data access.
Patent Information
- Application Number
- CN202411570881.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-02-14
AI Technical Summary
When accessing data not stored in the cache, the prior art has problems with low data acquisition efficiency, resulting in excessive database access pressure and slow response speed.
By setting the first cache and the second cache, a multi-layer cache architecture is formed. When data is not retrieved in the first cache, searching in the second cache will be attempted, increasing the chance that the data acquisition request will be hit by the cache.
It effectively reduces the number of times you get data directly from the database, thereby improving data acquisition efficiency, reducing database access pressure and improving response speed.
Smart Images
Figure CN119441298A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data acquisition method, device, electronic device and storage medium. Background Art
[0002] With the rapid development of the Internet and the increase in the number of Internet users, the user access level and data volume are getting larger and larger. When the number of requests is large, reading data directly from the database will cause excessive database access pressure and cause crashes. In addition, reading data from the database has a slow response speed. Therefore, hot data with large access volume but little data change is usually stored in the cache to improve the data reading and writing speed.
[0003] In the related art, when accessing data that is not stored in the cache, since the data does not exist in the cache, it is still necessary to obtain the data from the database, resulting in the same problem of low data acquisition efficiency when accessing such data. Therefore, it is urgent to provide a data acquisition method to solve the above technical problems. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a data acquisition method, device, electronic device and storage medium, which forms a multi-layer cache architecture by setting a first cache and a second cache. When data is not obtained in the first cache, it will try to find it in the second cache, which increases the chance that the data acquisition request is cached. This multi-level cache mechanism can effectively reduce the number of times data is directly obtained from the database, thereby improving data acquisition efficiency.
[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a data acquisition method, the method comprising:
[0006] receiving a first data acquisition request, wherein the request type of the first data acquisition request is a data query type;
[0007] When there is no data acquisition request matching the first data acquisition request in the first cache, constructing a second data acquisition request of a query statement type according to the first data acquisition request, wherein the first cache is used to store data acquisition requests of the data query type and data corresponding to each data acquisition request of the data query type;
[0008] When there is a target data acquisition request matching the second data acquisition request in the second cache, target data having a preset mapping relationship with the target data acquisition request is acquired and the target data is returned. The second cache is used to store data acquisition requests of query statement types and data corresponding to each data acquisition request of query statement type.
[0009] In some embodiments, the method further comprises:
[0010] When there is a data acquisition request matching the first data acquisition request in the first cache, first data having a preset mapping relationship with the first data acquisition request is acquired, and the first data is returned.
[0011] In some embodiments, after returning the first data, the method further includes:
[0012] Obtaining a target historical request number of the first data acquisition request;
[0013] Increment the target historical request count by one;
[0014] Determine the historical request count of data acquisition requests of each data query type;
[0015] The data acquisition requests of the data query type and the corresponding data stored in the first cache are updated according to each of the historical request times.
[0016] In some embodiments, updating the data acquisition request of the data query type and the corresponding data stored in the first cache according to each of the historical request times includes:
[0017] Sort each of the historical request times in descending order to obtain a sorted sequence;
[0018] Filtering out a plurality of target historical request times from the sorting sequence according to a preset filtering rule;
[0019] The data acquisition request of the data query type corresponding to each of the target historical request times and the corresponding data are stored in the first cache.
[0020] In some embodiments, filtering out a plurality of target historical request times from the sorting sequence according to a preset filtering rule includes:
[0021] Determine a preset number of historical request times before the sorting sequence as a target historical request time; or
[0022] A preset percentage of historical request times in the sorting sequence is determined as a target historical request time.
[0023] In some embodiments, constructing a second data acquisition request whose request type is a query statement type according to the first data acquisition request includes:
[0024] Acquire the data acquisition condition and data acquisition field included in the first data acquisition request;
[0025] Generate a data acquisition request template whose request type is a query statement type;
[0026] The data acquisition condition and the data acquisition field are respectively written into corresponding positions of the data acquisition request template to obtain a second data acquisition request of a query statement type.
[0027] In some embodiments, the method further comprises:
[0028] When there is no acquisition request matching the second data acquisition request in the second cache, searching the database for second data having a preset mapping relationship with the second data acquisition request, and returning the second data;
[0029] The second data acquisition request and the second data are stored in the second cache, and a mapping relationship between the second data acquisition request and the second data is established.
[0030] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a data acquisition device, comprising:
[0031] An acquisition unit, configured to receive a first data acquisition request, wherein the request type of the first data acquisition request is a data query type;
[0032] a construction unit, configured to construct, according to the first data acquisition request, a second data acquisition request of a query statement type when no data acquisition request matching the first data acquisition request exists in the first cache, wherein the first cache is configured to store data acquisition requests of the data query type and data corresponding to each data acquisition request of the data query type;
[0033] A return unit is used to obtain target data having a preset mapping relationship with the target data acquisition request and return the target data when there is a target data acquisition request matching the second data acquisition request in the second cache, and the second cache is used to store data acquisition requests of query statement type and data corresponding to each data acquisition request of query statement type.
[0034] In some embodiments, the apparatus is further configured to:
[0035] When there is a data acquisition request matching the first data acquisition request in the first cache, first data having a preset mapping relationship with the first data acquisition request is acquired, and the first data is returned.
[0036] In some embodiments, the apparatus is further configured to:
[0037] Obtaining a target historical request number of the first data acquisition request;
[0038] Increment the target historical request count by one;
[0039] Determine the historical request count of data acquisition requests of each data query type;
[0040] The data acquisition requests of the data query type and the corresponding data stored in the first cache are updated according to each of the historical request times.
[0041] In some embodiments, the apparatus is further configured to:
[0042] Sort each of the historical request times in descending order to obtain a sorted sequence;
[0043] Filtering out a plurality of target historical request times from the sorting sequence according to a preset filtering rule;
[0044] The data acquisition request of the data query type corresponding to each of the target historical request times and the corresponding data are stored in the first cache.
[0045] In some embodiments, the apparatus is further configured to:
[0046] Determine a preset number of historical request times before the sorting sequence as a target historical request time; or
[0047] A preset percentage of historical request times in the sorting sequence is determined as a target historical request time.
[0048] In some embodiments, a construction unit is used to:
[0049] Acquire the data acquisition condition and data acquisition field included in the first data acquisition request;
[0050] Generate a data acquisition request template whose request type is a query statement type;
[0051] The data acquisition condition and the data acquisition field are respectively written into corresponding positions of the data acquisition request template to obtain a second data acquisition request of a query statement type.
[0052] In some embodiments, the apparatus is further configured to:
[0053] The second data acquisition request and the second data are stored in the second cache, and a mapping relationship between the second data acquisition request and the second data is established.
[0054] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0055] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0056] The data acquisition method, device, electronic device and storage medium proposed in the present application receive a first data acquisition request, the request type of the first data acquisition request is a data query type; when there is no data acquisition request matching the first data acquisition request in the first cache, a second data acquisition request of a query statement type is constructed according to the first data acquisition request, and the first cache is used to store the data acquisition request of the data query type and the data corresponding to each data acquisition request of the data query type; when there is a target data acquisition request matching the second data acquisition request in the second cache, the target data with a preset mapping relationship established with the target data acquisition request is acquired, and the target data is returned, and the second cache is used to store the data acquisition request of the query statement type and the data corresponding to each data acquisition request of the query statement type. In this way, by setting the first cache and the second cache, a multi-layer cache architecture is formed. When the data is not obtained in the first cache, it will try to find it in the second cache, which increases the chance of the data acquisition request being cached. This multi-level cache mechanism can effectively reduce the number of times data is directly obtained from the database, thereby improving data acquisition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 is a flow chart of a data acquisition method provided in an embodiment of the present application;
[0058] Figure 2 is another flow chart of the data acquisition method provided in the embodiment of the present application;
[0059] Figure 3 is another flow chart of the data acquisition method provided in the embodiment of the present application;
[0060] Figure 4 yes Figure 3 Flow chart of step S304 in FIG.
[0061] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.
[0062] Figure 6 yes Figure 1 Flow chart of step S102 in FIG.
[0063] Figure 7 yes Figure 1 Another flow chart of step S102 in ;
[0064] Figure 8 is a structural schematic diagram of a data acquisition device provided in an embodiment of the present application;
[0065] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0067] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0069] First, some nouns involved in this application are analyzed:
[0070] queryData query: is an operation method used to obtain data in a specific system environment. It has the following characteristics and possible implementation methods: 1. Data acquisition: Its main purpose is to obtain data that meets specific requirements from the system. For example, in a data visualization system, a user may initiate a queryData query through an interface operation to obtain data used to generate a specific chart. 2. Meet business needs: According to different business scenarios and user needs, queryData query can adapt to various complex data acquisition requirements. For example, obtain sales data within a certain time period, relevant data of a specific product category, or data that meets specific filtering conditions and sorting requirements.
[0071] SQL query (Structured Query Language query) is a way of interacting with a database and obtaining the required data. SQL query uses structured query language (SQL) to send requests to the database management system (DBMS) to retrieve, manipulate and manage data. It allows users to extract data that meets specific conditions from the database and can sort, group, aggregate and perform other operations on the data.
[0072] SQL queries can be applied to: 1. Various information systems: such as enterprise resource planning (ERP) systems, customer relationship management (CRM) systems, etc. SQL queries are used to retrieve data that meets user needs. For example, query the inventory quantity of a specific product in the ERP system, and query the order history of a certain customer in the CRM system. 2. Generate data analysis reports: For example, by grouping and aggregating sales data, calculate the total sales of each region to provide data support for sales analysis. 3. Database administrators: SQL queries can be used to check the status of the database, such as querying the structure of the table, index status, and finding duplicate data.
[0073] Based on this, the embodiments of the present application provide a data acquisition method, device, electronic device and storage medium, which aim to reduce the number of times data is directly acquired from a data source, thereby improving the overall performance of the system.
[0074] The data acquisition method, device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data acquisition method in the embodiments of the present application is described.
[0075] The data acquisition method provided in the embodiment of the present application relates to the field of data processing technology. The data acquisition method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data acquisition method, etc., but is not limited to the above forms.
[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0078] Figure 1 is an optional flow chart of the data acquisition method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S103.
[0079] Step S101: receiving a first data acquisition request, where the request type of the first data acquisition request is a data query type.
[0080] Among them, the data acquisition request is a request instruction received by the system for acquiring data, which contains the user or business logic's demand information for data. The data acquisition request can be input by the user through the mouse and / or keyboard, or it can be a request sent by the user to the electronic device through a remote device, which is not limited here. The data query type is a specific data acquisition request type, which may be a query method customized by the system, used to initiate data query operations at the system level. Its operation logic is closely related to the system architecture and data processing flow.
[0081] For example, in a visual data viewing (data acquisition, data query, etc.) scenario, the data query type may be a queryData query type, which is used to obtain chart data in the form of a chart. The electronic device receives a data acquisition request with a request type of a data query type (queryData query type). This request may come from an operation of the user interface, such as a user selecting to view a chart or data range in a data visualization tool, thereby triggering this data acquisition request.
[0082] Step S102, when there is no data acquisition request matching the first data acquisition request in the first cache, construct a second data acquisition request whose request type is a query statement type according to the first data acquisition request, and the first cache is used to store data acquisition requests of the data query type and data corresponding to each data acquisition request of the data query type.
[0083] The first cache is used to store data acquisition requests of the data query type and the data corresponding to each request. It is part of the system cache system and mainly manages caches for data query type requests so that data can be quickly provided when similar requests arrive in the future, reducing the time and system overhead of data acquisition.
[0084] Specifically, by checking whether there is a data acquisition request in the first cache that matches the received first data acquisition request. The matching here involves comparison of multiple aspects such as request parameters, query conditions, and data range involved. If there is no matching request in the first cache, the electronic device will construct a second data acquisition request with a request type of query statement type (SQL query type) based on the first data acquisition request. During the construction process, the system needs to extract key information in the first data acquisition request, such as data acquisition conditions (such as time range, data category, etc.) and data acquisition fields (such as specific data columns). Subsequently, the electronic device will generate a second data acquisition request of query statement type.
[0085] Step S103, when there is a target data acquisition request matching the second data acquisition request in the second cache, the target data having a preset mapping relationship with the target data acquisition request is acquired, and the target data is returned. The second cache is used to store data acquisition requests of query statement type and data corresponding to each data acquisition request of query statement type.
[0086] The second cache is used to store query statement type data acquisition requests and the data corresponding to each request. It is also part of the system cache system, focusing on caching SQL query type requests. When certain conditions are met, data can be directly obtained from here to improve system performance. The preset mapping relationship is a data association rule pre-set in the second cache, which is used to find data associated with the data acquisition request in the second cache. This mapping relationship ensures the accuracy and consistency of data obtained from the second cache.
[0087] Specifically, if there is a matching request in the second cache, the system obtains the target data associated with the target data acquisition request according to the preset mapping relationship, and returns it as the final result. This step demonstrates the function of the second cache and the importance of the preset mapping relationship. By properly utilizing the second cache and the mapping relationship, the required data can be quickly obtained.
[0088] Steps S101 to S103 shown in the embodiment of the present application are as follows: by receiving a first data acquisition request, the request type of the first data acquisition request is a data query type; when there is no data acquisition request matching the first data acquisition request in the first cache, a second data acquisition request of a query statement type is constructed according to the first data acquisition request, and the first cache is used to store the data acquisition request of the data query type and the data corresponding to each data query type data acquisition request; when there is a target data acquisition request matching the second data acquisition request in the second cache, the target data with a preset mapping relationship established with the target data acquisition request is acquired, and the target data is returned, and the second cache is used to store the data acquisition request of the query statement type and the data corresponding to each query statement type data acquisition request. In this way, by setting the first cache and the second cache, a multi-layer cache architecture is formed. When the data is not obtained in the first cache, it will try to find it in the second cache, which increases the chance of the data acquisition request being cached. This multi-level cache mechanism can effectively reduce the number of times data is directly obtained from the database, thereby improving data acquisition efficiency.
[0089] See also Figure 2 In some embodiments, the data acquisition method may also include but is not limited to step S201:
[0090] Step S201: when there is a data acquisition request matching the first data acquisition request in the first cache, first data having a preset mapping relationship with the first data acquisition request is acquired, and the first data is returned.
[0091] Among them, after the electronic device receives the first data acquisition request, it first searches in the first cache. The first cache stores data acquisition requests of data query types and data corresponding to data acquisition requests of each data query type. The system compares the received first data acquisition request with the existing requests in the first cache to determine whether there is a matching request. The "match" here means that the key information of the request (such as query conditions, data fields involved, relevant parameters of the request, etc.) is highly consistent.
[0092] Specifically, if a matching request is found in the first cache, the electronic device will then determine the preset mapping relationship established with the first data acquisition request. This preset mapping relationship is pre-set in the system and is used to accurately associate the request with the corresponding data. It ensures that the data related to the request can be correctly obtained in the cache. According to the determined preset mapping relationship, the electronic device obtains the first data associated with it from the first cache. This first data is the data corresponding to the received first data acquisition request, which has been cached at some point in the past. After obtaining the first data, the system returns it to the initiator of the request (which may be a user interface, other business modules, etc.) as the final result, thereby completing the processing of the data acquisition request.
[0093] In this way, when there is a matching request in the first cache, data is directly obtained from the cache, avoiding complex data query and processing processes, and greatly reducing the time for data acquisition. This is very critical for application scenarios that require fast response, such as real-time data analysis, interactive user interfaces, etc., and can significantly improve the system's response speed and enhance user experience.
[0094] See also Figure 3 In some embodiments, the data acquisition method may also include but is not limited to steps S301 to S304:
[0095] Step S301, obtaining the target historical request number of the first data acquisition request.
[0096] In order to avoid the problem of long-term storage of some infrequently queried data in the first cache causing a large cache occupancy, the first cache only stores some data with a high query frequency so that these data can be quickly queried. Therefore, after the first data is returned, the target historical request number of the first data acquisition request is obtained.
[0097] Step S302, increase the target historical request count by one.
[0098] Since data query is performed again through the first data acquisition request this time, the target historical request number obtained in step S301 is increased by one, indicating that the request is executed once again.
[0099] Step S303: determining the historical request count of data acquisition requests of each data query type.
[0100] The electronic device traverses all data query type data acquisition requests (not limited to data acquisition requests stored in the first cache, but also includes data acquisition requests of data query types not stored in the first cache).
[0101] Specifically, for each data query type request, the electronic device will determine the corresponding historical request count.
[0102] Step S304: updating the data acquisition request of the data query type and the corresponding data stored in the first cache according to each of the historical request times.
[0103] The electronic device updates the data in the first cache according to the determined historical request times of each data query type request.
[0104] If a request has a high number of historical requests, it may mean that the data set corresponding to this request is more likely to be accessed again. The system can consider adjusting the storage location of this request and its corresponding data in the cache to the first cache that is easier to access, or adopt some optimization strategies, such as extending its retention time in the cache.
[0105] At the same time, for requests with a lower historical request count and their corresponding data, corresponding adjustments can be made according to the cache management strategy. For example, if the cache space is tight, you can consider clearing the data corresponding to requests with a lower historical request count first to free up cache space for more frequently accessed data.
[0106] In this way, by storing only data with a high query frequency, the cache is prevented from occupying too much space with infrequently accessed data, ensuring that the cache stores data that is more likely to be queried again. This can improve the cache hit rate and reduce unnecessary cache search time, thereby improving the overall performance of the system. The data in the cache is continuously updated according to the number of historical requests, so that the cache can adapt to changes in query requirements at different stages. For example, some data may have a low query frequency in a specific period, but as the business changes, the query frequency may increase, and the system can adjust the cache content in time to adapt to this change.
[0107] See also Figure 4 In some embodiments, step S304 may include but is not limited to steps S401 to S403:
[0108] Step S401, sorting each of the historical request times in descending order to obtain a sorting sequence.
[0109] The historical request times of data acquisition requests of all data query types are collected. These historical request times can be obtained from previously recorded storage locations (such as database tables, memory data structures, etc.).
[0110] Specifically, a sorting algorithm (such as quick sort, merge sort, etc.) is used to sort the historical request times in descending order.
[0111] For example, if the number of historical requests is [20, 15, 30, 10], the sorted sequence obtained after sorting is [30, 20, 15, 10].
[0112] Step S402: Filter out multiple target historical request times from the sorting sequence according to a preset filtering rule.
[0113] Among them, the preset screening rules can be set according to the system requirements and resource conditions.
[0114] For example, the preset filtering rule is to filter out the first three historical request times. For the above sorted sequence [30, 20, 15, 10], the filtered target historical request times are [30, 20, 15].
[0115] Step S403: storing the data acquisition request of the data query type corresponding to each target historical request number and the corresponding data in the first cache.
[0116] According to the filtered target historical request times, the corresponding data acquisition request of the specific data query type is determined, which can be determined by querying the mapping relationship established before or according to the unique identifier of the request.
[0117] Specifically, the data acquisition requests of these data query types and the corresponding data are stored in the first cache. If the first cache space is limited, it may be necessary to decide whether to replace the existing cache content according to a cache replacement strategy (such as a least recently used algorithm, etc.).
[0118] For example, for a request with a target historical request count of 20, assuming that the corresponding data is a specific set of sales data, the system stores the request and the corresponding sales data in the first cache so that they can be quickly retrieved the next time they are queried.
[0119] In this way, by sorting the historical request times and filtering out the data corresponding to high-frequency requests and storing them in the first cache, the probability of hitting the first cache during subsequent data queries is greatly increased. For example, for frequently queried sales data, if its historical request times are high, it is stored in the first cache, and the next time there is a similar query, it can be directly obtained from the cache without the need for complex data retrieval operations, thereby improving the system's response speed.
[0120] See also Figure 5 In some embodiments, step S402 may include but is not limited to step S501:
[0121] Step S501, determining a previous preset number of historical request times in the sorting sequence as a target historical request time; or determining a previous preset percentage of historical request times in the sorting sequence as a target historical request time.
[0122] Among them, the value of the preset number is determined, and this value can be set according to the needs and resource conditions of the system. For example, it can be set to the first 10 historical request times. From the historical request times sequence that has been sorted from large to small, select the first preset number of historical request times. Determine these selected historical request times as the target historical request times.
[0123] For example, suppose the sorted historical request count sequence is [50, 45, 40, 35, 30, 25, 20, 15, 10, 5]. If the preset number is set to 3, then the target historical request count is [50, 45, 40].
[0124] It is also possible to: determine a preset percentage value, such as setting it to the first 30% of the number of historical requests.
[0125] Calculate the total number of historical requests in the sorted sequence. For example, suppose there are 100 historical requests. Calculate the specific number based on the preset percentage. If it is 30%, it is 100*0.3=30. Select the historical request number calculated before from the sorted sequence as the target historical request number.
[0126] For example, suppose the sorted historical request count sequence has 100 values, which are arranged in descending order. If the preset percentage is set to 20%, then the target historical request count is the first 100*0.2=20 historical request counts. For example, the first 20 historical request counts may be [800, 750, 700, ..., 150].
[0127] In this way, the amount of high-frequency data to be cached can be accurately determined based on specific business scenarios and system resource conditions. For example, if system resources are limited but it is necessary to ensure that the most frequently queried data can be quickly obtained, a smaller preset number can be set to ensure that only the most critical high-frequency data is stored in the first cache, improving the pertinence and efficiency of the cache. Determining the target number of historical requests with specific numbers makes cache management clearer and more controllable. Developers can make adjustments based on actual conditions to adapt to different performance requirements and resource constraints.
[0128] See also Figure 6 In some embodiments, step S102 may include but is not limited to steps S601 to S603:
[0129] Step S601: Acquire the data acquisition condition and data acquisition field included in the first data acquisition request.
[0130] When receiving the first data acquisition request, the system parses the content of the request. It identifies the data acquisition condition part, which may include filtering conditions such as time range, specific product category, region, etc. For example, the data acquisition condition is "the time range is the past month, and the product category is electronic products". At the same time, the data acquisition field is determined, that is, which specific data items need to be obtained. For example, the data acquisition field is "product name, sales quantity, sales amount".
[0131] For example, the first data acquisition request is to obtain the sales data of mobile phone products on an e-commerce platform in the past week, including product name, sales quantity and sales amount. Then the data acquisition condition is "time range is the past week, product category is mobile phone", and the data acquisition fields are "product name, sales quantity, sales amount".
[0132] Step S602: Generate a data acquisition request template whose request type is a query statement type.
[0133] Among them, a basic query statement template is generated according to the database type and query language specification used by the system. This template usually includes the basic structure of the query, such as the SELECT statement for selecting the fields to be obtained, the FROM statement for specifying which table to obtain data from, and the WHERE statement for setting the filtering conditions.
[0134] For example, for a common relational database, assuming the SQL query language is used, the generated template may be "SELECT [data acquisition field] FROM [table name] WHERE [data acquisition condition]".
[0135] Step S603: Write the data acquisition condition and the data acquisition field into corresponding positions of the data acquisition request template respectively to obtain a second data acquisition request of a query statement type.
[0136] The data acquisition field determined in S601 is replaced with the "[data acquisition field]" part in the template. The data acquisition condition is replaced with the "[data acquisition condition]" part in the template. After the replacement, a second data acquisition request of a complete query statement type is obtained.
[0137] For example, if the table name is "sales_data", the data acquisition fields "product name, sales quantity, sales amount" and the data acquisition condition "time range is the past week, and the product category is mobile phone" are filled in the template respectively, and the second data acquisition request obtained is "SELECT product name, sales quantity, sales amount FROM sales_data WHERE time range is the past week AND product category is mobile phone".
[0138] In this way, the standardized template generation and filling process can be efficiently executed by the system, reducing the time overhead of data acquisition. Compared with building complex query statements from scratch every time, this method can generate the second data acquisition request faster and improve the response speed of the system. Generating templates for specific database types and query language specifications can make full use of the database's query optimization mechanism. The database can better understand and execute these standardized query statements, improve query efficiency, and reduce system resource consumption.
[0139] See also Figure 7 In some embodiments, step S102 may also include but is not limited to steps S701 to S702:
[0140] Step S701: when there is no acquisition request matching the second data acquisition request in the second cache, searching the database for second data having a preset mapping relationship with the second data acquisition request, and returning the second data.
[0141] If no request matching the current second data acquisition request is found in the second cache, the system will initiate a query to the database according to the information in the second data acquisition request, such as the conditions and fields in the query statement.
[0142] Specifically, the database searches the stored data according to the received query request. This process may involve operations such as scanning the database table and using the index to quickly locate the data that meets the conditions. Once the database finds the second data that has a preset mapping relationship with the second data acquisition request, it returns the data to the system.
[0143] Step S702: store the second data acquisition request and the second data in the second cache, and establish a mapping relationship between the second data acquisition request and the second data.
[0144] After the system obtains the second data from the database, the second data acquisition request and the corresponding second data are stored in the second cache.
[0145] At the same time, a mapping relationship between the second data acquisition request and the second data is established. This mapping relationship can be implemented in various ways, for example, a data structure (such as a hash table) can be used to store the corresponding relationship between the request and the data, so that the corresponding cache data can be quickly found according to the request in subsequent queries.
[0146] In this way, when the second cache misses, the data is queried from the database and returned, avoiding the user waiting for a long database query process, reducing the user's waiting time, and improving the system's response speed. For example, in a real-time data analysis system, users can obtain the required data for decision-making more quickly. The data obtained from the database is stored in the second cache and a mapping relationship is established, so that the next time the same query request can directly obtain data from the cache, avoiding repeated database query operations, greatly improving data query efficiency.
[0147] See also Figure 8 The embodiment of the present application also provides a data acquisition device, which can implement the above data acquisition method, and the device includes:
[0148] An acquisition unit, configured to receive a first data acquisition request, wherein the request type of the first data acquisition request is a data query type;
[0149] a construction unit, configured to construct, according to the first data acquisition request, a second data acquisition request of a query statement type when no data acquisition request matching the first data acquisition request exists in the first cache, wherein the first cache is configured to store data acquisition requests of the data query type and data corresponding to each data acquisition request of the data query type;
[0150] A return unit is used to obtain target data having a preset mapping relationship with the target data acquisition request and return the target data when there is a target data acquisition request matching the second data acquisition request in the second cache, and the second cache is used to store data acquisition requests of query statement type and data corresponding to each data acquisition request of query statement type.
[0151] In some embodiments, the apparatus is further configured to:
[0152] When there is a data acquisition request matching the first data acquisition request in the first cache, first data having a preset mapping relationship with the first data acquisition request is acquired, and the first data is returned.
[0153] In some embodiments, the apparatus is further configured to:
[0154] Obtaining a target historical request number of the first data acquisition request;
[0155] Increment the target historical request count by one;
[0156] Determine the historical request count of data acquisition requests of each data query type;
[0157] The data acquisition requests of the data query type and the corresponding data stored in the first cache are updated according to each of the historical request times.
[0158] In some embodiments, the apparatus is further configured to:
[0159] Sort each of the historical request times in descending order to obtain a sorted sequence;
[0160] Filtering out a plurality of target historical request times from the sorting sequence according to a preset filtering rule;
[0161] The data acquisition request of the data query type corresponding to each of the target historical request times and the corresponding data are stored in the first cache.
[0162] In some embodiments, the apparatus is further configured to:
[0163] Determine a preset number of historical request times before the sorting sequence as a target historical request time; or
[0164] A preset percentage of historical request times in the sorting sequence is determined as a target historical request time.
[0165] In some embodiments, a construction unit is used to:
[0166] Acquire the data acquisition condition and data acquisition field included in the first data acquisition request;
[0167] Generate a data acquisition request template whose request type is a query statement type;
[0168] The data acquisition condition and the data acquisition field are respectively written into corresponding positions of the data acquisition request template to obtain a second data acquisition request of a query statement type.
[0169] In some embodiments, the apparatus is further configured to:
[0170] The second data acquisition request and the second data are stored in the second cache, and a mapping relationship between the second data acquisition request and the second data is established.
[0171] The data acquisition device provided by the embodiment of the present application receives a first data acquisition request through an acquisition unit, and the request type of the first data acquisition request is a data query type; the construction unit constructs a second data acquisition request of a query statement type according to the first data acquisition request when there is no data acquisition request matching the first data acquisition request in the first cache, and the first cache is used to store the data acquisition request of the data query type and the data corresponding to each data acquisition request of the data query type; the return unit obtains the target data with a preset mapping relationship established with the target data acquisition request when there is a target data acquisition request matching the second data acquisition request in the second cache, and returns the target data, and the second cache is used to store the data acquisition request of the query statement type and the data corresponding to each data acquisition request of the query statement type. In this way, a multi-layer cache architecture is formed by setting the first cache and the second cache. When the data is not obtained in the first cache, it will try to find it in the second cache, which increases the chance that the data acquisition request is hit by the cache. This multi-level cache mechanism can effectively reduce the number of times data is directly obtained from the database, thereby improving data acquisition efficiency. When there is a matching request in the first cache, data is directly obtained from the cache, which avoids complex data query and processing processes and greatly reduces the time for data acquisition. This is very critical for application scenarios that require fast response, such as real-time data analysis and interactive user interfaces, and can significantly improve the system's response speed and user experience. By only storing data with high query frequency, the cache is prevented from occupying too much space for infrequently accessed data, ensuring that the cache stores data that is more likely to be queried again. This can improve the cache hit rate and reduce unnecessary cache search time, thereby improving the overall performance of the system. The data in the cache is continuously updated according to the number of historical requests, so that the cache can adapt to changes in query requirements at different stages. For example, some data may have a low query frequency in a certain period, but as the business changes, the query frequency may increase, and the system can adjust the cache content in time to adapt to this change. By sorting the number of historical requests and filtering out the data corresponding to high-frequency requests and storing them in the first cache, the probability of hitting the first cache during subsequent data queries is greatly increased. For example, for sales data that is frequently queried, if its historical request number is high, it is stored in the first cache, and the next time there is a similar query, it can be directly obtained from the cache without the need for complex data retrieval operations, thereby improving the system's response speed. The amount of high-frequency data to be cached can be accurately determined based on specific business scenarios and system resource conditions. For example, if system resources are limited but the most frequently queried data needs to be quickly retrieved, a smaller preset number can be set to ensure that only the most critical high-frequency data is stored in the first cache, improving the pertinence and efficiency of the cache. Determining the target historical request count with a specific number makes cache management more clear and controllable.Developers can make adjustments based on actual conditions to adapt to different performance requirements and resource constraints. The standardized template generation and filling process can be efficiently executed by the system, reducing the time overhead of data acquisition. Compared with building complex query statements from scratch every time, this method can generate the second data acquisition request faster and improve the response speed of the system. Generating templates for specific database types and query language specifications can make full use of the query optimization mechanism of the database. The database can better understand and execute these standardized query statements, improve query efficiency, and reduce system resource consumption. When the second cache misses, query data from the database and return it, avoiding directly letting users wait for a long database query process, reducing user waiting time, and improving the response speed of the system. For example, in a real-time data analysis system, users can obtain the required data faster to make decisions. The data obtained from the database is stored in the second cache and a mapping relationship is established, so that the next same query request can directly obtain data from the cache, avoiding repeated database query operations, and greatly improving data query efficiency.
[0172] The specific implementation of the data acquisition device is substantially the same as the specific implementation of the above-mentioned data acquisition method, and will not be described in detail herein.
[0173] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data acquisition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0174] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0175] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0176] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data acquisition method of the embodiment of this application;
[0177] Input / output interface 903, used to implement information input and output;
[0178] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0179] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0180] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0181] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data acquisition method when executed by a processor.
[0182] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0183] The data acquisition method, data acquisition device, electronic device and storage medium provided by the embodiment of the present application receive a first data acquisition request, the request type of the first data acquisition request is a data query type; when there is no data acquisition request matching the first data acquisition request in the first cache, a second data acquisition request of a query statement type is constructed according to the first data acquisition request, and the first cache is used to store the data acquisition request of the data query type and the data corresponding to each data query type data acquisition request; when there is a target data acquisition request matching the second data acquisition request in the second cache, the target data with a preset mapping relationship established with the target data acquisition request is acquired, and the target data is returned, and the second cache is used to store the data acquisition request of the query statement type and the data corresponding to each data acquisition request of the query statement type. In this way, by setting the first cache and the second cache, a multi-layer cache architecture is formed. When the data is not obtained in the first cache, it will try to find it in the second cache, which increases the chance of the data acquisition request being cached. This multi-level cache mechanism can effectively reduce the number of times data is directly obtained from the database, thereby improving data acquisition efficiency.
[0184] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0185] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0186] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0187] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0188] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0189] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0191] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0194] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A data acquisition method, characterized in that: include: receiving a first data acquisition request, wherein the request type of the first data acquisition request is a data query type; When there is no data acquisition request matching the first data acquisition request in the first cache, constructing a second data acquisition request of a query statement type according to the first data acquisition request, wherein the first cache is used to store data acquisition requests of the data query type and data corresponding to each data acquisition request of the data query type; When there is a target data acquisition request matching the second data acquisition request in the second cache, target data having a preset mapping relationship with the target data acquisition request is acquired and the target data is returned. The second cache is used to store data acquisition requests of query statement types and data corresponding to each data acquisition request of query statement type.
2. The data acquisition method according to claim 1, characterized in that: The method further comprises: When there is a data acquisition request matching the first data acquisition request in the first cache, first data having a preset mapping relationship with the first data acquisition request is acquired, and the first data is returned.
3. The data acquisition method according to claim 2, characterized in that: After returning the first data, the method further includes: Obtaining a target historical request number of the first data acquisition request; Increment the target historical request count by one; Determine the historical request count of data acquisition requests of each data query type; The data acquisition requests of the data query type and the corresponding data stored in the first cache are updated according to each of the historical request times.
4. The data acquisition method according to claim 3, characterized in that: The updating of the data acquisition request of the data query type and the corresponding data stored in the first cache according to each of the historical request times includes: Sort each of the historical request times in descending order to obtain a sorted sequence; Filtering out a plurality of target historical request times from the sorting sequence according to a preset filtering rule; The data acquisition request of the data query type corresponding to each of the target historical request times and the corresponding data are stored in the first cache.
5. The data acquisition method according to claim 4, characterized in that: The step of filtering out a plurality of target historical request times from the sorting sequence according to a preset filtering rule includes: Determine a preset number of historical request times before the sorting sequence as a target historical request time; or A preset percentage of historical request times in the sorting sequence is determined as a target historical request time.
6. The data acquisition method according to claim 1, characterized in that: The step of constructing a second data acquisition request of a query statement type according to the first data acquisition request includes: Acquire the data acquisition condition and data acquisition field included in the first data acquisition request; Generate a data acquisition request template whose request type is a query statement type; The data acquisition condition and the data acquisition field are respectively written into corresponding positions of the data acquisition request template to obtain a second data acquisition request of a query statement type.
7. The data acquisition method according to claim 1, characterized in that: The method further comprises: When there is no acquisition request matching the second data acquisition request in the second cache, searching the database for second data having a preset mapping relationship with the second data acquisition request, and returning the second data; The second data acquisition request and the second data are stored in the second cache, and a mapping relationship between the second data acquisition request and the second data is established.
8. A data acquisition device, characterized in that: include: An acquiring unit, configured to receive a first data acquisition request, wherein the request type of the first data acquisition request is a data query type; a construction unit, configured to construct, according to the first data acquisition request, a second data acquisition request of a query statement type when no data acquisition request matching the first data acquisition request exists in the first cache, wherein the first cache is configured to store data acquisition requests of the data query type and data corresponding to each data acquisition request of the data query type; A return unit is used to obtain target data having a preset mapping relationship with the target data acquisition request and return the target data when there is a target data acquisition request matching the second data acquisition request in the second cache, and the second cache is used to store data acquisition requests of query statement type and data corresponding to each data acquisition request of query statement type.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data acquisition method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data acquisition method according to any one of claims 1 to 7 is implemented.