Data query method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a data query method and device, equipment, a storage medium and a product. The method comprises the following steps: under the condition that a data volume difference value corresponding to a query data volume in a current time window is in a first preset interval, according to the query data volume in the current time window, determining continuous change times of window duration; and determining the window duration of the next time window according to the continuous change times, thereby carrying out data query until the query covers full data. The window duration of the next window is accurately decided on the basis of the corresponding data volume difference in combination with the continuous change trend of the window duration, so that the query window can adapt to the change of the data density, the whole query process is ensured to be operated in a high-performance interval, system bumping caused by too large or too small single query data volume is effectively prevented, and the query efficiency is improved. Therefore, the Elasticsearch search engine can efficiently and stably complete query, cleaning and screening tasks of a large amount of data.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data query method, apparatus, device, storage medium and product. Background Technology

[0002] Elasticsearch is a distributed, open-source search engine. As such, its storage structure is based on an inverted index. This structure maps words in documents to document identifiers, combined with distributed storage (Lucene index) of shards, replication, and field segmentation analysis, achieving efficient word segmentation and retrieval capabilities. However, existing technologies have significant drawbacks when performing large-scale full data queries on Elasticsearch: First, they rely on adjusting the maximum query volume of Elasticsearch and using paginated batch filtering. Limited by Elasticsearch's mechanism that the current page can only be queried after the previous page has been queried, in multi-partition scenarios, each partition node needs to repeat this operation, leading to high CPU core utilization and low performance. Second, using a fixed query range, it cannot adapt to fluctuations in data distribution. When the number of documents surges within a certain time window, it easily triggers deep pagination problems, which also severely consume system resources and may even crash the entire Elasticsearch service. Summary of the Invention

[0003] This invention provides a data query method, apparatus, device, storage medium, and product that can solve the technical problem that existing technologies cannot perform efficient and stable full data queries on Elasticsearch.

[0004] In a first aspect, embodiments of the present invention provide a data query method, the method comprising: If the difference in the amount of data queried in the current time window is within a first preset range, the number of consecutive changes in the window duration is determined based on the amount of data queried in the current time window; wherein, the number of consecutive changes in the window duration is used to characterize the number of times the window duration has been continuously adjusted in the same direction up to the current time window during the full data query process; The window duration of the next time window is determined based on the number of consecutive changes in the window duration. Data queries are performed according to the window duration of the next time window until the query covers all data.

[0005] In one feasible implementation, the step of determining the number of consecutive changes in window duration based on the amount of query data within the current time window includes: If the amount of query data in the current time window is greater than a preset data volume threshold, the first number of changes recorded in the first counter is determined, wherein the first number of changes is used to characterize the number of times the window duration has been continuously adjusted in the increasing direction up to the current time window during the full data query process; If the amount of data queried in the current time window is less than or equal to a preset data volume threshold, determine the second number of changes recorded in the second counter. The second number of changes is used to characterize the number of times the window duration has been continuously adjusted in the decreasing direction up to the current time window during the full data query process.

[0006] In one feasible implementation, before the step of determining the first number of changes recorded in the first counter when the amount of query data in the current time window is greater than a preset amount of data, the method further includes: If the window duration of the next time window in an adjacent time window has a doubling relationship with the window duration of the previous time window in an adjacent time window, the number of first changes recorded in the first counter is increased. If the window duration of the next time window in an adjacent time window does not satisfy the doubling relationship with the window duration of the previous time window in an adjacent time window, the first change count recorded in the first counter is reset to zero.

[0007] In one feasible implementation, the step of determining the window duration of the next time window based on the number of consecutive changes in the window duration includes: If the number of consecutive changes in the window duration is equal to the number of changes recorded in the first counter, the number of changes threshold is compared with the number of changes. If the first number of changes is less than or equal to the number of changes threshold, the window duration of the next time window is determined based on the window duration of the current time window and the preset reduction ratio. If the first number of changes is greater than the number of changes threshold, the window duration of the next time window is determined according to the duration relationship between the window duration of the current time window and the window duration of the historical time window, wherein the window number of the historical time window is obtained by subtracting the number of changes threshold from the window number of the current time window.

[0008] In one feasible implementation, the step of determining the window duration of the next time window based on the duration relationship between the current time window and the window duration of historical time windows includes: If the window duration of the current time window and the window duration of the historical time window satisfy the window duration doubling relationship, the window duration of the next time window is determined based on the window duration of the current time window and a preset reduction ratio. If the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration doubling relationship, a fixed duration value is determined based on the window duration of the current time window and a preset adjustment ratio, and the window duration of the next time window is determined based on the duration difference between the window duration of the current time window and the fixed duration value.

[0009] In one feasible implementation, the step of determining the window duration of the next time window based on the number of consecutive changes in the window duration includes: If the number of consecutive changes in window duration during the full data query process is the second number of changes recorded in the second counter, the number of changes threshold and the second number of changes are compared. If the second number of changes is less than or equal to the number of changes threshold, the window duration of the next time window is determined based on the window duration of the current time window and a preset increase ratio. If the second number of changes is greater than the number of changes threshold, the window duration of the next time window is determined according to the duration relationship between the window duration of the current time window and the window duration of the historical time window, wherein the historical window number of the historical time window is obtained by subtracting the number of changes threshold from the second number of changes.

[0010] In one feasible implementation, the step of determining the window duration of the next time window based on the duration relationship between the current time window and the window duration of historical time windows includes: If the window duration of the current time window and the window duration of the historical time window satisfy the window duration doubling relationship, the window duration of the next time window is determined according to the window duration of the current time window and a preset increase ratio. If the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration multiplication relationship, a fixed duration value is determined based on the window duration of the current time window and a preset adjustment ratio, and the window duration of the next time window is determined based on the sum of the window durations of the current time window and the fixed duration value.

[0011] In one feasible implementation, before the step of determining the number of consecutive changes in window duration based on the query data volume in the current time window when the difference in data volume corresponding to the query data volume in the current time window is within a first preset range, the method further includes: Calculate the initial difference between the amount of query data in the current time window and the preset data volume threshold; The absolute value of the initial difference is used as the data volume difference corresponding to the query data volume under the current time window; Alternatively, the ratio between the absolute value of the initial difference and the preset data volume threshold can be used as the data volume difference corresponding to the query data volume in the current time window.

[0012] In one feasible implementation, the method further includes: If the difference between the data volume and the query data volume in the current time window is within the second preset range and the query data volume in the current time window is greater than the preset data volume threshold, a fixed duration value is determined based on the window duration of the current time window and the preset adjustment ratio, and the window duration of the next time window is determined based on the duration difference between the window duration of the current time window and the fixed duration value. If the difference between the data volume and the query data volume in the current time window is within the second preset range and the query data volume in the current time window is less than or equal to the preset data volume threshold, the window duration of the next time window is determined based on the sum of the window duration of the current time window and the fixed duration value. If the difference in data volume corresponding to the query data volume within the current time window falls within a third preset interval, the window duration of the current time window is used as the window duration of the next time window. The upper limit of the third preset interval is less than or equal to the lower limit of the second preset interval. Secondly, embodiments of the present invention provide a data query device, the device comprising: The processing module is used to determine the number of consecutive changes in window duration based on the amount of data queried in the current time window when the difference in the amount of data queried in the current time window is within a first preset range; wherein, the number of consecutive changes in window duration is used to characterize the number of times the window duration has been continuously adjusted in the same direction up to the current time window during the full data query process; The processing module is also used to determine the window duration of the next time window based on the number of consecutive changes in the window duration; The query module is used to query data according to the window duration of the next time window until the query covers all data.

[0013] Thirdly, embodiments of the present invention provide a data query device, the device comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the data query method as described in the first aspect.

[0014] Fourthly, embodiments of the present invention provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the data query method as described in the first aspect is implemented.

[0015] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the data query method as described in the first aspect.

[0016] The data query method, apparatus, device, storage medium, and product of this invention, based on the difference in data volume corresponding to the actual query data volume of the current actual window and combined with the continuous change trend of the window duration, accurately determines the window duration of the next window, enabling the query window to adapt to changes in data density. This ensures that the entire query process operates within a high-performance range, effectively preventing system turbulence caused by excessively large or small single query data volume, and eliminating the huge CPU overhead caused by deep pagination calculations. This allows the Elasticsearch search engine to efficiently and stably complete large-scale data query, cleaning, and filtering tasks, expanding its application boundaries. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of an embodiment of a data query method provided by the present invention; Figure 2 This is a schematic flowchart of a second embodiment of a data query method provided by the present invention; Figure 3 This is a flowchart illustrating a data query method according to Embodiment 3 of the present invention. Figure 4 This is a flowchart illustrating Embodiment 4 of a data query method provided by the present invention; Figure 5 This is a schematic diagram illustrating the data volume change of a data query method provided in Embodiment 4 of the present invention; Figure 6 This is a schematic diagram of the structure of a data query device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a data query device provided in an embodiment of the present invention. Detailed Implementation

[0019] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0021] In Elasticsearch, based on the principle of inverted indexes, data is stored as multiple small parts called "shards," each of which is a Lucene index. Shards can be replicated and distributed across multiple nodes in the cluster, achieving high availability and horizontal scalability. Each shard contains an inverted index of the document and all its associated fields. The field data may also undergo some tokenization and analysis processing, such as stop word removal and case conversion. Therefore, during a search, Elasticsearch quickly finds documents containing the query terms using the inverted index and returns the relevant results. Furthermore, Elasticsearch uses a built-in caching mechanism to improve search performance and supports dynamically adding, updating, and deleting documents to keep the index up-to-date.

[0022] Currently, large-scale data cleaning of Elasticsearch is required in various scenarios, including IoT alarm message filtering, user log filtering, big data middleware development, and others. Due to its storage structure limitations, Elasticsearch is only suitable for word segmentation retrieval, not for full data cleaning, filtering, or numerical extraction. Existing technologies have significant drawbacks when performing large-scale full data queries on Elasticsearch: 1. They rely on adjusting the maximum query volume of Elasticsearch and using paginated batch filtering. This is limited by Elasticsearch's mechanism that the current page can only be queried after the previous page has been queried. In multi-partition scenarios, each partition node needs to repeat this operation, leading to high CPU core utilization and low performance. 2. Using a fixed query range fails to adapt to fluctuations in data distribution. When the number of documents surges within a certain time window, it easily triggers deep pagination problems, which also severely consume system resources and may even crash the entire Elasticsearch service.

[0023] To address the problems of the prior art, embodiments of this application provide a data query method, apparatus, device, storage medium, and product. The data query method provided in this application embodiment will be described first below.

[0024] Figure 1 A flowchart illustrating a data query method provided in one embodiment of this application is shown. Figure 1 As shown, the method may include steps 210-230: Step 210: If the difference in the amount of data corresponding to the amount of data queried in the current time window is within the first preset range, determine the number of consecutive changes in the window duration based on the amount of data queried in the current time window; wherein, the number of consecutive changes in the window duration is used to characterize the number of times the window duration has been continuously adjusted in the same direction up to the current time window during the full data query process.

[0025] The execution entity in this embodiment can be a data query device, which is a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone. The window sniffing algorithm in the data query device dynamically adjusts the window to divide the dynamic highest availability time interval of Elasticsearch, finds its optimal search state, and splits the full query into N time segments. By completing the query in each time interval, it ultimately achieves full data query and large-scale data cleaning in Elasticsearch.

[0026] The current time window refers to the time segment in the process of querying data in full data query that has just been completed or is in progress. It is the basic unit of data query. Its time range is divided sequentially based on the total time range of full data, and it is continuous and does not overlap with the adjacent previous time window and the adjacent next time window.

[0027] When the current time window is the first time window since the start of a full data query, the window duration of the current time window can be a pre-set T1; or it can be determined as follows: obtain the total number N of all documents to be queried in Elasticsearch and the total duration T covered by the query; calculate the theoretical total number of time windows M = N / N0 based on the preset ideal number of documents per window N0. Finally, based on the theoretical total number of time windows and the total duration covered by the query, the window duration of the first time window T1 = T / M.

[0028] The query data volume N1 within the current time window refers to the total amount of data retrieved from all shards of the Elasticsearch cluster after the query operation within the current time window has been completed. The data volume difference refers to the difference between the query data volume within the current time window and the preset data volume threshold N0. The data volume difference can be the absolute value of the difference between the query data volume N1 and the preset data volume threshold N0, or it can be the absolute difference between the two. This embodiment does not limit the specific definition of the data volume difference. The preset data volume threshold refers to the preset ideal document volume for a single window.

[0029] The number of consecutive changes in window duration is a count variable used to record the number of times the window duration of the time window has been adjusted in the same direction (e.g., continuously increasing or continuously decreasing) up to the current time window during the full data query process.

[0030] The first preset interval refers to a specific range of data difference. Preferably, the first preset interval is defined as greater than a preset first threshold. If the difference between the query data volume and the preset data volume threshold is within the first preset interval under the current time window, it indicates a significant deviation between the query data volume under the current time window and the preset data volume threshold, and the time window needs to be adjusted.

[0031] If the difference in the amount of data queried in the current time window is within the first preset range, the deviation direction between the amount of data queried in the current time window and the preset data threshold is determined based on the relationship between the amount of data queried in the current time window and the preset data threshold; based on the deviation direction between the two, the number of consecutive changes in the window duration in the corresponding direction is obtained.

[0032] In one feasible implementation, step 210 may include steps A11-A12: Step A11: If the amount of query data in the current time window is greater than a preset data volume threshold, determine the first number of changes recorded in the first counter. The first number of changes is used to characterize the number of times the window duration has been continuously adjusted in the increasing direction up to the current time window during the full data query process.

[0033] Step A12: If the amount of query data in the current time window is less than or equal to the preset data amount threshold, determine the second number of changes recorded in the second counter. The second number of changes is used to characterize the number of times the window duration has been continuously adjusted in the decreasing direction up to the current time window during the full data query process.

[0034] A counter is a register or variable that stores integer values ​​in memory. Specifically, the first counter counts the number of times the window duration has been continuously adjusted in the increasing direction up to the current time window during the full data query process, thus obtaining the first change count; the second counter counts the number of times the window duration has been continuously adjusted in the decreasing direction up to the current time window during the full data query process, thus obtaining the second change count.

[0035] If the difference in data volume corresponding to the query data volume within the current time window falls within a first preset interval, the relationship between the query data volume T1 within the current time window and a preset data volume threshold is determined. If the query data volume within the current time window is greater than the preset data volume threshold, it indicates that the window duration of the current time window may be too long, resulting in an excessive load on a single query. Therefore, the first change count recorded in the first counter needs to be invoked. If the query data volume within the current time window is less than or equal to the preset data volume threshold, it indicates that the window duration of the current time window may be too short, resulting in low efficiency for a single query. Therefore, the second change count recorded in the second counter needs to be invoked. In this embodiment, based on the relationship between the query data volume and the preset data volume, the change count corresponding to the current deviation direction is obtained, laying the foundation for the adaptive adjustment of the subsequent window duration.

[0036] In one possible implementation, steps B11-B12 may be included before step A11: Step B11: If the window duration of the next time window in an adjacent time window satisfies a doubling relationship with the window duration of the previous time window in an adjacent time window, increase the number of first changes recorded in the first counter.

[0037] Step B12: If the window duration of the next time window in an adjacent time window does not satisfy the doubling relationship with the window duration of the previous time window in an adjacent time window, clear the first change count recorded in the first counter.

[0038] In some embodiments, the first counter may be dynamically updated based on the following rules: During the full query process, after determining the window duration of the next time window, the relationship between the window durations of the adjacent previous and next time windows is immediately judged. The ratio of the window duration of the next time window to the window duration of the previous time window is calculated. If the window duration of the next time window is greater than the window duration of the previous time window, and the window duration of the next time window is a preset multiple (e.g., 2 times) of the window duration of the previous time window, then it is determined that the two satisfy a multiplication relationship, indicating that the adjustment of the window duration this time belongs to a continuous multiplication in the direction of increase, and the first change count recorded in the first counter is incremented by 1.

[0039] If the window duration of the next time window is less than or equal to the window duration of the previous time window; or if the window duration of the next time window is greater than the window duration of the previous time window, but the window duration of the next time window is not a preset multiple (e.g., 2 times) of the window duration of the previous time window, then it is determined that the two doubling relationships do not exist, indicating that the adjustment direction of the window duration this time does not follow the trend of continuous doubling. At the end of this trend, the first number of changes recorded in the first counter is cleared to zero.

[0040] In this embodiment, by determining whether the lengths of adjacent windows satisfy the doubling relationship, the first change number of the first counter is dynamically updated, accurately recording the trend of continuous doubling adjustment of window duration in the increasing direction, providing an accurate counting basis for subsequent adjustments to prevent unlimited doubling of windows, and ensuring the dynamic balance of window length during the full data query process.

[0041] In some embodiments, the second counter may be dynamically updated based on the following rules: During the full query process, after determining the window duration of the next time window, the relationship between the window durations of the adjacent previous and next time windows is immediately judged. The ratio of the window duration of the next time window to the window duration of the previous time window is calculated. If the window duration of the next time window is less than the window duration of the previous time window, and the window duration of the next time window is a preset multiple (e.g., 0.5 times) of the window duration of the previous time window, it is determined that the two satisfy a multiplication-reduction relationship, indicating that the adjustment of the window duration this time belongs to a continuous multiplication-reduction in the direction of decrease, and the second change count recorded in the second counter is incremented by 1.

[0042] If the window duration of the next time window is greater than or equal to the window duration of the previous time window; or if the window duration of the next time window is less than the window duration of the previous time window, but the window duration of the next time window is not a preset multiple (e.g., 0.5 times) of the window duration of the previous time window, then it is determined that the two doubling relationship is not satisfied, indicating that the adjustment direction of the window duration this time is not the end of the continuous doubling trend, and the second change count recorded in the second counter is cleared to zero.

[0043] In this embodiment, by determining whether the lengths of adjacent windows satisfy the doubling relationship, the second number of changes of the second counter is dynamically updated, accurately recording the trend of continuous doubling adjustment of window duration in the decreasing direction, providing an accurate counting basis for subsequent adjustments to prevent unlimited doubling of windows, and ensuring the dynamic balance of window length during the full data query process.

[0044] In one possible implementation, steps C11-C13 may be included before step 210: Step C11: Calculate the initial difference between the amount of query data in the current time window and the preset data volume threshold.

[0045] Step C12: The absolute value of the initial difference is used as the data volume difference corresponding to the query data volume under the current time window.

[0046] Step C13: The ratio between the absolute value of the initial difference and the preset data volume threshold is taken as the data volume difference corresponding to the query data volume under the current time window.

[0047] Calculate the difference between the current data volume N1 and the preset data volume threshold N0 within the current time window to obtain the initial difference = N1 - N0; then calculate the absolute value of the initial difference. This serves as the difference in data volume corresponding to the query data volume within the current time window. Alternatively, the absolute value of the initial difference can be used. The ratio between the preset data volume threshold N0 and the This represents the difference in data volume corresponding to the data volume queried within the current time window.

[0048] In subsequent comparisons, the absolute value of the initial difference can be determined. Is it within the first preset range (i.e.) It can also determine the absolute value of the initial difference; The ratio between the preset data volume threshold N0 and the Is it within the first preset range (i.e.) ).

[0049] In this embodiment, by calculating the absolute difference or relative ratio between the current query volume and the preset threshold, a standardized quantitative basis is provided for accurately judging the degree of data deviation and triggering the corresponding window adjustment strategy.

[0050] Step 220: Determine the window duration of the next time window based on the number of consecutive changes in the window duration.

[0051] It should be noted that the next time window refers to the next time segment that will be processed after the current time window has been completed, and its time range is based on the end time of the current time window.

[0052] Optionally, the number of consecutive changes in window duration and a preset continuous adjustment threshold are used. The relationship between the values ​​can determine the window duration of the next time window, as follows: If the number of consecutive changes is greater than the preset continuous adjustment threshold, it indicates that the window duration has been adjusted too much in the same direction (such as the number of consecutive doubling / decreasing exceeds the upper limit). At this time, a fast pullback mechanism is triggered to determine the window duration of the next time window to be adjusted in the opposite direction (for example, if it has been continuously increasing before, it will be decreasing now; if it has been continuously decreasing before, it will be increasing now).

[0053] If the number of consecutive changes is less than or equal to the preset consecutive adjustment threshold, it indicates that the number of consecutive adjustments to the window duration has not reached the upper limit, and the adjustment continues in the normal direction (for example, if it was previously continuously increased due to insufficient data, it will continue to increase now; if it was previously continuously decreased due to excessive data, it will continue to decrease now).

[0054] Optionally, in addition to the above methods, the window duration of the next time window can be determined based on the relationship between the number of consecutive changes in window duration and the dynamic threshold corresponding to the current time window (dynamically set based on the current window number), as follows: If the number of consecutive changes is greater than the dynamic threshold corresponding to the current window number, it indicates that the number of consecutive adjustments to the window duration in the same direction has exceeded the reasonable range of the current stage, triggering a fast pull-back mechanism, and deciding to adjust the window duration of the next time window in the opposite direction; if the number of consecutive changes is less than or equal to the dynamic threshold corresponding to the current window number, it indicates that the number of consecutive adjustments to the window duration is within a reasonable range, and the adjustment continues in the normal direction.

[0055] In this embodiment, the adjustment range can be calculated based on the current window duration according to a preset ratio, or increased or decreased according to a preset fixed value. This embodiment does not limit the specific adjustment method and adjustment range.

[0056] Step 230: Perform data queries according to the window duration of the next time window until the query covers all data.

[0057] Based on the duration of the next time window, a new time window is divided, with the time range starting from the end time of the current time window and the length being the duration of the next time window. Within the next time window, data from all shards of the Elasticsearch cluster within that time range is queried, obtaining the new amount of data to be queried. Steps 210-230 are repeated until the time ranges of all time windows completely cover the total time range of the full data, thus completing the full data query.

[0058] This embodiment accurately determines the window duration of the next window based on the difference between the actual query data volume in the current window and the continuous trend of window duration changes. This allows the query window to adapt to changes in data density, ensuring that the entire query process runs within a high-performance range. It effectively prevents system turbulence caused by excessively large or small query data volumes and eliminates the huge CPU overhead caused by deep pagination calculations. This enables the Elasticsearch search engine to efficiently and stably complete large-scale query, cleaning, and filtering tasks, expanding its application boundaries.

[0059] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The data query method, step 220, may include steps 310-330: Step 310: If the number of consecutive changes in the window duration is the first number of changes recorded in the first counter, compare the number of changes threshold with the first number of changes.

[0060] Step 320: If the first number of changes is less than or equal to the number of changes threshold, determine the window duration of the next time window based on the window duration of the current time window and the preset reduction ratio.

[0061] Step 330: If the first number of changes is greater than the number of changes threshold, determine the window duration of the next time window based on the duration relationship between the window duration of the current time window and the window duration of the historical time window, wherein the window number of the historical time window is obtained by subtracting the number of changes threshold from the window number of the current time window.

[0062] The preset reduction ratio is a pre-defined percentage (e.g., 50%) used to shorten the window duration. In this embodiment, the preset reduction ratio should be less than 100%. The historical time window refers to the time window in the full query process that has been queried before the current time window. Its corresponding window number is obtained by subtracting the change number threshold from the window number of the current time window.

[0063] The amount of query data N1 in the current time window is greater than the preset data volume threshold N0, and the absolute value of the initial difference is... Within the first preset range (i.e. In the case of: or, if the amount of query data N1 in the current time window is greater than the preset data volume threshold N0, and the absolute value of the initial difference is... The ratio between the preset data volume threshold N0 and the Within the first preset range (i.e. In the case of ), obtain the first number of changes recorded in the first counter, and compare the first number of changes C1 with a preset threshold number of changes. (For example 2) Compare them and determine the size relationship between them.

[0064] If the first number of changes C1 is less than or equal to the number of changes threshold This indicates that although the continuous trend of change is not yet too strong, the amount of query data has exceeded the capacity. In this case, the window duration of the next time window needs to be obtained by multiplying the current time window duration by the preset reduction ratio. For example, if the preset reduction ratio is 50%, then the window duration of the next time window is half the window duration of the current time window.

[0065] If the first number of changes C1 is greater than the number of changes threshold This indicates that the window duration has been excessively multiplied, and it is necessary to further determine the window duration of the next time window by considering the duration relationship between the current time window and the historical time window. For example, if the current time window duration is much longer than the historical time window duration, it is necessary to quickly pull back and significantly reduce the window duration of the next time window; if the current time window duration is less than or equal to the historical time window duration, or if the current time window duration is longer than the historical time window duration but not a multiple of the historical time window duration (e.g., twice), then it is only necessary to slightly reduce the window duration of the next time window or keep it unchanged. This embodiment does not limit the adjustment range or method.

[0066] In one feasible implementation, step 330 may include steps D11-D12: Step D11: If the window duration of the current time window and the window duration of the historical time window satisfy the window duration doubling relationship, determine the window duration of the next time window based on the window duration of the current time window and the preset reduction ratio.

[0067] Step D12: If the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration doubling relationship, determine a fixed duration value based on the window duration of the current time window and a preset adjustment ratio, and determine the window duration of the next time window based on the duration difference between the window duration of the current time window and the fixed duration value.

[0068] In some embodiments, the amount of query data N1 in the current time window is greater than the preset data volume threshold N0, and the absolute value of the initial difference is... Within the first preset range (i.e. In the case of: or, if the amount of query data N1 in the current time window is greater than the preset data volume threshold N0, and the absolute value of the initial difference is... The ratio between the preset data volume threshold N0 and the Within the first preset range (i.e. In the case where the first number of changes C1 is greater than the number of changes threshold, Then, it is further determined whether the window duration of the current time window and the window duration of the historical time windows satisfy the window duration doubling relationship. In this embodiment, the window duration doubling relationship is the current time window... The window duration is the historical time window. Window duration The window duration multiplication relationship can also be set to other conditions, and this embodiment does not impose any restrictions on this.

[0069] If the window duration of the current time window and the window duration of the historical time windows satisfy a doubling relationship, it indicates that the current window duration is due to continuous... If excessive growth occurs due to a doubling adjustment, a significant contraction is needed to bring it back to a reasonable range. In this case, the window duration of the next time window should be calculated by multiplying the current window duration by a preset reduction ratio. For example, if the preset reduction ratio is 50%, the window duration of the next time window will be half the window duration of the current time window.

[0070] If the window duration of the current time window and the window duration of the historical time windows do not satisfy the doubling relationship of window duration, it indicates that the increase in the current window duration is not strictly due to continuous... This doubling of time necessitates a small, fixed adjustment to avoid excessive contraction. In this case, the current time window duration is multiplied by a preset adjustment ratio to obtain the fixed duration. Subtracting this fixed duration from the current time window duration gives the next time window duration. For example, if the preset adjustment ratio is 20%, then the fixed duration equals the current time window duration. 20%; the window duration of the next time window = the window duration of the current time window - a fixed duration value. Besides the above method, the fixed duration value can also be calculated using other methods, and this embodiment does not impose any restrictions on this.

[0071] In this implementation, by determining whether the duration of the current time window and the duration of the historical time window satisfy a doubling relationship, two scenarios, excessive doubling and non-excessive doubling, are distinguished. Preset reduction ratio and fixed duration difference adjustment strategies are adopted respectively to achieve fine-grained control of the window duration. This ensures both rapid correction in the case of excessive adjustment and stable adjustment in the case of non-excessive adjustment, ultimately balancing the efficiency of full data query and the load pressure of Elasticsearch.

[0072] This embodiment compares a threshold number of changes with the first number of changes recorded in a first counter when the number of consecutive changes in the window duration is equal to the first number of changes recorded in the first counter. If the first number of changes is less than or equal to the threshold number, the window duration of the next time window is determined based on the current time window duration and a preset reduction ratio. If the first number of changes is greater than the threshold number, the window duration of the next time window is determined based on the duration relationship between the current time window duration and the historical time window durations, where the window index of the historical time window is obtained by subtracting the threshold number of changes from the current time window index. Through dynamic threshold judgment and case-based adjustment logic, fine-grained control over the continuous doubling adjustment of the window duration is achieved, preventing unlimited increases in window duration and ensuring dynamic balance of window duration and Elasticsearch query efficiency during full data querying.

[0073] Based on any one or more embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step 220 in the data query method may include steps 410-430: Step 410: If the number of consecutive changes in window duration during the full data query process is the second number of changes recorded in the second counter, compare the number of changes threshold with the second number of changes.

[0074] Step 420: If the second number of changes is less than or equal to the number of changes threshold, determine the window duration of the next time window based on the window duration of the current time window and the preset increase ratio.

[0075] Step 430: If the second number of changes is greater than the number of changes threshold, determine the window duration of the next time window based on the duration relationship between the window duration of the current time window and the window duration of the historical time window, wherein the historical window number of the historical time window is obtained by subtracting the number of changes threshold from the second number of changes.

[0076] The preset increase ratio is a pre-set value for increasing the window duration (e.g., 200%), and in this embodiment, the preset increase ratio should be greater than 100%.

[0077] The amount of query data N1 within the current time window is less than or equal to the preset data volume threshold N0, and the absolute value of the initial difference is... Within the first preset range (i.e. In the case of: or, if the amount of query data N1 in the current time window is less than or equal to the preset data volume threshold N0, and the absolute value of the initial difference is... The ratio between the preset data volume threshold N0 and the Within the first preset range (i.e. In the case of ), obtain the second number of changes recorded in the second counter, and compare the second number of changes C2 with the preset number of changes threshold. (For example 2) Compare them and determine the size relationship between them.

[0078] If the second number of changes C2 is less than or equal to the number of changes threshold This indicates that while the continuous trend is not yet too strong, the efficiency of querying data is relatively low. In this case, the window duration of the next time window needs to be multiplied by the preset increase ratio based on the current time window duration. For example, if the preset increase ratio is 200%, then the window duration of the next time window will be twice the window duration of the current time window.

[0079] If the second number of changes C2 is greater than the number of changes threshold This indicates that the window duration has been excessively reduced continuously, and it is necessary to further determine the window duration of the next time window by considering the duration relationship between the current time window and the historical time window. For example, if the current time window duration is much shorter than the historical time window duration, it is necessary to quickly pull back and significantly increase the window duration of the next time window; if the current time window duration is greater than or equal to the historical time window duration, or if the current time window duration is less than the historical time window duration but not a multiple of the historical time window duration (e.g., 0.5 times), then it is only necessary to slightly increase or keep the window duration of the next time window unchanged. This embodiment does not limit the adjustment range or method.

[0080] In one feasible implementation, step 430 may further include steps E11-E13: Step E11: If the window duration of the current time window and the window duration of the historical time window satisfy the window duration multiplication relationship, determine the window duration of the next time window based on the window duration of the current time window and a preset increase ratio.

[0081] Step E12: If the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration multiplication relationship, a fixed duration value is determined based on the window duration of the current time window and a preset adjustment ratio. The window duration of the next time window is determined based on the sum of the window durations of the current time window and the fixed duration value.

[0082] In some embodiments, the amount of query data N1 in the current time window is less than or equal to a preset data volume threshold N0, and the absolute value of the initial difference is... Within the first preset range (i.e. In the case of: or, if the amount of query data N1 in the current time window is less than or equal to the preset data volume threshold N0, and the absolute value of the initial difference is... The ratio between the preset data volume threshold N0 and the Within the first preset range (i.e. In the case where the second number of changes C2 is greater than the number of changes threshold, Then, it is further determined whether the window duration of the current time window and the window duration of the historical time windows satisfy the window duration doubling relationship. In this embodiment, the window duration doubling relationship is the current time window... The window duration is the historical time window. Window duration The window duration multiplication factor can also be set to other conditions, and this embodiment does not impose any restrictions on this.

[0083] If the window duration of the current time window and the window duration of the historical time windows satisfy a doubling relationship, it indicates that the current window duration is due to continuous... If the reduction is excessive due to a doubling adjustment, a significant contraction is needed to bring it back to a reasonable range. In this case, the window duration of the next time window needs to be multiplied by a preset increase ratio based on the current window duration. For example, if the preset increase ratio is 200%, then the window duration of the next time window will be twice the window duration of the current time window.

[0084] If the window duration of the current time window and the window duration of the historical time windows do not satisfy the window duration doubling relationship, it indicates that the increase in the current window duration is not strictly due to continuous... This doubling of the time window necessitates a small, fixed adjustment to avoid excessive contraction. In this case, the current time window duration is multiplied by a preset adjustment ratio to obtain the fixed duration. Adding this fixed duration to the current time window duration gives the next time window duration. For example, if the preset adjustment ratio is 20%, then the fixed duration = the current time window duration. 20%; the window duration of the next time window = the window duration of the current time window + a fixed duration value. Besides the above method, the fixed duration value can also be calculated using other methods, and this embodiment does not impose any restrictions on this.

[0085] In this implementation, by determining whether the duration of the current time window satisfies the doubling relationship with the duration of the historical time window, two scenarios are distinguished: excessive doubling and non-excessive doubling. Adjustment strategies with preset increase ratios and fixed duration differences are adopted respectively to achieve fine-grained control over the window duration. This ensures both rapid correction in cases of excessive adjustment and stable adjustment in non-excessive scenarios, ultimately balancing the efficiency of full data query and the load pressure on Elasticsearch.

[0086] This embodiment compares the number of consecutive changes in window duration during the full data query process with the second number of changes recorded in the second counter. If the second number of changes is less than or equal to the threshold, the window duration of the next time window is determined based on the current time window duration and a preset increase ratio. If the second number of changes is greater than the threshold, the window duration of the next time window is determined based on the duration relationship between the current time window duration and the historical time window durations, where the historical window number is obtained by subtracting the threshold from the second number of changes. Through dynamic threshold judgment and case-based adjustment logic, fine-grained control over the continuous doubling and reduction of window duration is achieved, preventing unlimited increases in window duration and ensuring a dynamic balance between window duration and Elasticsearch query efficiency during the full data query process.

[0087] Based on any one or more embodiments of this application, in the fourth embodiment of this application, the content that is the same as or similar to the embodiments described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 The data query method may include steps 510-530: Step 510: If the difference between the data volume corresponding to the query data volume in the current time window is within the second preset range and the query data volume in the current time window is greater than the preset data volume threshold, determine the fixed duration value according to the window duration of the current time window and the preset adjustment ratio, and determine the window duration of the next time window according to the duration difference between the window duration of the current time window and the fixed duration value.

[0088] The second preset interval refers to a specific range of data difference values. Preferably, the second preset interval is defined as less than or equal to a preset first threshold. And greater than the preset second threshold The interval. Where a preset first threshold is defined. >Preset second threshold .

[0089] If the difference in data volume corresponding to the query data volume within the current time window is within the second preset interval and the query data volume within the current time window is greater than the preset data volume threshold, for example, the absolute value of the initial difference... Located in the second preset interval (i.e. And N1 > N0; or, the absolute value of the initial difference. The ratio between the preset data volume threshold N0 and the Located in the second preset interval (i.e. If N1 > N0, it indicates that the amount of query data in the current time window is too large, but the deviation is not serious. In this case, the window duration of the current time window can be slightly reduced to obtain the window duration of the next time window.

[0090] In some embodiments, when the difference in data volume corresponding to the query data volume in the current time window is within a second preset interval and the query data volume in the current time window is greater than a preset data volume threshold, a fixed duration value is obtained by multiplying the window duration of the current time window by a preset adjustment ratio. The fixed duration value is then subtracted from the window duration of the current time window to obtain the window duration of the next time window. For example, if the preset adjustment ratio is 20%, then the fixed duration value = window duration of the current time window × 20%; the window duration of the next time window = window duration of the current time window - fixed duration value. Besides the above method, other calculation methods can also be used for the fixed duration value, and this embodiment does not limit this.

[0091] Step 520: If the difference in the amount of data queried in the current time window is within the second preset interval and the amount of data queried in the current time window is less than or equal to the preset data volume threshold, determine the window duration of the next time window based on the sum of the window duration of the current time window and the fixed duration value.

[0092] If the difference in data volume corresponding to the query data volume within the current time window is within the second preset interval and the query data volume within the current time window is less than or equal to the preset data volume threshold, for example, the absolute value of the initial difference. Located in the second preset interval (i.e. And N1≤N0; or, the absolute value of the initial difference. The ratio between the preset data volume threshold N0 and the Located in the second preset interval (i.e. If N1≤N0, it means that the amount of query data in the current time window is relatively small, but the deviation is not serious. In this case, the window duration of the current time window can be increased slightly to obtain the window duration of the next time window.

[0093] In some embodiments, when the difference in data volume corresponding to the query data volume in the current time window is within a second preset interval and the query data volume in the current time window is less than or equal to a preset data volume threshold, a fixed duration value is obtained by multiplying the window duration of the current time window by a preset adjustment ratio. The fixed duration value is then added to the window duration of the current time window to obtain the window duration of the next time window. For example, if the preset adjustment ratio is 20%, then the fixed duration value = window duration of the current time window × 20%; the window duration of the next time window = window duration of the current time window + fixed duration value. Besides the above method, other calculation methods can also be used for the fixed duration value, and this embodiment does not limit this.

[0094] Step 530: If the difference in the amount of data corresponding to the amount of data queried in the current time window is within the third preset interval, the window duration of the current time window is used as the window duration of the next time window, wherein the upper limit of the third preset interval is less than or equal to the lower limit of the second preset interval.

[0095] The third preset interval refers to a specific range of data difference values. Preferably, the third preset interval is defined as less than or equal to a preset second threshold. The range.

[0096] If the difference in data volume corresponding to the query data volume within the current time window falls within the third preset interval, for example, the absolute value of the initial difference... In the third preset interval (i.e. In the case of ), or, the absolute value of the initial difference. The ratio between the preset data volume threshold N0 and the In the third preset interval (i.e. In this case, it indicates that the deviation between the amount of data queried in the current time window and the preset data threshold is relatively slight. In this case, the window duration of the current time window is directly used as the window duration of the next time window. Keeping the window duration stable can reduce the interference of frequent adjustments on query efficiency and ensure the smoothness of the full data query process.

[0097] The following is an example illustrating the overall data query process. The specific values ​​and adjustment methods used are merely illustrative for ease of understanding: Step 1: Start the process. Step 2: Compare the query data volume N1 within the current time window with the preset data volume threshold N0: a) If the query data volume N1 within the current time window is greater than the preset data volume threshold N0, and... (or In the case where the first change count C1 recorded in the first counter is less than or equal to the change count threshold, When the window duration of the next time window is half the window duration of the current time window; when the first change count C1 recorded in the first counter is greater than the change count threshold. And the current time window The window duration is the historical time window. Window duration The duration of the next time window is half the duration of the current time window; when the first change count C1 recorded in the first counter is greater than the change count threshold. And the current time window The window duration is not the same as the historical time window. Window duration The duration of the next time window = the duration of the current time window - a fixed duration value. The fixed duration value is calculated as follows: fixed duration = length of the current time window × 20%.

[0098] b) The amount of data N1 queried in the current time window is greater than the preset data volume threshold N0, and (or The window duration of the next time window = the window duration of the current time window - a fixed duration value.

[0099] c) The amount of data queried within the current time window is less than or equal to a preset data volume threshold, and (or In the case where the second change count C2 recorded in the second counter is less than or equal to the change count threshold, When the window duration of the next time window is twice the window duration of the current time window; when the second change count C2 recorded in the second counter is greater than the change count threshold. And the current time window The window duration is the historical time window. Window duration When the window duration of the next time window is twice the window duration of the current time window; when the second change count C2 recorded in the second counter is greater than the change count threshold. And the current time window The window duration is not the same as the historical time window. Window duration When the window duration of the next time window is equal to the window duration of the current time window plus a fixed duration value, the window duration of the next time window is equal to the window duration of the current time window.

[0100] d) The amount of data queried within the current time window is less than or equal to the preset data volume threshold, and (or If the window duration of the next time window is equal to the window duration of the current time window plus a fixed duration value, then the window duration of the next time window is equal to the window duration of the current time window.

[0101] e) in (or If the window duration of the next time window is equal to the window duration of the current time window, then the window duration of the next time window is equal to the window duration of the current time window.

[0102] Step 3: Use the window duration of the next time window to perform an Elasticsearch data query, obtain the amount of data to be queried, and repeat Step 2 until the Elasticsearch query for the entire dataset is complete. For ease of understanding, a threshold based on the number of changes is used here. For example, Figure 5 As shown, at the start of the program, the first time window (segment 1) has a duration of 15 minutes, and the amount of data retrieved is 2000. This satisfies condition c in step 2 above. Therefore, the second time window is increased by a factor of 30, resulting in a duration of 30 minutes. Using the second time window (segment 2), an Elasticsearch query reveals only 50,000 data points, indicating that the next time window can be increased further. Entering the third time window (segment 3), with a duration of 60 minutes, the amount of data retrieved reaches 200,000, exceeding twice the optimal value of 100,000. Therefore, the next query time window is decreased, with the calculation method showing a factor of 3. Entering the fourth time window (segment 4), with a duration of 30 minutes, the amount of data retrieved reaches 60,000, which is less than the optimal range of 100,000-120,000. Since the transition from the third to the fourth time window involved a factor of 3, the next time window increases by a fixed duration. By repeating the above steps, the duration of the time window is dynamically expanded or shrunk to ensure that the total number of Elasticsearch queries is within a reasonable range, ultimately achieving a full query of the data.

[0103] In this embodiment, when the difference in the amount of data queried in the current time window falls within a second preset interval and the amount of data queried in the current time window exceeds a preset data volume threshold, a fixed duration value is determined based on the window duration of the current time window and a preset adjustment ratio. The window duration of the next time window is then determined based on the difference between the window duration of the current time window and the fixed duration value. When the difference in the amount of data queried in the current time window falls within a second preset interval and the amount of data queried in the current time window is less than or equal to a preset data volume threshold, the window duration of the next time window is determined based on the sum of the window duration of the current time window and the fixed duration value. When the difference in the amount of data queried in the current time window falls within a third preset interval, the window duration of the current time window is used as the window duration of the next time window, wherein the upper limit of the third preset interval is less than or equal to the lower limit of the second preset interval. By using the above method, the appropriate duration adjustment strategy is selected according to the deviation of the difference between the data volume and the data volume queried under the current time window, thereby achieving fine-grained dynamic control of the window duration and ensuring the efficiency and stability of full data query.

[0104] like Figure 6 As shown, this application embodiment provides a data query device 200, which may include a processing module 201 and a query module 202; Processing module 201 is used to determine the number of consecutive changes in window duration based on the query data volume under the current time window when the difference in data volume corresponding to the query data volume under the current time window is within a first preset range; wherein, the number of consecutive changes in window duration is used to characterize the number of times the window duration has been continuously adjusted in the same direction up to the current time window during the full data query process.

[0105] The processing module 201 is further configured to determine the window duration of the next time window based on the number of consecutive changes in the window duration.

[0106] The query module 202 is used to query data according to the window duration of the next time window until the query covers all data.

[0107] Optionally, the processing module 201 is further configured to: If the amount of query data in the current time window is greater than a preset data volume threshold, the first number of changes recorded in the first counter is determined, wherein the first number of changes is used to characterize the number of times the window duration has been continuously adjusted in the increasing direction up to the current time window during the full data query process; If the amount of data queried in the current time window is less than or equal to a preset data volume threshold, determine the second number of changes recorded in the second counter. The second number of changes is used to characterize the number of times the window duration has been continuously adjusted in the decreasing direction up to the current time window during the full data query process.

[0108] Optionally, the processing module 201 is further configured to: If the window duration of the next time window in an adjacent time window has a doubling relationship with the window duration of the previous time window in an adjacent time window, the number of first changes recorded in the first counter is increased. If the window duration of the next time window in an adjacent time window does not satisfy the doubling relationship with the window duration of the previous time window in an adjacent time window, the first change count recorded in the first counter is reset to zero.

[0109] Optionally, the processing module 201 is further configured to: If the number of consecutive changes in the window duration is equal to the number of changes recorded in the first counter, the number of changes threshold is compared with the number of changes. If the first number of changes is less than or equal to the number of changes threshold, the window duration of the next time window is determined based on the window duration of the current time window and the preset reduction ratio. If the first number of changes is greater than the number of changes threshold, the window duration of the next time window is determined according to the duration relationship between the window duration of the current time window and the window duration of the historical time window, wherein the window number of the historical time window is obtained by subtracting the number of changes threshold from the window number of the current time window.

[0110] Optionally, the processing module 201 is further configured to: If the window duration of the current time window and the window duration of the historical time window satisfy the window duration doubling relationship, the window duration of the next time window is determined based on the window duration of the current time window and a preset reduction ratio. If the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration doubling relationship, a fixed duration value is determined based on the window duration of the current time window and a preset adjustment ratio, and the window duration of the next time window is determined based on the duration difference between the window duration of the current time window and the fixed duration value.

[0111] Optionally, the processing module 201 is further configured to: If the number of consecutive changes in window duration during the full data query process is the second number of changes recorded in the second counter, the number of changes threshold and the second number of changes are compared. If the second number of changes is less than or equal to the number of changes threshold, the window duration of the next time window is determined based on the window duration of the current time window and a preset increase ratio. If the second number of changes is greater than the number of changes threshold, the window duration of the next time window is determined according to the duration relationship between the window duration of the current time window and the window duration of the historical time window, wherein the historical window number of the historical time window is obtained by subtracting the number of changes threshold from the second number of changes.

[0112] Optionally, the processing module 201 is further configured to: If the window duration of the current time window and the window duration of the historical time window satisfy the window duration doubling relationship, the window duration of the next time window is determined according to the window duration of the current time window and a preset increase ratio. If the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration multiplication relationship, a fixed duration value is determined based on the window duration of the current time window and a preset adjustment ratio, and the window duration of the next time window is determined based on the sum of the window durations of the current time window and the fixed duration value.

[0113] Optionally, the processing module 201 is further configured to: Calculate the initial difference between the amount of query data in the current time window and the preset data volume threshold; The absolute value of the initial difference is used as the data volume difference corresponding to the query data volume under the current time window; Alternatively, the ratio between the absolute value of the initial difference and the preset data volume threshold can be used as the data volume difference corresponding to the query data volume in the current time window.

[0114] Optionally, the processing module 201 is further configured to: If the difference between the data volume and the query data volume in the current time window is within the second preset range and the query data volume in the current time window is greater than the preset data volume threshold, a fixed duration value is determined based on the window duration of the current time window and the preset adjustment ratio, and the window duration of the next time window is determined based on the duration difference between the window duration of the current time window and the fixed duration value. If the difference between the data volume and the query data volume in the current time window is within the second preset range and the query data volume in the current time window is less than or equal to the preset data volume threshold, the window duration of the next time window is determined based on the sum of the window duration of the current time window and the fixed duration value. If the difference in the amount of data corresponding to the amount of data queried in the current time window is within the third preset interval, the window duration of the current time window is used as the window duration of the next time window, wherein the upper limit of the third preset interval is less than or equal to the lower limit of the second preset interval.

[0115] Figure 7 A schematic diagram of the hardware structure of the data query device provided in an embodiment of the present invention is shown.

[0116] The data query device may include a processor 301 and a memory 302 storing computer program instructions.

[0117] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0118] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 302 may include removable or non-removable (or fixed) media, or memory 302 may be non-volatile solid-state memory. Memory 302 may be internal or external to the integrated gateway disaster recovery device.

[0119] In one instance, memory 302 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0120] Memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0121] The processor 301 reads and executes computer program instructions stored in the memory 302 to achieve... Figure 1 The data query method in the illustrated embodiment.

[0122] In one example, the data query device may further include a communication interface 303 and a bus 304. For example, Figure 7 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 304 and complete communication with each other.

[0123] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0124] Bus 304 includes hardware, software, or both, that couples components of a data query device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 304 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0125] This data query device can achieve a combination based on data query methods. Figures 1-5 Describes the data query methods.

[0126] Furthermore, in conjunction with the data query methods described in the above embodiments, this invention can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data query methods described in the above embodiments.

[0127] This application also provides a computer program product, including a computer program, which, when executed, implements any of the data query methods described in the above embodiments.

[0128] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0129] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0130] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0131] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0132] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A data query method, characterized by, The method comprises: In the case that the data volume difference corresponding to the query data volume under the current time window is in the first preset interval, the number of continuous changes of the window length is determined according to the query data volume under the current time window; wherein the number of continuous changes of the window length represents the number of continuous adjustments of the window length in the same direction up to the current time window in the full-data query process; The window length of the next time window is determined according to the number of continuous changes of the window length; Data query is performed according to the window length of the next time window until the query covers the full data.

2. The method of claim 1, wherein, The step of determining the number of continuous changes of the window length according to the query data volume under the current time window comprises: In the case that the query data volume under the current time window is greater than the preset data volume threshold, the first change number recorded in the first counter is determined, wherein the first change number represents the number of continuous adjustments of the window length in the increasing direction up to the current time window in the full-data query process; In the case that the query data volume under the current time window is less than or equal to the preset data volume threshold, the second change number recorded in the second counter is determined, wherein the second change number represents the number of continuous adjustments of the window length in the decreasing direction up to the current time window in the full-data query process.

3. The method of claim 2, wherein, The step of determining the first change number recorded in the first counter in the case that the query data volume under the current time window is greater than the preset data volume threshold further comprises: In the case that the window length of the next time window in the adjacent time window and the window length of the previous time window in the adjacent time window satisfy the multiplication relationship, the first change number recorded in the first counter is increased; In the case that the window length of the next time window in the adjacent time window and the window length of the previous time window in the adjacent time window do not satisfy the multiplication relationship, the first change number recorded in the first counter is cleared.

4. The method of claim 1, wherein, The step of determining the window length of the next time window according to the number of continuous changes of the window length comprises: In the case that the number of continuous changes of the window length is the first change number recorded in the first counter, the change number threshold and the first change number are compared; In the case that the first change number is less than or equal to the change number threshold, the window length of the next time window is determined according to the window length of the current time window and a preset reduction ratio; In the case that the first change number is greater than the change number threshold, the window length of the next time window is determined according to the length relationship between the window length of the current time window and the window length of the historical time window, wherein the window serial number of the historical time window is obtained by subtracting the change number threshold from the window serial number of the current time window.

5. The method of claim 4, wherein, The step of determining the window length of the next time window according to the length relationship between the window length of the current time window and the window length of the historical time window comprises: in a case where the window duration of the current time window and the window duration of the historical time window satisfy a window duration multiplication relationship, determining the window duration of the next time window according to the window duration of the current time window and a preset reduction ratio; in a case where the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration multiplication relationship, determining a duration fixed value according to the window duration of the current time window and a preset adjustment ratio, and determining the window duration of the next time window according to a duration difference between the window duration of the current time window and the duration fixed value.

6. The method of claim 1, wherein, The step of determining the window duration of the next time window according to the number of continuous changes of the window duration includes: in a case where the number of continuous changes of the window duration in the full-data query process is a second change number recorded in the second counter, comparing the change number threshold value with the second change number; in a case where the second change number is less than or equal to the change number threshold value, determining the window duration of the next time window according to the window duration of the current time window and a preset increase ratio; in a case where the second change number is greater than the change number threshold value, determining the window duration of the next time window according to a duration relationship between the window duration of the current time window and the window duration of a historical time window, wherein a historical window serial number of the historical time window is obtained by subtracting the change number threshold value from the second change number.

7. The method of claim 6, wherein, The step of determining the window duration of the next time window according to the duration relationship between the window duration of the current time window and the window duration of the historical time window includes: in a case where the window duration of the current time window and the window duration of the historical time window satisfy a window duration multiplication relationship, determining the window duration of the next time window according to the window duration of the current time window and a preset reduction ratio; in a case where the window duration of the current time window and the window duration of the historical time window do not satisfy the window duration multiplication relationship, determining a duration fixed value according to the window duration of the current time window and a preset adjustment ratio, and determining the window duration of the next time window according to a duration difference between the window duration of the current time window and the duration fixed value.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: calculating an initial difference value between the query data volume in the current time window and the preset data volume threshold value; taking an absolute value of the initial difference value as the data volume difference value corresponding to the query data volume in the current time window; or, taking a ratio between the absolute value of the initial difference value and the preset data volume threshold value as the data volume difference value corresponding to the query data volume in the current time window.

9. The method of claim 1, wherein, The method further includes: In a case where the data volume difference corresponding to the query data volume in the current time window is in the second preset interval and the query data volume in the current time window is greater than the preset data volume threshold, a fixed duration value is determined according to a window duration of the current time window and a preset adjustment ratio, and a window duration of a next time window is determined according to a duration difference between the window duration of the current time window and the fixed duration value. In a case where the data volume difference corresponding to the query data volume in the current time window is in the second preset interval and the query data volume in the current time window is less than or equal to the preset data volume threshold, a window duration of a next time window is determined according to a sum of a window duration of the current time window and the fixed duration value. In a case where the data volume difference corresponding to the query data volume in the current time window is in a third preset interval, a window duration of a next time window is the window duration of the current time window, where an upper limit value of the third preset interval is less than or equal to a lower limit value of the second preset interval.

10. A data query apparatus, characterized by comprising: The apparatus includes: The processing module is configured to, in a case where the data volume difference corresponding to the query data volume in the current time window is in a first preset interval, determine a number of continuous changes of the window duration according to the query data volume in the current time window, where the number of continuous changes of the window duration represents a number of times of continuous adjustment of the window duration in the same direction in the full-data query process up to the current time window. The processing module is further configured to determine a window duration of a next time window according to the number of continuous changes of the window duration. The query module is configured to perform data query according to the window duration of the next time window until the query covers full data.

11. A data query device, characterized by The data query device includes a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the data query method in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer program instructions are stored on the computer readable storage medium and are executed by the processor to implement the data query method in any one of claims 1-9.

13. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the electronic device to cause the electronic device to perform the data query method in any one of claims 1-9.

Images