Data retrieval method, device, equipment and computer readable storage medium

Abstract

The application provides a data retrieval method, device and equipment and a computer readable storage medium, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The method comprises the following steps: in response to a data retrieval request carrying retrieval information, obtaining source data and a sampling rate corresponding to the source data; the source data is divided into at least two data units for data storage; based on the sampling rate corresponding to the source data, determining a unit sampling rate corresponding to the at least two data units and a data sampling rate corresponding to the data in the data unit; and combining the unit sampling rate, the data sampling rate and the retrieval information, performing data sampling on the source data to obtain target data as a retrieval result of the data retrieval request. Through the application, the number of target data can be effectively reduced, and the efficiency of data retrieval is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to big data and data retrieval technology, and in particular, to a data retrieval method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the rapid development of social informatization and networking, data is growing explosively. Users' retrieval and statistical analysis requirements for massive data are increasing.

[0003] In related technologies, the retrieval operation for massive data often consumes high system resources due to the large amount of data read, and the response to user retrieval requests is timed out. Or part of the result data is returned within the response time, affecting the accuracy of the data retrieval process. SUMMARY

[0004] The embodiments of the present application provide a data retrieval method, device, electronic equipment, computer readable storage medium and computer program product, which can effectively reduce the number of target data and improve the efficiency of data retrieval.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] The embodiments of the present application provide a data retrieval method:

[0007] In response to a data retrieval request carrying retrieval information, source data and a sampling rate corresponding to the source data are obtained;

[0008] The source data is divided into at least two data units for data storage;

[0009] Based on the sampling rate corresponding to the source data, a unit sampling rate corresponding to the at least two data units and a data sampling rate corresponding to the data in the data unit are determined;

[0010] The source data is sampled based on the unit sampling rate, the data sampling rate and the retrieval information to obtain target data;

[0011] The retrieval result including the target data is returned.

[0012] The embodiments of the present application provide a data retrieval device, which comprises:

[0013] The acquisition module is configured to obtain source data and a sampling rate corresponding to the source data in response to a data retrieval request carrying retrieval information; wherein the source data is divided into at least two data units for data storage;

[0014] determining, based on the sampling rate corresponding to the source data, a unit sampling rate corresponding to the at least two data units and a data sampling rate corresponding to data in the data unit;

[0015] sampling the source data to obtain target data in combination with the unit sampling rate, the data sampling rate, and the retrieval information;

[0016] returning the retrieval result including the target data.

[0017] In the foregoing solution, the obtaining module is further configured to parse the data retrieval request to determine a sampling rate mode for the source data.

[0018] When the sampling rate mode is a specified sampling rate mode, the sampling rate carried in the data retrieval request is taken as the sampling rate corresponding to the source data.

[0019] In the foregoing solution, the obtaining module is further configured to parse the data retrieval request to determine a sampling rate mode for the source data.

[0020] When the sampling rate mode is an intelligent sampling rate mode, an estimated data amount corresponding to the source data and a processing amount threshold when the source data is processed are obtained.

[0021] The sampling rate corresponding to the source data is determined in combination with the processing amount threshold and the estimated data amount.

[0022] In the foregoing solution, when the data unit is a data slice, the determining module is further configured to obtain a slice quantity of data slices corresponding to the source data.

[0023] The slice sampling rate corresponding to the at least two data slices is determined as the unit sampling rate in combination with the sampling rate corresponding to the source data and the slice quantity, and the ratio of the sampling rate corresponding to the source data to the slice sampling rate is taken as the data sampling rate corresponding to data in the data slice.

[0024] In the foregoing solution, when the data unit is a data block, the source data is divided into at least two data slices, and each data slice is divided into at least two data blocks. The determining module is further configured to obtain a slice quantity of data slices corresponding to the source data, and determine a slice sampling rate corresponding to the at least two data slices in combination with the sampling rate corresponding to the source data and the slice quantity.

[0025] An sampling rate threshold for the at least two data blocks is obtained.

[0026] determine an intermediate sampling rate for the at least two data blocks in combination with the sampling rate corresponding to the source data and the slice sampling rate;

[0027] determine a unit sampling rate corresponding to the at least two data units and a data sampling rate corresponding to data in the data units based on the sampling rate threshold and the intermediate sampling rate.

[0028] In the foregoing solution, the determining module is further configured to determine a product of the sampling rate corresponding to the source data and the number of slices;

[0029] perform an upward rounding on the product to obtain a number of sampling slices;

[0030] determine a ratio of the number of sampling slices to the number of slices, and use the ratio as a slice sampling rate corresponding to the at least two data slices.

[0031] In the foregoing solution, the sampling module is further configured to sample the at least two data units based on the unit sampling rate to obtain at least one target data unit;

[0032] sample data in each of the target data units based on the data sampling rate to obtain sampling data;

[0033] perform data retrieval in the sampling data based on the retrieval information to obtain the target data.

[0034] In the foregoing solution, the sampling module is further configured to determine a data sampling operator corresponding to data in the data units based on the data sampling rate;

[0035] sample data in the data units based on the data sampling operator to obtain sampling data;

[0036] wherein a ratio of a data amount of the sampling data to a data amount of the data in the data units is equal to the data sampling rate.

[0037] In the foregoing solution, when the data sampling operator is a modulo operation, the sampling module is further configured to obtain an index value corresponding to each piece of data in the data units;

[0038] perform a modulo operation on each of the index values to obtain a modulus value corresponding to each piece of data;

[0039] when the modulus value matches a preset modulus value, use data indicated by the corresponding index value as the sampling data.

[0040] In the foregoing solution, when the data sampling operator is a hash operation, the sampling module is further configured to obtain an index value corresponding to each piece of data in the data units;

[0041] hashing the index values to obtain hash values corresponding to the data;

[0042] when the hash values do not reach a hash value threshold, taking the data indicated by the corresponding index values as the sampling data.

[0043] In the above scheme, the search information includes a search operator and a query statement, and the sampling module is further configured to sample the source data based on the unit sampling rate, the data sampling rate and the search operator to obtain initial sampling data.

[0044] based on the query statement, performing data search in the initial sampling data to obtain the target data.

[0045] In the above scheme, when the search result further includes a statistical result corresponding to the target data, the returning module is further configured to obtain a data statistical method corresponding to the target data.

[0046] based on the data statistical method, performing statistical analysis on the target data to obtain the statistical result.

[0047] An electronic device is provided in an embodiment of the present application, and includes:

[0048] a memory configured to store executable instructions;

[0049] a processor configured to execute the executable instructions stored in the memory to implement the data search method provided in the embodiments of the present application.

[0050] A computer readable storage medium is provided in an embodiment of the present application, and stores executable instructions for causing a processor to execute the data search method provided in the embodiments of the present application.

[0051] A computer program product is provided in an embodiment of the present application, and includes a computer program or instructions for causing a processor to execute the data search method provided in the embodiments of the present application.

[0052] The embodiments of the present application have the following beneficial effects:

[0053] The embodiments of the present application can determine the unit sampling rate of at least two data units and the data sampling rate of data in a data unit based on the sampling rate by analyzing the received data search request to obtain search information and a sampling rate for source data, and then sampling the source data based on the unit sampling rate, the data sampling rate and the search information to obtain target data, thereby greatly reducing the number of target data, reducing the occupation of computing resources and response time, and improving the efficiency of data search. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a schematic diagram of an architecture of a data retrieval system provided by an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of a structure of an electronic device 500 of a data retrieval method provided by an embodiment of the present application;

[0056] Figure 3 is a schematic diagram of a flow of a data retrieval method provided by an embodiment of the present application;

[0057] Figure 4 is a schematic diagram of a visual interface of a data retrieval request provided by an embodiment of the present application;

[0058] Figure 5 is a flowchart of a method for obtaining a sampling rate in an intelligent sampling rate mode provided by an embodiment of the present application;

[0059] Figure 6 is a flowchart of a method for determining a fragment sampling rate provided by an embodiment of the present application;

[0060] Figure 7 is a flowchart of a method for determining a unit sampling rate and a data sampling rate provided by an embodiment of the present application;

[0061] Figure 8 is a flowchart of a data sampling method provided by an embodiment of the present application;

[0062] Figure 9 is a flowchart of a data sampling method based on a modulo operation provided by an embodiment of the present application;

[0063] Figure 10 is a flowchart of a data sampling method based on a hash operation provided by an embodiment of the present application;

[0064] Figure 11 is a flowchart of a data sampling method provided by an embodiment of the present application;

[0065] Figure 12 is a flowchart of a data statistical analysis determined based on a data sampling method provided by an embodiment of the present application;

[0066] Figure 13A is a schematic diagram of data fragment sampling provided by an embodiment of the present application;

[0067] Figure 13B is a schematic diagram of a fragment sampling rate value provided by an embodiment of the present application;

[0068] Figure 14 is a schematic diagram of a fragment sampling rate correction provided by an embodiment of the present application;

[0069] Figure 15is another sampling rate diagram provided by the embodiment of the present application.

[0070] Figure 16 is an experimental result analysis diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0072] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0073] If the similar description of "first / second" appears in the application file, the following description is added, in the following description, the terms "first\second\third" related only to distinguish similar objects, and do not represent the specific order of the objects, and it can be understood that "first\second\third" can be interchanged with the specific order or sequence as allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0075] The related data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0076] Before the embodiments of the present application are further described in detail, the terms and terms related to the embodiments of the present application are explained, and the terms and terms related to the embodiments of the present application are applicable to the following explanation.

[0077] 1) Distributed storage system: is the data dispersed storage in multiple independent devices. The traditional network storage system uses centralized storage server to store all data, storage server becomes the bottleneck of system performance, and is the focus of reliability and security, can not meet the needs of large-scale storage applications. Distributed network storage system uses scalable system structure, uses multiple storage servers to share storage load, uses location server to locate storage information, which not only improves the reliability, availability and access efficiency of the system, but also is easy to expand.

[0078] 2) Retrieval: is the instruction issued by the searcher to the computer, and is also the language of man-machine dialogue. Retrieval expresses the retrieval intention of the searcher. Retrieval is usually composed of retrieval words, logical operators, wildcard symbols, etc.

[0079] 3) Data fragmentation: refers to the data in the distributed database can be copied in each physical database in the network site.

[0080] Based on the above explanation of the terms and terminologies involved in the embodiments of the present application, the data retrieval system provided by the embodiments of the present application is described below. Referring to Figure 1 , Figure 1 is the architecture diagram of the data retrieval system provided by the embodiments of the present application. To realize supporting an exemplary application, the terminal (exemplarily shows terminal 400-1 and terminal 400-2) connects the server 200 through the network 300, the network 300 can be a wide area network or a local area network, or a combination of the two, and uses wireless or wired link to realize data transmission.

[0081] The terminal (such as terminal 400-1 and terminal 400-2) is installed and runs a data retrieval client, which is used to provide a graphical operation interface to the user, so that the user can input retrieval information for data retrieval of source data through the graphical operation interface, and receive a retrieval instruction for the source data, and send a data retrieval request carrying the retrieval information to the server 200.

[0082] The server 200 is used to respond to the data retrieval request carrying the retrieval information, acquire the source data and the sampling rate corresponding to the source data; wherein the source data is divided into at least two data units for data storage; based on the sampling rate corresponding to the source data, determine the unit sampling rate corresponding to the at least two data units and the data sampling rate corresponding to the data in the data unit; further, combine the unit sampling rate, the data sampling rate and the retrieval information to sample the source data to obtain target data; finally, return the retrieval result including the target data to the terminal (such as terminal 400-1 and terminal 400-2).

[0083] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0084] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0085] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. The backend services of technical network systems require substantial computing and storage resources. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can be achieved through cloud computing.

[0086] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 providing a data retrieval method according to an embodiment of this application. In practical applications, the electronic device 500 can be... Figure 1 The server or terminal shown is exemplified by electronic device 500. Figure 1 Taking the domain name resolution node shown as an example, an electronic device implementing the data retrieval method of this application embodiment will be described. The electronic device 500 provided in this application embodiment includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0087] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0088] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0089] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0090] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0091] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0092] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0093] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0094] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0095] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0096] In some embodiments, the data sampling device provided in this application can be implemented in software. Figure 2 A sampling device 555 for data stored in memory 550 is shown. It may be software in the form of programs and plug-ins, including the following software modules: acquisition module 5551, determination module 5552, sampling module 5553, and return module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0097] In other embodiments, the data retrieval device provided in this application can be implemented using a combination of hardware and software. As an example, the data retrieval device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data retrieval method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0098] Based on the above description of the data retrieval system and electronic device provided in the embodiments of this application, the data retrieval method provided in the embodiments of this application will be described below. In some embodiments, the data retrieval method provided in the embodiments of this application may be implemented by a server or a terminal alone, or by a server and a terminal working together. The following describes the data retrieval method provided in the embodiments of this application using the implementation of a server as an example.

[0099] See Figure 3 , Figure 3 This is a flowchart illustrating the data retrieval method provided in this application embodiment. The data retrieval method provided in this application embodiment includes:

[0100] In step 101, the server responds to the data retrieval request carrying retrieval information by obtaining the source data and the sampling rate corresponding to the source data.

[0101] In practice, the server can obtain the sampling rate X for the source data in two modes: specified sampling rate mode and intelligent sampling rate mode. Specified sampling rate mode uses a specified sampling rate, while intelligent sampling rate mode uses an intelligent sampling rate. It should be noted that the source data is divided into at least two data units for data storage.

[0102] In some embodiments, the server may obtain the sampling rate corresponding to the source data by parsing the received data retrieval request to determine the sampling rate mode for the source data; when the sampling rate mode is the specified sampling rate mode, the sampling rate carried in the data retrieval request is used as the sampling rate corresponding to the source data.

[0103] In actual implementation, after the server receives a data retrieval request for the source data, it parses the data retrieval request and obtains the corresponding parsing results. When the parsing results include a specific value of the sampling rate, it indicates that the sampling rate mode for the data is the specified sampling rate mode, and the sampling rate in the parsing results is the specified sampling rate.

[0104] For example, see Figure 4 , Figure 4 This is a schematic diagram of the visual interface for a data retrieval request provided in an embodiment of this application. The diagram provides the user with an input interface for performing a data retrieval. In the input box shown as number 1, the user enters the retrieval information. When the sampling rate mode for the source data is a specified sampling rate mode, the specified sampling rate can be entered in the control shown as number 2. When the sampling rate mode for the source data is a smart sampling rate mode, the control shown as number 2 can display "smart" and does not accept user input. Then, by clicking the "Execute" control, a data retrieval request is generated. After receiving the data retrieval request, the server parses the corresponding information, performs the corresponding data retrieval operation, returns the retrieval results, and displays the retrieval results in the area shown as number 3 in the diagram.

[0105] In some embodiments, see Figure 5 , Figure 5 This is a flowchart of the method for obtaining the sampling rate in the intelligent sampling rate mode provided in the embodiments of this application, combined with... Figure 5 The steps shown are explained.

[0106] Step 1011: The server parses the data retrieval request to determine the sampling rate pattern for the source data.

[0107] In actual implementation, after the server receives a data retrieval request for the source data, it parses the data retrieval request and obtains the corresponding parsing result. When the parsing result does not include the specific value of the sampling rate, it indicates that the sampling rate mode for the data is the intelligent sampling rate mode, and the sampling rate in the parsing result is the intelligent sampling rate.

[0108] Step 1012: When the sampling rate mode is intelligent sampling rate mode, obtain the estimated data volume corresponding to the source data and the processing volume threshold when processing the source data.

[0109] In actual implementation, when the sampling rate mode for the data is the intelligent sampling rate mode, the server first establishes the processing volume threshold D0 (i.e. the amount of data that the system can process within a reasonable time) when processing the source data. D0 is a positive integer. Then, it determines the estimated data volume D corresponding to the source data. D is a positive integer. Finally, it determines the sampling rate X according to the intelligent sampling rate calculation formula.

[0110] Sampling rate X = min(1, D0 / D*c) (1)

[0111] Where c is the reserve parameter, c≤1.

[0112] For example, if the server is set to process data within an acceptable timeframe (e.g., 30 seconds), with a processing threshold D0 = 10 million and an estimated data volume D = 20 million, the sampling rate X can be set to a maximum of 0.5. When the sampling rate X = 0.5, the server's computational capacity for sampling is fully utilized. Since D0 and D are both estimated information and may contain errors, a pre-flow parameter c can be set to ensure the accuracy of data sampling, allowing the server to maintain a balanced data sampling state.

[0113] Step 1013: Combine the processing volume threshold and the estimated data volume to determine the sampling rate corresponding to the source data.

[0114] In practice, the server can determine the sampling rate X based on the above intelligent sampling rate calculation formula, combined with the processing volume threshold and the estimated data volume.

[0115] In step 102, based on the sampling rate corresponding to the source data, the unit sampling rate corresponding to at least two data units and the data sampling rate of the data within the corresponding data unit are determined.

[0116] In practical implementation, in distributed storage systems, source data is often divided into at least two data units for distributed storage, and multiple data shards can be distributed across multiple devices. Data within a data unit is managed through these units. Therefore, when sampling the source data at a determined sampling rate X, sampling can begin with at least two corresponding data units, followed by sampling the data within each data unit. The server can first sample the at least two corresponding data units based on the unit sampling rate to determine the sampled data units. Then, the server continues to sample the data within each sampled data unit based on the data sampling rate to obtain the target data. This distributes the read load across different data unit nodes while reducing I / O resource consumption.

[0117] Regarding the unit sampling rate, in some embodiments, when the data unit is a data fragment, the server can determine the unit sampling rate in the following way: the server obtains the number of data fragments corresponding to the source data; combining the sampling rate corresponding to the source data and the number of fragments, the server determines the fragment sampling rate corresponding to at least two data fragments as the unit sampling rate; after determining the fragment sampling rate, the server determines the ratio of the sampling rate corresponding to the source data to the fragment sampling rate, and uses the obtained ratio as the data sampling rate of the data within the corresponding data fragment.

[0118] In practice, the data unit used to store source data can be a data shard. The server manages the data through sharding. Sharding ensures that data within different data shards is evenly distributed within the same time period. The server samples multiple data shards storing the source data based on a shard sampling rate. The shard sampling rate is represented by ns / S, where S is the number of data shards used to store the source data (S≥1), and ns is the number of sampled data shards (1≤ns≤S). It should be noted that the sampling rate and the shard sampling rate have the following relationship: Sampling rate X = shard sampling rate * other sampling rates, denoted by P, i.e., sampling rate X = ns / S * P. When the source data is only divided into data shards for storage, P can be used to characterize the data sampling rate within each data shard.

[0119] In some embodiments, see Figure 6 , Figure 6 This is a flowchart of the method for determining the segmented sampling rate provided in the embodiments of this application, combined with... Figure 6 The method shown illustrates how to determine the sampling rate for data fragments.

[0120] Step 201: The server determines the product of the sampling rate and the number of fragments corresponding to the source data.

[0121] In practice, since the sampling rate for data fragments is a coarse-grained sampling rate, the supported fragment sampling rates are 1 / S, 2 / S, ..., (S-1) / S, 1. That is, ns takes values ​​of 1, 2, ..., (S-1), S. After parsing the data retrieval request, the server obtains the sampling rate X. To determine the fragment sampling rate, the server can first determine the number of data fragments to be sampled, i.e., the size of ns.

[0122] Step 202: Round the product up to obtain the number of sampling segments.

[0123] In practical implementation, the server can determine the number of data fragments for sampling within the same time period based on the following formula: ns = cell(X*S). That is, first determine the sampling rate X and the number of data fragments S corresponding to the source data, then multiply the sampling rate X by the number of data fragments S, and round the product up using the cell function to determine the number of sampling fragments ns.

[0124] For example, if the sampling rate for the source data is set to X = 30% and the number of data fragments is S = 5, then according to ns = cell(X*S), we can determine that ns = cell(30%*5) = 2, that is, the number of sampling fragments is determined to be 2. The server samples 2 data fragments from the 5 data fragments storing the source data.

[0125] Step 203: Determine the ratio of the number of sampling fragments to the number of fragments, and use the ratio as the fragment sampling rate corresponding to at least two data fragments.

[0126] Continuing with the previous example, the server samples 2 data fragments from the 5 data fragments storing the source data. The ratio of the number of sampled fragments to the total number of fragments is determined to be 2 / 5. Therefore, 2 / 5 = 40% is taken as the fragment sampling rate for the data fragments.

[0127] In some embodiments, see Figure 7 , Figure 7 This is a flowchart illustrating the method for determining the unit sampling rate and data sampling rate provided in an embodiment of this application. In this embodiment, the data unit is a data block, and the source data is still divided into at least two data shards for storage. Each data shard is further divided into at least two data blocks, and the source data is ultimately stored in the data blocks. Figure 3 , combined Figure 7 The steps shown illustrate the implementation process of step 102.

[0128] Step 1021: The server obtains the number of data fragments corresponding to the source data, and determines the fragment sampling rate corresponding to at least two data fragments by combining the sampling rate and the number of fragments corresponding to the source data.

[0129] It should be noted that in this embodiment, the data unit is a data block, the source data is divided into at least two data shards, and each data shard is divided into at least two data blocks.

[0130] In practical implementation, in distributed data storage systems, servers can also manage source data in the form of data blocks, including data encoding and compression within data blocks. Disk I / O itself is also performed in blocks. When source data is divided into at least two data shards, each containing at least two data blocks, the block sampling rate and the data sampling rate within a data block can be determined by the sampling rate X and the shard sampling rate ns / S. Specifically, the sampling rate X, shard sampling rate ns / S, block sampling rate P1, and data sampling rate P2 within a block satisfy the following relationship: Sampling rate X = ns / S * P1 * P2. It should be noted that when ns / S = 1, i.e., when the shard sampling rate for a data shard is 1, the sampling rate X = P1 * P2. Here, ns / S = 1 includes two cases: one is when the number of data shards S = 1, in which case that data shard will definitely be accessed; the other is when the number of data shards S > 1, in which case each data shard will be accessed.

[0131] In actual implementation, the number of data fragments S > 1 for the source data, and each data fragment is divided into at least two data blocks. In this case, the sampling rate X = fragment sampling rate ns / S * block sampling rate P1 * intra-block data sampling rate P2. Based on Figure 7 The method shown for determining the fragment sampling rate determines the fragment sampling rate ns / S for the data fragment, and then determines the product P1*P2 based on the sampling rate X = ns / S*P1*p2.

[0132] Because when the server accesses data within a data block, it also accesses other data nearby within the same block, and may even read the entire block. (The target requirement is to access a single piece of data, but in practice, 100 pieces of data are accessed to access that single piece of data, which is a case of IO amplification). In sampling scenarios, because the sampled data is accessed in a uniform distribution, the actual IO amplification is significant, affecting the final sampling and reading efficiency. Therefore, in practical implementation, the sampling rate P can be split into two parts: P = P1 * P2. P1 is the block sampling rate (block-level sampling rate) for the data block; if a block is not sampled, then the data within that entire block will not be sampled. P2 is the sampling rate for the data within the data block (block-level sampling rate) (when a block is determined to be sampled, each document within it is then checked against P2 to determine whether it should be sampled). Therefore, the method for determining the block sampling rate P1 and the data sampling rate P2 can be adjusted according to the current number of server IOs and the IO amplification. That is, the size of P1 and P2 can be determined by a preset adjustment strategy. However, regardless of the adjustment strategy, it must be ensured that P1, P2, ns / S, and sampling rate X satisfy the condition of sampling rate X = ns / S * P1 * p2.

[0133] Step 1022: Obtain the sampling rate threshold for at least two data blocks.

[0134] In practice, a sampling rate threshold can be preset for at least two data blocks. It should be noted that the setting of the sampling rate threshold is related to the number of I / O operations in the system and the I / O amplification during data access.

[0135] Step 1023: Combine the sampling rate corresponding to the source data and the fragment sampling rate to determine the intermediate sampling rate for at least two data blocks.

[0136] In practice, since the sampling rate X = ns / S*P1*p2, the server can determine the value of P1*P2 based on the number of data fragments S and the sampling rate X. This value can be regarded as the intermediate sampling rate for a data block, that is, this value is the product of the block sampling rate for at least two data blocks within the data fragment and the data sampling rate within the block.

[0137] For example, if the sampling rate X = 0.002 and the fragment sampling rate ns / S = 1 / 10, that is, sampling one data fragment within 10 data fragments, then the intermediate sampling rate P1*P2 = 0.02 can be determined.

[0138] Step 1024: Based on the sampling rate threshold and the intermediate sampling rate, determine the unit sampling rate corresponding to at least two data units and the data sampling rate of the data within the corresponding data unit.

[0139] In actual implementation, the server determines the block sampling rate P1 and the intra-block data sampling rate P2 based on the sampling rate threshold and the intermediate sampling rate determined in step 1023.

[0140] In some embodiments, the server may determine the unit sampling rate and the data sampling rate of the corresponding data unit in the following ways: when the intermediate sampling rate reaches the sampling rate threshold, the server determines that the value of the block sampling rate corresponding to at least two data blocks is equal to 1, and uses the intermediate sampling rate as the data sampling rate of the data in the corresponding data block; when the intermediate sampling rate does not reach the sampling rate threshold, the server determines the ratio of the intermediate sampling rate to the sampling rate threshold; and uses the ratio as the block sampling rate corresponding to at least two data blocks, and uses the sampling rate threshold as the data sampling rate of the data in the corresponding data block.

[0141] In practice, the server compares the intermediate sampling rate with the sampling rate threshold. When the intermediate sampling rate reaches the sampling rate threshold, it indicates that the data distribution within the data block is relatively concentrated, and each data block within the data fragment can be sampled. That is, the block sampling rate P1 = 1 for the data block, and the data sampling rate for the data within the block is directly set to the intermediate sampling rate, i.e., P2 = sampling rate X / (ns / S). When the intermediate sampling rate does not reach the sampling rate threshold, it indicates that the data distribution within the data block is relatively scattered. In this case, in order to transform scattered access into concentrated access and reduce the I / O amplification effect, the sampling rate threshold can be directly used as the data sampling rate for the data within the block, and the ratio of the intermediate sampling rate to the sampling rate threshold can be used as the block sampling rate for the data block.

[0142] For example, in the diagram, the sampling rate X = 0.002 and the fragment sampling rate ns / S = 1 / 10 are set. That is, within the same time period, one data fragment is sampled from 10 data fragments. Then, the intermediate sampling rate P = P1 * P2 = 0.02 can be determined. When each data block in the data fragment is used as a sampled data block (P1 = 1), 2% of the data is sampled for each data block. If the number of data blocks is too large, the server access will be scattered, and the IO amplification will be obvious. In order to achieve centralized access, a data sampling rate threshold of 20% is set, that is, the minimum sampled data in each data block is 20%. Since the current intermediate sampling rate P = 0.02 = 2%, which is less than the sampling rate threshold, P2 is directly set to 20%, that is, 20% of the data is sampled for each data block. The block sampling rate P1 for the data block is 10%, that is, at least 1 / 10 of the data blocks in the sampled data fragment are accessed. In this way, by sampling the data blocks, some scattered access can be turned into centralized access, thereby greatly reducing the IO amplification effect.

[0143] In step 103, the source data is sampled by combining the unit sampling rate, the data sampling rate, and the retrieval information to obtain the target data as the retrieval result of the data retrieval request.

[0144] In some embodiments, see Figure 8 , Figure 8 This is a flowchart of the data sampling method provided in the embodiments of this application, combined with... Figure 8 The steps shown are explained.

[0145] Step 1031: The server samples at least two data units based on the unit sampling rate to obtain at least one target data unit.

[0146] In practice, the server samples data units according to a determined unit sampling rate to determine at least one data unit to be accessed, i.e., at least one target data unit. It should be noted that when the data unit is a data fragment, the unit sampling rate is the same as the fragment sampling rate. In this case, the server samples at least two data fragments storing the source data according to the fragment sampling rate to obtain the target data fragment (i.e., the data fragment the server wants to access). When the data unit is a data block, and the system storing the source data does not support data fragmentation, the unit sampling rate is the same as the block sampling rate. When there are multiple data fragments, and each data fragment is divided into at least two data blocks, the unit sampling rate includes both the fragment sampling rate and the block sampling rate. In this case, the server samples at least two data fragments storing the source data according to the fragment sampling rate to obtain at least one target data fragment, and then samples the data blocks within the target data fragment according to the block sampling rate to obtain at least one target data block (i.e., the data block within the target data fragment to be accessed).

[0147] For example, the number of data shards is set to 10, and the shard sampling rate is 0.1. That is, one data shard (target data shard) is determined from the 10 data shards for access. The target data shard includes 20 data blocks. According to the block sampling rate of 0.1, the server samples 2 data blocks from these 20 data blocks as the target data blocks to be accessed by the server. That is, the server will read data from these two target data blocks.

[0148] Step 1032: Based on the data sampling rate, sample the data in each target data unit to obtain sampled data.

[0149] In practice, once the server determines the target data unit (data fragment or data block) to be accessed, it can sample the data within the data unit according to the data sampling rate to determine the sampled data.

[0150] In some embodiments, the server may sample data within a data unit in the following manner: the server determines a data sampling operator for the data within the corresponding data unit based on a data sampling rate; the server samples the data within the data unit based on the data sampling operator to obtain sampled data; wherein the ratio of the amount of sampled data to the amount of data within the data unit is equal to the data sampling rate.

[0151] In practical implementation, after determining the data sampling rate within a data unit, the server can use data sampling operators to determine which data within the data unit can be used as sampled data. In other words, the data sampling operator can be understood as a sampling function or filtering condition used to sample the data within the data unit. Finally, the ratio of the amount of sampled data obtained by the server through the data sampling operator to the total amount of data in the current target data unit is equal to the data sampling rate. Data sampling operators include, but are not limited to: modulo operation, hash operation, and random probability selection.

[0152] In some embodiments, the server can sample data from the target data unit by random probability selection, that is, as long as the ratio of the amount of sampled data to the amount of data in the target data unit is equal to the data sampling rate.

[0153] For example, if the data sampling rate is set to 20%, the server can randomly sample 20% of the data from the target data unit.

[0154] In some embodiments, see Figure 9 , Figure 9 This is a flowchart of a data sampling method based on modulo operation provided in an embodiment of this application. (Combined with...) Figure 9 The steps shown illustrate the data sampling process when the data sampling operator is a modulo operation.

[0155] Step 301a: The server obtains the index value corresponding to each data in the data unit.

[0156] In practice, when the data sampling operator is a modulo operation, the server obtains the index value used to indicate the data. The index value can be a data identifier or a pointer to the address of the data storage, but it must be ensured that the index value is a numeric type.

[0157] Step 302a: Take the modulo of each index value to obtain the modulo value of each corresponding data.

[0158] In practice, the server performs a modulo operation on each index value to obtain the corresponding modulo value. For example, the server modulo the index value of the data by 10, and the modulo value can be 0, 1, 2, ..., 9.

[0159] Step 303a: When the modulus value matches the preset modulus value, the data indicated by the corresponding index value is used as the sampled data.

[0160] In practice, the server matches the modulus value obtained in step 302a with preset modulus values. If a modulus value matches a preset modulus value, the data indicated by the corresponding index value is used as the sampled data. It should be noted that there can be multiple preset modulus values, such as preset modulus 1, preset modulus 1, ..., preset modulus N, etc. Applicable scenarios include: when matching the modulus value with preset modulus 1, if the ratio of the sampled data volume to the total data volume within the data unit is less than the data sampling rate, the server can use preset modulus 2 to sample more data. That is, matching the modulus value with preset modulus 2 and using the matching data as sampled data, until the ratio of the sampled data volume to the total data volume within the data unit equals the data sampling rate. In practical applications, the server can also assign a priority to each preset modulus value and select a target preset modulus value to match with the modulus value obtained in step 302a based on the priority of the preset modulus value.

[0161] For example, a data unit stores 10,000 data entries with a data sampling rate of 20%. Preset modulo values ​​are set to 1, 2, and 3. A modulo operation is performed between the index values ​​of the 10,000 data entries and 8. This results in 1,000 sampled data entries with a modulo value of 1, which is less than the required 2,000 entries for the data sampling rate. The server then uses data entries with a modulo value of 2 as sampled data. When there are 1,500 sampled data entries with a modulo value of 2, 100 of them are selected, and the server's sampling process for that data unit ends. When there are 500 sampled data entries with a modulo value of 2, the number of sampled data entries still does not reach 2,000. The server continues to use data entries with a modulo value of 3 as sampled data until the number of sampled data entries reaches 2,000, at which point the server's sampling process for that data unit ends.

[0162] In some embodiments, see Figure 10 , Figure 10 This is a flowchart of a data sampling method based on hash operations provided in an embodiment of this application. (Combined with...) Figure 10 The steps shown illustrate the data sampling process when the data sampling operator is a hash operation.

[0163] Step 301b: The server obtains the index value corresponding to each data in the data unit.

[0164] In practice, when the data sampling operator is a hash operation, the server obtains the index value used to indicate the data. The index value can be a data identifier or a pointer to the address of the data storage, but it must be ensured that the index value is a numeric type.

[0165] Step 302b: Hash the index value to obtain the hash value of each corresponding data.

[0166] In practice, the server performs a hash operation on each index value to obtain the corresponding hash value.

[0167] Step 303b: When the hash value does not reach the hash value threshold, the data indicated by the corresponding index value is used as the sampling data.

[0168] For example, a hash function hashFunc is defined, and the index value of the data is used as the input information of hashFunc to obtain the corresponding hash value. When the hash value is less than the hash value threshold, the sampled data indicated by the corresponding index value is sampled.

[0169] Step 1033: Based on the retrieval information, perform data retrieval in the sampled data to obtain the target data.

[0170] In practice, the server retrieves the sampled data based on the retrieval information carried in the data retrieval request to obtain the target data. Retrieval information may include retrieval operators and query statements, as well as custom query conditions, query statements, and search expressions.

[0171] In some embodiments, see Figure 11 , Figure 11 This is a flowchart of the data sampling method provided in an embodiment of this application. In this case, the retrieval information includes retrieval operators and query statements, combined with... Figure 11 The steps shown are explained.

[0172] Step 401: The server combines the unit sampling rate, the data sampling rate, and the retrieval operator to sample the source data and obtain the initial sampled data.

[0173] In practice, the server samples data units based on the unit sampling rate to determine at least one target data unit to be accessed. Then, it samples the target data unit according to the data sampling rate to obtain intermediate sampled data. Next, it retrieves the retrieval operator carried in the data retrieval request and samples the intermediate sampled data to obtain initial sampled data.

[0174] For example, the server samples user log data. The data retrieval request carries the retrieval information "msg:error|select avg(rate)group by date", where the retrieval operator is "msg:error". This is a custom key:value retrieval operator, meaning it samples error logs of type "error" from the user log data. By sampling the operator "msg:error", the server obtains error log messages, which is also a way to filter the source data. It should be noted that the specific form of the retrieval operator can be determined according to the specific sampling requirements.

[0175] Step 402: Based on the query statement, perform data retrieval in the initial sampled data to obtain the target data.

[0176] Continuing from the previous example, the server retrieves the query statement "select avg(rate)group by date" from the retrieval information "msg:error|select avg(rate)group by date", and performs a query on the sampled data obtained based on the sampling rate X and the retrieval operator to obtain the target data.

[0177] In some embodiments, the server may perform statistical analysis on the target data in the following ways: when the search results also include statistical results for the target data, the server obtains the data statistical method corresponding to the target data; based on the data statistical method, the server performs statistical analysis on the target data to obtain statistical results.

[0178] In practice, the server can perform statistical analysis on the sampled data obtained based on the sampling rate X and the retrieval operator. The specific statistical method can be determined according to actual needs. The statistical method includes, but is not limited to, calculating the average (avg), calculating the number (count), and finding the top N items (topN).

[0179] Continuing from the previous example, "select avg(rate)group by date" calculates the average avg of the sampled data, that is, the average value of the rate field based on the date.

[0180] By applying the embodiments of this application, the server samples the source data by determining the unit sampling rate of the corresponding data unit and the data sampling rate within each data unit. This significantly reduces the amount of data involved in query analysis and, while ensuring the accuracy of the customer's query results, reduces the occupation of system resources and response time. Simultaneously, during the data sampling process, shard-level sampling technology reduces the number of I / O operations, and block-level sampling technology reduces I / O amplification, achieving highly efficient data sampling and retrieval.

[0181] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0182] In relevant massive data retrieval and analysis systems, services such as data writing, data querying, and data analysis are provided. Further aggregation and analysis of the retrieved datasets can be performed, such as averages, counts, histograms, and top-N operations. However, when dealing with massive datasets, the following problems often arise: excessive data volume leading to high I / O resource consumption; excessive computational load resulting in high computational resource consumption; and overall query and analysis timeouts, failing to yield results. The main reason for these problems is the sheer size of the retrieval result set, leading to an excessive amount of data for subsequent analysis. For retrieval and analysis systems that do not provide data sampling capabilities, when faced with massive datasets, they generally experience timeouts without results or excessively long execution times. While some retrieval and analysis systems offer basic sampling capabilities, these are highly dependent on the user's own data fields, making I / O amplification very likely and resulting in poor actual performance. Other data detection services, when analyzing large amounts of data, return partial data calculation results that differ significantly from the actual calculation results.

[0183] Based on this, embodiments of this application provide a data retrieval method, which can also be understood as a retrieval-based method. This data retrieval method samples data during the retrieval phase by specifying a sampling rate / intelligent sampling rate, reducing the amount of data involved in the analysis and thus avoiding resource and timeout issues encountered in the aforementioned massive data analysis. Simultaneously, during the data sampling process, shard-level sampling technology reduces the number of I / O operations, and block-level sampling technology reduces I / O amplification, achieving highly efficient data sampling and retrieval.

[0184] In practical implementation, the data retrieval method provided in this application can be used in massive data retrieval and analysis systems, including but not limited to log retrieval systems and user data analysis systems. See also Figure 4 The visual interface interaction process is as follows: First, the user enters the query analysis statement through the query statement input box in the interface. Then, the user sets the sampling rate, which can be set by specifying the sampling rate or by setting the intelligent sampling rate. Finally, the user clicks the "Execute Analysis" function button in the interface to send a data retrieval and analysis request to the server and receives the data analysis results returned by the server.

[0185] Next, the data retrieval method provided in this application embodiment will be described from a technical implementation perspective. In actual implementation, the server can obtain the sampling rate X for massive data in two modes: specified sampling rate mode and intelligent sampling rate mode. Specified sampling rate mode corresponds to a specified sampling rate, while intelligent sampling rate mode corresponds to an intelligent sampling rate. After receiving a data retrieval request for massive data, the server parses the request and obtains the corresponding parsing result. When the parsing result includes a specific value of the sampling rate, it indicates that the sampling rate mode for the data is specified sampling rate mode, and the sampling rate in the parsing result is the specified sampling rate. When the parsing result does not include a specific value of the sampling rate, it indicates that the sampling rate mode for the data is intelligent sampling rate mode, and the sampling rate in the parsing result is the intelligent sampling rate.

[0186] In actual implementation, when the sampling rate mode for the data is a specified sampling rate mode, the sampling rate carried by the received data retrieval request is read as the sampling rate X.

[0187] In practical implementation, when the data sampling rate mode is set to intelligent sampling rate mode, the server can determine a reasonable sampling rate X in the following way: First, it is necessary to establish a system support limit of L (i.e., the amount of data the system can process within a reasonable time). The estimated amount of customer query results data is N. A recommended sampling rate X can be: Sampling rate X = min(1, L / N*c), where c is a system reserve parameter, c≤1. For example, if the server can support L of 10 million records and N of 20 million records within an acceptable time range for the user (e.g., 30 seconds), theoretically, the sampling rate X could be set to 0.5 (10 million / 20 million). However, this would completely fill the server's computing power used for sampling. Since L and N are estimated, there will be some error. Therefore, in practical applications, a system pre-flow parameter c can be set to ensure the server maintains a good sampling state.

[0188] In some embodiments, see Figure 12 , Figure 12 This is a flowchart of the data statistical analysis determined by the data-based retrieval method provided in the embodiments of this application, combined with... Figure 12 The steps shown are explained.

[0189] Step 501: The server performs time segmentation on at least two data segments storing the data to be sampled, and determines the segment sampling rate.

[0190] In practical implementation, in distributed storage systems, the data to be sampled is often stored across multiple devices and managed through data sharding. This ensures that data from the same time period is evenly distributed across different data shards, achieving load balancing. Since the data in different shards is evenly distributed for the same time period, data sampling can be achieved by selecting a portion of the data from a specific shard for statistical analysis. This distributes the read load across different data shard nodes while reducing I / O resource consumption.

[0191] The sampling method described above for data sharding can be called shard sampling, or shard-level sampling. The server implements shard sampling based on the shard sampling rate, which can be represented by ns / S. S is the number of data shards, S≥1, and ns is the number of accessed data shards, 1≤ns≤S. It should be noted that the sampling rate and the shard sampling rate have the following relationship: Sampling rate X = Shard sampling rate * Other sampling rates, denoted by P, i.e., Sampling rate X = ns / S * P.

[0192] For example, see Figure 13A , Figure 13A This is a schematic diagram of data fragmentation sampling provided in an embodiment of this application. In the diagram, the number of data fragments corresponding to the data to be sampled is set to S = 5, and the fragmentation sampling rate is 40%. In the diagram, the fragmentation sampling rate determined based on the data fragments is 40% = 2 / 5, where 5 is the number of data fragments. Within the same time period (t1, t2, etc.), two different fragments are sampled from the five fragments. This achieves a 40% fragmentation sampling rate. The fragmentation sampling process can be completed simply by reading the data within the data fragments represented by the gray shaded area in the diagram, thereby reducing the number of I / O operations.

[0193] As can be seen from the above examples, the sharding sampling rate for data sharding only supports coarse-grained sharding sampling rates such as 1 / S, 2 / S, ..., S / S. See [link to relevant documentation]. Figure 13B , Figure 13B This is a schematic diagram of the fragment sampling rate provided in the embodiments of this application. Taking S=5 as an example, the corresponding fragment sampling rate can be 20%=1 / 5, 40%=2 / 5, 60%=3 / 5, 80%=4 / 5 and 100%=5 / 5, etc., which is the number of data fragments accessed in the same time period, ns. The method of determining ns can be ns=cell(X*S), where X is the aforementioned intelligent sampling rate or specified sampling rate, and S is the number of data fragments corresponding to the data to be sampled.

[0194] For the above ns / s = 40%, when the sampling rate X = 40%, we can directly sample 2 out of 5 data segments and sample 100% of the data within the sampled data segments to meet the requirement of a 40% sampling rate for the data to be sampled.

[0195] In practical applications, see Figure 14 , Figure 14 This is a schematic diagram of the fragment sampling rate correction provided in the embodiment of this application. In the figure, the fragment sampling rate for data fragments can support coarse-grained fragment sampling rates such as 1 / S, 2 / S, ..., S / S. Assuming that the fragment sampling rate for data fragments is 3 / S, while the actual required sampling rate X is less than 3 / S and greater than 2 / S, it can be corrected by other sampling rates P. P can also be called the correction coefficient for correcting the fragment sampling rate.

[0196] For example, taking S=5, when the sampling rate X=50%, the fragmented sampling rate ns / s=40% cannot reach the sampling rate X. The correction of P makes ns / S*P=50%.

[0197] It should be noted that when the retrieval and analysis system itself does not support data sharding, it can be assumed that the number of shards S = 1, then ns = 1, and the sharding sampling rate at the corresponding data sharding level is 1.

[0198] Step 502: Parse the query statement and combine it with the sampling operator conditions to sample the data.

[0199] In practice, the server parses the query statement carried in the data retrieval request and converts it into a logical query tree, containing logical combinations such as AND (&), OR (||), and NOT (!). Sampling operators for the AND condition are added to the logical query tree to achieve the combination of query conditions and sampling operators. Here, the sampling operator can be understood as a sampling function or a filtering condition.

[0200] For example, such as Figure 4 In the search results, the search information output on the search interface is "msg:error|select avg(rate)group by date". Here, msg:error can be regarded as a sampling operator, which is displayed in key:value format. For message log systems, it filters message information of type error; "select avg(rate)group by date" is the corresponding query statement.

[0201] Step 503: Combine the block sampling rate and sampling operator for the data block to perform data block sampling.

[0202] Here, sampling is performed at the block sampling rate of the data block, which is actually a block-level sampling process. In storage systems, data is generally managed in blocks, and data encoding and compression are performed within the block. Disk I / O itself is also performed in blocks. Therefore, when accessing a document within a data block, data near the document within the block is also accessed, and even the entire block of data may be read. (The target requirement is to access a single piece of data, but in reality, 100 pieces of data are accessed to access that single piece of data, which is a case of I / O amplification). In sampling scenarios, because the sampled data is accessed evenly, the actual I / O amplification is significant, affecting the final sampling and reading efficiency. Based on this, in practical implementation, the other sampling rate P can be split into two parts: P = P1 * P2. P1 is the block sampling rate (block-level sampling rate) for the data block. If a block is not sampled, then the data within the entire block will not be sampled. P2 is the sampling rate for the data within the data block (block-level sampling rate) (when a block is determined to be sampled, each document within it is then checked against P2 to determine whether it should be sampled). In other words, the sampling rate X = ns / S * P1 * P2 is guaranteed to be equal to the product of the data shard sampling rate ns / S, the block sampling rate P1 for the data blocks within the data shard, and the data sampling rate P2 for the data within the data blocks.

[0203] For example, see Figure 15 , Figure 15 This diagram illustrates other sampling rates provided in this application's embodiments. In the diagram, when the sampling rate X = 0.002, the fragment sampling rate is 1 / 10 = 0.1, and other sampling rates P = 2%, each data block within a data fragment needs to be accessed (block sampling rate P1 = 1), and 2% of the data is sampled from each data block. This results in scattered data access within data blocks, easily causing I / O amplification. When the block sampling rates P1 = 10% and P2 = 20%, only 1 / 10 of the data blocks in at least two data fragments are accessed, and 20% of the data is sampled from each accessed data block. This makes I / O more concentrated, only 1 / 10 of the original. Thus, through block-level sampling, some scattered access is transformed into concentrated access, thereby significantly reducing the I / O amplification effect.

[0204] Step 504: Sample the data within the data block by combining the data sampling rate and sampling operator.

[0205] Here, the data sampling rate refers to the sampling rate used when sampling data within a data block. Specifically, it's the sampling rate of the data when performing intra-block sampling, and can be represented by P2. The server can generate a data sampling operator for retrieval based on the data sampling rate. This data sampling operator must satisfy the following condition: after traversing all document IDs within the data block, the number of documents obtained through the sampling operator divided by the total number of documents equals the intra-block sampling rate (i.e., the final data sampling rate for the data within the data block).

[0206] In practice, available data sampling operators include, but are not limited to: modulo operation, hash operation, and random probability selection.

[0207] For example, taking hashing as the data sampling operator, a hash function `hashfunc` is defined. The input parameter is the document identifier (`doc_id`), and the output is the hash value corresponding to the document ID. The obtained hash value is compared with a preset threshold. When the hash value is less than the threshold, the data corresponding to the document ID is used as the sampling data. The threshold is related to the block sampling rate.

[0208] hashfunc(doc_id)<Int.MAX*P2.=> The doc_id sample

[0209] The hashfunc(doc_id) > Int.MAX*P2. => This means the doc_id will not be sampled.

[0210] Step 505: Search the retrieval system.

[0211] Here, the query operations performed in the retrieval system include steps such as index query, document scoring, and document sorting. Note that the query conditions at this point already include the sampling operator.

[0212] Step 505: Obtain the list of sampled data after sampling.

[0213] In practice, the server retrieves sampled data that meets both the query conditions and the sampling conditions, based on the aforementioned sampling of data shards, data blocks, and data within data blocks. This data can be a document ID.

[0214] Step 506: Extract relevant field data from the sampled data and perform statistical analysis.

[0215] Here, the data fields are analyzed based on the document ID list in S6, and the final statistical analysis is completed and returned to the user. The data volume has been reduced according to the sampling rate, so the processing time is significantly reduced.

[0216] See Figure 16 ,Figure 16 This is an analysis chart of the experimental results provided in an embodiment of this application. In the chart, when the sampling rate X = 1 (full data retrieval), the server retrieves all data, returning a total of 150,000,000 records in 13,423 milliseconds, and the calculated statistical result (i.e., average value) is approximately 1.66. By setting different sampling rates X, the number of data records is reduced, and the statistical result is determined. For example, when the sampling rate X = 0.3, the time taken is 4,635 milliseconds, returning 45,000,006 records, and the statistical result is approximately 1.64; when the sampling rate X = 0.1, the time taken is 1,621 milliseconds, returning 15,000,001 records, and the statistical result is approximately 1.64; when the sampling rate X = 0.001, the time taken is 102 milliseconds, returning 149,990 records, and the statistical result is approximately 1.64. It can be determined that the average value determined by sampling has a very small error compared to the average value calculated directly for the full data. That is, while ensuring that the avg operator meets customer needs, the analysis time is significantly reduced as the sampling rate is increased, thus improving the availability of the entire query system.

[0217] This application embodiment, through a retrieval sampling method, can significantly reduce the number of documents returned during the retrieval stage, thereby reducing the number of documents involved in query analysis. While ensuring the accuracy of customer query results, it greatly reduces the occupation of system resources and the time spent on query analysis. At the same time, 1. the user query analysis time is reduced, improving the product experience; system resource occupation is reduced, providing more query concurrency capabilities; and system availability is effectively improved.

[0218] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0219] The following description continues to illustrate the exemplary structure of the data retrieval device 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the live interactive device 555 in the memory 540 may include:

[0220] The acquisition module 5551 is used to acquire source data and the sampling rate corresponding to the source data in response to a data retrieval request carrying retrieval information; wherein the source data is divided into at least two data units for data storage;

[0221] The determining module 5552 is used to determine the unit sampling rate corresponding to the at least two data units and the data sampling rate corresponding to the data within the data units based on the sampling rate corresponding to the source data;

[0222] The sampling module 5553 is used to combine the unit sampling rate, the data sampling rate and the retrieval information to sample the source data to obtain the target data.

[0223] The return module 5554 is used to return the search results including the target data.

[0224] In some embodiments, the acquisition module is further configured to parse the data retrieval request to determine the sampling rate mode for the source data; when the sampling rate mode is a specified sampling rate mode, the sampling rate carried in the data retrieval request is used as the sampling rate corresponding to the source data.

[0225] In some embodiments, the acquisition module is further configured to parse the data retrieval request to determine the sampling rate mode for the source data; when the sampling rate mode is the intelligent sampling rate mode, to acquire the estimated data volume corresponding to the source data and the processing volume threshold when processing the source data; and to determine the sampling rate corresponding to the source data by combining the processing volume threshold and the estimated data volume.

[0226] In some embodiments, when the data unit is a data shard, the determining module is further configured to obtain the number of shards corresponding to the source data; combine the sampling rate corresponding to the source data and the number of shards to determine the shard sampling rate corresponding to the at least two data shards as the unit sampling rate, and use the ratio of the sampling rate corresponding to the source data to the shard sampling rate as the data sampling rate of the data within the corresponding data shard.

[0227] In some embodiments, when the data unit is a data block, the source data is divided into at least two data shards, and each data shard is divided into at least two data blocks. The determining module is further configured to obtain the number of shards corresponding to the source data, and determine the shard sampling rate corresponding to the at least two data shards by combining the sampling rate corresponding to the source data and the number of shards; obtain a sampling rate threshold for the at least two data blocks; determine an intermediate sampling rate for the at least two data blocks by combining the sampling rate corresponding to the source data and the shard sampling rate; and determine the unit sampling rate corresponding to the at least two data units and the data sampling rate corresponding to the data within the data unit based on the sampling rate threshold and the intermediate sampling rate.

[0228] In some embodiments, the determining module is further configured to determine the product of the sampling rate corresponding to the source data and the number of fragments; round up the product to obtain the number of sampling fragments; determine the ratio of the number of sampling fragments to the number of fragments, and use the ratio as the fragment sampling rate corresponding to the at least two data fragments.

[0229] In some embodiments, the sampling module is further configured to sample the at least two data units based on the unit sampling rate to obtain at least one target data unit; sample the data in each of the target data units based on the data sampling rate to obtain sampled data; and perform data retrieval in the sampled data based on the retrieval information to obtain the target data.

[0230] In some embodiments, the sampling module is further configured to determine a data sampling operator corresponding to the data within the data unit based on the data sampling rate; and to sample the data within the data unit based on the data sampling operator to obtain sampled data; wherein the ratio of the amount of sampled data to the amount of data within the data unit is equal to the data sampling rate.

[0231] In some embodiments, when the data sampling operator is a modulo operation, the sampling module is further configured to obtain the index value corresponding to each data in the data unit; perform modulo operation on each index value to obtain the modulo value corresponding to each data; and when the modulo value matches a preset modulo value, use the data indicated by the corresponding index value as the sampling data.

[0232] In some embodiments, when the data sampling operator is a hash operation, the sampling module is further configured to obtain the index value corresponding to each data in the data unit; hash the index value to obtain the hash value corresponding to each data; and when the hash value does not reach the hash value threshold, use the data indicated by the corresponding index value as the sampling data.

[0233] In some embodiments, the retrieval information includes a retrieval operator and a query statement. The sampling module is further configured to combine the unit sampling rate, the data sampling rate, and the retrieval operator to perform data sampling on the source data to obtain initial sampled data; and to perform data retrieval on the initial sampled data based on the query statement to obtain the target data.

[0234] In some embodiments, when the search results also include statistical results corresponding to the target data, the return module is further configured to obtain the data statistical method corresponding to the target data; and perform statistical analysis on the target data based on the data statistical method to obtain the statistical results.

[0235] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data retrieval method described in this application.

[0236] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the data retrieval method provided in this application, for example... Figure 3 The data retrieval method is shown.

[0237] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0238] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0239] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0240] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0241] In summary, the embodiments of this application sample massive amounts of data, reducing the amount of data involved in the analysis and effectively avoiding resource and response timeout issues encountered during the retrieval and analysis of massive amounts of data. Simultaneously, during the data sampling process, shard-level sampling technology reduces the amount of I / O, and block-level sampling technology reduces I / O amplification, achieving highly efficient data sampling and retrieval.

[0242] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data retrieval method, characterized in that, The method includes: In response to a data retrieval request carrying retrieval information, the source data and the sampling rate corresponding to the source data are obtained; The source data is divided into at least two data units for data storage; when the data unit is a data block, the source data is divided into at least two data shards, and each data shard is divided into at least two data blocks. Obtain the number of data fragments corresponding to the source data, and combine the sampling rate corresponding to the source data and the number of fragments to determine the fragment sampling rate corresponding to the at least two data fragments; Obtain the sampling rate threshold for the at least two data blocks; By combining the sampling rate corresponding to the source data and the segmented sampling rate, an intermediate sampling rate is determined for the at least two data blocks; Based on the sampling rate threshold and the intermediate sampling rate, determine the unit sampling rate corresponding to the at least two data units and the data sampling rate corresponding to the data within the data unit; By combining the unit sampling rate, the data sampling rate, and the retrieval information, the source data is sampled to obtain the target data. Returns the search results including the target data.

2. The method as described in claim 1, characterized in that, The step of obtaining the sampling rate corresponding to the source data includes: Parse the data retrieval request to determine the sampling rate pattern for the source data; When the sampling rate mode is a specified sampling rate mode, the sampling rate carried in the data retrieval request is used as the sampling rate corresponding to the source data.

3. The method as described in claim 1, characterized in that, The step of obtaining the sampling rate corresponding to the source data includes: Parse the data retrieval request to determine the sampling rate pattern for the source data; When the sampling rate mode is the intelligent sampling rate mode, the estimated data volume corresponding to the source data and the processing volume threshold when processing the source data are obtained. By combining the processing volume threshold and the estimated data volume, the sampling rate corresponding to the source data is determined.

4. The method as described in claim 1, characterized in that, The method further includes: When the data unit is a data fragment, obtain the number of fragments corresponding to the source data; By combining the sampling rate corresponding to the source data and the number of fragments, the fragment sampling rate corresponding to the at least two data fragments is determined as the unit sampling rate, and the ratio of the sampling rate corresponding to the source data to the fragment sampling rate is taken as the data sampling rate of the data within the corresponding data fragment.

5. The method as described in claim 4, characterized in that, The step of determining the shard sampling rate corresponding to the at least two data shards by combining the sampling rate corresponding to the source data and the number of shards includes: Determine the product of the sampling rate corresponding to the source data and the number of fragments; The product is rounded up to obtain the number of sampling segments; Determine the ratio of the number of sampling fragments to the number of fragments, and use the ratio as the fragment sampling rate corresponding to the at least two data fragments.

6. The method as described in claim 1, characterized in that, The step of combining the unit sampling rate, the data sampling rate, and the retrieval information to sample the source data to obtain the target data includes: Based on the unit sampling rate, at least two data units are sampled to obtain at least one target data unit; Based on the data sampling rate, the data in each of the target data units is sampled to obtain sampled data; Based on the retrieval information, data retrieval is performed on the sampled data to obtain the target data.

7. The method as described in claim 6, characterized in that, The step of sampling data in each target data unit based on the data sampling rate to obtain sampled data includes: Based on the data sampling rate, determine the data sampling operator corresponding to the data within the data unit; Based on the data sampling operator, the data within the data unit is sampled to obtain sampled data; The ratio of the amount of sampled data to the amount of data within the data unit is equal to the data sampling rate.

8. The method as described in claim 7, characterized in that, When the data sampling operator is a modulo operation, the step of sampling the data within the data unit based on the data sampling operator to obtain sampled data includes: Obtain the index value corresponding to each data in the data unit; Take the modulo of each index value to obtain the modulo value of each data. When the modulus value matches the preset modulus value, the data indicated by the corresponding index value is used as the sampled data.

9. The method as described in claim 7, characterized in that, When the data sampling operator is a hash operation, the step of sampling the data within the data unit based on the data sampling operator to obtain sampled data includes: Obtain the index value corresponding to each data in the data unit; Hash the index value to obtain the hash value of each of the data; When the hash value does not reach the hash value threshold, the data indicated by the corresponding index value is used as the sampling data.

10. The method as described in claim 1, characterized in that, The retrieval information includes retrieval operators and query statements. The step of combining the unit sampling rate, the data sampling rate, and the retrieval information to sample the source data and obtain the target data includes: By combining the unit sampling rate, the data sampling rate, and the retrieval operator, the source data is sampled to obtain initial sampled data; Based on the query statement, data retrieval is performed on the initial sampled data to obtain the target data.

11. The method as described in claim 1, characterized in that, When the search results also include statistical results corresponding to the target data, the method further includes: Obtain the data statistics method corresponding to the target data; Based on the aforementioned data statistics method, statistical analysis is performed on the target data to obtain the statistical results.

12. A data retrieval device, characterized in that, The device includes: The acquisition module is used to acquire source data and the sampling rate corresponding to the source data in response to a data retrieval request carrying retrieval information; wherein the source data is divided into at least two data units for data storage; The determination module is used to divide the source data into at least two data fragments when the data unit is a data block, and each data fragment is divided into at least two data blocks; Obtain the number of data fragments corresponding to the source data, and combine the sampling rate corresponding to the source data and the number of fragments to determine the fragment sampling rate corresponding to the at least two data fragments; Obtain the sampling rate threshold for the at least two data blocks; By combining the sampling rate corresponding to the source data and the segmented sampling rate, an intermediate sampling rate is determined for the at least two data blocks; Based on the sampling rate threshold and the intermediate sampling rate, determine the unit sampling rate corresponding to the at least two data units and the data sampling rate corresponding to the data within the data unit; A sampling module is used to combine the unit sampling rate, the data sampling rate, and the retrieval information to sample the source data and obtain the target data. The return module is used to return the search results including the target data.

13. The apparatus according to claim 12, characterized in that, The acquisition module is specifically used for: Parse the data retrieval request to determine the sampling rate pattern for the source data; When the sampling rate mode is a specified sampling rate mode, the sampling rate carried in the data retrieval request is used as the sampling rate corresponding to the source data.

14. The apparatus according to claim 12, characterized in that, The acquisition module is specifically used for: Parse the data retrieval request to determine the sampling rate pattern for the source data; When the sampling rate mode is the intelligent sampling rate mode, the estimated data volume corresponding to the source data and the processing volume threshold when processing the source data are obtained. By combining the processing volume threshold and the estimated data volume, the sampling rate corresponding to the source data is determined.

15. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data retrieval method according to any one of claims 1 to 11.

16. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the data retrieval method according to any one of claims 1 to 11.

17. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the data retrieval method according to any one of claims 1 to 11.

Images