Data query method and device, storage medium and electronic equipment

Abstract

The invention discloses a data query method and device, a storage medium and electronic equipment, a server can map large-scale historical data into a bit array through multiple rounds of data mapping calculation, so that whether real-time data is newly-added data or not can be queried based on the bit array, and due to the fact that specific values of the historical data are not stored in the bit array, the data query efficiency is improved. According to the method, each historical data is mapped with each data value contained in the bit array, so that whether each historical data is newly-added historical data or not can be represented through simple data values contained in the bit array, occupation of storage resources for data query is reduced, the data query efficiency is improved, and the data query efficiency is improved. And the processing requirement of real-time data is met.

Description

Technical Field

[0001] This specification relates to the field of big data processing technology, and in particular to a data query method, device, storage medium, and electronic device. Background Art

[0002] With the rapid development of information technology, big data has become a core force driving the digital transformation of various industries. In fields such as advertising, finance, and e-commerce, a company's ability to process billions of data in real time is closely linked to its market competitiveness. For example, in key processes such as advertising delivery, user behavior analysis, and risk control, the efficiency of real-time deduplication and judgment of the entire historical data is particularly important to the company's core competitiveness.

[0003] Typically, real-time deduplication of all historical data requires database queries. However, when processing billions of data points, database query methods may suffer from poor query efficiency and concurrency, leading to data delays in high-concurrency scenarios. This significantly impacts the real-time nature of the business.

[0004] Therefore, how to improve the efficiency of real-time deduplication judgment of large-scale data is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a data query method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a data query method, including:

[0008] Get real-time data;

[0009] Processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determining a data value at a position corresponding to the mapping value in a pre-generated bit array, wherein the data values ​​in the bit array are obtained by mapping the historical data in advance using the data mapping algorithm. For any data, the mapping value corresponding to the data is used to indicate the position corresponding to the data in the pre-generated bit array;

[0010] A query result for the real-time data is determined according to the data value, and a task is executed according to the query result, where the query result is used to indicate whether the real-time data is newly added data.

[0011] Optionally, before the step of acquiring real-time data, the method further includes:

[0012] Get the access key of the preset file system;

[0013] A data acquisition request is generated according to the access key, and the data acquisition request is sent to the file system to obtain initialization data from the file system, and a bit array is determined based on the initialization data, wherein the initialization data includes historical data for constructing the bit array or the bit array saved by the file system.

[0014] Optionally, the step of processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data specifically includes:

[0015] In the case where there are multiple preset data mapping algorithms, the real-time data is processed in sequence by each data mapping algorithm according to the order in which each data mapping algorithm is used in the data processing process to obtain a mapping value corresponding to the real-time data.

[0016] Optionally, the step of generating a bit array specifically includes:

[0017] Get each historical data and initial bit array;

[0018] For each historical data, the historical data is processed according to a preset data mapping algorithm to obtain a basic mapping value corresponding to the historical data, and the data value at the corresponding position of the basic mapping value in the initial bit array is adjusted to obtain an adjusted bit array.

[0019] Optionally, the step of obtaining each historical data and the initial bit array specifically includes:

[0020] Obtain the expected false alarm rate for each historical data and user input;

[0021] determining an expected number of data values ​​included in the initial bit array based on each historical data, the expected false alarm rate, and a specified association relationship, wherein the specified association relationship is used to represent an association between the expected number of data values ​​included in the to-be-generated bit array, the total number of each historical data, and the expected false alarm rate;

[0022] An initial bit array is generated according to an expected number of data values ​​contained in the initial bit array.

[0023] Optionally, the step of processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determining a data value at a position corresponding to the mapping value in a pre-generated bit array specifically includes:

[0024] Determining a data query dimension corresponding to the real-time data;

[0025] According to the data query dimension, a bit array matching the data query dimension is determined from the pre-generated bit arrays as a target bit array;

[0026] The real-time data is processed according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and a data value at a position corresponding to the mapping value in the target bit array is determined.

[0027] Optionally, the step of processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data specifically includes:

[0028] When multiple real-time data are obtained, determining the data generation time of each real-time data;

[0029] According to the data generation time of each real-time data, each real-time data is processed in turn according to the preset data mapping algorithm to obtain the mapping value corresponding to each real-time data.

[0030] This specification provides a data query device, including:

[0031] Acquisition module, used to obtain real-time data;

[0032] a mapping module, configured to process the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determine a data value at a position corresponding to the mapping value in a pre-generated bit array, wherein the data values ​​in the bit array are obtained by mapping the historical data in advance using the data mapping algorithm. For any data, the mapping value corresponding to the data is used to indicate the corresponding position of the data in the pre-generated bit array;

[0033] The query module is used to determine a query result for the real-time data according to the data value, and perform a task according to the query result, wherein the query result is used to indicate whether the real-time data is newly added data.

[0034] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0035] In the data query method provided in this specification, real-time data is obtained, and the real-time data is processed according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and the data value at the corresponding position of the mapping value in the pre-generated bit array is determined, wherein the data value in the bit array is obtained by mapping the historical data in advance through the data mapping algorithm. For any data, the mapping value corresponding to the data is used to represent the corresponding position of the data in the pre-generated bit array, and then the query result for the real-time data is determined according to the data value, and the task is executed according to the query result. The query result here is used to indicate whether the real-time data is new data.

[0036] It can be seen from the above method that the server can map large-scale historical data into a bit array through multiple rounds of data mapping calculations, so that it can query whether the real-time data is new data based on the bit array. Since the bit array does not store the specific value of the historical data, it maps each historical data with each data value contained in the bit array. It can then use the simple data values ​​contained in the bit array to characterize whether each historical data is new historical data, thereby reducing the storage resources occupied by data queries and improving the efficiency of data deduplication judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0038] Figure 1 A flowchart of a data query method provided in this specification;

[0039] Figure 2 This is a schematic diagram of the process of obtaining real-time data provided in this manual;

[0040] Figure 3 A schematic diagram of the query process of real-time data provided in this manual;

[0041] Figure 4 A schematic diagram of a data query device provided in this specification;

[0042] Figure 5 This manual provides a corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0044] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0045] Figure 1 The following is a flowchart of a data query method provided in this specification, which includes the following steps:

[0046] S101: Acquire real-time data.

[0047] In this specification, the business platform can obtain the access key of the preset file system, and then generate a data acquisition request based on the obtained access key, and send the generated data acquisition request to the preset file system to obtain initialization data from the file system, and then determine the bit array based on the obtained initialization data.

[0048] The file system mentioned above may be a system for persistently storing a large number of files (such as image data, video data, document data, etc.), such as an Object Storage Service (OSS) system, a Distributed File System (HDFS), etc.

[0049] The access keys mentioned above may include: an Access Key (AK) for identifying an identity and a Secret Key (SK) for verifying an identity.

[0050] The aforementioned initialization data includes historical data used to construct a bit array or a bit array stored in a file system (ie, a bit array required for subsequent data query is generated in advance and stored in a file system).

[0051] From the above content, it can be seen that the business platform can obtain two types of initialization data from the file system. When the above initialization data is historical data used to construct a bit array, the business platform can construct the bit array used in the subsequent data query process based on the historical data obtained from the file system for constructing the bit array. When the above initialization data is the bit array saved by the file system, the bit array saved by the file system can be directly used as the bit array used in the subsequent data query process.

[0052] Furthermore, after the business platform completes the above initialization operation, it can obtain real-time data sent by the terminal device used by the user, and query whether the real-time data sent by the user is new data, obtain the query results, and then perform tasks based on the obtained query results.

[0053] The above-mentioned real-time data may refer to data generated at an extremely fast speed and requiring immediate processing and response. The above-mentioned real-time data may be determined based on the actual business scenario. For example, in an advertising push business scenario, the above-mentioned real-time data may be the identification data of users who click on a specified advertisement. The business platform may then be required to query the identification data of different users that are continuously generated to determine whether the identification data of each continuously generated user is a new user (i.e., a user who has never clicked on the specified advertisement before) or a user who has clicked repeatedly.

[0054] Another example: in an e-commerce business scenario, the above-mentioned real-time data can be the fused data obtained by splicing the identification information of the user who visits a store and the identification information of this store. The business platform can then be required to query the continuously generated fused data to determine whether the fused data is the query result of new data. Based on whether the fused data is the query result of new data, it can be determined whether the user corresponding to the user's identification information in the fused data is a new user of the store corresponding to the identification information of the store in the fused data.

[0055] It should be noted that there are many ways for the service platform to obtain the real-time data sent by the user. For example, the service platform directly receives the real-time data sent by the client used by the user.

[0056] For another example: the business platform can receive real-time data sent by the terminal device used by the user through a preset message queue component. Specifically, the above-mentioned message queue component includes at least one queue for storing messages. The above-mentioned message queue component can receive real-time data sent by the terminal device used by the user through each queue, and convert the received real-time data into a byte stream, that is, a real-time data stream, and then transmit the real-time data stream to the business platform, so that the business platform can determine each real-time data based on the real-time data stream. The above-mentioned message queue component can be, for example: a distributed messaging system (Apache Kafka, Kafka), a rabbit message queue (RabbitMQ), etc.

[0057] In this specification, the execution entity used to implement the data query method can refer to a designated device such as a server set up on a business platform, or it can refer to a terminal device such as a desktop computer or a laptop computer. For the sake of convenience of description, the data query method provided in this specification is explained below using the server as the execution entity as an example.

[0058] The server for data management provided by the aforementioned business platform may refer to a separate server provided in the business platform, or may refer to a distributed server cluster provided in the business platform, or may refer to a cloud server.

[0059] The above describes in detail the method for the business platform to obtain real-time data. The following fully describes the specific process of the server set by the business platform for data management to obtain real-time data, such as Figure 2 shown.

[0060] Figure 2 This is a schematic diagram of the real-time data acquisition process provided in this manual.

[0061] Combine Figure 2 It can be seen that the server can obtain the access key of the preset file system through the distributed stream processing engine (Apache Flink, Flink) preset in the server, and then generate a data acquisition request based on the obtained access key, and send the generated data acquisition request to the preset file system to obtain initialization data from the file system, and then broadcast the initialization data to the parallel instances (also called parallel threads) of all preset processing tasks, so that each parallel instance determines the bit array used in the subsequent data query process.

[0062] The parallel instance mentioned above can be the basic task execution unit in Flink. The data query method mentioned above can be executed in parallel through the parallel instance, thereby improving the data query efficiency.

[0063] Furthermore, the server can obtain real-time data streams from the message queue component Kafka through Flink to obtain various real-time data, and then use the data processing module Redisson to determine whether the real-time data is a query result of new data for each real-time data.

[0064] S102: Process the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determine the data value of the mapping value at a corresponding position in a pre-generated bit array. The data value in the bit array is obtained by mapping the historical data in advance through the data mapping algorithm. For any data, the mapping value corresponding to the data is used to represent the corresponding position of the data in the pre-generated bit array.

[0065] Furthermore, after obtaining the real-time data, the server can process the obtained real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determine the data value of the mapping value corresponding to the real-time data at the corresponding position in the pre-generated bit array.

[0066] The data values ​​in the above-mentioned bit array are obtained by mapping the historical data in advance through a data mapping algorithm. For any data, the mapping value corresponding to the data is used to represent the corresponding position of the data in the pre-generated bit array.

[0067] It should be noted that a bit array contains multiple data values, wherein each data value corresponds to a mapping value, which is used to represent the order of each data value in the bit array, that is, to represent the position of each data value in the bit array, and the initial value of each data value in the bit array is 0. When it is necessary to map historical data into the bit array through a data mapping algorithm, the mapping value corresponding to each historical data can be determined through the data mapping algorithm, so that the data value of the mapping value corresponding to the historical data at the corresponding position in the bit array can be adjusted to 1. Then, when performing data query, the mapping value corresponding to the real-time data to be queried can be determined through the data mapping algorithm. If the real-time data to be queried is not new data, it means that the data value of the mapping value corresponding to the real-time data to be queried at the corresponding position in the bit array has been adjusted to 1. Therefore, when it is determined that the data value of the mapping value corresponding to the real-time data to be queried at the corresponding position in the bit array is 1, it can be determined that the real-time data to be queried is not new data.

[0068] On the contrary, when it is determined that the data value of the mapping value corresponding to the real-time data to be queried at the corresponding position in the bit array is 0, it means that the data value of the mapping value corresponding to the real-time data to be queried at the corresponding position in the bit array has not been adjusted to 1 before, that is, the real-time data to be queried is not historical data, but new data.

[0069] Specifically, the server can obtain various historical data and the initial bit array, and process each historical data according to a preset data mapping algorithm to obtain the basic mapping value corresponding to the historical data, and adjust the data value at the corresponding position of the basic mapping value in the initial bit array to obtain the adjusted bit array as the bit array used in subsequent data query processes.

[0070] For example: assuming that the size of the above-mentioned bit array is 10 (i.e., there are 10 data values ​​in the bit array), the above-mentioned data mapping algorithm is to take the remainder of the data to be processed to 10 as the mapping value of the data to be processed. At this time, for the historical data with a value of 5, the remainder algorithm can be used to determine that the mapping value of this historical data with a value of 5 is 5, that is, the 5th data value in the bit array can be adjusted from 0 to 1. When performing data query, if the value of the real-time data is 3, the remainder algorithm can be used to determine that the mapping value of this real-time data is 3, and the 3rd data value in the bit array is 0. At this time, it can be said that the real-time data with a value of 3 is new data rather than historical data. If the value of the real-time data is 5, the remainder algorithm can be used to determine that the mapping value of this real-time data is 5, and the 5th data value in the bit array is 1. At this time, it can be said that the real-time data with a value of 5 is historical data rather than new data.

[0071] From the above content, it can be seen that there may be false alarms when querying whether the real-time data is new data through the above method. For example, when the real-time data is 15, the mapping value determined by the remainder algorithm is also 5. At this time, the real-time data with a value of 15 will be determined as historical data, resulting in a false alarm. In order to avoid such false alarms, the server can use multiple data mapping algorithms.

[0072] Furthermore, when the server may use multiple data mapping algorithms, the server may process the real-time data in sequence through each data mapping algorithm according to the order in which the data mapping algorithms are used in the data processing process to obtain the mapping value corresponding to the real-time data.

[0073] For example: for historical data with a value of 5, first use the square median method to determine the mapping value of the historical data in this processing, and then use the remainder algorithm to determine the mapping value of the historical data in the previous processing as the final mapping value of the historical data with a value of 5.

[0074] The data mapping algorithm in the above content may be, for example, a message digest algorithm (Message-Digest Algorithm 5, MD5), a cryptographic hash function (Blowfish Cryptography Hash, bcrypt), a reversible hash function (Advanced Encryption Standard, AES), and the like.

[0075] In addition, it can be seen from the above content that the possibility of false alarms when querying whether real-time data is new data through the above method is not only related to the data mapping algorithm adopted, but also related to the size of the bit array. That is, the larger the size of the bit array used, the lower the probability of false alarms. However, the larger the bit array used, the more memory it occupies.

[0076] Therefore, in this specification, the method for the server to map historical data to obtain a bit array through a data mapping algorithm can be to obtain the expected false alarm rate of each historical data and user input, and then determine the expected number of data values ​​contained in the initial bit array based on the obtained historical data, the expected false alarm rate of user input and the specified association relationship, so that the initial bit array can be generated based on the expected number of data values ​​contained in the determined initial bit array, and then each historical data can be processed according to the preset data mapping algorithm to obtain the basic mapping value corresponding to the historical data, and the data value at the corresponding position of the basic mapping value in the initial bit array can be adjusted to obtain the adjusted bit array.

[0077] The above-mentioned specified association relationship is used to express the association between the expected number of data values ​​contained in the bit array to be generated, the total number of historical data, and the expected false alarm rate. In actual applications, there can be multiple formulas for representing the above-mentioned specified association relationship. The following only uses one of the multiple formulas for representing the above-mentioned specified association relationship as an example for explanation, and other formulas for representing the above-mentioned specified association relationship are not specifically limited. The specific formulas are as follows:

[0078]

[0079] In the above formula, m is the expected number of data values ​​contained in the bit array to be generated, n is the total number of historical data, and p is the expected false alarm rate.

[0080] In addition, in actual application scenarios, the real-time data received by the server may be of different data query dimensions, and at this time the server needs to query the real-time data of different data query dimensions.

[0081] Based on this, the above-mentioned bit arrays can be multiple, wherein for each data query dimension, there is a bit array corresponding to the data query dimension. At this time, after obtaining the real-time data, the server can determine the data query dimension corresponding to the real-time data, and according to the data query dimension corresponding to the real-time data, determine the bit array that matches the data query dimension corresponding to the real-time data from the bit arrays contained in the pre-generated bit array set. As the target bit array, the real-time data is processed according to the preset data mapping algorithm to obtain the mapping value corresponding to the real-time data, and determine the data value of the mapping value corresponding to the real-time data at the corresponding position in the target bit array. Then, according to the data value of the mapping value corresponding to the real-time data at the corresponding position in the target bit array, the query result of the real-time data can be determined.

[0082] The aforementioned data query dimensions can refer to different aspects used to describe and organize data during a data query. For example, in an ad push business scenario, the query may be to determine whether the user's identification data in real-time data is newly added. In this case, the data query dimension is the user's identification data.

[0083] Another example: In an e-commerce business scenario, the content that needs to be queried is whether the fused data obtained by concatenating the identification information of the user who visited a certain store and the identification information of this store is new data, in order to determine whether the user who visited a certain store is a new user of this store. At this time, the data query dimensions are the user's identification information and the identification information of this store.

[0084] S103: Determine a query result for the real-time data according to the data value, and execute a task according to the query result, wherein the query result is used to indicate whether the real-time data is newly added data.

[0085] In this specification, if the server determines that the real-time data is new data based on the data value at the corresponding position in the pre-generated bit array of the mapping value corresponding to the real-time data, the data value at the corresponding position in the pre-generated bit array of the mapping value corresponding to the real-time data can be updated, and the query result corresponding to the real-time data can be displayed to the user. If it is determined that the real-time data is not new data, the query result corresponding to the real-time data can be directly displayed to the user.

[0086] In addition, the server can also send the query results corresponding to the real-time data to the file system for persistent storage, and can execute tasks based on the query results corresponding to the real-time data.

[0087] The above tasks can be determined according to specific business scenarios. For example, in an advertising delivery business scenario, the server can generate a pie chart representing the proportion of new users in a specified time period based on the query results corresponding to the real-time data generated in the specified time period, and display the pie chart representing the proportion of new users in the specified time period to the user.

[0088] For another example: In a risk control business scenario, the server can determine the identification information of users who frequently access the specified business in a short period of time based on the query results corresponding to each real-time data, and then mark the users who frequently access the specified business in a short period of time as risky users.

[0089] The above describes the data query method provided in this manual in detail. The following is a complete description of the process of the server querying whether the real-time data is new data. Figure 3 shown.

[0090] Figure 3This is a schematic diagram of the query process of real-time data provided in this manual.

[0091] Combine Figure 3 It can be seen that the server can obtain real-time data from Kafka through Flink, and then process the real-time data according to the preset data mapping algorithm for each real-time data through the data processing module Redisson to obtain the mapping value corresponding to the real-time data, and determine the data value of the mapping value corresponding to the real-time data at the corresponding position in the pre-generated bit array, and determine the query result for the real-time data based on the data value of the mapping value corresponding to the real-time data at the corresponding position in the pre-generated bit array. If it is determined that the real-time data is new data, the data value of the mapping value corresponding to the real-time data at the corresponding position in the pre-generated bit array can be updated, and the query result corresponding to the real-time data can be output. If it is determined that the real-time data is not new data, the query result corresponding to the real-time data can be directly output.

[0092] It should be noted that in order to ensure the timeliness of real-time data processing, when the server obtains multiple real-time data through Flink, it can determine the data generation time of each real-time data. Then, based on the data generation time of each real-time data, it can process each real-time data in turn according to the preset data mapping algorithm to obtain the mapping value corresponding to each real-time data.

[0093] In addition, since real-time data is continuously generated, the real-time data obtained by the server through Flink is often obtained continuously. At this time, the server can immediately query the data after obtaining each real-time data, without having to wait for all real-time data to be obtained before performing data query.

[0094] In addition, in order to ensure the timeliness of the service processing real-time data while improving the server's throughput in processing real-time data, the server can also divide part of the received real-time data into a micro-batch for processing when obtaining multiple real-time data through Flink.

[0095] From the above content, it can be seen that in fields such as advertising, finance, e-commerce, etc., real-time data will be generated frequently and on a large scale. This requires that the time interval from the generation of real-time data to the processing of real-time data and the generation of processing results must be very short. The server can map large-scale historical data into a bit array through multiple rounds of data mapping calculations, so that it can query whether the real-time data is new data based on the bit array. Since the bit array does not store the specific value of the historical data, it maps each historical data with the data values ​​contained in the bit array, and then the simple data values ​​contained in the bit array can be used to characterize whether each historical data is new historical data, so as to reduce the storage resource occupation of data query and improve the efficiency of data query to meet the processing requirements of real-time data.

[0096] The above is one or more implementation data query methods of this specification. Based on the same idea, this specification also provides a corresponding data query device, such as Figure 4 shown.

[0097] Figure 4 A schematic diagram of a data query device provided in this specification, including:

[0098] Acquisition module 401, used to acquire real-time data;

[0099] A mapping module 402 is configured to process the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determine the data value at the corresponding position in a pre-generated bit array of the mapping value. The data values ​​in the bit array are obtained by mapping the historical data in advance using the data mapping algorithm. For any data, the mapping value corresponding to the data is used to indicate the corresponding position of the data in the pre-generated bit array.

[0100] The query module 403 is used to determine a query result for the real-time data according to the data value, and perform a task according to the query result, wherein the query result is used to indicate whether the real-time data is newly added data.

[0101] Optionally, the acquisition module 401 is specifically used to obtain an access key of a preset file system; generate a data acquisition request based on the access key, and send the data acquisition request to the file system to obtain initialization data from the file system, and determine a bit array based on the initialization data, wherein the initialization data includes historical data used to construct a bit array or a bit array saved by the file system.

[0102] Optionally, the mapping module 402 is specifically used to, when there are multiple preset data mapping algorithms, process the real-time data in sequence through each data mapping algorithm according to the order in which each data mapping algorithm is used in the data processing process to obtain the mapping value corresponding to the real-time data.

[0103] Optionally, the apparatus further includes: a generating module 404;

[0104] The generation module 404 is specifically used to obtain various historical data and an initial bit array; for each historical data, the historical data is processed according to a preset data mapping algorithm to obtain a basic mapping value corresponding to the historical data, and the data value at the corresponding position of the basic mapping value in the initial bit array is adjusted to obtain an adjusted bit array.

[0105] Optionally, the generation module 404 is specifically used to obtain each historical data and the expected false alarm rate of user input; determine the expected number of data values ​​contained in the initial bit array based on each historical data, the expected false alarm rate and a specified association relationship, and the specified association relationship is used to represent the association between the expected number of data values ​​contained in the bit array to be generated, the total number of each historical data and the expected false alarm rate; generate the initial bit array based on the expected number of data values ​​contained in the initial bit array.

[0106] Optionally, the mapping module 402 is specifically used to determine the data query dimension corresponding to the real-time data; based on the data query dimension, determine the bit array that matches the data query dimension from the pre-generated bit arrays as the target bit array; process the real-time data according to the preset data mapping algorithm to obtain the mapping value corresponding to the real-time data, and determine the data value of the mapping value at the corresponding position in the target bit array.

[0107] Optionally, the mapping module 402 is specifically used to, when multiple real-time data are obtained, determine the data generation time of each real-time data; according to the data generation time of each real-time data, process each real-time data in turn according to a preset data mapping algorithm to obtain a mapping value corresponding to each real-time data.

[0108] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a data query method.

[0109] This manual also provides Figure 5 The one shown corresponds to Figure 1 Schematic diagram of the electronic equipment. Figure 5As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0110] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always program the improved process flow into the hardware circuit to obtain the corresponding hardware circuit structure. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0111] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0112] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0113] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0114] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0118] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0119] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0120] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0122] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0124] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0125] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A data query method, characterized in that: include: Get real-time data; Processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determining a data value at a position corresponding to the mapping value in a pre-generated bit array, wherein the data values ​​in the bit array are obtained by mapping the historical data in advance using the data mapping algorithm. For any data, the mapping value corresponding to the data is used to indicate the position corresponding to the data in the pre-generated bit array; A query result for the real-time data is determined according to the data value, and a task is executed according to the query result, where the query result is used to indicate whether the real-time data is newly added data.

2. The method according to claim 1, wherein Before the step of acquiring real-time data, the method further includes: Get the access key of the preset file system; A data acquisition request is generated according to the access key, and the data acquisition request is sent to the file system to obtain initialization data from the file system, and a bit array is determined based on the initialization data, wherein the initialization data includes historical data for constructing the bit array or the bit array saved by the file system.

3. The method according to claim 1, wherein The step of processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data specifically includes: In the case where there are multiple preset data mapping algorithms, the real-time data is processed in sequence by each data mapping algorithm according to the order in which each data mapping algorithm is used in the data processing process to obtain a mapping value corresponding to the real-time data.

4. The method according to claim 1, wherein The step of generating a bit array specifically includes: Get each historical data and initial bit array; For each historical data, the historical data is processed according to a preset data mapping algorithm to obtain a basic mapping value corresponding to the historical data, and the data value at the corresponding position of the basic mapping value in the initial bit array is adjusted to obtain an adjusted bit array.

5. The method according to claim 4, wherein The step of obtaining each historical data and the initial bit array specifically includes: Obtain the expected false alarm rate for each historical data and user input; determining an expected number of data values ​​included in the initial bit array based on each historical data, the expected false alarm rate, and a specified association relationship, wherein the specified association relationship is used to represent an association between the expected number of data values ​​included in the to-be-generated bit array, the total number of each historical data, and the expected false alarm rate; An initial bit array is generated according to an expected number of data values ​​contained in the initial bit array.

6. The method according to claim 1, wherein The step of processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determining a data value at a position corresponding to the mapping value in a pre-generated bit array specifically includes: Determining a data query dimension corresponding to the real-time data; According to the data query dimension, a bit array matching the data query dimension is determined from the pre-generated bit arrays as a target bit array; The real-time data is processed according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and a data value at a position corresponding to the mapping value in the target bit array is determined.

7. The method according to claim 1, wherein The step of processing the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data specifically includes: When multiple real-time data are obtained, determining the data generation time of each real-time data; According to the data generation time of each real-time data, each real-time data is processed in turn according to the preset data mapping algorithm to obtain the mapping value corresponding to each real-time data.

8. A data query device, characterized in that: include: Acquisition module, used to obtain real-time data; a mapping module, configured to process the real-time data according to a preset data mapping algorithm to obtain a mapping value corresponding to the real-time data, and determine a data value at a position corresponding to the mapping value in a pre-generated bit array, wherein the data values ​​in the bit array are obtained by mapping the historical data in advance using the data mapping algorithm. For any data, the mapping value corresponding to the data is used to indicate the corresponding position of the data in the pre-generated bit array; The query module is used to determine a query result for the real-time data according to the data value, and perform a task according to the query result, wherein the query result is used to indicate whether the real-time data is newly added data.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images