Data query method, device, electronic device and readable medium

Abstract

The embodiment of the present invention provides a data query method, device, electronic device and readable medium, the method comprising: receiving multiple query tasks sent by a feature caller; determining target feature configuration data from preset feature configuration data according to the query tasks; the target feature configuration data includes target semantic configuration data and target query configuration data; calculating the target feature semantics according to the target semantic configuration data; merging query tasks with the same target feature semantics into a merged query task; acquiring offline feature data matching the target feature semantics of the merged query task from a preset offline feature library as matching offline feature data; determining the target offline feature data corresponding to each query task from the matching offline feature data according to the target query configuration data corresponding to each query task; and returning the target offline feature data to the feature caller corresponding to the query task. System resources are saved and query efficiency is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a data query method, a data query device, an electronic device, and a computer-readable medium. Background Art

[0002] When training an offline model or algorithm, a large amount of offline feature data is required as samples. The usual method is to backtrack the offline feature data, that is, to query and obtain the target offline feature data from the offline feature library according to the needs.

[0003] Specifically, when backtracking offline feature data, there are often a large number of query tasks at the same time. Each query task has feature configuration data configured for the offline feature data. The current offline feature data backtracking system does not analyze and distinguish the feature configuration data when executing query tasks, but queries each feature configuration data as a separate task. The same data in the feature configuration data will be queried repeatedly, resulting in a waste of system resources and low query efficiency. Summary of the invention

[0004] The embodiments of the present invention provide a data query method, device, electronic device and computer-readable storage medium to solve the problem of excessive resource consumption and low efficiency of offline feature data query in the prior art.

[0005] In order to solve the above problem, an embodiment of the present invention discloses a data query method, which includes:

[0006] Receive multiple query tasks sent by the feature caller;

[0007] Determining target feature configuration data from preset feature configuration data according to the query task; the target feature configuration data includes target semantic configuration data and target query configuration data;

[0008] Calculate the target feature semantics according to the target semantic configuration data;

[0009] Merging the query tasks with the same semantics of the target features into a merged query task;

[0010] Acquire offline feature data semantically matching the target feature of the combined query task from a preset offline feature library as matching offline feature data;

[0011] According to the target query configuration data corresponding to each query task, determining the target offline feature data corresponding to each query task from the matching offline feature data;

[0012] The target offline feature data is returned to the feature caller corresponding to the query task.

[0013] Optionally, the offline feature data has attribute data, the attribute data has corresponding attribute items, and the attribute items include data source, filtering conditions, screening range, time range and data indicators.

[0014] Optionally, before receiving the query task sent by the feature caller, the method further includes:

[0015] Receive a submitted configuration request; the configuration request includes feature configuration data; the feature configuration data includes semantic configuration data, query configuration data and calculation method;

[0016] Determining target attribute data from attribute data of the offline feature data according to the semantic configuration data;

[0017] The feature semantics is calculated based on the target attribute data using a hash algorithm;

[0018] The offline feature data is stored in an offline feature library based on the corresponding feature semantics.

[0019] Optionally, the attribute items of the semantic configuration data include data source, filtering condition and filtering range.

[0020] Optionally, storing the offline feature data in an offline feature library based on the corresponding feature semantics includes:

[0021] The offline feature data with the same feature semantics are merged and stored in the offline feature library.

[0022] Optionally, the merging the offline feature data with the same feature semantics and storing them in the offline feature library includes:

[0023] The offline feature data with the same feature semantics and the same time range are merged and stored in the offline feature library.

[0024] Optionally, the query configuration data includes a time range and a data indicator.

[0025] Optionally, returning the target offline feature data to the feature caller corresponding to the query task includes:

[0026] Calculating the target offline feature data according to the target calculation method in the target feature configuration data to obtain aggregated index data;

[0027] The aggregated indicator data is returned to the feature caller corresponding to the query task.

[0028] Optionally, the calculation method includes at least one of total calculation, sum calculation, average value calculation and de-duplication counting.

[0029] The embodiment of the present invention further discloses a data query device, which includes:

[0030] A query task receiving module, used for receiving multiple query tasks sent by the feature caller;

[0031] A target feature configuration data determination module, used to determine target feature configuration data from preset feature configuration data according to the query task; the target feature configuration data includes target semantic configuration data and target query configuration data;

[0032] A target feature semantics calculation module, used to calculate the target feature semantics according to the target semantics configuration data;

[0033] A query task merging module, used to merge query tasks with the same target feature semantics into a merged query task;

[0034] A matching offline feature data acquisition module is used to acquire offline feature data that semantically matches the target feature of the merged query task from a preset offline feature library as matching offline feature data;

[0035] A target offline feature data determination module, configured to determine the target offline feature data corresponding to each query task from the matching offline feature data according to the target query configuration data corresponding to each query task;

[0036] The target offline feature data returning module is used to return the target offline feature data to the feature caller corresponding to the query task.

[0037] Optionally, the offline feature data has attribute data, the attribute data has corresponding attribute items, and the attribute items include data source, filtering conditions, screening range, time range and data indicators.

[0038] Optionally, the device also includes: a configuration request receiving module, used to receive a submitted configuration request; the configuration request includes feature configuration data; the feature configuration data includes semantic configuration data, query configuration data and calculation method; a target attribute data determination module, used to determine the target attribute data from the attribute data of the offline feature data according to the semantic configuration data; a feature semantic calculation module, used to use a hash algorithm to calculate the feature semantics based on the target attribute data; an offline feature data storage module, used to store the offline feature data in an offline feature library based on the corresponding feature semantics.

[0039] Optionally, the attribute items of the semantic configuration data include data source, filtering condition and filtering range.

[0040] Optionally, the offline feature data storage module is further used to merge the offline feature data with the same feature semantics and store them in the offline feature library.

[0041] Optionally, the offline feature data storage module is further used to merge the offline feature data with the same feature semantics and the same time range and store them in the offline feature library.

[0042] Optionally, the query configuration data includes a time range and a data indicator.

[0043] Optionally, the target offline feature data returning module is further used to calculate the target offline feature data according to the target calculation method to obtain aggregated index data; and return the aggregated index data to the feature caller corresponding to the query task.

[0044] Optionally, the calculation method includes at least one of total calculation, sum calculation, average value calculation and de-duplication counting.

[0045] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0046] The memory is used to store computer programs;

[0047] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0048] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0049] The embodiment of the present invention further discloses a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the embodiment of the present invention.

[0050] The embodiments of the present invention include the following advantages:

[0051] In an embodiment of the present invention, by receiving multiple query tasks sent by a feature caller, target feature configuration data is determined from preset feature configuration data according to the query tasks, wherein the target feature configuration data includes target semantic configuration data and target query configuration data, the target feature semantics is calculated according to the target semantic configuration data, and query tasks with the same target feature semantics are merged into a merged query task, and then, offline feature data matching the target feature semantics of the merged query task is obtained from a preset offline feature library as matching offline feature data, and then, according to the target query configuration data corresponding to each query task, the target offline feature data corresponding to each query task is determined from the matching offline feature data, and returned to the feature caller corresponding to each query task. After receiving multiple query tasks from the feature caller, the embodiment of the present invention calculates the target feature semantics of the query task, and then merges the query tasks with the same target feature semantics into a merged query task to query the offline feature data in the offline feature library, thereby avoiding repeated queries when backtracking offline feature data, saving system resources, and improving query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of steps of an embodiment of a data query method of the present invention;

[0053] Figure 2 It is a flow chart of offline feature data storage steps of a data query method embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of a feature configuration page of the present invention;

[0055] Figure 4 It is a characteristic configuration calculation schematic diagram in the prior art;

[0056] Figure 5 It is a characteristic configuration calculation schematic diagram of a data query method embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of offline feature data storage in an embodiment of a data query method of the present invention;

[0058] Figure 7 It is a flow chart of a data query method embodiment of the present invention;

[0059] Figure 8 It is a structural block diagram of an embodiment of a data query device of the present invention. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Currently, when training offline models or algorithms, a large amount of offline feature data is required as samples. The usual method is to trace back the offline feature data. When tracing back the offline feature data, the offline feature data usually has the characteristics of multiple dimensional selections and a long feature backtracking time.

[0062] The following problems exist when using the existing feature library for offline model training:

[0063] 1. When a new offline feature data is added, although all offline feature data after the added time point can be automatically saved, the data before the added time point usually requires manual query tasks to trace back, which is not automated enough.

[0064] 2. Generally, it takes a long time to trace back offline feature data, and there is a lot of offline feature data that needs to be traced back. Currently, Redis (Remote Dictionary Server) storage is usually used. Although the query time can be reduced to milliseconds, it occupies a large storage space and the storage medium is expensive, which will greatly increase the storage cost.

[0065] 3. Due to the existence of time windows for offline feature data, each piece of offline feature data may exist in different time windows, that is, each piece of offline feature data may have different start and end times. Therefore, when storing and querying offline feature data, it is necessary to calculate the time window of each piece of offline feature data. This complex calculation will consume a lot of offline computing resources and has a high implementation cost in the offline computing framework.

[0066] 4. When there are multiple query tasks performing queries at the same time, it is necessary to create a query task to execute the query steps for the feature configuration corresponding to each query task. Among them, when there are multiple feature configurations with the same feature configuration data, it will cause a waste of system resources and low query efficiency.

[0067] In this regard, one of the core inventive points of an embodiment of the present invention is to store and query offline feature data according to feature semantics. After receiving multiple query tasks from the feature caller, the target feature semantics corresponding to the target feature configuration data of the query task can be calculated, and then the query tasks with the same target feature semantics can be merged into a merged query task, and then the offline feature data is queried in the offline feature library, thereby avoiding redundant queries during offline feature data backtracking, saving system resources, and improving query efficiency.

[0068] Reference Figure 1 , shows a flow chart of steps of a data query method embodiment of the present invention, which may specifically include the following steps:

[0069] Step 102: Receive multiple query tasks sent by the feature caller.

[0070] In a specific implementation, an embodiment of the present invention can be applied to a data query platform, which can simultaneously receive multiple query tasks sent by the feature caller and process these query tasks simultaneously. Specifically, the query task can be a query request for offline feature data, and the query request can carry a target feature configuration identifier, which can be a number, letter, symbol or other custom identifier, such as "10001", "abcd" or "target feature configuration identifier 1001". Among them, offline feature data can include operational data of platforms, APPs (Applications) and web pages, as well as historical behavior data of users.

[0071] Specifically, the offline feature data has attribute data, the attribute data has corresponding attribute items, and the attribute items include data source, filtering conditions, filtering range, time range and data indicators.

[0072] Among them, the data source is used to distinguish the source of offline feature data, such as platform data source, APP data source or web data source, etc. The filtering condition is a custom condition used to limit the offline feature data, such as scene, gender or level, etc. The screening range is used to distinguish offline feature data by type, such as user, IP address or region, etc. The time range is used to distinguish offline feature data by time period, such as the last hour, 1 pm to 2 pm, etc. The data indicator is used to distinguish the behavior or operation of offline feature data, such as posting, commenting or liking.

[0073] For example, the attribute data corresponding to the offline feature data A is the number of posts made by user A in the 123 scene of the APP in the last hour, the attribute data corresponding to the offline feature data B is the number of posts made by user B in the 123 scene of the APP in the last hour, and the attribute data corresponding to the offline feature data C is the number of comments made by user C in the 123 scene of the APP in the last hour. If the attribute data corresponding to the offline feature data A is divided according to the attribute items, it is: the number of posts (data indicators) of user A (filtering range) in the 123 scene (filtering condition) of the APP (data source) in the last hour (time range); the same applies to offline feature data B and C.

[0074] Step 104: determining target feature configuration data from preset feature configuration data according to the query task; the target feature configuration data includes target semantic configuration data and target query configuration data.

[0075] Among them, the target feature configuration data includes target semantic configuration data and target query configuration data. Specifically, the target semantic configuration data is the configuration data used to calculate feature semantics, and the target query configuration data can be the configuration data in the target feature configuration data except the target semantic configuration data, which can be used to query the target offline feature data. Specifically, the target semantic configuration data can be configured for the data source, filtering conditions, and screening range of the offline feature data, and the target query configuration data can be configured for the time range and data indicators of the offline feature data. When the target semantic configuration data of multiple target feature configuration data are the same, the target offline feature data can be determined through the target query configuration data.

[0076] For example, assuming there are target feature configuration data A100 and target feature configuration data A200, the target semantic configuration data of A100 is APP data source, 123 scenarios and users, and the target query configuration data is the last hour and the number of posts; the target semantic configuration data of A200 is APP data source, 123 scenarios and users, and the target query configuration data is the last day and the number of comments; then, when the target semantic configuration data of A100 and A200 are the same, different target offline feature data can be queried through the target query configuration data.

[0077] In a specific implementation, the query task sent by the feature caller may include a target feature configuration identifier, and in the query task, the time range to be queried may also be set; wherein the feature configuration identifier is used to uniquely identify a feature configuration, and the target feature configuration data may be determined from the preset feature configuration data according to the feature configuration identifier. The feature configuration is a pre-set configuration item for querying offline feature data, specifically, configuration for the attribute data of offline feature data.

[0078] Step 106: Calculate the target feature semantics according to the target semantic configuration data.

[0079] Furthermore, the target feature semantics can be calculated based on the target semantic configuration data. The target feature semantics can be numbers, letters, symbols or other custom identifiers. The target semantic configuration data and the target feature semantics are in a corresponding relationship. The target feature semantics calculated by the same target semantic configuration data are also the same. On the contrary, the target feature semantics calculated by different target semantic configuration data are different.

[0080] Step 108: Merge the query tasks with the same target feature semantics into a merged query task.

[0081] In a specific implementation, when there are multiple query tasks with the same target feature semantics, it means that the target semantic configuration data of the multiple query tasks are the same. For the same target semantic configuration data, only one query operation is required. Therefore, the multiple query tasks can be merged into a merged query task.

[0082] Step 110: Obtain offline feature data that semantically matches the target feature of the combined query task from a preset offline feature library as matching offline feature data.

[0083] In a specific implementation, offline feature data can be stored in an offline feature library according to feature semantics, specifically in the form of key-value (i.e., key-value pairs), with feature semantics as the key and offline feature data as the value, and the feature semantics and offline feature data are stored in the offline feature library accordingly. One possible scenario is that different offline feature data have the same feature semantics, so the same feature semantics may correspond to multiple different offline feature data.

[0084] In a specific implementation, after obtaining the target feature semantics, the target feature semantics can be compared with the feature semantics of all offline feature data in the offline feature library one by one, and then the matching offline feature data whose feature semantics matches the target feature semantics can be obtained.

[0085] Step 112: According to the target query configuration data corresponding to each query task, determine the target offline feature data corresponding to each query task from the matching offline feature data.

[0086] After the merged query task obtains the matching offline feature data with the same target semantic configuration data, since different offline feature data may have the same feature semantics in the offline feature library, the matching offline feature data may include multiple offline feature data. The corresponding target offline feature data can be further determined from the matching offline feature data with the target query configuration data corresponding to each query task in the merged query task. The target offline feature data is the offline feature data that meets the target feature configuration data of the query task.

[0087] Step 114: Return the target offline feature data to the feature caller corresponding to the query task.

[0088] In a specific implementation, after the target offline feature data of each query task is determined, the target offline feature data will be returned to the feature caller corresponding to each query task.

[0089] In the above data query method, by receiving multiple query tasks sent by the feature caller, the target feature configuration data is determined from the preset feature configuration data according to the query task, wherein the target feature configuration data includes target semantic configuration data and target query configuration data, the target feature semantics is calculated according to the target semantic configuration data, and the query tasks with the same target feature semantics are merged into a merged query task, and then, the offline feature data matching the target feature semantics of the merged query task is obtained from the preset offline feature library as the matching offline feature data, and then, according to the target query configuration data corresponding to each query task, the target offline feature data corresponding to each query task is determined from the matching offline feature data, and returned to the feature caller corresponding to each query task. After receiving multiple query tasks from the feature caller, the embodiment of the present invention calculates the target feature semantics of the query task, and then merges the query tasks with the same target feature semantics into a merged query task to query the offline feature data in the offline feature library, thereby avoiding repeated queries when the offline feature data is backtracked, saving system resources, and improving query efficiency.

[0090] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0091] In an exemplary embodiment, referring to Figure 2 Before step 102, receiving the query task sent by the feature caller, the method further includes:

[0092] Step 202: receiving a submitted configuration request; the configuration request includes feature configuration data; the feature configuration data includes semantic configuration data, query configuration data and calculation method;

[0093] In a specific implementation, the embodiment of the present invention can be applied to a data query platform, which provides a feature configuration page, such as Figure 3 As shown, the feature configuration page provides configuration items for the above attribute items. The user can set the corresponding feature configuration data in each configuration item, and preset the calculation method for calculating the aggregate index value.

[0094] Among them, the feature configuration data includes semantic configuration data and query configuration data. In the specific implementation, the attribute items of the semantic configuration data include data source, filter conditions and filter range; the query configuration data includes time range and data indicators. In one example, the user can first configure the semantic configuration data, for example, configure the data source as the APP data source, configure the filter conditions as the 123 scenes, and configure the filter range as the user; then configure the query configuration data, for example, set the time range to the last hour, and configure the data indicator to the number of posts. That is, the corresponding feature configuration data can be obtained: the number of posts by users in the last hour in the 123 scenes in the APP data source. The calculation method can also be configured, where the calculation method is used to calculate the target offline feature data, thereby obtaining the aggregate indicator value. After the configuration is completed and submitted, a feature configuration can be obtained.

[0095] Step 204: determining target attribute data from the attribute data of the offline feature data according to the semantic configuration data;

[0096] Step 206: Using a hash algorithm to calculate feature semantics based on the target attribute data;

[0097] Step 208: Store the offline feature data in an offline feature library based on the corresponding feature semantics.

[0098] Furthermore, the attribute data of the offline feature data that conforms to the feature configuration and feature configuration data is determined as the target attribute data, namely, the APP data source, 123 scenario and user; and then the feature semantics can be calculated based on the target attribute data, specifically, the calculation can be performed using a hash algorithm, that is, a unique hash value is calculated for the target attribute data through a hash algorithm, and this hash value is the feature semantics corresponding to the target attribute data, and then the offline feature data that conforms to the target attribute data can be stored in the offline feature library according to the feature semantics.

[0099] In a specific implementation, multiple feature configurations can be pre-set according to actual needs, and the feature configurations can be stored in a feature configuration library, which can be a relational database. When the semantic configuration data of the feature configuration in the feature configuration library is modified, the semantic features can be recalculated for the modified semantic configuration data, and the offline feature data can be re-stored in the offline feature library according to the new feature semantics. For example, the user modifies the feature configuration data in the above example, specifically modifies the filter conditions in the semantic configuration data, such as configuring the filter conditions to 456 scenes, then the modified feature configuration data is: the number of posts by users in the 456 scenes in the APP data source within the last hour. Since the semantic configuration data of this feature configuration has changed, it is necessary to recalculate the feature semantics of this feature configuration, and re-store the offline feature data that meets the new feature configuration in the offline feature library according to the new feature semantics.

[0100] In one example, when there are multiple query tasks at the same time, it is necessary to calculate the feature configurations carried in the query tasks to determine the target offline feature data corresponding to the query tasks.

[0101] Reference Figure 4 , shows a schematic diagram of feature configuration calculation. As shown in the figure, feature 1, feature 2, feature 3, feature 4, until feature n are query tasks for offline feature data, all of which have target feature configuration data. For each target feature configuration data, a query task needs to be started by the offline feature scheduling system for calculation, corresponding to task 1, task 2, task 3, task 4, until task n. When similar or identical feature configuration data exist, it will cause a waste of system resources.

[0102] Reference Figure 5 , shows a schematic diagram of feature configuration calculation of an embodiment of a data query method of the present invention. As shown in the figure, feature 1, feature 2, feature 3, feature 4 and feature 5 are query tasks for offline feature data, and each has target feature configuration data. When performing feature configuration calculation, the feature semantics are first calculated according to the three target attribute data of screening range (dim), filtering condition (filter) and data source (source) through hash (hash algorithm). The feature semantics of feature 1, feature 2, feature 3, feature 4 and feature 5 are semantic 1, semantic 2, semantic 1, semantic 3, and semantic 2 respectively. Then, feature configurations with the same feature semantics can be merged into one query task, that is, features 1 and 3 are merged into SQL_1 task, features 2 and 5 are merged into SQL_2 task, and feature 4 is SQL_3 task. It can be seen that by merging query tasks with the same feature semantics, the occupation of system resources can be reduced and the query efficiency can be improved.

[0103] In the above exemplary embodiments, by calculating the feature semantics of offline feature data and feature configuration, the offline feature semantics can be stored in the offline feature library according to the feature semantics, and the semantic feature configuration is preset in the feature configuration, without the need to manually select the target attribute data, thereby simplifying the calculation steps of the feature semantics. After receiving multiple query tasks from the feature caller, the target feature semantics of the target feature configuration data corresponding to the query task can be calculated, and then the query tasks with the same target feature semantics can be merged into a merged query task to query the offline feature data; redundant query calculations during offline feature data backtracking are avoided, system resources are saved, and query efficiency is improved.

[0104] In an exemplary embodiment, the step 208 of storing the offline feature data in an offline feature library based on the corresponding feature semantics includes:

[0105] The offline feature data with the same feature semantics are merged and stored in the offline feature library.

[0106] In a specific implementation, if a plurality of offline feature data have the same feature semantics, the offline feature data may be merged, and the merged offline feature data may be stored according to the same feature semantics.

[0107] In one example, suppose there are offline feature data D and E, the attribute data corresponding to the offline feature data D is the number of posts made by user D in the 123 scene of the APP in the last hour, and the attribute data corresponding to the offline feature data E is the number of comments made by user E in the 123 scene of the APP in the last day; if the feature semantics are calculated based on the data source, filtering conditions and filtering range, the feature semantics of the offline feature data D and E are the same, assuming that the feature semantics is "TZYY1", therefore, the offline feature data D and E can be merged and stored, specifically, the data indicators of the offline feature data D and E can be merged, and the time range is added after the data indicators to distinguish them, that is, "number of posts (last hour) + number of comments (last day)"; then the offline features D and E will be stored in the offline feature library in the form of "TZYY1"-"number of posts (last hour) + number of comments (last day)". Of course, the way and form of offline feature merging and storage here are only examples, and the embodiments of the present invention do not limit this.

[0108] In the above exemplary embodiment, offline feature data with the same feature semantics are merged and stored in an offline feature library, thereby achieving compression of offline feature data, reducing the occupancy rate of storage space by offline feature data, and lowering storage costs.

[0109] In an exemplary embodiment, the merging the offline feature data with the same feature semantics and storing them in the offline feature library includes:

[0110] The offline feature data with the same feature semantics and the same time range are merged and stored in the offline feature library.

[0111] In a specific implementation, if multiple offline feature data have the same feature semantics and the same time range, these offline feature data may be merged, and the merged offline feature data may be stored according to the same feature semantics.

[0112] In one example, suppose there are offline feature data F and G. The attribute data corresponding to the offline feature data F is the number of posts made by user F in the 123 scenes of the APP in the last two hours, and the attribute data corresponding to the offline feature data G is the number of comments made by user G in the 123 scenes of the APP in the last two hours. If the feature semantics are calculated based on the data source, filtering conditions and filtering range, the feature semantics of the offline feature data F and G are the same. Assuming the feature semantics is "TZYY2", the offline feature data F and G can be merged and stored. Specifically, the data indicators of the offline feature data F and G can be merged, that is, "number of posts + number of comments". Finally, the offline features F and G will be stored in the offline feature library in the form of "TZYY2"-"number of posts + number of comments". Of course, the way and form of offline feature merging and storage here are only examples, and the embodiments of the present invention do not limit this.

[0113] Reference Figure 6 , showing a schematic diagram of offline feature data storage in an embodiment of a data query method of the present invention. As shown in the figure, fid, dim, filter, source, and timestamp are attribute data corresponding to the offline feature data, and the data corresponding to indicator, such as Cf1:c1:v1, etc., are data indicators of the offline feature data. The left figure shows the distinction and storage based on the attribute data of the offline feature data. As long as the attribute data are different, the data indicators are stored separately. The right figure shows the distinction and storage based on the feature semantics, using dim, filter, source, and timestamp as the target attribute data for calculating the feature semantics, and merging and storing the offline feature data with the same feature semantics. Specifically, the data indicators of the offline feature data can be merged and stored.

[0114] In the specific implementation, offline feature data is usually huge in quantity and needs to be queried frequently. Therefore, a storage medium that can store a large amount of data and has a high QPS (Query Per Second) is required. As an example, HBase can be used as the storage medium for offline feature data. HBase is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. The technology comes from the Google paper "Bigtable: A Distributed Storage System for Structured Data" written by Fay Chang. It is a sub-project of the Apache Hadoop project and is suitable for storing unstructured data. Specifically, HBase has the following advantages: HBase data will be sorted and stored in the dictionary order of feature semantics and placed together in physical locations. This can be used to achieve efficient batch queries in time order and time range. HBase data is stored on HDFS (Hadoop Distributed File System) and supports massive data storage. HBase can achieve high query QPS by virtue of the characteristics of LSM (Log-Structured-Merge-Tree) tree, multi-shard storage of data, and multi-service query. You can use the BulkLoad (data import function in HBase) function to efficiently import a large amount of feature data from the data warehouse into HBase in batches every day.

[0115] In the above exemplary embodiment, offline feature data with the same feature semantics and the same time range can be merged and stored in the offline feature library, thereby further compressing and storing the offline feature data, reducing the occupancy rate of the storage space by the offline feature data, and reducing the storage cost; in addition, using HBase as the storage medium improves the storage reliability of the offline feature data and improves the efficiency of storage and query.

[0116] In an exemplary embodiment, the step 114, returning the target offline feature data to the feature caller corresponding to the query task, includes:

[0117] Calculating the target offline feature data according to the target calculation method in the target feature configuration data to obtain aggregated index data;

[0118] The aggregated indicator data is returned to the feature caller corresponding to the query task.

[0119] In a specific implementation, after determining the target offline feature data of the query task, the target offline feature data can also be calculated, wherein the calculation method includes at least one of total calculation, sum calculation, average calculation and deduplication counting. Specifically, assuming that the target offline feature data is the number of posts by all users in the 123 scenes of the APP in the last hour, then the offline feature data can be calculated using the target calculation method in the feature configuration data. Assuming that the target calculation method is average calculation, the aggregate index value is the average of the number of posts by users in the 123 scenes of the APP in the last hour. After calculating the aggregate index value, it is returned to the feature caller corresponding to the query task.

[0120] In the above exemplary embodiments, through the calculation method preset in the feature configuration data, after obtaining the target offline feature data, the target offline feature data can be further calculated according to the preset calculation method to obtain an aggregation index value, and returned to the feature caller, which can make the offline feature data calculation more flexible and simple and improve the efficiency of offline feature data calculation.

[0121] In order to enable those skilled in the art to better understand the technical solution of the embodiment of the present invention, an example is given below. Figure 7 The embodiments of the present invention are described.

[0122] Reference Figure 7 , shows a flow chart of a data query method embodiment of the present invention. As shown in the figure, feature configuration is stored in a feature configuration library according to feature semantics, and HBase is an offline feature library that stores multiple offline feature data.

[0123] 1. Receive multiple query tasks from the feature caller. The query tasks carry query parameters, which may include fid (feature configuration identifier), dim (filter range of offline feature data), and optional timestamp (time range);

[0124] 2. According to the query parameters in the query task, obtain the target feature configuration corresponding to the query task from the feature configuration library;

[0125] 3. The target feature configuration includes the target feature configuration data (screening range dim, data source source, filtering condition fileter, data indicator indicator, time range timeslice) and target calculation method;

[0126] 4. Calculate the target feature semantics according to the target semantic configuration data of the target feature configuration data; merge query tasks with the same target feature semantics into a merged query task;

[0127] 5. Query the matching offline feature data from HBase that semantically matches the target feature of the merge query task;

[0128] 6. After obtaining the matching offline feature data, determine the target offline feature data corresponding to each query task from the matching offline feature data according to the target query configuration data of the target feature configuration data corresponding to each query task;

[0129] 7. Calculate the target offline feature data according to the target calculation method of the target feature configuration data corresponding to each of the query tasks to obtain an aggregate index value;

[0130] 8. Return the aggregated index value to the feature caller corresponding to each query task.

[0131] In the above example, offline feature data is stored in HBase based on feature semantics, which improves the storage reliability of offline feature data. The target feature semantics of the target feature configuration data corresponding to the query task can be calculated, and then the query tasks with the same target feature semantics can be merged into a merged query task to query the offline feature data; and the aggregated indicator value corresponding to each query task is calculated and returned to the feature caller, avoiding redundant query calculations during offline feature data backtracking, saving system resources, and improving query efficiency.

[0132] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0133] Reference Figure 8 , shows a structural block diagram of a data query device embodiment of the present invention, which may specifically include the following modules:

[0134] The query task receiving module 802 is used to receive multiple query tasks sent by the feature caller;

[0135] The target feature configuration data determination module 804 is used to determine the target feature configuration data from the preset feature configuration data according to the query task; the target feature configuration data includes target semantic configuration data and target query configuration data;

[0136] A target feature semantics calculation module 806 is used to calculate the target feature semantics according to the target semantics configuration data;

[0137] A query task merging module 808, configured to merge query tasks with the same target feature semantics into a merged query task;

[0138] A matching offline feature data acquisition module 810 is used to acquire offline feature data semantically matching the target feature of the merged query task from a preset offline feature library as matching offline feature data;

[0139] A target offline feature data determination module 812 is used to determine the target offline feature data corresponding to each query task from the matching offline feature data according to the target query configuration data corresponding to each query task;

[0140] The target offline feature data returning module 814 is used to return the target offline feature data to the feature caller corresponding to the query task.

[0141] In an exemplary embodiment, the offline feature data has attribute data, and the attribute data has corresponding attribute items, and the attribute items include data source, filtering conditions, screening range, time range and data indicators.

[0142] In an exemplary embodiment, the device also includes: a configuration request receiving module, which is used to receive a submitted configuration request; the configuration request includes feature configuration data; the feature configuration data includes semantic configuration data, query configuration data and calculation method; a target attribute data determination module, which is used to determine the target attribute data from the attribute data of the offline feature data according to the semantic configuration data; a feature semantic calculation module, which is used to calculate the feature semantics based on the target attribute data using a hash algorithm; and an offline feature data storage module, which is used to store the offline feature data in an offline feature library based on the corresponding feature semantics.

[0143] In an exemplary embodiment, the attribute items of the semantic configuration data include data source, filtering condition and filtering range.

[0144] In an exemplary embodiment, the offline feature data storage module is further used to merge the offline feature data with the same feature semantics and store them in the offline feature library.

[0145] In an exemplary embodiment, the offline feature data storage module is further used to merge the offline feature data with the same feature semantics and the same time range and store them in the offline feature library.

[0146] In an exemplary embodiment, the query configuration data includes a time range and a data indicator.

[0147] In an exemplary embodiment, the target offline feature data returning module 814 is further used to calculate the target offline feature data according to the target calculation method to obtain aggregated index data; and return the aggregated index data to the feature caller corresponding to the query task.

[0148] In an exemplary embodiment, the calculation method includes at least one of total calculation, sum calculation, average calculation and de-duplication counting.

[0149] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0150] Preferably, an embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data query method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0151] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data query method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] An embodiment of the present invention provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned data query method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0153] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0155] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0156] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0161] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0162] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data query method, characterized in that: The method comprises: Receive multiple query tasks sent by the feature caller; Determining target feature configuration data from preset feature configuration data according to the query task; the target feature configuration data includes target semantic configuration data and target query configuration data; Calculate the target feature semantics according to the target semantic configuration data; Merging the query tasks with the same semantics of the target features into a merged query task; Acquire offline feature data semantically matching the target feature of the combined query task from a preset offline feature library as matching offline feature data; According to the target query configuration data corresponding to each query task, determining the target offline feature data corresponding to each query task from the matching offline feature data; The target offline feature data is returned to the feature caller corresponding to the query task.

2. The method according to claim 1, characterized in that The offline feature data has attribute data, and the attribute data has corresponding attribute items. The attribute items include data source, filtering conditions, screening range, time range and data indicators.

3. The method according to claim 2, characterized in that Before receiving the query task sent by the feature caller, the method further includes: Receive a submitted configuration request; the configuration request includes feature configuration data; the feature configuration data includes semantic configuration data, query configuration data and calculation method; Determining target attribute data from attribute data of the offline feature data according to the semantic configuration data; The feature semantics is calculated based on the target attribute data using a hash algorithm; The offline feature data is stored in an offline feature library based on the corresponding feature semantics.

4. The method according to claim 3, characterized in that The attribute items of the semantic configuration data include data source, filtering condition and filtering range.

5. The method according to claim 3, characterized in that: The storing the offline feature data in an offline feature library based on the corresponding feature semantics includes: The offline feature data with the same feature semantics are merged and stored in the offline feature library.

6. The method according to claim 5, characterized in that The step of merging the offline feature data with the same feature semantics and storing them in the offline feature library includes: The offline feature data with the same feature semantics and the same time range are merged and stored in the offline feature library.

7. The method according to claim 3, characterized in that The attribute items of the query configuration data include a time range and a data index.

8. The method according to claim 1, characterized in that The returning the target offline feature data to the feature caller corresponding to the query task includes: Calculating the target offline feature data according to the target calculation method in the target feature configuration data to obtain aggregated index data; The aggregated indicator data is returned to the feature caller corresponding to the query task.

9. The method according to claim 3, characterized in that: The calculation method includes at least one of total calculation, sum calculation, average value calculation and de-duplication counting.

10. A data query device, characterized in that: The device comprises: A query task receiving module, used for receiving multiple query tasks sent by the feature caller; A target feature configuration data determination module, used to determine target feature configuration data from preset feature configuration data according to the query task; the target feature configuration data includes target semantic configuration data and target query configuration data; A target feature semantics calculation module, used to calculate the target feature semantics according to the target semantics configuration data; A query task merging module, used to merge query tasks with the same target feature semantics into a merged query task; A matching offline feature data acquisition module is used to acquire offline feature data that semantically matches the target feature of the merged query task from a preset offline feature library as matching offline feature data; A target offline feature data determination module, configured to determine the target offline feature data corresponding to each query task from the matching offline feature data according to the target query configuration data corresponding to each query task; The target offline feature data returning module is used to return the target offline feature data to the feature caller corresponding to the query task.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 9 when executing the program stored in the memory.

12. One or more computer readable media having instructions stored thereon, which when executed by one or more processors cause the processors to perform the method of any one of claims 1 to 9.

Images