Real-time feature management method, computer program product, device and storage medium

Abstract

The embodiment of the invention provides a real-time feature management method, a computer program product, equipment and a storage medium. In the real-time feature generation process, the script used for generating the real-time features can be determined based on the generation mode description information configured by the user, and the script is operated once every preset time interval so as to query the latest source data related to the real-time features from the data warehouse and process the queried source data. And generating and storing a real-time characteristic value of the real-time characteristic. In addition, a degradation characteristic value corresponding to each real-time characteristic value can be determined and stored based on a degradation strategy configured by a user. When a query request for querying the real-time feature is received, if the current real-time feature value is unavailable, the corresponding degradation feature value can be utilized to replace the real-time feature value and return the real-time feature value to the online service system, so that stable operation of the online service system can be ensured, and accurate decision making of the online system can be ensured to the greatest extent.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular, to a real-time feature management method, a computer program product, a device, and a storage medium. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, the generation and management of real-time features have become particularly important in many fields. Real-time features can provide the latest data support for machine learning models, improve the accuracy and prediction ability of the models, and thus can provide accurate prediction data for online services to achieve refined operations. Taking the instant delivery scenario as an example, within a specific time and space, the estimation of delivery duration, the determination of delivery price, the generation of the pressure coefficient of the transportation capacity system, and the description and adjustment of the supply-demand relationship all depend on accurate and stable real-time feature input. During the real-time feature generation process, there may be delays in data processing and feature calculation. For some service scenarios with high real-time requirements, such delays may have a serious impact on online services. Therefore, it is necessary to provide a solution for scenarios where there are delays in real-time feature generation to ensure the stable operation of online services. Summary of the Invention

[0003] To overcome the above technical problems, the embodiments of this specification provide a real-time feature management method, a computer program product, a device, and a storage medium.

[0004] According to a first aspect of the embodiments of this specification, a real-time feature management method is provided. The method includes:

[0005] Providing an interaction interface for a user to create a real-time feature generation task through the interaction interface and configure task information of the real-time feature generation task, where the task information at least includes description information of a generation method of the real-time feature to be generated and a degradation strategy;

[0006] Determining a script for generating the real-time feature based on the description information of the generation method;

[0007] Running the script once every preset time to query the latest source data related to the real-time feature from a data warehouse, process the queried latest source data to obtain a real-time feature value of the real-time feature, and store it; wherein the source data stored in the data warehouse is updated in real time;

[0008] Determining and storing a degradation feature value corresponding to the real-time feature value based on the degradation strategy;

[0009] When receiving a query request for querying the real-time feature, if it is determined that the real-time feature value is unavailable, returning the degradation feature value.

[0010] In some embodiments, the script includes a query script for querying the latest source data related to the real-time feature from the data warehouse, and a processing script for processing the queried latest source data;

[0011] The generation method description information includes the query script and the processing script; or

[0012] The generation method description information includes a feature caliber, which is used to describe a screening strategy for screening source data related to the real-time feature from the data warehouse and a processing strategy for processing the screened source data. The query script is generated based on the screening strategy, and the processing script is generated based on the processing strategy.

[0013] In some embodiments, the processing strategy includes a calculation method for calculating the queried latest source data, and the calculation method includes one or more of the following:

[0014] Micro-batch calculation, graph calculation, and streaming calculation.

[0015] In some embodiments, the storage type of the source data in the data warehouse is determined based on the calculation method. Among them, in the scenario where the calculation method is micro-batch calculation, the storage type is hologres; in the scenario where the calculation method is graph calculation, the storage type is IGraph; in the scenario where the calculation method is streaming calculation, the storage type is a message queue.

[0016] In some embodiments, the real-time feature is a real-time feature in the instant delivery scenario, and the processing strategy includes the calculation time window length and the geographical fence identifier corresponding to the real-time feature.

[0017] In some embodiments, the source data stored in the data warehouse is obtained in the following manner:

[0018] Monitor the source data queues uploaded by each data source;

[0019] When each piece of source data is obtained, query the extended fields associated with the piece of source data from a preset data dimension table or system based on the content of the piece of source data;

[0020] Add the extended fields to the piece of source data as the extended source data, and store it in the data warehouse.

[0021] In some embodiments, the degraded feature value is stored in the form of main key - secondary key - value, where the main key is the standard name of the feature of the real-time feature, the secondary key is the time slice corresponding to the real-time feature value, and the value is the degraded feature value.

[0022] In some embodiments, when the status of the real-time feature is in a degraded state, it is determined that the real-time feature value is unavailable, where the degraded state is determined based on the following methods:

[0023] Monitor the output queue of the real-time feature. If no new real-time feature value is output from the output queue for a preset duration, switch the status of the real-time feature to the degraded state; and / or

[0024] When a degradation instruction for the real-time feature is received from the user through the interaction interface, switch the status of the real-time feature to the degraded state.

[0025] In some embodiments, the real-time feature is a real-time feature in an instant delivery scenario, and the task information further includes the feature standard name of the real-time feature. Obtaining and storing the real-time feature value of the real-time feature includes:

[0026] Store the real-time feature value in the online query database in key-value form to provide an online query service; where the key carries the feature standard name of the real-time feature and the geographical fence identifier corresponding to the real-time feature, and the value is the real-time feature value; and / or

[0027] Write the real-time feature value into the OPDS to provide an offline query service.

[0028] In some embodiments, the method further includes:

[0029] Receive a query request, where the query request carries the geographical fence identifier corresponding to the real-time feature to be queried;

[0030] Query the corresponding real-time feature value from the online query database based on the geographical fence identifier and return it;

[0031] Or

[0032] Receive a query request, where the query request carries geographical location information;

[0033] Convert the geographical location information into a geographical fence identifier;

[0034] Query the corresponding real-time feature value from the online query database based on the geographical fence identifier and return it.

[0035] In some embodiments, the real-time feature is a real-time feature in an instant delivery scenario, and each real-time feature is described by multiple metadata. The task information includes one or more of the multiple metadata, where the multiple metadata includes one or more of the following:

[0036] Feature caliber description, calculation time window length, and geofence type, feature standard name, degradation strategy;

[0037] Among them, the feature caliber is used to describe the screening strategy for screening source data related to the real-time feature from the data warehouse and the processing strategy for processing the screened source data; the feature standard name carries a prefix of the geofence type and a suffix of the calculation time window length.

[0038] According to the second aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the method mentioned in the first aspect above.

[0039] According to the third aspect of the embodiments of this specification, an electronic device is provided. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the computer program is executed, it implements the method mentioned in the first aspect above.

[0040] According to the fourth aspect of the embodiments of this specification, a computer storage medium is provided. A computer program is stored on the computer storage medium, and when the computer program is executed by a processor, it implements the method mentioned in the first aspect above.

[0041] Advantages of the embodiments of this specification: When a user creates a real-time feature generation task, in addition to configuring the description information of the generation method of the real-time feature, the user can also configure a corresponding degradation strategy based on the characteristics of each real-time feature. The degradation strategy can be used to determine the degradation feature value corresponding to each real-time feature value of the real-time feature. Among them, the degradation feature value can be a feature value that is determined based on historical data and is relatively close to the real-time feature value, or can accurately reflect the current situation of the real-time feature. During the real-time feature generation process, a script for generating the real-time feature can be determined based on the description information of the generation method configured by the user, and the script is run once every preset time interval to query the latest source data related to the real-time feature from the data warehouse, and process the queried source data to generate and store the real-time feature value of the real-time feature. At the same time, the degradation feature value corresponding to each real-time feature value can be determined based on the degradation strategy configured by the user and stored. When a query request for querying this real-time feature is received, if the current real-time feature value is unavailable due to various delays such as calculation and transmission, the corresponding degradation feature value can be used to replace the real-time feature value and returned to the online service system, thereby ensuring the stable operation of the online service system and maximizing the accurate decision-making of the online system.

[0042] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the embodiments of this specification. Brief Description of the Drawings

[0043] The accompanying drawings here are incorporated into the description and constitute a part of the embodiments of this description, showing embodiments that conform to the embodiments of this description, and are used together with the description to explain the principles of the embodiments of this description.

[0044] FIG. 1(a) is a schematic diagram of an application scenario shown in an exemplary embodiment of this description;

[0045] FIG. 1(b) is a schematic diagram of an application scenario shown in another exemplary embodiment of this description;

[0046] Figure 2 is a flowchart of a real-time feature management method shown in an exemplary embodiment of this description;

[0047] Figure 3 is a schematic diagram of pushing and storing generated real-time features shown in an exemplary embodiment of this description;

[0048] Figure 4 is a schematic diagram of a real-time feature management method shown in an exemplary embodiment of this description;

[0049] Figure 5 is a logic block diagram of an electronic device shown in an exemplary embodiment of this description. Detailed implementation manners

[0050] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of this description. On the contrary, they are merely examples of devices and methods consistent with some aspects of the embodiments of this description as detailed in the appended claims.

[0051] The terms used in the embodiments of this description are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of this description. The singular forms "a", "the", and "said" used in the embodiments of this description and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0052] It should be understood that although terms such as first, second, and third may be used in the embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to a determination".

[0053] Real-time features are widely used in some service scenarios that require quick response and dynamic decision-making. By extracting real-time features from real-time service data and using these real-time features to provide the latest data support for machine learning models, the accuracy and prediction ability of the models can be improved, and then accurate prediction data can be provided for online services so as to make accurate decisions based on the prediction data.

[0054] Taking the instant delivery scenario as an example, by extracting real-time features from real-time service data and using these real-time features for estimating delivery duration, determining delivery price, scheduling transportation capacity, etc., the dispatching method of riders can be optimized and the delivery efficiency can be improved. For example, by analyzing real-time features such as the real-time location of riders, real-time traffic conditions, and order distance, the dispatching system can more accurately estimate the delivery duration, and then the delivery tasks can be allocated based on the delivery duration, improving the delivery efficiency.

[0055] Generally, the generation and management of real-time features can be achieved through a real-time feature management platform. With the continuous growth of service complexity and data volume, it is usually required that the real-time feature management platform process a large amount of data in a short time, and the calculation difficulty is relatively high. Therefore, it is easy to have a situation where data processing and feature calculation may be delayed. For scenarios with high real-time requirements such as instant delivery, such a delay may have a serious impact on online services. For example, the failure to update real-time features in a timely manner may lead to decision-making delays or errors.

[0056] Taking the instant delivery scenario as an example, the dispatching system needs to dynamically adjust the delivery route and task allocation according to real-time features such as the real-time location of riders, the number of orders, and traffic conditions. If the update of these real-time features is delayed, for example, the number of riders within a certain geofence is delayed, and the data is not the current number of riders but the number of riders at a certain past moment, it may lead to untimely transportation capacity allocation, resulting in low delivery efficiency and even affecting user satisfaction.

[0057] Based on this, the embodiments of this specification provide a feature management method. When a user creates a real-time feature generation task, in addition to configuring the description information of the generation method of the real-time feature, the user can also configure the corresponding degradation strategy based on the characteristics of each real-time feature. The degradation strategy can be used to determine the degradation feature values corresponding to the real-time feature values of the real-time feature. Among them, the degradation feature value can be a feature value that is determined based on historical data and is relatively close to the real-time feature value, or can accurately reflect the current situation of the real-time feature. During the real-time feature generation process, the script used to generate the real-time feature can be determined based on the generation method description information configured by the user, and this script is run once every preset time interval to query the latest source data related to the real-time feature from the data warehouse, and process the queried source data to generate and store the real-time feature value of the real-time feature. At the same time, the degradation feature values corresponding to the real-time feature values can be determined based on the degradation strategy configured by the user and stored. When a query request for this real-time feature is received, if the current real-time feature value is unavailable due to various delays such as calculation and transmission, the corresponding degradation feature value can be used to replace the real-time feature value and returned to the online service system, so as to ensure the stable operation of the online service system and can maximize the accurate decision-making of the online system.

[0058] The real-time feature generation method provided by the embodiments of this specification can be executed by a pre-developed real-time feature management platform, and this real-time feature management platform can be deployed in a server or a server cluster. In some scenarios, in order to provide a complete solution from the cleaning of source data, the generation of real-time features to the query service of real-time features, multiple services such as the construction of a data warehouse, the generation and management of real-time features, and the query of real-time features can be integrated in this real-time feature management platform.

[0059] As shown in Figure 1(a), it is a schematic diagram of an application scenario according to an embodiment of this specification. The real-time feature management platform includes three functional modules: a data warehouse construction service, a real-time feature generation and management service, and a real-time feature query service. The three functional modules can be deployed on a server or a server cluster, and are communicatively connected to multiple service systems (such as Service System A and Service System B). The multiple service systems can send the source data generated in real time during the online service process to the data warehouse construction service of the real-time feature management platform, so that the data warehouse construction service can clean or perform various preprocessing on the service data, and then store it in the data warehouse for generating real-time features. The real-time feature generation and management service of the real-time feature management platform can provide an interactive interface through which users can create real-time feature generation tasks and configure relevant task information. After the real-time feature generation task goes online, the real-time feature generation and management service can execute the task at a preset time interval, continuously query relevant source data from the data warehouse, generate real-time features, and then store the generated real-time features in an online query database (such as a Redis database). When each service system needs to use real-time features, it can send a real-time feature query request to the real-time feature query service of the real-time feature management platform. The real-time feature query service can query the corresponding real-time feature value from the online query database and return it to the service system.

[0060] Considering that the calculation and storage delay of real-time features may cause the real-time feature values to be unavailable, in order to ensure the stable operation of the online service, as shown in Figure 1(b), when configuring task information, users can configure a degradation strategy for real-time features. The real-time feature generation and management service can generate corresponding degraded features based on the degradation strategy and store them. When receiving a query request from the service system to query real-time features, if it is determined that the current real-time feature value of the real-time feature is unavailable, the degraded feature value corresponding to the real-time feature value can be queried from the online query database and returned to the service system.

[0061] The following combines Figure 2 to introduce in detail the real-time feature management method provided by the embodiment of this specification. As Figure 2 shown, the real-time feature management method may include the following steps:

[0062] S202. Provide an interactive interface so that users can create a real-time feature generation task through the interactive interface and configure the task information of the real-time feature generation task. The task information at least includes description information of the generation method of the real-time feature to be generated and a degradation strategy;

[0063] In step S202, to facilitate the user in configuring the task information of the real-time feature generation task, an interactive interface can be provided through which the user can configure the task information of the real-time feature generation task. The task information may include various types of information related to the real-time feature to be generated, such as the task name, metadata for describing the real-time feature, the failure level caused when the real-time feature is unavailable, and other types of information.

[0064] Among them, the task information at least includes the description information of the generation method and the degradation strategy of the real-time feature to be generated. The description information of the generation method can be used to describe the generation method of the real-time feature. The description information of the generation method can be a text description of the real-time feature generation method or the code for generating the real-time feature, which is not limited in the embodiments of this specification. The degradation strategy is used to indicate how to determine a relatively reliable and accurate degradation feature value that can be used to replace the real-time feature value when the real-time feature value of the real-time feature is unavailable, so as to ensure the stable operation of the online service. The degradation strategy can describe whether each real-time feature can be degraded and the determination method of its degradation feature value in the degraded state. Usually, in order to ensure relatively accurate decision results, the degradation feature value can be determined based on historical data, and the specific generation method of the degradation feature value can be flexibly set according to the characteristics of the real-time feature to ensure that the degradation feature value can reflect the current situation of the real-time feature as accurately as possible.

[0065] For example, taking the real-time feature as the order volume in business district A in the past 10 minutes, the degradation feature value can be the historical value of the order volume in business district A at the same time slice in the past day, the average value of business district A at the same time slice in the past seven days, and so on. Among them, the time slice can be a certain time point or time period. For example, taking the current time as 10:30 on January 31, 2025, the past 10 minutes is the time slice from 10:20 to 10:30 on January 31, 2025. Therefore, the same time slice in the past day refers to 10:20 - 10:30 on January 30, 2025.

[0066] In some scenarios, to facilitate the unified management of real-time features, a set of metadata can be preset to describe a real-time feature. The metadata can include multiple dimensions, and for different application fields, the type of metadata can be flexibly set based on the characteristics of the real-time features in that application field. The task information can include one or more metadata of the real-time feature.

[0067] S204. Determine a script for generating the real-time feature based on the description information of the generation method;

[0068] In step S204, a script for generating the real-time feature can be determined based on the generation method description information. The script can include one or more. For example, the script can include a query script for querying source data related to the real-time feature from a data warehouse, and the script can also include a processing script for processing the queried source data. In some scenarios, the generation method description information can directly include the above scripts, that is, when the user configures the task information, the corresponding scripts can be entered. In some scenarios, the generation method description information can also be text information describing the real-time feature generation method, and the feature management platform can automatically generate the above scripts based on this text description.

[0069] S206. Run the script once every preset time to query the latest source data related to the real-time feature from the data warehouse, process the queried latest source data, obtain the real-time feature value of the real-time feature, and store it; where the source data stored in the data warehouse is updated in real time;

[0070] In S206, in order to generate a real-time feature, a data warehouse can be constructed. The data warehouse is a real-time data warehouse, that is, the source data in the data warehouse can be updated in real time. For example, real-time source data streams can be obtained from various data sources, and these source data streams can be uniformly cleaned and various preprocessings can be performed to obtain the source data for generating the real-time feature, and then stored in the data warehouse. After determining the above script, the script can be run once every preset time to query the latest source data related to the real-time feature from the data warehouse, process the queried latest source data, obtain the real-time feature value of the real-time feature, and store it. For example, if the real-time feature is the number of active riders in Business District A in the past 10 minutes, the script can be run once every 1 minute to query the relevant latest source data from the data warehouse and perform processing, so that the real-time feature value of the real-time feature can be obtained.

[0071] S208. Determine and store the degraded feature value corresponding to the real-time feature value based on the degradation policy;

[0072] In step S208, the degraded feature value corresponding to the real-time feature value can be determined and stored according to the above degradation policy. When the user configures the degradation policy, for some real-time features, the degraded feature value corresponding to the real-time feature value of the real-time feature can be directly configured. For some real-time features, the calculation method of the degraded feature value can be configured, so that the feature management platform can calculate the degraded feature value based on this calculation method. Among them, for different real-time features, the calculation methods can be different, and can be specifically set according to actual needs. Of course, for some real-time features that cannot be degraded, such as the real-time location of a rider, it can also be configured as non-degradable.

[0073] After determining the downgraded eigenvalue, the downgraded eigenvalue can be stored to facilitate the subsequent use of the downgraded eigenvalue to replace the above-mentioned real-time eigenvalue in scenarios where the real-time eigenvalue is unavailable.

[0074] S210. When receiving a query request for the real-time feature, if it is determined that the real-time eigenvalue is unavailable, the downgraded eigenvalue is returned.

[0075] In step S210, after receiving a query request for the real-time feature, if it is determined that the current real-time eigenvalue of the real-time feature is unavailable, the above-mentioned downgraded eigenvalue is returned. In the related art, if there is a delay in the current real-time eigenvalue of the real-time feature, a null value is usually returned when receiving a query request, which will affect the normal operation of the online service to a certain extent. In the embodiments of this specification, by pre-configuring the downgrading strategy for each real-time feature, when the real-time feature is unavailable, a downgraded eigenvalue that is relatively close to the current situation can be returned, thereby ensuring the stable operation of the online service.

[0076] In some embodiments, the real-time feature is a real-time feature in the instant delivery scenario. To facilitate the unified management of real-time features, each real-time feature is described by multiple metadata, and the above task information may include one or more of the multiple metadata. Considering that the real-time features in the instant delivery scenario are usually strongly correlated with time and space, the multiple metadata includes one or more of the following: feature caliber description, feature calculation time window length, and geographic fence type, feature standard name, downgrading strategy. Among them, the feature caliber is used to describe the screening strategy for screening source data related to the real-time feature from the data warehouse and the processing strategy for processing the screened source data, that is, it describes the generation method of the feature. Because the real-time features relied on in the instant delivery scenario are strongly correlated with time and space, each real-time feature corresponds to the length of the feature calculation time window and the geographic fence type. The geographic fence type includes cities, business districts, grids, aois (the smallest division unit of space), etc. In addition, for the unified management of real-time features in a standardized manner, for each real-time feature, a unified naming method can be adopted. The feature standard name carries a prefix of the geographic fence type and a suffix of the feature calculation time window length. For example, s1_create_cnt_m10, which means the order creation volume in a certain crowdsourcing business district in the past ten minutes, where s1 is the prefix of the geographic fence type and m10 is the suffix of the feature calculation time window length. When problems such as delays occur in the real-time feature production link, resulting in the unavailability of the real-time feature, users are supported to downgrade the real-time feature to a downgraded feature through a customized downgrading strategy. The supported downgrading strategies include historical values of the same time slice and the same subject id in the past day, the average value of the same time slice and the same subject id in the past seven days, and other strategies. For some real-time features that are not suitable for downgrading, such as the real-time location of riders, users are also supported to select non-downgradable.

[0077] In some embodiments, the script for generating real-time features can be written by the user himself. For example, the user can write a script based on the characteristics of the real-time features to be extracted in the interactive interface for generating the real-time features. For example, the script can include a query script for querying the latest source data related to the real-time features from the data warehouse, and a processing script for processing the queried latest source data. The generation method description information can include the above query script and the above processing script, that is, the real-time feature management platform can directly obtain the script from the generation method description information configured by the user.

[0078] In some embodiments, considering that the method of the user manually writing the script is relatively inefficient and requires the user to have a strong technical background, in order to improve the feature generation efficiency, the script for generating real-time features can also be automatically generated by the real-time feature management platform. For example, the user can only enter the text information for describing the generation method of the real-time features, and the real-time feature management can automatically generate the above script based on the text information. For example, in some scenarios, when setting the metadata of each real-time feature, the metadata can include the feature caliber, and the feature caliber is used to describe the screening strategy for screening the source data related to the real-time features from the data warehouse and the processing strategy for processing the screened source data. The generation method description information can include the feature caliber. After obtaining the feature caliber, the real-time feature management platform can automatically generate the above query script based on the screening strategy and automatically generate the above processing script based on the processing strategy.

[0079] In real-time feature generation, different types of source data and application scenarios have different requirements for calculation methods. Traditional real-time feature management platforms usually only support a single calculation method, thus limiting the flexibility and applicable scenarios of the real-time feature management platform. In order to improve the flexibility of the real-time feature management platform, enrich the applicable scenarios, and improve the efficiency of real-time feature generation, in some embodiments, the real-time feature management platform can support multiple calculation methods simultaneously, so that users can select the most suitable calculation method according to specific requirements, thereby being able to flexibly meet the needs of different types of source data and application scenarios. For example, the processing strategy includes a calculation method for calculating the latest source data queried, and this calculation method includes one or more of the following: micro-batch calculation, graph calculation, and streaming calculation. Among them, micro-batch calculation can efficiently process data within a time window, graph calculation can quickly analyze complex relationship data, and streaming calculation can process continuous data streams in real time. By combining these calculation methods, the real-time feature management platform can generate real-time features more efficiently and meet the real-time requirements of different application scenarios. By simultaneously supporting multiple calculation methods such as micro-batch calculation, graph calculation, and streaming calculation, the flexibility, adaptability, efficiency, and accuracy of the real-time feature management platform are improved. This integration of multiple calculation methods not only simplifies the platform architecture, reduces development and maintenance costs, but also enhances the scalability of the platform, enabling it to better meet the needs of different application scenarios.

[0080] Considering that different calculation methods have different requirements for data storage, in order to enable the real-time feature management platform to support multiple calculation methods, in some embodiments, when storing source data in the data warehouse, different storage types can also be selected based on the requirements of different application scenarios. For example, the storage type of source data in the data warehouse can be determined based on the above calculation methods. Among them, in the scenario where the calculation method is micro-batch calculation, the source data storage type can be HoloR es, in the scenario where the calculation method is graph calculation, the source data storage type can be Igraph, and in the scenario where the data warehouse is for streaming calculation, the source data storage type can be a message queue. Among them, HoloR es is suitable for micro-batch calculation, supports high-performance batch data processing and complex queries, and can efficiently process the aggregation and analysis of large-scale data. IGraph is suitable for graph calculation, supports efficient graph structure data processing and complex graph algorithms, and can quickly respond to graph query requests. Message queues (such as Kafka) are suitable for streaming calculation, support high-throughput, low-latency data transmission and buffering, and can process continuous data streams in real time. Different storage methods have their own advantages in different calculation scenarios. By selecting the most suitable storage type, resource utilization can be optimized, and storage and calculation costs can be reduced.

[0081] In some embodiments, the real-time feature is a real-time feature in the instant delivery scenario, and the processing strategy includes the calculation time window length corresponding to the real-time feature and the geofence identifier. In the instant delivery scenario, the spatial location of the rider changes over time. Therefore, the real-time features required in the instant delivery scenario are often strongly correlated with a certain time and space. Among them, the space refers to a variety of geofences, such as cities, business districts, grids, etc. By analyzing the real-time features within different time windows and different geofences, various service requirements such as the allocation of the transportation capacity system can be completed. Therefore, for any real-time feature in the instant delivery scenario, the real-time feature can be described by metadata such as the calculation time window length and the geofence identifier. That is, the processing strategy of the real-time feature can include the calculation time window length corresponding to the real-time feature and the geofence identifier. For example, if the real-time feature is the number of orders in Business District A in the past 10 minutes, its calculation time window length is 10 minutes, and the geofence identifier is Business District A. By using metadata such as the calculation time window length and the geofence identifier to describe the real-time feature, the processing method for processing the queried source data can be determined.

[0082] In the process of real-time feature generation, in addition to the information carried by a single data source, some additional information is often required. For example, taking the instant delivery scenario as an example, the rider data source carries information such as the rider's location information and indoor / outdoor information. However, when generating real-time features, it is also desired to obtain the information about the orders carried by the rider at the same time, but such information is not in the rider data source. Another example is that the waybill data source carries the status of the waybill and the location information of the user and merchant, but lacks the corresponding geofence information, and it is necessary to mark the corresponding geofence identifier according to the location information. The core problem faced in the above scenarios is how to achieve field expansion. The common solution in the industry is to use the multi-stream join capability of Flink. The multi-stream join capability of Flink is mainly used to merge the data streams of multiple data sources according to specific conditions to achieve field expansion. However, the multi-stream join of Flink has problems such as a complex computing network structure, a long link, difficulty in maintenance, and high resource consumption, and there is also a risk of out-of-order messages. In the process of expanding fields for the source data, in order to reduce resource consumption and complexity, when building a data warehouse, the real-time feature management platform can listen to the source data queues uploaded by each data source. When each piece of source data is obtained, the extended fields associated with the piece of source data can be queried from a preset data dimension table or service system based on the content of the piece of source data, and then the extended fields can be added to the piece of source data as the extended source data and stored in the data warehouse. Among them, different data sources can be data generated on different systems or software. Taking the instant delivery scenario as an example, different data sources can be the rider data source, the waybill data source, the merchant data source, and so on. The data dimension table can be a pre-constructed data table, and the data table records some relatively fixed data. For example, a data table that records the association relationships between different geofences. Each service system for providing source data can provide an interface. When building a data warehouse, the real-time feature management platform can send a query request to each service system through this interface. The query request can carry the relevant information of the fields to be extended, so as to query the extended fields related to the piece of source data from the service system and add them to the source data. The field expansion solution provided in the embodiments of this specification can avoid the complex multi-stream join network, reduce resource consumption, and can flexibly query and expand fields according to specific requirements, improving the flexibility of field expansion.

[0083] In some embodiments, considering that the degradation feature values corresponding to the real-time feature values in different time slices are different, in order to facilitate quick and efficient query of the degradation feature values, the degradation feature values can be stored in the form of main key - secondary key - value. Among them, the main key is the standard feature name of the real-time feature, the secondary key is the time slice corresponding to the real-time feature value, and the value is the degradation feature value. For example, the real-time feature is the number of orders in Business District A in the past 10 minutes. For the real-time feature values of this real-time feature in different time slices, the average value of the historical values in the same time slice in the past 7 days can be used as the degradation feature value of this real-time feature value. For example, taking the current time as 10:30 on January 31, 2025 as an example, the past 10 minutes is from 10:20 to 10:30 on January 31, 2025. For this time slice, the average value of the number of orders in Business District A in the same time slice in the past 7 days is 1000. Therefore, when storing this degradation feature value, it can be stored in the form of main key - secondary key - value. Among them, the main key is the standard feature name of the real-time feature, the secondary key is 10:20 - 10:30, and the value is 1000.

[0084] In some embodiments, in order to facilitate the management of real-time features, each real-time feature can include two states, a degradation state and a non-degradation state. Among them, if the current real-time feature value of the real-time feature does not have a delay, the state of the real-time feature is the non-degradation state. When in the non-degradation state, if a query request is received, the current real-time feature value can be directly returned. If the real-time feature has a delay in the current real-time feature value, the state of the real-time feature can be switched to the degradation state. When the state of the real-time feature is the degradation state, it means that the current real-time feature value is unavailable. At this time, if a query request is received, the degradation feature value corresponding to the current real-time feature value can be queried and the degradation feature value can be returned. Among them, the switching of the degradation state can be manually switched by the user or automatically switched by the real-time feature management platform. For example, in some embodiments, for each real-time feature, a state switching button can be set in the interaction interface. If the user finds that a certain real-time feature has a delay, a degradation instruction can be issued through this state switching button to switch the state of this real-time feature to the degradation state. Considering the scenario where the number of real-time features is large, the work of manually maintaining each real-time feature is heavy, consuming a lot of manpower and having low efficiency. Therefore, in some embodiments, the real-time feature management platform can also automatically monitor the output queue of the real-time feature. If the output queue has not output a new real-time feature value for more than a preset duration, the state of this real-time feature is automatically switched to the degradation state.

[0085] In some embodiments, the real-time feature is a real-time feature in the instant delivery scenario. The task information further includes the feature standard name of the real-time feature. When storing the real-time feature value of the real-time feature, the real-time feature value can be stored in the online query database in the key-value form to provide an online query service. Among them, the key carries the feature standard name of the real-time feature and the geographical fence identifier corresponding to the real-time feature, and the value is the real-time feature value. Among them, considering that real-time features are usually used by the online service system to provide real-time decisions. To facilitate the online service system to query the real-time feature in real time, the real-time feature can be stored in the online query database in the key-value form to provide an online query service. Among them, the online query database can be a database that can provide real-time data query functions, such as Redis database, Lindorm database, etc. For the real-time feature in the instant delivery scenario, since it is strongly correlated with time and space, to facilitate users to query the real-time feature based on spatial information, the key can carry the feature standard name of the real-time feature and the geographical fence identifier corresponding to the real-time feature. For example, the feature standard name and the geographical fence identifier can be concatenated as the key of the real-time feature. Among them, the validity period of the real-time feature can be set in the online query database. For example, 15 minutes, that is, the default validity period of each real-time feature value of the real-time feature is 15 minutes, and the real-time feature value becomes invalid after 15 minutes.

[0086] In some embodiments, the real-time feature values of the real-time feature in different time slices can also be written into ODPS to provide an offline query service. For example, users can obtain the offline real-time feature values through the ODPS table to perform tasks such as offline verification, analysis, and model training.

[0087] In some scenarios, such as Figure 3 As shown, considering that real-time feature values are continuously generated, when storing real-time feature values, the generated real-time feature values can be pushed to the message queue, and the Flink task consumes the message queue and writes them into TT (real-time data bus) and Redis database. Among them, the real-time feature values written into TT will be written into the ODPS table through the synchronization task for users to consume offline. The real-time feature values written into the Redis database can be stored in the key-value form for users to consume online. Among them, the key is obtained by concatenating the feature standard name and the geographical fence identifier, and the value is the real-time feature value. Among them, the real-time feature values in the Redis database are continuously updated, that is, the next real-time feature value can continuously overwrite the previous real-time feature value.

[0088] In some embodiments, the real-time feature management platform can provide an external real-time feature query capability. For example, it can receive a query request that carries the geofence identifier corresponding to the real-time feature to be queried, and then query the corresponding real-time feature value from the online query database based on the geofence identifier and return it.

[0089] In some embodiments, the query request can also carry geographical location information. After receiving the query request, the geographical location information can be converted into a geofence identifier, and then the corresponding real-time feature value can be queried from the online query database based on the geofence identifier and returned.

[0090] That is, the real-time feature management platform can support batch obtaining of real-time feature values through a list of geofence identifiers, and also support obtaining real-time feature values through coordinate points (i.e., geographical location information) and geofence types. When the user passes in coordinate points, the longitude and latitude of the coordinates can be converted into corresponding geofence identifiers through spatial determination, and then the online query database can be queried to obtain the real-time feature values and return them to the user.

[0091] Taking the instant delivery scenario as an example below, a specific application scenario is combined to introduce the real-time feature management method provided in the embodiments of this specification.

[0092] This embodiment provides a real-time feature management platform, which can include three modules: data warehouse construction, real-time feature generation and management, and real-time feature query service, so as to provide a complete solution from data cleaning, real-time feature generation and management to real-time feature consumption. Each module is introduced separately as follows:

[0093] 1. Data warehouse construction

[0094] (1) Data sources

[0095] The sources of data are diverse and can be waybill messages sent by service systems, location messages reported by riders, scheduling log buried points, etc. The forms can be various message queues and sls logs. Different data sources in different service systems can be cleaned and constructed through standardization and finally flow into the data warehouse for real-time feature generation.

[0096] (2) Field extension

[0097] During the real-time feature generation process, in addition to the information carried on a single data source, some additional information is often required. For example, the rider source carries the rider's location information, indoor / outdoor information, etc., but when generating real-time features, it is also desired to obtain the information about the orders carried by the rider at the same time, but such information is not in the rider source; another example is that the waybill source carries the status of the waybill and the location information of the user and merchant, but lacks the corresponding geofence information, and the corresponding geofence identifier needs to be marked according to the location information.

[0098] The core problem faced in the above scenario is how to implement field expansion. The common solution in the industry is to use the multi-stream join ability of Flink. However, the multi-stream join of Flink has problems such as complex computing network structure, long link, difficult to maintain, large resource consumption, etc., and there is also a risk of out-of-order messages. Therefore, in this embodiment, a self-developed tagging system is used to implement the field expansion of the data source. The tagging system can listen to the message queue corresponding to the data source. When receiving each message, it queries the pre-stored data dimension table according to the message content to obtain the extended fields, and then adds them to the message and sends them to Kafka. Alternatively, each service system providing source data can also provide an interface. The tagging system can send a query request to the service system based on this interface. The query request can carry relevant information such as the identifier of the extended field, so that the service system can return the extended field. The tagging system can add the extended field to the message and send it to Kafka.

[0099] (3) Data warehouse storage

[0100] When generating real-time features, the real-time feature management platform provided in this embodiment can support multiple calculation methods. In order to support multiple calculation methods, when storing source data, multiple storage types can also be selected to meet the requirements of different scenarios.

[0101] According to different application scenarios, different storage types can be used. For the micro-batch calculation scenario, hologres is selected as the storage type. For graph calculation, it is IGraph. For streaming calculation, it is the message queue. By combining different storage types, the platform can provide diverse feature production methods to adapt to different scenarios of users.

[0102] 2. Generation and management of real-time features

[0103] (1) Metadata management

[0104] For the instant delivery scenario, based on the characteristics of the real-time features of this scenario, a set of metadata can be set in advance, and each real-time feature can be described and managed through this set of metadata:

[0105] Feature caliber description: Chinese description of the feature screening method and aggregation method.

[0106] Aggregation window and subject type: Since the real-time features relied on in the instant delivery scenario are strongly related to time and space, each feature corresponds to an aggregation window and a subject type. Among them, the aggregation window refers to the length of the feature calculation time window, and the subject type refers to the aggregation subject of the feature, that is, "using the id of what subject to query the feature". The subject mainly includes various geographical fences, such as cities, business districts, grids, aois, etc.

[0107] Feature standard name: The unique query English key for a feature, with a unified naming convention, containing a subject type prefix and an aggregation window suffix. For example, s1_create_cnt_m10, which means the order creation volume in a certain crowdsourcing business district in the past ten minutes. Here, s1 is the subject type prefix and m10 is the aggregation window suffix.

[0108] Degradation strategy: When problems such as delays occur in the feature production link, resulting in the unavailability of real-time features, users are supported to degrade real-time features to historical features through custom degradation strategies. The supported degradation strategies include historical values of the same time slice and the same subject id in the past day, the average value of the same time slice and the same subject id in the past seven days, etc. For some real-time features that are not suitable for degradation, such as the real-time location of riders, users are also supported to select non-degradable.

[0109] (2) Feature generation

[0110] As Figure 4 shown, the feature management platform can provide an interactive interface. Users can create real-time feature generation tasks in the interactive interface and enter task information, which can include feature caliber, feature calculation method, degradation strategy, SLA signing (i.e., the fault level caused by the unavailability of real-time features), etc. The platform side can review the information entered by users. In order to unify the names of real-time features for standardized unified management of real-time features, the platform side can determine the feature standard name for the real-time feature, enter it, and then prepare data sources, etc. for the task. Users can enter scripts for generating real-time features in the interactive interface based on the feature calculation method. The platform side can review the scripts. After the review is passed, the task can be launched.

[0111] The platform provides three feature calculation methods: micro-batch calculation, graph calculation, and stream calculation. The following is an introduction to these three calculation methods:

[0112] Micro-batch calculation: Users enter the SQL for feature production and the python script for post-processing the SQL results on the platform. The scheduling module of the platform triggers a scheduling task at the minute level, executes the user's SQL and python script in the hologres computing engine, and pushes the calculated features to storage;

[0113] Graph calculation: IGraph is an online graph query engine service that supports Gremlin and PG query languages and provides stable and low-latency online query and update services. Graph calculation is based on the graph query ability of IGraph. Users configure Gremlin query scripts on the platform, and the query scripts will be executed during feature query and the results will be returned to users;

[0114] Stream computing: Stream computing is based on the Flink computing engine. Users write Flink SQL and Python custom functions on the platform. The platform assembles the user's script and data source declaration into a complete pyflink script, starts Flink to execute the user's feature calculation logic, and pushes the calculation results to storage.

[0115] (3) Feature storage

[0116] The calculated real-time features are pushed to the message queue, consumed by the Flink task, and written to TT and Redis. The feature data written to TT is written to the ODPS table through the synchronization task for offline consumption. Features for online consumption are written to Redis in key-value format, where the key is the concatenation of the feature standard name and the subject ID, and the value is the feature value. The feature lifecycle in Redis can be configured as needed and is set to 15 minutes by default.

[0117] 3. Real-time feature query service

[0118] (1) Online consumption

[0119] The platform provides real-time feature query capabilities, supports batch acquisition of real-time features through subject ID lists, and also supports acquisition of real-time features through coordinate points and subject types. When the user enters the coordinate point, the coordinate longitude and latitude will first be converted into the corresponding subject ID through spatial determination, and then Redis will be queried to obtain the real-time features and returned to the user.

[0120] (2) Offline consumption

[0121] The feature data written into TT will be written into the ODPS table through the synchronization task. Users can use the ODPS table to obtain offline features for offline verification, analysis, and model training.

[0122] (3) Feature degradation

[0123] The default declaration period for real-time features is 15 minutes. When a delay occurs in the feature production link, causing the writing of real-time features to exceed 15 minutes, the real-time features in Redis will expire. At this time, users will receive null values when querying. Such null values are often unacceptable to the model and will greatly affect the output of the model, thereby affecting the normal operation of the service system. To avoid the above problems, this embodiment provides a complete set of real-time data degradation strategies, including the production of degraded features, manual degradation, and automatic degradation.

[0124] i. Degrade data production

[0125] The platform will calculate and automatically generate degradation features based on historical data according to the degradation strategy set by the user. The degradation features can be updated at the daily level. After the degradation features are generated, they will be returned to the IGraph table through a synchronization task in the structure of main key - sub key - value. Among them, the main key is the standard name of the feature, the sub key is the time slice, and in the degradation state, the platform will query IGraph to obtain degradation data;

[0126] ii. Manual degradation

[0127] The platform realizes manual degradation through configuration distribution. When a certain real - time feature is unavailable, it can be modified through configuration to set the status of the corresponding real - time feature to the degradation state, so as not to return the real real - time feature to the user, but the degradation feature customized by the user, thus ensuring the stability of the online system.

[0128] iii. Automatic degradation

[0129] With the increase in the number of feature levels and production tasks, manual operation and maintenance of each real - time feature has become a heavy task. Therefore, a set of automatic degradation links is provided. Based on the CEP feature of Flink, the output source of real - time features is monitored. If no record is generated for a certain real - time feature within 14 minutes, a signal will be automatically sent to the platform. After receiving the signal, the platform can automatically set the status of the corresponding feature to the degradation state, so that when a query request is received, the degradation feature is returned to the user.

[0130] This embodiment conducts targeted modeling according to the characteristics of real - time features required in the field of instant delivery. By setting metadata to describe real - time features, it better meets the needs of the real - time delivery scenario and is also convenient for unified standardized management of real - time features. In addition, the platform uniformly standardizes and cleans the data source, provides standardized feature production capabilities externally, and the platform will uniformly control the online and change of features. Features developed by different users on the platform can be shared and reused, which can significantly reduce the data production cost. This embodiment also provides a complete data degradation scheme, supports users to customize degradation strategies, and flexibly switches between real - time features and degradation features through a combination of manual degradation and automatic degradation to ensure the stability of the online system.

[0131] Corresponding to the memory allocation method embodiment provided in this specification, this specification embodiment also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method mentioned in any of the above embodiments.

[0132] This specification embodiment also provides an electronic device, as Figure 5 shown, which is the structural schematic diagram of the electronic device of this specification embodiment. Except for Figure 5In addition to the illustrated processor 52 and memory 54, the device may generally further include other hardware, such as a forwarding chip responsible for processing packets, etc. In terms of the hardware structure, the device may also be a distributed device and may include multiple interface cards for expanding packet processing at the hardware level. The memory 54 stores computer instructions, and when the processor 52 executes the computer instructions, the methods mentioned in any of the above embodiments are implemented.

[0133] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0134] Since the part of the embodiments of this specification that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium and includes several instructions for causing a terminal device to execute all or part of the steps of the methods of the embodiments of this specification. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0135] The above are only the preferred embodiments of the embodiments of this specification and are not intended to limit the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the embodiments of this specification shall be included within the scope of protection of the embodiments of this specification.

Claims

1. A real-time feature management method, the method comprising: Providing an interactive interface for a user to create a real-time feature generation task through the interactive interface and configure task information of the real-time feature generation task, where the task information at least includes generation mode description information of the real-time feature to be generated and a degradation policy; Determining a script for generating the real-time feature based on the generation mode description information; Running the script once every preset time to query the latest source data related to the real-time feature from a data warehouse, and processing the queried latest source data to obtain a real-time feature value of the real-time feature and storing it; wherein, the source data stored in the data warehouse is updated in real time; Determining and storing a degradation feature value corresponding to the real-time feature value based on the degradation policy; When receiving a query request for querying the real-time feature, if it is determined that the real-time feature value is unavailable, returning the degradation feature value.

2. The method according to claim 1, characterized in that The script includes a query script for querying the latest source data related to the real-time feature from the data warehouse, and a processing script for processing the queried latest source data; The generation mode description information includes the query script and the processing script; or The generation mode description information includes a feature caliber, where the feature caliber is used to describe a screening strategy for screening source data related to the real-time feature from the data warehouse and a processing strategy for processing the screened source data, generating the query script based on the screening strategy, and generating the processing script based on the processing strategy.

3. The method according to claim 2, wherein The processing strategy includes a calculation method for calculating the queried latest source data, and the calculation method includes one or more of the following: Micro-batch calculation, graph calculation, and streaming calculation.

4. The method according to claim 3, characterized in that, The storage type of the source data in the data warehouse is determined based on the calculation method. Wherein, in the scenario where the calculation method is micro-batch calculation, the storage type is hologres; in the scenario where the calculation method is graph calculation, the storage type is IGraph; in the scenario where the calculation method is streaming calculation, the storage type is a message queue.

5. The method according to claim 2, wherein The real-time feature is a real-time feature in an instant delivery scenario, and the processing strategy includes a calculation time window length corresponding to the real-time feature and a geographic fence identifier.

6. The method according to claim 1, wherein The source data stored in the data warehouse is obtained based on the following method: Listening to source data queues uploaded by each data source; When obtaining each piece of source data, querying an extended field associated with the piece of source data from a preset data dimension table or system based on the content of the piece of source data; Adding the extended field to the piece of source data as the extended source data, and storing it in the data warehouse.

7. The method according to claim 1, characterized in that The degradation feature value is stored in the form of main key - secondary key - value, where the main key is the feature standard name of the real-time feature, the secondary key is the time slice corresponding to the real-time feature value, and the value is the degradation feature value.

8. The method according to claim 1, characterized in that When the status of the real-time feature is in a degraded state, it is determined that the real-time feature value is unavailable, where the degraded state is determined based on the following method: Monitor the output queue of the real-time feature. If no new real-time feature value is output from the output queue for a preset duration, switch the status of the real-time feature to the degraded status; and / or When receiving a degradation instruction for the real-time feature sent by the user through the interaction interface, switch the status of the real-time feature to the degraded status.

9. The method according to claim 1, wherein The real-time feature is a real-time feature in the instant delivery scenario, and the task information further includes the feature standard name of the real-time feature. Obtaining and storing the real-time feature value of the real-time feature includes: Storing the real-time feature value in the online query database in the key-value form to provide an online query service; wherein, the key carries the feature standard name of the real-time feature and the geographical fence identifier corresponding to the real-time feature, and the value is the real-time feature value; and / or Writing the real-time feature value into the OPDS to provide an offline query service.

10. The method according to claim 9, characterized in that The method further includes: Receiving a query request, where the query request carries the geographical fence identifier corresponding to the real-time feature to be queried; Querying the corresponding real-time feature value from the online query database based on the geographical fence identifier and returning it; Or Receiving a query request, where the query request carries geographical location information; Converting the geographical location information into a geographical fence identifier; Querying the corresponding real-time feature value from the online query database based on the geographical fence identifier and returning it.

Images