Offline feature fusion method and device, electronic equipment and storage medium

Abstract

The application provides an offline feature fusion method and device, electronic equipment and storage medium, and applies to the field of data processing. The method comprises the following steps: obtaining a pre-established feature configuration table corresponding to at least one machine learning model, wherein the feature configuration table at least comprises a model identifier representing the machine learning model, an offline feature identifier representing the offline feature, a hive data table identifier representing a hive data table to which the offline feature belongs, and a feature association primary key corresponding to the offline feature stored in the hive data table. According to the offline feature identifier and the hive data table identifier in the feature configuration table corresponding to the machine learning model, the offline feature is extracted from the hive data table to which the offline feature belongs. The offline feature comprises the feature association primary key corresponding to the offline feature, the offline feature identifier and the offline feature value. According to the feature association primary key, the extracted offline feature corresponding to at least one machine learning model is fused to obtain a fused offline feature.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an offline feature fusion method, apparatus, electronic device and storage medium. Background Technology

[0002] With the advent of the artificial intelligence era, machine learning models have become the mathematical engine of AI. They perform reasoning and predictions by processing massive amounts of data. Data scientists and algorithm scientists build different machine learning models based on application scenarios, and then use large amounts of sample data to repeatedly adjust and verify the effectiveness and accuracy of the models, ultimately enabling them to make accurate predictions about specific real-world situations.

[0003] Currently, the sample features used to train machine learning models are generally calculated by big data computing engines such as Hive. Different offline features are first calculated through different Hive tasks and then stored in a cache for use during model training. Since offline features calculated by different Hive tasks are stored in different Hive tables, the multiple offline features required by a particular machine learning model are often stored in different Hive tables. Therefore, to train the machine learning model, it is first necessary to extract the required offline features from different Hive tables, then combine these features to form multiple training samples, and finally input each training sample into the machine learning model sequentially.

[0004] Existing offline feature extraction methods first determine all Hive tables to which the required offline features belong, then directly associate these Hive tables to form a new Hive table (a wide table), and finally read the associated data from the wide table and write it into a cache for training. In this method, the association operations between different Hive tables are computationally intensive and slow. Furthermore, the associated feature data must first be stored in the wide table and then read and written to the cache, which is time-consuming and wastes storage and computational resources. Therefore, how to extract offline features required by machine learning models more efficiently has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide an offline feature fusion method, apparatus, electronic device, and storage medium. By pre-configuring a feature configuration table for the machine learning model, multiple Hive tables are directly accessed to extract the required offline features. Then, the extracted offline features are fused according to the feature association primary key to obtain fused offline features. This allows the fused offline features to be used directly to train the machine learning model. This method achieves the goal of flexibly extracting offline features. Simultaneously, the result of feature association fusion is directly written to a cache, avoiding wasted storage resources due to multiple writes.

[0006] The first aspect of this application provides an offline feature fusion method, which includes:

[0007] Obtain at least one pre-built feature configuration table corresponding to a machine learning model. The feature configuration table shall include at least the model identifier representing the machine learning model, the offline feature identifier representing the offline features, the Hive table identifier representing the Hive table to which the offline features belong, and the feature association primary key corresponding to the offline features stored in the Hive table.

[0008] Based on the offline feature identifier and Hive table identifier in the feature configuration table corresponding to at least one machine learning model, extract offline features from the Hive data table to which the offline features belong. Offline features include the feature association primary key, offline feature identifier, and offline feature value.

[0009] Based on the feature association primary key corresponding to the offline features, feature fusion is performed on the offline features corresponding to at least one extracted machine learning model to obtain fused offline features.

[0010] A second aspect of this application provides an offline feature fusion apparatus, which includes:

[0011] The acquisition unit is used to acquire at least one pre-established feature configuration table corresponding to a machine learning model. The feature configuration table includes at least a model identifier representing the machine learning model, an offline feature identifier representing the offline features, a Hive table identifier representing the Hive table to which the offline features belong, and the feature association primary key corresponding to the offline features stored in the Hive table.

[0012] The determining unit is used to extract offline features from the Hive data table to which the offline features belong, based on the offline feature identifier and Hive data table identifier in the feature configuration table corresponding to at least one machine learning model. The offline features include the feature association primary key, the offline feature identifier, and the offline feature value.

[0013] The processing unit is used to perform feature fusion on the offline features corresponding to at least one machine learning model based on the feature association primary key corresponding to the offline features, so as to obtain fused offline features.

[0014] A third aspect of this application also provides an electronic device, including: a memory and a processor, wherein the memory and the processor are coupled.

[0015] The memory is used to store one or more computer instructions.

[0016] The processor is used to execute one or more computer instructions to implement the offline feature fusion method described in the first aspect above.

[0017] A fourth aspect of this application also provides a computer-readable storage medium storing one or more computer instructions, characterized in that the instructions are executed by a processor to implement the offline feature fusion method described in any of the above technical solutions.

[0018] Compared with the prior art, the embodiments of this application have the following advantages:

[0019] The offline feature fusion method provided in this application first pre-establishes a feature configuration table based on the input of the machine learning model to be trained. This feature configuration table includes various information about the offline features required by the machine learning model, including the offline feature identifier, the Hive table identifier representing the Hive table to which the offline feature belongs, and the feature association key corresponding to the offline feature in that Hive table. This allows direct access to the original Hive table based on the feature configuration table, directly extracting offline features from the Hive table to which the offline features belong, and obtaining the feature association key, offline feature identifier, and offline feature value corresponding to the offline feature. Finally, based on the feature association key corresponding to each offline feature, multiple offline features are fused to form multiple feature samples for training the machine learning model. It is understandable that by using the feature configuration table, a single offline feature can be directly accessed from the Hive table, achieving the goal of flexible extraction of offline features. When offline features are added or deleted, only the extraction information in the feature configuration table needs to be modified. This greatly improves the reusability of the original features while efficiently providing the offline features required by the machine learning model. In addition, after obtaining multiple offline features, the offline features are directly fused and the fused result is directly written into the cache. This avoids the operation of fusion calculation between multiple Hive tables to obtain a wide table. At the same time, there is no need to first store the fused offline features in the wide table and then read the data from the wide table to write it into the cache, thereby saving computing and storage resources. The time consumed by the feature fusion process will also be greatly shortened. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating an offline feature fusion method provided in an embodiment of this application;

[0022] Figure 2 A flowchart illustrating another offline feature fusion method provided in this application embodiment;

[0023] Figure 3 A flowchart illustrating the training process of the machine learning model provided in the embodiments of this application;

[0024] Figure 4 This is a schematic diagram of the structure of an offline feature fusion device provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] This application provides an offline feature fusion method, apparatus, electronic device, and storage medium. By pre-configuring a feature configuration table for the machine learning model, multiple Hive tables are directly accessed to extract the required offline features. Then, the extracted offline features are fused according to the feature association primary key to obtain fused offline features. This allows the fused offline features to be used directly to train the machine learning model. This method achieves the goal of flexibly extracting offline features. Simultaneously, the result of feature association fusion is directly written to a cache, avoiding wasted storage resources due to multiple writes.

[0027] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described above. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar roles and are not intended to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] With the advent of the artificial intelligence era, machine learning models have become the mathematical engine of AI. They perform reasoning and predictions by processing massive amounts of data. Data scientists and algorithm scientists build different machine learning models based on application scenarios, and then use large amounts of sample data to repeatedly adjust and verify the effectiveness and accuracy of the models, ultimately enabling them to make accurate predictions about specific real-world situations.

[0030] Currently, the sample features used to train machine learning models are generally calculated by big data computing engines such as Hive. Different offline features are first calculated through different Hive tasks and then stored in a cache for use during model training. Since offline features calculated by different Hive tasks are stored in different Hive tables, the multiple offline features required by a particular machine learning model are often stored in different Hive tables. Therefore, to train the machine learning model, it is first necessary to extract the required offline features from different Hive tables, then combine these features to form multiple training samples, and finally input each training sample into the machine learning model sequentially.

[0031] Existing offline feature extraction methods first determine all Hive tables to which the required offline features belong, then directly associate these Hive tables to form a new Hive table (a wide table), and finally read the associated data from the wide table and write it into a cache for training. In this method, the association operations between different Hive tables are computationally intensive and slow. Furthermore, whenever a new feature is added or an old feature is removed, the structure of the wide table needs to be modified by adding or removing fields, resulting in high operational costs. On the other hand, because each machine learning model requires its own offline feature data, the resulting wide tables are unique and cannot be reused. Moreover, the associated feature data must first be stored in the wide table and then read and written to the cache, which is time-consuming and wastes storage and computational resources. Therefore, how to extract the offline features required by machine learning models more efficiently has become an urgent problem to be solved.

[0032] To address the aforementioned problems, this application provides an offline feature fusion method, apparatus, electronic device, and storage medium. First, a feature configuration table is pre-established based on the output requirements of the machine learning model to be trained. This allows direct access to the original Hive table based on the feature configuration table, enabling direct extraction of offline features from the Hive data table containing the offline features. Finally, the offline features are fused to form multiple fused offline features for training the machine learning model. This allows direct access to the Hive table to obtain individual offline features, achieving flexible extraction of offline features. After obtaining multiple offline features, they are directly fused without the need for fusion between Hive tables. This significantly saves computational and storage resources while reducing the time required for the feature fusion process. The method, apparatus, terminal, and computer-readable storage medium of this application will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0033] Figure 1 This is a flowchart illustrating an offline feature fusion method provided in an embodiment of this application. It should be noted that the steps shown in this flowchart can be executed in a computer system such as a set of computer-executable instructions, and in some cases, the released steps can be executed in a logical order different from that shown in the flowchart.

[0034] like Figure 1 As shown, this offline feature fusion method includes the following steps:

[0035] 101. Obtain the pre-established feature configuration table corresponding to the machine learning model.

[0036] Machine learning models typically determine an output based on multiple input features. For example, a machine learning model for recommending game items would take user item purchase and usage features as inputs. When using this model to obtain item recommendations, both purchase and usage features must be used as inputs. Building a machine learning model requires extensive training with a large amount of sample data (training features), repeatedly adjusting and validating its effectiveness and accuracy to better reflect reality and make accurate predictions. Therefore, training this model also requires using existing user item purchase and usage features.

[0037] However, the existing features are all obtained through different Hive tasks, and these features are stored in different Hive tables. For example, in the above example, user item purchase feature data and user item usage feature data are stored in different Hive tables. Therefore, when training a machine learning model, the offline features from different Hive tables need to be fused to obtain fused offline features.

[0038] Offline features, as opposed to real-time features, are feature data that has been calculated using big data computing engines like Hive over a period of time and stored in Hive tables. For example, T+1 offline features refer to feature data calculated using data from the previous day. These offline features can be stored in different Hive tables according to their computational convenience, and all the Hive tables together form an offline feature database (a collection of Hive tables).

[0039] Table 1 illustrates the structure of a Hive data table. Each Hive data table contains feature values ​​for multiple features. The Hive data table stores feature values ​​for different offline features based on offline feature identifiers. These offline feature identifiers can be feature names or feature symbols, without limitation. As shown in Table 1, this Hive data table includes feature association identifiers, offline feature identifiers, and feature values. The feature association identifier is the feature association primary key, used to link different offline features. For example, in the above example, the feature association primary key could be a user account name, thus linking user item purchase feature data and user item usage feature data belonging to the same user using the user account name. This Hive data table includes three offline features: Feature 1, Feature 2, and Feature 3, and various corresponding feature values. Each feature value corresponds to a specific offline feature.

[0040] Table 1

[0041]

[0042] This application embodiment utilizes a feature configuration table to extract the offline features required by a machine learning model from numerous offline features in an offline feature database. The process of establishing this feature configuration table involves first determining at least one offline feature required by the machine learning model based on its input, along with its associated primary key. Then, an offline feature identifier is determined for each offline feature. Based on the offline feature identifier, the Hive data table to which each offline feature belongs in the offline feature database is determined, and then the target column to which each offline feature belongs in its Hive data table is determined. Finally, the offline feature identifier, its Hive data table, its target column in the Hive data table, and its associated primary key are recorded for each offline feature, resulting in the feature configuration table corresponding to the machine learning model.

[0043] In other words, the feature configuration table should at least include a model identifier representing the machine learning model (e.g., the model name), an offline feature identifier representing the offline feature (e.g., the feature name), a Hive table identifier representing the Hive table to which the offline feature belongs (e.g., the Hive table name), and the feature association primary key corresponding to the offline feature stored in the Hive table. The feature association primary key is created within the Hive table and is used to query the content stored in the Hive table; it's similar to an index and identifies specific stored data within the Hive table.

[0044] In the example above, the feature configuration table needs to include the model name of the machine learning model, the feature name of the user item purchase feature data, the feature name of the user item usage feature, the Hive table name of the Hive data table to which the user item purchase feature data belongs, the Hive table name of the Hive data table to which the user item usage feature data belongs, and the feature association primary key such as user account information. Only in this way can the feature configuration table be used to query the various Hive data tables in the offline feature database to obtain offline features.

[0045] The feature configuration table indicates the correspondence between the machine learning model to be trained and the Hive data table, recording the methods for extracting target offline features from the offline feature database. The feature configuration table can also include the preprocessing method for each offline feature. That is, after obtaining the offline features according to the contents of the feature configuration table, the feature values ​​of each offline feature need to be preprocessed according to the corresponding preprocessing method before the subsequent offline feature fusion stage. The preprocessing methods for offline features include, but are not limited to, standardization and feature encoding, which will not be elaborated upon here.

[0046] 102. Extract offline features from the Hive data table to which the offline features belong, based on the offline feature identifier and the Hive data table identifier in the feature configuration table.

[0047] After establishing the feature configuration table in advance, when training a machine learning model, the feature configuration table is used to extract offline features to form training samples for training the machine learning model. When extracting relevant offline features, the corresponding Hive table is first located based on the Hive table identifier in the feature configuration table. Then, in that Hive table, a column of features corresponding to the target offline feature is determined based on the offline feature identifier, resulting in a column of offline feature values. Each offline feature value corresponds to a specific training sample. For example, suppose the Hive table identifier recorded in the feature configuration table corresponding to the machine learning model includes the table name of the Hive table shown in Table 1, and the recorded offline feature identifier includes the identifier of feature 2. Then, the Hive table shown in Table 1 needs to be located first, and the third column of data corresponding to feature 2 needs to be determined in that Hive table. The third column of data in that Hive table then needs to be extracted as the offline feature corresponding to the machine learning model. Each offline feature includes the feature association primary key, the offline feature identifier, and the offline feature value.

[0048] 103. Based on the feature association primary key corresponding to the offline features, perform feature fusion on the offline features corresponding to the extracted machine learning model to obtain fused offline features.

[0049] After obtaining a large number of offline features, feature fusion is needed to obtain fused offline features corresponding to the extracted machine learning model. These fused offline features serve as input to the machine learning model and can be used as input data in the training samples to train the model. For example, in the above example, to train a machine learning model that outputs user item recommendation results, it is necessary to determine a first Hive data table storing user item purchase features and a second Hive data table storing user item usage features based on the feature configuration table. Then, the data corresponding to the user item usage features in the first Hive data table and the data corresponding to the user item purchase features in the second Hive data table are determined. Next, each extracted data point corresponds to a feature association primary key, i.e., user account information. For example, each data point extracted from the first Hive data table is the feature value of the user item usage feature for a specific user, and each data point extracted from the second Hive data table is the feature value of the user item purchase feature for a specific user. Finally, the offline features obtained from the two data tables are associated based on the user account information (feature association primary key), thus obtaining the feature values ​​of the user item usage feature and the user item purchase feature for a specific account. In this way, the feature values ​​of a user's item usage and purchase characteristics can be input into the machine learning model as a fused offline feature.

[0050] Understandably, if the structure of the machine learning model is updated and the input changes, then only the feature configuration table needs to be modified. First, determine the offline feature identifier of the feature to be updated. Then, determine the identifier of the Hive table to which the offline feature identifier belongs in the offline feature database, and the target column of the feature to be updated within that Hive table. Finally, update the feature configuration table corresponding to the machine learning model based on this information.

[0051] For example, if the offline feature identifier to be updated is a newly added offline feature identifier, then you only need to add the Hive table identifier of the Hive table to the feature configuration table. You can also add information about the target column. Conversely, if the offline feature identifier to be updated is an offline feature identifier to be deleted, then delete the Hive table identifier of the Hive table to which that offline feature belongs. Then, based on the updated feature configuration table, you can re-extract the offline feature. This achieves flexible extraction of offline features, not only efficiently obtaining the offline features needed for machine learning models but also greatly improving the reusability of the original features.

[0052] The offline feature fusion method provided in this application first pre-establishes a feature configuration table based on the input of the machine learning model to be trained. This feature configuration table includes various information about the offline features required by the machine learning model, including the offline feature identifier, the Hive table identifier representing the Hive table to which the offline feature belongs, and the feature association key corresponding to the offline feature in that Hive table. This allows direct access to the original Hive table based on the feature configuration table, directly extracting offline features from the Hive table to which the offline features belong, and obtaining the feature association key, offline feature identifier, and offline feature value corresponding to the offline feature. Finally, based on the feature association key corresponding to each offline feature, multiple offline features are fused to form multiple feature samples for training the machine learning model. It is understandable that by using the feature configuration table, a single offline feature can be directly accessed from the Hive table, achieving the goal of flexible extraction of offline features. When offline features are added or deleted, only the extraction information in the feature configuration table needs to be modified. This greatly improves the reusability of the original features while efficiently providing the offline features required by the machine learning model. In addition, after obtaining multiple offline features, the offline features are directly fused and the fused result is directly written into the cache. This avoids the operation of fusion calculation between multiple Hive tables to obtain a wide table. At the same time, there is no need to first store the fused offline features in the wide table and then read the data from the wide table to write it into the cache, thereby saving computing and storage resources. The time consumed by the feature fusion process will also be greatly shortened.

[0053] Based on the methods described above, the process of extracting offline features and fusing offline features will be described in detail below. Figure 2 This is a flowchart illustrating another offline feature fusion method provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0054] 201. Based on the offline feature identifier and Hive data table identifier in the feature configuration table, extract the offline feature from the Hive data table to which the offline feature belongs.

[0055] The feature configuration table records the Hive data table identifiers of the offline features required by the machine learning model. Therefore, the offline features can be obtained simply by accessing the corresponding Hive data table based on the feature configuration identifier. In this embodiment, training samples can be acquired using the Spark computing engine. First, the feature configuration table information of a machine learning model is read row by row in Spark, and the information of each feature is obtained sequentially row by row. Then, the feature values ​​of the corresponding offline features are obtained from the corresponding Hive data table. Each time an offline feature is obtained, it is stored in the form of "feature association primary key, offline feature identifier, offline feature value". It is understandable that if the feature configuration table also includes the preprocessing method for offline features, the offline feature values ​​are first processed according to the preprocessing method, and then the processed feature values ​​are stored in the above form.

[0056] 202. Combine the extracted offline features into a vertical table arranged by row.

[0057] In this vertical table, each row stores an offline feature; that is, each row's data includes the feature association primary key, offline feature identifier, and offline feature value corresponding to an offline feature. Table 2 shows the format of the vertical table, which is: feature association primary key, offline feature identifier, and offline feature value. For example, in... Figure 1 In the example shown in the embodiment, the first row of data can be the feature value of the user item purchase feature corresponding to user 1, the second row of data can be the feature value of the user item usage feature corresponding to user 2, the third row of data can be the feature value of the user item purchase feature corresponding to user 3, and so on. That is, each row of data represents an offline feature, which serves as the basis for subsequent feature fusion.

[0058] Table 2

[0059]

[0060] In this process, if an offline feature is stored repeatedly in different Hive data tables, the same offline feature will be recorded repeatedly when extracting corresponding data based on the feature configuration table. In this case, different deduplication operations need to be performed depending on the situation, that is, feature aggregation processing of the samples. For example, two offline features with the same offline feature identifier and feature association primary key can be identified, and then the two features can be filtered according to their feature type. For example, if the offline feature is a user level feature, then the higher user level should be selected for the same player; that is, whichever offline feature records the higher user level should be kept, and the other offline feature should be deleted.

[0061] For example, if the offline feature is the online duration feature, then it is necessary to merge the two offline features. That is, to sum the feature values ​​of the two offline features with the same offline feature identifier and feature association primary key, and merge the two identical offline features into one.

[0062] 203. Based on the primary key associated with the features, perform row and column transformation on the vertical table to obtain a horizontal table with offline features arranged in columns.

[0063] After recording all extracted offline features using a vertical table, the vertical table needs to be transformed into a horizontal table with the offline features arranged in columns. Each row of each horizontal table stores the offline feature values ​​corresponding to all offline feature identifiers associated with a feature-associated primary key.

[0064] For example, the contents of the vertical table can be grouped according to the feature association primary key, grouping all offline features belonging to the same feature association primary key into one group. Then, a row-column transformation (pivot) is performed on all offline feature identifiers corresponding to each feature association primary key, which makes it easy to store the offline feature value corresponding to an offline feature identifier in a row of the horizontal table.

[0065] For example, in the above example, the grouped vertical table can be as shown in Table 3. The vertical table records each offline feature sequentially according to the feature association primary key. Then, the Spark computing engine's pivot function is used to transform the rows and columns of Table 3.

[0066] Table 3

[0067]

[0068]

[0069] Next, the vertical table after grouping is transformed into a row and column table to obtain a horizontal table after offline feature association and fusion, in the form of (feature association primary key, feature 1, feature 2, feature 3, ...), as shown in Table 4.

[0070] User 1 20 40 User 2 25 50 User 3 35 45

[0071] 204. Aggregate and deduplicate the offline features arranged by columns in the horizontal table to obtain fused offline features.

[0072] Understandably, the aggregation process for offline features belonging to the same feature association primary key can be performed either after row and column transformation or before. That is, aggregation can be skipped in the above steps, and after obtaining a horizontal table including all offline features, pivot processing can be applied to the aggregated offline feature identifiers corresponding to each feature association primary key. Understandably, if several offline features belonging to the same feature association primary key have the same offline feature identifier, then these offline features need to be aggregated. The aggregation process is performed according to the offline feature type corresponding to the offline feature identifier, including deduplication and merging operations, etc. Specific methods can be found in step 202 above, and will not be elaborated here.

[0073] 205. Store the fused offline features in the cache database.

[0074] After obtaining the horizontal table, the offline features belonging to the same feature-associated primary key in each row are called fused offline features. These fused offline features need to be stored in a cache database. The Spark engine can communicate with the cache database; after obtaining the horizontal table, the data in the horizontal table is written to the cache database so that during online inference, the fused offline features in the cache database are read to perform online inference and obtain the online inference results.

[0075] The following is a brief introduction to the process of using fused offline features for online inference. When training a machine learning model, multiple fused offline features for the model are first retrieved from the cache. Then, these fused offline features are input into the machine learning model to obtain the output result corresponding to each fused offline feature. Finally, the model parameters of the machine learning model are adjusted based on the output results until the output result of the machine learning model meets the training requirements. Figure 3 The training steps for a machine learning model include:

[0076] S1. Determine training samples based on fused offline features.

[0077] When multiple fused offline features are obtained, multiple training samples are determined based on these features. Each training sample includes a fused offline feature and the corresponding ground truth result. For example, in the above example, the training samples include a user's item purchase features and item usage features, as well as the corresponding ground truth item recommendation result. The ground truth item recommendation result for that user can be determined based on actual item recommendation rules and is used to evaluate the output of the machine learning model.

[0078] S2. Input the fused offline features into the machine learning model to be trained to obtain the output result of the fused offline features.

[0079] Then, each fused offline feature is input into the machine learning model to be trained, and the output result of each fused offline feature is obtained. For example, in the above example, the user item purchase feature and user item usage feature corresponding to a certain user are input into the machine learning model, and the machine learning model outputs the item recommendation output result corresponding to that user.

[0080] S3. Compare the output results with the actual results of fusing offline features, and calculate the loss value.

[0081] Next, the item recommendation output of the machine learning model for that user is compared with the actual item recommendation for that user to obtain the loss value. Understandably, the loss value is used to evaluate the difference between the item recommendation output and the actual item recommendation. Based on this difference, the model parameters of the machine learning model are adjusted inversely to make the output of the machine learning model closer to the actual item recommendation.

[0082] S4. Adjust the model parameters of the machine learning model based on the loss value.

[0083] By adjusting the model parameters of the machine learning model using the loss value, the output of the machine learning model can be made closer to the actual item recommendation results.

[0084] S5. When the preset training conditions are met, the training process of the machine learning model ends.

[0085] Training ends when the training reaches the preset training conditions. For example, the preset training conditions may be that the number of training iterations reaches a preset number, or the loss value is less than a preset loss threshold, etc.

[0086] Based on the same inventive concept, this invention also provides an offline feature fusion device, as described in the following embodiments. Figure 4 This is a schematic diagram of an offline feature fusion device provided in an embodiment of this application. The following is in conjunction with... Figure 4 The embodiments described below are described in detail. The embodiments described below are used to explain the technical solutions of this application and are not intended to limit actual use.

[0087] The offline feature fusion device includes:

[0088] The acquisition unit 401 is used to acquire at least one pre-established feature configuration table corresponding to a machine learning model. The feature configuration table includes at least a model identifier representing the machine learning model, an offline feature identifier representing the offline features, a Hive table identifier representing the Hive table to which the offline features belong, and the feature association primary key corresponding to the offline features stored in the Hive table.

[0089] The determining unit 402 is used to extract offline features from the Hive data table to which the offline features belong, based on the offline feature identifier and the Hive data table identifier in the feature configuration table corresponding to at least one machine learning model. The offline features include the feature association primary key, the offline feature identifier, and the offline feature value corresponding to the offline feature.

[0090] The processing unit 403 is used to perform feature fusion on the offline features corresponding to at least one machine learning model based on the feature association primary key corresponding to the offline features, so as to obtain fused offline features.

[0091] In an optional implementation, the offline feature fusion apparatus further includes an establishment unit 404.

[0092] The acquisition unit 401 is also used to acquire the offline feature identifier and feature association primary key corresponding to the machine learning model.

[0093] The determining unit 402 is further configured to determine, based on the offline feature identifier and the feature association primary key, the Hive data table to which the offline feature belongs in the Hive data table set, and the target column in the Hive data table to which the offline feature belongs. Each Hive data table in the Hive data table set is used to store the correspondence between the primary key and the feature.

[0094] Establishment unit 404 is used to create a feature configuration table corresponding to the machine learning model based on the offline feature identifier, feature association primary key, the Hive data table to which the offline feature belongs, and the target column.

[0095] In an optional implementation, the feature configuration table also includes a preprocessing method for offline feature values.

[0096] The processing unit 403 is further configured to preprocess the extracted offline feature values ​​according to the preprocessing method of the offline feature values ​​after the determining unit 402 extracts the offline features from the Hive data table to which the offline features belong.

[0097] The processing unit 403 is also used to perform feature fusion based on the preprocessed offline feature values.

[0098] In an optional implementation, processing unit 403 is specifically configured to combine the offline features corresponding to at least one machine learning model into a vertical table of offline features arranged in rows. Each row of the vertical table stores one offline feature. The vertical table is then transformed into a horizontal table of offline features arranged in columns based on the feature association key. Each row of the horizontal table stores the offline feature values ​​corresponding to all offline feature identifiers associated with a feature association key. The offline features arranged in columns in the horizontal table are aggregated and deduplicated to obtain fused offline features.

[0099] In an optional implementation, the offline feature fusion apparatus further includes a storage unit 405.

[0100] Storage unit 405 is used to store fused offline features in a cache database. During online inference, the fused offline features corresponding to at least one machine learning model in the cache database are read to perform online inference and obtain the online inference result.

[0101] In an optional implementation, the determining unit 402 is further configured to determine the offline feature identifier of the offline feature to be updated when the machine learning model performs a model update. This involves determining the first Hive data table corresponding to the offline feature identifier of the offline feature to be updated in the Hive data table set, and the first target column corresponding to the first Hive data table.

[0102] The processing unit 403 is also used to update the feature configuration table corresponding to the machine learning model based on the first hive data table and the first target column corresponding to the first hive data table.

[0103] In an optional implementation, the processing unit 403 is specifically configured to: if the offline feature identifier of the offline feature to be updated is a newly added offline feature identifier, then add a new target Hive data table as the Hive data table to which the offline feature belongs, and add a new target column of the target Hive data table as the target column of the Hive data table to which the offline feature belongs. If the offline feature identifier of the offline feature to be updated is an offline feature identifier to be deleted, then delete the target Hive data table corresponding to the offline feature to be updated from the Hive data table to which the offline feature belongs.

[0104] In an optional implementation, the determining unit 402 is specifically used to extract offline features from the Hive data table to which the updated offline features belong, based on the offline feature identifier and the Hive data table identifier in the updated feature configuration table.

[0105] In an optional implementation, processing unit 403 is specifically used to group the contents of the vertical table according to the feature association primary key. Based on the grouping result, all offline feature identifiers corresponding to each feature association primary key are determined. Pivot processing is performed on all offline feature identifiers corresponding to each feature association primary key, and the offline feature values ​​corresponding to all offline feature identifiers are stored in one row of the horizontal table.

[0106] In an optional implementation, processing unit 403 is further configured to aggregate all offline feature identifiers to obtain aggregated offline feature identifiers. Pivot processing is then performed on the aggregated offline feature identifier corresponding to each feature-associated primary key.

[0107] In this embodiment, the establishment unit 404 first pre-establishes a feature configuration table for the machine learning model to be trained based on its input. This feature configuration table includes various information about the offline features required by the machine learning model, including the offline feature identifier, the Hive table identifier representing the Hive table to which the offline feature belongs, and the feature association key corresponding to the offline feature in that Hive table. Thus, the determination unit 402 can directly access the original Hive table based on the feature configuration table, directly extracting offline features from the Hive table to which the offline features belong, obtaining the feature association key, offline feature identifier, and offline feature value corresponding to the offline feature. Finally, the processing unit 403 fuses multiple offline features based on the feature association key corresponding to each offline feature, forming multiple feature samples to train the machine learning model. It is understandable that by using the feature configuration table, a single offline feature can be directly accessed from the Hive table, achieving the goal of flexible extraction of offline features. When offline features are added or deleted, only the extraction information in the feature configuration table needs to be modified. This greatly improves the reusability of the original features while efficiently providing the offline features required by the machine learning model. In addition, after obtaining multiple offline features, the offline features are directly fused and the fused result is directly written into the cache. This avoids the operation of fusion calculation between multiple Hive tables to obtain a wide table. At the same time, there is no need to first store the fused offline features in the wide table and then read the data from the wide table to write it into the cache, thereby saving computing and storage resources. The time consumed by the feature fusion process will also be greatly shortened.

[0108] It should be noted that the information interaction and execution process between the modules / units in the offline feature fusion device are different from those in this application. Figures 1 to 3 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0109] The following describes an electronic device provided by an embodiment of this application. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 may be equipped with... Figure 5 The offline feature fusion device described in the corresponding embodiment is used to implement Figures 1 to 3 The functions correspond to those in the embodiments. Specifically, the electronic device 500 includes: a receiver 501, a transmitter 502, a processor 503, and a memory 504 (wherein the electronic device 500 may have one or more processors 503). Figure 5(Taking a processor as an example), processor 503 may include application processor 5031 and communication processor 5032. In some embodiments of this application, receiver 501, transmitter 502, processor 503 and memory 504 may be connected via a bus or other means.

[0110] Memory 504 may include read-only memory and random access memory, and provides instructions and data to processor 503. A portion of memory 504 may also include non-volatile random access memory (NVRAM). Memory 504 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0111] Processor 503 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0112] The methods disclosed in the embodiments of this application can be applied to processor 503, or implemented by processor 503. Processor 503 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 503 or by instructions in software form. Processor 503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 503 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 504, and processor 503 reads the information from memory 504 and, in conjunction with its hardware, completes the steps of the above method.

[0113] Receiver 501 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 502 can be used to output digital or character information through the first interface; transmitter 502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 502 may also include a display device such as a display screen.

[0114] In this embodiment of the application, the application processor 5031 in the processor 503 is used to execute... Figures 1 to 3 This corresponds to an offline feature fusion method in the embodiments. It should be noted that the specific manner in which the application processor 5031 executes each step differs from that in this application. Figures 1 to 3 The various method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figures 1 to 3 The corresponding method embodiments are the same, and for details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0115] This application provides a computer-readable storage medium, which includes computer instructions. When executed by a processor, the computer instructions are used to implement the technical solution of any offline feature fusion method in this application.

[0116] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 1 to 3 The illustrated embodiment describes the steps in an offline feature fusion method.

[0117] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. An offline feature fusion method for a game item recommendation model, characterized in that, The offline features include user item purchase features and user item usage features, and the offline feature fusion method includes: Obtain at least one pre-established feature configuration table corresponding to a game item recommendation model; the feature configuration table includes at least a model identifier representing the game item recommendation model, an offline feature identifier representing offline features, a Hive data table identifier representing the Hive data table to which the offline features belong, and a feature association primary key corresponding to the offline features stored in the Hive data table; Based on the offline feature identifier and the Hive data table identifier in the feature configuration table corresponding to the at least one game item recommendation model, the offline feature is extracted from the Hive data table to which the offline feature belongs; the offline feature includes the feature association primary key corresponding to the offline feature, the offline feature identifier, and the offline feature value; The offline features corresponding to the at least one game item recommendation model are combined into a vertical table with the offline features arranged in rows; each row of the vertical table stores one of the offline features; The vertical table is transformed into a horizontal table with offline features arranged in columns based on the feature association primary key; wherein, each row of the horizontal table stores the offline feature values ​​corresponding to all offline feature identifiers corresponding to a feature association primary key; The offline features arranged by columns in the horizontal table are aggregated and deduplicated to obtain fused offline features; the fused offline features are used to train the game item recommendation model.

2. The offline feature fusion method according to claim 1, characterized in that, The process of establishing the pre-established feature configuration table includes: Obtain the offline feature identifier and the feature association primary key corresponding to the game item recommendation model; The offline feature identifier and the feature association primary key are used to determine the Hive data table to which the offline feature belongs and the target column in the Hive data table to which the offline feature belongs; each Hive data table in the Hive data table set is used to store the correspondence between the primary key and the feature; Based on the offline feature identifier, the feature association primary key, the Hive data table to which the offline feature belongs, and the target column, establish the feature configuration table corresponding to the game item recommendation model.

3. The offline feature fusion method according to claim 2, characterized in that, The feature configuration table also includes a preprocessing method for the offline feature values; after extracting the offline features from the Hive data table to which the offline features belong, the offline feature fusion method further includes: The extracted offline feature values ​​are preprocessed according to the preprocessing method for the offline feature values. The feature fusion is performed based on the preprocessed offline feature values.

4. The offline feature fusion method according to claim 1 or 3, characterized in that, The step of fusing features of the extracted offline features corresponding to the at least one game item recommendation model based on the feature association primary key corresponding to the offline features to obtain fused offline features includes: The offline features corresponding to the at least one game item recommendation model are combined into a vertical table with the offline features arranged in rows; each row of the vertical table stores one of the offline features; The vertical table is transformed into a horizontal table with offline features arranged in columns based on the feature association primary key; wherein, each row of the horizontal table stores the offline feature values ​​corresponding to all offline feature identifiers corresponding to a feature association primary key; The offline features arranged by columns in the horizontal table are aggregated and deduplicated to obtain the fused offline features.

5. The offline feature fusion method according to claim 1, characterized in that, The offline feature fusion method further includes: The fused offline features are stored in a cache database; during online inference, online inference is performed by reading the fused offline features corresponding to at least one game item recommendation model in the cache database and obtaining the online inference result.

6. The offline feature fusion method according to claim 1, characterized in that, The offline feature fusion method further includes: When the game item recommendation model is updated, the offline feature identifier of the offline feature to be updated is determined; Determine the target Hive data table corresponding to the offline feature identifier of the offline feature to be updated in the Hive data table set, and the target column corresponding to the target Hive data table; Based on the target Hive data table and the corresponding target column in the target Hive data table, update the feature configuration table corresponding to the game item recommendation model.

7. The offline feature fusion method according to claim 6, characterized in that, The step of updating the feature configuration table corresponding to the game item recommendation model based on the target Hive data table and the corresponding target column in the target Hive data table includes: If the offline feature identifier of the offline feature to be updated is a newly added offline feature identifier, then the target Hive data table is added as the Hive data table to which the offline feature belongs, and the target column of the target Hive data table is added as the target column of the Hive data table to which the offline feature belongs; If the offline feature identifier of the offline feature to be updated is the same as the offline feature identifier to be deleted, then delete the target Hive data table corresponding to the offline feature to be updated from the Hive data table to which the offline feature belongs.

8. The offline feature fusion method according to any one of claims 6 to 7, characterized in that, Based on the offline feature identifier and the Hive data table identifier in the feature configuration table corresponding to the at least one game item recommendation model, the offline feature is extracted from the Hive data table to which the offline feature belongs, including: Based on the offline feature identifier and the Hive data table identifier in the updated feature configuration table, the offline feature is extracted from the Hive data table to which the updated offline feature belongs.

9. The offline feature fusion method according to claim 4, characterized in that, The step of performing row and column transformation on the vertical table based on the associated primary key of the features to obtain a horizontal table with offline features arranged in columns includes: The contents of the vertical table are grouped according to the primary key associated with the aforementioned features; Based on the grouping results, determine all offline feature identifiers corresponding to each of the aforementioned feature-associated primary keys; For each of the aforementioned feature-associated primary keys, perform pivot processing on all offline feature identifiers and store the offline feature values ​​corresponding to all offline feature identifiers in one row of the horizontal table.

10. The offline feature fusion method according to claim 9, characterized in that, The offline feature fusion method further includes: All the offline feature identifiers are aggregated to obtain the aggregated offline feature identifiers; The step of performing pivot processing on all offline feature identifiers corresponding to each of the aforementioned feature-associated primary keys includes: The pivot perspective processing is performed on the aggregated offline feature identifier corresponding to the primary key associated with each feature.

11. An offline feature fusion device for a game item recommendation model, characterized in that, The offline features include user item purchase features and user item usage features, and the offline feature fusion device includes: The acquisition unit is used to acquire at least one pre-established feature configuration table corresponding to a game item recommendation model; the feature configuration table includes at least a model identifier representing the game item recommendation model, an offline feature identifier representing offline features, a Hive data table identifier representing the Hive data table to which the offline features belong, and a feature association primary key corresponding to the offline features stored in the Hive data table; The determining unit is configured to extract the offline feature from the Hive data table to which the offline feature belongs, based on the offline feature identifier and the Hive data table identifier in the feature configuration table corresponding to the at least one game item recommendation model; the offline feature includes the feature association primary key corresponding to the offline feature, the offline feature identifier, and the offline feature value; The processing unit is configured to combine the offline features corresponding to the at least one game item recommendation model into a vertical table of offline features arranged in rows; each row of the vertical table stores one offline feature; the vertical table is transformed into a row-column table according to the feature association key to obtain a horizontal table of offline features arranged in columns; wherein each row of the horizontal table stores the offline feature values ​​corresponding to all offline feature identifiers corresponding to a feature association key; the offline features arranged in columns in the horizontal table are aggregated and deduplicated to obtain fused offline features; the fused offline features are used to train the game item recommendation model.

12. An electronic device, characterized in that, include: The memory and the processor are coupled; processor; A memory for storing a program for determining sample features, which, when read and executed by the processor, performs the offline feature fusion method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor using the offline feature fusion method as described in any one of claims 1-10.

Images