Double-embedded reasoning method and system

Abstract

The invention discloses a double-embedded reasoning method and system, and is applied to the technical field of system software, and the system comprises a back-end storage module which is used for storing an embedded table; the compression embedding module is used for converting the embedding table into a low-representation embedding item; the cache module is used for storing the input category type features and searching the category type features; if so, obtaining a high-representation embedding item; if the high representation embedding item is not hit, providing a low representation embedding item by a compression embedding module, asynchronously loading the high representation embedding item which is not hit from a rear-end storage module, and updating the high representation embedding item to a cache module; the feature interaction module is used for receiving the embedded item output by the cache module or the compression embedding module, and carrying out interaction processing on the embedded item and the obtained numerical features to generate composite features; the neural network module is used for predicting the composite features generated by the feature interaction module and outputting a result; through cooperative work among the modules, memory and delay challenges in a traditional recommendation system are solved, and reasoning performance is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of system software, and particularly to a dual-embedding inference method and system. Background Art

[0002] With the deep popularization of the Internet and the rapid expansion of the scale of Internet users, the high-speed growth of information data has led to a serious problem of information overload. How to mine potential value from massive data and provide more accurate content services for users is directly related to the survival and development of enterprises.

[0003] As an important means to solve the problem of information overload, the recommendation system has been widely applied. However, the recommendation effect is often closely related to the degree of model adaptation, and it is crucial to design a suitable recommendation model. In the recommendation system, the embedding model is widely used in various recommendation tasks because it can effectively represent categorical features. However, the embedding model needs to process a large amount of categorical data, which cannot be directly subjected to matrix operations. Therefore, each categorical data requires an independent embedding table to represent, which brings the problems of overly sparse and huge embedding table entries. In addition, the parameters of traditional deep neural networks can be shared, while the embedding table occupies most of the memory, making the storage and scheduling problems of the embedding table a challenge for researchers. Therefore, it is necessary to optimize the storage and scheduling of embedding table entries to improve the training and inference speed of the recommendation system.

[0004] To overcome these defects, the present application proposes a dual-embedding inference method and system. Summary of the Invention

[0005] The purpose of the present application is to provide a dual-embedding inference method and system, aiming to solve the efficiency problem of embedding table storage and retrieval in large-scale recommendation systems.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] In the first aspect, the present application provides a dual-embedding inference system, which specifically includes:

[0008] A backend storage module: used to store the embedding table;

[0009] A compressed embedding module, used to convert the embedding table into low-representation embedding entries;

[0010] A cache module, used to store the input categorical features, and perform a lookup on the categorical features; if a hit occurs, obtain the high-representation embedding entries; if a miss occurs, the compressed embedding module provides the low-representation embedding entries, asynchronously loads the missed high-representation embedding entries from the backend storage module, and updates them to the cache module;

[0011] A feature interaction module, configured to receive the embedding items output by the cache module or the compressed embedding module, interact with the obtained numerical features, and generate composite features;

[0012] A neural network module: configured to predict the composite features generated by the feature interaction module and output results.

[0013] In a second aspect, the present application provides a dual-embedding inference method, and the steps include:

[0014] Obtain categorical features and numerical features;

[0015] Perform feature interaction on the numerical features and the embedding items to generate composite features; wherein, the embedded categorical features include: during the inference process, query the categorical features, if the cache hits, extract high-representation embedding items from the categorical features; if the cache misses, trigger asynchronous update to obtain low-representation embedding items;

[0016] Train a neural network model based on historical data, input the composite features into the neural network model for prediction, and output results.

[0017] In a third aspect, the present application provides a computer device, which includes a processor and a memory coupled to the processor. Wherein, the memory stores program instructions for implementing a dual-embedding inference method; the processor is configured to execute the program instructions stored in the memory to implement a dual-embedding inference.

[0018] In a fourth aspect, the present application provides a storage medium, which stores program instructions that can be run by a processor, and the program instructions are used to execute a dual-embedding inference method.

[0019] The present application provides a dual-embedding inference method and system, which have the following beneficial effects:

[0020] (1) By combining traditional embedding tables with memory-efficient compressed embedding technology, effectively manage the embedding index and data access during the model inference process through an intelligent cache mechanism; the compressed embedding module effectively solves the sparsity and storage bottleneck problems of large-scale embedding tables through pre-computed rotation weights and block storage strategies;

[0021] (2) In the cache module, through the asynchronous cache update mechanism combined with hyper-threading technology, when the cache misses, the main thread does not need to wait for the backend storage to load, and directly performs approximate calculations through the compressed embedding module, reducing the inference latency and reducing the dependence on the backend storage module;

[0022] (3) This application divides hot data and cold data according to the access frequency of categorical features in the offline dataset; while taking into account the comprehensiveness and accuracy of the recommendation system, it realizes the optimization of performance and resource utilization, bringing a significant performance improvement to the recommendation system;

[0023] (4) This application does not rely on dedicated hardware and completely realizes efficient embedded table management and inference acceleration through software optimization, and is applicable to various resource environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic structural diagram of a dual-embedding inference system according to Embodiment 1 of this application;

[0025] Figure 2 It is a schematic flowchart of a dual-embedding inference method according to Embodiment 2 of this application;

[0026] Figure 3 It is a schematic structural diagram of a computer device according to Embodiment 3 of this application;

[0027] Figure 4 It is a schematic structural diagram of a storage medium according to Embodiment 4 of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0029] The following analyzes the solutions in the prior art in combination with related technologies.

[0030] Kurniawan et al. [1] proposed a three-layer cache structure to solve the problem of large-scale embedded table storage. By caching the most frequently accessed embedded data in memory, it avoids accessing the backend storage every time. Jain et al. [2] proposed a memory-efficient embedding technique to compress the embedded table size and reduce memory consumption through rotated embedding and approximate storage. Eisenman et al. [3] proposed a non-volatile memory-based solution to speed up the embedded lookup process by using non-volatile memory as the storage backend. Ke et al. [4] and Kwon et al. [5] accelerated the inference speed by using near-memory processing technology.

[0031] [1]D.H. Kurniawan, R. Wang, K.S. Zulkifli, F.A. Wiranata, J. Bent, Y. Vigfusson, and H.S. Gunawi, “Evstore: Storage and caching capabilities for scaling embedding tables in deep recommendation systems,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2023, pp. 281-294.

[0032] [2]R. Jain, S. Cheng, V. Kalagi, V. Sanghavi, S. Kaul, M. Arunachalam, K. Maeng, A. Jog, A. Sivasubramaniam, M.T. Kandemir et al., “Optimizing cpu performance for recommendation systems at-scale,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1-15.

[0033] [3]A. Eisenman, M. Naumov, D. Gardner, M. Smelyanskiy, S. Pupyrev, K. Hazelwood, A. Cidon, and S. Katti, “Bandana: Using non-volatile memory for storing deep learning models,” Proceedings of machine learning and systems, vol. 1, pp. 40-52, 2019.

[0034] [4]L.Ke, U.Gupta, B.Y.Cho, D.Brooks, V.Chandra, U.Diril, A.Firoozshahian, K.Hazelwood, B.Jia, H.-H.S.Lee et al., “Recnmp: Accelerating personalized recommendation with near-memory processing,” in 2020 ACM / IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2020, pp.790-803.

[0035] [5]Y.Kwon, Y.Lee, and M.Rhu, “Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning,” in Proceedings of the 52nd Annual IEEE / ACM International Symposium on Microarchitecture, 2019, pp.740-753.

[0036] Analyzing the drawbacks of the above prior arts, in the solution of [1] above, although the memory occupancy problem of large-scale embedding tables is solved, there is still the problem of long back-end latency. In the solution of [2] above, although this technology has significant advantages in memory efficiency, it usually leads to a decrease in model accuracy. In the solutions of [3-5] above, the inference latency is significantly improved, but it requires dedicated hardware support, and requires the update and iteration of the software and hardware environment of the recommendation system infrastructure, and is not applicable to the continuous expansion of large-scale embedding tables.

[0037] Therefore, when facing the storage problem of traditional embedding tables, the prior arts still face many problems such as model inference latency, model performance, memory and efficiency balance. In this application, traditional embedding tables and memory-efficient embedding tables are combined, and an asynchronous cache update mechanism is adopted to significantly reduce memory occupancy while ensuring inference performance.

[0038] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0039] Embodiment 1

[0040] Please refer to Figure 1 , which is a schematic structural diagram of a dual-embedding inference system according to Embodiment 1 of the present application; the specific content includes:

[0041] Backend storage module: used to store the embedding table;

[0042] Compression embedding module, used to convert the embedding table into low-representation embedding items;

[0043] Cache module, used to store the input categorical features, and perform lookups on the categorical features; if a hit occurs, obtain the high-representation embedding items; if a miss occurs, the compression embedding module provides low-representation embedding items, asynchronously loads the missing high-representation embedding items from the backend storage module, and updates them to the cache module;

[0044] Feature interaction module, used to receive the embedding items output by the cache module or the compression embedding module, and perform interactive processing with the obtained numerical features to generate composite features;

[0045] Neural network module: used to predict the composite features generated by the feature interaction module and output the results.

[0046] In this embodiment, the inference system is composed of a backend storage module, an offline training module, a compression embedding module, a cache module, a feature interaction module, and a neural network module. Each module will be described in detail below.

[0047] The backend storage module adopts a distributed key-value storage system such as Redis Cluster or Cassandra to manage and store the embedding table. The embedding table is stored in a sharded form, and each shard corresponds to a different feature category. During the dynamic update process, the backend storage module adopts a copy-on-write strategy, and the update operation only generates a new embedding table in the background to avoid blocking read requests.

[0048] The compression embedding module converts dynamic calculations into static weight storage by calculating rotation weights, which can reduce redundant calculations of trigonometric functions and matrix multiplications during the inference process. Thus, while maintaining the memory advantage, it significantly improves the inference efficiency. For example, when processing large-scale embedding data, traditional compression methods need to perform repeated calculations during each inference. However, this application pre-computes the weights to quickly obtain the required results, shortening the inference time, enabling the inference system to respond to requests faster, and improving the user experience. Additionally, it works in collaboration with the cache module. When the cache misses, the compression embedding module loads and decompresses the embedding item, and inputs the compressed low-representation embedding item into the feature interaction module.

[0049] During the inference process of the cache module, the categorical features are searched to determine whether there is a hit. If it is determined that there is a hit, the cache module extracts the high-representation embedding item from the categorical features, and processes the high-representation embedding item and the numerical features through the feature interaction module to obtain the composite feature.

[0050] If it is determined that there is a miss, asynchronous update is performed through the hyper-threading method, triggering the compression embedding module to output the low-representation embedding item for inference. Asynchronously, the missing high-representation embedding item is retrieved from the backend storage module and updated to the cache module; the low-representation embedding item and the numerical features are input into the feature interaction module to generate the composite feature. Among them, the missing high-representation embedding item is updated based on a preset replacement strategy, and the replacement strategy includes but is not limited to: LRU, LFU, DLIRS, or Cacheus.

[0051] It should be noted that when the cache misses, the inference system does not wait for the cache update to complete before continuing with subsequent operations. Instead, through the hyper-threading method, without affecting the main thread's processing of other tasks, the missing embeddings are asynchronously retrieved from the backend storage and updated to the cache, avoiding delays caused by waiting for cache updates and improving the overall throughput and response speed of the inference system. For example, when processing a large number of concurrent requests, the hyper-threading method enables multiple threads to execute alternately on a single physical core, allowing other threads to continue processing tasks such as feature interaction and neural network calculations for other requests while the cache is being updated, thus effectively enhancing the concurrent processing ability and efficiency of the inference system. Multiple different cache replacement strategies can be adopted in the cache module. Different replacement strategies have their own characteristics and are suitable for different scenarios and data access patterns, and can be dynamically adjusted according to the data access pattern.

[0052] The offline training module is connected to the backend storage module and the compression embedding module; the offline training module is used to classify the obtained offline data set according to a preset frequency threshold to obtain hot data and cold data. Through the embedding routing technology, the hot data is routed to the first embedding table, and the first embedding table is compressed using a one-to-one embedding mapping relationship; the cold data is routed to the second embedding table, and the second embedding table is compressed by rotation embedding or combined embedding; the second embedding table is optimized based on the first embedding table.

[0053] The feature interaction module is used to receive the embedding items output by the cache module or the compression embedding module, interact with the obtained numerical features to generate composite features. After feature interaction, the composite features are input into the neural network module for processing and prediction.

[0054] The neural network module is used to predict the composite features generated by the feature interaction module and output the results. The neural network module is a multi-layer perceptron (MLP), and its structure includes: an input layer for receiving the composite features output by the feature interaction module. Hidden layer: 3 fully connected layers with dimensions of 256, 128, and 64 respectively, and the activation function is Swish. Residual connections are introduced to prevent gradient disappearance. Output layer: 1 neuron, and the Sigmoid function is used to output the prediction results.

[0055] In summary, the embodiment 1 of the present application integrates traditional embedding and memory-efficient embedding technologies, and optimizes large-scale dynamic data inference through an asynchronous cache mechanism. Among them, the cache module uses asynchronous updates to avoid delays, the compression embedding module stores all embedding items in a memory-efficient manner, and the feature interaction module processes the mixed features and then inputs them into the neural network module to predict and output the results. It effectively solves the problems of high memory occupancy, cache miss latency, and performance efficiency trade-off in traditional recommendation systems, and does not require dedicated hardware. Software optimization can achieve efficient embedding table management.

[0056] Embodiment 2

[0057] Please refer to Figure 2 , which is a schematic flowchart of a dual-embedding inference method according to Embodiment 2 of the present application; the steps include:

[0058] Obtain categorical features and numerical features;

[0059] Perform feature interaction on the numerical features and embedding items to generate composite features; among them, the embedded categorical features include: during the inference process, query the categorical features, if the cache hits, extract the high-representation embedding items from the categorical features; if the cache misses, trigger asynchronous updates to obtain the low-representation embedding items;

[0060] A neural network model is trained based on historical data, and the composite features are input into the neural network model for prediction to output results.

[0061] In this embodiment, an intelligent caching mechanism is used to effectively manage the embedding index and data access during the model inference process. To avoid the inference latency caused by the traditional cache update mechanism, an asynchronous thread is introduced to perform the cache update operation. During the inference process, when querying categorical features, if the cache hits, high-representation embedding items are extracted from the categorical features; if the cache misses, asynchronous update is triggered to obtain low-representation embedding items. The cache update is executed in the background thread, thus avoiding the latency caused by the main thread waiting for the cache update. The main thread can continue to process other tasks while the cache is being updated, effectively reducing the inference latency.

[0062] Furthermore, since the traditional method cannot effectively capture the cold-hot distribution characteristics in the data during the training process, in the actual inference process, especially when the cold-hot data changes dynamically, the dual embeddings cannot achieve good alignment and fusion, which in turn leads to a decline in model performance. To solve this problem, this application adapts to the dual-embedding hybrid working mode during the actual inference in the training stage, enhances the collaborative working ability of the two embeddings, and ensures effective fusion in the inference stage. Specifically, the offline dataset is preprocessed, and hot data and cold data are distinguished according to a predefined frequency threshold. The hot data is routed to the first embedding table, and the cold data is routed to the second embedding table. Specifically, the hot data, that is, the data with high-frequency access, is routed to the first embedding table and compressed using a one-to-one embedding mapping relationship to ensure high precision of the model performance; while the cold data, that is, the data with low-frequency access, is routed to the second embedding table and compressed through rotation embedding or combined embedding to fit the access patterns of the cold-hot data in the actual inference process as much as possible, thereby optimizing the mixed expression form of the embedding data in the training stage and improving the model performance in the actual inference stage.

[0063] Experimenting with this inference method on multiple datasets shows that it can significantly improve the inference performance while reducing the memory footprint. On the Criteo dataset, this method reduces the memory footprint by 97% with only a 0.25% AUC loss, and at the same time, compared with the traditional synchronous cache strategy, the average inference latency is reduced by 13.7%, and the P99 latency is significantly reduced by 21.1%.

[0064] In summary, in Embodiment 2 of the present application, categorical and numerical features are first obtained, and the numerical features are interacted with the processed embedding terms to generate composite features. During inference, the categorical features are queried. If the cache hits, the high-representation embedding terms are extracted; if not, asynchronous update is triggered to obtain the low- and high-representation embedding terms, avoiding main thread latency. Additionally, a hybrid mode is adopted to divide cold data and hot data, enhancing the collaboration of the dual embeddings. This method effectively solves the problems of memory occupation, inference latency, and cold-hot data fusion, improves the model performance and adaptability, and achieves efficient inference.

[0065] Embodiment 3

[0066] Please refer to Figure 3 , which is a schematic structural diagram of a computer device according to Embodiment 3 of the present application. The computer device 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0067] The memory 52 stores program instructions for implementing the above-mentioned dual-embedding inference method.

[0068] The processor 51 is configured to execute the program instructions stored in the memory 52 to implement a dual-embedding inference.

[0069] Among them, the processor 51 can also be referred to as a CPU (Central Processing Unit).

[0070] The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0071] Embodiment 4

[0072] Please refer to Figure 4, which is a schematic structural diagram of the storage medium according to Embodiment 4 of the present application. The storage medium of the embodiments of the present application stores a program file 61 that can implement all the above methods. Among them, the program file 61 can be stored in the above storage medium in the form of a software product, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods according to various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, external hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, or devices such as computers, servers, mobile phones, and tablets.

[0073] It should be noted that in this document, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, apparatus, article, or method including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, apparatus, article, or method. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, apparatus, article, or method including that element.

[0074] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

[0075] Although the embodiments of the present application have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application. The scope of the present application is defined by the appended claims and their equivalents.

[0076] Certainly, the present invention can also have other various embodiments. Based on this embodiment, other embodiments obtained by those of ordinary skill in the art without any creative work belong to the scope protected by the present invention.

Claims

1. A dual-embedded inference system, characterized in that, Comprising: A backend storage module: used to store the embedding table; A compression embedding module, used to convert the embedding table into low-representation embedding items; A cache module, used to store the input categorical features and perform lookups on the categorical features; If a hit occurs, obtain the high-representation embedding items; If a miss occurs, the compression embedding module provides the low-representation embedding items, asynchronously loads the missed high-representation embedding items from the backend storage module, and updates them to the cache module; A feature interaction module, used to receive the embedding items output by the cache module or the compression embedding module, and perform interaction processing with the obtained numerical features to generate composite features; A neural network module: used to predict the composite features generated by the feature interaction module and output the results.

2. The dual-embedded inference system according to claim 1, wherein During the inference process, the cache module performs lookups on the categorical features to determine whether there is a hit; If it is determined that there is a hit, the cache module extracts high-representation embedding items from the categorical features, and processes the high-representation embedding items and the numerical features through the feature interaction module to obtain composite features; If it is determined that there is a miss, asynchronous update is performed through the hyper-threading method, triggering the compression embedding module to output low-representation embedding items for inference, and asynchronously loading the missed high-representation embedding items to update the cache module; Input the low-representation embedding items and the numerical features into the feature interaction module to generate composite features; Among them, the missed high-representation embedding items are updated based on a preset replacement strategy, and the replacement strategy includes but is not limited to: LRU, LFU, DLIRS, or Cacheus.

3. A dual-embedded inference system according to claim 1, characterized in that, The compression embedding module converts dynamic calculations into static weights for storage by calculating rotation weights.

4. A dual-embedded inference system according to claim 1, characterized in that, The system further includes an offline training module, which is connected to the backend storage module and the compression embedding module; The offline training module is used to classify the obtained offline dataset according to a preset frequency threshold to obtain hot data and cold data; Through the embedding routing technology, route the hot data to the first embedding table and the cold data to the second embedding table.

5. A dual-embedded inference system according to claim 4, characterized in that, The first embedding table is compressed using a one-to-one embedding mapping relationship, and the second embedding table is compressed through rotation embedding or combined embedding; the second embedding table is optimized based on the first embedding table.

6. A dual-embedding inference method, characterized in that, Comprising: Obtain categorical features and numerical features; Perform feature interaction on the numerical features and the embedding items to generate composite features; among them, the embedded categorical features include: during the inference process, query the categorical features, if the cache hits, extract high-representation embedding items from the categorical features; if the cache misses, trigger asynchronous update to obtain low-representation embedding items; Train a neural network model based on historical data, input the composite features into the neural network model for prediction, and output the results.

7. A dual-embedding inference method according to claim 6, characterized in that When the cache misses, asynchronous update is performed through the hyper-threading method; Update the cache for the missed categorical features, and this cache update process is executed in parallel with the inference process.

8. A dual-embedding inference method according to claim 6, characterized in that, The method further includes: dividing hot data and cold data according to the access frequency of categorical features in the offline dataset; Route the hot data to the first embedded table and the cold data to the second embedded table; the first embedded table is compressed using a one-to-one embedding mapping relationship, and the second embedded table is compressed by rotational embedding or combined embedding; the second embedded table is optimized based on the first embedded table.

9. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor. Among them, the memory stores program instructions for implementing a dual-embedding inference method according to any one of claims 6-8; the processor is configured to execute the program instructions stored in the memory to implement a dual-embedding inference.

10. A storage medium, characterized in that, Store program instructions that can be run by a processor, and the program instructions are used to execute a dual-embedding inference method according to any one of claims 6-8.

Images