A method and system for wordpiece construction based on sparse autoencoder

Abstract

The application discloses a kind of word element construction method and system based on sparse self-encoder, the method includes: obtaining the semantic embedding vector of article;Using weight sharing strategy, the similarity of codebook vector and original embedding is calculated by weight sharing codebook;Top-K sparsification is executed to similarity, the K codebook vectors of maximum similarity value are retained, the rest is zero, generate sparse representation;According to the position and numerical size of non-zero element in sparse representation, generate discrete word element sequence;Reconstruct semantic embedding by decoder;Joint reconstruction loss, orthogonal constraint and diversity regularization optimization model.The quality of article representation and recommendation performance are significantly improved by sparse representation and orthogonal constraint optimization.The present method directly learns the sparse representation of article semantic features using sparse self-encoder combined with trainable codebook, and ensures the independence and semantic uniqueness of codebook vector through orthogonal constraint, thereby effectively alleviating the imbalance and embedding collapse problem during training.

Description

Technical Field

[0001] This application relates to the field of recommender system technology, specifically a lexical construction method and system based on sparse autoencoders. Background Technology

[0002] With the widespread application of Large-Scale Language Models (LLMs) in recommender systems, using LLMs for recommendation has become a new paradigm. However, in generative recommender systems based on pre-trained large language models, the model's input and output are usually presented in the form of tokens. Therefore, how to transform user interaction sequences into discrete token sequences is a key step in data processing, directly affecting the performance of sequence recommendation tasks. Traditional methods assign a unique ID token to each item as its discrete representation. However, as the item set gradually increases, this method requires adding all item IDs as new tokens to the vocabulary of the pre-trained model, affecting the performance of the original model, increasing computational overhead, and limiting scalability. Some works have attempted to use the item title directly as an index for the input and output of large models. However, this method fails to fully utilize other information about the items, and the generation space of large models is enormous, so the results may not be able to map to real items.

[0003] While existing methods have addressed some of the above issues, they also have two main limitations:

[0004] 1. Imbalanced Training Problem. Existing vector quantization methods typically rely on multiple codebooks and hierarchically quantize the semantic features of items through residual connections. However, this design suffers from a severe imbalanced training problem. Specifically, each codebook usually maps input features to only its nearest single codebook vector, resulting in most codebook vectors remaining unactivated throughout training. For example, two different items may consistently be mapped to the same codebook vector, while other codebook vectors remain untrained. This imbalanced update distribution prevents the model from fully utilizing the entire codebook, leading to inefficient training and limiting the model's representational power.

[0005] 2. Embedding Collapse Problem. Codebook vectors in existing vector quantization methods often exhibit high similarity and dependency, leading to embedding collapse. Geometrically, this collapse manifests as linear dependency of codebook vectors, causing the dimension of the representation space to fall far below the theoretical upper limit. For example, in three-dimensional space, codebook vectors may only cover a two-dimensional subspace, significantly reducing the model's representational power and generalization performance. This embedding collapse problem not only limits the model's ability to capture semantic features of items but also prevents the generated discrete semantic IDs from effectively representing the semantic similarity between items. Summary of the Invention

[0006] This embodiment provides a lexical construction method, system, electronic device, and storage medium based on a sparse autoencoder to solve the problem in related technologies that the embedding collapse problem not only limits the model's ability to capture the semantic features of items, but also makes the generated discrete semantic IDs unable to effectively express the semantic similarity between items.

[0007] In a first aspect, embodiments of the present invention provide a lexical construction method based on a sparse autoencoder, the lexical construction method based on a sparse autoencoder comprising the following steps:

[0008] Obtain the semantic embedding vector of the item;

[0009] A weight-sharing strategy is adopted to calculate the similarity between the codebook vector and the original embedding through the weight-sharing codebook;

[0010] Top-K sparsity is applied to the similarity, retaining the K codebook vectors with the largest similarity values ​​and setting the rest to zero to generate sparse representations;

[0011] Generate a discrete word sequence based on the position and value of non-zero elements in the sparse representation;

[0012] Reconstruct the semantic embedding using a decoder;

[0013] Joint reconstruction loss, orthogonal constraints, and diversity regularization optimization model.

[0014] In an optional embodiment, the weight sharing strategy is:

[0015] This allows the encoder and decoder to share a single trainable codebook. Where N represents the codebook size and d is the dimension of the semantic embedding. For codebook The i-th codebook vector in the middle;

[0016] The encoding process is viewed as calculating the similarity between each codebook vector in the codebook and the original embedding, while the decoding process is viewed as reconstructing the original embedding based on highly similar codebook vectors.

[0017] In an optional embodiment, calculating the similarity between the codebook vector and the original embedding using a weight-shared codebook includes:

[0018] For a given embedding representation, its latent space representation is computed by the encoder:

[0019]

[0020] in, For semantic embedding of items, The product of the codebook matrix and the semantic embedding vector represents the inner product of all codebook vectors in the codebook with the semantic embedding of the item, which is used as the similarity score. For the first codebook C Each codebook vector.

[0021] In an optional embodiment, top-K sparsity is performed on the similarity, retaining the K codebook vectors with the highest similarity values ​​and setting the rest to zero, to generate a sparse representation, including:

[0022] Retain the K codebook vectors that are most relevant to the input embedding representation, and set the rest to zero to obtain a sparse representation. ;

[0023] ;

[0024]

[0025] in, Let be the value of the j-th element in the latent space representation of the i-th item. Let represent the indices of all non-zero elements in the sparse representation of the i-th item, that is, the indices corresponding to the top-K largest elements. Here, the top-K largest elements refer to the non-zero positions corresponding to the K codebook vectors with the highest similarity scores.

[0026] In an optional embodiment, the discrete word sequence is generated as follows:

[0027] Sort the non-zero elements in the sparse representation from largest to smallest, and convert the corresponding codebook indexes into lexical identifiers.

[0028] In an optional embodiment, reconstructing the semantic embedding via a decoder includes:

[0029] The original embedding is reconstructed during the decoding stage by weighted combination of different codebook vectors using a shared weight approach.

[0030] ;

[0031] in, For the output of the model, Let be the value of the j-th element in the latent space representation of the i-th item. Let represent the index of the top-K largest elements in the sparse representation of the i-th item, where the top-K largest elements refer to the non-zero positions corresponding to the K codebook vectors with the highest similarity scores. This represents the j-th codebook vector in the codebook.

[0032] In an optional embodiment, the joint reconstruction loss, orthogonality constraint, and diversity regularization optimization model includes:

[0033] Reconstruction loss: ;

[0034] in The original semantic embedding of the input. The reconstructed semantic embedding is the output of the model;

[0035] Orthogonal constraints: , ;

[0036] in, For the size of the batch data, Indicates single-heat operation. This means keeping only the vector. The value of the Lth largest element is set, and all other elements are set to zero. express The result after single-heat operation; Used to calculate the cosine similarity of two items i and j in the same batch at the Lth marker position;

[0037] Diversity regularization optimization: .

[0038] in Represents the identity matrix. Represents the codebook matrix.

[0039] Compared with existing technologies, the beneficial effects of the lexical construction method based on sparse autoencoders of the present invention are as follows:

[0040] This invention significantly improves the quality of item representation and recommendation performance through sparse representation and orthogonal constraint optimization. Specifically, this method utilizes a sparse autoencoder combined with a trainable codebook to directly learn sparse representations of item semantic features, and ensures the independence and semantic uniqueness of codebook vectors through orthogonal constraints, thereby effectively alleviating the problems of training imbalance and embedding collapse. Experimental results show that this method outperforms existing methods on multiple benchmark datasets, significantly improving the quality of item representation and the performance of the recommendation system.

[0041] Secondly, embodiments of the present invention provide a lexical construction system based on a sparse autoencoder, comprising:

[0042] The module retrieves the semantic embedding vector of the item;

[0043] The similarity calculation module adopts a weight-sharing strategy, which calculates the similarity between the codebook vector and the original embedding through the weight-sharing codebook.

[0044] The sparse representation generation module performs top-K sparsification on the similarity, retaining the K codebook vectors with the largest similarity values ​​and setting the rest to zero to generate sparse representations;

[0045] The discrete word sequence generation module generates discrete word sequences based on the position and value of non-zero elements in the sparse representation.

[0046] The reconstruction module reconstructs the semantic embedding through the decoder;

[0047] The joint optimization module combines joint reconstruction loss, orthogonal constraints, and diverse regularized optimization models.

[0048] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.

[0049] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the lexical construction method based on a sparse autoencoder as described in the first aspect.

[0050] Compared with the prior art, the beneficial effects of the lexical construction system, electronic device and storage medium based on sparse autoencoder of the present invention are the same as those of the lexical construction method based on sparse autoencoder described in the first aspect, so they will not be repeated here. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of the lexical construction method based on a sparse autoencoder in an embodiment of the present invention;

[0053] Figure 2 This is a flowchart of model training in an embodiment of the present invention;

[0054] Figure 3 This is a structural block diagram of the lexical construction system based on a sparse autoencoder in an embodiment of the present invention;

[0055] Figure 4 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0056] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0057] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0058] In recommendation systems based on large-scale language models (LLMs), obtaining discrete lexical representations of items is a crucial step in sequence recommendation tasks. However, existing vector quantization (VQ) methods suffer from significant limitations in handling training imbalance and embedding collapse. To overcome these challenges, this paper proposes a lexical construction framework (RecTok) based on a sparse autoencoder (SAE), which adaptively selects and learns important semantic features to discretely represent items. We further propose a novel optimization objective to address representation diversity and embedding collapse issues. Our core idea is to directly learn sparse representations of items through a sparse autoencoder and introduce orthogonal constraints and diversity regularization to optimize the codebook's representation space. Our RecTok method is then described in detail.

[0059] This invention provides a lexical construction method based on a sparse autoencoder, applicable to recommendation systems based on large-scale language models. Figure 1 This is a flowchart of the lexical construction method based on sparse autoencoders of the present invention, as shown below. Figure 1As shown, the process includes the following steps:

[0060] S100, Obtain the semantic embedding vector of the item;

[0061] S200. A weight-sharing strategy is adopted to calculate the similarity between the codebook vector and the original embedding through the weight-sharing codebook.

[0062] In this embodiment, the weight sharing strategy is as follows:

[0063] This allows the encoder and decoder to share a single trainable codebook. , where N represents the size of the codebook and d is the dimension of the semantic embedding;

[0064] The encoding process is viewed as calculating the similarity between each codebook vector in the codebook and the original embedding, while the decoding process is viewed as reconstructing the original embedding based on highly similar codebook vectors.

[0065] Specifically, this scheme employs a sparse autoencoder (SAE) to map continuous semantic embeddings to discrete code vectors. To simplify the model and ensure consistency in the semantic space, a weight-sharing strategy is adopted. More specifically, the encoder and decoder share a trainable codebook. The encoding process can be viewed as calculating the similarity between each codebook vector and the original embedding, while the decoding process can be viewed as reconstructing the original embedding based on highly similar codebook vectors. This design not only reduces the complexity of the model but also ensures consistency in the semantic space during the encoding and decoding processes.

[0066] The similarity between the codebook vector and the original embedding is calculated using a weight-shared codebook, including:

[0067] For a given embedding representation, its latent space representation is computed by the encoder:

[0068]

[0069] in, For semantic embedding of items, The product of the codebook matrix and the semantic embedding vector represents the inner product of all codebook vectors in the codebook with the semantic embedding of the item, which is used as the similarity score. For the first codebook C Each codebook vector.

[0070] S300. Perform top-K sparsification on the similarity, retain the K codebook vectors with the largest similarity values, and set the rest to zero to generate sparse representations;

[0071] Top-K sparsity is applied to the similarity, retaining the K codebook vectors with the highest similarity values ​​and setting the rest to zero, generating a sparse representation, including:

[0072] Retain the K codebook vectors that are most relevant to the input embedding representation, and set the rest to zero to obtain a sparse representation. ;

[0073] ;

[0074] ;

[0075] in, Let be the value of the j-th element in the latent space representation of the i-th item. Let represent the indices of all non-zero elements in the sparse representation of the i-th item, that is, the indices corresponding to the top-K largest elements. The top-K largest elements refer to the non-zero positions corresponding to the K codebook vectors with the highest similarity scores.

[0076] S400. Generate a discrete word sequence based on the position and value of the non-zero elements in the sparse representation.

[0077] In this embodiment, the discrete word sequence is generated as follows:

[0078] Sort the non-zero elements in the sparse representation from largest to smallest, and convert the corresponding codebook indexes into lexical identifiers.

[0079] Specifically, given an embedding representation The latent space representation is computed through the encoder. To ensure sparsity, a top-K sparsity mechanism is introduced, which retains the K codebook vectors most relevant to the input embedding and sets the rest to zero to obtain the sparse representation. The above sparse coding strategy enables It can be represented by K most relevant codebook vectors, with the information contained therein concentrated on these K most relevant codebook vectors, and these vectors are adaptively selected and updated during subsequent optimization. This is based on sparse representation. Discrete ID representations are obtained based on the index of each position and the size of the element. For example, a sparse representation of an item. The element at position 3 is the largest, followed by the element at position 1. Therefore, the discrete ID of this item is...<T_3> ,<T_1> .

[0080] S500, reconstructs semantic embeddings through a decoder;

[0081] Reconstructing semantic embeddings through a decoder includes:

[0082] The original embedding is reconstructed in the decoding stage by weighted combination of different codebook vectors using a shared weight approach.

[0083] The decoder weights are a matrix composed of all codebook vectors, which is also the transpose of the encoder weights. By using shared weights in the decoding stage to reconstruct the original embedding through weighted combination of different codebook vectors, the original input can be reconstructed while retaining the K most relevant codebook vectors selected based on semantic similarity during the encoding stage.

[0084] ;

[0085] in, For the output of the model, Let be the value of the j-th element in the latent space representation of the i-th item. This represents the index of the top-K largest element in the sparse representation of the i-th item. This represents the j-th codebook vector in the codebook.

[0086] S600, joint reconstruction loss, orthogonal constraints, and diversity regularization optimization models.

[0087] In this scheme, the optimization objective of the model consists of a reconstruction objective, orthogonality constraints, and diversity regularization. The reconstruction objective mainly focuses on the consistency between the sparse representation encoded by the sparse autoencoder and the original output, calculating the mean square error between the decoder output and the original input:

[0088] ;

[0089] in The original semantic embedding of the input. The reconstructed semantic embedding is the output of the model.

[0090] The diversity constraint minimizes the cosine similarity of items in the same batch at the Lth marker position, promoting balanced codebook utilization while preserving the semantic similarity of the first L-1 markers:

[0091] ;

[0092] ;

[0093] in, For the size of the batch data, Indicates single-heat operation. This means keeping only the vector. The value of the Lth largest element is set, and all other elements are set to zero. express The result after single-heat operation; Used to calculate the cosine similarity of two items i and j in the same batch at the Lth marker position;

[0094] Furthermore, to ensure the integrity of the codebook semantic space, the similarity and dependencies between codebook vectors should be reduced. Therefore, orthogonal constraints are used to further optimize the codebook's expressive power.

[0095] ;

[0096] in, Represents the identity matrix. Represents the codebook matrix.

[0097] Overall, the RecTok method is based on a simple and efficient sparse autoencoder. By converting sparse semantic representations into discrete IDs for subsequent LLM training, this method is not only computationally efficient but can also be scaled to large-scale datasets.

[0098] The RecTok method proposed in this study demonstrates significant advantages and positive results in the discrete item representation task of recommendation systems based on large language models. First, RecTok integrates a sparse autoencoder with a codebook structure, sparsely activates and optimizes the K most relevant codebook vectors, and introduces orthogonal constraints and diversity regularization to optimize codebook representation capabilities, effectively overcoming the training imbalance and embedding collapse problems of traditional vector quantization-based lexical construction methods. Compared with existing methods, this scheme not only enhances the ability to capture semantic features of items but also reduces computational complexity through single-codebook sparse encoding, achieving efficient alignment between recommendation tasks and the autoregressive properties of LLM.

[0099] The experimental results are shown in Table 1. RecTok comprehensively outperforms various sequence recommendation baselines on three datasets from the Amazon e-commerce platform (MusicalInstruments, Video Games, and Baby Products). On the Games dataset, RecTok achieves HR@1 of 0.0223 and HR@5 of 0.0637, representing improvements of 6.70% and 3.74% respectively compared to LC-Rec. On the Instruments dataset, HR@5 and NDCG@5 show improvements of 19.4% and 17.3% respectively compared to variants without sparse activation, validating the crucial role of sparse representation in semantic modeling. This method, through the collaborative design of sparse coding and geometric constraints, preserves fine-grained representations of item semantics while adapting to the autoregressive generative paradigm of LLM, demonstrating significant engineering application potential in large-scale dynamic scenarios such as e-commerce and content recommendation.

[0100] Table 1: CVR prediction results of different methods on the Criteo dataset

[0101]

[0102] For example, such as Figure 2 As shown, Figure 2 The flowchart for training the model and generating recommendation results is shown below.

[0103] Data collection and preprocessing:

[0104] Collect all items in the dataset and their corresponding text information, such as title, category, and description. Encode the item text information using a pre-trained text encoding model to obtain semantic embeddings.

[0105] Model initialization:

[0106] The codebook vector is initialized using a uniform distribution, and the encoder and decoder weights are bound together and transposed of each other.

[0107] Model parameter update:

[0108] The sparse autoencoder is optimized through a multi-objective joint optimization process, which is achieved by using optimization algorithms such as stochastic gradient descent to balance computational efficiency and performance.

[0109] Obtain the discrete representation of the item:

[0110] The trained model is used to infer the semantic embeddings of all items, and discrete lexical units are assigned based on the obtained sparse activation results.

[0111] Fine-tuning the language model:

[0112] Based on the mapping relationship from items to lexical units in the previous step, the user's interaction sequence is represented by lexical units, and a language model prompt word fine-tuning model is constructed.

[0113] Generate recommendation results:

[0114] After training, the next item is predicted in the form of lexical units, and finally mapped to the specific item ID or title using the reverse process of step 4.

[0115] The RecTok method proposed in this solution can be widely applied to language model-based sequence recommendation tasks in scenarios such as e-commerce recommendations, content platforms, and streaming media services. In practical applications, RecTok can effectively improve item labeling quality and recommendation accuracy in various large-scale dynamic scenarios. For example, in large e-commerce platforms, merchants can use RecTok to transform product metadata (such as titles and descriptions) into discrete labeled sequences through a sparse autoencoder, integrating them into the LLM recommendation generation process. This method retains the most relevant code vectors through top-K sparsity and avoids semantic collapse by combining orthogonal constraints, enabling LLM to more accurately capture the semantic associations of products in the user's historical interaction sequence, thereby optimizing the conversion rate of recommendation modules such as "You May Also Like". In video streaming media scenarios, RecTok can transform the textual metadata of film and television content (such as plot summaries and tags) into semantically compact labeled sequences, helping LLM model the temporal patterns of user viewing history and dynamically adjust recommendation strategies to adapt to real-time preference changes. For example, when a user watches science fiction films consecutively, the labeled sequences generated by RecTok can strengthen LLM's capture of the "science fiction" semantic cluster and increase the recommendation priority of similar content.

[0116] Similarly, in news platforms, RecTok can semantically encode the titles and summaries of news articles. The generated discrete tags can be effectively integrated into the LLM autoregressive recommendation model, solving the problem that traditional ID taggers cannot capture the timeliness of content. By encouraging balanced codebook utilization through diversity regularization, RecTok enables LLM to quickly update the tag representations of relevant topics when handling breaking news events, achieving real-time iteration of recommended content. Furthermore, in game app store recommendation scenarios, RecTok can transform game type, gameplay descriptions, and other metadata into structured tags, helping LLM model users' game preference sequences. Especially when handling newly released games, sparse encoding and codebook optimization can accelerate the semantic integration of new content and reduce cold-start recommendation errors. These application examples demonstrate that RecTok, through the deep coupling of semantic tagging and the autoregressive characteristics of LLM, exhibits flexible adaptability and engineering practicality in large-scale recommendation systems across different domains.

[0117] This invention also provides a lexical construction system based on a sparse autoencoder, which is used to implement the above-described method embodiments; details already described will not be repeated. The terms "module," "unit," and "subunit," etc., used below refer to combinations of software and / or hardware that achieve a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.

[0118] like Figure 3 As shown, Figure 3This is a structural block diagram of the lexical construction system based on a sparse autoencoder in this invention. The system includes:

[0119] Get module 101 to obtain the semantic embedding vector of the item;

[0120] The similarity calculation module 102 adopts a weight sharing strategy to calculate the similarity between the codebook vector and the original embedding through the weight-shared codebook.

[0121] The sparse representation generation module 103 performs top-K sparsification on the similarity, retains the K codebook vectors with the largest similarity values, and sets the rest to zero to generate sparse representations;

[0122] The discrete word sequence generation module 104 generates a discrete word sequence based on the position and value of the non-zero elements in the sparse representation.

[0123] Reconstruction module 105 reconstructs semantic embeddings using a decoder;

[0124] Joint optimization module 106, which includes joint reconstruction loss, orthogonal constraints, and diversity regularization optimization models.

[0125] Figure 4 A structural block diagram of the electronic device provided in the embodiments of the present invention, such as... Figure 4 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following methods:

[0126] S100, Obtain the semantic embedding vector of the item;

[0127] S200. A weight-sharing strategy is adopted to calculate the similarity between the codebook vector and the original embedding through the weight-sharing codebook.

[0128] S300. Perform top-K sparsification on the similarity, retain the K codebook vectors with the largest similarity values, and set the rest to zero to generate sparse representations;

[0129] S400. Generate a discrete word sequence based on the position and value of the non-zero elements in the sparse representation.

[0130] S500, reconstructs semantic embeddings through a decoder;

[0131] S600, joint reconstruction loss, orthogonal constraints, and diversity regularization optimization models.

[0132] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a wordpiece based on a sparse autoencoder, applied to a recommendation system based on a large-scale language model, characterized in that, The method comprises the following steps: obtaining a semantic embedding vector of an item; calculating the similarity between the codebook vector and the original embedding through the weight-shared codebook by adopting a weight-sharing strategy; performing top-K sparsification on the similarity, retaining the K codebook vectors with the largest similarity values and zeroing the rest to generate a sparse representation; generating a discrete token sequence according to the position and numerical size of the non-zero elements in the sparse representation; reconstructing the semantic embedding through a decoder, comprising: reconstructing the original embedding by weighted combination of different codebook vectors in the decoding stage in a weight-sharing manner, ； wherein, is an output of the model, is a value of the jth element in the latent space representation of the ith item, represents the index corresponding to the top-K largest elements in the sparse representation of the ith item, the top-K largest elements refer to the non-zero positions corresponding to the K codebook vectors with the highest similarity scores, represents the jth codebook vector in the codebook; optimizing the model jointly with reconstruction loss, orthogonal constraint and diversity regularization.

2. The sparse autoencoder-based token construction method of claim 1, wherein, The weight-sharing strategy is: Making an encoder and a decoder jointly use a trainable codebook where N denotes the size of the codebook, d is the dimension of the semantic embedding, for the codebook the i-th codebook vector in the codebook treating the encoding process as calculating the similarity between each codebook vector in the codebook and the original embedding, and treating the decoding process as reconstructing the original embedding based on the codebook vector with high similarity.

3. The sparse autoencoder-based token construction method of claim 2, wherein, calculating the similarity between the codebook vector and the original embedding through the weight-shared codebook, comprising: calculating the hidden space representation of the given embedding representation through the encoder: ； wherein, is a semantic embedding of the item, is a codebook matrix multiplication semantic embedding vector, representing that all codebook vectors in the codebook have a similarity as an inner product with the semantic embedding of the item, is a codebook vector in the codebook C. The codebook vector is the codebook vector with the highest similarity to the semantic embedding of the item. is a codebook vector in the codebook C. The codebook vector is the codebook vector with the highest similarity to the semantic embedding of the item. 4.The sparse autoencoder-based token construction method of claim 1, wherein, performing top-K sparsification on the similarity, retaining the K codebook vectors with the largest similarity values and zeroing the rest to generate a sparse representation, comprising: Retain the K codebook vectors most relevant to the input embedding representation, and zero out the rest to get a sparse representation ; ； ； wherein, is a value of the jth element in the latent space representation of the ith item, denotes the index corresponding to the top-K largest element in the sparse representation of the ith item, wherein the top-K largest element refers to the non-zero position corresponding to the K codebook vectors with the highest similarity score. 5.The sparse autoencoder-based token construction method of claim 1, wherein, The discrete token sequence generation method is: sorting the non-zero elements in the sparse representation in descending order of numerical value, and converting the corresponding codebook index into a token identifier.

6. The sparse autoencoder-based token construction method of claim 1, wherein, The optimization model jointly with reconstruction loss, orthogonal constraint and diversity regularization comprises: reconstruction loss: ; wherein, is an input raw semantic embedding, is a reconstructed semantic embedding output by the model; Orthogonal constraints: , ; wherein, is the size of the batch data, represents one-hot operation, represents only the reserved vector the value of the Lth largest element in the vector, and the rest of the elements are set to zero, represents the result after one-hot operation; is used to calculate the cosine similarity of i and j in the same batch at the Lth label position. Diversity regularized optimization: ; wherein denotes the identity matrix, denotes the codebook matrix.

7. A sparse autoencoder-based wordpiece construction system, comprising: comprising: an acquisition module that acquires a semantic embedding vector of an item; a similarity calculation module that calculates the similarity between the codebook vector and the original embedding through the weight-shared codebook by adopting a weight-sharing strategy; a sparse representation generation module that performs top-K sparsification on the similarity, retains the K codebook vectors with the largest similarity values and zeroing the rest to generate a sparse representation; a discrete token sequence generation module that generates a discrete token sequence according to the position and numerical size of the non-zero elements in the sparse representation; a reconstruction module that reconstructs the semantic embedding through a decoder, comprising: reconstructing the original embedding by weighted combination of different codebook vectors in the decoding stage in a weight-sharing manner, ； wherein, is an output of the model, is a value of the jth element in the latent space representation of the ith item, represents the index corresponding to the top-K largest elements in the sparse representation of the ith item, the top-K largest elements refer to the non-zero positions corresponding to the K codebook vectors with the highest similarity scores, represents the jth codebook vector in the codebook; a joint optimization module that optimizes the model jointly with reconstruction loss, orthogonal constraint and diversity regularization.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the sparse autoencoder-based token construction method according to any one of claims 1-6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the sparse autoencoder-based token construction method according to any one of claims 1-6.

Images