Model training method, online recall retrieval method, electronic device, and storage medium

Abstract

This disclosure provides a model training method, an online recall retrieval method, an electronic device, and a storage medium. The fine-tuning model training method includes: processing images in post samples using an image encoder in the fine-tuning model to obtain image features; processing post text in post samples using a text encoder in the fine-tuning model to obtain text features; processing tag information of post samples using a text encoder to obtain tag features; fusing image features and text features using a multilayer perceptron in the fine-tuning model to obtain fused features; aligning the fused features and tag features to obtain post features of the post samples, which are used as the output of the fine-tuning model; obtaining a loss function using image features, text features, and post features; and training the image encoder, text encoder, and multilayer perceptron using the loss function.

Description

Technical Field

[0001] This disclosure relates to the field of large model recommendation technology, and in particular to a model training method, an online recall retrieval method, an electronic device, and a storage medium. Background Technology

[0002] Current social media platforms offer a wealth of features, covering various interactive scenarios such as social matching, post interaction, and voice chat. To achieve recommendation capabilities that bridge the gap between post interaction and voice chat scenarios, a recommendation system capable of utilizing cross-modal and cross-scenario data needs to be built. Traditional recommendation systems typically rely on user profiling, collaborative filtering, and semantic matching to provide recommendations to users. Summary of the Invention

[0003] The inventors noted the following shortcomings in cross-scenario recommendation technologies.

[0004] 1) Feature disconnect: The content modalities differ significantly between different scenarios. For example, post interaction scenarios (such as post squares) contain text and images, while voice chat scenarios mainly contain text and audio. Traditional models cannot align the two types of content in the semantic space.

[0005] 2) Single distribution of user behavior: Most users' usage habits are concentrated on a single function, and cross-scenario behavior signals are sparse, which reduces the recall ability of I2I (Item to Item) and U2I (User to Item).

[0006] 3) System cold start: The current cross-scenario recommendation module is still in the cold start stage, which requires a lot of manual feature engineering and strategy parameter tuning, resulting in high development and maintenance costs.

[0007] Accordingly, this disclosure provides a model training method to construct a unified cross-modal representation model, enabling post interaction content and voice product interaction content to be compared within the same semantic space. Furthermore, this disclosure also provides an online recall retrieval method that utilizes a trained generative recall model to recommend matching voice products to users.

[0008] In a first aspect of this disclosure, a method for training a fine-tuning model is provided, comprising: processing an image in a post sample using an image encoder in the fine-tuning model to obtain image features; processing post text in the post sample using a text encoder in the fine-tuning model to obtain text features; processing tag information of the post sample using the text encoder to obtain tag features; fusing the image features and the text features using a multilayer perceptron in the fine-tuning model to obtain fused features; aligning the fused features and the tag features to obtain post features of the post sample, which are used as the output of the fine-tuning model; obtaining a loss function using the image features, the text features, and the post features; and training the image encoder, the text encoder, and the multilayer perceptron using the loss function.

[0009] In some embodiments, obtaining a loss function using the image features, the text features, and the post features includes: generating a first sub-loss function based on the alignment deviation between the image features and the text features; generating a second sub-loss function based on the alignment deviation between the fused features and the tag features in the post features; and generating the loss function based on the first sub-loss function and the second sub-loss function.

[0010] In some embodiments, generating the loss function based on the first sub-loss function and the second sub-loss function includes generating the loss function based on a weighted sum of the first sub-loss function and the second sub-loss function.

[0011] In some embodiments, the length of the post text is greater than a preset text threshold, and the number of tags in the post sample is less than a preset number threshold.

[0012] In a second aspect of this disclosure, a generative recall model training method is provided, comprising: processing real-time interactive data using a fine-tuned model trained by the fine-tuning model training method described in any of the above embodiments to obtain a data representation vector; encoding the data representation vector using a predetermined encoder and storing the obtained encoding information in an encoding database; constructing a user profile sample for a specified user, wherein the user profile sample includes basic data of the specified user and an encoded sample sequence of the first n historical behaviors of the specified user extracted from the encoding database, where n is a positive integer; generating a prompt template, wherein the prompt template includes the user profile sample and instructional prompt information; inputting the prompt template into the generative recall model so that the generative recall model outputs a predicted encoding of the (n+1)th historical behavior of the specified user; determining a loss function based on the deviation between the encoded sample of the (n+1)th historical behavior of the specified user extracted from the encoding database and the predicted encoding; and training the generative recall model using the loss function.

[0013] In some embodiments, the encoded information belonging to voice chat users in the encoded database is written into the voice chat user profile database; the encoded information belonging to non-voice chat users in the encoded database is written into the non-voice chat user profile database.

[0014] In some embodiments, the predetermined encoder is a residual quantization variational autoencoder (RQ-VAE).

[0015] In some embodiments, the basic data of the specified user includes at least one of the specified user's gender, age, and geographic location.

[0016] In some embodiments, when the specified user is a voice chat user, the generative recall model is a voice chat scenario generative recall model, and the encoded sample sequence includes encoded information of at least one of voice chat information, click behavior, and dwell time related to the voice product.

[0017] In some embodiments, the voice product includes at least one of voice chat rooms, voice parties, and online game voice channels.

[0018] In some embodiments, when the specified user is a non-voice chat user, the generative recall model is a non-voice chat scenario generative recall model, and the encoded sample sequence includes encoded information of at least one of the following: like behavior, favorite behavior, and comment content related to the post.

[0019] In a third aspect of this disclosure, an online recall retrieval method is provided, comprising: upon receiving a retrieval request sent by a target user, determining the chat behavior of the target user; extracting a user profile of the target user from a user profile database corresponding to the chat behavior, wherein the user profile includes basic data of the target user and an encoded sequence of the target user's recent historical behaviors; generating a prompt template, wherein the prompt template includes the user profile and instructional prompt information; inputting the prompt template into a generative recall model corresponding to the chat behavior, so that the generative recall model outputs a predicted encoding of the target user's next behavior, wherein the generative recall model is trained using the generative recall model training method described in any of the above embodiments; decoding the predicted encoding to obtain a candidate representation vector; extracting candidate voice products corresponding to the candidate representation vector from a voice chat recall database; and presenting product information of the candidate voice products to the target user.

[0020] In some embodiments, presenting the product information of the candidate voice products to the target user includes: determining whether the candidate voice products include a first target voice product, wherein the first target voice product is currently online and accessible; if the candidate voice products include only one first target voice product, then presenting the product information of the first target voice product to the target user.

[0021] In some embodiments, if the candidate voice products include multiple first target voice products, a recommendation value for each of the multiple first target voice products is determined based on online behavior data; and product information of a predetermined number of the multiple first target voice products is presented to the target user in descending order of recommendation value.

[0022] In some embodiments, the online behavioral data includes at least one of the following: popularity of the voice product, dwell time, and conversion rate.

[0023] In some embodiments, when the target user's chat behavior is voice chat behavior, the user profile database is a voice chat user profile database, the generative recall model is a voice chat generative recall model, and the encoding sequence includes encoding information of at least one of voice chat information, click behavior, and dwell time related to the voice product.

[0024] In some embodiments, if the target user's chat behavior is non-voice chat behavior, the user profile database is a non-voice chat user profile database, the generative recall model is a non-voice chat generative recall model, and the encoding sequence includes at least one of the following: like behavior, favorite behavior, and comment content related to the post.

[0025] In a fourth aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any of the above embodiments.

[0026] In a fifth aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0027] In a sixth aspect of this disclosure, a computer program product is provided, including computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any of the above embodiments.

[0028] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a fine-tuning model training method according to an embodiment of the present disclosure;

[0031] Figure 2 This is a flowchart illustrating a fine-tuning model training method according to another embodiment of the present disclosure;

[0032] Figure 3 This is a flowchart illustrating a generative recall model training method according to an embodiment of this disclosure.

[0033] Figure 4 This is a schematic diagram of a prompt template according to an embodiment of the present disclosure;

[0034] Figure 5 This is a flowchart illustrating a generative recall model training method according to another embodiment of this disclosure.

[0035] Figure 6 This is a schematic flowchart of an online recall retrieval method according to an embodiment of the present disclosure;

[0036] Figure 7 This is a flowchart illustrating an online recall retrieval method according to another embodiment of the present disclosure;

[0037] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0038] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0039] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0040] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0041] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0042] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0043] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0044] Figure 1 This is a flowchart illustrating a fine-tuning model training method according to an embodiment of the present disclosure. In some embodiments, the following fine-tuning model training method is performed by an electronic device, including steps 11-17.

[0045] In step 11, the image encoder in the fine-tuning model is used to process the images in the post samples to obtain image features.

[0046] For example, fine-tuning models include CN-CLIP (Chinese Contrastive Language-Image Pre-training) or other suitable machine learning models.

[0047] It should be noted that the post samples include images and post text, and each post sample also has tag information. The tag information is short text that involves the title of the voice product, how to play, and other content.

[0048] In some embodiments, for the selected post sample, the text length of the post text is greater than a preset text threshold, and the number of tags in the post sample is less than a preset number threshold.

[0049] If the length of the post text exceeds a preset text threshold, effective alignment between the text features corresponding to the post text and the image features corresponding to the images in the post sample can be achieved. If the number of labels in the post sample is less than a preset number threshold, alignment between the label features and the text and image features in the post sample can be ensured.

[0050] In step 12, the text encoder in the fine-tuning model is used to process the post text in the post sample to obtain text features.

[0051] In step 13, the tag information of the post samples is processed using a text encoder to obtain tag features.

[0052] In step 14, the image features and text features are fused using the multilayer perceptron (MLP) in the fine-tuning model to obtain fused features.

[0053] In step 15, the fused features and label features are aligned to obtain the post features of the post samples, which are used as the output of the fine-tuning model.

[0054] In step 16, the loss function is obtained using image features, text features, and post features.

[0055] In some embodiments, the step of obtaining the loss function using image features, text features, and post features includes steps S101-S103.

[0056] S101. Generate the first sub-loss function based on the alignment deviation between image features and text features.

[0057] S102. Generate a second sub-loss function based on the alignment deviation between the fusion features and tag features in the post features.

[0058] S103. Generate the loss function based on the first sub-loss function and the second sub-loss function.

[0059] For example, a loss function is generated based on the weighted sum of the first and second sub-loss functions.

[0060] In step 17, the image encoder, text encoder, and multilayer perceptron are trained using the loss function.

[0061] Figure 2 This is a flowchart illustrating a fine-tuning model training method according to another embodiment of this disclosure.

[0062] like Figure 2 As shown, a post sample includes an image, post text, and corresponding tags. The image encoder in the fine-tuning model is used to process the image in the post sample to obtain image features. The text encoder in the fine-tuning model is used to process the post text in the post sample to obtain text features. The text encoder is then used to process the tags in the post sample to obtain tag features. Finally, the multilayer perceptron in the fine-tuning model is used to fuse the image features and text features to obtain fused features.

[0063] For example, the image encoder is a CN-CLIP image encoder, and the text encoder is a CN-CLIP text encoder.

[0064] A first sub-loss function is generated based on the alignment deviation between image features and text features. A second sub-loss function is generated based on the alignment deviation between fused features and label features. The final loss function is generated by weighting the first and second sub-loss functions.

[0065] For example, the loss function is the InfoNCE loss function.

[0066] Next, the image encoder, text encoder, and multilayer perceptron are trained using the loss function.

[0067] In the fine-tuning model training method provided in the above embodiments of this disclosure, the image encoder, text encoder and multilayer perceptron in the fine-tuning model are trained so that the image features of the images in the post samples output by the image encoder are aligned with the text features of the post text in the post samples output by the text encoder, and the image-text features of the post samples output by the multilayer perceptron are aligned with the label features of the labels in the post samples output by the text encoder. This enables the trained fine-tuning model to map the post interaction content and the voice product interaction content to a unified semantic representation space, providing comparable feature vectors for the generative recall model used for recommending voice products.

[0068] Figure 3 This is a schematic flowchart of a generative recall model training method according to an embodiment of the present disclosure. In some embodiments, the following generative recall model training method is performed by an electronic device, including steps 31-37.

[0069] In step 31, the real-time interactive data is processed using a fine-tuning model to obtain a data representation vector.

[0070] It should be noted that real-time interactive data includes at least one of the following: voice product interaction information and post interaction information.

[0071] In some embodiments, the fine-tuning model utilizes Figure 1 or Figure 2 The model is trained using the fine-tuning model training method described in any of the embodiments.

[0072] In step 32, the data representation vector is encoded using a predetermined encoder, and the resulting encoded information is stored in the encoding database.

[0073] In some embodiments, the encoded information belonging to voice chat users in the encoded database is written into the voice chat user profile database, and the encoded information belonging to non-voice chat users in the encoded database is written into the non-voice chat user profile database.

[0074] It's important to clarify that voice chat users refer to those who actively engage in voice chat, while non-voice chat users are those who do not. By establishing a voice chat user profile database and a non-voice chat user profile database, we can quickly extract corresponding user profiles for each type of user.

[0075] In some embodiments, the predefined encoder is RQ-VAE (Residual Quantized Variational AutoEncoder).

[0076] It should be noted that during the encoding process, RQ-VAE can map the original multimodal interaction content into a dense sequence of discrete semantic tokens, which can then be uniformly input into a large model to achieve the semantic expression of user interests.

[0077] For example, using RQ-VAE to encode the data representation vector of a speech room yields three sets of encoded indices [45, 128, 312], corresponding to the following character forms:<a_45><b_128><c_312> .

[0078] In step 33, a user profile sample for a specified user is constructed, wherein the user profile sample includes the basic data of the specified user and the encoded sample sequence of the first n historical behaviors of the specified user extracted from the encoded database, where n is a positive integer.

[0079] In some embodiments, the basic data of a specified user includes at least one of the specified user's gender, age, and geographic location.

[0080] In step 34, a prompt template is generated, which includes a user profile sample and instructional prompt information.

[0081] For example, a user's basic attribute data includes: gender - female, age - 20. This user recently interacted with three voice products, the first of which had the following encoded information:<a_12><b_43><c_87> The encoding information of the second voice product is<a_55><b_64><c_101> The encoding information of the third voice product is<a_78><b_95><c_210> If the instruction prompt is "Please predict the next voice product the user might interact with," then the generated prompt template will be as follows: Figure 4 As shown.

[0082] In step 35, the prompt template is input into the generative recall model so that the generative recall model outputs the predicted code of the (n+1)th historical behavior of the specified user.

[0083] For example, generative recall models include the Qwen2.5-0.5B model or other suitable machine learning models.

[0084] It's important to note that, unlike traditional recall systems that rely on multi-source feature retrieval, generative recall models directly learn the generation distribution of user behavior sequences through language modeling, thereby achieving autoregressive prediction of potential interest targets. This process can be viewed as an "encoding-to-encoding" generative task, where the encoded representation of the next potential target is generated from the encoded sequence of user behavior.

[0085] In step 36, the loss function is determined based on the coded sample of the (n+1)th historical behavior of the specified user extracted from the coding database and the deviation of the predicted coding.

[0086] In step 37, the generative recall model is trained using the loss function.

[0087] It should be noted here that the training of the generative recall model described above can also be called SFT (Supervised Fine-Tuning). The goal of fine-tuning is to enable the model to learn the following two types of capabilities:

[0088] 1. Dependency modeling capability between RQ-VAE tokens: that is, understanding the temporal and semantic relationships between different product codes in the user behavior sequence, so as to output a code sequence that conforms to the real interest migration pattern during generation.

[0089] 2. Command compliance and recommendation intent modeling capability: Through command-based sample training, the model can accurately generate results that conform to the context logic based on commands such as "predict the next interactive product".

[0090] During training, a sliding window prediction mechanism can be employed. For example, using the user's first to nth historical interactions as input, the RQ-VAE code corresponding to the (n+1)th interaction is predicted. The window then slides forward, using the user's second to (n+1)th historical interactions as input, to predict the RQ-VAE code corresponding to the (n+2)th interaction. This process continues until the end of the sequence is reached. This approach not only enhances the model's ability to capture the evolution of user interests but also effectively improves sample utilization and prediction stability.

[0091] For example, such as Figure 5 As shown, the user's historical behavior data includes three recent historical behavior data points, which respectively include interaction data with product 1, product 2, and product 3. In this case, a prompt template is generated, which includes the user's basic attribute data, the encoded sample sequence of the user's three recent historical behavior data points, and the instructional prompt information, such as... Figure 2 Next, the prompt template will be input into the generative recall model so that the generative recall model can output the predicted code of the user's fourth historical behavior.<a_3><b_2><c_4> .

[0092] The user's fourth historical action was an interaction with product 4. In this case, the encoding of the user's fourth historical action data is obtained.<a_3><b_3><c_3> and according to the encoding<a_3><b_3><c_3> The predicted encoding of the user's fourth historical behavior output by the generative recall model.<a_3><b_2><c_4> The bias determines the loss function, and this loss function is used to train the generative recall model.

[0093] In some embodiments, when the specified user is a voice chat user, the generative recall model is a voice chat scenario generative recall model, and the encoded sample sequence includes encoded information of at least one of voice chat information, click behavior, and dwell time related to the voice product.

[0094] For example, voice products include at least one of voice chat rooms, voice parties, and online game voice channels.

[0095] In some embodiments, when the specified user is a non-voice chat user, the generative recall model is a non-voice chat scenario generative recall model, and the encoded sample sequence includes encoded information of at least one of the following: post-related liking behavior, collection behavior, and comment content.

[0096] In the generative recall model training method provided in the above embodiments of this disclosure, RQ-VAE is used to encode users' historical behavior data, which can map the original multimodal interaction content into a dense sequence of discrete semantic tokens, thereby uniformly inputting it into a large model to achieve semantic expression of user interests. Furthermore, by supervising the fine-tuning of the generative recall model, it can learn the ability to model dependencies between RQ-VAE tokens, as well as the ability to model instruction compliance and recommendation intent.

[0097] Figure 6 This is a schematic flowchart of an online recall retrieval method according to an embodiment of the present disclosure. In some embodiments, the following online recall retrieval method is performed by an electronic device, including steps 61-67.

[0098] In step 61, upon receiving a retrieval request from the target user, the target user's chat behavior is determined.

[0099] It should be noted that if the target user engages in voice chat, then the target user can be identified as a voice chat user. If the target user does not engage in voice chat, for example, only interacts with posts, then the target user can be identified as a non-voice chat user.

[0100] In step 62, the user profile of the target user is extracted from the user profile database corresponding to the chat behavior, wherein the user profile includes the target user's basic data and the encoded sequence of the target user's recent historical behavior.

[0101] It should be noted that if the target user engages in voice chat, the user profile database corresponding to this chat behavior is a voice chat user profile database, and the corresponding encoding sequence includes at least one of the following: voice chat information related to the voice product, click behavior, and dwell time. If the target user does not engage in voice chat, the user profile database corresponding to this chat behavior is a non-voice chat user profile database, and the corresponding encoding sequence includes at least one of the following: like behavior, favorite behavior, and comment content related to posts.

[0102] In step 63, a prompt template is generated, which includes a user profile and instructional prompt information.

[0103] In step 64, the prompt template is input into the generative recall model corresponding to the chat behavior, so that the generative recall model outputs the predicted code of the target user's next behavior.

[0104] It should be noted here that generative recall models utilize... Figure 3 or Figure 5 The generative recall model is trained using the training method shown in any of the embodiments.

[0105] In some embodiments, if the target user's chat behavior is voice chat, the generative recall model is a voice chat generative recall model. If the target user's chat behavior is non-voice chat, the generative recall model is a non-voice chat generative recall model.

[0106] It should be noted that when the target user's chat behavior is voice chat, the instructional prompt message included in the prompt template is "predict the next voice product the user may interact with". In this case, the voice chat generative recall model predicts the next voice product the user may interact with.

[0107] When the target user's chat behavior is non-voice chat behavior, the instructional prompt information included in the prompt template is "predict the next post that the user may interact with". In this case, the non-voice chat generative recall model predicts the next post that the user may interact with, and through cross-modal representation space mapping, uses the encoded vector of the post to perform similarity retrieval in the voice product database to recall semantically relevant voice product information.

[0108] In step 65, the prediction code is decoded to obtain the candidate representation vector.

[0109] For example, RQ-VAE can be used to decode predictive codes.

[0110] In step 66, candidate voice products corresponding to candidate representation vectors are extracted from the voice chat recall database.

[0111] In step 67, the product information of the candidate voice products is presented to the target user.

[0112] In some embodiments, the step of presenting product information of candidate voice products to target users includes steps S201-S204.

[0113] S201. Determine whether the candidate voice products include the first target voice product, wherein the first target voice product is currently online and accessible.

[0114] If the candidate voice products include only one first target voice product, then execute S202; if the candidate voice products include multiple first target voice products, then execute S203.

[0115] S202, Present the product information of the first target voice product to the target user.

[0116] This ensures that the voice products recommended to users are all currently online and accessible, thereby effectively improving the user experience.

[0117] S203. Determine the recommended value for each of the multiple primary target voice products based on online behavior data.

[0118] In some embodiments, online behavioral data includes at least one of the following: popularity of voice products, dwell time, and conversion rate.

[0119] In other words, voice products are scored based on their popularity, dwell time, and conversion rate, and a weighted sum is calculated based on the scores to obtain a recommendation value for the voice product.

[0120] S204. Present the product information of a predetermined number of first target voice products to the target user in descending order of recommendation value.

[0121] It's important to note that this approach utilizes candidate representation vectors to perform vector similarity retrieval in a voice chat recall database, selecting the most semantically similar, online, and accessible voice products as the recall results. This method ensures both the immediate availability of the recalled content and maintains the semantic generalization ability of the generative recall model.

[0122] Figure 7 This is a schematic flowchart of an online recall retrieval method according to another embodiment of the present disclosure.

[0123] It should be noted here that... Figure 7 The fine-tuning model in the middle utilizes Figure 1 , 2 The model is trained using the fine-tuning model training method described in any of the embodiments.

[0124] like Figure 7 As shown, a fine-tuning model is used to process real-time interactive data to obtain a data representation vector. Real-time interactive data includes at least one of voice product interaction information and post interaction information. The data representation vector is encoded using an RQ-VAE encoder, and the resulting encoded information is stored in an encoding database. Furthermore, based on user interaction, encoded information belonging to voice chat users in the encoding database is written into a voice chat user profile database, and encoded information belonging to non-voice chat users in the encoding database is written into a non-voice chat user profile database.

[0125] To train the voice chat generative recall model, user profile samples of voice chat users are generated. These user profile samples include basic data of the voice chat users and encoded sample sequences of the user's first n historical behaviors extracted from a voice chat user profile database, where n is a positive integer. Next, a prompt template is generated, which includes the user profile samples of the voice chat users and command-based prompt information. The prompt template is input into the voice chat generative recall model so that the model outputs the predicted encoding of the user's (n+1)th historical behavior. Based on the deviation between the encoded sample of the user's (n+1)th historical behavior extracted from the voice chat user profile database and the predicted encoding, a loss function is determined, and this loss function is then used to train the voice chat generative recall model.

[0126] To train the non-voice chat generative recall model, user profile samples of non-voice chat users are generated. These user profile samples include basic data of non-voice chat users and encoded sample sequences of the first n historical behaviors of non-voice chat users extracted from a non-voice chat user profile database, where n is a positive integer. Next, a prompt template is generated, which includes the user profile samples of non-voice chat users and command-line prompt information. The prompt template is input into the non-voice chat generative recall model so that the model outputs the predicted encoding of the (n+1)th historical behavior of the non-voice chat user. Based on the deviation between the encoded sample of the (n+1)th historical behavior of the non-voice chat user extracted from the non-voice chat user profile database and the predicted encoding, a loss function is determined, and this loss function is then used to train the non-voice chat generative recall model.

[0127] Next, as Figure 7 As shown, the target user's chat behavior is determined based on the search request sent by the target user. If the target user engages in voice chat, they are identified as a voice chat user. If the target user does not engage in voice chat, for example, only interacts with posts, they are identified as a non-voice chat user.

[0128] If the target user is a voice chat user, then the user profile of the target user is extracted from the voice chat user profile database. The user profile includes the target user's basic data and a encoded sequence of the target user's recent historical behaviors. The encoded sequence includes at least one of the following: voice chat information related to the voice product, click behavior, and dwell time. A prompt template is generated, which includes the target user's user profile and instructional prompt information.

[0129] Next, the prompt template is input into the voice chat generative recall model so that the model outputs a predictive code for the next likely voice product the target user might interact with. The predictive code is then decoded to obtain a candidate representation vector. Multiple candidate voice products corresponding to the candidate representation vectors are extracted from the voice chat recall database. These candidate voice products are then sorted using posterior data to present the product information of the selected target voice product to the target user.

[0130] If the target user is not a voice chat user, then the target user's user profile is extracted from the non-voice chat user profile database. This user profile includes the target user's basic data and a encoded sequence of the target user's recent historical behaviors. The encoded sequence includes at least one of the following: liking, saving, and commenting related to posts. A prompt template is generated, which includes the target user's user profile and instructional prompt information.

[0131] Next, the prompt template is input into the non-voice chat generative recall model, enabling it to output the next possible post for the target user's interaction. Through cross-modal representation space mapping, the encoded vector of this post is used to perform a similarity search in the voice product database to obtain the encoded information of semantically related voice products. This encoded information is then decoded to obtain candidate representation vectors. Multiple candidate voice products corresponding to the candidate representation vectors are extracted from the voice chat recall database. The candidate voice products are then ranked using posterior data to present the product information of the selected target voice product to the target user.

[0132] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0133] like Figure 8 As shown, the electronic device 80 can be represented in the form of a general computing device. The electronic device 80 includes a memory 81, a processor 82, and a bus 83 connecting different system components.

[0134] The memory 81 may include, for example, system memory, non-volatile storage media, etc. System memory may store, for example, an operating system, application programs, a boot loader, and other programs. System memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. Non-volatile storage media may store, for example, instructions for corresponding embodiments of at least one fine-tuning model training method, generative recall model training method, and online recall retrieval method being executed. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0135] The processor 82 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the acquisition module, the calculation module, and the adjustment module, can be implemented by executing instructions in the central processing unit (CPU) running memory to perform the corresponding steps, or by implementing dedicated circuits that perform the corresponding steps.

[0136] For example, processor 82 is configured to execute instructions stored in memory 81, such as Figure 1-3 The method involved in any of the embodiments in 5-7.

[0137] For example, when processor 82 executes an implementation such as Figure 1-2 When the method is described in any of the embodiments, the electronic device can serve as a fine-tuning model training device. The processor 82 executes the implementation as described above. Figure 3 , 5 When the method is described in any of the embodiments, the electronic device can serve as a generative recall model training device. The processor 82 executes an implementation as described above. Figure 6-7 When any of the embodiments of the method are involved, the electronic device can be used as an online recall retrieval device.

[0138] Bus 83 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0139] The interfaces 84, 85, and 86 of the electronic device 80, as well as the memory 81 and processor 82, can be connected via bus 83. Input / output interface 84 provides a connection interface for input / output devices such as monitors, mice, and keyboards. Network interface 85 provides a connection interface for various networked devices. Storage interface 86 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0140] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0141] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0142] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0143] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0144] This disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figure 1-3 The method involved in any of the embodiments in 5-7.

[0145] This disclosure also provides a computer program product, including computer instructions, wherein the computer instructions, when executed by a processor, implement as follows: Figure 1-3 The method involved in any of the embodiments in 5-7.

[0146] By implementing the above embodiments of this disclosure, cross-scenario semantic alignment and real-time online recommendation can be effectively achieved, effectively improving the recommendation quality of voice products in interactive platforms and enhancing the user experience when using related products. Applying the solution of this disclosure to social platforms increases the effective reach days by 0.3615%, the penetration rate of related products by 0.2898%, and the deep user rate by 0.5598%.

[0147] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.

[0148] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0149] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. An online recall retrieval method, comprising: Upon receiving a retrieval request from a target user, determine the target user's chat behavior; Extract the user profile of the target user from the user profile database corresponding to the chat behavior, wherein the user profile includes the basic data of the target user and the encoded sequence of the target user's recent historical behavior; Generate a prompt template, wherein the prompt template includes the user profile and instructional prompt information; The prompt template is input into the generative recall model corresponding to the chat behavior, so that the generative recall model outputs the predicted code of the next behavior of the target user, wherein the generative recall model is trained using the generative recall model training method; The predictive encoding is decoded to obtain the candidate representation vector; Extract candidate voice products corresponding to the candidate representation vector from the voice chat recall database; Present the product information of the candidate voice products to the target user; The generative recall model training method includes: processing real-time interactive data using a fine-tuned model trained by a fine-tuning model training method to obtain a data representation vector; encoding the data representation vector using a predetermined encoder and storing the resulting encoded information in an encoding database; constructing a user profile sample for a specified user, wherein the user profile sample includes the basic data of the specified user and an encoded sample sequence of the specified user's first n historical behaviors extracted from the encoding database, where n is a positive integer; generating a prompt template, wherein the prompt template includes the user profile sample and instructional prompt information; inputting the prompt template into the generative recall model so that the generative recall model outputs a predicted encoding of the specified user's (n+1)th historical behavior; determining a loss function based on the deviation between the encoded sample of the specified user's (n+1)th historical behavior extracted from the encoding database and the predicted encoding; and training the generative recall model using the loss function. The fine-tuning model training method includes: processing the image in the post sample using an image encoder in the fine-tuning model to obtain image features; processing the post text in the post sample using a text encoder in the fine-tuning model to obtain text features; processing the label information of the post sample using the text encoder to obtain label features; fusing the image features and the text features using a multilayer perceptron in the fine-tuning model to obtain fused features; aligning the fused features and the label features to obtain the post features of the post sample, which are used as the output of the fine-tuning model; obtaining a loss function using the image features, the text features, and the post features; and training the image encoder, the text encoder, and the multilayer perceptron using the loss function.

2. The online recall search method of claim 1, wherein, Presenting the product information of the candidate voice products to the target user includes: Determine whether the candidate voice products include a first target voice product, wherein the first target voice product is currently online and accessible. If the candidate voice products include only one first target voice product, then the product information of the first target voice product will be presented to the target user.

3. The online recall retrieval method according to claim 2 further includes: If the candidate voice products include multiple first target voice products, then the recommendation value of each of the multiple first target voice products is determined based on online behavior data; According to the order of recommendation value from largest to smallest, the product information of a predetermined number of the multiple first target voice products is presented to the target user.

4. The online recall retrieval method according to claim 3, wherein, The online behavioral data includes at least one of the following: popularity of voice products, dwell time, and conversion rate.

5. The online recall retrieval method according to any one of claims 1-4, wherein, When the target user's chat behavior is voice chat behavior, the user profile database is a voice chat user profile database, the generative recall model is a voice chat generative recall model, and the encoding sequence includes encoding information of at least one of voice chat information, click behavior, and dwell time related to the voice product.

6. The online recall retrieval method according to any one of claims 1-4, wherein, If the target user's chat behavior is non-voice chat behavior, the user profile database is a non-voice chat user profile database, the generative recall model is a non-voice chat generative recall model, and the encoding sequence includes at least one of the following: post-related liking behavior, collection behavior, and comment content.

7. The online recall search method of claim 1, wherein, The generative recall model training method also includes: Write the encoded information belonging to voice chat users in the encoded database into the voice chat user profile database; Write the encoded information belonging to non-voice chat users in the encoded database into the non-voice chat user profile database.

8. The online recall retrieval method according to claim 7, wherein, The predetermined encoder is a residual quantization variational autoencoder (RQ-VAE).

9. The online recall retrieval method according to claim 1, wherein, The basic data of the specified user includes at least one of the following: the user's gender, age, and geographical location.

10. The online recall retrieval method according to claim 1, wherein, When the specified user is a voice chat user, the generative recall model is a voice chat scenario generative recall model, and the encoded sample sequence includes encoded information of at least one of voice chat information, click behavior, and dwell time related to the voice product.

11. The online recall retrieval method according to claim 10, wherein, The voice products include at least one of voice chat rooms, voice parties, and online game voice channels.

12. The online recall retrieval method according to claim 1, wherein, When the specified user is a non-voice chat user, the generative recall model is a generative recall model for non-voice chat scenarios, and the encoded sample sequence includes encoded information of at least one of the following: like behavior, favorite behavior, and comment content related to the post.

13. The online recall search method of claim 1, wherein, The loss function obtained using the image features, the text features, and the post features includes: A first sub-loss function is generated based on the alignment deviation between the image features and the text features; A second sub-loss function is generated based on the alignment deviation between the fused features and the tag features in the post features; The loss function is generated based on the first sub-loss function and the second sub-loss function.

14. The online recall search method of claim 13, wherein, Generating the loss function based on the first sub-loss function and the second sub-loss function includes: The loss function is generated by weighting the first sub-loss function and the second sub-loss function.

15. The online recall retrieval method according to claim 1, wherein, The length of the post text is greater than a preset text threshold, and the number of tags in the post sample is less than a preset number threshold.

16. An electronic device comprising: Memory; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-15 based on memory-stored instruction execution.

17. A computer readable storage medium, wherein, A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-15.

18. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any one of claims 1-15.

Images