Model processing methods, model processing devices, computer program products and equipment

Abstract

This application discloses a model processing method, a model processing device, a computer program product, and an equipment. The method includes: acquiring content data of multimedia resources and interaction data of sample objects with respect to the multimedia resources; adaptively training a recommendation model using the content data and interaction data to obtain a first recommendation model; acquiring collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources and object collaborative features of sample objects, wherein the multimedia collaborative features represent the content representation of multimedia resources in the online recommendation scenario, and the object collaborative features represent the object representation of sample objects in the online recommendation scenario; optimizing and training the first recommendation model using the collaborative features to obtain a trained recommendation model; and using the trained recommendation model to recall multimedia resources recommended to objects. Using this application can improve the accuracy of recommending multimedia resources to objects.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model processing method, a model processing device, a computer program product, and an equipment. Background Technology

[0002] In the scenario of recommending multimedia resources, a trained model can be used to generate features of multimedia resources and features of the user to be recommended resources. Thus, multimedia resources can be recommended to the user based on the similarity between the features of the multimedia resources and the features of the user. For example, multimedia resources with features similar to the user's features can be recommended to the user.

[0003] The training model is usually trained directly using information from the multimedia resources themselves and the user's own information. Its training dimensions and methods are very limited, which means that the trained model cannot accurately generate the features of the multimedia resources and the user. Consequently, the multimedia resources recommended to the target audience based on the generated features are also inaccurate. Summary of the Invention

[0004] This application provides a model processing method, a model processing apparatus, a computer program product, and a device that can improve the accuracy of recommending multimedia resources to objects.

[0005] This application provides a model processing method, which includes:

[0006] Acquire the content data of multimedia resources in the multimedia resource set, and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set;

[0007] The recommendation model is adaptively trained using content data and interaction data to obtain the first recommendation model;

[0008] Acquire collaborative features; collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in the online recommendation scenario;

[0009] The first recommendation model is optimized and trained using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

[0010] This application provides a model processing apparatus, which includes:

[0011] The first acquisition module is used to acquire the content data of multimedia resources in the multimedia resource set and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set.

[0012] The training module is used to adaptively train the recommendation model using content data and interaction data to obtain the first recommendation model;

[0013] The second acquisition module is used to acquire collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in the online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in the online recommendation scenario.

[0014] The optimization module is used to optimize and train the first recommendation model using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

[0015] Optionally, if the collaborative features include multimedia collaborative features and object collaborative features, the optimization module uses the collaborative features to optimize and train the first recommendation model to obtain the trained recommendation model in the following ways:

[0016] The first recommendation model is optimized and trained using multimedia collaborative features to obtain the second recommendation model;

[0017] The second recommendation model is optimized and trained using object collaborative features to obtain the trained recommendation model.

[0018] Optionally, the recommendation model includes a multimedia model and an object model. The multimedia model is used to represent multimedia resources, and the object model is used to represent objects.

[0019] The adaptive training of the recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the recommendation model;

[0020] The optimization training of the first recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the first recommendation model;

[0021] The optimization training of the second recommendation model includes: keeping the model parameters of the multimedia model in the second recommendation model unchanged, and correcting the model parameters of the object model in the second recommendation model.

[0022] Optionally, the content data is a sequence of K sub-content data from the corresponding multimedia resource, where K is a positive integer; the optimization module uses multimedia collaborative features to optimize and train the first recommendation model to obtain the second recommendation model in the following ways:

[0023] The multimedia model in the first recommendation model is invoked to learn features for each sub-content data in the content data, generating sub-content features for each sub-content data.

[0024] The multimedia collaborative features and the K sub-content features of the K sub-content data are sequentially concatenated to obtain the first feature sequence;

[0025] The first recommendation model is optimized and trained based on the first feature sequence to obtain the second recommendation model;

[0026] Among them, there is one multimedia collaborative feature, and the multimedia collaborative feature and K sub-content features are K+1 elements in the first feature sequence. One element is located at the position of an element in the first feature sequence, and the multimedia collaborative feature is located at the position of the first element in the first feature sequence.

[0027] Optionally, the optimization module can optimize and train the first recommendation model based on the first feature sequence to obtain the second recommendation model in the following ways:

[0028] The multimedia model in the first recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence;

[0029] The object model in the first recommendation model is invoked to learn features from the interaction data, generating the second interaction features of the sample objects.

[0030] Obtain the second feature similarity between the multimedia fusion features and the second interaction features of the multimedia resources in the multimedia resource set;

[0031] The model parameters of the first recommendation model are corrected based on the similarity of the second feature to obtain the second recommendation model.

[0032] Optionally, the multimedia model in the first recommendation model has a first attention mask; the optimization module calls the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence, including the following methods:

[0033] The multimedia model in the first recommendation model is invoked to perform attention processing on the elements in the first feature sequence based on the first attention mask, generating fused sub-features corresponding to the position of each element in the first feature sequence.

[0034] The fusion sub-feature corresponding to the position of the last element in the first feature sequence is used as the multimedia fusion feature of the multimedia resources in the multimedia resource set.

[0035] The first attention mask is used to generate a fusion sub-feature corresponding to the i-th element position based on the element at the i-th element position in the first feature sequence and the elements at the element positions before the i-th element position, where i is a positive integer and i is less than or equal to K+1.

[0036] Optionally, the interaction data is a sequence of L multimedia resources with interactive behaviors for the sample objects, where L is a positive integer; the optimization module uses object collaborative features to optimize and train the second recommendation model to obtain the trained recommendation model in the following ways:

[0037] Obtain the multimedia fusion features of each of the L multimedia resources. The L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model.

[0038] The object collaborative features are concatenated with the L multimedia fusion features in the form of a prefix of the L multimedia fusion features to obtain the second feature sequence;

[0039] In the object model of the second recommendation model, add L prefix model parameters corresponding to multimedia fusion features, and correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model;

[0040] Among them, there are P object collaboration features, where P is a positive integer. The P object collaboration features and L multimedia fusion features are the P+L elements in the second feature sequence. One element is located at the position of one element in the second feature sequence. The P object collaboration features are located at the positions of the first P elements in the second feature sequence.

[0041] Optionally, the optimization module can correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model in the following ways:

[0042] The multimedia model in the second recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on content data and multimedia collaboration features;

[0043] The newly added second recommendation model with prefixed model parameters is invoked to generate object fusion features of the sample objects based on the second feature sequence.

[0044] Obtain the third feature similarity between the multimedia fusion features of multimedia resources in the multimedia resource set and the object fusion features of sample objects;

[0045] The training recommendation model is obtained by correcting the parameters of the newly added prefix model in the second recommendation model based on the third feature similarity.

[0046] Optionally, the object model in the second recommendation model has a second attention mask; the optimization module calls the newly added second recommendation model with prefixed model parameters to generate object fusion features of the sample objects based on the second feature sequence, including:

[0047] The newly added second recommendation model with prefixed model parameters is invoked. Based on the second attention mask, attention processing is performed on the elements in the second feature sequence to generate the fused sub-feature corresponding to each element position in the second feature sequence.

[0048] The fusion sub-feature corresponding to the position of the last element in the second feature sequence is used as the object fusion feature of the sample object;

[0049] The second attention mask is used to generate a fusion sub-feature corresponding to the j-th element position based on the element at the j-th element position in the second feature sequence and the elements at the element positions before the j-th element position, where j is a positive integer and j is less than or equal to P+L.

[0050] Optionally, the interaction data is a sequence of L multimedia resources with interactive behaviors for the sample object, where L is a positive integer; the training module uses content data and interaction data to adaptively train the recommendation model to obtain the first recommendation model in the following ways:

[0051] The recommendation model is invoked to learn features from the content data, generating multimedia content features of multimedia resources in the multimedia resource set.

[0052] The recommendation model is invoked to generate multimedia content features of L multimedia resources, and the first interaction features of the sample object are generated based on the L multimedia content features of the L multimedia resources.

[0053] Obtain the first feature similarity between the multimedia content features and the first interaction features of the multimedia resources in the multimedia resource set;

[0054] The model parameters of the recommendation model are corrected based on the similarity of the first feature to obtain the first recommendation model.

[0055] Optionally, the training module may adjust the model parameters of the recommendation model based on the first feature similarity to obtain the first recommendation model in the following ways:

[0056] The interactive behavior information of the sample object to the multimedia resources in the multimedia resource set is obtained. The interactive behavior information is used to indicate whether the sample object has interactive behavior or not interactive behavior to the multimedia resources in the multimedia resource set.

[0057] Based on the interaction behavior information, obtain the reference feature similarity between the sample object and the multimedia resources in the multimedia resource set;

[0058] By analyzing the difference between the first feature similarity and the reference feature similarity, the model parameters of the recommendation model are corrected to obtain the first recommendation model.

[0059] Optionally, the trained recommendation model includes a multimedia model and an object model, wherein the multimedia model is used to represent multimedia resources and the object model is used to represent objects; the aforementioned model processing device further includes a recommendation module, which is used for:

[0060] Get Q candidate multimedia resources and the target object to be recommended, where Q is a positive integer;

[0061] The multimedia model in the trained recommendation model is called to generate the multimedia fusion features of each candidate multimedia resource, and the object model in the trained recommendation model is called to generate the object fusion features of the target object.

[0062] Obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource;

[0063] The Q candidate multimedia resources are sorted in descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource to obtain the sorted Q candidate multimedia resources.

[0064] The top R candidate multimedia resources from the sorted Q candidate multimedia resources are recommended to the target object; R is a positive integer and R is less than Q.

[0065] Optionally, the recommendation module may use the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource in the following ways:

[0066] Obtain candidate content data and candidate multimedia collaborative features for any candidate multimedia resource; the candidate multimedia collaborative features are used to represent the content representation of any candidate multimedia resource in an online recommendation scenario.

[0067] The multimedia model in the trained recommendation model is invoked, and multimedia fusion features of any candidate multimedia resource are generated based on candidate content data and candidate multimedia collaborative features.

[0068] Optionally, the recommendation module may use methods such as calling the object model in the trained recommendation model to generate object fusion features of the target object, including:

[0069] Obtain the target interaction data and target object collaborative features of the target object; the target interaction data is a sequence of L target multimedia resources with interactive behaviors of the target object, where L is a positive integer; the target object collaborative features are used to represent the object representation of the target object in online recommendation scenarios.

[0070] The multimedia model in the trained recommendation model is invoked to generate multimedia fusion features for each target multimedia resource;

[0071] The object model in the trained recommendation model is invoked, and the object fusion feature of the target object is generated based on the L multimedia fusion features of L target multimedia resources and the target object collaborative features.

[0072] This application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method of this application.

[0073] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in the above-mentioned aspect.

[0074] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in the various alternative embodiments described above.

[0075] This application can acquire content data of multimedia resources in a multimedia resource set and interaction data of sample objects with respect to multimedia resources in the multimedia resource set; it can also use the content data and interaction data to adaptively train a recommendation model to obtain a first recommendation model; and it can acquire collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in an online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in an online recommendation scenario; and the first recommendation model can be optimized and trained using the collaborative features to obtain a trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to objects. As can be seen, the method proposed in this application can perform multi-stage progressive training on the recommendation model. In this multi-stage progressive training, the recommendation model can first undergo overall adaptive training, allowing it to initially learn information related to multimedia resources and objects. Then, collaborative features (such as multimedia collaborative features of multimedia resources and / or object collaborative features of sample objects) can be introduced to further optimize the training of the recommendation model, enabling it to learn the existing features of multimedia resources and / or sample objects in online recommendation scenarios. This can greatly improve the training effect of the recommendation model, resulting in a superior recommendation model. The trained recommendation model can then accurately recall multimedia resources recommended to objects. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a schematic diagram of the network architecture of a recommendation network provided in an embodiment of this application;

[0078] Figure 2 This is a schematic diagram of a model training scenario provided in an embodiment of this application;

[0079] Figure 3 This is a schematic flowchart of a model processing method provided in an embodiment of this application;

[0080] Figure 4 This is a schematic diagram illustrating the effect of a recommended interface for displaying multimedia resources provided in an embodiment of this application;

[0081] Figure 5This is a schematic diagram of a process for optimizing and training a first recommendation model according to an embodiment of this application;

[0082] Figure 6 This is a schematic diagram of the structure of a first attention mask provided in an embodiment of this application;

[0083] Figure 7 This is a schematic diagram of a scenario for optimizing and training a first recommendation model, provided in an embodiment of this application.

[0084] Figure 8 This is a schematic diagram of a process for optimizing and training a second recommendation model according to an embodiment of this application;

[0085] Figure 9 This is a schematic diagram of the structure of a second attention mask provided in an embodiment of this application;

[0086] Figure 10 This is a schematic diagram of a scenario for optimizing and training a second recommendation model, provided in an embodiment of this application.

[0087] Figure 11 This is a schematic diagram of a process for recommending multimedia resources to an object, provided in an embodiment of this application;

[0088] Figure 12 This is a schematic diagram illustrating a scenario of recommending multimedia resources to a user, as provided in an embodiment of this application.

[0089] Figure 13 This is a schematic diagram of the structure of a model processing device provided in an embodiment of this application;

[0090] Figure 14 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0091] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0092] First, it should be noted that all data collected in this application (such as content data of multimedia resources, interaction data of sample objects with respect to multimedia resources, multimedia collaborative features, and related data such as object collaborative features) were collected with the consent and authorization of the data subject (such as users, institutions, or enterprises), and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0093] Artificial intelligence (AI) is a new technical science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. This application primarily concerns machine learning within artificial intelligence.

[0094] LLM: Large Language Model, a natural language processing model based on deep learning, designed to understand and generate natural language text. LLMs typically consist of hundreds of millions to hundreds of billions of parameters and are capable of handling complex language tasks.

[0095] Embedding: A vector representation of data that has been embedded.

[0096] Prefix-tuning: An efficient parameter fine-tuning method designed to adapt to a specific task by adding a trainable "prefix" to the input sequence without adjusting all parameters of the entire model.

[0097] GNN: A deep learning model for processing and analyzing graph-structured data, which learns and propagates information by utilizing the relationships between nodes.

[0098] Please see Figure 1 , Figure 1 This is a schematic diagram of the network architecture of a recommendation network provided in an embodiment of this application. For example... Figure 1 As shown, this network architecture may include a server 200 and a terminal device cluster. The terminal device cluster may include one or more terminal devices; the number of terminal devices is not limited here. Figure 1 As shown, multiple terminal devices may specifically include terminal device 1, terminal device 2, terminal device 3, ..., terminal device n; as... Figure 1 As shown, terminal device 1, terminal device 2, terminal device 3, ..., terminal device n can all connect to server 200 via the network, so that each terminal device can interact with server 200 through the network connection.

[0099] like Figure 1 The server 200 shown can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart TV, in-vehicle terminal, smart home device, or other smart terminal. The following description uses the communication between terminal device 1 and server 200 as an example to illustrate the specific implementation of this application.

[0100] In this system, terminal device 1 can be a user's terminal, which may include a client. This client is used to receive and view multimedia resources recommended to the user. Server 200 can be a background server on the client. Server 200 can recall the multimedia resources recommended to the user and push them to the client, allowing the client to display the received multimedia resources on its front-end interface for the user to view. Server 200 can recall the multimedia resources recommended to the user using a trained recommendation model. The overall training process of this recommended model can be found below. Figure 2 As described in the corresponding embodiments, and specifically how the trained recommendation model recalls multimedia resources recommended to users, please refer to the following: Figure 11 The relevant descriptions in the corresponding embodiments.

[0101] Please see also Figure 2 , Figure 2 This is a schematic diagram of a model training scenario provided in an embodiment of this application. For example... Figure 2 As shown above, Figure 1 Server 200 can obtain a multimedia resource set, which may contain at least one multimedia resource for training the recommendation model. Server 200 can obtain content data of the multimedia resources in the multimedia resource set, as well as interaction data of sample objects with respect to the multimedia resources in the multimedia resource set. The content data reflects the specific content of the multimedia resources, and the interaction data reflects the interactive behavior of the sample objects with respect to the multimedia resources, such as click behavior.

[0102] In addition, server 200 can also acquire collaborative features, which may include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects. The multimedia collaborative features may be features learned and generated from multimedia resources in the online recommendation scenario, and the object collaborative features may be features learned and generated from sample objects in the online recommendation scenario. The online recommendation scenario in which the multimedia resources reside and the online recommendation scenario in which the sample objects reside may be different or the same. The following explanation, using the example of collaborative features including both multimedia and object collaborative features, illustrates how to train the recommendation model.

[0103] Therefore, the server can first use the aforementioned content data and interaction data to adaptively train the recommendation model (also known as pre-training) to obtain a first recommendation model. This first recommendation model can initially learn the content features of multimedia resources and the preference features of sample objects for multimedia resources. Then, the server 200 can incorporate multimedia collaborative features to optimize the first recommendation model, resulting in a second recommendation model. This second recommendation model can effectively incorporate multimedia collaborative features of multimedia resources. Furthermore, the server 200 can incorporate object collaborative features to further optimize the second recommendation model, resulting in a final trained recommendation model. This final trained recommendation model can then effectively incorporate object collaborative features of sample objects.

[0104] Using the method provided in this application, the recommendation model can be progressively trained through a pipeline, gradually injecting multimedia collaborative information and object collaborative information into the recommendation model. This allows the recommendation model to learn not only the content features of the multimedia resources themselves and the preference features of the sample objects for the multimedia resources, but also the rich features (such as multimedia collaborative features and object collaborative features) already existing in other sources (such as online recommendation scenarios) of the multimedia resources and sample objects. As a result, a recommendation model with excellent training performance can be obtained. Through this trained recommendation model, the multimedia resources recommended to the object can be accurately recalled.

[0105] Please see Figure 3 , Figure 3 This is a flowchart illustrating a model processing method provided in an embodiment of this application. The executing entity in this embodiment can be a model processing device (hereinafter referred to as a processing device), which can be a single computer device or a cluster of multiple computer devices. The computer device can be a server or other devices; this application does not impose any limitations on this.

[0106] like Figure 3 As shown, the method may include:

[0107] Step S101: Obtain the content data of the multimedia resources in the multimedia resource set and the interaction data of the sample objects with respect to the multimedia resources in the multimedia resource set.

[0108] Optionally, the processing device can acquire the content data of multimedia resources in a multimedia resource set. The multimedia resource set can be a collection of several multimedia resources used to train the recommendation model. The content data of the multimedia resources can be data used to refer to (or describe) the resource content of the multimedia resources. The multimedia resources can be any type of resource that needs to be recommended, such as video, image, article, product, or live broadcast room.

[0109] In this application, the content data of a multimedia resource can be text data (i.e., data in text format). If the multimedia resource itself is a text-type resource, such as an article, then the content data of the multimedia resource can be the text content contained within the multimedia resource itself; while if the multimedia resource itself is not a text-type resource, such as a video, image, or product, then the content data of the multimedia resource can be the text content contained in the text data used to describe the multimedia resource. This text data used to describe the multimedia resource can be called the text description data of the multimedia resource, and the text description data can describe the content of the multimedia resource in the form of text.

[0110] For example, if a multimedia resource is a product, its text description data can be used to describe information such as the product's type, color, function, model, and brand. Similarly, if a multimedia resource is an image, its text description data can be used to describe the image's type, objects contained within the image, its color, size, and source. Each multimedia resource can have one set of content data, and the specific number of multimedia resources in a multimedia resource set can be determined based on the actual application scenario.

[0111] Specifically, the content data of a multimedia resource can be a sequence consisting of K sub-content data of the multimedia resource, where K is a positive integer and represents the maximum length of the multimedia resource's content data. If the multimedia resource itself is a text-based resource, then the K sub-content data can be contained within the multimedia resource itself. These K sub-content data can be obtained by segmenting the multimedia resource into words, with each sub-content data being a word within that multimedia resource. The order of these K words within the multimedia resource is the same as their order within the multimedia resource's content data.

[0112] If the multimedia resource itself is not a text-type resource, then the K sub-content data can be contained in the text description data of the multimedia resource. The K sub-content data can be obtained by segmenting the text description data of the multimedia resource into words. That is, a sub-content data can be a word in the text description data. The order of the K words in the text description data of the multimedia resource is the same as the order in the content data of the multimedia resource.

[0113] Optionally, if the number of segments obtained by segmenting the multimedia resources themselves or the text description data of the multimedia resources is less than K, then zero elements can be added after the obtained segments to obtain the content data of the multimedia resources, so that the sequence length of the content data is still padded to K.

[0114] Therefore, the content data of multimedia resources can also be called a word sequence. This word sequence can contain multiple words. A word can be a word obtained by segmenting the multimedia resource itself or the text description data of the multimedia resource, or a supplementary 0 element. For example, if K is 6, the content data of a multimedia resource can be the sequence [t1, t2, t3, t4, t5, t6], where t1 to t6 can be 6 words obtained through word segmentation. As another example, if K is 6, the content data of a multimedia resource can also be the sequence [t1, t2, t3, t4, 0, 0], where t1 to t4 can be 4 words obtained through word segmentation, and the two 0s after t4 can be supplementary 0 elements to keep the sequence length at 6.

[0115] The processing device can also acquire interactive data of sample objects on multimedia resources within a multimedia resource set. Sample objects can be sample users, or there can be multiple sample objects; the number of sample objects can be determined based on the actual application scenario. The processing principle for each sample object is the same. The interactive data of a sample object on multimedia resources can be a sequence consisting of L multimedia resources with which the sample object has interactive behavior, where L is a positive integer and can be the maximum sequence length of the interactive data. Each sample object can have one set of interactive data.

[0116] Optionally, the interactive behavior of the sample object towards multimedia resources can be a click behavior, or other behaviors, such as the behavior of viewing multimedia resources for a duration exceeding a set duration threshold after clicking on them. This application does not impose any restrictions on this. For example, the L multimedia resources can be the top L multimedia resources with interactive behavior that the sample object has within a recent time period (i.e., a recent time period, such as within a week), such as the top L multimedia resources for which interactive behavior was executed first within that time period. The order of these L multimedia resources in the sample object's interaction data can be the order in which the sample object performs interactive behavior on these L multimedia resources.

[0117] If the number of multimedia resources with which the user has interactive behavior is less than L, the interaction data of the sample object with respect to the multimedia resources can be obtained by padding zero elements after the multimedia resources with which the user has interactive behavior, so that the sequence length of the interaction data is still kept to L.

[0118] For example, if L is 10, the interaction data of a sample object can be the sequence [u1, u2, u3, u4, u5, u6, u7, u8, u9, u10], where u1 to u10 can be 10 multimedia resources that the user has interactive behavior with, and the multimedia resources can exist in the sequence through the identifier of the multimedia resources; as another example, if L is 10, the content data of a multimedia resource can also be the sequence [u1, u2, u3, u4, u5, u6, 0, 0, 0, 0], where u1 to u6 can be 6 multimedia resources that the user has interactive behavior with, and the 4 0s after u6 can be supplementary 0 elements to keep the length of the sequence 10.

[0119] In summary, the content data of multimedia resources can reflect the semantic content of the multimedia resources themselves, while the interaction data of sample objects with multimedia resources can reflect the sample objects' interest and preferences for multimedia resources. For example, the more multimedia resources that sample objects trigger interactive behavior with, the more likely they are to be the multimedia resources that sample objects prefer.

[0120] Step S102: Adaptively train the recommendation model using content data and interaction data to obtain the first recommendation model.

[0121] Optionally, the processing device can use the content data of the multimedia resources in the aforementioned multimedia resource set and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set to adaptively train the recommendation model. The recommendation model after adaptive training can be referred to as the first recommendation model. This adaptive training can be a comprehensive training of the recommendation model as a whole, enabling the recommendation model to initially learn and perceive the semantic content of the multimedia resources and the interest preferences of sample objects with respect to the multimedia resources, that is, to enable the recommendation model to initially adapt to the application scenario of recommending multimedia resources to objects in this application. This adaptive training can also be referred to as pre-training in this application.

[0122] Optionally, the recommendation model to be trained in this application can be a commercially available pre-trained model, such as a publicly available open-source model that already possesses corresponding data processing capabilities, such as the ability to process text data. In this application, the recommendation model can be specifically adjusted for a particular recommendation scenario, such as recommending multimedia resources to an object. That is, training the recommendation model in this application can refer to adjusting the model in a specific direction for a specific application scenario, so that the adjusted recommendation model can be applied to that specific application scenario, such as recommending multimedia resources to an object.

[0123] The recommendation model can include a multimedia model and an object model. The multimedia model can be used to represent multimedia resources, and the object model can be used to represent objects. That is, the multimedia model can be used to generate features of multimedia resources, and the object model can be used to generate features of objects (i.e., users). Optionally, both the multimedia model and the object model can be LLM (Large Language Model). In other words, this application can use two LLM models to construct the recommendation model. These two LLM models can be identical. One LLM model can be used to represent multimedia resources (in which case it is the multimedia model), and the other LLM model can be used to represent objects (in which case it is the object model).

[0124] The adaptive training of the recommendation model can include simultaneously correcting the model parameters of both the multimedia model and the object model within the recommendation model. In other words, during this training phase, the model parameters of both the multimedia model and the object model need to be trained and corrected. The process of adaptive training of the recommendation model can be described as follows.

[0125] It should be noted that during adaptive training, this application can perform multiple rounds of iterative training on the recommendation model using multimedia resources and sample objects from the multimedia resource set. The multimedia resources and sample objects used in different rounds of iterative training can be different; that is, the content data of the multimedia resources and the interaction data of the sample objects used in different rounds of iterative training can be different. The training principle of each round of iterative training is the same. Therefore, the following description can be based on the iterative training process of any round of the recommendation model.

[0126] The processing device can invoke a recommendation model (which may be the multimedia model within the recommendation model) to perform feature learning (which may be feature embedding) on ​​the content data of multimedia resources in the multimedia resource set (such as the content data of multimedia resources in any round) to generate multimedia content features of the multimedia resources in the multimedia resource set. These multimedia content features reflect the semantic features of the multimedia resources. Feature learning on the content data of a multimedia resource can generate a corresponding multimedia content feature. The multimedia content features of the multimedia resource can be generated by the multimedia model within the recommendation model.

[0127] The process by which the processing device calls the recommendation model to learn features from the content data of multimedia resources in the multimedia resource set, in order to generate multimedia content features of the multimedia resources in the multimedia resource set, may include:

[0128] The processing device can invoke the multimedia model within the recommendation model to perform feature learning (such as feature embedding) on ​​each sub-content data in the multimedia resource's content data to generate sub-content features for each sub-content data. A sub-content data can have one sub-content feature, and the K sub-content features of the K sub-content data can constitute a feature sequence (which can be called the fourth feature sequence). A sub-content feature can be an element in this fourth feature sequence, and this element is located at the position of an element in the fourth feature sequence. Furthermore, the multimedia model in the recommendation model can have an attention mask (which can be called the native attention mask of the multimedia model).

[0129] The multimedia model in the recommendation model can use this native attention mask to apply attention to the elements in the fourth feature sequence to generate multimedia sub-features corresponding to each element position in the fourth feature sequence. The processing device can then use the multimedia sub-feature corresponding to the last element position in the fourth feature sequence as the multimedia content feature of the aforementioned multimedia resource. Specifically, the native attention mask can be used to generate the multimedia sub-feature corresponding to the c-th element position based on the element at the c-th element position in the fourth feature sequence and the elements at the element positions preceding the c-th element position. Therefore, the multimedia sub-feature corresponding to the last element position is a feature generated by fusing the features of the elements at each of the preceding element positions, and can serve as an accurate multimedia content feature of the multimedia resource. c is a positive integer, and c is less than or equal to K. In other words, when generating multimedia sub-features corresponding to each element position in the fourth feature sequence, only the elements at the current element position and the elements at the preceding element positions need to be considered, without needing to consider the elements at the element positions following the current element position.

[0130] The processing device can also invoke a recommendation model to perform feature learning (or feature embedding) on ​​the interaction data of a sample object (such as the interaction data of a sample object with respect to multimedia resources in a multimedia resource set at any round) to generate a first interaction feature for that sample object. This first interaction feature can reflect the sample object's interest in multimedia resources. Performing feature learning on the interaction data of a sample object can generate a corresponding first interaction feature, which can be a feature vector.

[0131] The process of the processing device calling the recommendation model to perform feature learning on the interaction data of the sample object in order to generate the first interaction feature of the sample object may include: the processing device may first call the recommendation model (which may be the multimedia model in the recommendation model) to generate multimedia content features of L multimedia resources in the interaction data of the sample object. The multimedia content feature of one of the L multimedia resources may also be obtained by the multimedia model in the recommendation model to perform feature learning on the content data of the multimedia resource. A multimedia content feature may be a feature vector.

[0132] The processing device can also invoke the recommendation model to generate the first interaction feature of the sample object using the L multimedia content features of the L multimedia resources. This includes: the L multimedia content features can form a feature sequence of length L (which can be called the third feature sequence), where a multimedia content feature is an element in the third feature sequence, and the element is located at the position of an element in the third feature sequence. Furthermore, the object model in the recommendation model can have an attention mask (which can be called the native attention mask of the object model).

[0133] The object model in the recommendation model can use this native attention mask to apply attention to the elements in the third feature sequence to generate interactive sub-features corresponding to each element position in the third feature sequence. The processing device can then use the interactive sub-feature corresponding to the last element position in the third feature sequence as the first interactive feature of the sample object. Specifically, this native attention mask can be used to generate the interactive sub-feature corresponding to the s-th element position based on the element at the s-th element position in the third feature sequence and the elements at the element positions preceding the s-th element position. Therefore, the interactive sub-feature corresponding to the last element position is a feature generated by fusing the features of the elements at all preceding element positions, and can be used as the accurate first interactive feature of the sample object. s is a positive integer, and s is less than or equal to L. In other words, when generating the interactive sub-features corresponding to each element position in the third feature sequence, only the elements at the current element position and the elements at the preceding element positions need to be considered, without needing to consider the elements at the element positions following the current element position.

[0134] Therefore, the processing device can obtain the feature similarity (which can be called the first feature similarity) between the multimedia content features of multimedia resources (such as multimedia resources in any round) in the multimedia resource set and the first interaction features of sample objects (which can be sample objects in any round). A multimedia content feature and a first interaction feature can have a feature similarity, that is, a multimedia resource and a sample object can have a feature similarity. Both the multimedia content feature and the first interaction feature can be feature vectors; therefore, the feature similarity between the multimedia content feature and the first interaction feature can be a vector similarity, which can be a cosine similarity.

[0135] The processing device can correct the model parameters of the recommendation model through the first feature similarity to obtain the first recommendation model, including: the processing device can obtain the interaction behavior information of the sample object (such as the sample object in any round mentioned above) with respect to the multimedia resources in the multimedia resource set (such as the multimedia resources in any round mentioned above). A sample object can have an interaction behavior information for a multimedia resource. The interaction behavior information can be used to indicate whether the sample object has an interaction behavior or not with the multimedia resource.

[0136] Therefore, the processing device can obtain the reference feature similarity between a sample object (such as a sample object in any of the aforementioned rounds) and multimedia resources in the multimedia resource set (such as sample objects in any of the aforementioned rounds) based on the interaction behavior information. A sample object and a multimedia resource can have one reference feature similarity. If the interaction behavior information of a sample object towards a multimedia resource indicates that the sample object has an interactive behavior towards that multimedia resource, then the reference feature similarity between the sample object and the multimedia resource can be the maximum value of the feature similarity (e.g., it can be 1). Conversely, if the interaction behavior information of a sample object towards a multimedia resource indicates that the sample object does not have an interactive behavior towards that multimedia resource, then the reference feature similarity between the sample object and the multimedia resource can be the minimum value of the feature similarity (e.g., it can be 0).

[0137] The processing device can correct the model parameters of the recommendation model by using the difference between the first feature similarity and the reference feature similarity to obtain a first recommendation model. The goal of correcting the model parameters is to minimize (e.g., minimize) the difference between the first feature similarity and the reference feature similarity, as shown in the following formula:

[0138]

[0139] Where 'a' can represent any multimedia resource (such as any multimedia resource in any round mentioned above), 'b' can represent any sample object (such as any sample object in any round mentioned above), 'A' represents all multimedia resources in any round, and 'B' represents all sample objects in any round. y represents the feature similarity between the first interaction feature of sample object b and the multimedia content feature of multimedia resource a (belonging to the predicted score of the recommendation model). a,b This represents the reference feature similarity between sample object b and multimedia resource a, used to reflect whether sample object b actually has interactive behavior with multimedia resource a.

[0140] The above L BPR The loss function is generated by the similarity between the first feature and the reference feature. This loss function can be BPR-loss (an interest-based recommendation loss). This loss function represents the feature generation bias of the recommendation model for multimedia content features and the first interaction feature. This loss function can be used to reflect the difference between the similarity between the first feature and the reference feature. The larger the loss function, the greater the difference between the similarity between the first feature and the reference feature; conversely, the smaller the loss function, the smaller the difference between the first feature and the reference feature. Therefore, this application can use this loss function to correct the model parameters of the recommendation model to obtain the first recommendation model. Correcting the model parameters of the recommendation model can make the value of the loss function tend to be minimized (e.g., tend to 0).

[0141] The processing device can use the above principle to perform several rounds of iterative training on the recommendation model (i.e., iteratively correct the model parameters of the recommendation model) to finally obtain the first recommendation model. For example, the first recommendation model can be obtained after iteratively training the recommendation model for a specified number of rounds.

[0142] Among them, the multimedia model in the first recommendation model is the multimedia model after adaptive training of the multimedia model in the recommendation model, and the object model in the first recommendation model is the object model after adaptive training of the object model in the recommendation model.

[0143] Step S103: Obtain collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in the online recommendation scenario.

[0144] Optionally, the processing device can acquire collaborative features, which may include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects. In other words, the collaborative features may only include multimedia collaborative features of multimedia resources in the multimedia resource set, or may only include object collaborative features of sample objects, or may include both multimedia collaborative features and object collaborative features simultaneously.

[0145] This multimedia collaborative feature can be used to represent the content representation of multimedia resources in an online recommendation scenario within a multimedia resource set. Specifically, it can be an existing feature of the multimedia resources in the online recommendation scenario, or a feature (such as an embedding) learned and generated from the multimedia resources within that online recommendation scenario. Alternatively, it could be a feature learned and generated from the multimedia resources using an existing recall model (such as a fine-grained GNN model) within that online recommendation scenario. The specific training principle of this recall model needs to be determined based on the specific data recommendation strategy within that online recommendation scenario.

[0146] In this online recommendation scenario, multimedia resources can be represented by their IDs (identifiers). Therefore, the multimedia collaborative feature can be the ID collaborative feature of the multimedia resource. This ID collaborative feature can be obtained by embedding the ID of the multimedia resource into the recall model. The ID collaborative feature can contain rich collaborative information about the multimedia resource in the online recommendation scenario. This collaborative information can include any information that the multimedia resource exhibits in this online recommendation scenario, such as user interaction information, attribute information of the multimedia resource in this online recommendation scenario, and other information related to the multimedia resource in the online recommendation scenario, etc. The online recommendation scenario in which the multimedia resource resides can be called the first online recommendation scenario. For example, if the multimedia resource is an article, the first online recommendation scenario can be the article recommendation scenario in the subscription account where the article is located; or if the multimedia resource is a short video, the first online recommendation scenario can be the video recommendation scenario in the video client where the short video is located; and so on.

[0147] The aforementioned object collaborative features can be used to represent the object representation of a sample object in an online recommendation scenario. That is, these object collaborative features can be features already present in the online recommendation scenario, or features learned and generated from the sample object in that online recommendation scenario (such as embeddings). For example, they can be features learned and generated from the sample object using an existing recall model in that online recommendation scenario (such as a fine-grained GNN model). The specific training principle of this recall model needs to be determined based on the specific data recommendation strategy in that online recommendation scenario.

[0148] In this online recommendation scenario, sample objects can be represented by their IDs (identifiers). Therefore, the collaborative feature of the object can be the ID collaborative feature of the sample object. This ID collaborative feature can be obtained by feature embedding the ID of the sample object through the recall model. The ID collaborative feature can contain rich collaborative information about the sample object in the online recommendation scenario. This collaborative information can include any information exhibited by the sample object in this online recommendation scenario, such as preference information for interacting with the data recommended in the online recommendation scenario, attribute information of the sample object in the online recommendation scenario, and other information related to the sample object in the online recommendation scenario, etc. The online recommendation scenario in which the sample object is located can be called a second online recommendation scenario. There can be one or more second online recommendation scenarios, such as P scenarios, where P is a positive integer. For example, the second online recommendation scenario could be a scenario of recommending friends' updates to users in WeChat Moments, a scenario of recommending products to users on a shopping platform, or a scenario of recommending short videos to users on a short video platform. The second online recommendation scenario can be different from the first online recommendation scenario. An online recommendation scenario can refer to a scenario that has already been put into application and used to recommend data to objects online.

[0149] Step S104: The first recommendation model is optimized and trained using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

[0150] Optionally, the processing device can use the acquired collaborative features to optimize and train the first recommendation model to obtain the final trained recommendation model. Specifically, if the collaborative features only include multimedia collaborative features, it indicates that only multimedia collaborative features can be introduced without object collaborative features to optimize and train the first recommendation model; if the collaborative features only include object collaborative features, it indicates that only object collaborative features can be introduced without multimedia collaborative features to optimize and train the first recommendation model; if the collaborative features include both multimedia and object collaborative features, it indicates that both multimedia and object collaborative features can be introduced simultaneously to optimize and train the first recommendation model. All three methods of optimizing and training the first recommendation model fall within the scope of protection of this application.

[0151] The specific features included in the collaborative features can be determined based on the actual application scenario. For example, in scenarios where saving training costs is necessary (or where the processing device has low performance), if a deeper learning of the features of multimedia resources is required, the collaborative features may only include multimedia collaborative features. Similarly, in scenarios where saving training costs is necessary, if a deeper learning of the features of sample objects is required, the collaborative features may only include object collaborative features. However, if saving training costs is not required, and a deeper learning of both the features of multimedia resources and the features of sample objects is needed, then the collaborative features may include both multimedia collaborative features and object collaborative features.

[0152] Among them, the most preferred solution in this application is that the collaborative features include both multimedia collaborative features and object collaborative features. The following is a specific description of how to optimize and train the first recommendation model, taking the example of collaborative features that include both multimedia collaborative features and object collaborative features.

[0153] The processing device can first optimize and train the first recommendation model using the aforementioned multimedia collaborative features (also known as fine-tuning, here referring to the first fine-tuning). This optimized first recommendation model can be called the second recommendation model, which is obtained by optimizing and training the model using the multimedia collaborative features of multimedia resources. The optimization training of the first recommendation model can include simultaneously correcting the model parameters of both the multimedia model and the object model within the first recommendation model. That is, during the optimization training of the first recommendation model, the model parameters of both the multimedia model and the object model need to be corrected.

[0154] In this model, the multimedia model in the second recommendation model is an optimized version of the multimedia model in the first recommendation model, and the object model in the second recommendation model is an optimized version of the object model in the first recommendation model. For details on how to optimize the first recommendation model using multimedia collaborative features to obtain the second recommendation model, please refer to the following... Figure 5 The description in the corresponding embodiment.

[0155] Furthermore, the processing device can use the aforementioned object collaborative features to continue optimizing and training the second recommendation model (also known as fine-tuning, which can be a second fine-tuning here). The second recommendation model after optimization and training can be used as the final trained recommendation model. This final trained recommendation model is the adjusted recommendation model, which is obtained by introducing the object collaborative features of the sample objects for optimization and training.

[0156] The optimization training of the second recommendation model may include: keeping the model parameters of the multimedia model in the second recommendation model unchanged (e.g., freezing the model parameters of the multimedia model in the second recommendation model without modification), and modifying the model parameters of the object model in the second recommendation model. That is, during the optimization training of the second recommendation model, the model parameters of the multimedia model do not need to be modified, but only the model parameters of the object model need to be modified.

[0157] This is because the object model generates object features based on features generated by the multimedia model from multimedia resources. Therefore, if the multimedia model is also being optimized and trained while introducing collaborative features, the features generated by the multimedia model from multimedia resources will change. This makes the object model's generation of object features based on the features generated by the multimedia model from multimedia resources unstable, ultimately leading to poor optimization training results for the object model. Therefore, this application, when optimizing and training the second recommendation model, does not need to modify the model parameters of the multimedia model, but only modifies the model parameters of the object model, thereby ensuring the stability and excellence of the object model's optimization training results.

[0158] Therefore, it is understandable that, when collaborative features include both multimedia collaborative features and object collaborative features, the optimization order of this application—first introducing multimedia collaborative features to optimize the first recommendation model and then introducing object collaborative features to optimize the second recommendation model—has specific reasons and considerations. By first introducing multimedia collaborative features to optimize the first recommendation model and then fixing the model parameters of the multimedia model in the first recommendation model after optimization training, this application achieves the fixation of the features generated by the multimedia resources through the multimedia model. Thus, when introducing object collaborative features to optimize the second recommendation model, the fixed features of the multimedia resources can be used to stably generate the features of the object in the object model, thereby achieving stable and accurate training of the object model.

[0159] The multimedia model in the trained recommendation model is the multimedia model optimized and trained from the multimedia model in the second recommendation model, and the object model in the trained recommendation model is the object model optimized and trained from the object model in the second recommendation model. For details on how to optimize and train the second recommendation model using object collaborative features to obtain the trained recommendation model, please refer to the following... Figure 8 The description in the corresponding embodiment.

[0160] Optionally, if the collaborative features include multimedia collaborative features, this application can directly use the second recommendation model, which is optimized and trained using the multimedia collaborative features, as the final trained recommendation model. If the collaborative features include object collaborative features, this application can directly introduce object collaborative features to optimize and train the first recommendation model, and the first recommendation model optimized and trained in this case can be directly used as the final trained recommendation model. Furthermore, regardless of which features the collaborative features include or which training stage the recommendation model is in, the principle for training the model can be the principle shown in formula (1) above.

[0161] It should be noted that the introduction of multimedia collaborative features in this application affects the way the multimedia model generates features for multimedia resources, while the introduction of object collaborative features affects the way the object model generates features for objects. Therefore, it can be understood that if collaborative features include multimedia collaborative features, then when only multimedia collaborative features are introduced to optimize the training of the first recommendation model, the principle by which the multimedia model generates features for multimedia resources can be described as follows: Figure 5 The principle of generating multimedia resource features (such as the multimedia fusion features described below) in the corresponding embodiments is the same. However, since object collaboration features are not introduced, the principle of generating object features (such as the second interaction features described below) by the object model can be the same as the principle of generating object features by the object model during adaptive training.

[0162] Similarly, if collaborative features include object collaborative features, when only object collaborative features are introduced to optimize and train the first recommendation model, the principle by which the object model generates object features can be described as follows: Figure 8 The principle of generating object features (such as object fusion features below) in the corresponding embodiment is the same. However, since multimedia collaborative features are not introduced, the principle of generating multimedia resource features by the multimedia model can be the same as the principle of generating multimedia resource features (such as multimedia content features above) by the multimedia model during adaptive training.

[0163] After training the aforementioned recommendation model, this application can apply this trained recommendation model to recall multimedia resources recommended to an object. That is, the trained recommendation model in this application can be used to recommend multimedia resources that match the object's interests. The specific recommendation process can be found below. Figure 10 The relevant description in the corresponding embodiment. The processing device can push the multimedia resources recommended to the user by the trained recommendation model to the client where the user views the multimedia resources, so that the client can display the recommended multimedia resources to the user in the recommendation interface (which belongs to the client interface) for the user to view.

[0164] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the effect of a recommended interface for displaying multimedia resources provided in an embodiment of this application. For example... Figure 4 As shown, the multimedia resources here can be articles. The recommendation interface can display a subscription recommendation list, which can then sequentially display the various multimedia resources (i.e., articles) recalled for the user, including the article "Special Topic Interpretation | Latest Research Progress in Large Language Models", the article "Simultaneous Training of Multiple Graph Neural Networks", and the article "The Market Value of xxx Company Soars", thus realizing the recommendation of multimedia resources for the user.

[0165] Furthermore, it should be noted that the first recommendation model, the second recommendation model, and the trained recommendation model mentioned above are actually the same model at different training stages. Also, since the model parameters of the object model are adjusted during the optimization training of the second recommendation model, rather than the multimedia model's parameters, the multimedia model in the second recommendation model can be the same as the multimedia model in the trained recommendation model.

[0166] By employing the method described in this application, a progressive training of the recommendation model is achieved through multiple stages: adaptive training, optimization training by introducing multimedia collaborative features, and optimization training by introducing object collaborative features. This allows the recommendation model to be progressively and deeply introduced into the model, enabling it to learn multimedia collaborative features and object collaborative features more effectively and deeply. Ultimately, a high-performing recommendation model is obtained, which can accurately recall and recommend multimedia resources to users.

[0167] This application can acquire content data of multimedia resources in a multimedia resource set and interaction data of sample objects with respect to multimedia resources in the multimedia resource set; it can also use the content data and interaction data to adaptively train a recommendation model to obtain a first recommendation model; and it can acquire collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in an online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in an online recommendation scenario; and the first recommendation model can be optimized and trained using the collaborative features to obtain a trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to objects. As can be seen, the method proposed in this application can perform multi-stage progressive training on the recommendation model. In this multi-stage progressive training, the recommendation model can first undergo overall adaptive training, allowing it to initially learn information related to multimedia resources and objects. Then, collaborative features (such as multimedia collaborative features of multimedia resources and / or object collaborative features of sample objects) can be introduced to further optimize the training of the recommendation model, enabling it to learn the existing features of multimedia resources and / or sample objects in online recommendation scenarios. This can greatly improve the training effect of the recommendation model, resulting in a superior recommendation model. The trained recommendation model can then accurately recall multimedia resources recommended to objects.

[0168] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating a process for optimizing and training a first recommendation model, as provided in an embodiment of this application. Figure 5 As shown, the process may include:

[0169] Step S201: Call the multimedia model in the first recommendation model to learn features for each sub-content data in the content data and generate sub-content features for each sub-content data.

[0170] Optionally, since the content data of multimedia resources in the multimedia resource set can be a sequence composed of K sub-content data of the multimedia resource, the processing device can call the multimedia model in the first recommendation model to perform feature learning (such as feature embedding processing) on ​​each sub-content data in the content data (such as the content data of multimedia resources in any round mentioned above) to generate sub-content features for each sub-content data. A sub-content data can have one sub-content feature, and a sub-content feature can be a feature vector. By performing feature learning on K sub-content data in a content data, K sub-content features of the K sub-content data can be generated. These K sub-content features can be used to characterize the content semantics of the multimedia resource.

[0171] Step S202: The multimedia collaborative features and the K sub-content features of the K sub-content data are sequentially concatenated to obtain the first feature sequence.

[0172] Optionally, the processing device can sequentially concatenate the multimedia collaborative features and the K sub-content features of the K sub-content data to obtain a first feature sequence. The multimedia collaborative features and the K sub-content features concatenated in the first feature sequence all belong to the same multimedia resource. Therefore, one multimedia resource can correspond to one first feature sequence, and the processing principles for each multimedia resource are the same and independent.

[0173] A multimedia resource typically has only one multimedia collaborative feature, which is the feature of that multimedia resource in the online recommendation scenario it belongs to. Therefore, it can be understood that the multimedia collaborative feature and K sub-content features concatenated in the first feature sequence are the K+1 elements in the first feature sequence, and one of these K+1 elements is located at an element position in the first feature sequence. The first feature sequence contains a total of K+1 element positions, and the multimedia collaborative feature can be located at the first element position in the first feature sequence. The concatenation order of the K sub-content features in the first feature sequence is the sorting of the K sub-content data to which the K sub-content features belong in the corresponding content data.

[0174] For example, the K sub-content features may include sub-content features 1 to 6 in sequence. Then a first feature sequence can be [multimedia collaboration feature, sub-content feature 1, sub-content feature 2, sub-content feature 3, sub-content feature 4, sub-content feature 5, sub-content feature 6]. The multimedia collaboration features in this first feature sequence and the content data to which sub-content features 1 to 6 belong can all be multimedia collaboration features and content data of the same multimedia resource.

[0175] Step S203: Optimize and train the first recommendation model based on the first feature sequence to obtain the second recommendation model.

[0176] Optionally, the processing device can call the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources in the multimedia resource set through the first feature sequence. A multimedia resource can have a multimedia fusion feature, which is the feature obtained by fusing the multimedia collaborative features of the multimedia resource and the content features (such as K sub-content features) generated by the multimedia model for the multimedia resource.

[0177] The process of generating the multimedia fusion feature may include: the multimedia model in the first recommendation model may have a first attention mask, and the processing device may call the multimedia model in the first recommendation model to perform attention processing on the elements in the first feature sequence through the first attention mask to generate the fusion sub-feature corresponding to each element position in the first feature sequence. A fusion sub-feature may also be a feature vector.

[0178] Thus, the processing device can use the fusion sub-feature corresponding to the last element position (i.e. the last element position) in the first feature sequence as the multimedia fusion feature of the multimedia resources in the multimedia resource set. This is because the fusion sub-feature corresponding to the last element position is a feature obtained by fusing the sub-content features at each of the previous element positions. Therefore, the fusion sub-feature corresponding to the last element position can be used to characterize the features of the first feature sequence as a whole.

[0179] The aforementioned first attention mask is an innovative and specific attention mask added to the multimedia model during the optimization training stage of the first recommendation model in this application. This attention mask can be used to generate fused sub-features corresponding to the positions of each element in the first feature sequence through the multimedia collaborative features spliced ​​in the first feature sequence.

[0180] Specifically, using this first attention mask, the fusion sub-feature corresponding to the i-th element position can be generated by using the element at the i-th element position in the first feature sequence and the elements at the element positions preceding the i-th element position. Here, i is a positive integer, and i is less than or equal to K+1. The i-th element position can be any element position in the first feature sequence. This means that when generating the fusion sub-feature corresponding to an element position in the first feature sequence, this application can focus on the elements at that element position and the elements preceding it, without considering the elements at the element positions following that element position. Since the multimedia collaborative feature is located at the first element position in the first feature sequence, the fusion sub-features corresponding to each element position of each sub-content feature in the first feature sequence are generated by combining this multimedia collaborative feature. This achieves the effect of introducing the multimedia collaborative feature into the fusion sub-features corresponding to each element position.

[0181] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a first attention mask provided in an embodiment of this application. For example... Figure 6 As shown, the dimension of the first attention mask is (K+1)*(K+1), which is the square of K+1, where K can be equal to 4. The token sequence here can refer to the aforementioned content data, where a sub-content data has a sub-content feature. The element position of a sub-content feature in the first feature sequence can correspond to a row and a column in the first attention mask. Specifically, any element in the first attention mask can be used to obtain the correlation between any two element positions in the first feature sequence. Each row and column of the first attention mask corresponds sequentially to each element position in the first feature sequence. For example, the first row and first column of the first attention mask can both correspond to the first element position in the first feature sequence, the second row and second column of the first attention mask can both correspond to the second element position in the first feature sequence, and so on.

[0182] In this context, an element with a value of 1 in the first attention mask indicates that the correlation between the two corresponding element positions in the first feature sequence needs to be obtained, while an element with a value of 0 in the first attention mask indicates that the correlation between the two corresponding element positions in the first feature sequence does not need to be obtained (or does not need to be concerned with). This can be understood as the correlation between the two corresponding element positions in the first feature sequence being infinite or 0. Here, an element with a value of 0 in the first attention mask means that the correlation between the current element position and the element positions after it in the first feature sequence does not need to be obtained.

[0183] For example, the first row and first column of the first attention mask correspond to the position of the first element in the first feature sequence (i.e., the position of the element where the multimedia collaborative feature is located). The element 1 in the first row and first column of the first attention mask can be used to obtain the correlation between the multimedia collaborative feature and itself.

[0184] For example, the second row and first column of the first attention mask correspond to the second element position and the first element position in the first feature sequence. The element 1 in the second row and first column of the first attention mask is used to obtain the correlation between the second element position (i.e. the element position where the first sub-content feature is located) and the first element position (i.e. the element position where the multimedia collaboration feature is located) in the first feature sequence.

[0185] For example, the third row and second column of the first attention mask corresponds to the third element position and the second element position in the first feature sequence. The element 1 in the third row and second column of the first attention mask is used to obtain the correlation between the third element position (i.e., the element position where the second sub-content feature is located in the first feature sequence) and the second element position (i.e., the element position where the first sub-content feature is located in the first feature sequence).

[0186] like Figure 6 As shown, element 1 in the first row and element 1 in the first column of the first attention mask can be used to obtain the relevance related to multimedia collaborative features, and Figure 6 Element 1 in the gray area can be used to obtain the correlation between the features of each sub-content data in the Token sequence.

[0187] For example, the fusion sub-feature corresponding to the i-th element position in the first feature sequence can be generated by weighted summation of the elements at the i-th element position and the elements at the i-th element position and the elements at the i-th element position before the i-th element position, based on the correlation between the i-th element position and the i-th element position itself and the elements at the i-th element position before the i-th element position. The correlation can be used as the weight.

[0188] The processing device can also invoke the object model in the first recommendation model to perform feature learning on the interaction data of the sample object to generate a second interaction feature of the sample object. This process may include: the processing device can first invoke the multimedia model in the first recommendation model to generate multimedia fusion features of L multimedia resources in the interaction data. The principle of generating the L multimedia fusion features of the L multimedia resources in the interaction data is the same as the principle of invoking the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources in the multimedia resource set. A multimedia fusion feature can be a feature vector. Therefore, the processing device can invoke the object model in the first recommendation model to generate the second interaction feature of the sample object through the native attention mask of the object model and the L multimedia fusion features. The principle of invoking the object model in the first recommendation model to generate the second interaction feature of the sample object through the native attention mask of the object model and the L multimedia fusion features is the same as the principle of invoking the object model in the recommendation model to generate the first interaction feature of the sample object through the native attention mask of the object model and the L multimedia content features, and will not be elaborated here. A sample object can have one second interaction feature, and a second interaction feature can be a feature vector.

[0189] The processing device can acquire the feature similarity between the multimedia fusion features of multimedia resources (which may include all multimedia resources in any round) in a multimedia resource set and the second interaction features of sample objects (which may include all sample objects in any round). This feature similarity can be called the second feature similarity. A multimedia resource and a sample object can have a second feature similarity. The calculation principle of the second feature similarity can be the same as that of the first feature similarity, such as the cosine similarity between feature vectors. The processing device can use the second feature similarity to correct the model parameters of the first recommendation model to obtain the second recommendation model.

[0190] The principle of correcting the model parameters of the first recommendation model through the second feature similarity to obtain the second recommendation model is the same as the principle of correcting the model parameters of the recommendation model through the first feature similarity to obtain the first recommendation model. Both can be expressed by the principle in formula (1) above. Furthermore, the first recommendation model can be trained through several rounds of iterative training. The multimedia resources used in each round of iterative training (i.e., the corresponding content data and multimedia collaborative features) can be different or partially the same. Similarly, the sample objects used in each round of iterative training (i.e., the corresponding interaction data) can also be different or partially the same, depending on the actual application scenario.

[0191] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating a scenario for optimizing and training a first recommendation model, as provided in an embodiment of this application. Figure 7 As shown, the Item Modeling (LLM) in the first recommendation model can be a multimedia model, and the User Modeling (LLM) in the first recommendation model can be an object model. When optimizing and training the first recommendation model, this application can incorporate multimedia collaborative features into the Item Modeling (LLM) for training. For example, the Item Modeling (LLM) can be called to generate multimedia fusion features of multimedia resources in one round using multimedia collaborative features. The multimedia fusion features generated by the Item Modeling (LLM) from the multimedia resources in the interaction data of the sample objects using multimedia collaborative features can be input into the User Modeling (LLM). Therefore, in the User Modeling (LLM), the second interaction features of the sample objects can be generated using the multimedia fusion features of the multimedia resources in the interaction data of the sample objects.

[0192] Therefore, the processing device can correct the model parameters of the first recommendation model by the difference between the multimedia fusion features of the multimedia resources in this round and the second interaction features of the sample objects (which can be reflected by the similarity of the second features mentioned above).

[0193] This application proposes an effective masking strategy (such as a feature fusion strategy based on the aforementioned first attention mask). This masking strategy can effectively incorporate multimedia collaborative features of multimedia resources to train the multimedia model. Since multimedia collaborative features contain rich collaborative information of multimedia resources (such as various information displayed by multimedia resources in online recommendation scenarios, such as various operation information triggered by users in online recommendation scenarios, etc.), the multimedia model trained in this way can have a very strong representation ability of multimedia resources.

[0194] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating a process for optimizing and training a second recommendation model, as provided in an embodiment of this application. Figure 8 As shown, the process may include:

[0195] Step S301: Obtain the multimedia fusion features of each of the L multimedia resources. The L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model.

[0196] Optionally, since the interaction data of the sample object is a sequence of L multimedia resources with interactive behavior, the processing device can obtain the multimedia fusion features of each of the L multimedia resources. The L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model. The generation principle of the L multimedia fusion features is the same as the principle of generating the multimedia fusion features of the multimedia resources in the multimedia resource set in step S203 above.

[0197] Step S302: The object collaborative features are concatenated with the L multimedia fusion features in the form of a prefix of the L multimedia fusion features to obtain the second feature sequence.

[0198] Optionally, this application employs a Prefix-tuning efficient fine-tuning paradigm to fuse the object collaborative features of sample objects. This application can concatenate the object collaborative features of the sample object with the aforementioned L multimedia fusion features in a prefix form (Prefix form) to obtain a second concatenated sequence. One sample object can correspond to one second concatenated sequence. The object collaborative features concatenated in the second concatenated sequence, as well as the interaction data to which the L multimedia fusion features concatenated in the second concatenated sequence belong, can all belong to the same sample object.

[0199] In this context, a sample object can be in at least one online recommendation scenario, and within that scenario, it can possess one object collaborative feature. Therefore, a sample object can have at least one object collaborative feature, such as P object collaborative features, where P is a positive integer (i.e., an integer greater than or equal to 1). For example, a multimedia resource can be an article, and this application can be applied to an article recommendation scenario. The P online recommendation scenarios in which the sample object is located can be other recommendation scenarios besides the article recommendation scenario, such as video recommendation scenarios and friend activity recommendation scenarios, etc.

[0200] Therefore, the P object collaboration features and L multimedia fusion features can be P+L elements in the second feature sequence, that is, the sequence length of the second feature sequence can be P+L. One of the P+L elements can be located at one element position in the second feature sequence. The second feature sequence can have a total of P+L element positions. The P object collaboration features can be located at the first P element positions in the second feature sequence. The order of the P object collaboration features in the second feature sequence can be arbitrary. The order of the L multimedia fusion features in the second feature sequence can be the order of the L multimedia resources to which the L multimedia fusion features belong in the corresponding interactive data.

[0201] For example, P object collaboration features may include object collaboration feature 1 and object collaboration feature 2, and L multimedia fusion features may include multimedia fusion feature 1 to multimedia fusion feature 6. Then, a second feature sequence obtained by concatenating the P object collaboration features and the L multimedia fusion features can be [object collaboration feature 1, object collaboration feature 2, multimedia fusion feature 1, multimedia fusion feature 2, multimedia fusion feature 3, multimedia fusion feature 4, multimedia fusion feature 5, multimedia fusion feature 6]. The interactive data of the multimedia resources to which object collaboration feature 1 and object collaboration feature 2, as well as multimedia fusion features 1 to multimedia fusion feature 6 belong, all belong to the same sample object.

[0202] Step S303: Add L prefix model parameters corresponding to multimedia fusion features to the object model in the second recommendation model, and correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model.

[0203] Optionally, since the object collaboration features of the aforementioned sample object are concatenated with the L multimedia fusion features of the L multimedia resources in the interaction data of the sample object in a prefix form, the processing device can add prefix model parameters corresponding to the L multimedia fusion features to the object model in the second recommendation model. The number of prefix model parameters can be the same as the number of multimedia fusion features concatenated in a prefix form, such as both being L. Each concatenated multimedia fusion feature can correspond to one prefix model parameter. This prefix model parameter can be used to fuse the multimedia fusion feature with other features (such as the concatenated object collaboration features) and can also be used to record the information learned from the multimedia fusion feature during the training process of the object model. This information is reflected through the trained prefix model parameters.

[0204] Therefore, in the process of optimizing and training the second recommendation model in this application, the original model parameters of the second recommendation model can be frozen, and only the newly added prefix model parameters in the second recommendation model can be corrected. Since the proportion of the prefix model parameters in the model parameters of the second recommendation model is less than 0.5%, such as about 0.35%, the second recommendation model can be optimized and trained very efficiently during this optimization training stage.

[0205] Specifically, the processing device can use the aforementioned second feature sequence to correct the prefix model parameters newly added in the second recommendation model to obtain the trained recommendation model. This process may include:

[0206] The processing device can invoke the multimedia model in the second recommendation model to generate multimedia fusion features of multimedia resources (such as multimedia resources in any round) in the multimedia resource set by using the content data and multimedia collaboration features of the multimedia resources in the multimedia resource set. A multimedia resource can have one multimedia fusion feature. The principle of invoking the multimedia model in the second recommendation model to generate multimedia fusion features of multimedia resources is the same as the principle of invoking the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources through the first feature sequence, and will not be repeated here.

[0207] The processing device can also call the newly added second recommendation model with prefixed model parameters to generate object fusion features of sample objects (which can be sample objects from any round) through the above-mentioned second feature sequence. A sample object can have one object fusion feature. This process can be described as follows.

[0208] The object model in the second recommendation model can have a second attention mask. The processing device can call the object model in the second recommendation model to perform attention processing on the elements in the second feature sequence through the second attention mask to generate the fusion sub-feature corresponding to each element position in the second feature sequence. Here, a fusion sub-feature can also be a feature vector.

[0209] Thus, the processing device can use the fusion sub-feature corresponding to the last element position (i.e. the last element position) in the second feature sequence as the object fusion feature of the sample object. Since the fusion sub-feature corresponding to the last element position is the feature obtained by fusing the elements at the previous element positions, the fusion sub-feature corresponding to the last element position can be used to characterize the fusion feature of the second feature sequence as a whole.

[0210] The aforementioned second attention mask is an innovative and specific attention mask added to the object model during the optimization training stage of the second recommendation model in this application. This attention mask can be used to generate fused sub-features corresponding to the positions of each element in the second feature sequence through the object collaborative features spliced ​​in the second feature sequence.

[0211] Specifically, using this second attention mask, the fusion sub-feature corresponding to the j-th element position can be generated by using the element at the j-th element position in the second feature sequence and the elements at the element positions preceding the j-th element position. Here, j is a positive integer, and j is less than or equal to P+L. The j-th element position can be any element position in the second feature sequence. This means that when generating the fusion sub-feature corresponding to an element position in the second feature sequence, this application can focus on the elements at that element position and the elements preceding it, without considering the elements at the element positions following that element position. Since the object collaborative feature is located at the first P element positions in the second feature sequence, the fusion sub-features corresponding to each element position of the L multimedia fusion features in the second feature sequence are all generated in conjunction with this object collaborative feature. This achieves the effect of introducing the object collaborative feature into the fusion sub-features corresponding to each element position.

[0212] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a second attention mask provided in an embodiment of this application. For example... Figure 9 As shown, the dimension of the first attention mask can be (P+L)*(P+L), that is, the square of P+L, where P can be equal to 2 and L can be equal to 4. The article sequence here can refer to the aforementioned interactive data, and the multimedia resource can be an article. One multimedia resource in the interactive data has a multimedia fusion feature. The element position of a multimedia fusion feature in the second feature sequence can correspond to a row and a column in the second attention mask. Specifically, any element in the second attention mask can be used to obtain the correlation between any two element positions in the second feature sequence. Each row and column of the second attention mask corresponds sequentially to each element position in the second feature sequence. For example, the first row and first column of the second attention mask can both correspond to the first element position in the second feature sequence, the second row and second column of the first attention mask can both correspond to the second element position in the second feature sequence, and so on.

[0213] In this context, elements with a value of 1 in the second attention mask indicate that the correlation between the two corresponding element positions in the second feature sequence needs to be obtained. Elements with a value of 0 in the first attention mask indicate that the correlation between the two corresponding element positions in the second feature sequence does not need to be obtained (or does not need to be concerned with). This can be understood as the correlation between the two corresponding element positions in the second feature sequence being infinite or 0. Here, elements with a value of 0 in the second attention mask indicate that the correlation between the current element position and the element positions after it in the second feature sequence does not need to be obtained.

[0214] Specifically, it's important to note that since there are two object collaborative features here, in the first and second rows (i.e., the first P rows) of the second attention mask, all elements except the first two (which are 1) are 0. This is because the elements in the first and second rows of the second attention mask are used to fuse P object collaborative features, thus requiring the calculation of the correlation between these P object collaborative features. In other words, the elements in the first P rows and first P columns of the second attention mask can be 1, while the elements in the first P rows (excluding the first P columns) and the elements in the first P columns (excluding the first P rows) can be 0. This design is for fusing the P object fusion features.

[0215] For example, the first row and first column of the second attention mask correspond to the position of the first element in the second feature sequence (i.e., the position of the element where the first object collaborative feature is located). The element 1 in the first row and first column of the second attention mask can be used to obtain the correlation between the first object collaborative feature and itself.

[0216] For example, the second row and first column of the second attention mask corresponds to the second element position and the first element position in the second feature sequence. The element 1 in the second row and first column of the second attention mask is used to obtain the correlation between the second element position (i.e., the element position where the second object co-feature is located) and the first element position (i.e., the element position where the first object co-feature is located) in the second feature sequence.

[0217] For example, the fourth row and third column of the second attention mask corresponds to the fourth and third element positions in the second feature sequence. The element 1 in the fourth row and third column of the second attention mask is used to obtain the correlation between the fourth element position (i.e., the element position where the second multimedia fusion feature is located) and the third element position (i.e., the element position where the first multimedia fusion feature is located) in the second feature sequence.

[0218] like Figure 9 As shown, elements 1 in columns 1 and 2 of the second attention mask can be used to obtain the relevance related to the object's collaborative features, while Figure 9 Element 1 in the gray area can be used to obtain the correlation between the various multimedia fusion features of each multimedia resource in the article sequence.

[0219] For example, the fusion sub-feature corresponding to the j-th element position in the first feature sequence can be generated by weighted summation of the elements at the j-th element position and the elements at the j-th element position and the elements at the element positions before the j-th element position based on the correlation between the j-th element position and the j-th element position itself and the element positions before the j-th element position. The correlation can be used as the weight.

[0220] The processing device can obtain the third feature similarity between the multimedia fusion features of multimedia resources (which can be multimedia resources from any round) in the multimedia resource set and the object fusion features of sample objects (which can be sample objects from any round). A multimedia resource and a sample object can have one third feature similarity. The calculation principle of this third feature similarity can be the same as that of the first feature similarity, such as the cosine similarity between feature vectors. The processing device can use this third feature similarity to correct the newly added prefix model parameters in the second recommendation model to obtain the trained recommendation model.

[0221] The principle of correcting the prefix model parameters through the third feature similarity to obtain the trained recommendation model is the same as the principle of correcting the recommendation model parameters through the first feature similarity to obtain the first recommendation model. Both can be expressed by the principle in formula (1) above. Furthermore, the second recommendation model can also be trained through several rounds of iterative training. The multimedia resources used in each round of iterative training (i.e., the corresponding content data and multimedia collaborative features) can be different or partially the same. Similarly, the sample objects used in each round of iterative training (i.e., the corresponding interaction data) can also be different or partially the same, depending on the specific application scenario.

[0222] It should be noted that the multimedia resources in the interaction data of the sample objects are used to generate features of the sample objects (such as object fusion features, first interaction features, and second interaction features). The multimedia resources in the interaction data do not calculate feature similarity (such as first feature similarity, second feature similarity, and third feature similarity) with the sample objects to correct model parameters. However, in each training round of each training stage, the multimedia resources that calculate feature similarity with the sample objects include all multimedia resources in that training round.

[0223] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating a scenario for optimizing and training a second recommendation model, as provided in an embodiment of this application. Figure 10As shown, the Item Modeling (LLM) in the second recommendation model can be a multimedia model, and the User Modeling (LLM) in the second recommendation model can be an object model. When optimizing and training the second recommendation model, this application can incorporate multimedia collaborative features into the Item Modeling (LLM) and integrate object collaborative features into the User Modeling (LLM) for training. For example, the Item Modeling (LLM) can be called to generate multimedia fusion features of multimedia resources for one round using multimedia collaborative features. The multimedia fusion features generated by the Item Modeling (LLM) from the multimedia resources in the interaction data of sample objects using multimedia collaborative features can be input into the User Modeling (LLM). Therefore, the User Modeling (LLM) can generate object fusion features of sample objects using the multimedia fusion features of multimedia resources in the interaction data of sample objects and the object collaborative features.

[0224] Therefore, the processing device can correct the model parameters of the second recommendation model by the difference between the multimedia fusion features of the multimedia resources in this round and the object fusion features of the sample objects (which can be reflected by the third feature similarity mentioned above), such as correcting the prefix model parameters added in the form of Prefix in the second recommendation model.

[0225] The recommendation model trained as described above is the final recommendation model obtained through progressive training. The above process in this application proposes an effective masking strategy (such as a feature fusion strategy based on the second attention mask mentioned above). This masking strategy can effectively incorporate the object collaborative features of the sample objects to train the object model. Since the object collaborative features contain rich collaborative information of the sample objects (such as various information of the sample objects in online recommendation scenarios, such as the sample objects' preference information for the data recommended in online recommendation scenarios, etc.), the object model can have a very strong representation ability for objects.

[0226] In summary, in this application, the three training stages—adaptive training of the recommendation model, optimization training of the first recommendation model, and optimization training of the second recommendation model—can all be trained using the aforementioned multimedia resource set and interactive data of several sample objects targeting the multimedia resources in that set. Furthermore, for each training stage, the model can undergo multiple rounds of iterative training. Each round of iterative training can utilize a subset of the multimedia resources in the set and a subset of the sample objects. Within a single training stage, the multimedia resources and sample objects used in different rounds of model training can be different.

[0227] Furthermore, in any training phase and any round of model training, the model parameters can be corrected by the feature similarity between the features generated by each multimedia resource in that round and the features generated by the sample objects in that round. The specific principle of the correction can be found in the principle shown in formula (1) above. For example, in the training phase of adaptive training of the recommendation model, the model parameters of the recommendation model can be corrected by the first feature similarity between the multimedia content features generated by the multimedia resources in the multimedia resource set and the first interaction features generated by the sample objects; similarly, in the training phase of optimization training of the first recommendation model, the model parameters of the first recommendation model can be corrected by the second feature similarity between the multimedia fusion features generated by the multimedia resources in the multimedia resource set and the second interaction features generated by the sample objects; similarly, in the training phase of optimization training of the second recommendation model, the model parameters of the second recommendation model can be corrected by the third feature similarity between the multimedia fusion features generated by the multimedia resources in the multimedia resource set and the object fusion features generated by the sample objects.

[0228] Please see Figure 11 , Figure 11 This is a schematic diagram illustrating a process for recommending multimedia resources to an object, provided in an embodiment of this application. In this embodiment, a trained recommendation model obtained when the aforementioned collaborative features simultaneously include multimedia collaborative features and object collaborative features is used as an example to recall multimedia resources recommended to the object. Figure 11 As shown, the process may include:

[0229] Step S401: Obtain Q candidate multimedia resources and the target object for resource recommendation, where Q is a positive integer.

[0230] Optionally, the processing device can acquire Q candidate multimedia resources and a target object for resource recommendation, where Q is a positive integer. The Q candidate multimedia resources may include all multimedia resources to be recommended to the target object (i.e., the user). The target object may be a user who needs multimedia resource recommendations, and the number of target objects can be determined according to the actual application scenario. Since the principle of recommending multimedia resources to each target object is the same, the following description uses the process of recommending multimedia resources to a single target object as an example. Optionally, the Q candidate multimedia resources may overlap (i.e., be the same) with the multimedia resources in the aforementioned multimedia resource set, or they may not overlap, depending on the actual application scenario.

[0231] Step S402: Invoke the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource, and invoke the object model in the trained recommendation model to generate object fusion features for the target object.

[0232] Optionally, the processing device can invoke the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource. The processing device can obtain the content data and multimedia collaborative features of the candidate multimedia resources. Therefore, the processing device can invoke the multimedia model in the trained recommendation model to generate multimedia fusion features for the candidate multimedia resources using the content data and multimedia collaborative features. A candidate multimedia resource can have one multimedia fusion feature, which can be a feature vector. The method for obtaining the content data and multimedia collaborative features of the candidate multimedia resources can be the same as the method for obtaining the content data and multimedia collaborative features of the multimedia resources in the multimedia resource set described above.

[0233] For example, the processing device can acquire the content data (referred to as candidate content data) of any candidate multimedia resource, and the multimedia collaborative features (referred to as candidate multimedia collaborative features) of that candidate multimedia resource. These candidate multimedia collaborative features can be used to represent the content representation of that candidate multimedia resource in an online recommendation scenario. Therefore, the processing device can invoke the multimedia model in the trained recommendation model to generate the multimedia fusion features of that candidate multimedia resource using the candidate content data and the candidate multimedia collaborative features.

[0234] The principle of generating multimedia fusion features of candidate multimedia resources by calling the multimedia model in the trained recommendation model through the content data and multimedia collaborative features of candidate multimedia resources is the same as the principle of generating multimedia fusion features of multimedia resources in the multimedia resource set by calling the multimedia model in the first recommendation model in step S203 above through the content data and multimedia collaborative features of multimedia resources in the multimedia resource set (such as the first feature sequence obtained through the content data and multimedia collaborative features). It will not be elaborated here.

[0235] Furthermore, the processing device can also invoke the object model in the trained recommendation model to generate object fusion features of the target object. Similarly, the processing device can obtain the interaction data of the target object with respect to Q candidate multimedia resources, as well as the object collaboration features of the target object. Therefore, the processing device can invoke the object model in the trained recommendation model to generate object fusion features of the target object through the interaction data and object collaboration features of the target object. The method for obtaining the interaction data and object collaboration features of the target object can be the same as the method for obtaining the interaction data and object collaboration features of the sample objects mentioned above.

[0236] For example, the processing device can acquire the interaction data of the target object with respect to Q candidate multimedia resources (which can be called target interaction data) and the object collaboration features of the target object (which can be called target object collaboration features). The target interaction data is a sequence of L target multimedia resources with interactive behavior of the target object. The L target multimedia resources can be multimedia resources among the aforementioned Q candidate multimedia resources. The target object collaboration features can be used to represent the object representation of the target object in online recommendation scenarios.

[0237] Therefore, the processing device can first call the multimedia model in the trained recommendation model to generate multimedia fusion features for each target multimedia resource. The principle behind generating the multimedia fusion features of the target multimedia resources is the same as the principle behind calling the multimedia model in the first recommendation model to generate multimedia fusion features for the multimedia resources in the multimedia resource set in step S203 above.

[0238] Therefore, the multimedia fusion features of the L target multimedia resources can be input into the object model in the trained recommendation model to call the object model to generate the object fusion features of the target object through the L multimedia fusion features of the L target multimedia resources and the target object collaborative features. The object fusion features can be a feature vector.

[0239] The principle of generating object fusion features of target objects by calling the object model in the trained recommendation model through the interaction data of target objects (such as L multimedia fusion features generated from L target multimedia resources in the interaction data of target objects) and object collaborative features is the same as the principle of generating object fusion features of sample objects by calling the second recommendation model with newly added prefix model parameters in step S303 above through the interaction data of sample objects (such as L multimedia fusion features generated from L multimedia resources in the interaction data) and object collaborative features. It will not be elaborated here.

[0240] Step S403: Obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource.

[0241] Optionally, the processing device can obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource. The object fusion features of the target object and the multimedia fusion features of a candidate multimedia resource can have a feature similarity, which can be a cosine similarity.

[0242] The greater the feature similarity between the object fusion feature of the target object and the multimedia fusion feature of a candidate multimedia resource, the greater the likelihood that the target object is interested in the candidate multimedia resource, such as the greater the probability that the target object will perform interactive behavior on the candidate multimedia resource. Conversely, the smaller the feature similarity between the object fusion feature of the target object and the multimedia fusion feature of a candidate multimedia resource, the less likely that the target object is interested in the candidate multimedia resource, such as the less likely that the target object will perform interactive behavior on the candidate multimedia resource.

[0243] Step S404: Sort the Q candidate multimedia resources according to the descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource, to obtain the sorted Q candidate multimedia resources.

[0244] Optionally, the processing device can sort the Q candidate multimedia resources according to the descending order of feature similarity between the object fusion feature of the target object and the multimedia fusion feature of each candidate multimedia resource, to obtain a sorted set of Q candidate multimedia resources. Among these sorted Q candidate multimedia resources, those with higher feature similarity between their multimedia fusion feature and the object fusion feature of the target object are ranked earlier, while those with lower feature similarity are ranked later.

[0245] Step S405: Recommend the top R candidate multimedia resources from the sorted Q candidate multimedia resources to the target object; R is a positive integer and R is less than Q.

[0246] Optionally, the processing device can select the top R candidate multimedia resources from the Q ranked candidate multimedia resources as the multimedia resources recalled for the target object, and recommend these R candidate multimedia resources to the target object (e.g., push these R candidate multimedia resources to the target object's client). R is a positive integer, and R is less than Q. That is, the topR (R with the highest feature similarity) method can be used to recommend multimedia resources to the target object.

[0247] Alternatively, a similarity threshold can be set. Candidate multimedia resources among the Q candidate multimedia resources whose feature similarity between the multimedia fusion features and the object fusion features of the target object is greater than or equal to the similarity threshold can be used as multimedia resources recalled for the target object, and the recalled multimedia resources can be recommended to the target object.

[0248] Furthermore, if the aforementioned trained recommendation model was trained by incorporating multimedia collaborative features but not object collaborative features, then the principle by which the multimedia model in the trained recommendation model generates candidate multimedia resource features is similar to the above. Figure 5 The principle of generating features of multimedia resources (such as multimedia fusion features) in the corresponding embodiment is the same, and the principle of generating features of target objects in the object model of the trained recommendation model is the same as the principle of generating features of sample objects (such as the first interaction features mentioned above) during the adaptive training of the recommendation model.

[0249] If the aforementioned trained recommendation model was trained by incorporating object collaborative features but not multimedia collaborative features, then the principle by which the object model in the trained recommendation model generates the features of the target object is the same as described above. Figure 8 The principle of generating features of sample objects (such as object fusion features) in the corresponding embodiment is the same. The principle of generating features of multimedia resources in the multimedia model of the trained recommendation model is the same as the principle of generating features of multimedia resources (such as the multimedia content features mentioned above) in the process of adaptive training of the recommendation model.

[0250] This application can recall multimedia resources recommended to the target object by using the feature similarity between the features generated by each candidate multimedia resource and the target object after training the recommendation model. For example, it can recall candidate multimedia resources with feature similarity in the top R and recommend them to the target object.

[0251] Please see Figure 12 , Figure 12 This is a schematic diagram illustrating a scenario for recommending multimedia resources to a user, as provided in an embodiment of this application. Figure 12 As shown, this application can have two application stages: an offline application stage and an online application stage.

[0252] During this offline phase, the processing device can invoke the trained recommendation model to generate user representations (such as object fusion features) for each candidate user (e.g., target object), and can also invoke the trained recommendation model to generate article representations (such as multimedia fusion features) for each candidate multimedia resource (which could be a candidate article). The processing device can store the user representations of each candidate user and the article representations of each candidate multimedia resource in memory (e.g., an in-memory database). The candidate users and candidate multimedia resources can be continuously updated.

[0253] In the subsequent online phase, when a requesting user (which can be any candidate user) needs to view recommended multimedia resources, the processing device can retrieve the user representation of the requesting user and the article representations of each candidate article from memory. Thus, the processing device can calculate the feature similarity between the user representation of the requesting user and the article representations of each candidate article, and can select the top R (i.e., the R articles with the highest feature similarity) candidate articles to recommend to the requesting user.

[0254] Through the above process, namely the excellent trained recommendation model obtained in this application, accurate recommendation of multimedia resources for the target is achieved.

[0255] To verify the accuracy and effectiveness of the recommendation model trained using this application for multimedia resource recall, the following tests were conducted, which demonstrated the effectiveness of the model training method and the accuracy of multimedia resource recall.

[0256] Test 1:

[0257] Application solutions AUC ACC Base 0.90346248 0.91045101 Base+ICF (part of this application) 0.93465374 0.93429425 Base+UCF (part of this application) 0.93865503 0.93900055 Base+ICF+UCF (This application) 0.94965339 0.94450351

[0258] Table 1

[0259] Wherein, Base represents the first recommendation model obtained after pre-training the aforementioned recommendation model in this application; ICF represents model training incorporating multimedia collaborative features (corresponding to training with multimedia collaborative features included, or training with multimedia collaborative features but without object collaborative features); UCF represents model training incorporating object collaborative features (corresponding to the case where collaborative features include object collaborative features, or training with object collaborative features but without multimedia collaborative features); and ICF+UCF represents model training incorporating both multimedia collaborative features and object collaborative features (corresponding to the case where collaborative features simultaneously include multimedia collaborative features and object collaborative features). AUC is a classification evaluation metric, with a higher value indicating better recommendation performance; ACC is an evaluation metric for recommendation accuracy, with a higher value indicating more accurate recommendations.

[0260] Therefore, as shown in Table 1 above, when using the method provided in this application, the AUC and ACC values ​​for model training with the introduction of multimedia collaborative features are greater than those for adaptive training of the recommendation model alone; the AUC and ACC values ​​for model training with the introduction of object collaborative features are also greater than those for the introduction of multimedia collaborative features; and the AUC and ACC values ​​for model training with the simultaneous introduction of both multimedia and object collaborative features are greater than those for the introduction of only object collaborative features. This indicates that the simultaneous introduction of multimedia and object collaborative features for model training in this application yields the best results and accuracy in multimedia resource recommendation.

[0261] Test 2:

[0262]

[0263]

[0264] Table 2

[0265] The multimedia resources can be articles. This application conducted five recall tests on articles in the subscription account, each using seven days of data. Recall 50 represents the recall rate when recalling 50 articles, Recall 100 represents the recall rate when recalling 100 articles, and Recall 150 represents the recall rate when recalling 150 articles. As shown in Table 2 above, compared to the Base model (i.e., the first recommendation model after adaptive training of the recommendation model in this application), the recommendation model trained using the multimedia collaborative features and object collaborative features introduced in this application has the highest recall rates in Recall 50, Recall 100, and Recall 150. This clearly demonstrates the effectiveness of introducing multimedia collaborative features and object collaborative features for model training in this application.

[0266] In summary, the test results above definitively confirm the accuracy and effectiveness of the method provided in this application for recalling and recommending multimedia resources.

[0267] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a model processing device provided in an embodiment of this application. For example... Figure 13 As shown, the model processing device 1 may include: a first acquisition module 11, a training module 12, a second acquisition module 13, and an optimization module 14.

[0268] The first acquisition module 11 is used to acquire the content data of multimedia resources in the multimedia resource set and the interaction data of the sample object with respect to the multimedia resources in the multimedia resource set.

[0269] Training module 12 is used to adaptively train the recommendation model using content data and interaction data to obtain the first recommendation model;

[0270] The second acquisition module 13 is used to acquire collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in the online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in the online recommendation scenario.

[0271] The optimization module 14 is used to optimize and train the first recommendation model using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

[0272] Optionally, if the collaborative features include multimedia collaborative features and object collaborative features, the optimization module 14 uses the collaborative features to optimize and train the first recommendation model to obtain the trained recommendation model in the following ways:

[0273] The first recommendation model is optimized and trained using multimedia collaborative features to obtain the second recommendation model;

[0274] The second recommendation model is optimized and trained using object collaborative features to obtain the trained recommendation model.

[0275] Optionally, the recommendation model includes a multimedia model and an object model. The multimedia model is used to represent multimedia resources, and the object model is used to represent objects.

[0276] The adaptive training of the recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the recommendation model;

[0277] The optimization training of the first recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the first recommendation model;

[0278] The optimization training of the second recommendation model includes: keeping the model parameters of the multimedia model in the second recommendation model unchanged, and correcting the model parameters of the object model in the second recommendation model.

[0279] Optionally, the content data is a sequence of K sub-content data belonging to the multimedia resource, where K is a positive integer; the optimization module 14 uses multimedia collaborative features to optimize and train the first recommendation model to obtain the second recommendation model in the following ways:

[0280] The multimedia model in the first recommendation model is invoked to learn features for each sub-content data in the content data, generating sub-content features for each sub-content data.

[0281] The multimedia collaborative features and the K sub-content features of the K sub-content data are sequentially concatenated to obtain the first feature sequence;

[0282] The first recommendation model is optimized and trained based on the first feature sequence to obtain the second recommendation model;

[0283] Among them, there is one multimedia collaborative feature, and the multimedia collaborative feature and K sub-content features are K+1 elements in the first feature sequence. One element is located at the position of an element in the first feature sequence, and the multimedia collaborative feature is located at the position of the first element in the first feature sequence.

[0284] Optionally, the optimization module 14 optimizes and trains the first recommendation model based on the first feature sequence to obtain the second recommendation model in the following ways:

[0285] The multimedia model in the first recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence;

[0286] The object model in the first recommendation model is invoked to learn features from the interaction data, generating the second interaction features of the sample objects.

[0287] Obtain the second feature similarity between the multimedia fusion features and the second interaction features of the multimedia resources in the multimedia resource set;

[0288] The model parameters of the first recommendation model are corrected based on the similarity of the second feature to obtain the second recommendation model.

[0289] Optionally, the multimedia model in the first recommendation model has a first attention mask; the optimization module 14 calls the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence, including:

[0290] The multimedia model in the first recommendation model is invoked to perform attention processing on the elements in the first feature sequence based on the first attention mask, generating fused sub-features corresponding to the position of each element in the first feature sequence.

[0291] The fusion sub-feature corresponding to the position of the last element in the first feature sequence is used as the multimedia fusion feature of the multimedia resources in the multimedia resource set.

[0292] The first attention mask is used to generate a fusion sub-feature corresponding to the i-th element position based on the element at the i-th element position in the first feature sequence and the elements at the element positions before the i-th element position, where i is a positive integer and i is less than or equal to K+1.

[0293] Optionally, the interaction data is a sequence of L multimedia resources with interactive behaviors of the sample objects, where L is a positive integer; the optimization module 14 uses object collaborative features to optimize and train the second recommendation model to obtain the trained recommendation model, including:

[0294] Obtain the multimedia fusion features of each of the L multimedia resources. The L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model.

[0295] The object collaborative features are concatenated with the L multimedia fusion features in the form of a prefix of the L multimedia fusion features to obtain the second feature sequence;

[0296] In the object model of the second recommendation model, add L prefix model parameters corresponding to multimedia fusion features, and correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model;

[0297] Among them, there are P object collaboration features, where P is a positive integer. The P object collaboration features and L multimedia fusion features are the P+L elements in the second feature sequence. One element is located at the position of one element in the second feature sequence. The P object collaboration features are located at the positions of the first P elements in the second feature sequence.

[0298] Optionally, the optimization module 14 can correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model in the following ways:

[0299] The multimedia model in the second recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on content data and multimedia collaboration features;

[0300] The newly added second recommendation model with prefixed model parameters is invoked to generate object fusion features of the sample objects based on the second feature sequence.

[0301] Obtain the third feature similarity between the multimedia fusion features of multimedia resources in the multimedia resource set and the object fusion features of sample objects;

[0302] The training recommendation model is obtained by correcting the parameters of the newly added prefix model in the second recommendation model based on the third feature similarity.

[0303] Optionally, the object model in the second recommendation model has a second attention mask; the optimization module 14 calls the newly added second recommendation model with prefixed model parameters to generate object fusion features of the sample objects based on the second feature sequence, including:

[0304] The newly added second recommendation model with prefixed model parameters is invoked. Based on the second attention mask, attention processing is performed on the elements in the second feature sequence to generate the fused sub-feature corresponding to each element position in the second feature sequence.

[0305] The fusion sub-feature corresponding to the position of the last element in the second feature sequence is used as the object fusion feature of the sample object;

[0306] The second attention mask is used to generate a fusion sub-feature corresponding to the j-th element position based on the element at the j-th element position in the second feature sequence and the elements at the element positions before the j-th element position, where j is a positive integer and j is less than or equal to P+L.

[0307] Optionally, the interaction data is a sequence of L multimedia resources with interactive behaviors for the sample object, where L is a positive integer; the training module 12 uses content data and interaction data to adaptively train the recommendation model to obtain the first recommendation model in the following ways:

[0308] The recommendation model is invoked to learn features from the content data, generating multimedia content features of multimedia resources in the multimedia resource set.

[0309] The recommendation model is invoked to generate multimedia content features of L multimedia resources, and the first interaction features of the sample object are generated based on the L multimedia content features of the L multimedia resources.

[0310] Obtain the first feature similarity between the multimedia content features and the first interaction features of the multimedia resources in the multimedia resource set;

[0311] The model parameters of the recommendation model are corrected based on the similarity of the first feature to obtain the first recommendation model.

[0312] Optionally, the training module 12 may revise the model parameters of the recommendation model based on the first feature similarity to obtain the first recommendation model in the following ways:

[0313] The interactive behavior information of the sample object to the multimedia resources in the multimedia resource set is obtained. The interactive behavior information is used to indicate whether the sample object has interactive behavior or not interactive behavior to the multimedia resources in the multimedia resource set.

[0314] Based on the interaction behavior information, obtain the reference feature similarity between the sample object and the multimedia resources in the multimedia resource set;

[0315] By analyzing the difference between the first feature similarity and the reference feature similarity, the model parameters of the recommendation model are corrected to obtain the first recommendation model.

[0316] Optionally, the trained recommendation model includes a multimedia model and an object model, wherein the multimedia model is used to represent multimedia resources and the object model is used to represent objects; the above-mentioned model processing device further includes a recommendation module 15, which is used for:

[0317] Get Q candidate multimedia resources and the target object to be recommended, where Q is a positive integer;

[0318] The multimedia model in the trained recommendation model is called to generate the multimedia fusion features of each candidate multimedia resource, and the object model in the trained recommendation model is called to generate the object fusion features of the target object.

[0319] Obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource;

[0320] The Q candidate multimedia resources are sorted in descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource to obtain the sorted Q candidate multimedia resources.

[0321] The top R candidate multimedia resources from the sorted Q candidate multimedia resources are recommended to the target object; R is a positive integer and R is less than Q.

[0322] Optionally, the recommendation module 15 may generate multimedia fusion features for each candidate multimedia resource by calling the multimedia model in the trained recommendation model, including:

[0323] Obtain candidate content data and candidate multimedia collaborative features for any candidate multimedia resource; the candidate multimedia collaborative features are used to represent the content representation of any candidate multimedia resource in an online recommendation scenario.

[0324] The multimedia model in the trained recommendation model is invoked, and multimedia fusion features of any candidate multimedia resource are generated based on candidate content data and candidate multimedia collaborative features.

[0325] Optionally, the recommendation module 15 may use the object model in the trained recommendation model to generate object fusion features of the target object in the following ways:

[0326] Obtain the target interaction data and target object collaborative features of the target object; the target interaction data is a sequence of L target multimedia resources with interactive behaviors of the target object, where L is a positive integer; the target object collaborative features are used to represent the object representation of the target object in online recommendation scenarios.

[0327] The multimedia model in the trained recommendation model is invoked to generate multimedia fusion features for each target multimedia resource;

[0328] The object model in the trained recommendation model is invoked, and the object fusion feature of the target object is generated based on the L multimedia fusion features of L target multimedia resources and the target object collaborative features.

[0329] According to one embodiment of this application, Figure 3 The steps involved in the model processing method shown can be derived from... Figure 13 The model processing device 1 shown is used to execute the process. For example, Figure 3 Step S101 shown can be performed by Figure 13 The first acquisition module 11 in the process is executed. Figure 3 Step S102 shown can be performed by Figure 13 The training module 12 in the middle is used to execute; Figure 3 Step S103 shown can be performed by Figure 13 The second acquisition module 13 in the middle is used to execute, Figure 3 Step S104 shown can be performed by Figure 13 The optimization module 14 in the middle is used to execute it.

[0330] According to one embodiment of this application, Figure 13 The modules in the model processing device 1 shown can be individually or entirely combined into one or more units, or some of the units can be further divided into multiple functionally smaller sub-units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by multiple units, or the function of multiple modules can be implemented by one unit. In other embodiments of this application, the model processing device 1 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0331] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0332] According to one embodiment of this application, a computer program capable of executing the steps involved in the corresponding methods shown in the various embodiments of this application can be run on a general-purpose computer device (which may include processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM)) to construct a system as described in the embodiments of this application. Figure 13 The model processing apparatus 1 shown above. The computer program described above can be recorded on a computer-readable recording medium, and can be loaded into the computer device described above through the computer-readable recording medium and run therein.

[0333] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 14 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. In some embodiments, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 14 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0334] exist Figure 14In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0335] Acquire the content data of multimedia resources in the multimedia resource set, and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set;

[0336] The recommendation model is adaptively trained using content data and interaction data to obtain the first recommendation model;

[0337] Acquire collaborative features; collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in the online recommendation scenario;

[0338] The first recommendation model is optimized and trained using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

[0339] It should be understood that the computer device 1000 described in the embodiments of this application can execute the model processing methods described in the embodiments of this application, and can also execute the methods described above. Figure 13 The description of the model processing device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0340] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program. When a processor executes this computer program, it can perform the model processing methods described in the various embodiments of this application; therefore, they will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0341] As an example, the aforementioned computer program can be deployed and executed on a single computer device, or deployed and executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can form a blockchain network.

[0342] The aforementioned computer-readable storage medium can be an internal storage unit of the computer device, such as a hard drive or memory. It can also be an external storage device, such as a plug-in hard drive, smart media card (SMC), secure digital card (SD) card, or flash card. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. This computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. It can also be used to temporarily store data that has been output or will be output.

[0343] This application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the model processing methods described in the embodiments of this application; therefore, these will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0344] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0345] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0346] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A model processing method, characterized in that, The method includes: Acquire the content data of multimedia resources in the multimedia resource set, and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set; The recommendation model is adaptively trained using the content data and the interaction data to obtain a first recommendation model; Acquire collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of the sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in an online recommendation scenario, and the object collaborative features are used to represent the object representation of the sample objects in an online recommendation scenario; The first recommendation model is optimized and trained using the collaborative features to obtain a trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

2. The method according to claim 1, characterized in that, If the collaborative features include the multimedia collaborative features and the object collaborative features, then the step of optimizing and training the first recommendation model using the collaborative features to obtain the trained recommendation model includes: The first recommendation model is optimized and trained using the aforementioned multimedia collaborative features to obtain the second recommendation model; The second recommendation model is optimized and trained using the object collaborative features to obtain the trained recommendation model.

3. The method according to claim 2, characterized in that, The recommendation model includes a multimedia model and an object model. The multimedia model is used to represent multimedia resources, and the object model is used to represent objects. The adaptive training of the recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the recommendation model; The optimization training of the first recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the first recommendation model; The optimization training of the second recommendation model includes: keeping the model parameters of the multimedia model in the second recommendation model unchanged, and correcting the model parameters of the object model in the second recommendation model.

4. The method according to claim 3, characterized in that, The content data is a sequence consisting of K sub-content data of the respective multimedia resource, where K is a positive integer; the step of optimizing and training the first recommendation model using the multimedia collaborative features to obtain the second recommendation model includes: The multimedia model in the first recommendation model is invoked to perform feature learning on each sub-content data in the content data, and to generate sub-content features for each sub-content data. The multimedia collaborative features and the K sub-content features of the K sub-content data are sequentially concatenated to obtain the first feature sequence; The first recommendation model is optimized and trained based on the first feature sequence to obtain the second recommendation model; Wherein, the multimedia collaboration feature is one, and the multimedia collaboration feature and the K sub-content features are K+1 elements in the first feature sequence, with one element located at one element position in the first feature sequence, and the multimedia collaboration feature located at the first element position in the first feature sequence.

5. The method according to claim 4, characterized in that, The step of optimizing and training the first recommendation model based on the first feature sequence to obtain the second recommendation model includes: The multimedia model in the first recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence; The object model in the first recommendation model is invoked to perform feature learning on the interaction data, thereby generating the second interaction feature of the sample object; Obtain the second feature similarity between the multimedia fusion features of the multimedia resources in the multimedia resource set and the second interaction features; The model parameters of the first recommendation model are corrected based on the second feature similarity to obtain the second recommendation model.

6. The method according to claim 5, characterized in that, The multimedia model in the first recommendation model has a first attention mask; the step of calling the multimedia model in the first recommendation model to generate multimedia fusion features of the multimedia resources in the multimedia resource set based on the first feature sequence includes: The multimedia model in the first recommendation model is invoked to perform attention processing on the elements in the first feature sequence based on the first attention mask, generating fused sub-features corresponding to the position of each element in the first feature sequence. The fusion sub-feature corresponding to the position of the last element in the first feature sequence is used as the multimedia fusion feature of the multimedia resources in the multimedia resource set. The first attention mask is used to generate a fusion sub-feature corresponding to the i-th element position based on the element at the i-th element position in the first feature sequence and the elements at the element positions before the i-th element position, where i is a positive integer and i is less than or equal to K+1.

7. The method according to claim 3, characterized in that, The interactive data is a sequence of L multimedia resources with interactive behaviors of the sample object, where L is a positive integer; the step of optimizing and training the second recommendation model using the object collaborative features to obtain the trained recommendation model includes: Obtain the multimedia fusion features of each of the L multimedia resources, wherein the L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model; The object collaborative features are concatenated with the L multimedia fusion features in the form of a prefix to obtain the second feature sequence; In the object model of the second recommendation model, add prefix model parameters corresponding to the L multimedia fusion features, and correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model; There are P object collaboration features, where P is a positive integer. The P object collaboration features and the L multimedia fusion features are P+L elements in the second feature sequence. One element is located at one element position in the second feature sequence. The P object collaboration features are located at the first P element positions in the second feature sequence.

8. The method according to claim 7, characterized in that, The step of correcting the prefix model parameters based on the second feature sequence to obtain the trained recommendation model includes: The multimedia model in the second recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the content data and the multimedia collaborative features; The second recommendation model, which has been newly added with the prefix model parameters, is invoked to generate object fusion features of the sample object based on the second feature sequence; Obtain the third feature similarity between the multimedia fusion features of the multimedia resources in the multimedia resource set and the object fusion features of the sample objects; The newly added prefix model parameters in the second recommendation model are corrected based on the third feature similarity to obtain the trained recommendation model.

9. The method according to claim 8, characterized in that, The object model in the second recommendation model has a second attention mask; the invocation of the second recommendation model with the newly added prefix model parameters generates object fusion features of the sample object based on the second feature sequence, including: The second recommendation model with the added prefix model parameters is invoked to perform attention processing on the elements in the second feature sequence based on the second attention mask, generating fused sub-features corresponding to the position of each element in the second feature sequence. The fusion sub-feature corresponding to the position of the last element in the second feature sequence is used as the object fusion feature of the sample object; The second attention mask is used to generate a fusion sub-feature corresponding to the j-th element position based on the element at the j-th element position in the second feature sequence and the elements at the element positions before the j-th element position, where j is a positive integer and j is less than or equal to P+L.

10. The method according to claim 1, characterized in that, The interaction data is a sequence of L multimedia resources with interactive behaviors of the sample object, where L is a positive integer; the step of adaptively training the recommendation model using the content data and the interaction data to obtain a first recommendation model includes: The recommendation model is invoked to perform feature learning on the content data, generating multimedia content features of multimedia resources in the multimedia resource set; The recommendation model is invoked to generate multimedia content features of the L multimedia resources, and the first interaction features of the sample object are generated based on the L multimedia content features of the L multimedia resources. Obtain the first feature similarity between the multimedia content features of the multimedia resources in the multimedia resource set and the first interaction feature; The model parameters of the recommendation model are corrected based on the first feature similarity to obtain the first recommendation model.

11. The method according to claim 10, characterized in that, The step of correcting the model parameters of the recommendation model based on the first feature similarity to obtain the first recommendation model includes: The interaction behavior information of the sample object with respect to the multimedia resources in the multimedia resource set is obtained. The interaction behavior information is used to indicate whether the sample object has interactive behavior or not interactive behavior with the multimedia resources in the multimedia resource set. Based on the interaction behavior information, obtain the reference feature similarity between the sample object and the multimedia resources in the multimedia resource set; The model parameters of the recommendation model are corrected by the difference between the first feature similarity and the reference feature similarity to obtain the first recommendation model.

12. The method according to claim 1, characterized in that, The trained recommendation model includes a multimedia model and an object model. The multimedia model is used to represent multimedia resources, and the object model is used to represent objects. The method further includes: Get Q candidate multimedia resources and the target object to be recommended, where Q is a positive integer; The multimedia model in the trained recommendation model is invoked to generate multimedia fusion features for each candidate multimedia resource, and the object model in the trained recommendation model is invoked to generate object fusion features for the target object. Obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource; The Q candidate multimedia resources are sorted in descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource to obtain the sorted Q candidate multimedia resources. The top R candidate multimedia resources from the sorted Q candidate multimedia resources are recommended to the target object; R is a positive integer and R is less than Q.

13. The method according to claim 12, characterized in that, The step of calling the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource includes: Acquire candidate content data and candidate multimedia collaborative features of any candidate multimedia resource; the candidate multimedia collaborative features are used to represent the content representation of any candidate multimedia resource in an online recommendation scenario. The multimedia model in the trained recommendation model is invoked, and multimedia fusion features of any candidate multimedia resource are generated based on the candidate content data and the candidate multimedia collaborative features.

14. The method according to claim 12, characterized in that, The step of generating the object fusion feature of the target object by calling the object model in the trained recommendation model includes: The target interaction data and the target object collaborative features of the target object are obtained; the target interaction data is a sequence of L target multimedia resources with interactive behavior of the target object, where L is a positive integer; the target object collaborative features are used to represent the object representation of the target object in online recommendation scenarios. The multimedia model in the trained recommendation model is invoked to generate multimedia fusion features for each target multimedia resource; The object model in the trained recommendation model is invoked, and based on the L multimedia fusion features of the L target multimedia resources and the target object collaborative features, the object fusion features of the target object are generated.

15. A model processing device, characterized in that, The method includes: The first acquisition module is used to acquire the content data of multimedia resources in the multimedia resource set and the interaction data of the sample object with respect to the multimedia resources in the multimedia resource set. The training module is used to adaptively train the recommendation model using the content data and the interaction data to obtain a first recommendation model; The second acquisition module is used to acquire collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of the sample object, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in the online recommendation scenario, and the object collaborative features are used to represent the object representation of the sample object in the online recommendation scenario; An optimization module is used to optimize and train the first recommendation model using the collaborative features to obtain a trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

16. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-14.

17. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the steps of the method according to any one of claims 1-14.

Images