Method and apparatus for recommending media data, and computer device and storage medium

Abstract

The present application relates to a method and apparatus for recommending media data, and a computer device, a storage medium and a computer program product. The method can be applied to the field of artificial intelligence, for example, applied to a scenario of determining, by means of an intelligent terminal, target media data that a target object is interested in. The method comprises: extracting a media representation vector and a text representation vector from media data and corresponding description text (step 202); performing knowledge retrieval in a knowledge graph according to the media representation vector, so as to obtain an entity subgraph corresponding to the media data, and determining an entity representation vector corresponding to the entity subgraph (step 204); performing feature fusion processing on the media representation vector, the text representation vector and the entity representation vector, so as to obtain a knowledge-enhanced vector (step 206); and acquiring target media data on the basis of the knowledge-enhanced vector, and recommending the target media data to a target object (step 208).

Description

Method, device, computer equipment and storage medium for recommending media data

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 18, 2023, with application number 2023108802404, and invention name “Method, device, computer equipment and storage medium for recommending media data”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for recommending media data. Background Art

[0003] With the development of internet technology, media browsing has become increasingly popular among a wide range of users. In related technologies, recommendation systems can identify other media that a user might be interested in based on the media content they have previously viewed. However, these identified other media often have a high degree of similarity to the previously viewed content, which can easily lead to simplistic recommendations.

[0004] Summary of the Invention

[0005] According to various embodiments provided by the present application, a method, apparatus, computer device, computer-readable storage medium, and computer program product for recommending media data are provided.

[0006] In a first aspect, the present application provides a method for recommending media data, which is executed by a server and includes:

[0007] Extract media representation vectors and text representation vectors from media data and the description text of the media data; perform knowledge retrieval in the knowledge graph based on the media representation vectors to obtain an entity subgraph of the media data, and determine the entity representation vector of the entity subgraph; perform feature fusion processing on the media representation vector, the text representation vector and the entity representation vector to obtain a knowledge enhancement vector; and obtain target media data based on the knowledge enhancement vector, and recommend the target media data to the target object.

[0008] In a second aspect, the present application further provides a device for recommending media data. The device includes:

[0009] A vector extraction module, configured to extract a media representation vector and a text representation vector from the media data and the description text of the media data;

[0010] A first knowledge retrieval module is used to perform knowledge retrieval in the knowledge graph based on the media representation vector, obtain an entity subgraph of the media data, and determine the entity representation vector of the entity subgraph;

[0011] A first fusion module is used to perform feature fusion processing on the media representation vector, the text representation vector, and the entity representation vector to obtain a knowledge enhancement vector; and

[0012] The recommendation module is used to obtain target media data based on the knowledge enhancement vector and recommend the target media data to the target object.

[0013] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the media data recommendation method provided in the first aspect when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the media data recommendation method provided in the first aspect.

[0015] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, implements the media data recommendation method provided in the first aspect.

[0016] In a sixth aspect, the present application provides a method for processing a recommendation model, which is executed by a server and includes:

[0017] Based on the feature extraction model, a first media training vector and a first text training vector are extracted from the first sample media data and the corresponding first sample text; based on the knowledge retrieval model, knowledge retrieval processing is performed on the first media training vector and the knowledge graph to obtain a training subgraph of the first sample media data, and the entity training vector of the training subgraph is determined; based on the knowledge enhancement model, feature fusion processing is performed on the first media training vector, the first text training vector and the entity training vector to obtain a knowledge enhancement training vector; based on the knowledge enhancement training vector and the sample label, a visual loss value and a language loss value are determined; based on the knowledge enhancement training vector and the training subgraph, a knowledge retrieval loss value is determined; based on the visual loss value, the language loss value and the knowledge retrieval loss value, the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model are adjusted to obtain an enhanced vector extraction model; and a recommendation model is determined based on the enhanced vector extraction model and the classification model; the recommendation model is used to extract the knowledge enhancement vector based on the media data, the descriptive text and the knowledge graph, and determine the interest type based on the knowledge enhancement vector, so as to obtain the target media data based on the interest type and recommend the target media data to the target object.

[0018] In a seventh aspect, the present application further provides a device for processing a recommendation model. The device comprises:

[0019] A training vector extraction module, configured to extract a first media training vector and a first text training vector from the first sample media data and the corresponding first sample text based on a feature extraction model;

[0020] a second knowledge retrieval module, configured to perform knowledge retrieval processing on the first media training vector and the knowledge graph based on the knowledge retrieval model, obtain a training subgraph of the first sample media data, and determine an entity training vector of the training subgraph;

[0021] A second fusion module is configured to perform feature fusion processing on the first media training vector, the first text training vector, and the entity training vector based on the knowledge enhancement model to obtain a knowledge enhancement training vector;

[0022] A first loss value determination module is used to determine a visual loss value and a language loss value based on the knowledge-enhanced training vector and the sample label;

[0023] A second loss value determination module is used to determine a knowledge retrieval loss value based on the knowledge enhancement training vector and the training subgraph;

[0024] a parameter adjustment module, configured to adjust parameters of the feature extraction model, the knowledge retrieval model, and the knowledge enhancement model based on the visual loss value, the language loss value, and the knowledge retrieval loss value to obtain an enhanced vector extraction model; and

[0025] The recommendation model determination module is used to determine the recommendation model based on the enhancement vector extraction model and the classification model; the recommendation model is used to extract the knowledge enhancement vector based on the media data, descriptive text and knowledge graph, and determine the interest type based on the knowledge enhancement vector, so as to obtain the target media data based on the interest type and recommend the target media data to the target object.

[0026] In an eighth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for processing the recommendation model provided in the sixth aspect is implemented.

[0027] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for processing the recommendation model provided in the sixth aspect.

[0028] In a tenth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for processing the recommendation model provided in the sixth aspect.

[0029] The details of one or more embodiments of the present application are set forth in the following drawings and description. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are merely exemplary embodiments of this application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0031] FIG1 is a diagram illustrating an application environment of a method for recommending media data according to an embodiment;

[0032] FIG2 is a schematic flow chart of a method for recommending media data in one embodiment;

[0033] FIG3 is a schematic diagram of extracting a media representation vector in one embodiment;

[0034] FIG4 is a schematic diagram of extracting a text representation vector in one embodiment;

[0035] FIG5 is a schematic diagram of determining target media data based on media data, description text, and a knowledge graph in one embodiment;

[0036] FIG6 is a schematic diagram of determining an entity representation vector in one embodiment;

[0037] FIG7 is a schematic diagram of determining a knowledge enhancement vector in one embodiment;

[0038] FIG8 is a schematic diagram of the structure of a recommendation model in one embodiment;

[0039] FIG9 is a schematic diagram of a method for recommending media data in another embodiment;

[0040] FIG10 is a schematic diagram of a method for processing a recommendation model in one embodiment;

[0041] FIG11 is a schematic diagram of determining a first media training vector in one embodiment;

[0042] FIG12 is a schematic diagram of determining a first text training vector in one embodiment;

[0043] FIG13 is a schematic diagram illustrating determining a knowledge enhancement training vector during the training process of an enhancement vector extraction model according to an embodiment;

[0044] FIG14 is a schematic diagram of a method for processing a recommendation model in another embodiment;

[0045] FIG15 is a structural block diagram of a device for recommending media data according to an embodiment;

[0046] FIG16 is a block diagram of a processing device for a recommendation model according to an embodiment;

[0047] FIG17 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] In this specification and the accompanying drawings, substantially the same or similar steps and elements are denoted by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance or ranking.

[0050] The media data recommendation method provided in the embodiments of the present application can be applied in the application environment shown in FIG1 . In this embodiment, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or placed on a cloud or other network server. The media data recommendation method can be executed by terminal 102, by server 104, or by a collaborative process between terminal 102 and server 104.

[0051] Taking the media data recommendation method executed by server 104 as an example, server 104 can extract media representation vectors and text representation vectors from the media data and the description text of the media data; server 104 can perform knowledge retrieval in the knowledge graph based on the media representation vector, obtain the entity subgraph of the media data, and determine the entity representation vector of the entity subgraph; server 104 can perform feature fusion processing on the media representation vector, text representation vector and entity representation vector to obtain a knowledge enhancement vector; server 104 can also obtain target media data based on the knowledge enhancement vector and recommend the target media data to the target object.

[0052] The terminal 102 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, IoT device, or portable wearable device. IoT device may be a smart speaker, smart TV, smart air conditioner, or smart car device. Portable wearable device may be a smart watch, smart bracelet, or head-mounted device.

[0053] The server 104 may be an independent physical server or a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).

[0054] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0055] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus), or a network, and this application does not impose any limitation thereto.

[0056] In some embodiments, as shown in FIG2 , a method for recommending media data is provided. The method is performed by the server or terminal in FIG1 , or can be performed collaboratively by the server and terminal in FIG1 . The method is described using the server in FIG1 as an example, and includes the following steps:

[0057] Step 202: extract a media representation vector and a text representation vector from the media data and the description text of the media data.

[0058] The media data refers to the media data currently being viewed by the target user, or may also be media data previously viewed by the target user. Specifically, the media data may be a video, image, or live stream. The target user is the user. It should be noted that in media data recommendations, the media data currently being viewed or previously viewed by the user may be of interest to the user. Recommending media data based on potential media interest can improve the compatibility of the recommended media data with the user's preferences.

[0059] The description text is used to describe the content of the media data; for example, the media data is a video, such as the video content is a kitten eating fish, the description text of the video can be: "The newly bought dried fish has arrived, and the kitten eats it with relish"; for example, the media data is a picture, such as the picture content is a pitcher pitching in a baseball game, the description text of the picture can be: "Baseball player pitching".

[0060] The media representation vector is obtained by extracting features from media data and is used to reflect the content of the media data; the text representation vector is obtained by extracting features from the description text and is used to reflect the content of the description text.

[0061] In some embodiments, the server can obtain the media data that the target object is browsing and the description text of the media data. The server can also obtain the media data that the target object has browsed and the description text of the media data. The server can extract media representation vectors from the media data and text representation vectors from the description text through a feature extraction model.

[0062] Exemplarily, the server inputs the media data and the description text into a feature extraction model, and extracts a media representation vector of the media data and a text representation vector of the description text through the feature extraction model.

[0063] In some embodiments, step 202 includes: performing feature extraction on the media data using an image feature extraction model to obtain a media representation vector; and extracting a text representation vector from the description text of the media data using a text feature extraction model.

[0064] Among them, the image feature extraction model includes a first self-attention layer and a visual feedforward layer; the text feature extraction model includes a second self-attention layer and a text feedforward layer.

[0065] As shown in Figure 3, the media data is input into the image feature extraction model, the initial representation vector of the media data is output through the first self-attention layer, and the initial representation vector of the media data is processed through the visual feedforward layer to obtain the media representation vector.

[0066] As shown in Figure 4, the description text is input into the text feature extraction model, the initial representation vector of the description text is output through the second self-attention layer, and the initial representation vector of the description text is processed through the text feedforward layer to obtain the text representation vector.

[0067] In the above embodiment, the media representation vector of the media data is extracted through the image feature extraction model, and the text representation vector of the descriptive text is extracted through the text feature extraction model, so that the media representation vector can reflect the content of the media data and the text representation vector can reflect the content of the descriptive text, thereby improving the quality of the media representation vector and the text representation vector.

[0068] In some embodiments, features are extracted from media data using an image feature extraction model to obtain a media representation vector, including: when the media data is a video, features of multiple image frames in the video are extracted using an image feature extraction model to obtain a media representation vector; when the media data is an image, features are extracted from multiple image blocks of the image using an image feature extraction model to obtain a media representation vector.

[0069] The multiple image frames may be partial image frames in a video, the number of the multiple image frames may be a first preset number, and the size of the image frames may be a preset size. The first preset number and the preset size may be set according to actual needs, and the embodiments of the present application do not limit the first preset number and the preset size.

[0070] Among them, multiple image blocks can be obtained by segmenting the image, the number of multiple image blocks can be a first preset number, and the size of the image block can be a preset size, that is, the size of the image block is the same as the size of the image frame, and the number of multiple image blocks is the same as the number of multiple image frames.

[0071] When the media data is a video, the server can sample a first preset number of image frames from the video, and fill or crop the first preset number of image frames so that the sizes of the first preset number of image frames are all preset sizes; the server inputs the first preset number of image frames into the image representation extraction model, and outputs a media representation vector through the image representation extraction model. The media representation vector includes an image sub-representation vector for each image frame, that is, the media representation vector includes a first preset number of image sub-representation vectors.

[0072] When the media data is an image, the server can divide the image into a first preset number of image blocks, and fill or crop the first preset number of image blocks so that the sizes of the first preset number of image blocks are all preset sizes; the server inputs the first preset number of image blocks into the image representation extraction model, and outputs a media representation vector through the image representation extraction model, where the media representation vector includes the image sub-representation vector of each image block, that is, the media representation vector includes the first preset number of image sub-representation vectors.

[0073] In some embodiments, the media data may also be a live broadcast room. When the media data is a live broadcast room, features of multiple live broadcast frames in the live broadcast room are extracted through image features to obtain a media representation vector.

[0074] Among them, the multiple live picture frames can be partial picture frames of the pictures played in the live broadcast room, the number of the multiple live picture frames can be a first preset number, and the size of the live picture frame can be a preset size.

[0075] In the above embodiment, the media data can be a video or an image, so that the media data recommendation method can be applied to scenarios of browsing videos or browsing images to recommend target media data, thereby improving the applicability of the media data recommendation method.

[0076] Step 204 , perform knowledge retrieval in the knowledge graph based on the media representation vector, obtain the entity subgraph of the media data, and determine the entity representation vector of the entity subgraph.

[0077] Among them, the knowledge graph includes multiple entities and the relationships between multiple entities; the knowledge graph belongs to a node-connection graph, the nodes are used to represent entities, and the connections between nodes are used to represent the relationships between nodes; multiple sets of entity relationships can be obtained through the knowledge graph; illustratively, the entity relationship can be obtained from the knowledge graph: {E1, r1, E2}, E1 is the first entity, E2 is the second entity, r1 is the entity relationship between the first entity and the second entity, for example, the first entity is a ball, the second entity is a table tennis ball, and the relationship is an ownership relationship; in the node-connection graph, the first entity is represented by node E1, the second entity is represented by node E2, and the entity relationship r1 between the first entity and the second entity is represented by the connection r1 between node E1 and node E2.

[0078] In actual applications, the knowledge graph can be constructed by the background of an application that browses media data. For example, the target object browses media data in an instant messaging application, and the knowledge graph is constructed by the background of the instant messaging application.

[0079] Among them, the entity subgraph is a partial graph of the knowledge graph. The entity subgraph involves multiple entities. Multiple entities are some of all entities included in the knowledge graph. The entity subgraph can be used to reflect the relationship between multiple entities in the knowledge graph.

[0080] Exemplarily, all entities included in the knowledge graph are: E1, E2, E3, ..., En, and multiple entities included in the entity subgraph are: E1, E2, ..., Eu, and the entity subgraph is used to reflect the relationship between E1, E2, ..., Eu in the knowledge graph.

[0081] The entity representation vector is used to reflect the relationships between multiple entities and multiple entities in the entity subgraph.

[0082] In some embodiments, the server determines the representation vectors of multiple entities in the knowledge graph, selects entities related to the media data from the multiple entities in the knowledge graph based on the representation vectors of each entity and the media representation vector, and determines the entity subgraph of the media data based on the multiple entities related to the media data and the relationship between the multiple entities in the knowledge graph. The server extracts features from the entity subgraph to obtain the entity representation vector.

[0083] In some embodiments, the server selects multiple entities related to the media data from multiple entities in the knowledge graph based on multiple entity representation vectors and media representation vectors. The server may determine the similarity between the media representation vector and the multiple entity representation vectors, sort the multiple entity representation vectors in descending order of similarity, and obtain an entity representation vector sequence. In the entity representation vector sequence, the server selects multiple target entity representation vectors arranged in the first second preset number, and uses the entities represented by the multiple target entity representation vectors as the multiple entities related to the media data.

[0084] In some embodiments, after the server determines multiple entities related to the media data, it can obtain the neighbor entities of each entity in the knowledge graph, and determine the entity subgraph of the media data based on the multiple entities, the neighbor entities of each entity, and the relationship between the multiple entities and the multiple neighbor entities in the knowledge graph.

[0085] Step 206 : Perform feature fusion processing on the media representation vector, the text representation vector, and the entity representation vector to obtain a knowledge enhancement vector.

[0086] The knowledge enhancement vector is obtained by fusing the knowledge information of the entity into the media representation vector and the text representation vector.

[0087] In some embodiments, the server can obtain the preset weights of the media representation vector, text representation vector and entity representation vector respectively, and perform weighted summation based on the media representation vector, text representation vector, entity representation vector, the preset weights of the media representation vector, the preset weights of the text representation vector and the preset weights of the entity representation vector to obtain a knowledge enhancement vector.

[0088] It should be noted that the sum of the preset weights of the media representation vector, the preset weights of the text representation vector, and the preset weights of the entity representation vector is 1. When the preset weights of the media representation vector, the preset weights of the text representation vector, and the preset weights of the entity representation vector are the same, it is equivalent to averaging the media representation vector, the text representation vector, and the entity representation vector.

[0089] In some embodiments, the server may concatenate the media representation vector and the text representation vector to obtain a first concatenated vector, and the server may extract features from the first concatenated vector through a self-attention network to obtain a first fusion vector; the server may obtain weights of the first fusion vector and the text representation vector respectively, and perform weighted processing on the first fusion vector and the text representation vector based on the weight of the first fusion vector and the weight of the text representation vector to obtain a knowledge enhancement vector.

[0090] In some embodiments, the server may concatenate the media representation vector and the entity representation vector to obtain a second concatenated vector, and concatenate the text representation vector and the entity representation vector to obtain a third concatenated vector; perform feature extraction on the second concatenated vector through a self-attention network to obtain a second fusion vector, and perform feature extraction on the third concatenated vector through a self-attention network to obtain a third fusion vector; the server obtains the weights of the second fusion vector and the third fusion vector respectively, and performs weighted processing on the second fusion vector and the third fusion vector according to the weight of the second fusion vector and the weight of the third fusion vector to obtain a knowledge enhancement vector.

[0091] In some embodiments, the server concatenates the media representation vector, the text representation vector, and the entity representation vector to obtain a concatenated vector, and performs feature extraction on the concatenated vector through a self-attention network to obtain a knowledge enhancement vector.

[0092] In some embodiments, the server performs weighted summation of the media representation vector, the text representation vector, and the entity representation vector based on the preset weights of the media representation vector, the text representation vector, and the entity representation vector to obtain a fusion vector; and performs feature extraction on the fusion vector through a self-attention network to obtain a knowledge enhancement vector.

[0093] Step 208: Acquire target media data based on the knowledge enhancement vector and recommend the target media data to the target object.

[0094] The target media data is media data recommended to the target object.

[0095] In some embodiments, since the knowledge enhancement vector is obtained by fusing the knowledge information of the entity in the media representation vector and the text representation vector, the knowledge enhancement vector can reflect the content of the media data and the descriptive text, as well as the entity information related to the media data. Therefore, by obtaining the target media data based on the knowledge enhancement vector, the target media data can be obtained, whose content is related to the media data and the descriptive text, and whose entity is related to the media data.

[0096] In some embodiments, step 208 includes: classifying the knowledge enhancement vector to obtain the interest type of the target object; acquiring target media data according to the interest type, and recommending the target media data to the target object.

[0097] The interest type is a type that the target object may be interested in, and the number of interest types can be one or more; the number of target media data can be one or more.

[0098] In some embodiments, the server inputs the knowledge enhancement vector into a classification model, uses the classification model to determine the predicted probability that the knowledge enhancement vector belongs to each preset type, selects a target probability from the multiple predicted probabilities, and the selected target probability is greater than the unselected predicted probabilities. The predicted type corresponding to the selected target probability is used as the target object's interest type. It should be noted that when a single target probability is selected, a single interest type is determined; when multiple target probabilities are selected, multiple interest types are determined.

[0099] When the interest type is one, the server can select target media data from multiple candidate media data belonging to the interest type; the server obtains the popularity values ​​of multiple candidate media data, and can select the candidate media data with the largest popularity value from the multiple candidate media data as the target media data, or can select multiple candidate media data with larger popularity values ​​from the multiple candidate media data as the target media data; the popularity value of the target media data is greater than the popularity value of the unselected candidate media data; in actual applications, the popularity value of the candidate media data can be determined based on the number of views, comments and likes of the candidate media data.

[0100] When there are multiple interest types, the server may select target media data belonging to each interest type from multiple candidate media data belonging to the multiple interest types, to obtain multiple target media data.

[0101] When there is only one target media data, the server can send the target media data to the terminal used by the target object to browse the media data; when the target object triggers the operation of switching to the next media data during browsing the media data, the terminal can play the target media data.

[0102] When there are multiple target media data, the server can sort the multiple target media data in descending order according to the popularity value to obtain a target media data list, and send the target media data list to the terminal used by the target object to browse the media data; the terminal can display the target media data list in the recommendation area of ​​the browsing page of the media data, and the terminal responds to the trigger operation for any target media data in the target media data list and plays the target media data that receives the trigger operation; when the target object triggers the operation of switching to the next media data while browsing the media data, the terminal can play the target media data ranked first in the target media data list.

[0103] In the above embodiment, the interest type of the target object is determined based on the knowledge enhancement vector, and target media data whose content is related to the media data and the descriptive text, and whose entity is related to the media data, can be obtained, thereby improving the correlation between the target media data and the media data. Furthermore, the target media data may be the media data that the target object is interested in, thereby improving the media recommendation effect. In this way, the target object does not need to perform manual search to obtain and browse the media data of interest, thereby reducing the resource consumption caused by manual search and improving resource utilization.

[0104] Exemplarily, as shown in FIG5 , the server extracts the media representation vector of the media data, extracts the text representation vector of the description text of the media data, searches the knowledge graph based on the media representation vector to obtain an entity subgraph, and determines the entity representation vector of the entity subgraph. The server fuses the media representation vector, the text representation vector, and the entity representation vector to obtain a knowledge enhancement vector, obtains the target media data based on the knowledge enhancement vector, and recommends the target media data to the target object.

[0105] In the above-mentioned media data recommendation method, a media representation vector and a text representation vector are extracted from the media data and the description text of the media data. An entity subgraph is retrieved from the knowledge graph based on the media representation vector, and the entity representation vector of the entity subgraph is determined. Feature fusion processing is performed on the media representation vector, the text representation vector, and the entity representation vector to obtain a knowledge enhancement vector. Based on the knowledge enhancement vector, target media data is recommended to the target object. An entity subgraph related to the content of the media data is retrieved from the knowledge graph based on the media representation vector. Then, based on the entity subgraph, an entity representation vector related to the content of the media data can be obtained. The media representation vector, the text representation vector, and the entity representation vector are fused to obtain a knowledge enhancement vector, so that the knowledge enhancement vector can reflect the content of the media data and the description text, as well as entity information related to the content of the media data. Therefore, based on the knowledge enhancement vector, target media data whose content is similar to the media data and whose entities are related to the entities of the media data can be obtained, thereby improving the relevance of the target media data to the media data. Therefore, the target media data may be media data of interest to the target object, thereby improving the media recommendation effect. In this way, the target object can obtain and browse the media data of interest without performing manual search, reducing resource consumption caused by manual search and improving resource utilization.

[0106] In some embodiments, knowledge retrieval is performed in a knowledge graph based on a media representation vector to obtain an entity subgraph of the media data, and an entity representation vector of the entity subgraph is determined, including: retrieving a target entity related to the media data in the knowledge graph based on the media representation vector; determining the entity subgraph of the media data based on the target entity and the knowledge graph; and extracting features of multiple entities in the entity subgraph to obtain an entity representation vector.

[0107] The target entity is an entity in the knowledge graph that is related to the content of the media data. For example, if the media data is an image of a baseball player pitching on a field, the target entity may include, but is not limited to, baseball, player, pitching, and field.

[0108] The entity representation vector includes entity sub-representation vectors of multiple entities in the entity sub-graph.

[0109] In some embodiments, the server obtains multiple initial entity vectors in the knowledge graph, determines the correlations between the media representation vectors and the multiple initial entity vectors, selects candidate correlations from the correlations between the media representation vectors and the multiple initial entity vectors, and uses the entity represented by the initial entity vector used to calculate the candidate correlation as the target entity related to the media data.

[0110] Selecting candidate correlations from the correlations between the media representation vector and the multiple initial entity vectors can be done by arranging the correlations between the media representation vector and the multiple initial entity vectors in descending order to obtain an initial correlation sequence, and selecting a third preset number of candidate correlations arranged at the front in the initial correlation sequence.

[0111] In some embodiments, determining an entity subgraph of media data based on the target entity and the knowledge graph may be determining the relationship between target entities based on the knowledge graph, and determining the entity subgraph based on the target entity and the relationship between the target entities.

[0112] The server extracts features from the entity subgraph to obtain entity sub-representation vectors of the multiple entities in the entity subgraph, and determines an entity feature representation vector based on the entity sub-representation vectors of the multiple entities in the entity subgraph.

[0113] In the above embodiment, the target entity related to the media data is retrieved in the knowledge graph through the media representation vector, and then the entity subgraph of the media entity is determined based on the target entity. Since the entity subgraph includes the target entity and the relationship between the target entities, the entity representation vector is determined based on the entity subgraph, so that the entity representation vector can more accurately reflect the entity in the media data, thereby improving the accuracy of the entity representation vector.

[0114] In some embodiments, the media representation vector includes at least two image sub-representation vectors; based on the media representation vector, retrieving the target entity related to the media data in the knowledge graph includes: obtaining multiple initial entity vectors in the knowledge graph; retrieving candidate entities in the knowledge graph based on the multiple initial entity vectors and at least two image sub-representation vectors; and selecting the target entity related to the media data from the candidate entities.

[0115] Among them, the initial entity vector is the representation vector of multiple entities in the knowledge graph; the initial entity vector can be predetermined by the encoder.

[0116] Among them, when the media data is an image, the media representation vector is obtained by feature extraction of at least two image blocks obtained by dividing the image, and then at least two image sub-representation vectors are used to represent the at least two image blocks; when the media data is a video, the media representation vector is obtained by feature extraction of at least two image frames in the video, and then at least two image sub-representation vectors are used to represent the at least two image frames.

[0117] In some embodiments, for each initial entity vector, the correlation between the initial entity vector and each image sub-representation vector is determined, and a candidate entity is determined based on the correlation between the initial entity vector and each image sub-representation vector.

[0118] Determining the candidate entity based on the correlation between the initial entity vector and each image sub-representation vector can be done by determining whether there is at least one correlation between the initial entity vector and each image sub-representation vector that falls within a preset interval. If so, determining that the entity represented by the initial entity vector is a candidate entity; if not, determining that the entity represented by the initial entity vector is not a candidate entity.

[0119] Among them, the correlation degree belonging to the preset interval is greater than the correlation degree not belonging to the preset interval. The preset interval can be set according to actual needs, and the embodiment of the present application does not limit the specific range of the preset interval.

[0120] In some embodiments, for each image sub-representation vector, the correlation between the image sub-representation vector and each initial entity vector is determined, and the candidate entity is determined based on the correlation between the image sub-representation vector and each initial entity vector.

[0121] Determining the candidate entity based on the correlation between the image sub-representation vector and each initial entity vector can be done by determining a correlation set of the image sub-representation vector based on the correlation between the image sub-representation vector and each initial entity vector, selecting multiple higher correlations from the correlation set of the image sub-representation vector, and using the initial entity vector used to calculate the higher correlation as the initial entity vector associated with the image sub-representation vector, and then using the entity represented by the initial entity vector associated with the image sub-representation vector as the candidate entity associated with the image sub-representation vector.

[0122] Selecting a higher correlation from the set of correlations of the image sub-representation vectors may involve selecting a fourth preset number of correlations from the set of correlations in descending order of correlation. The fourth preset number may be set based on actual needs, and the specific value of the fourth preset number is not limited in this embodiment of the application.

[0123] In some embodiments, candidate entities are retrieved based on at least two image sub-representation vectors; among the candidate entities, selecting the target entity related to the media data can be to obtain at least one target entity from the candidate entities retrieved based on each image sub-representation vector to obtain the target entity related to the media data; among the candidate entities retrieved based on each image sub-representation vector, obtaining at least one target entity can be to determine the candidate correlation between the candidate entity and the image sub-representation vector, and obtaining at least one target entity from the candidate entities based on the candidate correlation, and the candidate correlation between the target entity and the image sub-representation vector is greater than the candidate correlation between other candidate entities and the image sub-representation vector.

[0124] In some embodiments, candidate entities are retrieved based on at least two image sub-representation vectors; among the candidate entities, selecting a target entity related to the media data may be selecting at least one target entity from all candidate entities retrieved based on at least two image sub-representation vectors; for example, for a candidate entity retrieved based on each image sub-representation vector, the correlation between the image sub-representation vector and the candidate entity is used as the candidate correlation of the candidate entity; the server sorts all candidate entities retrieved based on at least two image sub-representation vectors in descending order of candidate correlation to obtain a candidate entity sequence, and in the candidate entity sequence, selects the fifth preset number of candidate entities arranged in front, and uses the fifth preset number of candidate entities as the target entities related to the media data. The fifth preset number can be set according to actual needs, and the embodiment of the present application does not limit the specific value of the fifth preset number.

[0125] In the above embodiment, based on the initial entity vector and at least two image sub-representation vectors in the knowledge graph, candidate entities related to the image sub-representation vector are retrieved in the knowledge graph, and among the candidate entities related to the image sub-representation vector, the target entity related to the media data is selected, so that the determined target entity is related to multiple image sub-representation vectors of the media data, and then the target entity can reflect the content of the media data, thereby improving the accuracy of retrieving the target entity related to the media data.

[0126] In some embodiments, candidate entities are retrieved in a knowledge graph based on multiple initial entity vectors and at least two image sub-representation vectors, including: determining a correlation set of at least two image sub-representation vectors based on multiple initial entity vectors and at least two image sub-representation vectors; the correlation set includes the correlations between the image sub-representation vectors and the initial entity vectors respectively; and selecting candidate entities related to the at least two image sub-representation vectors in the knowledge graph based on the correlation set.

[0127] The candidate entities associated with at least two image sub-representation vectors include candidate entities associated with each image sub-representation vector.

[0128] In some embodiments, for each image sub-representation vector, the server determines the correlation between the image sub-representation vector and multiple initial entity vectors, and determines a correlation set of the image sub-representation vector based on the correlation between the image sub-representation vector and multiple initial entity vectors.

[0129] Exemplarily, the correlation between the image sub-representation vector and the initial entity vector is determined as shown in formula (1).

[0130] Formula (1): sim(v i ,m j )=f v (v i ) T f e (m j );

[0131] Among them, f v (v i ) is the image sub-representation vector, f v (v i ) T is the transposed vector of the image sub-representation vector, f e (m j ) is the initial entity vector, sim(v i ,m j ) is the correlation between the image sub-representation vector and the initial entity vector.

[0132] The server arranges multiple relevances in the relevance set in descending order, selects the sixth preset number of target relevances in the sorted relevance set, obtains the initial entity representation vectors of the sixth preset number of target relevances, and uses the entities represented by the obtained multiple initial entity representation vectors as candidate entities related to the image sub-representation vector.

[0133] Exemplarily, assuming that the media representation vector includes s image sub-representation vectors and the sixth preset number is t, s*t candidate entities can be obtained.

[0134] In the above embodiment, in the correlation set of each image sub-representation vector, a higher correlation is obtained, and candidate entities related to the image sub-representation vector are selected based on the higher correlation, so that the candidate entities can reflect the content represented by the image sub-representation vector, thereby improving the accuracy of the selected candidate entities.

[0135] In some embodiments, based on the target entity and the knowledge graph, an entity subgraph of the media data is determined, including: determining the neighbor nodes of each target entity in the knowledge graph; determining the extended entity based on multiple target entities and the neighbor nodes of each target entity, and determining the relationship between the extended entities in the knowledge graph; determining the entity subgraph of the media data based on the extended entity and the relationship between the extended entities.

[0136] Among them, the neighbor nodes of the target entity can be the first-order neighbor nodes of the target entity, or can include the first-order neighbor nodes and second-order neighbor nodes of the target entity; the extended entity includes the target entity and the entities represented by the neighbor nodes;.

[0137] Exemplarily, the entities included in the knowledge graph can be represented as: {E1, E2, …, En}, and based on the media representation vector V, multiple target entities related to the media are retrieved in {E1, E2, …, En}: {E1, E2, …, Eq}, and the first-order neighbor nodes of the multiple target entities are determined in the knowledge graph to expand the target entities and obtain the extended entities: {E1, E2, …, Eu}, and an entity subgraph G is built based on the extended entities {E1, E2, …, Eu} in the knowledge graph.

[0138] In the above embodiment, neighbor nodes of the target entity are obtained in the knowledge graph to expand the target entity, so that the entities included in the entity subgraph are richer, thereby improving the quality of the entity representation vector of the entity subgraph.

[0139] In some embodiments, based on the media representation vector, multiple target entities related to the media data are retrieved in the knowledge graph, including: retrieving multiple target entities related to the media data in the knowledge graph based on the media representation vector through the retrieval sub-model of the knowledge retrieval model; determining the entity subgraph of the media data based on the target entity and the knowledge graph, including: building a network through the subgraph of the knowledge retrieval model, and determining the entity subgraph of the media data based on multiple target entities and the knowledge graph; extracting features of multiple entities in the entity subgraph to obtain the entity representation vector, including: extracting features of multiple entities in the entity subgraph through the graph neural network of the knowledge retrieval model to obtain the entity representation vector.

[0140] Among them, the knowledge retrieval model includes a retrieval sub-model, a sub-graph construction network and a graph neural network; as shown in Figure 6, the server inputs the media representation vector and the knowledge graph into the retrieval sub-model, and retrieves the target entity related to the media data through the retrieval sub-model; the target entity and the knowledge graph are input into the sub-graph construction network, and the entity sub-graph of the media data is output through the sub-graph construction network; the entity sub-graph is input into the graph neural network, and the features of multiple entities in the entity sub-graph are extracted through the graph neural network to obtain the entity representation vector.

[0141] In some embodiments, the retrieval sub-model can encode multiple entities in the knowledge graph to obtain initial entity vectors of the multiple entities, and determine a correlation set of the multiple image sub-representation vectors based on the initial entity vectors and the multiple image sub-representation vectors in the media representation vector. Based on the correlation set, candidate entities related to the multiple image sub-representation vectors are selected in the knowledge graph, and among the multiple candidate entities, multiple target entities related to the media data are selected.

[0142] The target entity and the knowledge graph are input into the subgraph construction network. The subgraph construction network can obtain the neighbor nodes of the target entity in the knowledge graph, determine the extended entity based on the target entity and the neighbor nodes, determine the relationship between the extended entities in the knowledge graph, and determine the entity subgraph based on the extended entity and the relationship between the extended entities.

[0143] For each entity in the entity subgraph, a node trajectory graph is obtained based on the entity's random walk within the entity subgraph. This node trajectory graph is then fed into a graph neural network, which then outputs a representation vector for the entity. Similarly, the representation vector of each entity in the entity subgraph is determined, and the entity representation vector is obtained based on each entity's representation vector. In practical applications, graph neural networks can be GNNs (Graph Neural Networks).

[0144] In the above embodiment, the retrieval sub-model, sub-graph construction network and graph neural network in the knowledge retrieval model are used to retrieve target entities related to the media data, and an entity sub-graph of the media data is constructed. The entity representation vectors of multiple entities in the entity sub-graph are extracted, so that the entity representation vector can more accurately reflect the entities in the media data, thereby improving the accuracy of the entity representation vector.

[0145] In some embodiments, feature fusion processing is performed on the media representation vector, the text representation vector and the entity representation vector to obtain a knowledge enhancement vector, including: splicing the media representation vector, the text representation vector and the entity representation vector, and during the splicing process, adding a separation element between the media representation vector and the text representation vector, and adding a separation element between the text representation vector and the entity representation vector to obtain a splicing vector; and feature fusion processing is performed on the splicing vector through a knowledge enhancement model to obtain a knowledge enhancement vector.

[0146] Among them, the knowledge enhancement model includes a regularization layer, an encoder and a feedforward network layer.

[0147] The separator element in the splicing vector can be used to distinguish the media representation vector, text representation vector and entity representation vector in the splicing vector.

[0148] The knowledge enhancement vector includes: a media enhancement vector, a text enhancement vector and an entity enhancement vector; when the splicing vector includes a separation element, the knowledge enhancement vector also includes a separation element for distinguishing the media enhancement vector, the text enhancement vector and the entity enhancement vector.

[0149] In some embodiments, the media representation vector is {v1, v2,…, vn}, the text representation vector is {t1, t2,…, tn}, and the entity representation vector is {e1, e2,…, en}. The server splices the media representation vector, the text representation vector, and the entity representation vector, and during the splicing process, adds a separator element [sep] between the media representation vector and the text representation vector, and between the text representation vector and the entity representation vector to obtain the spliced ​​vector {v1, v2,…, vn[sep]t1, t2,…, tn[sep]e1, e2,…, en}.

[0150] In some embodiments, as shown in Figure 7, the server inputs the splicing vector into the knowledge enhancement model, randomly discards the splicing vector through the regularization layer to reduce the amount of processed data, and obtains a regularized vector; for example, the splicing vector {v1, v2,…, vn[sep]t1, t2,…, tn[sep]e1, e2,…, en} is processed through the regularization layer to obtain the regularized vector {v1, 0,…, vn[sep]0, t2,…, tn[sep]e1, e2,…, 0}.

[0151] The regularized vector is processed by the encoder to achieve multimodal fusion of the media representation vector, text representation vector and entity representation vector in the regularized vector to obtain a fusion vector; in practical applications, the encoder can be implemented through a multi-head attention network; for example, the regularized vector {v1,0,…, vn[sep]0,t2,…, tn[sep]e1,e2,…, 0} is multimodally fused by the encoder to obtain a fusion vector {a1,a2,…, an[sep]b1,b2,…, bn[sep]c1,c2,…, cn}.

[0152] The fused vector is activated through the feedforward network layer to obtain a knowledge-enhanced vector. Compared with the fused vector, the knowledge-enhanced vector strengthens the expression of media data, descriptive text and entities. For example, the fused vector {a1, a2, …, an[sep] b1, b2, …, bn[sep] c1, c2, …, cn} is activated and fused through the feedforward network layer to obtain the knowledge-enhanced vector {x1, x2, …, xn[sep] y1, y2, …, yn[sep] z1, z2, …, zn}.

[0153] It should be noted that the knowledge enhancement vector includes the media enhancement vector {x1, x2, ..., xn}, the text enhancement vector {y1, y2, ..., yn} and the entity enhancement vector {z1, z2, ..., zn}.

[0154] In the above embodiment, the media representation vector, the text representation vector and the entity representation vector are spliced ​​to obtain a spliced ​​vector, and the spliced ​​vector is feature fused through the indication enhancement model to obtain a knowledge enhancement vector. The multimodal representation vector is fused so that the knowledge enhancement vector can reflect the media data, the descriptive text and the content, as well as the entity information related to the content of the media data. Based on the knowledge enhancement vector, the target media data whose content is similar to the media data and whose entities are related to the entities of the media data can be obtained, thereby improving the media recommendation effect. In this way, the target object does not need to perform manual search to obtain and browse the media data of interest, thereby reducing the resource consumption caused by manual search and improving resource utilization.

[0155] In some embodiments, the method for recommending media data further includes: extracting a first media training vector and a first text training vector from the first sample media data and the first sample text of the first sample media data based on a feature extraction model; performing knowledge retrieval processing on the first media training vector and the knowledge graph based on a knowledge retrieval model to obtain a training subgraph of the first sample media data, and determining an entity training vector of the training subgraph; performing feature fusion processing on the first media training vector, the first text training vector, and the entity training vector based on a knowledge enhancement model to obtain a knowledge enhancement training vector; determining a visual loss value and a language loss value based on the knowledge enhancement training vector and the sample label of the first sample media data; determining a knowledge retrieval loss value based on the knowledge enhancement training vector and the training subgraph; adjusting parameters of the feature extraction model, the knowledge retrieval model, and the knowledge enhancement model based on the visual loss value, the language loss value, and the knowledge retrieval loss value to obtain an enhanced vector extraction model; determining a recommendation model based on the enhanced vector extraction model and the classification model; the recommendation model is used to extract a knowledge enhancement vector based on the media data, the descriptive text, and the knowledge graph, and determine an interest type based on the knowledge enhancement vector, so as to obtain target media data based on the interest type and recommend the target media data to the target object.

[0156] In some embodiments, the recommendation method for media data can be applied to a recommendation model. As shown in Figure 8, the recommendation model includes: an enhanced vector extraction model and a classification model; the enhanced vector extraction model includes: an image feature extraction model, a text feature extraction model, a knowledge retrieval model and a knowledge enhancement model; the enhanced vector extraction model is obtained by adjusting the parameters of the pre-trained feature extraction model, the knowledge retrieval model and the knowledge enhancement model; wherein, the pre-trained feature extraction model includes: a pre-trained image feature extraction model and a text feature extraction model.

[0157] In practical applications, the media data, descriptive text and knowledge graph are processed by the enhancement vector model in the recommendation model to obtain the knowledge enhancement vector, and the knowledge enhancement vector is classified by the classification model in the recommendation model to obtain the interest type of the target object, so as to obtain the target media data based on the interest type and recommend the target media data to the target object.

[0158] In some embodiments, the knowledge-enhanced training vector includes a media-enhanced training vector and a text-enhanced training vector; the sample labels include a shielded sub-image label and a shielded word label; based on the knowledge-enhanced training vector and the sample label of the first sample media data, the visual loss value and the language loss value are determined, including: obtaining the shielded sub-image training vector in the media-enhanced training vector, and determining the visual loss value based on the shielded sub-image training vector and the shielded sub-image label; classifying the text-enhanced training vector to obtain the shielded word prediction probability, and determining the language loss value based on the shielded word prediction probability and the shielded word label.

[0159] In some embodiments, the knowledge enhancement training vector also includes an entity enhancement training vector; based on the knowledge enhancement training vector and the training subgraph, the knowledge retrieval loss value is determined, including: selecting an entity enhancement training vector pair in the entity enhancement training vector, and determining a first score of the entity enhancement training vector pair; obtaining an entity negative sample pair in the training subgraph, and determining a second score of the entity negative sample pair; the entity negative sample pair includes two training entities that do not have an entity relationship in the training subgraph; and determining the knowledge retrieval loss value based on the first score and the second score.

[0160] In some embodiments, the first sample media data and the first sample text belong to a sample set, and the sample set also includes second sample media data and second sample text; the method also includes: performing feature extraction on the second sample media data and the second sample text to obtain a second media training vector and a second text training vector; determining an image-text contrast loss value based on the second media training vector, the second text training vector, the first media training vector and the first text training vector; adjusting the parameters of the pre-trained feature extraction model, the pre-trained knowledge retrieval model and the pre-trained knowledge enhancement model based on the visual loss value, the language loss value and the knowledge retrieval loss value to obtain a post-trained recommendation model, including: adjusting the parameters of the pre-trained feature extraction model, the pre-trained knowledge retrieval model and the pre-trained knowledge enhancement model based on the visual loss value, the language loss value, the knowledge retrieval loss value and the image-text contrast loss value to obtain a post-trained recommendation model.

[0161] In some embodiments, an image-text contrast loss value is determined based on the second media training vector, the second text training vector, the first media training vector, and the first text training vector, including: determining a first similarity based on the first media training vector and the second text training vector; determining a second similarity based on the first text training vector and the second media training vector; and determining an image-text contrast loss value based on the first similarity, the second similarity, the first similarity label, and the second similarity label.

[0162] In some embodiments, as shown in FIG9 , a method for recommending media data includes:

[0163] Step 901: When the media data is a video, features of multiple image frames in the video are extracted using an image feature extraction model to obtain a media representation vector. When the media data is an image, features of multiple image blocks are extracted using the image feature extraction model to obtain a media representation vector. The media representation vector includes at least two image sub-representation vectors.

[0164] Step 902: extracting a text representation vector from the description text of the media data using a text feature extraction model;

[0165] Step 903: Obtain initial entity vectors of multiple entities in the knowledge graph using the retrieval sub-model of the knowledge retrieval model. Determine a correlation set of the at least two image sub-representation vectors based on the multiple initial entity vectors and the at least two image sub-representation vectors; the correlation set includes correlations between the image sub-representation vectors and the initial entity vectors. Based on the correlation set, select candidate entities in the knowledge graph that are related to the at least two image sub-representation vectors; and select multiple target entities related to the media data from among the candidate entities.

[0166] Step 904: Construct a network using the subgraph of the knowledge retrieval model to determine neighbor nodes of each target entity in the knowledge graph; determine extended entities based on the multiple target entities and the neighbor nodes of each target entity, and determine relationships between the extended entities in the knowledge graph; and determine an entity subgraph of the media data based on the extended entities and the relationships between the extended entities.

[0167] Step 905: Extract features of multiple entities in the entity subgraph through the graph neural network of the knowledge retrieval model to obtain entity representation vectors;

[0168] Step 906: Concatenate the media representation vector, the text representation vector, and the entity representation vector. During the concatenation process, a separator element is added between the media representation vector and the text representation vector, and between the text representation vector and the entity representation vector to obtain a concatenated vector. Feature fusion processing is performed on the concatenated vector using a knowledge enhancement model to obtain a knowledge enhancement vector.

[0169] Step 907 : Classify the knowledge enhancement vector to obtain the interest type of the target object; obtain target media data based on the interest type, and recommend the target media data to the target object.

[0170] In the above-mentioned media data recommendation method, a media representation vector and a text representation vector are extracted from the media data and the description text of the media data. An entity subgraph is retrieved from the knowledge graph based on the media representation vector, and the entity representation vector of the entity subgraph is determined. Feature fusion processing is performed on the media representation vector, the text representation vector, and the entity representation vector to obtain a knowledge enhancement vector. Based on the knowledge enhancement vector, target media data is recommended to the target object. An entity subgraph related to the content of the media data is retrieved from the knowledge graph based on the media representation vector. Then, based on the entity subgraph, an entity representation vector related to the content of the media data can be obtained. The media representation vector, the text representation vector, and the entity representation vector are fused to obtain a knowledge enhancement vector, so that the knowledge enhancement vector can reflect the content of the media data and the description text, as well as entity information related to the content of the media data. Therefore, based on the knowledge enhancement vector, target media data whose content is similar to the media data and whose entities are related to the entities of the media data can be obtained, thereby improving the relevance of the target media data to the media data. Therefore, the target media data may be media data of interest to the target object, thereby improving the media recommendation effect. In this way, the target object can obtain and browse the media data of interest without performing manual search, reducing resource consumption caused by manual search and improving resource utilization.

[0171] In some embodiments, as shown in FIG10 , a method for processing a recommendation model is provided. The method can be executed by a server or a terminal. The method is described using the server execution as an example and includes the following steps:

[0172] Step 1002 : extracting a first media training vector and a first text training vector from the first sample media data and the first sample text of the first sample media data based on a feature extraction model.

[0173] Among them, the feature extraction model includes a pre-trained image feature extraction model and a text feature extraction model.

[0174] In some embodiments, feature extraction is performed on the first sample media data using a pre-trained image feature extraction model to obtain a first media training vector, and feature extraction is performed on the first sample text using a pre-trained text feature extraction model to obtain a first text training vector.

[0175] In some embodiments, the pre-trained image feature extraction model can be implemented using a first bidirectional encoding model (Transformer), which includes multiple image encoders. The first sample media data includes multiple sample images and masked sub-images. An initial sample image can be divided to obtain multiple sample images, and some of the multiple sample images can be masked to obtain the first media sample data including the multiple sample images and the masked sub-images. Alternatively, image frames in a sample video can be sampled to obtain multiple sample images, and some of the multiple sample images can be masked to obtain the first media sample data.

[0176] Exemplarily, as shown in FIG11 , the initial sample image is divided into N sample images, and the N sample images are masked by a visual mask model to mask part of the N sample images to obtain first sample media data; the first bidirectional coding model includes L image encoders, and the first sample media data is processed by the L image encoders to obtain a first media training vector.

[0177] In some embodiments, the pre-trained image feature extraction model can be implemented using a second bidirectional encoding model, which includes multiple text encoders. The first sample text includes multiple words and blocked words; the initial sample text can be segmented to obtain multiple words, and some of the multiple words can be blocked to obtain the first sample text including the multiple words and blocked words.

[0178] Exemplarily, as shown in Figure 12, the initial sample is segmented to obtain words, for example, the initial sample is: "A baseball player throwing a ball in a game", and the words are: "A", "baseball", "player", "throwing", "a", "ball", "in", "a" and "game"; the words are masked by the text mask model to shield some of the words, and a start mark is added in front of the masked sample text to obtain a first sample text, for example, the first sample text includes: "[cls]", "A", "[MASK]", "[MASK]", "throwing", "a", "[MASK]", "in", "a", "game"; the second bidirectional encoding model includes L text encoders, and the first sample text is processed by L text encoders to obtain a first text training vector.

[0179] Step 1004: Based on the knowledge retrieval model, perform knowledge retrieval processing on the first media training vector and the knowledge graph to obtain a training subgraph of the first sample media data, and determine the entity training vector of the training subgraph.

[0180] Among them, the knowledge retrieval model in this step is a pre-trained knowledge retrieval model, and the pre-trained knowledge retrieval model includes: a pre-trained retrieval sub-model, a pre-trained sub-graph construction network and a pre-trained graph neural network.

[0181] In some embodiments, the first media training vector and the knowledge graph are processed by a pre-trained retrieval sub-model to retrieve training entities related to the first sample media data, the training entities and the knowledge graph are processed by a pre-trained sub-graph construction network to construct a training sub-graph of the first sample media data, and features are extracted from the training sub-graph by a pre-trained graph neural network to obtain an entity training vector of the training sub-graph.

[0182] Step 1006 : Based on the knowledge enhancement model, feature fusion processing is performed on the first media training vector, the first text training vector, and the entity training vector to obtain a knowledge enhancement training vector.

[0183] The knowledge enhancement model in this step is a pre-trained knowledge enhancement model, which includes: a pre-trained regularization layer, a pre-trained encoder, and a pre-trained feedforward layer.

[0184] In some embodiments, the server splices the first media training vector, the first text training vector and the entity training vector, and adds separation elements between the first media training vector and the first text training vector, and between the first text training vector and the entity training vector during the splicing process to obtain a training splicing vector, and randomly discards the training splicing vector through a pre-training regularization layer to obtain a training regularization vector; fuses the training regularization vector through a pre-training encoder to achieve multimodal fusion of the first media training vector, the first text training vector and the entity training vector in the training regularization vector to obtain a training fusion vector; activates the training fusion vector through a pre-training feedforward layer to obtain a knowledge-enhanced training vector.

[0185] It should be noted that the knowledge enhancement training vector includes media enhancement training vector, text enhancement training vector and entity enhancement training vector.

[0186] In some embodiments, the pre-trained encoder includes: a self-attention layer, a first normalization layer, a feed-forward layer, and a second normalization layer; the process of the self-attention layer processing the training regularization vector is shown in formula (2).

[0187] Formula (2):

[0188] in, is the representation vector output by the self-attention layer, is the training regularization vector, the self-attention layer includes multiple attention heads, is the parameter of the mth attention head, is the value parameter of the mth attention head, is the query parameter of the mth attention head, is the key parameter of the mth attention head.

[0189] The process of processing the representation vector and training regularization vector output by the self-attention layer through the first normalization layer is shown in formula (3).

[0190] Formula (3):

[0191] in, is the representation vector output by the first normalization layer, is the representation vector output by the self-attention layer, is the training regularization vector.

[0192] The process of processing the representation vector output by the first normalization layer through the feedforward layer is shown in formula (4).

[0193] Formula (4):

[0194] in, is the representation vector output by the feedforward layer, and are the network parameters of the feedforward layer, is the representation vector output by the first normalization layer.

[0195] The process of processing the representation vector output by the feedforward layer and the representation vector output by the first normalization layer through the second normalization layer is shown in formula (5).

[0196] Formula (5):

[0197] in, is the representation vector output by the second normalization layer, i.e. the training fusion vector, is the representation vector output by the first normalization layer, is the representation vector output by the feedforward layer.

[0198] Step 1008: Determine the visual loss value and the language loss value based on the knowledge-enhanced training vector and the sample label.

[0199] Among them, the sample labels include shielded sub-image labels and shielded word labels; the visual loss value is used to reflect the difference between the media enhancement training vector and the shielded sub-image label, and the language loss value is used to reflect the difference between the predicted probability corresponding to the text enhancement training vector and the shielded word label.

[0200] In some embodiments, step 1008 includes: obtaining a shielded sub-image enhancement vector in a media enhancement training vector, and determining a visual loss value based on the shielded sub-image enhancement vector and the shielded sub-image label; performing classification processing on the text enhancement training vector to obtain a shielded word prediction probability, and determining a language loss value based on the shielded word prediction probability and the shielded word label.

[0201] Among them, the media enhancement training vector includes a shielded sub-image enhancement vector of the shielded sub-image; the shielded sub-image enhancement vector is a feature vector obtained by reconstructing the shielded sub-image through the first sample media data, the first sample text and the knowledge graph.

[0202] The text enhancement training vector includes a shielded word enhancement vector of the shielded word; the shielded word enhancement vector is a feature vector obtained by predicting the shielded word through the first sample media data, the first sample text and the knowledge graph.

[0203] In some embodiments, the server obtains a masked sub-image enhancement vector from the media enhancement training vector, obtains a masked sub-image label for the masked sub-image enhancement vector, and calculates a visual loss value based on the masked sub-image enhancement vector and the masked sub-image label for the masked sub-image enhancement vector. It should be noted that the masked sub-image enhancement vector and the masked sub-image label for the masked sub-image enhancement vector correspond to the same masked sub-image.

[0204] For example, the visual loss value can be determined by a cross entropy loss function, as shown in formula (6).

[0205] Formula (6):

[0206] in, is the visual loss value, y(I) msk is the masked sub-image label, p(I) msk is the masked sub-image enhancement vector, and H(·) is the cross entropy loss function.

[0207] In some embodiments, when there are multiple shielded sub-image enhancement vectors, the loss value of the shielded sub-image enhancement vector can be determined based on each shielded sub-image enhancement vector and the corresponding shielded sub-image label, and the average value can be calculated based on the respective loss values ​​of the multiple shielded sub-image enhancement vectors to obtain the visual loss value.

[0208] In some embodiments, the server may classify the text enhancement training vector using a classifier to obtain a blocked word prediction probability. The server then obtains the blocked word enhancement vector from the text enhancement training vector and calculates the language loss value using the blocked word prediction probability and blocked word label. It should be noted that the blocked word prediction probability and blocked word label correspond to the same blocked word.

[0209] For example, the language loss value can be determined by a cross entropy loss function, as shown in formula (7).

[0210] Formula (7):

[0211] in, is the language loss value, y(T) msk is the blocked word label, p(T) msk is the predicted probability of the masked word, and H(·) is the cross entropy loss function.

[0212] In the above embodiment, the visual loss value is determined by predicting the representation vector of the masked sub-image, and the language loss value is determined by predicting the representation vector of the masked word, thereby improving the accuracy of the visual loss value and the language loss value, so as to facilitate the subsequent adjustment of the model parameters through the visual loss value and the language loss value.

[0213] Step 1010: Determine a knowledge retrieval loss value based on the knowledge-enhanced training vector and the training subgraph.

[0214] The knowledge enhancement training vector includes an entity enhancement training vector, and the entity enhancement training vector includes multiple entity enhancement sub-vectors.

[0215] In some embodiments, for each entity enhancer vector, the server can determine the target entity enhancer vector of the entity enhancer vector among other entity enhancer vectors, and determine the entity positive sample pair through the entity enhancer vector and the target entity enhancer vector; the server obtains the entity negative sample pair in the training subgraph, and the server determines the knowledge retrieval loss value based on the entity positive sample pair and the entity negative sample pair.

[0216] In some embodiments, step 1010 includes: obtaining an entity positive sample pair in the training subgraph, and determining a first score of the entity positive sample pair based on the entity enhanced training vector; obtaining an entity negative sample pair in the training subgraph, and determining a second score of the entity negative sample pair based on the entity enhanced training vector; the entity negative sample pair includes two training entities that do not have an entity relationship in the training subgraph; and determining a knowledge retrieval loss value based on the first score and the second score.

[0217] Among them, the two entities included in the entity positive sample pair have an entity relationship in the training subgraph, and the two entities included in the entity negative sample pair do not have an entity relationship in the training subgraph.

[0218] In some embodiments, the server obtains positive sample pairs of entities having entity relationships in the training subgraph, and for each entity positive sample pair, obtains the entity enhancement subvectors of the two entities in the entity positive sample pair in the entity enhancement training vector, and determines the first score of the entity positive sample pair based on the entity enhancement subvectors of the two entities in the entity positive sample pair.

[0219] The server obtains entity negative sample pairs in the training subgraph in which no entity relationship exists. For each entity negative sample pair, the server obtains the entity enhancement subvectors of the two entities in the entity negative sample pair from the entity enhancement training vector, and determines the second score of the entity positive sample pair based on the entity enhancement subvectors of the two entities in the entity negative sample pair.

[0220] Exemplarily, the entities included in the training subgraph are: E1, E2, E3, E4, E5, wherein there is no entity relationship between E2 and E3, and there is no entity relationship between E4 and E5, and then the entity negative sample pairs include {E2, E3} and {E4, E5}, and the second score of {E2, E3} is determined according to the entity enhancer vectors of E2 and E3 respectively, and the second score of {E4, E5} is determined according to the entity enhancer vectors of E4 and E5 respectively.

[0221] In some embodiments, the knowledge retrieval loss value is determined according to the first score and the second score, and reference may be made to formula (8).

[0222] Formula (8):

[0223] in, is the knowledge retrieval loss value, (h, t) is the entity positive sample pair, φ r (h,t) is the first score, (h′,t′) is the entity negative sample pair, φ r (h′,t′+γ) is the second score, φ r (·) is the scoring function, (h,r,t)∈S indicates that there is an entity relationship r between entity h and entity t in the training subgraph S, and γ is a preset parameter.

[0224] In some embodiments, a knowledge retrieval loss value is determined based on the knowledge enhanced training vector and the training subgraph, including: determining an entity sample pair based on the training subgraph, and determining a score of the entity sample pair based on the entity enhanced training vector; when the score belongs to the positive sample interval, the entity sample pair is regarded as an entity positive sample pair; when the score does not belong to the positive sample interval, the entity sample pair is regarded as an entity negative sample pair.

[0225] The entity sample pairs are determined according to the training subgraph, and the entities included in the training subgraph are combined in pairs to obtain entity sample pairs.

[0226] In the above embodiment, entity positive sample pairs and entity negative sample pairs are determined by training subgraphs, and the first score of the entity positive sample pairs and the second score of the entity negative sample pairs are determined according to the entity enhanced training vector, so that the knowledge retrieval loss value can be used to reflect the gap between training entities with entity relationships and training entities without entity relationships, thereby improving the accuracy of the knowledge retrieval loss value.

[0227] Step 1012: Based on the visual loss value, the language loss value, and the knowledge retrieval loss value, the parameters of the feature extraction model, the knowledge retrieval model, and the knowledge enhancement model are adjusted to obtain an enhanced vector extraction model.

[0228] Among them, the enhanced vector extraction model includes: a trained feature extraction model, a trained knowledge retrieval model and a trained knowledge enhancement model, and the trained feature extraction model includes a trained image feature extraction model and a trained text feature extraction model.

[0229] In some embodiments, the server superimposes the visual loss value, the language loss value, and the knowledge retrieval loss value to obtain a total loss value, and adjusts the parameters of the pre-trained feature extraction model, the pre-trained knowledge retrieval model, and the pre-trained knowledge enhancement model according to the total loss value until the pre-trained feature extraction model, the pre-trained knowledge retrieval model, and the pre-trained knowledge enhancement model converge to obtain an enhanced vector extraction model.

[0230] In practical applications, the AdamW optimizer can be used to adjust the parameters of the pre-trained feature extraction model, the pre-trained knowledge retrieval model, and the pre-trained knowledge enhancement model according to the preset learning rate and the preset weight decay. The AdamW optimizer is used to update the parameters of the neural network based on the gradient to minimize the total loss value. The preset learning rate can be set according to actual needs, such as the preset learning rate can be 5e-5, and the preset weight decay can be set according to actual needs, such as the preset weight decay can be 0.02.

[0231] Step 1014, determining a recommendation model based on the enhancement vector extraction model and the classification model; the recommendation model is used to extract the knowledge enhancement vector based on the media data, descriptive text and knowledge graph, and determine the interest type based on the knowledge enhancement vector, so as to obtain the target media data based on the interest type and recommend the target media data to the target object.

[0232] Among them, the recommendation model includes an enhanced vector extraction model and a classification model.

[0233] In some embodiments, the enhanced vector extraction model is connected to a trained classification model to generate a recommendation model. In practical applications, the media data currently being viewed by the target subject, the descriptive text of the media data, and the knowledge graph are input into the recommendation model. The enhancement vector extraction model of the recommendation model determines the knowledge enhancement vector, and the classification model of the recommendation model outputs the interest type of the knowledge enhancement vector. This allows the target media data to be acquired based on the interest type and recommended to the target subject.

[0234] It should be noted that the enhanced vector extraction model can be a pre-trained model. After obtaining the enhanced vector extraction model through pre-training, the enhanced vector extraction model can be used in the downstream task of media data recommendation. The parameters of the enhanced vector extraction model are fixed, and the parameters of the initial classification model are adjusted to obtain the trained classification model. The recommendation model is determined based on the enhanced vector extraction model and the trained classification model.

[0235] In the processing method of the above-mentioned recommendation model, the first media training vector and the first text training vector are extracted by the feature extraction model, the entity training vector is retrieved in the knowledge graph according to the first media training vector, and the first media training vector, the first text training vector and the entity training vector are subjected to feature fusion processing to obtain the knowledge enhancement training vector, that is, the related entities with the first media sample data are retrieved in the knowledge graph, the entity training vector of the related entity, the first media training vector and the first text quantity are fused to realize multimodal data interaction, enhance the representation of the first media sample data, the first text sample and the related entities, and improve the quality of the knowledge enhancement training vector; the combined visual loss value, the language loss value and the The parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model are adjusted by using the value and the knowledge retrieval loss value, so that in the process of parameter adjustment, the content information of the first media sample data and the first text sample can be learned, and the entity information related to the first media sample data can also be learned, thereby improving the quality of the trained enhanced vector extraction model, and then improving the quality of the recommendation model including the enhanced vector extraction model. Based on the recommendation model, it is determined that the target media data is recommended to the target object, which can improve the media recommendation effect. In this way, the target object does not need to perform manual search to obtain and browse the media data of interest, reducing the resource consumption caused by manual search and improving resource utilization.

[0236] In some embodiments, the first sample media data and the first sample text belong to a sample set, and the sample set also includes second sample media data and second sample text; the processing method of the recommendation model also includes: performing feature extraction on the second sample media data and the second sample text to obtain a second media training vector and a second text training vector; determining an image-text contrast loss value based on the second media training vector, the second text training vector, the first media training vector and the first text training vector; adjusting the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model based on the visual loss value, the language loss value and the knowledge retrieval loss value to obtain an enhanced vector extraction model, including: adjusting the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model based on the visual loss value, the language loss value, the knowledge retrieval loss value and the image-text contrast loss value to obtain an enhanced vector extraction model.

[0237] The image-text comparison loss value may reflect the similarity between the sample media data and the sample text of the sample media data, and the difference between the similarity between the sample media data and the sample text of other sample media data.

[0238] In some embodiments, the server may perform feature extraction on the second sample media data and the second sample text through a feature extraction model to obtain a second media training vector and a second text training vector; the server determines a first candidate similarity based on the first media training vector and the first text training vector, determines a second candidate similarity based on the first media training vector and the second text training vector, determines a third candidate similarity based on the first text training vector and the second media training vector, and determines an image-text contrast loss value based on the first candidate similarity, the second candidate similarity, and the third candidate similarity.

[0239] In some embodiments, the server superimposes the visual loss value, language loss value, knowledge retrieval loss value and image-text comparison loss value to obtain a total loss value, and adjusts the parameters of the pre-trained feature extraction model, the pre-trained knowledge retrieval model and the pre-trained knowledge enhancement model according to the total loss value until the pre-trained feature extraction model, the pre-trained knowledge retrieval model and the pre-trained knowledge enhancement model converge to obtain an enhanced vector extraction model.

[0240] In the above embodiment, the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model are adjusted jointly by the visual loss value, the language loss value, the knowledge retrieval loss value and the image-text contrast loss value, so that in the process of parameter adjustment, the content information of the first media sample data and the first text sample can be learned, the entity information related to the first media sample data can be learned, and the similar content between the first media sample data and the first text sample can be learned, thereby improving the quality of the trained enhanced vector extraction model, and then improving the quality of the recommendation model including the enhanced vector extraction model. Determining to recommend target media data to the target object based on the recommendation model can improve the media recommendation effect. In this way, the target object does not need to perform manual search to obtain and browse the media data of interest, thereby reducing the resource consumption caused by manual search and improving resource utilization.

[0241] In some embodiments, an image-text contrast loss value is determined based on the second media training vector, the second text training vector, the first media training vector, and the first text training vector, including: determining a first similarity based on the first media training vector and the second text training vector; determining a second similarity based on the first text training vector and the second media training vector; and determining an image-text contrast loss value based on the first similarity, the second similarity, the first similarity label, and the second similarity label.

[0242] The first similarity label may be the similarity between the first media training vector and the first text training vector, and the second similarity label may be the similarity between the first text training vector and the first media training vector.

[0243] For example, the first similarity tag may be y i2t =S(I*T), the second similarity label can be y t2i =S(T*I), where I is the first media training vector, T is the first text training vector, S(·) is the similarity calculation function, and y i2t It can be y t2i The transposed form of .

[0244] In some embodiments, the sample set includes multiple second media sample data, and accordingly, the sample set includes multiple second sample texts, and thus the number of second media training vectors is multiple, and the number of second text training vectors is multiple; for the first media training vector, the server determines the first similarity between the first media training vector and the multiple second text training vectors respectively, and for the first text training vector, the server determines the second similarity between the first text training vector and the multiple second media training vectors respectively.

[0245] The server determines a first target similarity based on first similarities between the first media training vector and the plurality of second text training vectors, and determines a second target similarity based on second similarities between the first text training vector and the plurality of second media training vectors.

[0246] For example, as shown in formula (9).

[0247] Formula (9):

[0248] Among them, s (i2t) is the first target similarity, s i ,T k > is the first media training vector I i and the kth second text training vector T k The first similarity between them, K is the number of sample texts in the sample set, when k=i, T i is the first media training vector I i The first text training vector.

[0249] For example, as shown in formula (10).

[0250] Formula (10):

[0251] Among them, s (t2i) is the second target similarity, s i ,T i > is the first text training vector T i and the kth second media training vector I k The second similarity between them, K is the number of second media sample data in the sample set, when k=i, I i is the first text training vector T i The corresponding first media training vector.

[0252] The server can calculate the loss value between the first target similarity and the first similarity label through the cross-entropy loss function, calculate the loss value between the second target similarity and the second similarity label through the cross-entropy loss function, and determine the image-text comparison loss value based on the loss value between the first target similarity and the first similarity label, and the loss value between the second target similarity and the second similarity label; the image-text comparison loss value is added to the model training process, and through training, the similarity between the extracted media data with corresponding relationships and the representation vectors of the description text is greater, and the similarity between the media data without corresponding relationships and the representation vectors of the description text is smaller.

[0253] For example, as shown in formula (11).

[0254] ​​Formula (11):

[0255] in, is the image-text contrast loss value, y i2t is the first similarity label, s i2t is the first target similarity, y t2i is the second similarity label, s t2i is the second target similarity.

[0256] In the above embodiment, the image contrast loss value is determined based on the first similarity, the second similarity, the first similarity label and the second similarity label, and the image contrast loss value is added to the process of adjusting the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model to train the enhanced vector extraction model, thereby improving the quality of the enhanced vector extraction model.

[0257] In some embodiments, as shown in FIG13 , the training process of the enhanced vector extraction model includes:

[0258] Dividing the initial sample image into N sample images, performing mask processing on the N sample images using a visual mask model to obtain first sample media data; processing the first sample media data using a pre-trained image feature extraction model to obtain a first media training vector;

[0259] Perform word segmentation on the initial sample and obtain N t words, and N t The words are masked and a start tag is added before the masked sample text to obtain a first sample text; the first sample text is processed by a pre-trained text feature extraction model to obtain a first text training vector;

[0260] Inputting the knowledge graph and the first media training vector into a pre-trained knowledge retrieval model, and determining a training subgraph and an entity training vector of the training subgraph through the pre-trained knowledge retrieval model; the pre-trained knowledge retrieval model includes: a pre-trained retrieval submodel, a pre-trained subgraph construction network, and a pre-trained graph neural network;

[0261] Splicing the first media training vector, the first text training vector, and the entity training vector, and adding a separator element between the first media training vector and the first text training vector, and between the first text training vector and the entity training vector, to obtain a training splicing vector;

[0262] The training concatenated vectors are subjected to feature fusion by a pre-training knowledge enhancement model to obtain knowledge enhancement training vectors. The pre-training knowledge enhancement model includes: a pre-training regularization layer, a pre-training encoder, and a pre-training feedforward layer. The knowledge enhancement training vectors include media enhancement training vectors, text enhancement training vectors, and entity enhancement training vectors.

[0263] According to the knowledge enhanced training vector and sample labels, the visual loss value and the language loss value are determined. According to the knowledge enhanced training vector and the training subgraph, the knowledge retrieval loss value is determined. Based on the visual loss value, the language loss value and the knowledge retrieval loss value, the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model are adjusted to obtain the enhanced vector extraction model.

[0264] In some embodiments, as shown in FIG14 , a method for processing a recommendation model includes:

[0265] Step 1401: extracting a first media training vector and a first text training vector from first sample media data and a first sample text of the first sample media data based on a feature extraction model;

[0266] Step 1402: Based on the knowledge retrieval model, perform knowledge retrieval processing on the first media training vector and the knowledge graph to obtain a training subgraph of the first sample media data, and determine an entity training vector of the training subgraph;

[0267] Step 1403: Based on the knowledge enhancement model, feature fusion processing is performed on the first media training vector, the first text training vector, and the entity training vector to obtain a knowledge enhancement training vector; the knowledge enhancement training vector includes the media enhancement training vector, the text enhancement training vector, and the entity enhancement training vector.

[0268] Step 1404: Obtain a masked sub-image enhancement vector from the media enhancement training vector, and determine a visual loss value based on the masked sub-image enhancement vector and the masked sub-image label; perform classification processing on the text enhancement training vector to obtain a masked word prediction probability, and determine a language loss value based on the masked word prediction probability and the masked word label;

[0269] Step 1405: Obtain a positive entity pair in the training subgraph and determine a first score for the positive entity pair based on the entity augmented training vector; obtain a negative entity pair in the training subgraph and determine a second score for the negative entity pair based on the entity augmented training vector; the negative entity pair includes two training entities that do not have an entity relationship in the training subgraph; determine a knowledge retrieval loss value based on the first score and the second score;

[0270] Step 1406: Perform feature extraction on the second sample media data and the second sample text to obtain a second media training vector and a second text training vector; determine a first similarity based on the first media training vector and the second text training vector; determine a second similarity based on the first text training vector and the second media training vector; and determine an image-text contrast loss value based on the first similarity, the second similarity, the first similarity label, and the second similarity label.

[0271] Step 1407: Based on the visual loss value, the language loss value, the knowledge retrieval loss value, and the image-text contrast loss value, the parameters of the feature extraction model, the knowledge retrieval model, and the knowledge enhancement model are adjusted to obtain an enhanced vector extraction model.

[0272] Step 1408, determining a recommendation model based on the enhancement vector extraction model and the classification model; the recommendation model is used to extract the knowledge enhancement vector based on the media data, descriptive text and knowledge graph, and determine the interest type based on the knowledge enhancement vector, so as to obtain the target media data based on the interest type and recommend the target media data to the target object.

[0273] In some embodiments, the quality of the enhanced vector extraction model is tested and compared with other models in related technologies. The comparison results are shown in Table 1.

[0274] Table 1

[0275] Among them, KAT (Knowledge Augmented Transformer, a knowledge conversion model); REVIVE (a visual question answering model); ALBEF (Align before Fuse: Vision and Language Representation Learning with Momentum Distillation, a visual language representation model based on momentum distillation); BLIP (a visual language multimodal model); REVEAL (Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory, a multi-source and multimodal visual language pre-training model); VL-BERT is a general visual language model; UNITER (UNiversal Image-TExt Representation Learning, a multimodal pre-training model); OSCAR (Object-Semantics Aligned Pre-training for Vision-Language Tasks, a multimodal pre-training model); SimVLM, a simple visual language pre-training model under weak supervision.

[0276] Wiki data refers to Wikipedia data. #image12M refers to 12 million images, #image129M refers to 129 million images, and other content related to #image is similar. CC12M refers to 12 million image-text pairs, and WIT (Wikipedia-based Image Text Dataset-GitHub) is a Wikipedia-based image-text dataset.

[0277] Integrating knowledge-based downstream tasks, the enhanced vector extraction model achieved improved accuracy compared to KAT, REVIVE, ALBEFF, BLIP, and REVEAL on the OK-VQA (Outside Knowledge-Visual Question Answering) dataset, and achieved significant relative accuracy gains compared to the currently advanced REVIVE and BLIP. Compared to REVEAL, the enhanced vector extraction model demonstrated superior performance while utilizing fewer knowledge graph resources. On the AOK-VQA dataset, the enhanced vector extraction model also achieved improved accuracy compared to ALBEF, BLIP, and REVEAL.

[0278] Combined with downstream tasks of general visual language, the enhanced vector extraction model also shows improved accuracy compared to VL-BERT, UNITER, OSCAR, and ALBEF in the VQA-v2 (Visual Question Answering-v2) dataset and with a basic amount of data training. In the VQA-v2 dataset, and with a large amount of data training, the enhanced vector extraction model is also quite competitive.

[0279] Combined with downstream tasks of general visual language, the enhanced vector extraction model also shows improved accuracy compared to VL-BERT, UNITER, OSCAR and ALBEF in the SNLI-VE (Stanford Natural Language Inference-Visual Entailment) dataset.

[0280] In some embodiments, the ability of the enhanced vector extraction model to retrieve entities is tested, and the enhanced vector extraction model is compared with the existing multimodal entity retrieval model. The comparison results are shown in Table 2.

[0281] Table 2

[0282] Among them, ViT+BERT (Vision Transformer+Bidirectional Encoder Representation from Transformers) is a visual transformer plus language representation model, ResNet is a residual network, and CLIP uses text as a supervisory signal to train a transferable visual model. It can be seen that among the scores of each model under six indicators, the enhanced vector extraction model has better scores in five indicators than the existing multimodal entity retrieval model.

[0283] In the processing method of the above-mentioned recommendation model, the first media training vector and the first text training vector are extracted by the feature extraction model, the entity training vector is retrieved in the knowledge graph according to the first media training vector, and the first media training vector, the first text training vector and the entity training vector are subjected to feature fusion processing to obtain the knowledge enhancement training vector, that is, the related entities with the first media sample data are retrieved in the knowledge graph, the entity training vector of the related entity, the first media training vector and the first text quantity are fused to realize multimodal data interaction, enhance the representation of the first media sample data, the first text sample and the related entities, and improve the quality of the knowledge enhancement training vector; the combined visual loss value, the language loss value and the The parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model are adjusted by using the value and the knowledge retrieval loss value, so that in the process of parameter adjustment, the content information of the first media sample data and the first text sample can be learned, and the entity information related to the first media sample data can also be learned, thereby improving the quality of the trained enhanced vector extraction model, and then improving the quality of the recommendation model including the enhanced vector extraction model. Based on the recommendation model, it is determined that the target media data is recommended to the target object, which can improve the media recommendation effect. In this way, the target object does not need to perform manual search to obtain and browse the media data of interest, reducing the resource consumption caused by manual search and improving resource utilization.

[0284] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0285] Based on the same inventive concept, embodiments of the present application also provide a media data recommendation device for implementing the aforementioned media data recommendation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more media data recommendation device embodiments provided below can be referenced to the limitations of the media data recommendation method described above.

[0286] In some embodiments, as shown in FIG15 , a media data recommendation apparatus is provided, comprising: a vector extraction module 1501 , a first knowledge retrieval module 1502 , a first fusion module 1503 , and a recommendation module 1504 , wherein:

[0287] A vector extraction module 1501 is configured to extract a media representation vector and a text representation vector from the media data and the description text of the media data;

[0288] A first knowledge retrieval module 1502 is configured to perform knowledge retrieval in the knowledge graph based on the media representation vector, obtain an entity subgraph of the media data, and determine the entity representation vector of the entity subgraph;

[0289] A first fusion module 1503 is configured to perform feature fusion processing on the media representation vector, the text representation vector, and the entity representation vector to obtain a knowledge enhancement vector;

[0290] The recommendation module 1504 is configured to obtain target media data based on the knowledge enhancement vector and recommend the target media data to the target object.

[0291] In some embodiments, the vector extraction module 1501 includes: a media representation vector extraction unit and a text representation vector extraction unit;

[0292] a media representation vector extraction unit, configured to extract features from media data using an image feature extraction model to obtain a media representation vector;

[0293] The text representation vector extraction unit is used to extract a text representation vector from the description text of the media data through a text feature extraction model.

[0294] In some embodiments, the media representation vector extraction unit is also used to extract features of multiple image frames in the video through an image feature extraction model to obtain a media representation vector when the media data is a video; and to extract features of multiple image blocks of the image through an image feature extraction model to obtain a media representation vector when the media data is an image.

[0295] In some embodiments, the first knowledge retrieval module 1502 includes:

[0296] A target entity determination unit, configured to retrieve multiple target entities related to the media data in the knowledge graph based on the media representation vector;

[0297] An entity subgraph determining unit, configured to determine an entity subgraph of media data based on multiple target entities and a knowledge graph;

[0298] The entity representation vector determination unit is used to extract features of multiple entities in the entity subgraph to obtain entity representation vectors.

[0299] In some embodiments, the media representation vector includes at least two image sub-representation vectors; the target entity determination unit is further used to obtain multiple initial entity vectors in the knowledge graph; based on the multiple initial entity vectors and at least two image sub-representation vectors, candidate entities are retrieved in the knowledge graph; among the candidate entities, multiple target entities related to the media data are selected.

[0300] In some embodiments, the target entity determination unit also includes a candidate entity retrieval subunit, which is used to determine a correlation set of at least two image sub-representation vectors based on multiple initial entity vectors and at least two image sub-representation vectors; the correlation set includes the correlation between the image sub-representation vectors and the initial entity vectors respectively; based on the correlation set, candidate entities related to the at least two image sub-representation vectors are selected in the knowledge graph.

[0301] In some embodiments, the entity subgraph determination unit is further used to determine the neighbor nodes of each target entity in the knowledge graph; determine the extended entity based on multiple target entities and the neighbor nodes of each target entity, and determine the relationship between the extended entities in the knowledge graph; determine the entity subgraph of the media data based on the extended entity and the relationship between the extended entities. In some embodiments, the target entity determination unit is further used to retrieve multiple target entities related to the media data in the knowledge graph based on the media representation vector through the retrieval submodel of the knowledge retrieval model; the entity subgraph determination unit is further used to construct a network through the subgraph of the knowledge retrieval model, and determine the entity subgraph corresponding to the media data based on the multiple target entities and the knowledge graph; the entity representation vector determination unit is further used to extract features of multiple entities in the entity subgraph through the graph neural network of the knowledge retrieval model to obtain the entity representation vector.

[0302] In some embodiments, the first fusion module 1503 is also used to splice the media representation vector, the text representation vector and the entity representation vector, and in the splicing process, add a separation element between the media representation vector and the text representation vector, and add a separation element between the text representation vector and the entity representation vector to obtain a spliced ​​vector; the spliced ​​vector is subjected to feature fusion processing through the knowledge enhancement model to obtain a knowledge enhancement vector.

[0303] In some embodiments, the recommendation module 1504 is further configured to classify the knowledge enhancement vector to obtain the interest type of the target object; obtain target media data based on the interest type, and recommend the target media data to the target object.

[0304] Each module in the aforementioned media data recommendation apparatus may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in the form of hardware, or may be stored in a memory within the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0305] Based on the same inventive concept, embodiments of the present application also provide a media data recommendation device for implementing the aforementioned media data recommendation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more media data recommendation device embodiments provided below can be referenced to the limitations of the media data recommendation method described above.

[0306] In some embodiments, as shown in FIG16 , a device for processing a recommendation model is provided, comprising: a training vector extraction module 1601, a second knowledge retrieval module 1602, a second fusion module 1603, a first loss value determination module 1604, a second loss value determination module 1605, a parameter adjustment module 1606, and a recommendation model determination module 1607; wherein,

[0307] A training vector extraction module 1601 is configured to extract a first media training vector and a first text training vector from the first sample media data and the first sample text of the first sample media data based on a feature extraction model;

[0308] A second knowledge retrieval module 1602 is configured to perform knowledge retrieval processing on the first media training vector and the knowledge graph based on the knowledge retrieval model to obtain a training subgraph of the first sample media data and determine an entity training vector of the training subgraph;

[0309] The second fusion module 1603 is configured to perform feature fusion processing on the first media training vector, the first text training vector, and the entity training vector based on the knowledge enhancement model to obtain a knowledge enhancement training vector;

[0310] A first loss value determination module 1604 is configured to determine a visual loss value and a language loss value based on the knowledge-enhanced training vector and the sample labels;

[0311] A second loss value determination module 1605 is configured to determine a knowledge retrieval loss value based on the knowledge enhancement training vector and the training subgraph;

[0312] A parameter adjustment module 1606 is configured to adjust parameters of the feature extraction model, the knowledge retrieval model, and the knowledge enhancement model based on the visual loss value, the language loss value, and the knowledge retrieval loss value to obtain an enhanced vector extraction model;

[0313] The recommendation model determination module 1607 is used to determine the recommendation model based on the enhancement vector extraction model and the classification model; the recommendation model is used to extract the knowledge enhancement vector based on the media data, descriptive text and knowledge graph, and determine the interest type based on the knowledge enhancement vector, so as to obtain the target media data based on the interest type and recommend the target media data to the target object.

[0314] In some embodiments, the knowledge enhancement training vector includes a media enhancement training vector and a text enhancement training vector; the sample label includes a shielded sub-image label and a shielded word label; the first loss value determination module 1604 is used to obtain the shielded sub-image enhancement vector in the media enhancement training vector, and determine the visual loss value based on the shielded sub-image enhancement vector and the shielded sub-image label; the text enhancement training vector is classified to obtain the shielded word prediction probability, and the language loss value is determined based on the shielded word prediction probability and the shielded word label.

[0315] In some embodiments, the second loss value determination module 1605 is further used to obtain entity positive sample pairs in the training subgraph, and determine a first score of the entity positive sample pairs based on the entity enhancement training vector; obtain entity negative sample pairs in the training subgraph, and determine a second score of the entity negative sample pairs based on the entity enhancement training vector; the entity negative sample pairs include two training entities that do not have an entity relationship in the training subgraph; and determine the knowledge retrieval loss value based on the first score and the second score.

[0316] In some embodiments, the first sample media data and the first sample text belong to a sample set, and the sample set also includes second sample media data and second sample text; the processing device of the recommendation model further includes a third loss value determination module for performing feature extraction on the second sample media data and the second sample text to obtain a second media training vector and a second text training vector; and determining an image-text contrast loss value based on the second media training vector, the second text training vector, the first media training vector, and the first text training vector;

[0317] Accordingly, the parameter adjustment module 1606 is used to adjust the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model based on the visual loss value, the language loss value, the knowledge retrieval loss value and the image-text contrast loss value to obtain the enhanced vector extraction model.

[0318] In some embodiments, the third loss value determination module includes an image-text comparison loss value determination unit, which is used to determine the first similarity based on the first media training vector and the second text training vector; determine the second similarity based on the first text training vector and the second media training vector; and determine the image-text comparison loss value based on the first similarity, the second similarity, the first similarity label and the second similarity label.

[0319] Each module in the aforementioned recommendation model processing device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0320] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 17. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to recommend models, target media data, and sample sets. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for recommending media data or a method for processing a recommendation model.

[0321] Those skilled in the art will understand that the structure shown in Figure 17 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0322] In one embodiment, a computer device is provided, comprising a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the computer device implements the aforementioned method for recommending media data or the method for processing a recommendation model. In another embodiment, a computer-readable storage medium is provided, storing the computer program. When the processor executes the computer program, the computer device implements the aforementioned method for recommending media data or the method for processing a recommendation model.

[0323] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the above-mentioned method for recommending media data or the method for processing a recommendation model. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0324] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0325] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0326] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for recommending media data, wherein: Executed by a server, the method includes: Extracting a media representation vector and a text representation vector from the media data and a description text of the media data; Performing knowledge retrieval in the knowledge graph according to the media representation vector to obtain an entity subgraph of the media data, and determining an entity representation vector of the entity subgraph; Performing feature fusion processing on the media representation vector, the text representation vector, and the entity representation vector to obtain a knowledge enhancement vector; and Target media data is acquired based on the knowledge enhancement vector, and the target media data is recommended to a target object.

2. The method according to claim 1, wherein: The step of extracting a media representation vector and a text representation vector from the media data and the description text of the media data includes: Extract features from media data using an image feature extraction model to obtain a media representation vector; and A text representation vector is extracted from the description text of the media data through a text feature extraction model.

3. The method according to claim 2, wherein: The step of extracting features from media data using an image feature extraction model to obtain a media representation vector includes: When the media data is a video, extracting features of a plurality of image frames in the video by using an image feature extraction model to obtain a media representation vector; and When the media data is an image, the image feature extraction model is used to extract features from multiple image blocks of the image to obtain a media representation vector.

4. The method according to any one of claims 1 to 3, wherein: The performing knowledge retrieval in the knowledge graph according to the media representation vector to obtain the entity subgraph of the media data and determining the entity representation vector of the entity subgraph includes: Retrieving multiple target entities related to the media data in the knowledge graph based on the media representation vector; Determining an entity subgraph of the media data based on the multiple target entities and the knowledge graph; and Features of multiple entities in the entity subgraph are extracted to obtain entity representation vectors.

5. The method according to claim 4, wherein: The media representation vector includes at least two image sub-representation vectors; and the retrieving multiple target entities related to the media data in the knowledge graph based on the media representation vector includes: Get the initial entity vectors of multiple entities in the knowledge graph; Retrieving candidate entities in the knowledge graph based on the multiple initial entity vectors and the at least two image sub-representation vectors; and Among the candidate entities, multiple target entities related to the media data are selected.

6. The method according to claim 5, wherein: The retrieving candidate entities in the knowledge graph according to the plurality of initial entity vectors and the at least two image sub-representation vectors comprises: Determining a correlation set of the at least two image sub-representation vectors according to the plurality of initial entity vectors and the at least two image sub-representation vectors; the correlation set includes correlations between the image sub-representation vectors and the initial entity vectors respectively; and Based on the relevance set, candidate entities related to the at least two image sub-representation vectors are selected in the knowledge graph.

7. The method according to claim 4, wherein: The determining, based on the multiple target entities and the knowledge graph, an entity subgraph of the media data includes: Determining neighbor nodes of each of the target entities in the knowledge graph; Determine an extended entity according to the multiple target entities and the neighboring nodes of each of the target entities; Determining the relationship between the extended entities in the knowledge graph; and An entity subgraph of the media data is determined according to the extended entities and the relationship between the extended entities.

8. The method according to claim 4, wherein: The retrieving a plurality of target entities related to the media data in the knowledge graph based on the media representation vector includes: Retrieving multiple target entities related to the media data in the knowledge graph based on the media representation vector through a retrieval sub-model of the knowledge retrieval model; The determining, based on the target entity and the knowledge graph, an entity subgraph corresponding to the media data includes: Constructing a network through a subgraph of the knowledge retrieval model, and determining an entity subgraph corresponding to the media data based on the plurality of target entities and the knowledge graph; and The extracting features of the multiple entities in the entity subgraph to obtain an entity representation vector comprises: The features of multiple entities in the entity subgraph are extracted through the graph neural network of the knowledge retrieval model to obtain an entity representation vector.

9. The method according to any one of claims 1 to 8, wherein: The performing feature fusion processing on the media representation vector, the text representation vector and the entity representation vector to obtain a knowledge enhancement vector includes: splicing the media representation vector, the text representation vector and the entity representation vector, and in the splicing process, adding a separation element between the media representation vector and the text representation vector, and adding a separation element between the text representation vector and the entity representation vector, to obtain a splicing vector; and The concatenated vector is subjected to feature fusion processing through a knowledge enhancement model to obtain a knowledge enhancement vector.

10. The method according to any one of claims 1 to 9, wherein: The acquiring target media data based on the knowledge enhancement vector and recommending the target media data to the target object includes: Classifying the knowledge enhancement vector to obtain the interest type of the target object; and Target media data is acquired according to the interest type, and the target media data is recommended to the target object.

11. A method for processing a recommendation model, wherein: The method comprises: Extracting a first media training vector and a first text training vector from the first sample media data and the first sample text of the first sample media data based on the feature extraction model; Based on the knowledge retrieval model, perform knowledge retrieval processing on the first media training vector and the knowledge graph to obtain a training subgraph of the first sample media data, and determine an entity training vector of the training subgraph; Based on the knowledge enhancement model, feature fusion processing is performed on the first media training vector, the first text training vector, and the entity training vector to obtain a knowledge enhancement training vector; Enhance the training vector and the sample label according to the knowledge, and determine the visual loss value and the language loss value; Determining a knowledge retrieval loss value based on the knowledge enhanced training vector and the training subgraph; Based on the visual loss value, the language loss value and the knowledge retrieval loss value, adjusting the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model to obtain an enhanced vector extraction model; and A recommendation model is determined based on the enhancement vector extraction model and the classification model; the recommendation model is used to extract knowledge enhancement vectors based on media data, description text and knowledge graph, and determine interest types based on the knowledge enhancement vectors, so as to obtain target media data based on the interest type and recommend the target media data to a target object.

12. The method according to claim 11, wherein: The knowledge enhancement training vector includes a media enhancement training vector and a text enhancement training vector; the sample label includes a shielded sub-image label and a shielded word label; Determining the visual loss value and the language loss value based on the knowledge enhanced training vector and the sample label includes: Obtaining a shielded sub-image enhancement vector in the media enhancement training vector, and determining a visual loss value according to the shielded sub-image enhancement vector and the shielded sub-image label; and The text enhancement training vector is classified to obtain a blocked word prediction probability, and a language loss value is determined based on the blocked word prediction probability and the blocked word label.

13. The method according to claim 11, wherein: The knowledge enhancement training vector also includes an entity enhancement training vector; The step of determining the knowledge retrieval loss value based on the knowledge enhanced training vector and the training subgraph includes: Acquire entity positive sample pairs in the training subgraph, and determine a first score of the entity positive sample pairs according to the entity enhanced training vector; Acquire an entity negative sample pair in the training subgraph, and determine a second score of the entity negative sample pair according to the entity enhanced training vector; the entity negative sample pair includes two training entities that do not have an entity relationship in the training subgraph; and A knowledge retrieval loss value is determined according to the first score and the second score.

14. The method according to any one of claims 11 to 13, wherein: The first sample media data and the first sample text belong to a sample set, and the sample set also includes second sample media data and second sample text; the method further includes: Performing feature extraction on the second sample media data and the second sample text to obtain a second media training vector and a second text training vector; Determining an image-text contrast loss value according to the second media training vector, the second text training vector, the first media training vector, and the first text training vector; and The adjusting the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model based on the visual loss value, the language loss value and the knowledge retrieval loss value to obtain the enhanced vector extraction model includes: Based on the visual loss value, the language loss value, the knowledge retrieval loss value and the image-text contrast loss value, the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model are adjusted to obtain an enhanced vector extraction model.

15. The method according to claim 14, wherein: The step of determining the image-text contrast loss value according to the second media training vector, the second text training vector, the first media training vector, and the first text training vector includes: determining a first similarity based on the first media training vector and the second text training vector; determining a second similarity based on the first text training vector and the second media training vector; and An image-text comparison loss value is determined according to the first similarity, the second similarity, the first similarity label, and the second similarity label.

16. A device for recommending media data, wherein: The device comprises: A vector extraction module, used to extract a media representation vector and a text representation vector from the media data and the description text of the media data; A first knowledge retrieval module is used to perform knowledge retrieval in the knowledge graph according to the media representation vector, obtain an entity subgraph of the media data, and determine an entity representation vector of the entity subgraph; A first fusion module is used to perform feature fusion processing on the media representation vector, the text representation vector and the entity representation vector to obtain a knowledge enhancement vector; and A recommendation module is used to obtain target media data based on the knowledge enhancement vector and recommend the target media data to a target object.

17. A processing device for a recommendation model, wherein: The device comprises: A training vector extraction module, used to extract a first media training vector and a first text training vector from the first sample media data and the corresponding first sample text based on a feature extraction model; A second knowledge retrieval module is used to perform knowledge retrieval processing on the first media training vector and the knowledge graph based on the knowledge retrieval model to obtain a training subgraph of the first sample media data and determine an entity training vector of the training subgraph; A second fusion module is used to perform feature fusion processing on the first media training vector, the first text training vector and the entity training vector based on a knowledge enhancement model to obtain a knowledge enhancement training vector; The first loss value determination module is used to determine the visual loss value and Language loss value; A second loss value determination module, used to determine a knowledge retrieval loss value based on the knowledge enhancement training vector and the training subgraph; a parameter adjustment module, configured to adjust the parameters of the feature extraction model, the knowledge retrieval model and the knowledge enhancement model based on the visual loss value, the language loss value and the knowledge retrieval loss value to obtain an enhanced vector extraction model; and A recommendation model determination module is used to determine a recommendation model based on the enhancement vector extraction model and the classification model; the recommendation model is used to extract a knowledge enhancement vector based on media data, description text and knowledge graph, and determine an interest type based on the knowledge enhancement vector, so as to obtain target media data based on the interest type and recommend the target media data to a target object.

18. A computer device comprising a memory and one or more processors, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processors, the one or more processors execute the steps of the method according to any one of claims 1 to 15.

19. One or more non-volatile readable storage media having computer readable instructions stored thereon, wherein when the computer readable instructions are executed by the processor, the one or more processors implement the steps of any one of the methods of claims 1 to 15.

20. A computer-readable instruction product, comprising computer-readable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

Images