Model training methods, devices, multimedia recommendation methods, equipment, and storage media

Abstract

The training method for the multimedia feature extraction model in this application includes: determining multiple training samples from multiple multimedia resources, each training sample including an anchor multimedia resource, a first multimedia resource, and a second multimedia resource; the anchor multimedia resource is any one of the multiple multimedia resources; the first object set corresponding to the first multimedia resource intersects with the anchor object set corresponding to the anchor multimedia resource; the second object set corresponding to the second multimedia resource does not intersect with the anchor object set; sorting the multiple training samples to obtain a target sample matrix; inputting the target sample matrix into a preset multimedia feature extraction model for feature extraction processing to obtain a multimedia feature matrix; determining triplet loss information based on the multimedia feature matrix; and training the preset multimedia feature extraction model based on the triplet loss information to obtain the target multimedia feature extraction model. Introducing object information during the generation of training samples on the multimedia side can improve the accuracy of the target multimedia feature extraction model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to model training methods, apparatus, multimedia recommendation methods, devices, and storage media. Background Technology

[0002] In the field of multimedia recommendation (such as advertising recommendation), existing multimedia recommendation systems generate multimedia-side features and object-side features independently, that is, they are generated separately. Taking the commonly used multimodal feature embedding in multimedia-side features as an example, the corresponding multimedia-side feature generation model is generally trained using the multimodal materials of the multimedia itself as input (video frames, titles, audio, etc.). After the multimedia-side feature generation model converges, in the testing phase, a piece of multimodal material of the test multimedia is input, and the output result of a certain layer of the model is output according to business needs. This result is taken as a multidimensional feature embedding and used together with the object features generated by the object-side feature generation model as input to the subsequent models in the recommendation model to obtain the recommended multimedia.

[0003] It is evident that in existing multimedia recommendation technologies, the process of generating multimedia-side features is unrelated to the object-side, failing to fully explore object information and multimedia materials. This results in low accuracy of multimedia recommendations and an inability to push multimedia content that meets the object's needs. Summary of the Invention

[0004] In view of the above-mentioned technical problems, this application proposes a model training method, apparatus, multimedia recommendation method, device and storage medium.

[0005] According to one aspect of this application, a method for training a multimedia feature extraction model is provided, comprising:

[0006] Multiple training samples are determined from multiple multimedia resources. Each training sample includes an anchor multimedia resource, a first multimedia resource, and a second multimedia resource. The anchor multimedia resource is any one of the multiple multimedia resources. The first object set corresponding to the first multimedia resource intersects with the anchor object set corresponding to the anchor multimedia resource. The second object set corresponding to the second multimedia resource does not intersect with the anchor object set.

[0007] The multiple training samples are sorted to obtain the target sample matrix;

[0008] The target sample matrix is ​​input into a preset multimedia feature extraction model for feature extraction processing to obtain a multimedia feature matrix. The number of rows in the multimedia feature matrix is ​​the same as the number of rows in the target sample matrix.

[0009] Based on the multimedia feature matrix, the triplet loss information is determined;

[0010] The preset multimedia feature extraction model is trained based on the triplet loss information until the preset conditions are met, thus obtaining the target multimedia feature extraction model.

[0011] According to another aspect of this application, a multimedia recommendation method is provided, comprising:

[0012] Determine the object to be recommended and the target multimedia corresponding to the object to be recommended;

[0013] The target multimedia is input into the target multimedia feature extraction model, and feature extraction processing is performed on the target multimedia to obtain the multimedia features corresponding to the target multimedia. The target multimedia feature extraction model is obtained according to the training method described above.

[0014] The multimedia features corresponding to the target multimedia are input into the prediction model, and multiple preset multimedia are predicted based on the multimedia features corresponding to the target multimedia to obtain the prediction probability of each of the multiple preset multimedia.

[0015] Based on the predicted probabilities of each of the multiple preset multimedia resources, determine the multimedia resources to be recommended from the multiple preset multimedia resources;

[0016] The multimedia to be recommended is recommended to the target audience.

[0017] According to another aspect of this application, a multimedia feature extraction model training apparatus is provided, comprising:

[0018] A determination module is used to determine multiple training samples from multiple multimedia resources. Each training sample includes an anchor multimedia resource, a first multimedia resource, and a second multimedia resource. The anchor multimedia resource is any one of the multiple multimedia resources. The first object set corresponding to the first multimedia resource intersects with the anchor object set corresponding to the anchor multimedia resource. The second object set corresponding to the second multimedia resource does not intersect with the anchor object set.

[0019] The sorting module is used to sort the multiple training samples to obtain the target sample matrix;

[0020] The feature extraction module is used to input the target sample matrix into a preset multimedia feature extraction model, perform feature extraction processing, and obtain a multimedia feature matrix, wherein the number of rows in the multimedia feature matrix is ​​the same as the number of rows in the target sample matrix;

[0021] The triplet loss determination module is used to determine triplet loss information based on the multimedia feature matrix.

[0022] The model training module is used to train the preset multimedia feature extraction model based on the triplet loss information until the preset conditions are met, thereby obtaining the target multimedia feature extraction model.

[0023] According to another aspect of this application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the training method or multimedia recommendation method of the multimedia feature extraction model described above.

[0024] According to another aspect of this application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the training method of the multimedia feature extraction model or the multimedia recommendation method described above.

[0025] In the training samples of this application, each training sample includes an anchor multimedia, a first multimedia, and a second multimedia. The first object set corresponding to the first multimedia intersects with the anchor object set corresponding to the anchor multimedia, while the second object set corresponding to the second multimedia does not intersect with the anchor object set. It can be seen that there is an object-based correlation between the anchor multimedia and the first multimedia, but no object-based correlation between the anchor multimedia and the second multimedia. That is, the training samples incorporate object information. Model training based on such training samples can combine object information with multimedia materials, fully explore the relationship between object information and multimedia materials, obtain more accurate multimedia features, and thus improve the accuracy of subsequent multimedia recommendations. In addition, in the model training process, this application takes each multimedia as a whole as the unit, which can ensure the robustness of multimedia feature extraction and effectively adapt to different recommendation scenarios.

[0026] Other features and aspects of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0027] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0028] Figure 1 This diagram illustrates an application system provided according to an embodiment of the present application.

[0029] Figure 2 This diagram illustrates a flowchart of a training method for a multimedia feature extraction model according to an embodiment of this application.

[0030] Figure 3 The diagram illustrates a flowchart of a method for determining multiple training samples from multiple multimedia sources according to an embodiment of this application.

[0031] Figure 4 The diagram illustrates a method for sorting multiple training samples to obtain a target sample matrix according to an embodiment of this application.

[0032] Figure 5 The diagram illustrates a method according to an embodiment of this application for inputting a target sample matrix into a preset multimedia feature extraction model, performing feature extraction processing, and obtaining a multimedia feature matrix.

[0033] Figure 6 The diagram shows a flowchart of a method for determining triplet loss information based on a multimedia feature matrix according to an embodiment of this application.

[0034] Figure 7 The diagram illustrates a method for grouping multimedia features in a multimedia feature matrix to obtain a target feature matrix according to an embodiment of this application.

[0035] Figure 8 This diagram illustrates a flowchart of a training method for a multimedia feature extraction model according to another embodiment of this application.

[0036] Figure 9 This diagram illustrates a data processing method for training a multimedia feature extraction model according to an embodiment of this application.

[0037] Figure 10 The diagram shows a flowchart of a multimedia recommendation method according to an embodiment of this application.

[0038] Figure 11 This diagram illustrates a block diagram of a multimedia feature extraction model training apparatus according to an embodiment of the present application.

[0039] Figure 12 This diagram illustrates an electronic device for training a multimedia feature extraction model or for multimedia recommendation, according to an embodiment of this application. Detailed Implementation

[0040] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0041] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0042] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0043] Please see Figure 1 , Figure 1 This diagram illustrates an application system according to an embodiment of this application. The application system can be used for training the multimedia feature extraction model of this application. Figure 1 As shown, the application system may include at least server 01 and terminal 02.

[0044] In this embodiment of the application, server 01 can be used for multimedia resource processing, such as multimedia resource search processing. Server 01 may include an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0045] In this embodiment, terminal 02 can receive and display target multimedia resources. Terminal 02 may include physical devices such as smartphones, desktop computers, tablets, laptops, smart voice interaction devices, smart home appliances, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, in-vehicle terminals, and aircraft. Physical devices may also include software running on them, such as applications. In this embodiment, the operating system running on terminal 02 may include, but is not limited to, Android, iOS, Linux, and Windows.

[0046] In the embodiments described in this specification, the terminal 02 and the server 01 can be directly or indirectly connected through wired or wireless communication, and this application does not limit this connection.

[0047] In a specific embodiment, when server 02 is a distributed system, this distributed system can be a blockchain system. When the distributed system is a blockchain system, it can be formed by multiple nodes (any form of computing device connected to the network, such as servers or user terminals). These nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In a distributed system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer. Specifically, the functions of each node in the blockchain system may include:

[0048] Routing is a basic function of nodes, used to support communication between nodes.

[0049] In addition to routing capabilities, nodes can also have the following functions:

[0050] 2) Applications are deployed in the blockchain to implement specific business needs. They record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to a temporary block.

[0051] It should be noted that in the specific embodiments of this application, data related to user information is involved. When the following embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0052] This application utilizes machine learning (ML) technology for model training. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0053] Figure 2This document illustrates a flowchart of a training method for a multimedia feature extraction model according to an embodiment of this application. The multimedia feature extraction model can refer to a model used to extract multimedia features in a multimedia recommendation system. Multimedia recommendation systems may include advertising recommendation systems, short video recommendation systems, music recommendation systems, etc. A multimedia recommendation system can be a dual-tower model or a single-tower model; this disclosure does not limit its scope.

[0054] Combination Figures 2 to 9 The training method for the multimedia feature extraction model provided in the embodiments of this specification may include:

[0055] Step S11: Determine multiple training samples from multiple multimedia resources. Each training sample includes an anchor multimedia resource, a first multimedia resource, and a second multimedia resource. The anchor multimedia resource is any one of the multiple multimedia resources. The first object set corresponding to the first multimedia resource intersects with the anchor object set corresponding to the anchor multimedia resource. The second object set corresponding to the second multimedia resource does not intersect with the anchor object set.

[0056] In the embodiments of this specification, the type of multimedia can be video, audio, image, or text, and this disclosure does not limit this. Each training sample includes three multimedia elements, and the training sample can also be called a triplet sample. The anchor multimedia element in the training sample can be called an anchor sample, the first multimedia element can be called a positive sample of the anchor multimedia element, and the second multimedia element can be called a negative sample of the anchor multimedia element. The anchor multimedia element can be determined from multiple multimedia elements, and each multimedia element in the multiple multimedia elements can serve as the anchor multimedia element.

[0057] The anchor object set can be a collection of objects corresponding to anchor multimedia, and these objects can be called anchor objects. The first object set can be a collection of objects corresponding to the first multimedia, and these objects can be called first objects. The second object set can be a collection of objects corresponding to the second multimedia, and these objects can be called second objects. Here, an object can be a user.

[0058] In one example, if any object meets the preset corresponding conditions, then the object can be the object corresponding to the multimedia; the preset corresponding conditions can be performing a preset operation on the multimedia; this disclosure does not limit this; the preset operation can be clicking the watch button, clicking the forward button, or clicking the purchase button, etc.; for example, if the object clicks the purchase button corresponding to the multimedia, then the object can be the object corresponding to that multimedia.

[0059] In another example, if an object meets preset tag conditions, then the object can be the object corresponding to the multimedia content. The preset tag conditions can be that the object's category tag matches the multimedia category tag of the multimedia content. For example, if the multimedia content is a comedy video, then the multimedia category tag of that multimedia content is "comedy". The category tag to which the object belongs can be a tag chosen by the object itself. If the category tag chosen by the object is "comedy", then the object's object category tag is "comedy", and the object can be the object corresponding to that multimedia content.

[0060] In the embodiments of this specification, in the same training sample, the first object set corresponding to the first multimedia intersects with the anchor object set corresponding to the anchor multimedia, indicating that at least one object has performed a preset operation on both the anchor multimedia and the first multimedia. This further demonstrates that there is an object-based association between the anchor multimedia and the first multimedia. In the same training sample, the second object set corresponding to the second multimedia does not intersect with the anchor object set, indicating that no object has simultaneously performed a preset operation on both the anchor multimedia and the second multimedia. This further demonstrates that there is no object-based association between the anchor object and the second multimedia.

[0061] In one example, we can first identify an object, then determine the anchor multimedia and the first multimedia from the multimedia corresponding to that object, then determine the set of anchor objects corresponding to the anchor multimedia, then determine the multimedia corresponding to each object in the anchor object set, and finally take any multimedia other than the multimedia corresponding to each object in the anchor object set as the second multimedia, thus obtaining a training sample. Other training samples can also be obtained by following the above steps.

[0062] Step S12: Sort the multiple training samples to obtain the target sample matrix.

[0063] In the embodiments of this specification, the sorting process may be to arrange the P multimedia elements of multiple training samples into P columns (P being the total number of multimedia elements in the multiple training samples) to obtain a target sample matrix of 1 row and P columns (1 row × P columns); the multiple column information of the target sample matrix corresponds to the multiple multimedia elements of the multiple training samples, and the P column information of the target sample matrix corresponds one-to-one with the P multimedia elements.

[0064] In other examples, the row and column values ​​of the target sample matrix can also be other values, and this disclosure does not limit them, as long as the target sample matrix can represent multiple multimedia of multiple training samples.

[0065] Step S13: Input the target sample matrix into the preset multimedia feature extraction model, perform feature extraction processing, and obtain the multimedia feature matrix. The number of rows in the multimedia feature matrix is ​​the same as the number of rows in the target sample matrix.

[0066] In the embodiments of this specification, a preset multimedia feature extraction model can perform feature extraction processing on each multimedia element in the target sample matrix to obtain multimedia features corresponding to each multimedia element in the target sample matrix. The multimedia features can be CLS features (cls emb). The multimedia features corresponding to each multimedia element in the target sample matrix form a multimedia feature matrix.

[0067] The multimedia feature matrix has the same number of rows as the target sample matrix. In one example, both the target sample matrix and the multimedia feature matrix are 1-row × P-column matrices. The P columns of the target sample matrix correspond one-to-one with the P multimedia elements, and the P columns of the multimedia feature matrix correspond one-to-one with the P multimedia features.

[0068] Step S14: Determine the triplet loss information based on the multimedia feature matrix.

[0069] In the embodiments of this specification, the multimedia features corresponding to each training sample can be substituted into the triplet loss function to obtain triplet loss information. Triplet loss information can be determined in an unsupervised manner. In one example, the multimedia features corresponding to each training sample can be obtained from the multimedia feature matrix, and the triplet loss information can be determined based on the multimedia features corresponding to each training sample.

[0070] In the embodiments of this specification, a preset triplet loss function can be obtained, and the mathematical expression of the triplet loss function is as follows:

[0071] L=max(d(a,p)-d(a,n)+margin,0)

[0072] Where d(a,p) represents the distance between the anchor multimedia and the first multimedia in the same training sample, d(a,n) represents the distance between the anchor multimedia and the second multimedia in the same training sample, and margin is a constant greater than 0.

[0073] Step S15: Train the preset multimedia feature extraction model based on the triplet loss information until the preset conditions are met, and obtain the target multimedia feature extraction model.

[0074] In this embodiment of the specification, if the triplet loss information does not meet the preset conditions, the model parameters of the preset multimedia feature extraction model can be adjusted based on the triplet loss information. The adjusted model is then used as the new preset multimedia feature extraction model, and the process returns to step S13 until the obtained triplet loss information meets the preset conditions. The preset conditions can be that the value of the triplet loss information is within a preset range.

[0075] In the embodiments of this specification, each training sample includes an anchor multimedia, a first multimedia, and a second multimedia. The first object set corresponding to the first multimedia can intersect with the anchor object set corresponding to the anchor multimedia, while the second object set corresponding to the second multimedia does not intersect with the anchor object set. It can be seen that there can be a correlation between the anchor multimedia and the first multimedia based on anchor objects, where the anchor object is an object in the anchor object set. There is no correlation between the anchor multimedia and the second multimedia based on objects. That is, the training sample incorporates object information (object behavior or object preferences, etc.). Model training based on such training samples can combine object information with multimedia materials, integrate object information into the multimedia-side feature generation process, fully explore the relationship between object information and multimedia materials, obtain more accurate multimedia features, and thus improve the accuracy of subsequent multimedia recommendations.

[0076] In the model training process described in this specification, each multimedia component is treated as a whole, which ensures the robustness of multimedia feature extraction and makes it less susceptible to the impact of data quality (such as text-image matching errors, first and second class labeling errors, etc.), thus effectively adapting to different recommendation scenarios.

[0077] Because object information is diverse (e.g., due to changes in object interests), and the embodiments of this specification determine training samples based on different object information, the training samples can reflect the diversity of object information. This can improve the diversity of multimedia to be recommended in subsequent actual recommendations, thereby increasing the exposure of various multimedia while meeting the needs of the objects to be recommended.

[0078] As object information is updated, the correspondence between objects and multimedia will also be updated accordingly, causing the training samples to change accordingly, thereby obtaining multimedia features that can more fully reflect object information. It can be seen that the target multimedia feature extraction model obtained by using the embodiments of this specification can obtain more accurate multimedia features, thereby improving the accuracy of subsequent recommendations and recommending multimedia that meets the needs of the objects to be recommended.

[0079] Training a model based on triplet loss information can target the final recommendation tendency. In multimedia recommendation applications, it is more direct than other losses, thus bringing better results to subsequent recommendation models.

[0080] Combination Figure 3 As shown, in one possible implementation, step S11 may include:

[0081] Step S111: Determine the anchor multimedia from multiple multimedia sources.

[0082] In one example, the anchor multimedia can be randomly selected from multiple multimedia resources. In another example, the anchor multimedia can be a multimedia resource that meets the anchor multimedia condition from multiple multimedia resources, such as having been viewed a preset number of times. In other examples, the anchor multimedia can also be a multimedia resource whose corresponding object meets the anchor object condition, such as having viewed a preset number of multimedia resources. Objects that meet the anchor object condition can be identified as candidate objects, and a multimedia resource can be randomly selected from the multimedia resources corresponding to the candidate object as the anchor multimedia.

[0083] Step S112: Determine the anchor object set, candidate multimedia set, and second multimedia set corresponding to the anchor multimedia. The anchor object set is a set of objects on which a preset operation has been performed on the anchor multimedia. The candidate multimedia set includes multimedia on which the preset operation has been performed on each anchor object in the anchor object set. The second multimedia set includes multimedia other than the candidate multimedia set among multiple multimedia.

[0084] In the embodiments of this specification, based on each anchor multimedia, a set of anchor objects, a set of candidate multimedia, and a second set of multimedia can be determined respectively. The preset operation can be clicking a watch button, share button, or purchase button in the multimedia, or it can be browsing the multimedia for a preset duration, etc., which is not limited in this disclosure. For example, if an object clicks the purchase button corresponding to the multimedia, then that object can be the object corresponding to that multimedia.

[0085] Step S113: Select a first multimedia set from the candidate multimedia set. The first multimedia set includes multimedia in the candidate multimedia set except for the anchor multimedia.

[0086] In the embodiments of this specification, the candidate multimedia set corresponding to the anchor multimedia includes the anchor multimedia. The anchor multimedia can be removed from the candidate multimedia set to obtain the first multimedia set.

[0087] Step S114: Arrange the anchor multimedia, the first multimedia, and the second multimedia according to the preset multimedia arrangement order to obtain multiple training samples; the first multimedia is any one of the first multimedia set, and the second multimedia is any one of the second multimedia set.

[0088] In the embodiments of this specification, the training samples corresponding to each anchor point multimedia are determined one by one.

[0089] In one example, each multimedia element in the first multimedia set can be used to construct a training sample, and each multimedia element in the second multimedia set can be used to construct a training sample. The first multimedia element in the training sample can be randomly determined from the first multimedia set, and the second multimedia element can be randomly determined from multimedia elements other than the candidate multimedia set.

[0090] In another example, a subset of multimedia can be randomly selected from the second multimedia set, and the second multimedia can be randomly selected from the subset.

[0091] In the embodiments of this specification, the anchor multimedia, the first multimedia corresponding to the anchor multimedia, and the second multimedia corresponding to the anchor multimedia can be used to obtain a training sample. The multimedia arrangement order of the three multimedias in the training sample can be: anchor multimedia, first multimedia, and second multimedia.

[0092] In one example, let's assume the preset action is clicking the "watch" button. Let C be the total set of all multimedia content. Each object can correspond to a multimedia list, which includes all multimedia content that the object clicks to watch within a preset time period (e.g., one week). Objects with ≥2 multimedia content in their multimedia lists are identified as candidate objects, and U is the set of candidate objects. The set of candidate objects and the set of multimedia lists clicked by each candidate object is denoted as U. For the object ui, its corresponding multimedia list includes ni multimedia elements, which are respectively

[0093] A multimedia c is randomly selected from set C. m For each anchor multimedia, find a list of all multimedia that contains it, and then take the union of all other multimedia in each list except for the anchor multimedia. This set is called the first multimedia set corresponding to the current anchor multimedia. The second multimedia set corresponding to the current anchor multimedia is NEG. m =C\(POS m ∪c m That is, the total set excluding POS. m and c m itself.

[0094] Based on C m POS m and NEG m Construct training samples (triplets); each triplet contains 3 multimedia elements, where the first element is always c. m The second position is POS. m Any multimedia file, the 3rd character is NEG. m Any multimedia. Due to NEGm The amount of data in Chinese and English media is often very large, and using all of it would lead to severe data imbalance. Therefore, it is possible to use NEG... m Randomly select n used n negative samples (e.g., n) used =10), so for c m Its corresponding training samples include POS m ·n used indivual.

[0095] Training sample data is constructed for at least one anchor multimedia element, resulting in a training sample set T = {a}. i pos ij ,neg ijk}, where a represents the anchor point, i = 0, ..., C-1; j = 0, ..., |POS i |-1; k=0,...,||NEG i -1. |POS i |,|NEG i | represent POS respectively i NEG i The number of multimedia elements. Therefore, the embodiments of this specification can obtain multiple training samples.

[0096] In one possible implementation, in each training sample, the anchor multimedia, the first multimedia, and the second multimedia are arranged in a preset multimedia arrangement order; two adjacent training samples in the target sample matrix are different training samples.

[0097] In the embodiments of this specification, based on step S114, it can be ensured that the arrangement order of the three multimedia elements in different training samples is the same. In step S12, multiple training samples can be randomly sorted on a per-sample basis; during the random sorting process, one training sample is arranged adjacent to another training sample; the adjacent arrangement can be adjacent in the matrix rows or adjacent in the matrix columns, and this disclosure does not limit this; each row information of the target sample matrix corresponds to at least one training sample, or each column information of the target sample matrix corresponds to at least one training sample. In one example, the target sample matrix is ​​a 1-row × P-column matrix, and the row information of the target sample matrix corresponds to all training samples.

[0098] In the embodiments of this specification, in step S12, multiple training samples can be sorted on a training sample basis. The resulting target sample matrix can then be used to determine the triplet loss information in step S14, based on the multimedia features corresponding to the training samples.

[0099] Combination Figure 6As shown, in one possible implementation, step S14 may include:

[0100] Step S141: Group the multimedia features in the multimedia feature matrix to obtain the target feature matrix. The target feature matrix consists of the sample features corresponding to multiple training samples. Each sample feature is the multimedia feature corresponding to the anchor multimedia, the first multimedia, and the second multimedia.

[0101] In this embodiment, the order of multimedia features in the multimedia feature matrix corresponds to the order of multimedia in the target sample matrix. The target sample matrix is ​​obtained by sorting the training samples. Multimedia belonging to the same training sample in the target sample matrix can be arranged adjacently. In step S141, the three multimedia corresponding to the training sample can be treated as a whole unit, and the multimedia feature matrix can be grouped to obtain the sample features corresponding to each training sample. Each sample feature includes the multimedia features of the three multimedia in the training sample.

[0102] Step S142: Determine the triplet loss information based on multiple sample features of the target feature matrix.

[0103] In the embodiments of this specification, multiple sample features can be substituted into the triplet loss function to obtain triplet loss information.

[0104] Combination Figure 4 As shown, in one possible implementation, step S12 may include:

[0105] Step S121: Perform inter-sample permutation on multiple training samples to obtain a reference sample matrix, where multiple rows of the reference sample matrix correspond to multiple training samples respectively.

[0106] In this embodiment, the sample arrangement process can involve arranging multiple training samples into multiple rows to obtain a reference sample matrix (Q rows × 3 columns). Simultaneously, the arrangement of the three multimedia elements within each training sample remains unchanged during this process. Each row of the reference sample matrix corresponds to one training sample, and the number of rows in the reference sample matrix is ​​equal to the number of training samples, Q. Each row of the reference sample matrix includes three columns, each corresponding to one of the three multimedia elements of the training sample. The first column can represent the anchor multimedia element, the second column can represent the second multimedia element, and the third column can represent the third multimedia element. The arrangement of multiple training samples in the reference sample matrix can be random. In this embodiment, Q*3 = P.

[0107] Step S122: Perform sample row and column transformation on the reference sample matrix to obtain the target sample matrix, which has one row.

[0108] In the embodiments of this specification, the sample row and column transformation process can be to transform the reference sample matrix (Q rows × 3 columns) into a matrix of 1 row × P columns, wherein the s-th row and 1-th column of the reference sample matrix can be transformed into the (3*s-2)-th column, the s-th row and 2-th column of the reference sample matrix can be transformed into the (3*s-1)-th column, and the s-th row and 3-th column of the reference sample matrix can be transformed into the (3*s)-th column, thereby obtaining the target sample matrix of 1 row × P columns.

[0109] In this embodiment of the specification, in step S13, the target sample matrix is ​​input into a preset multimedia feature extraction model to obtain a multimedia feature matrix of 1 row × P columns. Combined with... Figure 9 As shown, Figure 9 The dashed arrow on the left indicates that the order of multimedia features in the multimedia feature matrix corresponds to the order of multimedia features in the target sample matrix.

[0110] Combination Figure 7 As shown, in one possible implementation, step S141 may include:

[0111] Step S1411: Based on multiple training samples, perform feature row and column transformation on the multimedia feature matrix to obtain the target feature matrix. The feature row and column transformation is the inverse transformation of the sample row and column transformation. The number of rows in the target feature matrix is ​​the number of multiple training samples.

[0112] In the embodiments of this specification, the grouping process in step S141 may include performing feature row and column transformation processing on the multimedia feature matrix; the feature row and column transformation processing on the multimedia feature matrix may be performed based on the transformation relationship between the reference sample matrix and the target sample matrix to obtain the target feature matrix. Combined with Figure 9 As shown, Figure 9 The dashed arrow on the right indicates that the order of multimedia features in the target feature matrix corresponds to the order of multimedia features in the reference sample matrix. In one example, the feature row and column transformation process can be to transform a 1-row × P-column multimedia feature matrix into a Q-row × 3-column matrix. The row and column of the transformed matrix can be determined based on the quotient (t) and remainder (w) of p ÷ 3 in the p-th column of the multimedia feature matrix. Specifically, the row of the transformed p-th column is the (t+1)-th row; when w is not equal to 0, the column of the transformed p-th column is the w-th column; when w is equal to 0, the column of the transformed p-th column is the 3rd column, thus obtaining the Q-row × 3-column target feature matrix.

[0113] The Q rows of information in the target feature matrix can correspond one-to-one with the Q training samples. Each row of information in the target feature matrix can include three columns, which correspond to three multimedia features respectively. The first multimedia feature corresponds to the anchor multimedia, the second multimedia feature corresponds to the first multimedia, and the third multimedia feature corresponds to the second multimedia.

[0114] Combination Figure 5 As shown, in one possible implementation, the preset multimedia feature extraction model includes an encoding model, a fusion model, and a segmentation model.

[0115] Step S13 may include:

[0116] Step S131: Input the target sample matrix into the encoding model, encode the target sample matrix to obtain a single-modal feature matrix.

[0117] In the embodiments of this specification, the format of each multimedia element in the target sample matrix can be a unimodal identifier sequence format. The number of target sample matrices can be one or more; when there is only one target sample matrix, the modality of the identifier sequence can be one of the following: text modality, video modality, and audio modality; when there are multiple target sample matrices, the modality of the identifier sequence can include multiple modalities such as text modality, video frame modality, and audio modality.

[0118] In this embodiment of the specification, before step S131, after obtaining the target sample matrix, the multimedia in the target sample matrix can be decomposed and sequenced. Each multimedia is decomposed to obtain two types of single-modal data (text modal data and video modal data). Then, the single-modal data is sequenced to obtain the identifier sequences of the two single modalities of each multimedia. For example, long sentences in the text modality can be processed into word sequences, and video modality can be processed into video frame sequences. Thus, a target sample matrix in text modal identifier sequence format and a target sample matrix in video modal identifier sequence format are obtained.

[0119] In other examples, before step S131, all multimedia of all training samples can be decomposed and sequenced to obtain training samples in identifier sequence format; then the training samples in identifier sequence format are sorted to obtain the target sample matrix in unimodal identifier sequence format.

[0120] In step S131, the encoding model can be a single-modal encoding model. This model encodes the multimedia in the single-modal identifier sequence format within the target sample matrix, obtaining the single-modal features corresponding to each multimedia in the target sample matrix. Target sample matrices in different modal identifier sequence formats can be input into the corresponding encoding model for their respective modalities. For text modalities, common text modal encoding models include Text-RCNN and BERT; for video modalities, common video modal encoding models include C3D, EfficientNet, and VideoSwin. For each identifier sequence, the corresponding modal encoding model can output a fixed-dimensional single-modal feature (single-modal embedding). In one example, the dimension can be 768. Taking the case where a multimedia is decomposed into two types of single-modal data (e.g., text modal data and video modal data) as an example, in step S131, for each multimedia, the corresponding text modal features and video modal features can be obtained, thus obtaining the text modal feature matrix and video modal feature matrix corresponding to the target sample matrix.

[0121] Step S132: Input the single-modal feature matrix into the fusion model, perform fusion processing on the single-modal feature matrix to obtain the fused modal feature matrix.

[0122] In the embodiments of this specification, the fusion model can be a fusion transformation model (transformer model), generally a multi-layer transformer structure. The fusion model can be used to fuse and transform the input single-modal features to obtain fused modal features. The fused modal feature matrix can include the fused modal features corresponding to each multimedia in the target sample matrix, and one multimedia can correspond to one fused modal feature.

[0123] In the embodiments of this specification, the parameters of the fusion model are adjustable and can be adjusted accordingly based on the number of types of input single-modal feature matrices to ensure that the fusion model can process one type of single-modal feature matrix or process multiple types of single-modal feature matrices simultaneously.

[0124] In one example, combined Figure 9 As shown, taking two single modalities (text modality and video modality) as examples, the text modality feature matrix and the video modality feature matrix obtained in step S131 are input into the fusion model together, and modal interaction is performed on the text modality feature matrix and the video modality feature matrix to obtain the fused modality feature matrix.

[0125] In another example, taking a modality (text modality) as an example, the text modality feature matrix obtained in step S131 is input into the fusion model, and modal interaction is performed on the text modality feature matrix to obtain the fused modality feature matrix.

[0126] Step S133: Input the fusion modal feature matrix into the segmentation model, and perform segmentation processing on each fusion modal feature of the fusion modal feature matrix to obtain the multimedia feature matrix.

[0127] In the embodiments of this specification, a fused modal feature may include a multimedia feature (cls emb) and a modality-specific feature. Taking the case where a multimedia is decomposed into two single-modal data (e.g., text modal data and video modal data) as an example, the first (0th) bit of the fused modal feature is a cls flag bit (multimedia feature cls emb), which gathers information from all text and video frames and can be regarded as a comprehensive representation of the multimedia; the following bits are the features corresponding to the text modality and the features corresponding to the visual modality, respectively.

[0128] In the embodiments described in this specification, the segmentation model can segment the fused modal features to obtain multimedia features corresponding to the multimedia. The multimedia feature matrix may include the multimedia features corresponding to each multimedia in the target sample matrix.

[0129] Combination Figure 8 As shown, in one possible implementation, the method may further include:

[0130] Step S16: Determine the single-mode loss information based on the single-mode feature matrix.

[0131] In the embodiments of this specification, after step S131, the single-modal features in the single-modal feature matrix can be substituted into the single-modal loss function to obtain the single-modal loss information. Different single-modal loss functions correspond to different modalities. In one example, masked text modeling loss information (MLM) can be determined based on the text modal feature matrix, and masked video frame modeling loss information (MFM) can be determined based on the video modal feature matrix. Both masked text modeling loss information and masked video frame modeling loss information can be determined in an unsupervised manner.

[0132] Step S17: Determine the reference loss information based on the fused modality feature matrix. The reference loss information includes one or more of the cross-entropy loss information and the contrastive loss information.

[0133] In the embodiments of this specification, a fused modal feature may include a multimedia feature (cls emb) and a feature corresponding to the modality. In the segmentation process of step S133, the segmentation model may also obtain the fused feature corresponding to the multimedia.

[0134] In one example, a multimedia presentation is decomposed into two single-modal data (e.g., text modality data and video modality data). The fused modality features can include multimedia features (cls emb), features corresponding to the text modality, and features corresponding to the visual modality. In step S17, in addition to obtaining the multimedia features cls emb, the features corresponding to the text modality and the features corresponding to the visual modality are also obtained. Then, the features corresponding to the text modality and the features corresponding to the visual modality are averaged in terms of the number of dimensions to obtain the text fusion features (merged text emb) and the video fusion features (merged video emb). Thus, each multimedia presentation can obtain one multimedia feature cls emb, one text fusion feature merged text emb, and one video fusion feature merged video emb. Based on the target sample matrix, a multimedia feature matrix, a text fusion feature matrix, and a video fusion feature matrix can be obtained. Further, cross-entropy loss information can be determined based on the multimedia feature matrix. The cross-entropy loss information can be primary or secondary industry classification cross-entropy loss information, and can be determined in a supervised manner. The contrastive loss information can be determined based on the text fusion feature matrix and the video fusion feature matrix. This contrastive loss information can be image-text matching contrastive learning loss information, or it can be determined in an unsupervised manner. The multimedia feature cls emb can be 10-dimensional, the text fusion feature merged text emb can be 768-dimensional, and the video fusion feature merged video emb can be 768-dimensional; this disclosure does not impose any limitations on these dimensions.

[0135] In another example, if a multimedia dataset is decomposed into a single modal data (e.g., text modality data), then the fused modality features can include multimedia features (cls emb) and features corresponding to the text modality. Each multimedia dataset can yield a multimedia feature cls emb and a text fusion feature merged text emb. Based on the target sample matrix, a multimedia feature matrix and a text fusion feature matrix can be obtained. Furthermore, cross-entropy loss information can be determined based on the multimedia feature matrix.

[0136] Step S15 may include:

[0137] Step S151: Based on the triplet loss information and the target loss information, train the preset multimedia feature extraction model until the preset conditions are met to obtain the target multimedia feature extraction model. The target loss information includes one or more of the single-modal loss information and the reference loss information.

[0138] In the embodiments of this specification, the triplet loss information can be combined with any loss information from the target loss information to determine the target multimedia feature extraction model. The target loss information can be one or more of the following: text modeling loss information, masked video frame modeling loss information, cross-entropy loss information, and contrastive loss information.

[0139] In the embodiments of this specification, the triplet loss information and the target loss information can be weighted. Different weights can be set for different losses as needed to obtain the overall loss information, perform gradient calculation and backpropagation, and train the preset multimedia feature extraction model.

[0140] The embodiments in this specification are based on multiple loss function information, including triplet loss information, to train the model, which can obtain a more accurate target multimedia feature extraction model, thereby improving the accuracy of subsequent multimedia recommendations.

[0141] This specification's embodiments introduce training samples based on object information (such as object behavior or object preferences). The essence of these embodiments is that, given a multimedia resource (which can be viewed as the object's current behavior), the model learns which multimedia resources are more likely to appear alongside it (should be recommended) and which are less likely to appear alongside it (should not be recommended). In the training samples, the first multimedia resource and the anchor multimedia resource can be recommended together, while the second multimedia resource and the anchor multimedia resource cannot be recommended together. This supervision process is actually free from the limitations of specific modalities, treating each multimedia resource as a whole as the unit. Therefore, it is more robust and relatively less affected by data quality (such as image-text matching errors, first and second class labeling errors, etc.), and can adapt to different recommendation scenarios. Training the model based on triplet loss information allows for targeting the final recommendation tendency. In the context of multimedia recommendation, this approach is more direct than other losses, thus leading to better performance for subsequent recommendation models.

[0142] Combination Figure 10 As shown in the embodiments of this specification, a multimedia recommendation method is provided and applied to a multimedia recommendation system. In one example, the multimedia recommendation system can be a single-tower model, which may include a prediction model and a target multimedia feature extraction model on the multimedia side.

[0143] The multimedia recommendation methods provided in the embodiments of this specification include:

[0144] Step S21: Determine the object to be recommended and the target multimedia corresponding to the object to be recommended.

[0145] In the embodiments of this specification, the object to be recommended can be any object, and the target multimedia can be the multimedia on which the object to be recommended performs a preset operation. In one example, when any object is detected performing a preset operation, that object can be identified as the object to be recommended, and the multimedia on which that object performs the preset operation can be identified as the target multimedia. The preset operation can be clicking a watch button, share button, or purchase button in the multimedia, or it can be browsing the multimedia for a preset duration, etc. This disclosure does not limit this.

[0146] Step S22: Input the target multimedia into the target multimedia feature extraction model, perform feature extraction processing on the target multimedia, and obtain the multimedia features corresponding to the target multimedia. The target multimedia feature extraction model is obtained according to the training method of the multimedia feature extraction model described above.

[0147] In the embodiments of this specification, before step S22, the target multimedia is first decomposed to obtain at least one single-modal data; then the at least one single-modal data is sequenced to obtain at least one single-modal identifier sequence.

[0148] In the embodiments of this specification, the target multimedia feature extraction model may include a target encoding model, a target fusion model, and a target segmentation model. In step S22, at least one single-modal identifier sequence can be input into its corresponding single-modal target encoding model to obtain at least one single-modal feature; at least one single-modal feature can be input into the target fusion model to obtain a fused modal feature; and the fused modal feature can be input into the target segmentation model to obtain a multimedia feature (cls emb) corresponding to the target multimedia.

[0149] Step S23: Input the multimedia features corresponding to the target multimedia into the prediction model, and perform prediction processing on multiple preset multimedia based on the multimedia features corresponding to the target multimedia to obtain the prediction probability of each of the multiple preset multimedia.

[0150] In the embodiments of this specification, the preset multimedia may belong to a preset database corresponding to the prediction model. In a single-tower model, the prediction model of the single-tower model can perform prediction processing on multiple preset multimedia based on multimedia features to obtain the prediction probability corresponding to each preset multimedia. The prediction probability can characterize the probability that the preset multimedia will be used by the target object to perform a target operation. Optionally, the prediction processing can be click-through rate prediction processing, the target operation can be clicking the watch button, and the prediction probability can be the click-through rate prediction probability; the prediction processing can also be conversion rate prediction processing, the target operation can be clicking the buy button, and the prediction probability can be the conversion rate prediction probability; this disclosure does not limit it in this way.

[0151] Step S24: Based on the predicted probabilities of each of the multiple preset multimedia, determine the multimedia to be recommended from the multiple preset multimedia.

[0152] In one example, multiple preset multimedia resources can be sorted by their predicted probabilities (click-through rate prediction probability or conversion rate prediction probability) from highest to lowest, and the preset multimedia resource with the highest predicted probability can be identified as the multimedia resource to be recommended.

[0153] In other examples, the effective cost per mille (ECPM) of multiple preset multimedia resources can be determined based on their respective click-through rate prediction probability, conversion rate prediction probability, and list price. The preset multimedia resource with the highest ECPM is then selected as the one to be recommended. The list price of a multimedia resource can be the price paid by the multimedia producer to the multimedia platform for promotion on the platform.

[0154] Step S25: Recommend the multimedia to be recommended to the target audience.

[0155] In the embodiments of this specification, after determining the multimedia to be recommended, the multimedia to be recommended can be pushed to the recommended object through SMS, voice, pop-up window or other means.

[0156] In the embodiments of this specification, in the training samples of the target multimedia feature extraction model, the first object set corresponding to the first multimedia intersects with the anchor object set corresponding to the anchor multimedia, while the second object set corresponding to the second multimedia does not intersect with the anchor object set. This indicates that there is a correlation between the anchor multimedia and the first multimedia based on anchor objects, where the anchor object is an object within the anchor object set. There is no object-based correlation between the anchor multimedia and the second multimedia; that is, the training samples incorporate object information (object behavior or object preferences, etc.). Training the model based on such training samples allows for the combination of object information with multimedia materials, integrating object information into the multimedia-side feature generation process, fully exploring the correlation between object information and multimedia materials, and obtaining more accurate multimedia features. Therefore, using the target multimedia feature extraction model of the embodiments of this specification can yield more accurate multimedia features, thereby improving the accuracy of subsequent recommendations.

[0157] In another example, the multimedia recommendation system can be a dual-tower model, which may include a prediction model, a target multimedia feature extraction model on the multimedia side, and an object feature extraction model on the object side. The target multimedia feature extraction model can be used to generate corresponding multimedia features based on the target multimedia. The object feature extraction model can be used to generate corresponding object features based on object information (information about the object to be recommended). The prediction model of the dual-tower model can obtain full features based on multimedia features and object features, and perform prediction processing on multiple preset multimedia programs based on the full features to obtain the prediction probability corresponding to each preset multimedia program.

[0158] Combination Figure 11 As shown in the embodiments of this specification, a multimedia feature extraction model training device is also provided, which may include:

[0159] The determination module 10 is used to determine multiple training samples from multiple multimedias. Each training sample includes an anchor multimedia, a first multimedia, and a second multimedia. The anchor multimedia is any one of the multiple multimedias. The first object set corresponding to the first multimedia intersects with the anchor object set corresponding to the anchor multimedia. The second object set corresponding to the second multimedia does not intersect with the anchor object set.

[0160] The sorting module 20 is used to sort multiple training samples to obtain the target sample matrix;

[0161] The feature extraction module 30 is used to input the target sample matrix into a preset multimedia feature extraction model, perform feature extraction processing, and obtain a multimedia feature matrix. The number of rows in the multimedia feature matrix is ​​the same as the number of rows in the target sample matrix.

[0162] Triple-state loss determination module 40 is used to determine triple-state loss information based on the multimedia feature matrix;

[0163] The model training module 50 is used to train a preset multimedia feature extraction model based on triplet loss information until the preset conditions are met, thus obtaining the target multimedia feature extraction model.

[0164] In one possible implementation, the triplet loss determination module 40 may include:

[0165] The grouping unit is used to group the multimedia features in the multimedia feature matrix to obtain the target feature matrix. The target feature matrix consists of the sample features corresponding to multiple training samples. Each sample feature is the multimedia feature corresponding to the anchor multimedia, the first multimedia, and the second multimedia.

[0166] The triplet loss determination unit is used to determine triplet loss information based on multiple sample features of the target feature matrix.

[0167] In one possible implementation, in each training sample, the anchor multimedia, the first multimedia, and the second multimedia are arranged in a preset multimedia arrangement order;

[0168] In the target sample matrix, two adjacent training samples are different training samples.

[0169] In one possible implementation, the sorting module 20 may include:

[0170] The sample arrangement unit is used to perform sample arrangement processing on multiple training samples to obtain a reference sample matrix. The multiple rows of the reference sample matrix correspond to multiple training samples respectively.

[0171] The sample row and column transformation unit is used to perform sample row and column transformation on the reference sample matrix to obtain the target sample matrix, which has one row.

[0172] Grouping units may include:

[0173] The feature row and column transformation subunit is used to perform feature row and column transformation processing on the multimedia feature matrix based on multiple training samples to obtain the target feature matrix. The feature row and column transformation processing is the inverse transformation processing of the sample row and column transformation processing, and the number of rows in the target feature matrix is ​​the number of multiple training samples.

[0174] In one possible implementation, the preset multimedia feature extraction model includes an encoding model, a fusion model, and a segmentation model;

[0175] Feature extraction module 30 may include:

[0176] The encoding unit is used to input the target sample matrix into the encoding model, encode the target sample matrix, and obtain a single-modality feature matrix.

[0177] The fusion unit is used to input the single-modal feature matrix into the fusion model, perform fusion processing on the single-modal feature matrix, and obtain the fused modal feature matrix;

[0178] The segmentation unit is used to input the fusion modal feature matrix into the segmentation model, and to segment each fusion modal feature of the fusion modal feature matrix to obtain the multimedia feature matrix.

[0179] In one possible implementation, the method may also include:

[0180] The single-modal loss determination module is used to determine single-modal loss information based on the single-modal feature matrix.

[0181] The reference loss determination module is used to determine reference loss information based on the fused modality feature matrix. The reference loss information includes one or more of cross-entropy loss information and contrastive loss information.

[0182] Model training module 50 includes:

[0183] The model training unit is used to train a preset multimedia feature extraction model based on triplet loss information and target loss information until preset conditions are met, thereby obtaining a target multimedia feature extraction model. The target loss information includes one or more of single-modal loss information and reference loss information.

[0184] In one possible implementation, the determining module 10 may include:

[0185] The first determining unit is used to determine the anchor multimedia from multiple multimedia sources;

[0186] The second determining unit is used to determine the anchor object set, the candidate multimedia set, and the second multimedia set corresponding to the anchor multimedia. The anchor object set is a set of objects on which a preset operation has been performed on the anchor multimedia. The candidate multimedia set includes multimedia on which the preset operation has been performed by each anchor object in the anchor object set. The second multimedia set includes multimedia other than the candidate multimedia set among multiple multimedia sets.

[0187] A filtering unit is used to filter out a first multimedia set from the candidate multimedia set. The first multimedia set includes multimedia in the candidate multimedia set except for the anchor multimedia.

[0188] The arrangement unit is used to arrange the anchor multimedia, the first multimedia, and the second multimedia according to a preset multimedia arrangement order to obtain multiple training samples; the first multimedia is any one of the first multimedia set, and the second multimedia is any one of the second multimedia set.

[0189] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0190] Figure 12 This diagram illustrates a block diagram of an electronic device for training a multimedia feature extraction model or for multimedia recommendation, according to an embodiment of this application. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for a multimedia feature extraction model or a multimedia recommendation method.

[0191] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0192] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a training method for a multimedia feature extraction model or a multimedia recommendation method as described in the embodiments of this application.

[0193] In an exemplary embodiment, a storage medium is also provided, which, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the multimedia resource processing method in the embodiments of this application.

[0194] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the training method of the multimedia feature extraction model or the multimedia recommendation method in the embodiments of this application.

[0195] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0196] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0197] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for a multimedia feature extraction model, characterized in that, include: Multiple training samples are determined from multiple multimedia sources, each training sample including anchor multimedia, first multimedia, and second multimedia. The anchor multimedia is any one of the plurality of multimedia; The first object set corresponding to the first multimedia intersects with the anchor object set corresponding to the anchor multimedia, and at least one object has performed a preset operation on both the anchor multimedia and the first multimedia; the second object set corresponding to the second multimedia does not intersect with the anchor object set, and no object has performed a preset operation on either the anchor multimedia or the second multimedia; the anchor object set is the set of objects on which the anchor multimedia has performed a preset operation; the preset operation includes clicking the watch button, clicking the share button, or clicking the purchase button; The multiple training samples are sorted to obtain the target sample matrix; The target sample matrix is ​​input into the encoding model in the preset multimedia feature extraction model for encoding processing to obtain a single-modality feature matrix; The single-modal feature matrix is ​​input into the fusion model in the preset multimedia feature extraction model for fusion processing to obtain the fused modal feature matrix; The fused modality feature matrix is ​​input into the segmentation model in the preset multimedia feature extraction model for segmentation processing to obtain the multimedia feature matrix; The number of rows in the multimedia feature matrix is ​​the same as the number of rows in the target sample matrix; Based on the multimedia feature matrix, the triplet loss information is determined; The preset multimedia feature extraction model is trained based on the triplet loss information until the preset conditions are met, thus obtaining the target multimedia feature extraction model. The target multimedia feature extraction model is a model used to extract multimedia features in a multimedia recommendation system.

2. The training method for the multimedia feature extraction model according to claim 1, characterized in that, The step of determining the triplet loss information based on the multimedia feature matrix includes: The multimedia features in the multimedia feature matrix are grouped to obtain a target feature matrix. The target feature matrix consists of the sample features corresponding to each of the multiple training samples. Each sample feature is the multimedia feature corresponding to the anchor multimedia, the first multimedia, and the second multimedia. The triplet loss information is determined based on multiple sample features of the target feature matrix.

3. The training method for the multimedia feature extraction model according to claim 2, characterized in that, The process of sorting the multiple training samples to obtain the target sample matrix includes: The multiple training samples are arranged to obtain a reference sample matrix, and the multiple rows of the reference sample matrix correspond to the multiple training samples respectively. The reference sample matrix is ​​subjected to a sample row and column transformation process to obtain the target sample matrix, wherein the target sample matrix has one row.

4. The training method for the multimedia feature extraction model according to claim 3, characterized in that, The process of grouping the multimedia features in the multimedia feature matrix to obtain the target feature matrix includes: Based on the multiple training samples, the multimedia feature matrix is ​​subjected to feature row and column transformation processing to obtain the target feature matrix. The feature row and column transformation processing is the inverse transformation processing of the sample row and column transformation processing, and the number of rows of the target feature matrix is ​​the number of the multiple training samples.

5. The training method for the multimedia feature extraction model according to claim 4, characterized in that, The method further includes: Based on the single-modal feature matrix, determine the single-modal loss information; Based on the fused modality feature matrix, reference loss information is determined, which includes one or more of cross-entropy loss information and contrastive loss information. The step of training the preset multimedia feature extraction model based on the triplet loss information until the preset conditions are met to obtain the target multimedia feature extraction model includes: Based on the triplet loss information and the target loss information, the preset multimedia feature extraction model is trained until the preset conditions are met, thereby obtaining the target multimedia feature extraction model. The target loss information includes one or more of the single-modal loss information and the reference loss information.

6. The training method for the multimedia feature extraction model according to claim 1, characterized in that, The step of determining multiple training samples from multiple multimedia sources includes: The anchor multimedia is determined from the plurality of multimedia; The anchor object set, candidate multimedia set, and second multimedia set corresponding to the anchor multimedia are determined. The anchor object set is a set of objects on which a preset operation has been performed on the anchor multimedia. The candidate multimedia set includes multimedia on which the preset operation has been performed by each anchor object in the anchor object set. The second multimedia set includes multimedia other than the candidate multimedia set among the plurality of multimedia. A first multimedia set is selected from the candidate multimedia set, and the first multimedia set includes multimedia in the candidate multimedia set other than the anchor multimedia. According to a preset multimedia arrangement order, the anchor multimedia, the first multimedia, and the second multimedia are arranged to obtain the multiple training samples; the first multimedia is any one of the first multimedia set, and the second multimedia is any one of the second multimedia set.

7. A multimedia recommendation method, characterized in that, include: Determine the object to be recommended and the target multimedia corresponding to the object to be recommended; The target multimedia is input into the target multimedia feature extraction model, and feature extraction processing is performed on the target multimedia to obtain the multimedia features corresponding to the target multimedia, wherein the target multimedia feature extraction model is obtained by the training method according to any one of claims 1 to 6; The multimedia features corresponding to the target multimedia are input into the prediction model, and multiple preset multimedia are predicted based on the multimedia features corresponding to the target multimedia to obtain the prediction probability of each of the multiple preset multimedia. Based on the predicted probabilities of each of the multiple preset multimedia resources, determine the multimedia resources to be recommended from the multiple preset multimedia resources; The multimedia to be recommended is recommended to the target audience.

8. A training device for a multimedia feature extraction model, characterized in that, include: The determination module is used to determine multiple training samples from multiple multimedia, each training sample including anchor multimedia, first multimedia, and second multimedia; The anchor multimedia is any one of the plurality of multimedia; The first object set corresponding to the first multimedia intersects with the anchor object set corresponding to the anchor multimedia, and at least one object has performed a preset operation on both the anchor multimedia and the first multimedia; the second object set corresponding to the second multimedia does not intersect with the anchor object set, and no object has performed a preset operation on either the anchor multimedia or the second multimedia; the anchor object set is the set of objects on which the anchor multimedia has performed a preset operation; the preset operation includes clicking the watch button, clicking the share button, or clicking the purchase button; The sorting module is used to sort the multiple training samples to obtain the target sample matrix; The feature extraction module is used to input the target sample matrix into a preset multimedia feature extraction model for feature extraction processing to obtain a multimedia feature matrix. The number of rows in the multimedia feature matrix is ​​the same as the number of rows in the target sample matrix. The preset multimedia feature extraction model includes an encoding model, a fusion model, and a segmentation model. The triplet loss determination module is used to determine triplet loss information based on the multimedia feature matrix. The model training module is used to train the preset multimedia feature extraction model based on the triplet loss information until the preset conditions are met, so as to obtain the target multimedia feature extraction model. The target multimedia feature extraction model is a model used to extract multimedia features in a multimedia recommendation system; The feature extraction module includes: An encoding unit is used to input the target sample matrix into the encoding model for encoding processing to obtain a single-modality feature matrix; A fusion unit is used to input the single-modal feature matrix into the fusion model for fusion processing to obtain a fused modal feature matrix; The segmentation unit is used to input the fused modality feature matrix into the segmentation model for segmentation processing to obtain the multimedia feature matrix.

9. The apparatus according to claim 8, characterized in that, The triplet loss determination module includes: A grouping unit is used to group the multimedia features in the multimedia feature matrix to obtain a target feature matrix. The target feature matrix consists of the sample features corresponding to each of the multiple training samples. Each sample feature is the multimedia feature corresponding to the anchor multimedia, the first multimedia, and the second multimedia. The triplet loss determination unit is used to determine the triplet loss information based on multiple sample features of the target feature matrix.

10. The apparatus according to claim 9, characterized in that, The sorting module includes: The sample arrangement unit is used to perform sample arrangement processing on the plurality of training samples to obtain a reference sample matrix, wherein the multiple rows of the reference sample matrix correspond to the plurality of training samples respectively. The sample row and column transformation unit is used to perform sample row and column transformation processing on the reference sample matrix to obtain the target sample matrix, wherein the target sample matrix has one row. Grouping units include: The feature row-column transformation subunit is used to perform feature row-column transformation processing on the multimedia feature matrix based on the multiple training samples to obtain the target feature matrix. The feature row-column transformation processing is the inverse transformation processing of the sample row-column transformation processing, and the number of rows of the target feature matrix is ​​the number of the multiple training samples.

11. The apparatus according to claim 8, characterized in that, The device further includes: The single-modal loss determination module is used to determine single-modal loss information based on the single-modal feature matrix. The reference loss determination module is used to determine reference loss information based on the fused modality feature matrix, wherein the reference loss information includes one or more of cross-entropy loss information and contrastive loss information; The model training module includes: The model training unit is used to train the preset multimedia feature extraction model based on the triplet loss information and the target loss information until the preset conditions are met, thereby obtaining the target multimedia feature extraction model. The target loss information includes one or more of the single-modal loss information and the reference loss information.

12. The apparatus according to claim 8, characterized in that, The determining module includes: A first determining unit is configured to determine the anchor multimedia from the plurality of multimedia; The second determining unit is used to determine the anchor object set, the candidate multimedia set, and the second multimedia set corresponding to the anchor multimedia. The anchor object set is a set of objects on which a preset operation has been performed on the anchor multimedia. The candidate multimedia set includes multimedia on which the preset operation has been performed by each anchor object in the anchor object set. The second multimedia set includes multimedia on which the multiple multimedia sets are not on the candidate multimedia set. A filtering unit is configured to filter out a first multimedia set from the candidate multimedia set, wherein the first multimedia set includes multimedia in the candidate multimedia set other than the anchor multimedia. The arrangement unit is used to arrange the anchor multimedia, the first multimedia, and the second multimedia according to a preset multimedia arrangement order to obtain the plurality of training samples; the first multimedia is any one of the first multimedia set, and the second multimedia is any one of the second multimedia set.

13. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the training method of the multimedia feature extraction model according to any one of claims 1 to 6, or the processor is configured to perform the multimedia recommendation method according to claim 7.

14. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the training method of the multimedia feature extraction model according to any one of claims 1 to 6, or when the computer program instructions are executed by the processor, they implement the multimedia recommendation method according to claim 7.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the training method of the multimedia feature extraction model according to any one of claims 1 to 6, or the multimedia recommendation method according to claim 7.

Images