Multimedia resource model training and similarity determination method and device

Abstract

The present application relates to the field of artificial intelligence technology, and provides a method and device for model training and similarity determination of multimedia resources, in order to solve the problem of complex training of convolutional neural network models in related technologies. In the present application, similar query requests correspond to similar multimedia resources. In order to better construct positive sample pairs, the multimedia resources corresponding to similar query requests in the embodiments of the present application are screened using a time window. The reason why a time window can be used is that under the same or similar query requests, two access operations with similar access times are often accesses to extremely similar multimedia resources. Therefore, the positive sample pairs screened based on the time window have high practicality and accuracy.

Description

Background Art

[0002] There are abundant video resources on the Internet. With the accumulation of time and the increase of users, the number of videos is huge. Some video-based applications usually use video feature expressions. One application is to recommend videos by calculating the similarity between videos.

[0003] The similarity between videos generally requires converting the videos into feature expressions, such as embedding features, and then calculating the distance between the feature expressions of two videos as the similarity between the two videos.

[0004] In the related art, convolutional neural network (CNN) is often used to extract the feature expression of the video. However, training a convolutional neural network requires positive sample pairs and negative sample pairs. The labeling of positive and negative sample pairs is complex, difficult, and has low output. Therefore, the training of the convolutional neural network model in the related art is complex. Summary of the invention

[0005] The embodiments of the present application provide a method and device for model training and similarity determination of multimedia resources, which are used to solve the problem of complex training of convolutional neural network models in related technologies.

[0006] In a first aspect, the present application provides a method for training a feature extraction model of multimedia resources, the method comprising:

[0007] Constructing positive sample pairs using multimedia resources accessed by the same account object in the multimedia resource set, wherein the access times of two samples in the positive sample pair are taken from the same time window; and,

[0008] A negative sample pair is constructed using multimedia resources that have not been accessed and multimedia resources that have been accessed in the multimedia resource set; wherein multimedia resources matched by similar query requests constitute the multimedia resource set, and query requests with similarity higher than a similarity threshold constitute the similar query requests;

[0009] The positive sample pairs and the negative sample pairs are used to train a feature extraction model, wherein the feature extraction model is used to extract similarities between multimedia resources.

[0010] Optionally, the feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer;

[0011] The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource;

[0012] The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource;

[0013] The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource;

[0014] The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

[0015] Optionally, the second feature is a parameter of the attention level of a multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to perform feature extraction on the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to perform feature extraction on the content of the first multimedia resource to obtain the first feature of the first multimedia resource;

[0016] The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

[0017] Optionally, the adopting the positive sample pair and the negative sample pair to train a feature extraction model includes:

[0018] Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair;

[0019] Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features;

[0020] Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs;

[0021] A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

[0022] Optionally, the method further includes:

[0023] Acquire the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set;

[0024] Screening out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set;

[0025] The method of constructing positive sample pairs using multimedia resources in a multimedia resource set accessed by the same account object specifically includes:

[0026] For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

[0027] Optionally, the loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but has different signs.

[0028] Optionally, the loss function is used to minimize the distance between two samples in a positive sample pair and maximize the degree of difference between two samples in a negative sample pair.

[0029] Optionally, the loss function is used to minimize the following value:

[0030]

[0031] Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

[0032] Optionally, the method further includes:

[0033] The multimedia resource set is constructed based on the following method:

[0034] Determining a similarity between each of the plurality of query requests;

[0035] Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set;

[0036] The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

[0037] Optionally, the method further includes:

[0038] For each query request in the query request set, if the last accessed multimedia resource associated with the query request has a specified operation record, the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object are marked as a positive sample pair.

[0039] Optionally, the multiple query requests satisfy at least one of the following conditions:

[0040] The query time interval is less than the first time duration;

[0041] Query request for the same account object.

[0042] Optionally, if the multiple query requests satisfy a query time interval less than the first duration or are query requests for the same account object, the similarity threshold used by the multiple query requests with a query time interval less than the first duration is less than the similarity threshold used by the query requests for the same account object.

[0043] Optionally, the specified operation record includes at least one of the following:

[0044] The playback duration is longer than the specified duration;

[0045] Like, forward, favorite, follow, and comment.

[0046] Optionally, the attention level parameter includes at least one of the following parameters:

[0047] Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

[0048] In a second aspect, the present application further provides a method for determining similarity of multimedia resources, the method comprising:

[0049] Acquire a first multimedia resource and a second multimedia resource;

[0050] Using a feature extraction model to extract feature expressions of the first multimedia resource and the second multimedia resource respectively;

[0051] Determining the similarity between the first multimedia resource and the second multimedia resource based on the respective feature expressions of the first multimedia resource and the second multimedia resource;

[0052] The feature extraction model is pre-trained based on positive sample pairs and negative samples, where:

[0053] Query requests with similarity higher than a similarity threshold constitute query requests of the same type; multimedia resources matched by query requests of the same type constitute the multimedia resource set;

[0054] The positive sample pair is constructed by using multimedia resources in the multimedia resource set that are accessed by the same account object, and the access time of two samples in the positive sample pair is taken from the same time window;

[0055] The negative sample pairs are constructed by using the multimedia resources that have not been accessed and the multimedia resources that have been accessed in the multimedia resource set.

[0056] Optionally, the feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer;

[0057] The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource;

[0058] The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource;

[0059] The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource;

[0060] The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

[0061] Optionally, the second feature is a parameter of the attention level of a multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to perform feature extraction on the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to perform feature extraction on the content of the first multimedia resource to obtain the first feature of the first multimedia resource;

[0062] The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

[0063] Optionally, using the positive sample pair and the negative sample pair to train a feature extraction model includes:

[0064] Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair;

[0065] Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features;

[0066] Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs;

[0067] A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

[0068] Optionally, the loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but has different signs.

[0069] Optionally, the method further includes:

[0070] The multimedia resource set is constructed based on the following method:

[0071] Determining a similarity between each of the plurality of query requests;

[0072] Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set;

[0073] The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

[0074] Optionally, the method further includes:

[0075] For each query request in the query request set, if the last accessed multimedia resource associated with the query request has a specified operation record, the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object are marked as a positive sample pair.

[0076] Optionally, the multiple query requests satisfy at least one of the following conditions:

[0077] The query time interval is less than the first time duration;

[0078] Query request for the same account object.

[0079] Optionally, if the multiple query requests satisfy a query time interval less than the first duration or are query requests for the same account object, the similarity threshold used by the multiple query requests with a query time interval less than the first duration is less than the similarity threshold used by the query requests for the same account object.

[0080] Optionally, the method further includes:

[0081] Acquire the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set;

[0082] Screening out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set;

[0083] The multimedia resources accessed by the same account object in the multimedia resource set are used to construct positive sample pairs, including:

[0084] For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

[0085] Optionally, the loss function is used to minimize the distance between two samples in a positive sample pair and maximize the degree of difference between two samples in a negative sample pair.

[0086] Optionally, the loss function is used to minimize the following value:

[0087]

[0088] Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

[0089] Optionally, the specified operation record includes at least one of the following:

[0090] The playback duration is longer than the specified duration;

[0091] Like, forward, favorite, follow, and comment.

[0092] Optionally, the attention level parameter includes at least one of the following parameters:

[0093] Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

[0094] In a third aspect, the present application further provides a training device for a feature extraction model of multimedia resources, the device comprising:

[0095] The sample pair construction module is configured to construct a positive sample pair using multimedia resources accessed by the same account object in the multimedia resource set, wherein the access time of the two samples in the positive sample pair is taken from the same time window; and to construct a negative sample pair using multimedia resources that have not been accessed and multimedia resources that have been accessed in the multimedia resource set; wherein multimedia resources matched by the same type of query requests constitute the multimedia resource set, and query requests with a similarity higher than a similarity threshold constitute the same type of query requests;

[0096] The training module is configured to use the positive sample pairs and the negative sample pairs to train a feature extraction model, wherein the feature extraction model is used to extract similarities between multimedia resources.

[0097] Optionally, the feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer;

[0098] The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource;

[0099] The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource;

[0100] The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource;

[0101] The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

[0102] Optionally, the second feature is a parameter of the attention level of a multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to perform feature extraction on the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to perform feature extraction on the content of the first multimedia resource to obtain the first feature of the first multimedia resource;

[0103] The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

[0104] Optionally, the step of training a feature extraction model using the positive sample pair and the negative sample pair is performed, and the training module is specifically configured as follows:

[0105] Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair;

[0106] Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features;

[0107] Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs;

[0108] A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

[0109] Optionally, the device further comprises:

[0110] A candidate resource determination module is configured to obtain the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set;

[0111] A positive sample resource set determination module is configured to filter out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set;

[0112] The method of constructing positive sample pairs by using multimedia resources in a multimedia resource set accessed by the same account object is performed, and the sample pair construction module is specifically configured as follows:

[0113] For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

[0114] Optionally, the loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but has different signs.

[0115] Optionally, the loss function is used to minimize the distance between two samples in a positive sample pair and maximize the degree of difference between two samples in a negative sample pair.

[0116] Optionally, the loss function is used to minimize the following value:

[0117]

[0118] Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

[0119] Optionally, the device further comprises:

[0120] The multimedia resource set construction module is configured to construct the multimedia resource set based on the following method:

[0121] Determining a similarity between each of the plurality of query requests;

[0122] Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set;

[0123] The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

[0124] Optionally, the sample pair construction module is further configured to: for each query request in the query request set, if the last accessed multimedia resource associated with the query request has a specified operation record, then mark the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object as a positive sample pair.

[0125] Optionally, the multiple query requests satisfy at least one of the following conditions:

[0126] The query time interval is less than the first time duration;

[0127] Query request for the same account object.

[0128] Optionally, if the multiple query requests satisfy a query time interval less than the first duration or are query requests for the same account object, the similarity threshold used by the multiple query requests with a query time interval less than the first duration is less than the similarity threshold used by the query requests for the same account object.

[0129] Optionally, the specified operation record includes at least one of the following:

[0130] The playback duration is longer than the specified duration;

[0131] Like, forward, favorite, follow, and comment.

[0132] Optionally, the attention level parameter includes at least one of the following parameters:

[0133] Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

[0134] In a fourth aspect, the present application further provides a device for determining similarity of multimedia resources, the device comprising:

[0135] An acquisition module, configured to acquire a first multimedia resource and a second multimedia resource;

[0136] A feature expression extraction module, configured to respectively extract feature expressions of the first multimedia resource and the second multimedia resource using a feature extraction model;

[0137] a similarity determination module, configured to determine the similarity between the first multimedia resource and the second multimedia resource based on the feature expressions of each of the first multimedia resource and the second multimedia resource;

[0138] The feature extraction model is pre-trained based on positive sample pairs and negative samples, where:

[0139] Query requests with similarity higher than a similarity threshold constitute query requests of the same type; multimedia resources matched by query requests of the same type constitute the multimedia resource set;

[0140] The positive sample pair is constructed by using multimedia resources in the multimedia resource set that are accessed by the same account object, and the access time of two samples in the positive sample pair is taken from the same time window;

[0141] The negative sample pairs are constructed by using the multimedia resources that have not been accessed and the multimedia resources that have been accessed in the multimedia resource set.

[0142] Optionally, the feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer;

[0143] The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource;

[0144] The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource;

[0145] The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource;

[0146] The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

[0147] Optionally, the second feature is a parameter of the attention level of a multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to perform feature extraction on the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to perform feature extraction on the content of the first multimedia resource to obtain the first feature of the first multimedia resource;

[0148] The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

[0149] Optionally, the device further includes: a training module configured to train a feature extraction model using the positive sample pair and the negative sample pair based on the following method:

[0150] Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair;

[0151] Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features;

[0152] Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs;

[0153] A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

[0154] Optionally, the loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but has different signs.

[0155] Optionally, the device further comprises:

[0156] The multimedia resource set construction module is configured to construct the multimedia resource set based on the following method:

[0157] Determining a similarity between each of the plurality of query requests;

[0158] Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set;

[0159] The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

[0160] Optionally, the device further comprises:

[0161] The sample pair construction module is configured to, for each query request in the query request set, mark the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object as a positive sample pair if the last multimedia resource accessed associated with the query request has a specified operation record.

[0162] Optionally, the multiple query requests satisfy at least one of the following conditions:

[0163] The query time interval is less than the first time duration;

[0164] Query request for the same account object.

[0165] Optionally, if the multiple query requests satisfy a query time interval less than the first duration or are query requests for the same account object, the similarity threshold used by the multiple query requests with a query time interval less than the first duration is less than the similarity threshold used by the query requests for the same account object.

[0166] Optionally, the device further comprises:

[0167] A candidate resource determination module is configured to obtain the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set;

[0168] A positive sample resource set determination module is configured to filter out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set;

[0169] The sample pair construction module uses the multimedia resources accessed by the same account object in the multimedia resource set to construct positive sample pairs based on the following method:

[0170] For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

[0171] Optionally, the loss function is used to minimize the distance between two samples in a positive sample pair and maximize the degree of difference between two samples in a negative sample pair.

[0172] Optionally, the loss function is used to minimize the following value:

[0173]

[0174] Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

[0175] Optionally, the specified operation record includes at least one of the following:

[0176] The playback duration is longer than the specified duration;

[0177] Like, forward, favorite, follow, and comment.

[0178] Optionally, the attention level parameter includes at least one of the following parameters:

[0179] Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

[0180] In a fifth aspect, the present application further provides an electronic device, including:

[0181] processor;

[0182] a memory for storing instructions executable by the processor;

[0183] The processor is configured to execute the instructions to implement any method provided in the first aspect and / or the second aspect of the present application.

[0184] In the sixth aspect, an embodiment of the present application further provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any method provided in the first aspect and / or second aspect of the present application.

[0185] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any method provided in the first aspect and / or the second aspect of the present application.

[0186] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects: the same or similar multimedia resources can be searched through different query requests. Therefore, there is a certain similarity between the query requests for the same or similar multimedia resources. In other words, similar query requests correspond to similar multimedia resources. In order to better construct positive sample pairs, in the embodiments of the present application, the multimedia resources corresponding to similar query requests are screened using a time window. The reason why the time window can be used is that under the same or similar query requests, two access operations with similar access times are often accesses to extremely similar multimedia resources. Therefore, the positive sample pairs screened based on the time window have high practicality and accuracy.

[0187] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0188] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0189] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0190] Figure 2 One of the flow charts of a sample construction and training method of a feature extraction model for multimedia resources provided in one embodiment of the present application;

[0191] Figure 3 A second flowchart of a method for training a feature extraction model for multimedia resources provided in an embodiment of the present application;

[0192] Figure 4 A schematic diagram of the structure of a feature extraction model for multimedia resources provided in one embodiment of the present application;

[0193] Figure 5 A third flowchart of a method for training a feature extraction model for multimedia resources provided in an embodiment of the present application;

[0194] Figure 6 A flowchart of a method for determining similarity of multimedia resources provided in one embodiment of the present application;

[0195] Figure 7 A structural block diagram of a training device for a feature extraction model of multimedia resources provided in an embodiment of the present application;

[0196] Figure 8 A structural block diagram of a device for determining similarity of multimedia resources provided in an embodiment of the present application;

[0197] Fig. 9 The figure is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0198] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0199] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0200] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0201] (1) In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0202] (2) "and / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0203] (3) Servers provide services for terminals. The services provided include providing resources to terminals and storing terminal data. Servers correspond to applications installed on terminals and work in conjunction with applications on terminals.

[0204] (4) Terminal device, which can refer to software APP (Application) or client. It has a visual display interface and can interact with users; it corresponds to the server and provides local services to customers. For software applications, except for some applications that only run locally, they are generally installed on ordinary client terminals and need to cooperate with the server to run. After the development of the Internet, more commonly used applications include short video applications, email clients for sending and receiving emails, and instant messaging clients. For this type of application, there needs to be corresponding servers and service programs in the network to provide corresponding services, such as database services, configuration parameter services, etc. In this way, specific communication connections need to be established between the client terminal and the server to ensure the normal operation of the application.

[0205] (5) Multimedia resources. In the embodiments of the present application, multimedia resources refer to various types of resources that can be accessed on the network, such as audio resources, video resources, web resources, etc.

[0206] (6) Feature expression refers to the information used to describe the characteristics of multimedia resources. Feature expression can extract high-level features from multimedia resources for subsequent application use.

[0207] In view of the problem that the labeling of positive and negative sample pairs in the related art is complicated, difficult, and has low output, resulting in complicated training of the convolutional neural network model, the embodiments of the present application provide a solution.

[0208] The method provided in the embodiment of the present application is not only applicable to videos, but also applicable to feature extraction of any multimedia resources such as audio resources, web page resources, etc.

[0209] The inventive concept of the embodiment of the present application can be summarized as follows: based on the user's access record to multimedia resources, the same or similar access requests are obtained to construct an access request set, and then from the multimedia resource set corresponding to the access request set, multimedia resources accessed within the same time window are selected to construct positive sample pairs, and multimedia resources that have not been accessed and multimedia resources that have been accessed are selected to construct negative sample pairs. Thus, based on the same or similar access requests, the same or similar multimedia resources can be screened out, and then based on the user's access behavior, multimedia resources within the same time window can be screened from the same or similar multimedia resources to ensure the accuracy of the construction of the positive sample pairs. In addition, based on the multimedia resources that have not been accessed and the multimedia resources that have been accessed, automatic mining and labeling of negative sample pairs is achieved. Thus, the present application can simplify the labeling of positive and negative sample pairs.

[0210] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs. It should be noted that the user information involved in this application is obtained based on user authorization.

[0211] refer to Figure 1, which is a schematic diagram of an application scenario of a training method for a post-processing model of speech recognition results and a processing method provided in an embodiment of the present application. The application scenario includes multiple terminal devices 101 (including terminal device 101-1, terminal device 101-2, ... terminal device 101-n), and also includes a server 102. Among them, the terminal device 101 and the server 102 are connected via a wireless or wired network, and the terminal device 101 includes but is not limited to electronic devices such as desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, smart TVs, etc. The server 102 can be a server cluster or a cloud computing center composed of a single server, a plurality of servers. The server 102 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0212] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 1 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0213] The user can send a query request to the server 102 based on the terminal device 101, and the server 102 can search for relevant multimedia resources for the user based on the query request, and then push it to the terminal device 101 for display. The user can filter out multimedia resources that meet the search intention based on the displayed results for access. Thus, the server 102 can mine samples that meet the user's search intention corresponding to the user's query request based on the user's access operation of his query request, and further the multimedia resources that have not been accessed are samples that do not meet the user's search intention. Thus, the same or similar search intentions can be analyzed, and samples of the same or similar search intentions can be mined to construct positive sample pairs, and then samples that do not belong to the search intention and samples that belong to the search intention can construct negative sample pairs, thereby realizing automatic mining and labeling of positive and negative sample pairs. And the constructed positive and negative sample pairs meet the user's search intention, can be applied to different query requests and are not limited to the cognitive situation of the labeling personnel themselves, so the categories of the constructed positive and negative sample pairs can be richer, and then the training of the feature extraction model is more complete, which can enable the feature extraction model to learn more features and extract features that can express deep semantics.

[0214] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application.

[0215] For ease of understanding, the training method of the feature extraction model for multimedia resources and the method for users to determine the similarity between multimedia resources in this application are respectively described below.

[0216] 1. Training of feature extraction model

[0217] (1) Labeling of positive and negative sample pairs

[0218] The same or similar multimedia resources can be searched through different query requests. Therefore, there is a certain similarity between query requests for the same or similar multimedia resources. In other words, similar query requests correspond to similar multimedia resources. In order to better construct positive sample pairs, in the embodiment of the present application, the multimedia resources corresponding to similar query requests are screened using a time window. The reason why the time window can be used is that under the same or similar query requests, two access operations with similar access times are often accesses to extremely similar multimedia resources. Therefore, the positive sample pairs screened based on the time window have high practicality and accuracy.

[0219] In view of this, in the embodiment of the present application, query requests with similarity higher than the similarity threshold constitute the same type of query requests, and the matching multimedia resources of the same type of query requests constitute a multimedia resource set. During implementation, the multimedia resource set can be constructed based on the following method, such as Figure 2 As shown, the following steps are included:

[0220] In step 201, the similarity of each query request among a plurality of query requests is determined.

[0221] In the embodiment of the present application, feature extraction may be performed on the query request to obtain semantic features of the query request, and then the similarity between the query requests may be determined based on the distance between the semantic features.

[0222] The query request may be triggered based on a search keyword, and the query request for extracting semantic features may include the search keyword and may further include a query category. The query category is specified by the user. For example, the user searches for a specific category under the general category of TV series, such as costume drama, suspense drama, production year, actor, etc.

[0223] In step 202, at least two query requests with a similarity higher than a corresponding similarity threshold are obtained to obtain a query request set;

[0224] During implementation, in order to better mine positive sample pairs, the multiple query requests used to construct the query request set must meet at least one of the following conditions:

[0225] Condition 1: The query time interval is less than the first duration.

[0226] The query request set constructed by the query requests that meet the condition and whose similarity is higher than the similarity threshold can realize the modeling of the user's short-term interests.

[0227] If the embedding features of two query requests are highly similar, and the time interval between the two query requests is less than 30 minutes, such queries will be merged into one query request set session.

[0228] Condition 2: Query request for the same account object. This condition is used to model the user's long-term interests. This condition can be used to model the user's long-term interests and obtain positive sample pairs.

[0229] In implementation, the similarity threshold used in short-term interest modeling in condition 1 and the similarity threshold used in long-term interest modeling in condition 2 may be different. For example, the condition for short-term interest modeling may be looser, and the similarity threshold used is lower than the similarity threshold used for long-term interest modeling.

[0230] In specific implementation, for example, in condition 1, short-term interest modeling requires that the similarity of each query request in the same query request set is relatively high, and the similarity threshold used in long-term interest modeling is relatively high, which can reflect that it is the same query request of the same user.

[0231] In step 203, the multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

[0232] For example, the multimedia resources corresponding to query request 1 include a and b, the multimedia resources corresponding to query request 2 include b and c, and the multimedia resources corresponding to query request 3 include d and e. Then the multimedia resource set corresponding to query requests 1-3 includes a, b, c, d, and e.

[0233] Based on the multimedia resource set, we can build positive and negative samples to train the complete model, such as Figure 2 The steps include:

[0234] In step 204, multimedia resources in the multimedia resource set accessed by the same account object are used to construct positive sample pairs, wherein the access times of the two samples in the positive sample pair are taken from the same time window.

[0235] For example, both long-term interest modeling and short-term interest modeling can obtain corresponding query request sets. Each query request in the query request set obtains a corresponding query result, that is, the multimedia resource corresponding to the query request. Each query structure in the same query request set constitutes a multimedia resource set. Which multimedia resources are accessed in the query result corresponding to each query request is information that can be determined. Therefore, for the multimedia resources accessed in the multimedia resource set, that is, similar multimedia resources, in order to ensure a high degree of similarity, the multimedia resources accessed in the same time window are obtained to construct positive sample pairs.

[0236] For example, a sequence of multimedia resources is obtained by sorting according to access time, and according to the size of the time window, any two accessed multimedia resources in the same time window are obtained to construct a positive sample pair.

[0237] If the query time interval between query request A and query request B is less than the first duration, and the similarity is higher than the similarity threshold, the multimedia resources searched and displayed based on query request A and the multimedia resources searched and displayed based on query request B are shown in Table 1:

[0238]

[0239] Then, the sequence (a1, a2, b1, a3, b2) is sorted in the order of access time. The time window size is 2. For a1, the positive sample pairs include [a1, a2] and [a1, b1]. For a2, the positive sample pairs include [a1, a2], [a2, b1] and [a2, a3], and so on.

[0240] In some embodiments, in order to further ensure the accuracy of the constructed positive sample pairs, in the embodiments of the present application, the multimedia resource set corresponding to the query request set can be screened to select the effectively accessed multimedia resources to construct the positive sample pairs. Figure 3 The following steps are shown:

[0241] In step 301, the accessed multimedia resources in the multimedia resource set are obtained to obtain a candidate resource set.

[0242] For example, if the multimedia resources corresponding to query request A are a1, a2, a3 and a4, among which the first three are accessed, then a1, a2 and a3 are used to construct the candidate resource set, and a4 is used as the multimedia resource that has not been accessed.

[0243] In step 302, multimedia resources with designated operation records are screened out from the candidate resource set to obtain a positive sample resource set.

[0244] The designated operation records are used to filter out multimedia resources with effective access, also known as effective clicks. The designated operation records may include at least one of the following: a playback duration greater than a designated duration, and at least one of the following operation attributes: like, forward, favorite, follow, comment, etc.

[0245] For example, if short video a1 is accessed and the playback time is longer than the specified time, a1 has the specified operation record. Or if a1 is liked, converted, collected, followed or commented on after being accessed, a1 also has the specified operation record, and a1 is a validly accessed multimedia resource.

[0246] The multimedia resources that are effectively accessed in the candidate resource set are screened out to obtain a positive sample resource set, and then in step 303, for the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

[0247] Therefore, by screening positive samples with specified operation records, it is possible to screen out noise data of invalid accesses, thereby improving the accuracy of the constructed positive sample pairs.

[0248] In another embodiment, in order to increase the diversity and quantity of positive sample pairs, the multimedia resources that are satisfactorily accessed are defined in the embodiment of the present application. The multimedia resources that are satisfactorily accessed are: the multimedia resources that are last accessed in a query request. If the multimedia resource has a specified operation record, then the multimedia resource is a multimedia resource that is satisfactorily accessed. Then the multimedia resources that are satisfactorily accessed and the multimedia resources that are accessed / validly accessed by the same account object both constitute positive sample pairs.

[0249] In step 205, negative sample pairs are constructed using the multimedia resources that have not been accessed and the multimedia resources that have been accessed in the multimedia resource set.

[0250] It should be noted that the execution order of the operation of constructing a positive sample pair in step 204 and the operation of constructing a negative sample pair in step 205 is not limited, that is, step 204 can be executed first and then step 205, or step 205 can be executed first and then step 204, or step 204 and step 205 can be executed simultaneously, all of which are applicable to the embodiments of the present application.

[0251] (2) Parameter training process of feature extraction model

[0252] After the positive and negative sample pairs are obtained, in step 206 , the positive sample pairs and the negative sample pairs may be used to train a feature extraction model.

[0253] Among them, the structure of the feature extraction model to be trained is as follows Figure 4As shown: the feature extraction model to be trained is a multi-tower network structure, each tower includes a convolutional neural network and a first fully connected layer, the first fully connected layer of each tower is connected to the second fully connected layer, and the second fully connected layer is connected to the logistic regression layer to classify and predict whether the sample pair is a positive sample pair or a negative sample pair. Among them:

[0254] The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract a first feature of a first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract a first feature of a second multimedia resource;

[0255] The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource;

[0256] The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource;

[0257] The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

[0258] The convolutional neural network is used to extract features of the content of multimedia resources and the parameter Ctr of the degree of attention of multimedia resources to obtain the first feature, or to extract features of the content of multimedia resources to obtain the first feature. The parameter of the degree of attention is also referred to as the second feature hereinafter, and may include at least one of the following parameters: click index parameters such as click volume or click rate, attention index parameters such as attention volume and attention rate, like index parameters such as like volume or like rate, comment index parameters such as the total number of comments or comment rate, the comment rate can be understood as the ratio between the users who posted the comments and the users who visited; forwarding index parameters such as forwarding volume or forwarding rate, the forwarding rate can be understood as the ratio of the number of forwarding users to the number of visiting users, etc.

[0259] The first fully connected layer is used to perform feature extraction on the first feature and the attention level parameter to obtain a feature expression of the multimedia resource.

[0260] Figure 4 FIG. 2 shows two tower-structured feature extraction models to be trained. Figure 4 In the embodiment, the first fully connected layer and the second fully connected layer may each include one or more fully connected layers. Figure 4 The training process includes Figure 5 Steps shown:

[0261] In step 501, the feature expressions of the first multimedia resource and the second multimedia resource in any sample pair are obtained, that is, the feature expressions output by the first fully connected layer.

[0262] In step 502, a second fully connected layer is used to extract feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features, that is, features input to the softmax logistic regression layer.

[0263] In step 503, the classification features are classified to obtain the predicted category of any sample pair, and the classification categories to be divided include positive sample pairs and negative sample pairs.

[0264] Softmax performs classification processing based on classification features to determine whether the two input videos are positive sample pairs or negative sample pairs. Based on the predicted category, the labeled category can be compared. In step 504, the predicted category and the labeled category are used to determine the loss value, and the model parameters of the feature extraction model are adjusted based on the loss value.

[0265] During implementation, the loss function used in training the feature extraction model includes the feature similarity of the two samples in the positive sample pair and the degree of difference between the two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but with different signs. Therefore, a simple formula can be used to determine the loss value. This allows the loss of both the positive and negative sample pairs to be included in the calculation of the loss. The loss function is used to minimize the distance between the two samples in the positive sample pair and maximize the degree of difference between the two samples in the negative sample pair. Therefore, model training can be performed based on the ability to classify similar multimedia resources and to divide different multimedia resources as much as possible, thereby improving the accuracy of the model in extracting features of multimedia resources.

[0266] During implementation, the loss function used is shown in formula (1), and the training goal is to make the result of formula 1 as small as possible.

[0267]

[0268] Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

[0269] It should be noted that, taking video as an example, the video can be sampled and the sampled image sequence can be input into the feature extraction model, or a single frame image of the video can be input into the feature extraction model. When the image sequence is input, the final feature expression is the feature expression of the entire image sequence. When a single frame image is input, the feature expression of each frame image of the video can be obtained, and then the feature expression of the image sequence can be obtained by splicing, or each frame sampled image of the first video and each frame sampled image of the other video are classified and predicted respectively, and finally the predicted category of each frame sampled image is compared with the corresponding sample category to obtain the loss value.

[0270] 2. Determine similarity based on feature extraction model

[0271] After the feature extraction model is trained, the similarity of multimedia resources can be determined based on the model. It should be noted that the feature extraction model only needs to be trained once, and the model can be reused to determine similarity later. Figure 6 As shown, it is a flow chart of a method for determining similarity using the aforementioned feature extraction model in an embodiment of the present application, comprising the following steps:

[0272] In step 601, a first multimedia resource and a second multimedia resource are obtained;

[0273] In step 602, feature extraction models are used to extract feature expressions of the first multimedia resource and the second multimedia resource respectively;

[0274] In step 603, the similarity between the first multimedia resource and the second multimedia resource is determined based on the feature expressions of each of the first multimedia resource and the second multimedia resource.

[0275] During implementation, the cosine distance between two embedding features is used to obtain the similarity between two multimedia resources.

[0276] Based on the same inventive concept, the present application provides a training device for a feature extraction model of multimedia resources, such as Figure 7 As shown, the device 700 includes:

[0277] The sample pair construction module 701 is configured to construct a positive sample pair using multimedia resources accessed by the same account object in the multimedia resource set, wherein the access time of the two samples in the positive sample pair is taken from the same time window; and construct a negative sample pair using multimedia resources that have not been accessed and multimedia resources that have been accessed in the multimedia resource set; wherein multimedia resources matched by the same type of query requests constitute the multimedia resource set, and query requests with a similarity higher than a similarity threshold constitute the same type of query requests;

[0278] The training module 702 is configured to use the positive sample pairs and the negative sample pairs to train a feature extraction model, wherein the feature extraction model is used to extract similarities between multimedia resources.

[0279] Optionally, the feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer;

[0280] The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource;

[0281] The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource;

[0282] The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource;

[0283] The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

[0284] Optionally, the second feature is a parameter of the attention level of a multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to perform feature extraction on the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to perform feature extraction on the content of the first multimedia resource to obtain the first feature of the first multimedia resource;

[0285] The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

[0286] Optionally, the step of training a feature extraction model using the positive sample pair and the negative sample pair is performed, and the training module is specifically configured as follows:

[0287] Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair;

[0288] Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features;

[0289] Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs;

[0290] A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

[0291] Optionally, the device further comprises:

[0292] A candidate resource determination module is configured to obtain the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set;

[0293] A positive sample resource set determination module is configured to filter out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set;

[0294] The method of constructing positive sample pairs by using multimedia resources in a multimedia resource set accessed by the same account object is performed, and the sample pair construction module is specifically configured as follows:

[0295] For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

[0296] Optionally, the loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but has different signs.

[0297] Optionally, the loss function is used to minimize the distance between two samples in a positive sample pair and maximize the degree of difference between two samples in a negative sample pair.

[0298] Optionally, the loss function is used to minimize the following value:

[0299]

[0300] Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

[0301] Optionally, the device further comprises:

[0302] The multimedia resource set construction module is configured to construct the multimedia resource set based on the following method:

[0303] Determining a similarity between each of the plurality of query requests;

[0304] Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set;

[0305] The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

[0306] Optionally, the sample pair construction module is further configured to: for each query request in the query request set, if the last accessed multimedia resource associated with the query request has a specified operation record, then mark the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object as a positive sample pair.

[0307] Optionally, the multiple query requests satisfy at least one of the following conditions:

[0308] The query time interval is less than the first time duration;

[0309] Query request for the same account object.

[0310] Optionally, if the multiple query requests satisfy a query time interval less than the first duration or are query requests for the same account object, the similarity threshold used by the multiple query requests with a query time interval less than the first duration is less than the similarity threshold used by the query requests for the same account object.

[0311] Optionally, the specified operation record includes at least one of the following:

[0312] The playback duration is longer than the specified duration;

[0313] Like, forward, favorite, follow, and comment.

[0314] Optionally, the attention level parameter includes at least one of the following parameters:

[0315] Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

[0316] Based on the same inventive concept, the present application embodiment also provides a device for determining the similarity of multimedia resources, such as Figure 8 As shown, the device 800 includes:

[0317] An acquisition module 801 is configured to acquire a first multimedia resource and a second multimedia resource;

[0318] A feature expression extraction module 802 is configured to respectively extract feature expressions of the first multimedia resource and the second multimedia resource using a feature extraction model;

[0319] A similarity determination module 803 is configured to determine the similarity between the first multimedia resource and the second multimedia resource based on the feature expressions of the first multimedia resource and the second multimedia resource respectively;

[0320] The feature extraction model is pre-trained based on positive sample pairs and negative samples, where:

[0321] Query requests with similarity higher than a similarity threshold constitute query requests of the same type; multimedia resources matched by query requests of the same type constitute the multimedia resource set;

[0322] The positive sample pair is constructed by using multimedia resources in the multimedia resource set that are accessed by the same account object, and the access time of two samples in the positive sample pair is taken from the same time window;

[0323] The negative sample pairs are constructed by using the multimedia resources that have not been accessed and the multimedia resources that have been accessed in the multimedia resource set.

[0324] Optionally, the feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer;

[0325] The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource;

[0326] The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource;

[0327] The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource;

[0328] The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

[0329] Optionally, the second feature is a parameter of the attention level of a multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to perform feature extraction on the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to perform feature extraction on the content of the first multimedia resource to obtain the first feature of the first multimedia resource;

[0330] The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

[0331] Optionally, the device further includes: a training module configured to train a feature extraction model using the positive sample pair and the negative sample pair based on the following method:

[0332] Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair;

[0333] Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features;

[0334] Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs;

[0335] A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

[0336] Optionally, the loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but has different signs.

[0337] Optionally, the device further comprises:

[0338] The multimedia resource set construction module is configured to construct the multimedia resource set based on the following method:

[0339] Determining a similarity between each of the plurality of query requests;

[0340] Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set;

[0341] The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

[0342] Optionally, the device further comprises:

[0343] The sample pair construction module is configured to, for each query request in the query request set, mark the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object as a positive sample pair if the last multimedia resource accessed associated with the query request has a specified operation record.

[0344] Optionally, the multiple query requests satisfy at least one of the following conditions:

[0345] The query time interval is less than the first time duration;

[0346] Query request for the same account object.

[0347] Optionally, if the multiple query requests satisfy a query time interval less than the first duration or are query requests for the same account object, the similarity threshold used by the multiple query requests with a query time interval less than the first duration is less than the similarity threshold used by the query requests for the same account object.

[0348] Optionally, the device further comprises:

[0349] A candidate resource determination module is configured to obtain the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set;

[0350] A positive sample resource set determination module is configured to filter out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set;

[0351] The sample pair construction module uses the multimedia resources accessed by the same account object in the multimedia resource set to construct positive sample pairs based on the following method:

[0352] For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

[0353] Optionally, the loss function is used to minimize the distance between two samples in a positive sample pair and maximize the degree of difference between two samples in a negative sample pair.

[0354] Optionally, the loss function is used to minimize the following value:

[0355]

[0356] Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

[0357] Optionally, the specified operation record includes at least one of the following:

[0358] The playback duration is longer than the specified duration;

[0359] Like, forward, favorite, follow, and comment.

[0360] Optionally, the attention level parameter includes at least one of the following parameters:

[0361] Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

[0362] After introducing the model training method and processing method of multimedia resources and related devices according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0363] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0364] In some possible implementations, the electronic device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the multimedia information editing method according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute steps such as a model training method and a similarity determination method for multimedia resources.

[0365] Refer to the following Fig. 9 The electronic device 130 according to this embodiment of the present application is described. Fig. 9 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0366] like Fig. 9 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0367] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0368] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0369] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0370] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0371] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 132 including instructions, and the instructions can be executed by the processor 131 of the device 700 or the processor 131 of the device 800 to complete the training method and post-processing method of the post-processing model of the speech recognition result. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0372] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by the processor 131, implements any one of the model training method and similarity determination method for multimedia resources provided in the present application.

[0373] In an exemplary embodiment, various aspects of the model training method and similarity determination method for multimedia resources provided by the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the model training method and similarity determination method for multimedia resources according to various exemplary embodiments of the present application described above in this specification.

[0374] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0375] The program product of the model training method and similarity determination method for multimedia resources of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, an apparatus or a device or used in combination with it.

[0376] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0377] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0378] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0379] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0380] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0381] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0382] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0383] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0384] These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0385] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0386] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for training a feature extraction model for multimedia resources, characterized in that: The method comprises: A positive sample pair is constructed by using multimedia resources in a multimedia resource set that are accessed by the same account object, wherein the access time of two samples in the positive sample pair is taken from the same time window; the constructing of the positive sample pair includes: obtaining a sequence of multimedia resources by sorting the access time, and obtaining any two accessed multimedia resources in the same time window to construct a positive sample pair according to the time window size; and, A negative sample pair is constructed using multimedia resources that have not been accessed and multimedia resources that have been accessed in the multimedia resource set; wherein multimedia resources matched by similar query requests constitute the multimedia resource set, and query requests with similarity higher than a similarity threshold constitute the similar query requests; Using the positive sample pair and the negative sample pair to train a feature extraction model, wherein the feature extraction model is used to extract similarities between multimedia resources; The method further comprises: Acquire the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set; Screening out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set; The method of constructing positive sample pairs using multimedia resources in a multimedia resource set accessed by the same account object specifically includes: For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

2. The method according to claim 1, characterized in that The feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer; The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource; The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource; The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource; The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

3. The method according to claim 2, characterized in that The second feature is a parameter of the attention level of the multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to extract features from the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to extract features from the content of the first multimedia resource to obtain the first feature of the first multimedia resource; The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

4. The method according to claim 2 or 3, characterized in that: The adopting the positive sample pair and the negative sample pair to train the feature extraction model comprises: Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair; Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features; Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs; A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

5. The method according to claim 1, characterized in that: The loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but with different signs.

6. The method according to claim 5, characterized in that The loss function is used to minimize the distance between two samples in a positive sample pair and maximize the difference between two samples in a negative sample pair.

7. The method according to claim 5 or 6, characterized in that: The loss function is used to minimize the following value: Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

8. The method according to claim 1, characterized in that: The method further comprises: The multimedia resource set is constructed based on the following method: Determining a similarity between each of the plurality of query requests; Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set; The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

9. The method according to claim 8, characterized in that The method further comprises: For each query request in the query request set, if the last accessed multimedia resource associated with the query request has a specified operation record, the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object are marked as a positive sample pair.

10. The method according to claim 8, characterized in that The multiple query requests satisfy at least one of the following conditions: The query time interval is less than the first time duration; Query request for the same account object.

11. The method according to claim 10, characterized in that If the multiple query requests satisfy the query time interval being less than the first duration or the query requests of the same account object, then the similarity threshold used by the multiple query requests with the query time interval being less than the first duration is less than the similarity threshold used by the query requests of the same account object.

12. The method according to claim 1 or 9, characterized in that: The specified operation record includes at least one of the following: The playback duration is longer than the specified duration; Like, forward, favorite, follow, and comment.

13. The method according to claim 3, characterized in that The attention level parameter includes at least one of the following parameters: Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

14. A method for determining similarity of multimedia resources, characterized in that: The method comprises: Acquire a first multimedia resource and a second multimedia resource; Using a feature extraction model to extract feature expressions of the first multimedia resource and the second multimedia resource respectively; Determining the similarity between the first multimedia resource and the second multimedia resource based on the respective feature expressions of the first multimedia resource and the second multimedia resource; The feature extraction model is pre-trained based on positive sample pairs and negative sample pairs, where: Query requests with similarity higher than a similarity threshold constitute query requests of the same type; multimedia resources matched by query requests of the same type constitute the multimedia resource set; The positive sample pair is constructed by using multimedia resources in the multimedia resource set that are accessed by the same account object, and the access time of two samples in the positive sample pair is taken from the same time window; constructing the positive sample pair includes: obtaining a sequence of multimedia resources according to the access time, and obtaining any two accessed multimedia resources in the same time window according to the time window size to construct the positive sample pair; The negative sample pairs are constructed by using the multimedia resources that have not been accessed and the multimedia resources that have been accessed in the multimedia resource set; The method further comprises: Acquire the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set; Screening out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set; The multimedia resources accessed by the same account object in the multimedia resource set are used to construct positive sample pairs, including: For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

15. The method according to claim 14, characterized in that The feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer; The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource; The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource; The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource; The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

16. The method according to claim 15, characterized in that The second feature is a parameter of the attention level of the multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to extract features from the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to extract features from the content of the first multimedia resource to obtain the first feature of the first multimedia resource; The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

17. The method according to claim 15 or 16, characterized in that The positive sample pair and the negative sample pair are used to train a feature extraction model, comprising: Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair; Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features; Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs; A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

18. The method according to claim 14, characterized in that The loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but with different signs.

19. The method according to claim 14, characterized in that The method further comprises: The multimedia resource set is constructed based on the following method: Determining a similarity between each of the plurality of query requests; Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set; The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

20. The method according to claim 19, characterized in that The method further comprises: For each query request in the query request set, if the last accessed multimedia resource associated with the query request has a specified operation record, the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object are marked as a positive sample pair.

21. The method according to claim 19, characterized in that The multiple query requests satisfy at least one of the following conditions: The query time interval is less than the first time duration; Query request for the same account object.

22. The method according to claim 21, characterized in that If the multiple query requests satisfy the query time interval being less than the first duration or the query requests of the same account object, then the similarity threshold used by the multiple query requests with the query time interval being less than the first duration is less than the similarity threshold used by the query requests of the same account object.

23. The method according to claim 18, characterized in that The loss function is used to minimize the distance between two samples in a positive sample pair and maximize the difference between two samples in a negative sample pair.

24. The method according to claim 18 or 23, characterized in that The loss function is used to minimize the following value: Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

25. The method according to claim 20 or 14, characterized in that The specified operation record includes at least one of the following: The playback duration is longer than the specified duration; Like, forward, favorite, follow, and comment.

26. The method according to claim 16, characterized in that The attention level parameter includes at least one of the following parameters: Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

27. A training device for a feature extraction model of multimedia resources, characterized in that: The device comprises: The sample pair construction module is configured to construct a positive sample pair using multimedia resources accessed by the same account object in a multimedia resource set, wherein the access time of two samples in the positive sample pair is taken from the same time window; the construction of the positive sample pair is configured to: obtain a sequence of multimedia resources by sorting the access time, and obtain any two accessed multimedia resources in the same time window according to the time window size to construct a positive sample pair; and construct a negative sample pair using multimedia resources that have not been accessed and multimedia resources that have been accessed in the multimedia resource set; wherein multimedia resources matched by the same type of query requests constitute the multimedia resource set, and query requests with a similarity higher than a similarity threshold constitute the same type of query requests; A training module is configured to train a feature extraction model using the positive sample pair and the negative sample pair, wherein the feature extraction model is used to extract similarities between multimedia resources; The device also includes: A candidate resource determination module is configured to obtain the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set; A positive sample resource set determination module is configured to filter out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set; The method of constructing positive sample pairs by using multimedia resources in a multimedia resource set accessed by the same account object is performed, and the sample pair construction module is specifically configured as follows: For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

28. The device according to claim 27, characterized in that The feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer; The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource; The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource; The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource; The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

29. The device according to claim 28, characterized in that The second feature is a parameter of the attention level of the multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to extract features from the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to extract features from the content of the first multimedia resource to obtain the first feature of the first multimedia resource; The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

30. The device according to claim 28 or 29, characterized in that The training of the feature extraction model using the positive sample pair and the negative sample pair is performed, and the training module is specifically configured as follows: Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair; Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features; Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs; A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

31. The device according to claim 27, characterized in that The loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but with different signs.

32. The device according to claim 31, characterized in that The loss function is used to minimize the distance between two samples in a positive sample pair and maximize the difference between two samples in a negative sample pair.

33. The device according to claim 31 or 32, characterized in that The loss function is used to minimize the following value: Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

34. The device according to claim 27, characterized in that The device also includes: The multimedia resource set construction module is configured to construct the multimedia resource set based on the following method: Determining a similarity between each of the plurality of query requests; Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set; The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

35. The device according to claim 34, characterized in that The sample pair construction module is further configured to: for each query request in the query request set, if the last accessed multimedia resource associated with the query request has a specified operation record, mark the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object as a positive sample pair.

36. The device according to claim 34, characterized in that The multiple query requests satisfy at least one of the following conditions: The query time interval is less than the first time duration; Query request for the same account object.

37. The device according to claim 36, characterized in that If the multiple query requests satisfy the query time interval being less than the first duration or the query requests of the same account object, then the similarity threshold used by the multiple query requests with the query time interval being less than the first duration is less than the similarity threshold used by the query requests of the same account object.

38. The device according to claim 27 or 35, characterized in that The specified operation record includes at least one of the following: The playback duration is longer than the specified duration; Like, forward, favorite, follow, and comment.

39. The device according to claim 29, characterized in that The attention level parameter includes at least one of the following parameters: Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

40. A device for determining similarity of multimedia resources, characterized in that: The device comprises: An acquisition module, configured to acquire a first multimedia resource and a second multimedia resource; A feature expression extraction module, configured to respectively extract feature expressions of the first multimedia resource and the second multimedia resource using a feature extraction model; a similarity determination module, configured to determine the similarity between the first multimedia resource and the second multimedia resource based on the feature expressions of each of the first multimedia resource and the second multimedia resource; The feature extraction model is pre-trained based on positive sample pairs and negative sample pairs, where: Query requests with similarity higher than a similarity threshold constitute query requests of the same type; multimedia resources matched by query requests of the same type constitute the multimedia resource set; The positive sample pair is constructed by using the multimedia resources in the multimedia resource set that are accessed by the same account object, and the access time of the two samples in the positive sample pair is taken from the same time window; constructing the positive sample pair is configured as follows: obtaining a sequence of multimedia resources by sorting the access time, and obtaining any two accessed multimedia resources in the same time window to construct the positive sample pair according to the time window size; The negative sample pairs are constructed by using the multimedia resources that have not been accessed and the multimedia resources that have been accessed in the multimedia resource set; The device also includes: A candidate resource determination module is configured to obtain the accessed multimedia resources in the multimedia resource set to obtain a candidate resource set; A positive sample resource set determination module is configured to filter out multimedia resources with specified operation records from the candidate resource set to obtain a positive sample resource set; The sample pair construction module uses the multimedia resources accessed by the same account object in the multimedia resource set to construct positive sample pairs based on the following method: For the positive sample resource set, positive sample pairs are constructed using multimedia resources accessed by the same account object.

41. The device according to claim 40, characterized in that The feature extraction model is a dual-tower network structure, wherein each tower includes a convolutional neural network and a first fully connected layer; The convolutional neural network of the first tower network structure in the dual-tower network structure is used to extract the first feature of the first multimedia resource; the convolutional neural network of the second tower network structure in the dual-tower network structure is used to extract the first feature of the second multimedia resource; The first fully connected layer of the first tower network structure is used to extract the first feature of the first multimedia resource and the second feature of the first multimedia resource to obtain a feature expression of the first multimedia resource; The first fully connected layer of the second tower network structure is used to extract the first feature of the second multimedia resource and the second feature of the second multimedia resource to obtain a feature expression of the second multimedia resource; The feature expression of the first multimedia resource and the feature expression of the second multimedia resource are used to determine the similarity between the first multimedia resource and the second multimedia resource.

42. The device according to claim 41, characterized in that The second feature is a parameter of the attention level of the multimedia resource, and the convolutional neural network of the first tower network structure is specifically used to extract features from the content of the first multimedia resource and the parameter of the attention level of the first multimedia resource to obtain the first feature of the first multimedia resource, or to extract features from the content of the first multimedia resource to obtain the first feature of the first multimedia resource; The convolutional neural network of the second tower network structure is specifically used to extract features from the content of the second multimedia resource and the attention level parameter of the second multimedia resource to obtain the first feature of the second multimedia resource, or to extract features from the content of the second multimedia resource to obtain the first feature of the second multimedia resource.

43. The device according to claim 41 or 42, characterized in that The apparatus further includes: a training module configured to train a feature extraction model using the positive sample pair and the negative sample pair based on the following method: Obtaining respective feature expressions of a first multimedia resource and a second multimedia resource in any sample pair; Using a second fully connected layer to extract features from feature expressions of the first multimedia resource and the second multimedia resource to obtain classification features; Performing classification processing on the classification features to obtain a predicted category of any sample pair, wherein the classification categories to be classified include positive sample pairs and negative sample pairs; A loss value is determined using the predicted category and the labeled category, and a model parameter of the feature extraction model is adjusted based on the loss value.

44. The device according to claim 40, characterized in that The loss function used in training the feature extraction model includes the feature similarity of two samples in the positive sample pair and the degree of difference between two samples in the negative sample pair, and the expression of the feature similarity of the two samples in the positive sample pair is the same as the expression of the degree of difference between the two samples in the negative sample pair, but with different signs.

45. The device according to claim 40, characterized in that The device also includes: The multimedia resource set construction module is configured to construct the multimedia resource set based on the following method: Determining a similarity between each of the plurality of query requests; Obtain at least two query requests whose similarity is higher than a corresponding similarity threshold to obtain a query request set; The multimedia resource set is constructed by using the multimedia resources corresponding to each query request in the query request set.

46. ​​The device according to claim 45, characterized in that The device also includes: The sample pair construction module is configured to, for each query request in the query request set, mark the multimedia resource and any multimedia resource in the multimedia resource set accessed by the same account object as a positive sample pair if the last multimedia resource accessed associated with the query request has a specified operation record.

47. The device according to claim 45, characterized in that The multiple query requests satisfy at least one of the following conditions: The query time interval is less than the first time duration; Query request for the same account object.

48. The device according to claim 47, characterized in that If the multiple query requests satisfy the query time interval being less than the first duration or the query requests of the same account object, then the similarity threshold used by the multiple query requests with the query time interval being less than the first duration is less than the similarity threshold used by the query requests of the same account object.

49. The device according to claim 44, characterized in that The loss function is used to minimize the distance between two samples in a positive sample pair and maximize the difference between two samples in a negative sample pair.

50. The device according to claim 44 or 49, characterized in that The loss function is used to minimize the following value: Among them, Dp indicates that sample l and sample c are a positive sample pair, Dn indicates that sample l and sample c are a negative sample pair, Vc is the feature expression of sample c, and Vl is the feature expression of sample l.

51. The device according to claim 46 or 40, characterized in that The specified operation record includes at least one of the following: The playback duration is longer than the specified duration; Like, forward, favorite, follow, and comment.

52. The device according to claim 42, characterized in that The attention level parameter includes at least one of the following parameters: Click volume index parameters, attention volume index parameters, like volume index parameters, comment volume index parameters, and forwarding volume index parameters.

53. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as described in any one of claims 1-26.

54. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 26.

55. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 26 is implemented.

Images