Feature extraction model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a feature extraction model training method and device, electronic equipment and storage medium, belonging to the technical field of computer. The method comprises: determining a plurality of sample resource categories corresponding to each geographical area based on resource label information in a resource database corresponding to each geographical area; obtaining associated data of sample multimedia resources under each sample resource category based on resource identifiers corresponding to the resource label information in the resource database corresponding to a plurality of geographical areas; and training a to-be-trained feature extraction model based on the sample resource categories corresponding to a plurality of geographical areas and the associated data of sample multimedia resources under each sample resource category, to obtain a target feature extraction model. In the present disclosure, the target feature extraction model has the ability to represent multimedia resources in the resource database corresponding to a plurality of geographical areas, and there is no need to train a feature extraction model for each geographical area, thereby saving the human cost required for training the model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a feature extraction model training method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of multimedia technology, multimedia resources gradually become an indispensable entertainment means in people's life around the world. In the multimedia resource scene, the multimedia resources are usually recommended to users by using the resource features corresponding to the multimedia resources, or the multimedia resources are classified or recalled by using the resource features corresponding to the multimedia resources. Due to the diversification of multimedia resources in the resource database corresponding to different geographical regions, a feature extraction model is usually trained for each geographical region, so as to obtain the resource features corresponding to the multimedia resources in the resource database corresponding to each geographical region based on the feature extraction model corresponding to each geographical region. However, this method needs to train multiple feature extraction models, which wastes a lot of human cost and has low training efficiency. SUMMARY

[0003] The present disclosure provides a feature extraction model training method and device, electronic equipment and storage medium, which can save human cost and improve the training efficiency of the feature extraction model. The technical solutions of the present disclosure are as follows:

[0004] According to an aspect of an embodiment of the present disclosure, a feature extraction model training method is provided, comprising:

[0005] determining a plurality of sample resource categories corresponding to each geographical region based on resource label information in the resource database corresponding to each geographical region;

[0006] obtaining associated data of sample multimedia resources under each sample resource category based on resource identifiers corresponding to the resource label information in the resource database corresponding to a plurality of geographical regions, the resource identifier indicating the sample multimedia resources, and the associated data including at least one of sample images, sample texts or sample audios;

[0007] training a to-be-trained feature extraction model based on the sample resource categories corresponding to the plurality of geographical regions and the associated data of the sample multimedia resources under each sample resource category, to obtain a target feature extraction model; the target feature extraction model is used to obtain resource features corresponding to target multimedia resources.

[0008] In some embodiments, the determining the plurality of sample resource categories corresponding to each geographical region based on the resource label information in the resource database corresponding to each geographical region comprises:

[0009] obtaining a plurality of resource label information in the resource database corresponding to the geographical region;

[0010] Based on the multiple resource tag information, a tag coefficient is determined for each resource tag information, and the tag coefficient indicates the importance of the resource tag information;

[0011] Based on the tag coefficients of the multiple resource tag information, select target resource tag information whose tag coefficients belong to the target interval from the multiple resource tag information;

[0012] Based on the tag information of each target resource, a sample resource category is determined to obtain multiple sample resource categories corresponding to the geographical region.

[0013] In some embodiments, determining a sample resource category based on each target resource tag information to obtain multiple sample resource categories corresponding to the geographic region includes:

[0014] The multiple geographical regions are sorted according to their corresponding region identifiers;

[0015] According to the sorting order of the multiple geographical regions, a resource category identifier is generated for the target resource tag information corresponding to each geographical region, so as to obtain multiple resource category identifiers corresponding to each geographical region. The multiple resource category identifiers corresponding to the geographical region indicate multiple sample resource categories corresponding to the geographical region. Different geographical regions have different resource category identifiers.

[0016] In some embodiments, training the feature extraction model to obtain the target feature extraction model based on the sample resource categories corresponding to the plurality of geographical regions and the associated data of sample multimedia resources under each sample resource category includes:

[0017] For the associated data of sample multimedia resources under any sample resource category, the associated data is processed based on the feature extraction model to be trained to obtain the sample resource features corresponding to the sample multimedia resources.

[0018] The sample resource features corresponding to the sample multimedia resources are subjected to category prediction processing to obtain the predicted resource category to which the sample multimedia resources belong;

[0019] Based on the sample resource category and the predicted resource category, the feature extraction model to be trained is trained to obtain the target feature extraction model.

[0020] In some embodiments, processing the associated data based on the feature extraction model to be trained to obtain the sample resource features corresponding to the sample multimedia resources includes:

[0021] When the associated data includes at least two of the sample image, the sample text, or the sample audio, feature extraction is performed on the associated data based on the feature extraction model to be trained to obtain data features, wherein the data features include at least two of the image features corresponding to the sample image, the text features corresponding to the sample text, or the audio features corresponding to the sample audio.

[0022] Based on the feature extraction model to be trained, the features contained in the data features are fused to obtain the multi-modal features corresponding to the sample multimedia resources, and the multi-modal features are the sample resource features corresponding to the sample multimedia resources.

[0023] In some embodiments, the sample multimedia resource is a sample video, and the associated data includes the sample image, which includes a cover image and multiple video frames; the step of processing the associated data based on the feature extraction model to be trained to obtain the sample resource features corresponding to the sample multimedia resource includes:

[0024] Based on the feature extraction model to be trained, features are extracted from the cover image and each video frame respectively to obtain the cover features corresponding to the cover image and the video frame features corresponding to each video frame;

[0025] Based on the feature extraction model to be trained, the cover feature and multiple video frame features are fused to obtain the visual features corresponding to the sample video;

[0026] The cover feature and the visual feature are determined as the sample resource features corresponding to the sample video.

[0027] In some embodiments, the predicted resource category includes a first resource category and a second resource category, wherein the first resource category is obtained by performing category prediction processing on the cover features; the second resource category is obtained by performing category prediction processing on the visual features; and training the feature extraction model to be trained based on the sample resource category and the predicted resource category to obtain the target feature extraction model includes:

[0028] Based on the sample resource category and the first resource category, a first loss value is determined;

[0029] Based on the sample resource category and the second resource category, a second loss value is determined;

[0030] Based on the sum of the first loss value and the second loss value, the feature extraction model to be trained is trained to obtain the target feature extraction model.

[0031] In some embodiments, the predicted resource category further includes a third resource category, and the sample resource features corresponding to the sample multimedia resources further include multimodal features. The multimodal features are obtained by fusing at least two of image features, text features, or audio features. The image features are features corresponding to sample images in the associated data, the text features are features corresponding to sample text in the associated data, and the audio features are features corresponding to sample audio in the associated data. The method further includes:

[0032] The multi-modal features are subjected to category prediction processing to obtain the third resource category;

[0033] Based on the sample resource category and the third resource category, a third loss value is determined;

[0034] The step of training the feature extraction model to be trained based on the sum of the first loss value and the second loss value to obtain the target feature extraction model includes:

[0035] The target feature extraction model is trained based on the sum of the first loss value, the second loss value, and the third loss value.

[0036] In some embodiments, after training the feature extraction model to be trained based on the sample resource categories corresponding to the plurality of geographical regions and the associated data of sample multimedia resources under each sample resource category to obtain the target feature extraction model, the method further includes:

[0037] Obtain associated data of the target multimedia resource, wherein the associated data of the target multimedia resource includes at least one of images, text, or audio;

[0038] Based on the target feature extraction model, the associated data of the target multimedia resource is processed to obtain the resource features corresponding to the target multimedia resource;

[0039] Based on the resource characteristics corresponding to the target multimedia resource, the target multimedia resource is recommended.

[0040] According to another aspect of the embodiments of this disclosure, a feature extraction model training apparatus is provided, comprising:

[0041] The determining unit is configured to perform an operation based on resource tag information in the resource database corresponding to each geographic region to determine multiple sample resource categories corresponding to each geographic region.

[0042] The acquisition unit is configured to execute resource identifiers corresponding to resource tag information in a resource database corresponding to multiple geographical regions, and acquire associated data of sample multimedia resources under each sample resource category. The resource identifiers indicate the sample multimedia resources, and the associated data includes at least one of sample images, sample text, or sample audio.

[0043] The training unit is configured to train the feature extraction model to be trained based on the sample resource categories corresponding to the multiple geographical regions and the associated data of sample multimedia resources under each sample resource category, so as to obtain the target feature extraction model; the target feature extraction model is used to obtain the resource features corresponding to the target multimedia resources.

[0044] In some embodiments, the determining unit is configured to perform the following actions: acquiring multiple resource tag information from a resource database corresponding to the geographic region; determining a tag coefficient for each resource tag information based on the multiple resource tag information, wherein the tag coefficient indicates the importance of the resource tag information; selecting target resource tag information whose tag coefficient belongs to a target interval from the multiple resource tag information based on the tag coefficients of the multiple resource tag information; and determining a sample resource category based on each target resource tag information to obtain multiple sample resource categories corresponding to the geographic region.

[0045] In some embodiments, the determining unit is configured to sort the plurality of geographic regions according to the region identifiers corresponding to the plurality of geographic regions; and generate a resource category identifier for the target resource tag information corresponding to each geographic region according to the sorting order of the plurality of geographic regions, so as to obtain a plurality of resource category identifiers corresponding to each geographic region. The plurality of resource category identifiers corresponding to the geographic region indicate a plurality of sample resource categories corresponding to the geographic region, and the resource category identifiers corresponding to different geographic regions are different.

[0046] In some embodiments, the training unit is configured to perform association data of sample multimedia resources under any sample resource category, process the association data based on the feature extraction model to be trained, obtain sample resource features corresponding to the sample multimedia resources, perform category prediction processing on the sample resource features corresponding to the sample multimedia resources, obtain the predicted resource category to which the sample multimedia resources belong, and train the feature extraction model to be trained based on the sample resource category and the predicted resource category to obtain the target feature extraction model.

[0047] In some embodiments, the training unit is configured to perform feature extraction on the associated data based on the feature extraction model to be trained, when the associated data includes at least two of the sample image, the sample text, or the sample audio, to obtain data features, wherein the data features include at least two of the image features corresponding to the sample image, the text features corresponding to the sample text, or the audio features corresponding to the sample audio; and to fuse the features contained in the data features based on the feature extraction model to be trained to obtain multi-modal features corresponding to the sample multimedia resource, wherein the multi-modal features are sample resource features corresponding to the sample multimedia resource.

[0048] In some embodiments, the sample multimedia resource is a sample video, and the associated data includes the sample image, which includes a cover image and multiple video frames. The training unit is configured to perform feature extraction on the cover image and each video frame based on the feature extraction model to be trained, to obtain cover features corresponding to the cover image and video frame features corresponding to each video frame; to fuse the cover features and multiple video frame features based on the feature extraction model to be trained, to obtain visual features corresponding to the sample video; and to determine the cover features and the visual features as sample resource features corresponding to the sample video.

[0049] In some embodiments, the predicted resource category includes a first resource category and a second resource category, wherein the first resource category is obtained by performing category prediction processing on the cover features; the second resource category is obtained by performing category prediction processing on the visual features; the training unit is configured to perform the following operations: determining a first loss value based on the sample resource category and the first resource category; determining a second loss value based on the sample resource category and the second resource category; and training the feature extraction model to be trained based on the sum of the first loss value and the second loss value to obtain the target feature extraction model.

[0050] In some embodiments, the predicted resource category further includes a third resource category, and the sample resource features corresponding to the sample multimedia resources further include multi-modal features. The multi-modal features are obtained by fusing at least two of image features, text features, or audio features. The image features are features corresponding to sample images in the associated data, the text features are features corresponding to sample text in the associated data, and the audio features are features corresponding to sample audio in the associated data. The training unit is further configured to perform category prediction processing on the multi-modal features to obtain the third resource category; determine a third loss value based on the sample resource category and the third resource category; and train the feature extraction model to be trained based on the sum of the first loss value, the second loss value, and the third loss value to obtain the target feature extraction model.

[0051] In some embodiments, the apparatus further includes:

[0052] The acquisition unit is further configured to acquire associated data of the target multimedia resource, wherein the associated data of the target multimedia resource includes at least one of images, text, or audio.

[0053] The processing unit is configured to perform processing on the associated data of the target multimedia resource based on the target feature extraction model to obtain the resource features corresponding to the target multimedia resource;

[0054] The recommendation unit is configured to recommend the target multimedia resource based on the resource features corresponding to the target multimedia resource.

[0055] According to another aspect of the embodiments of this disclosure, an electronic device is provided, the electronic device comprising:

[0056] One or more processors;

[0057] Memory used to store the executable program code of the processor;

[0058] The processor is configured to execute the program code to implement the aforementioned feature extraction model training method.

[0059] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which, when the program code in the computer-readable storage medium is executed by the processor of an electronic device, enables the electronic device to perform the above-described feature extraction model training method.

[0060] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the above-described feature extraction model training method.

[0061] In the solution provided by this disclosure, a feature extraction model is jointly trained based on sample resource categories corresponding to multiple geographical regions and the associated data of sample resources under each sample resource category. This enables the trained target feature extraction model to represent multimedia resources in resource databases corresponding to multiple geographical regions, and also ensures the accuracy of resource features obtained based on the feature extraction model. Subsequently, only this target feature extraction model is needed to obtain resource features corresponding to multimedia resources in databases corresponding to multiple geographical regions, eliminating the need to train a feature extraction model for each geographical region. This saves the manpower cost required for training the model and improves the training efficiency of the feature extraction model.

[0062] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0064] Figure 1 This is a schematic diagram illustrating an implementation environment according to an exemplary embodiment.

[0065] Figure 2 This is a flowchart illustrating a feature extraction model training method according to an exemplary embodiment.

[0066] Figure 3 This is a flowchart illustrating another feature extraction model training method according to an exemplary embodiment.

[0067] Figure 4 This is a flowchart illustrating a feature extraction model training method according to an exemplary embodiment.

[0068] Figure 5 This is a block diagram of a feature extraction model training device according to an exemplary embodiment.

[0069] Figure 6 This is a block diagram of another feature extraction model training apparatus according to an exemplary embodiment.

[0070] Figure 7 This is a block diagram illustrating a terminal according to an exemplary embodiment.

[0071] Figure 8 This is a block diagram illustrating a server according to an exemplary embodiment. Detailed Implementation

[0072] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0073] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0074] It should be noted that the information (including but not limited to resource tagging information and resource identifiers) and data (including but not limited to associated data) disclosed herein are all authorized by the user or by all parties in full, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the resource tagging information involved in this application was obtained under full authorization.

[0075] In this embodiment of the disclosure, the feature extraction model training method can be executed by an electronic device. In some embodiments, the electronic device is provided as a terminal or a server. When the electronic device is provided as a terminal, the operations performed by the feature extraction model training method are implemented by the terminal; when it is provided as a server, the operations performed by the feature extraction model training method are implemented by the server. In some embodiments, the terminal is at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, MP3 (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), and laptop computer. In some embodiments, the server is at least one of the following: a server, multiple servers, a cloud computing platform, and a virtualization center.

[0076] In some embodiments, the electronic device is provided as a server. Figure 1 This is a schematic diagram illustrating an implementation environment for a feature extraction model training method according to an exemplary embodiment. Taking an electronic device provided as a server as an example, see [link to example]. Figure 1 The implementation environment specifically includes: terminal 101 and server 102.

[0077] Server 102 is used to train a feature extraction model, and provides services to terminal 101 based on the trained feature extraction model. For example, server 102 provides multimedia resource recommendation services and multimedia resource search services to terminal 101 based on the trained feature extraction model. Taking the example of server 102 providing multimedia resource search services to terminal 101 based on the trained feature extraction model, terminal 101 sends a resource search request to server 102. Server 102, based on the trained feature extraction model, obtains resource features corresponding to multiple multimedia resources. Based on the similarity between the resource features corresponding to multiple multimedia resources and the features of the information carried in the resource search request, server 102 recommends multimedia resources to terminal 101 that match the information carried in the resource search request.

[0078] In some embodiments, terminal 101 installs an application provided by server 102. After training a target feature extraction model, server 102 provides services to the application based on the target feature extraction model. Terminal 101 interacts with server 102 based on the application, and server 102 then provides services to terminal 101 based on the target feature extraction model.

[0079] Figure 2 This is a flowchart illustrating a feature extraction model training method according to an exemplary embodiment, such as... Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0080] In step S201, the electronic device determines multiple sample resource categories corresponding to each geographical region based on the resource tag information in the resource database corresponding to each geographical region.

[0081] In this embodiment of the disclosure, each geographical region corresponds to a resource database, which is used to store multimedia resources published by users within that geographical region. These multimedia resources include text, images, or videos. Resource tag information is used to describe the tags associated with the multimedia resources. For example, the resource tag information is the topic tag selected by the user when publishing the multimedia resource, such as "XX Movie" or "About the 2023 New Year," etc.

[0082] In this embodiment of the disclosure, the content of multimedia resources in resource databases corresponding to different geographical regions is diverse. Therefore, the resource categories to which the multimedia resources in resource databases corresponding to different geographical regions belong are different. Therefore, based on the resource tag information in the resource database corresponding to each geographical region, multiple sample resource categories corresponding to each geographical region are determined. The multiple sample resource categories corresponding to any geographical region are also the resource categories to which the multimedia resources in the resource database corresponding to that geographical region belong.

[0083] In step S202, the electronic device obtains the associated data of sample multimedia resources under each sample resource category based on the resource identifier corresponding to the resource tag information in the resource database corresponding to multiple geographical regions. The resource identifier indicates the sample multimedia resource, and the associated data includes at least one of sample image, sample text, or sample audio.

[0084] In this embodiment of the disclosure, for any geographical region, a resource database includes a correspondence between resource identifiers and resource tag information, with each resource identifier corresponding to one or more resource tag information. After determining multiple sample resource categories corresponding to the geographical region, the resource identifier corresponding to each sample resource category can be determined based on the correspondence between resource identifiers and resource tag information in the resource database. Then, based on the resource identifier corresponding to each sample resource category, the associated data of sample multimedia resources under each sample resource category is obtained.

[0085] The associated data describes the sample multimedia resource, and its content is related to that resource. For example, if the sample multimedia resource is a sample video, the associated data includes sample images that are video frames or cover images from the video; sample text that is the video title, text identified from video frames or cover images, or text obtained through speech recognition of audio from the video; and sample audio that is the audio from the video.

[0086] In step S203, the electronic device trains the feature extraction model to be trained based on the sample resource categories corresponding to the multiple geographical regions and the associated data of sample multimedia resources under each sample resource category, to obtain the target feature extraction model, which is used to obtain the resource features corresponding to the target multimedia resources.

[0087] In this embodiment of the disclosure, the feature extraction model to be trained is trained based on the sample resource categories corresponding to the multiple geographical regions and the associated data of sample multimedia resources under each sample resource category. This realizes the scheme of jointly training the feature extraction model through the associated data of sample multimedia resources in the resource databases corresponding to multiple geographical regions. This enables the trained target feature extraction model to have the ability to represent resources and to represent multimedia resources in the resource databases corresponding to multiple geographical regions, thereby improving the applicability of the target feature extraction model.

[0088] In the solution provided by this disclosure, a feature extraction model is jointly trained based on sample resource categories corresponding to multiple geographical regions and the associated data of sample resources under each sample resource category. This enables the trained target feature extraction model to represent multimedia resources in resource databases corresponding to multiple geographical regions, and also ensures the accuracy of resource features obtained based on the feature extraction model. Subsequently, only this target feature extraction model is needed to obtain resource features corresponding to multimedia resources in databases corresponding to multiple geographical regions, eliminating the need to train a feature extraction model for each geographical region. This saves the manpower cost required for training the model and improves the training efficiency of the feature extraction model.

[0089] In some embodiments, based on resource tag information in the resource database corresponding to each geographic region, multiple sample resource categories corresponding to each geographic region are determined, including:

[0090] Retrieve multiple resource tag information from the resource database corresponding to a geographic region;

[0091] Based on multiple resource tag information, a tag coefficient is determined for each resource tag information, and the tag coefficient indicates the importance of the resource tag information.

[0092] Based on the tag coefficients of multiple resource tag information, select target resource tag information whose tag coefficients belong to the target range;

[0093] Based on the tag information of each target resource, a sample resource category is determined to obtain multiple sample resource categories corresponding to the geographic region.

[0094] In this embodiment, the tag coefficient of resource tag information can reflect the importance of resource tag information. The larger the tag coefficient, the more important the resource tag information is, and the more it can reflect that multimedia resources in the resource database can be distinguished based on the resource tag information. If the tag coefficient is too large, the number of multimedia resources corresponding to the resource tag information will be too small, making the resource tag information unsuitable as a sample resource category for end users to classify resources. Therefore, according to the tag coefficients of multiple resource tag information, resource tag information whose tag coefficients belong to the target range is selected as the target tag information to filter out resource tag information that is popular or meaningless, so as to ensure the accuracy of the obtained target resource tag information, and thus ensure the accuracy of the subsequently determined sample resource category.

[0095] In some embodiments, a sample resource category is determined based on each target resource tag information to obtain multiple sample resource categories corresponding to a geographic region, including:

[0096] Sort multiple geographical regions according to their corresponding region identifiers;

[0097] Based on the sorting order of multiple geographic regions, a resource category identifier is generated for the target resource label information corresponding to each geographic region, so as to obtain multiple resource category identifiers corresponding to each geographic region. The multiple resource category identifiers corresponding to the geographic region indicate multiple sample resource categories corresponding to the geographic region. The resource category identifiers corresponding to different geographic regions are different.

[0098] In this embodiment of the disclosure, the region identifier indicates the corresponding geographical region. The multiple geographical regions are sorted according to the region identifiers corresponding to the multiple geographical regions, and resource category identifiers corresponding to each geographical region are generated separately to ensure that the generated resource category identifiers are not repeated, thus ensuring the diversity of the determined sample resource categories and ensuring that the feature extraction model trained subsequently can be applied to multiple geographical regions.

[0099] In some embodiments, based on sample resource categories corresponding to multiple geographical regions and the associated data of sample multimedia resources under each sample resource category, the feature extraction model to be trained is trained to obtain the target feature extraction model, including:

[0100] For the associated data of sample multimedia resources under any sample resource category, the associated data is processed based on the feature extraction model to be trained to obtain the sample resource features corresponding to the sample multimedia resources.

[0101] The sample resource features corresponding to the sample multimedia resources are subjected to category prediction processing to obtain the predicted resource category to which the sample multimedia resources belong;

[0102] Based on the sample resource category and the predicted resource category, the feature extraction model to be trained is trained to obtain the target feature extraction model.

[0103] In this embodiment of the disclosure, the sample resource features corresponding to the sample multimedia resources are obtained based on the feature extraction model to be trained, and the predicted resource category is obtained based on the sample resource features. Based on the sample resource category to which the sample multimedia resources belong and the predicted resource category, it is possible to determine whether the predicted resource category is accurate, reflecting the accuracy of the sample resource features, and thus reflecting the accuracy of the feature extraction model. That is, the sample resource category to which the sample multimedia resources belong and the predicted resource category are only used as a supervision signal. Based on the sample resource category to which the sample multimedia resources belong and the predicted resource category, the feature extraction model is trained to improve the accuracy of the feature extraction model.

[0104] In some embodiments, based on the feature extraction model to be trained, the associated data is processed to obtain the sample resource features corresponding to the sample multimedia resources, including:

[0105] When the associated data includes at least two of the sample images, sample text, or sample audio, feature extraction is performed on the associated data based on the feature extraction model to be trained to obtain data features. The data features include at least two of the image features corresponding to the sample images, the text features corresponding to the sample text, or the audio features corresponding to the sample audio.

[0106] Based on the feature extraction model to be trained, the features contained in the data features are fused to obtain the multi-modal features corresponding to the multimedia resources of the sample. The multi-modal features are the sample resource features corresponding to the multimedia resources of the sample.

[0107] In this embodiment of the disclosure, since the data features include features of multiple modalities of the sample multimedia resource, the features of different modalities can characterize the sample multimedia resource from different perspectives. By fusing the features contained in the data features, multimodal features of the sample multimedia resource are obtained. By using the multimodal features of the sample resource as the sample resource features corresponding to the sample multimedia resource, the content contained in the sample resource features is enriched, making the multimodal features more representative of the sample resource and ensuring the accuracy of the features.

[0108] In some embodiments, the sample multimedia resource is a sample video, and the associated data includes sample images, which include a cover image and multiple video frames; based on the feature extraction model to be trained, the associated data is processed to obtain the sample resource features corresponding to the sample multimedia resource, including:

[0109] Based on the feature extraction model to be trained, feature extraction is performed on the cover image and each video frame to obtain the cover features corresponding to the cover image and the video frame features corresponding to each video frame.

[0110] Based on the feature extraction model to be trained, the cover features and features of multiple video frames are fused to obtain the visual features corresponding to the sample video.

[0111] Cover features and visual features are identified as the sample resource features corresponding to the sample videos.

[0112] In this embodiment of the disclosure, the cover image may be different from the video frames contained in the sample video. The cover image can reflect different meanings of the sample video, so the cover features corresponding to the cover image can also represent the sample video independently. Therefore, both the cover features and visual features are determined as the sample video features corresponding to the sample video, so that the sample video features can more comprehensively represent the sample video and ensure the accuracy of the sample video features.

[0113] In some embodiments, the predicted resource category includes a first resource category and a second resource category. The first resource category is obtained by performing category prediction processing on cover features; the second resource category is obtained by performing category prediction processing on visual features. Based on the sample resource category and the predicted resource category, the feature extraction model to be trained is trained to obtain the target feature extraction model, including:

[0114] The first loss value is determined based on the sample resource category and the first resource category;

[0115] The second loss value is determined based on the sample resource category and the second resource category;

[0116] Based on the sum of the first and second loss values, the feature extraction model to be trained is trained to obtain the target feature extraction model.

[0117] In this embodiment of the disclosure, both the first resource category and the second resource category are predicted resource categories. Since the associated data of the sample multimedia resources include different types of information, the feature extraction model is trained by combining cover features and visual features to improve the accuracy of the feature extraction model in extracting features of different types of information, thereby improving the accuracy of the feature extraction model.

[0118] In some embodiments, the predicted resource category further includes a third resource category, and the sample resource features corresponding to the sample multimedia resources further include multi-modal features. The multi-modal features are obtained by fusing at least two of image features, text features, or audio features. The image features are the features corresponding to the sample images in the associated data, the text features are the features corresponding to the sample text in the associated data, and the audio features are the features corresponding to the sample audio in the associated data. The method further includes:

[0119] The multimodal features are subjected to category prediction processing to obtain the third resource category;

[0120] The third loss value is determined based on the sample resource category and the third resource category;

[0121] Based on the sum of the first and second loss values, the feature extraction model to be trained is trained to obtain the target feature extraction model, including:

[0122] Based on the sum of the first, second, and third loss values, the feature extraction model to be trained is trained to obtain the target feature extraction model.

[0123] In this embodiment of the disclosure, since the associated data of the sample multimedia resources includes different types of information, the feature extraction model to be trained can extract multiple resource features of the sample multimedia resources, predict multiple predicted resource categories through multiple resource features, train the feature extraction model to be trained, so as to improve the accuracy of the feature extraction model in extracting features of different types of information, thereby improving the accuracy of the feature extraction model.

[0124] In some embodiments, after training the feature extraction model to be trained based on sample resource categories corresponding to multiple geographical regions and the associated data of sample multimedia resources under each sample resource category, and obtaining the target feature extraction model, the method further includes:

[0125] Obtain associated data of the target multimedia resource, wherein the associated data of the target multimedia resource includes at least one of images, text or audio;

[0126] Based on the target feature extraction model, the associated data of the target multimedia resources are processed to obtain the resource features corresponding to the target multimedia resources;

[0127] Based on the resource characteristics corresponding to the target multimedia resources, recommendations are made for the target multimedia resources.

[0128] In this embodiment of the disclosure, resource features corresponding to target multimedia resources are obtained based on the target feature extraction model obtained through training, and the target multimedia resources are recommended based on the resource features, so as to recommend the target multimedia resources to users who are interested, thereby ensuring the accuracy of resource recommendation.

[0129] The above Figure 2 The diagram shown is merely the basic process of this disclosure. The following section further elaborates on the solution provided in this disclosure based on a specific implementation method, using the resource database as a video database, resource tag information as video tag information, sample resource categories as sample video categories, sample multimedia resources as sample videos, resource identifiers as video identifiers, and sample resource features as sample video features as an example. Figure 3 This is a flowchart illustrating another feature extraction model training method according to an exemplary embodiment, such as... Figure 3 As shown, the method is performed by an electronic device and includes the following steps:

[0130] In step S301, the electronic device acquires multiple video tag information from the video database corresponding to any one of the multiple geographical regions.

[0131] In this embodiment of the disclosure, based on video databases corresponding to multiple geographical regions, video tag information is obtained from the video database corresponding to each geographical region, and the sample video category corresponding to each geographical region is determined accordingly.

[0132] In some embodiments, if the video database corresponding to a geographic region includes video identifiers and corresponding video tag information, then step S301 includes: obtaining video tag information corresponding to each video identifier from the video database corresponding to the geographic region, deduplicating the obtained video tag information, and obtaining multiple video tag information corresponding to the geographic region.

[0133] In some embodiments, the video tag information in the video database corresponding to the geographical region exists in any form, such as text or other strings. For example, the video tag information is a hashtag.

[0134] In step S302, the electronic device determines the tag coefficient of each video tag information based on the multiple video tag information, and the tag coefficient indicates the importance of the video tag information.

[0135] In this embodiment of the disclosure, since there may be meaningless or irrelevant video tag information among the multiple video tag information, in order to ensure the accuracy of the subsequently determined sample video categories, it is necessary to determine the tag coefficient of each video tag information to indicate the importance of each video tag information, so as to filter out high-quality video tag information from multiple video tag information and ensure the accuracy of the subsequently generated sample video categories.

[0136] The tag coefficient is represented by a numerical value. The larger the tag coefficient of a video tag, the more important the video tag is, indicating that the video tag is meaningful. The smaller the tag coefficient of a video tag, the less important the video tag is, indicating that the video tag is meaningless.

[0137] In some embodiments, the TF-IDF (Term Frequency–Inverse Document Frequency, a commonly used weighting technique for information retrieval and data mining) algorithm is employed to determine the tag coefficient for each video tag based on the multiple video tag information. The tag coefficient obtained using the TF-IDF algorithm is the TF-IDF score of the video tag information.

[0138] In step S303, the electronic device selects target video tag information whose tag coefficients belong to the target interval from the multiple video tag information based on the tag coefficients of the multiple video tag information.

[0139] In this embodiment, the tag coefficient of video tag information reflects the importance of the video tag information. The larger the tag coefficient, the more important the video tag information is, and the more it can distinguish videos in the video database. The smaller the tag coefficient, the less important the video tag information is, and the less accurately it can distinguish videos in the video database. However, if the tag coefficient is too large, the number of videos corresponding to the video tag information will be too small, making it impossible for the video tag information to be used as a sample video category for end users to classify videos. Therefore, according to the tag coefficients of multiple video tag information, video tag information whose tag coefficients belong to the target range is selected as the target tag information to filter out video tag information that is trending or meaningless, such as meaningless video tag information like "aaaa" or "cccc", to ensure the accuracy of the obtained target video tag information, and thus ensure the accuracy of the subsequently determined sample video category.

[0140] In this embodiment, the TF-IDF algorithm is used to filter multiple target video tag information from the video database corresponding to each geographical region. That is, each geographical region corresponds to multiple target video tag information. For different geographical regions, the target video tag information corresponding to different geographical regions may contain the same video tag information.

[0141] The target interval can be any numerical range. In some embodiments, the target interval is a pre-set numerical range, or it is a target interval determined based on the tag coefficients of multiple video tag information corresponding to the geographical region.

[0142] In some embodiments, determining the target interval includes: sorting the multiple video tag information in ascending order of the tag coefficients, and using the tag coefficients of the video tag information at the target sequence number and the tag coefficients of the video tag information at the end of the sequence to form the target interval.

[0143] The target number can be any value, such as 100 or 1000.

[0144] It should be noted that the present embodiment uses a target range to filter out target video tag information. In another embodiment, it is not necessary to perform the above step S303. Instead, other methods are adopted to filter out target video tag information from multiple video tag information based on the tag coefficients of multiple video tag information.

[0145] In some embodiments, the process of filtering target video tag information from multiple video tag information includes: sorting the multiple video tag information in ascending order of tag coefficients, filtering out a first number of video tag information that are ranked first, and using the remaining video tag information after filtering as the target video tag information. The first number can be any number.

[0146] In step S304, the electronic device determines a sample video category based on the tag information of each target video to obtain multiple sample video categories corresponding to the geographical area.

[0147] In this embodiment of the disclosure, after determining multiple target video tag information corresponding to the geographical area, a sample video category is determined based on one target video tag information, thereby enabling the determination of multiple sample video categories corresponding to the geographical area.

[0148] This embodiment of the disclosure uses geographical regions as units, determining multiple sample video categories for each geographical region. The final number of sample video categories is the sum of the multiple sample video categories corresponding to the various geographical regions. For example, if each geographical region has 10 sample video categories, and there are 10 geographical regions in total, then 100 sample video categories are obtained.

[0149] In some embodiments, the process of determining multiple sample video categories corresponding to each geographic region includes: generating a video category identifier for each target video tag information to obtain multiple video category identifiers corresponding to the geographic region, wherein the video category identifiers corresponding to different geographic regions are different.

[0150] The multiple video category identifiers corresponding to the geographical region indicate the multiple sample video category identifiers corresponding to that geographical region. These video category identifiers can be represented in any form; for example, they can be represented as strings or as numbers.

[0151] In this embodiment of the disclosure, among the target video tag information corresponding to multiple geographical regions, there may be the same video tag information in the target video tag information corresponding to different geographical regions. However, after obtaining multiple video category identifiers corresponding to each geographical region in the manner described above, there are no identical video category identifiers in the video category identifiers corresponding to different geographical regions. This ensures that the determined sample video categories retain the diversity of different geographical regions, thereby ensuring that the determined sample video categories are diverse and that the feature extraction model trained subsequently can be applied to multiple geographical regions.

[0152] For example, if a video is tagged with "love", the video tag is represented in the same way in databases for multiple geographic regions. However, considering that users in different geographic regions may express love in different ways, the video tag is retained in different geographic regions, and a video category identifier corresponding to the video tag is set for each geographic region. The video category identifier corresponding to the tag is different in different geographic regions.

[0153] In some embodiments, the process of determining multiple sample video categories corresponding to each geographic region includes: sorting the multiple geographic regions according to the region identifiers corresponding to the multiple geographic regions; generating a video category identifier for each target video tag information corresponding to each geographic region according to the sorting order of the multiple geographic regions, so as to obtain multiple video category identifiers corresponding to each geographic region. The multiple video category identifiers corresponding to the geographic region indicate multiple sample video categories corresponding to the geographic region, and the video category identifiers corresponding to different geographic regions are different.

[0154] The generation of video category identifiers can employ any algorithm. For example, the video category identifiers can be identified by serial numbers according to the order in which they are generated. For instance, given three geographical regions, each with 10 video tag information entries, after generating a video category identifier for each tag, the 10 video category identifiers for geographical region 1 would be 0-9, for geographical region 2 10-19, and for geographical region 3 20-29.

[0155] In this embodiment of the disclosure, the region identifier indicates the corresponding geographical region. The multiple geographical regions are sorted according to the region identifiers corresponding to the multiple geographical regions, and video category identifiers corresponding to each geographical region are generated separately to ensure that the generated video category identifiers are not repeated, thus ensuring the diversity of the determined sample video categories and ensuring that the feature extraction model trained subsequently can be applied to multiple geographical regions.

[0156] In some embodiments, the process of determining multiple sample video categories corresponding to each geographic region includes: aligning target video tag information corresponding to multiple geographic regions, encoding the aligned target video tag information to obtain multiple video category identifiers, wherein each target video tag information in each geographic region corresponds to a video category identifier, and the video category identifiers corresponding to different geographic regions are different.

[0157] In this embodiment of the disclosure, since different geographical regions may contain text in different languages, or the same geographical region may contain text in different languages, a feature alignment method is used to generate multiple video category identifiers to ensure that the generated video category identifiers can represent the sample video category corresponding to each geographical region, thereby ensuring the accuracy of the sample video category.

[0158] It should be noted that the embodiments disclosed herein use the tag coefficients of video tag information to determine multiple sample video categories corresponding to each geographical region. However, in another embodiment, it is not necessary to perform the above steps S301-S304. Instead, other methods are adopted to determine multiple sample video categories corresponding to each geographical region based on the video tag information in the video database corresponding to each geographical region.

[0159] In step S305, the electronic device obtains the associated data of sample videos under each sample video category based on the video identifier corresponding to the video tag information in the video database corresponding to multiple geographical regions. The video identifier indicates the sample video, and the associated data includes at least one of sample image, sample text, or sample audio.

[0160] In this embodiment of the disclosure, the video database corresponding to each geographical region stores the correspondence between video tag information and video identifier, and each sample video category corresponds to a video tag information. Therefore, for any sample video category corresponding to any geographical region, the video identifier corresponding to the sample video category can be determined based on the correspondence between video tag information and video identifier in the video database corresponding to the geographical region. Based on the video identifier corresponding to the sample video category, the sample videos under the sample video category can be determined, and thus the associated data of the sample videos under the sample video category can be obtained.

[0161] In some embodiments, the process of obtaining associated data of sample videos under a sample video category includes the following two methods.

[0162] In the first method, the associated data corresponding to the video identifier is pre-stored in the database: if the associated data corresponding to the video identifier is stored in the video database corresponding to the geographical region, the associated data corresponding to the video identifier corresponding to the sample video category is obtained from the video database corresponding to the geographical region, that is, the associated data of the sample video under the sample video category is obtained; or, if the associated data corresponding to the video identifier is stored in other databases, the associated data corresponding to the video identifier corresponding to the sample video category is obtained from other databases.

[0163] In this context, "other databases" refers to databases other than the video databases corresponding to geographic regions. In some embodiments, these other databases are used to store associated data corresponding to video identifiers in multiple video databases representing geographic regions.

[0164] In this embodiment of the disclosure, the associated data corresponding to the video identifier is pre-stored in the video database or other database corresponding to the geographical region. Therefore, after determining the video identifier corresponding to each sample video category based on multiple sample categories corresponding to the geographical region and the video identifier corresponding to the video tag information in the video database corresponding to the geographical region, the associated data of the sample video under each sample video category can be obtained from the video database or other database corresponding to the geographical region based on the determined video identifier.

[0165] The second method involves extracting the associated data corresponding to the video identifier from the video database corresponding to the geographic region: Given a determined video identifier for a sample video category, a sample video corresponding to the video identifier is retrieved from the video database corresponding to that geographic region, and the associated data for that sample video is extracted from that sample video.

[0166] In this embodiment, the video database corresponding to the geographical region stores not only video identifiers corresponding to video tag information, but also the videos corresponding to those identifiers. Therefore, it is possible to determine sample videos under the sample video category from the video database corresponding to the geographical region, and extract the associated data of those sample videos. This method only requires obtaining the associated data of the sample videos, eliminating the need to generate and store the associated data of multiple videos in advance, thereby saving storage resources.

[0167] In some embodiments, the process of extracting associated data from the sample video includes: extracting an image from the sample video to obtain a sample image, and determining the sample image as associated data of the sample video; or, determining the video title text corresponding to the sample video as associated data of the sample video; or, extracting an image from the sample video, performing text recognition on the extracted image to obtain a first text, and determining the first text as associated data of the sample video; or, performing speech recognition on sample audio in the sample video to obtain a second text, and determining the first text as associated data of the sample video; or, determining sample audio in the sample video as associated data of the sample video.

[0168] In this embodiment of the disclosure, the sample image is any image from the sample video; for example, the sample image includes the cover image of the sample video or a video frame from the sample video. The sample text includes the video title text, first text, or second text corresponding to the sample video.

[0169] In this embodiment of the disclosure, OCR (Optical Character Recognition) can be used to perform text recognition on the extracted images; and ASR (Automatic Speech Recognition) can be used to perform speech recognition on the sample audio in the sample video.

[0170] In some embodiments, the sample image includes video frames from a sample video, and the process of extracting video frames from the sample video includes: extracting a second number of video frames from the sample video, wherein the time interval between any two adjacent video frames in the second number of video frames is equal.

[0171] The second quantity can be any quantity, for example, the second quantity can be 4 or 5, etc.

[0172] In step S306, the electronic device processes the associated data of sample videos under any sample video category based on the feature extraction model to be trained, and obtains the sample video features corresponding to the sample video.

[0173] In this embodiment of the disclosure, the associated data contains content related to the sample video, meaning the associated data can characterize the sample video. The feature extraction model is used to obtain the video features corresponding to the video. Based on the feature extraction model to be trained, the associated data is processed to obtain sample video features used to characterize the sample video. The feature extraction model to be trained is a feature extraction model that has not yet been fully trained.

[0174] In some embodiments, step S306 includes: when the associated data of the sample video includes at least two of sample images, sample text, or sample audio, performing feature extraction on the associated data of the sample video based on the feature extraction model to be trained to obtain data features, the data features including at least two of image features corresponding to the sample image, text features corresponding to the sample text, or audio features corresponding to the sample audio; and fusing the features contained in the data features based on the feature extraction model to be trained to obtain multi-modal features corresponding to the sample video, the multi-modal features being sample video features corresponding to the sample video.

[0175] In this embodiment, the associated data of the sample video includes at least two of the following: sample image, sample text, or sample audio. Based on the feature extraction model to be trained, features are extracted from each data item in the associated data to obtain the features of each data item, i.e., the data features. These data features correspond one-to-one with the at least two data items in the associated data. Since the data features include features from multiple modalities of the sample video, such as image and text features, or image and audio features, different modal features can characterize the sample video from different perspectives. By fusing the features included in the data features, the sample video features corresponding to the sample video are obtained, i.e., the multi-modal features of the sample video. By using the multi-modal features of the sample video as the corresponding sample video features, the content of the sample video features is enriched, making the multi-modal features better characterize the sample video and ensuring the accuracy of the features.

[0176] In some embodiments, step S306 above includes the following three methods.

[0177] The first method involves the following steps 1-3: The associated data of the sample video includes sample images, which include the cover image of the sample video and multiple video frames.

[0178] Step 1: Based on the feature extraction model to be trained, perform feature extraction on the cover image and each video frame to obtain the cover features corresponding to the cover image and the video frame features corresponding to each video frame.

[0179] Among them, the cover feature is used to represent the cover image, and the video frame feature is used to represent the video frame. Both the cover feature and the video frame feature can be represented in any form. For example, both the cover feature and the video frame feature can be represented in the form of feature vectors.

[0180] Step 2: Based on the feature extraction model to be trained, fuse the cover features and features from multiple video frames to obtain the visual features corresponding to the sample video.

[0181] In this embodiment of the disclosure, both cover features and video frame features can characterize the sample video from a visual perspective. Therefore, the cover features and video frame features are fused to obtain the visual features of the sample video, thereby characterizing the sample video from a visual perspective.

[0182] In some embodiments, step 2 includes: weighted fusion of features from multiple video frames based on the feature extraction model to be trained to obtain fused features; weighted fusion of cover features and fused features to obtain visual features corresponding to the sample video.

[0183] In this embodiment, a weighted approach is used. First, features from multiple video frames are fused. Then, the fused features are fused with the cover features to ensure that the resulting visual features fully incorporate both the cover and video frame features, thus improving the accuracy of the visual features. For the weighted fusion method, a self-attention weighted fusion approach can be employed, where multiple video frame features are weighted and fused, and the cover and fused features are also weighted and fused to ensure that the features are fully integrated, thereby guaranteeing the accuracy of the resulting visual features.

[0184] Step 3: Determine the cover features and visual features as the sample video features corresponding to the sample video.

[0185] In this embodiment of the disclosure, the cover image may be different from the video frames contained in the sample video. The cover image can reflect different meanings of the sample video, so the cover features corresponding to the cover image can also represent the sample video independently. Therefore, both the cover features and visual features are determined as the sample video features corresponding to the sample video, so that the sample video features can more comprehensively represent the sample video and ensure the accuracy of the sample video features.

[0186] The second approach involves using sample video-related data, including sample text. This sample text comprises video title text, a first text, and a second text. The first text is obtained through text recognition of images in the sample video, and the second text is obtained through speech recognition of sample audio in the sample video. Based on the feature extraction model to be trained, features are extracted from the video title text, the first text, and the second text to obtain the title features corresponding to the video title text, the first feature corresponding to the first text, and the second feature corresponding to the second text. Based on the feature extraction model to be trained, the title features, the first feature, and the second feature are fused to obtain the text features corresponding to the sample text. These text features are then used as the sample video features corresponding to the sample video.

[0187] In this embodiment of the disclosure, the sample text contains multiple types of text, and each type of text can characterize the sample video. Therefore, the features of the multiple types of text contained in the sample text are fused to enrich the content of the obtained text features, thereby improving the accuracy of the text features and thus improving the accuracy of the sample video features.

[0188] In some embodiments, a self-attention mechanism is employed to fuse the title features, the first text, and the second features to obtain the text features corresponding to the sample text.

[0189] The third approach involves using sample audio as the associated data for the sample video: based on the feature extraction model to be trained, features are extracted from the sample audio to obtain the audio features corresponding to the sample audio; the audio features corresponding to the sample audio are then used as the sample video features corresponding to the sample video.

[0190] It should be noted that the above three methods are only examples of obtaining sample video features corresponding to sample videos, with the sample video's associated data including any one of sample images, sample text, or sample audio. In another embodiment, the sample video's associated data includes any two or three of sample images, sample text, or sample audio. Based on the content contained in the associated data, the above three methods for obtaining sample video features corresponding to sample videos are combined to obtain the sample video features corresponding to the sample video.

[0191] In some embodiments, the associated data of the sample video includes at least one of the sample image, sample text, or sample audio, and the sample video features corresponding to the sample video include cover features, visual features, and multimodal features.

[0192] Specifically, when the associated data includes sample images and sample text, and the sample images include cover images and multiple video frames, the multimodal features are obtained by fusing the cover features corresponding to the cover image, the video frame features corresponding to the multiple video frames, and the text features corresponding to the sample text. When the associated data includes sample images and sample audio, and the sample images include cover images and multiple video frames, the multimodal features are obtained by fusing the cover features corresponding to the cover image, the video frame features corresponding to the multiple video frames, and the audio features corresponding to the sample audio. When the associated data includes sample images, sample text, and sample audio, and the sample images include cover images and multiple video frames, the multimodal features are obtained by fusing the cover features corresponding to the cover image, the video frame features corresponding to the multiple video frames, the text features corresponding to the sample text, and the audio features corresponding to the sample audio.

[0193] It should be noted that the embodiments disclosed herein are only illustrated using the example of associated data including sample images, sample text, or sample audio. In another embodiment, the associated data also includes other data. In this case, when obtaining the sample video features corresponding to the sample video, the features of other data are obtained based on the feature extraction model to be trained, and the features of other data are combined to obtain the multi-modal features corresponding to the sample video.

[0194] For example, if the associated data also includes user information, which indicates the user who posted the sample video, then based on the feature extraction model to be trained, features are extracted from the user information to obtain user features. That is, the data features corresponding to the associated data also include user features. Then, based on the feature extraction model to be trained, the features contained in the data features are fused to obtain the multi-modal features corresponding to the sample video, which is to say, the sample video features corresponding to the sample video are obtained.

[0195] In step S307, the electronic device performs category prediction processing on the sample video features corresponding to the sample video to obtain the predicted video category to which the sample video belongs.

[0196] The predicted video category is one or more of the sample video categories corresponding to multiple geographical regions.

[0197] In some embodiments, step S307 includes: processing the sample video features corresponding to the sample video to obtain a category vector, the category vector including values ​​of multiple dimensions, the number of dimensions contained in the category vector being equal to the total number of sample video categories corresponding to multiple geographical regions; determining the sample video category corresponding to the dimension with the largest value in the category vector as the predicted video category to which the sample video belongs, or determining the sample video category corresponding to the third number of dimensions in the category vector as the predicted video category to which the sample video belongs.

[0198] In this system, the value of the third dimension is greater than the values ​​of other dimensions in the category vector, and the third dimension can be any number. The value of each dimension represents the probability that the sample video belongs to the sample video category corresponding to that dimension; the sample video category corresponding to the dimension with the largest value is the predicted video category to which the sample video is likely to belong.

[0199] In step S308, the electronic device trains the feature extraction model to be trained based on the sample video category to which the sample video belongs and the predicted video category, thereby obtaining the target feature extraction model.

[0200] In this embodiment of the disclosure, the sample video features corresponding to the sample video are obtained based on the feature extraction model to be trained, and the predicted video category is obtained based on the sample video features. Based on the sample video category to which the sample video belongs and the predicted video category, it is possible to determine whether the predicted video category is accurate, reflecting the accuracy of the sample video features, and thus reflecting the accuracy of the feature extraction model. That is, the sample video category to which the sample video belongs and the predicted video category are only used as a supervision signal. Based on the sample video category to which the sample video belongs and the predicted video category, the feature extraction model to be trained is trained to improve the accuracy of the obtained target feature extraction model.

[0201] In some embodiments, step S308 includes: determining a loss value based on the sample video category and the predicted video category, and training the feature extraction model to be trained based on the determined loss value to obtain the target feature extraction model.

[0202] In this embodiment of the disclosure, the loss value can reflect the difference between the sample video category to which the sample video belongs and the predicted video category. This can be used to train the feature extraction model and improve the accuracy of the feature extraction model.

[0203] In some embodiments, when the sample video features corresponding to the sample video include cover features and visual features, the process of training the feature extraction model includes: performing category prediction processing on the cover features to obtain a first video category to which the sample video belongs; performing category prediction processing on the visual features to obtain a second video category to which the sample video belongs; and training the feature extraction model to be trained based on the sample video category, the first video category, and the second video category to obtain a target feature extraction model.

[0204] In this embodiment of the disclosure, both the first video category and the second video category are predicted video categories. Since the associated data of the sample videos includes different types of information, the feature extraction model to be trained is trained by combining cover features and visual features to improve the accuracy of the feature extraction model in extracting features of different types of information, thereby improving the accuracy of the obtained target feature extraction model.

[0205] In some embodiments, when a first video category and a second video category are determined, a first loss value is determined based on the sample video category and the first video category, a second loss value is determined based on the sample video category and the second video category, and the feature extraction model to be trained is trained based on the sum of the first loss value and the second loss value to obtain the target feature extraction model.

[0206] In this embodiment of the disclosure, each loss value can reflect the difference between the sample video category to which the sample video belongs and the predicted video category. Training the feature extraction model with the sum of multiple loss values ​​can improve the accuracy of the feature extraction model.

[0207] In some embodiments, when the sample video features corresponding to the sample video include multi-modal features, the process of training the feature extraction model includes: performing multi-modal feature category prediction processing to obtain the third video category to which the sample video belongs; and training the feature extraction model based on the sample video category and the third video category.

[0208] It should be noted that the above two methods can be combined. That is, the feature extraction model to be trained is obtained based on the sample video category, the first video category, the second video category, and the third video category. In this embodiment, the feature extraction model is trained with features from different categories to ensure the accuracy of the feature extraction model.

[0209] In some embodiments, when a first video category, a second video category, and a third video category are determined, a first loss value is determined based on the sample video category and the first video category, a second loss value is determined based on the sample video category and the second video category, a third loss value is determined based on the sample video category and the third video category, and the feature extraction model to be trained is trained based on the sum of the first loss value, the second loss value, and the third loss value to obtain the target feature extraction model.

[0210] In this embodiment of the disclosure, each loss value can reflect the difference between the sample video category to which the sample video belongs and the predicted video category. Training the feature extraction model with the sum of multiple loss values ​​can improve the accuracy of the feature extraction model.

[0211] It should be noted that steps S306-S308 above use the associated data of sample videos under any sample video category as an example to train the feature extraction model to obtain the target feature extraction. In another embodiment, following steps S306-S308 above, based on sample video categories corresponding to multiple geographical regions and the associated data of sample videos under each sample video category, the feature extraction model to be trained is iteratively trained to obtain the target feature extraction model. During the iterative training process, the feature extraction model obtained in the previous iteration is used as the feature extraction model to be trained in the current iteration. Then, the current iteration training is performed according to steps S306-S308 above, and the above steps are repeated. When the number of iterations reaches the threshold, or when the sum of the loss values ​​obtained in the current iteration is less than the loss threshold, the iterative training is stopped, and the currently obtained feature extraction model is used as the target feature extraction model.

[0212] It should be noted that the present embodiment only trains the feature extraction model by predicting the predicted video category to which the sample video belongs. In another embodiment, it is not necessary to perform the above steps S306-S308. Instead, other methods are adopted to train the feature extraction model to be trained based on the sample video categories corresponding to multiple geographical regions and the associated data of sample videos under each sample video category, so as to obtain the target feature extraction model.

[0213] The solution provided in this disclosure uses sample resource categories corresponding to multiple geographical regions and associated data of sample resources under each category to jointly train a feature extraction model. This enables the trained target feature extraction model to represent multimedia resources in resource databases corresponding to multiple geographical regions, while also ensuring the accuracy of resource features obtained based on the feature extraction model. Subsequently, only this target feature extraction model is needed to obtain resource features corresponding to multimedia resources in databases corresponding to multiple geographical regions, eliminating the need to train a separate feature extraction model for each geographical region. This saves the manpower cost required for model training and improves the training efficiency of the feature extraction model. Furthermore, by using a single feature extraction model to unify the multi-modal representation of global videos, it helps the recommendation system to better distribute short videos globally.

[0214] In the above Figure 2 to Figure 3 Based on the embodiments shown, this disclosure constructs a video tag information preprocessing module and adopts encoding mechanisms, multi-task learning mechanisms, and multimodal feature mechanisms to train a feature extraction model for globally unified video multimodal representation, so as to subsequently utilize the video features corresponding to the video to help the video be distributed globally. The process of training the feature extraction model includes the following three stages.

[0215] The first stage involves building training data for the feature extraction model:

[0216] Video tag information corresponding to each geographic region is obtained from video databases covering multiple geographic regions globally. Using the TF-IDF algorithm, a tag coefficient for each video tag is determined, region by region. For any given geographic region, the video tags are sorted based on their tag coefficients. Then, target video tags for that geographic region are filtered out to remove trending or meaningless tags, resulting in target video tags for each geographic region. For any target video tag within any geographic region, multiple video identifiers corresponding to that tag are retrieved from the database for that region. These multiple target video tags and their corresponding video identifiers are used as training data for that geographic region, resulting in training data for multiple geographic regions. This training data is then used to train the feature extraction model.

[0217] The second stage involves encoding the target video tag information:

[0218] For target video tag information corresponding to multiple geographic regions, each geographic region corresponds to multiple target video tag information, and different geographic regions may have the same video tag information. Following the order of the region identifiers of the multiple geographic regions, video identifiers are generated sequentially for each geographic region's target video tag information. That is, for each geographic region, a video category identifier is generated for each target video tag information corresponding to that geographic region, resulting in multiple video category identifiers for that geographic region. These multiple video category identifiers indicate multiple sample video categories corresponding to that geographic region, thus obtaining multiple video category identifiers for different geographic regions. For example, the multiple geographic regions include geographic region 1, geographic region 2, and geographic region 3. Geographic region 1 corresponds to 30,000 target video tag information, geographic region 2 corresponds to 20,000 target video tag information, and geographic region 3 corresponds to 30,000 target video tag information. Following the arrangement of region identifiers for three geographic regions, video category identifiers are generated for the 30,000 target video tags corresponding to geographic region 1, ranging from 0 to 29,999. Then, video category identifiers are generated for the target video tags corresponding to geographic region 2, starting encoding from 30,000, resulting in a range of 30,000-49,999. Similarly, video category identifiers are generated for the target video tags corresponding to geographic region 3, starting encoding from 50,000, resulting in a range of 50,000-79,999. The sum of the sample video categories indicated by the video category identifiers corresponding to multiple geographic regions is used as the sample video categories for training the feature extraction model.

[0219] The target video tag information corresponding to multiple geographical regions is encoded in the above manner, so that the target video tag information corresponding to each geographical region has a corresponding video category identifier. Even if different geographical regions have the same target video tag information, the target video tag information has different video category identifiers in different geographical regions. This preserves the influence of the differences in language text or expression in different geographical regions on the sample video category, so that the feature extraction model can be trained with the sample video category determined in the future, which can further improve the global unified video representation capability of the feature extraction model.

[0220] The third stage, such as Figure 4 As shown, feature extraction and model training:

[0221] Based on the training data corresponding to multiple geographical regions obtained in the first stage, the associated data of sample videos under each sample video category is acquired. This associated data includes cover images, multiple video frames, video title text, first text, second text, and sample audio. Taking four video frames as an example, the feature extraction model is iteratively trained based on the sample video categories corresponding to multiple geographical regions and the associated data of sample videos under each sample video category.

[0222] Taking four video frames as an example, based on the image feature extraction sub-model in the feature extraction model, features are extracted from the cover image and the four video frames respectively, resulting in cover features corresponding to the cover image and video frame features corresponding to each video frame. Both the cover features and video frame features are 512-dimensional feature vectors. This feature extraction sub-model is either a CNN (Convolutional Neural Networks) or a Transformer (a type of neural network). Based on the text feature extraction sub-model in the feature extraction model, features are extracted from the video title text, the first text, and the second text respectively, resulting in title features corresponding to the video title text, the first feature corresponding to the first text, and the second feature corresponding to the second text. These features are also 512-dimensional feature vectors. This text feature extraction sub-model is a multilingual BERT (Bidirectional Encoder Representation from Transformers, a type of neural network), such as multilingual (a multilingual model) or MdeBERTa (a neural network). Based on the audio feature extraction sub-model in the feature extraction model, feature extraction is performed on the sample audio to obtain audio features, where the audio features are 128-dimensional feature vectors. The audio feature extraction sub-model is VGGish (a type of neural network).

[0223] Based on the feature extraction model, the features of four video frames are weighted and fused to obtain a fused feature, which is a 512-dimensional feature vector. This fused feature is then weighted and fused with the cover feature to obtain a visual feature, which is also a 512-dimensional feature vector. Based on the feature extraction model, the title feature, the first feature, and the second feature are weighted and fused to obtain a text feature, which is also a 512-dimensional feature vector. Based on the Multilayer Perceptron (MLP) in the feature extraction model, the audio features are transformed to obtain a 512-dimensional feature vector. Based on the feature extraction model, the cover feature, visual feature, and the transformed audio feature are weighted and fused to obtain a multimodal feature, which is also a 512-dimensional feature vector.

[0224] Based on the first classification model, the cover features are processed to obtain the first video category to which the sample video belongs; based on the second classification model, the visual features are classified to obtain the second video category to which the sample video belongs; based on the second classification model, the multimodal features are classified to obtain the third video category to which the sample video belongs. The first, second, and third classification models can all be multilayer perceptrons.

[0225] Based on the sample video categories and the first video category, a first loss value is determined. Based on the sample video categories and the second video category, a second loss value is determined. Based on the sample video categories and the third video category, a third loss value is determined. The sum of the first, second, and third loss values ​​is determined. Based on the determined sum, the feature extraction model is adjusted using a gradient inversion method.

[0226] In this embodiment, the sample video belongs to multiple sample video categories, and the predicted first, second, and third video categories may each be multiple. A multi-video category approach is used to determine the loss value, and based on the determined loss value, the feature extraction model is adjusted. The overall loss is calculated through loss fusion, and then gradient backpropagation is performed on the shared feature extraction to update the weights. This approach has the advantages of low-level weight sharing and high-level discriminative fusion, thereby improving the accuracy of the feature extraction model.

[0227] In this embodiment, the cost of obtaining video tag information is low, and the feature extraction model produces multiple types of features simultaneously, which can meet the various business needs of recommendation, search, and other directions, such as recall / coarse ranking / fine ranking.

[0228] In the above Figure 3 Based on the embodiments shown, this disclosure can also obtain video features corresponding to a video based on a trained target feature extraction model, and can be applied to various scenarios. Taking a video recommendation scenario as an example, the process of recommending a video based on the target feature extraction model includes: obtaining the associated data of the target video, which includes at least one of images, text, or audio; processing the associated data of the target video based on the target feature extraction model to obtain the video features corresponding to the target video; and recommending the target video based on the video features corresponding to the target video.

[0229] In this embodiment of the disclosure, video features corresponding to the target video are obtained based on the trained feature extraction model, and the target video is recommended using the video features so as to recommend the target video to users who are interested, thereby ensuring the accuracy of video recommendation.

[0230] In this embodiment of the disclosure, the trained target feature extraction model is used to obtain various types of features of the video, namely cover features, visual features and multimodal features. The target feature extraction model can be applied in various scenarios, such as recommendation, search and other scenarios, involving many links such as recall, coarse ranking, fine ranking and diversity.

[0231] For example, a target feature extraction model can be used to obtain video features corresponding to a video, and cosine similarity can be used to calculate whether two videos are similar in content. Alternatively, in a video recommendation scenario based on the SIM model (Search-based Interest Model), the target feature extraction model can be used to obtain video features corresponding to the video currently being watched by the user, and then the target feature extraction model can be used to obtain video features corresponding to multiple videos in the video database. A similarity retrieval method can be used to retrieve the top K videos from the video database, and then these videos are fed into the exact search unit (ESU) of the SIM model for fine matching, so as to recommend more accurate videos to the user.

[0232] It should be noted that the above Figure 3 to Figure 4 The illustrated embodiment uses video as an example of multimedia resources. In another embodiment, the multimedia resources can be other resources, such as text or images. In this case, the trained target feature extraction model is used to obtain the resource features corresponding to the text or images.

[0233] It should be noted that the above embodiments use multimedia resources as an example for illustration. In another embodiment, the sample multimedia resources include sample videos, sample text, and sample images. Therefore, the above embodiments are followed. Figure 3 The embodiment shown trains the feature extraction model to be trained based on sample videos, sample texts, and sample images to obtain the target feature extraction model. The target feature extraction model has the ability to acquire features corresponding to videos, texts, or images. In other words, the target feature extraction model has the ability to acquire resource features corresponding to three types of multimedia resources. Based on this target feature extraction model, resource features corresponding to any multimedia resource can be acquired.

[0234] It should be noted that all the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this disclosure, and will not be described in detail here.

[0235] Figure 5 This is a block diagram illustrating a feature extraction model training apparatus according to an exemplary embodiment. (Refer to...) Figure 5 The device includes:

[0236] The determining unit 501 is configured to perform the determination of multiple sample resource categories corresponding to each geographic region based on the resource tag information in the resource database corresponding to each geographic region.

[0237] The acquisition unit 502 is configured to perform resource identification based on resource tag information in a resource database corresponding to multiple geographical regions, and acquire associated data of sample multimedia resources under each sample resource category. The resource identification indicates the sample multimedia resources, and the associated data includes at least one of sample images, sample text, or sample audio.

[0238] Training unit 503 is configured to train the feature extraction model to be trained based on the associated data of sample resource categories corresponding to multiple geographical regions and sample multimedia resources under each sample resource category, so as to obtain the target feature extraction model; the target feature extraction model is used to obtain the resource features corresponding to the target multimedia resources.

[0239] In some embodiments, the determining unit 501 is configured to perform the following actions: obtaining multiple resource tag information from a resource database corresponding to a geographic region; determining a tag coefficient for each resource tag information based on the multiple resource tag information, wherein the tag coefficient indicates the importance of the resource tag information; selecting target resource tag information whose tag coefficient belongs to a target interval from the multiple resource tag information based on the tag coefficients of the multiple resource tag information; and determining a sample resource category based on each target resource tag information to obtain multiple sample resource categories corresponding to the geographic region.

[0240] In some embodiments, the determining unit 501 is configured to sort multiple geographic regions according to the region identifiers corresponding to the multiple geographic regions; and generate a resource category identifier for the target resource label information corresponding to each geographic region according to the sorting order of the multiple geographic regions, so as to obtain multiple resource category identifiers corresponding to each geographic region. The multiple resource category identifiers corresponding to the geographic region indicate multiple sample resource categories corresponding to the geographic region, and the resource category identifiers corresponding to different geographic regions are different.

[0241] In some embodiments, the training unit 503 is configured to perform association data of sample multimedia resources under any sample resource category, process the association data based on the feature extraction model to be trained, obtain sample resource features corresponding to the sample multimedia resources, perform category prediction processing on the sample resource features corresponding to the sample multimedia resources, obtain the predicted resource category to which the sample multimedia resources belong, and train the feature extraction model to be trained based on the sample resource category and the predicted resource category to obtain the target feature extraction model.

[0242] In some embodiments, the training unit 503 is configured to perform feature extraction on the associated data based on the feature extraction model to be trained, when the associated data includes at least two of the sample images, sample text, or sample audio, to obtain data features, wherein the data features include at least two of the image features corresponding to the sample images, the text features corresponding to the sample text, or the audio features corresponding to the sample audio; and to fuse the features contained in the data features based on the feature extraction model to be trained to obtain multi-modal features corresponding to the sample multimedia resources, wherein the multi-modal features are sample resource features corresponding to the sample multimedia resources.

[0243] In some embodiments, the sample multimedia resource is a sample video, and the associated data includes sample images, which include a cover image and multiple video frames; the training unit 503 is configured to perform feature extraction based on the feature extraction model to be trained, extracting features from the cover image and each video frame respectively, to obtain cover features corresponding to the cover image and video frame features corresponding to each video frame; based on the feature extraction model to be trained, fusing the cover features and multiple video frame features to obtain visual features corresponding to the sample video; and determining the cover features and visual features as sample resource features corresponding to the sample video.

[0244] In some embodiments, the predicted resource category includes a first resource category and a second resource category. The first resource category is obtained by performing category prediction processing on the cover features; the second resource category is obtained by performing category prediction processing on the visual features. The training unit 503 is configured to perform the following operations: determine a first loss value based on the sample resource category and the first resource category; determine a second loss value based on the sample resource category and the second resource category; and train the feature extraction model to be trained based on the sum of the first loss value and the second loss value to obtain the target feature extraction model.

[0245] In some embodiments, the predicted resource category further includes a third resource category, and the sample resource features corresponding to the sample multimedia resources further include multi-modal features. The multi-modal features are obtained by fusing at least two of the following: image features, text features, or audio features. The image features are the features corresponding to the sample images in the associated data, the text features are the features corresponding to the sample text in the associated data, and the audio features are the features corresponding to the sample audio in the associated data. The training unit 503 is further configured to perform category prediction processing on the multi-modal features to obtain the third resource category; determine a third loss value based on the sample resource category and the third resource category; and train the feature extraction model to be trained based on the sum of the first loss value, the second loss value, and the third loss value to obtain the target feature extraction model.

[0246] In some embodiments, the apparatus further includes:

[0247] The acquisition unit 502 is further configured to acquire associated data of the target multimedia resource, the associated data of the target multimedia resource including at least one of images, text or audio.

[0248] The processing unit 504 is configured to execute a target feature extraction model to process the associated data of the target multimedia resource to obtain the resource features corresponding to the target multimedia resource.

[0249] Recommendation unit 505 is configured to recommend target multimedia resources based on the resource features corresponding to the target multimedia resources.

[0250] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the feature extraction model training apparatus and the feature extraction model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0251] In an exemplary embodiment, an electronic device is also provided, the electronic device comprising:

[0252] One or more processors;

[0253] Memory used to store processor-executable program code;

[0254] The processor is configured to execute the program code to implement the aforementioned feature extraction model training method.

[0255] In some embodiments, when the electronic device is provided as a terminal Figure 7 This is a block diagram illustrating a terminal 700 according to an exemplary embodiment. The terminal... Figure 7 A structural block diagram of a terminal 700 provided in an exemplary embodiment of the present disclosure is shown. Typically, the terminal 700 includes a processor 701 and a memory 702.

[0256] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0257] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one program code, which is executed by the processor 701 to implement the feature extraction model training method provided in the method embodiments of this disclosure.

[0258] In some embodiments, the terminal 700 may also optionally include a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, and a power supply 708.

[0259] Peripheral device interface 703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702 and peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0260] The radio frequency (RF) circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 704 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this disclosure.

[0261] Display screen 705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 705, which serves as the front panel of terminal 700; in other embodiments, there may be at least two display screens 705, respectively disposed on different surfaces of terminal 700 or in a folded design; in still other embodiments, display screen 705 may be a flexible display screen, disposed on a curved or folded surface of terminal 700. Furthermore, display screen 705 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 705 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0262] The camera assembly 706 is used to acquire images or videos. Optionally, the camera assembly 706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0263] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 701 for processing, or input to the radio frequency circuit 704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0264] Power supply 708 is used to power the various components in terminal 700. Power supply 708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 708 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0265] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on terminal 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0266] In some embodiments, when the electronic device is provided as a server Figure 8 This is a block diagram illustrating a server 800 according to an exemplary embodiment. The server 800 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 801 and one or more memories 802. The memory 802 stores at least one line of program code, which is loaded and executed by the processor 801 to implement the feature extraction model training method provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 800 may also include other components for implementing device functions, which will not be elaborated upon here.

[0267] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0268] In an exemplary embodiment, a computer program product includes a computer program / instruction that, when executed by a processor, implements the above-described feature extraction model training method.

[0269] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0270] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training a feature extraction model, characterized in that, The method includes: Based on the resource tag information in the resource database corresponding to each geographic region, multiple sample resource categories corresponding to each geographic region are determined. Based on the resource identifiers corresponding to the resource tag information in the resource database corresponding to multiple geographical regions, the associated data of sample multimedia resources under each sample resource category is obtained, wherein the resource identifiers indicate the sample multimedia resources; The sample multimedia resources are sample videos, and the associated data includes sample images, which include cover images and multiple video frames. For the associated data of sample multimedia resources under any sample resource category, based on the feature extraction model to be trained, features are extracted from the cover image and each video frame to obtain the cover features corresponding to the cover image and the video frame features corresponding to each video frame. Based on the feature extraction model to be trained, the cover features and features from multiple video frames are fused to obtain the visual features corresponding to the sample video. The sample resource features, which are composed of the cover features and the visual features, are subjected to category prediction processing to obtain the predicted resource category to which the sample video belongs; Based on the sample resource categories and predicted resource categories corresponding to the multiple geographical regions, the feature extraction model to be trained is trained to obtain the target feature extraction model; the target feature extraction model is used to obtain the resource features corresponding to the target multimedia resources.

2. The method according to claim 1, characterized in that, The determination of multiple sample resource categories corresponding to each geographical region based on resource tag information in the resource database corresponding to each geographical region includes: Obtain multiple resource tag information from the resource database corresponding to the geographical region; Based on the multiple resource tag information, a tag coefficient is determined for each resource tag information, and the tag coefficient indicates the importance of the resource tag information; Based on the tag coefficients of the multiple resource tag information, select target resource tag information whose tag coefficients belong to the target interval from the multiple resource tag information; Based on the tag information of each target resource, a sample resource category is determined to obtain multiple sample resource categories corresponding to the geographical region.

3. The method according to claim 2, characterized in that, The step of determining a sample resource category based on each target resource tag information to obtain multiple sample resource categories corresponding to the geographical region includes: The multiple geographical regions are sorted according to their corresponding region identifiers; According to the sorting order of the multiple geographical regions, a resource category identifier is generated for the target resource tag information corresponding to each geographical region, so as to obtain multiple resource category identifiers corresponding to each geographical region. The multiple resource category identifiers corresponding to the geographical region indicate multiple sample resource categories corresponding to the geographical region. Different geographical regions have different resource category identifiers.

4. The method according to claim 1, characterized in that, The method further includes: If the associated data also includes at least one of sample text or sample audio, feature extraction is performed on the associated data based on the feature extraction model to be trained to obtain data features. The data features include at least two of the image features corresponding to the sample image, the text features corresponding to the sample text, or the audio features corresponding to the sample audio. Based on the feature extraction model to be trained, the features contained in the data features are fused to obtain the multi-modal features corresponding to the sample multimedia resources, and the multi-modal features are the sample resource features corresponding to the sample multimedia resources.

5. The method according to claim 1, characterized in that, The predicted resource category includes a first resource category and a second resource category. The first resource category is obtained by performing category prediction processing on the cover features; the second resource category is obtained by performing category prediction processing on the visual features. The step of training the feature extraction model to be trained based on the sample resource categories and predicted resource categories corresponding to the multiple geographical regions to obtain the target feature extraction model includes: Based on the sample resource category and the first resource category, a first loss value is determined; Based on the sample resource category and the second resource category, a second loss value is determined; Based on the sum of the first loss value and the second loss value, the feature extraction model to be trained is trained to obtain the target feature extraction model.

6. The method according to claim 5, characterized in that, The predicted resource category further includes a third resource category, and the sample resource features corresponding to the sample multimedia resources further include multi-modal features. The multi-modal features are obtained by fusing at least one of image features and text features or audio features. The text features are the features corresponding to the sample text in the associated data, and the audio features are the features corresponding to the sample audio in the associated data. The method further includes: The multi-modal features are subjected to category prediction processing to obtain the third resource category; Based on the sample resource category and the third resource category, a third loss value is determined; The step of training the feature extraction model to be trained based on the sum of the first loss value and the second loss value to obtain the target feature extraction model includes: The target feature extraction model is trained based on the sum of the first loss value, the second loss value, and the third loss value.

7. The method according to any one of claims 1-6, characterized in that, After training the feature extraction model based on the sample resource categories and predicted resource categories corresponding to the multiple geographical regions to obtain the target feature extraction model, the method further includes: Obtain associated data of the target multimedia resource, wherein the associated data of the target multimedia resource includes images; Based on the target feature extraction model, the associated data of the target multimedia resource is processed to obtain the resource features corresponding to the target multimedia resource; Based on the resource characteristics corresponding to the target multimedia resource, the target multimedia resource is recommended.

8. A feature extraction model training device, characterized in that, The device includes: The determining unit is configured to perform an operation based on resource tag information in the resource database corresponding to each geographic region to determine multiple sample resource categories corresponding to each geographic region. The acquisition unit is configured to execute resource identifiers corresponding to resource tag information in a resource database corresponding to multiple geographical regions, and acquire associated data of sample multimedia resources under each sample resource category, wherein the resource identifiers indicate the sample multimedia resources; The sample multimedia resources are sample videos, and the associated data includes sample images, which include cover images and multiple video frames. The training unit is configured to perform feature extraction on the cover image and each video frame based on the feature extraction model to be trained, obtaining cover features corresponding to the cover image and video frame features corresponding to each video frame. Based on the feature extraction model to be trained, the cover features and multiple video frame features are fused to obtain the visual features corresponding to the sample video. Category prediction processing is performed on the sample resource features composed of the cover features and the visual features to obtain the predicted resource category to which the sample video belongs. Based on the sample resource categories and predicted resource categories corresponding to the multiple geographical regions, the feature extraction model to be trained is trained to obtain a target feature extraction model. The target feature extraction model is used to obtain the resource features corresponding to the target multimedia resources.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the feature extraction model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the feature extraction model training method as described in any one of claims 1 to 7.

Images