Processing method, device, electronic device and medium based on feature extraction model

By acquiring and fusing the text and resource features of video information, training the feature extraction model, the problem of low accuracy of video resource retrieval in the prior art is solved, and the accuracy of the search results is improved.

CN113869272BActive Publication Date: 2024-12-06BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111193177.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-13
Publication Date
2024-12-06
Estimated Expiration
2041-10-13

AI Technical Summary

Technical Problem

The existing feature extraction model has low accuracy in video resource retrieval, making it difficult to effectively match the search information and video features entered by users.

Method used

By obtaining the search features and associated video information of sample search information, combining text features and resource features to generate video features, and using these features to train the feature extraction model to improve the accuracy of the model.

Benefits of technology

The feature extraction model's ability to identify text information and video resources in video information is enhanced, the accuracy of search results is improved, and the retrieved video information is highly correlated with the search information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113869272B_ABST
    Figure CN113869272B_ABST
Patent Text Reader

Abstract

The present disclosure is about a processing method, device, electronic device and storage medium based on a feature extraction model, and belongs to the field of computer technology. The method includes: obtaining the retrieval feature of sample retrieval information; based on the feature extraction model, obtaining the first text feature of the first text information in the first video information associated with the sample retrieval information and the first resource feature of the first video resource, fusing the first text feature with the first resource feature to obtain the first video feature; based on the feature extraction model, obtaining the second resource feature of the second video resource in the second video information not associated with the sample retrieval information, fusing the first text feature with the second resource feature to obtain the second video feature; based on the matching relationship between the first video feature and the second video feature and the retrieval feature, respectively, the feature extraction model is trained to improve the accuracy of the feature extraction model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a processing method, device, electronic device and medium based on a feature extraction model. Background Art

[0002] With the development of computer technology, video resources are becoming more and more abundant, and users' demand for searching video resources is also increasing. In the related search method, based on the feature extraction model, the search feature of the search information input by the user and the video feature of at least one video information are obtained, and based on the search feature and the video feature, the video information associated with the search information is selected, but the accuracy of the feature extraction model is poor. Summary of the invention

[0003] The present disclosure provides a processing method, device, electronic device and medium based on a feature extraction model, which improves the accuracy of the feature extraction model.

[0004] According to one aspect of an embodiment of the present disclosure, a processing method based on a feature extraction model is provided, the method comprising:

[0005] Acquire a search feature of sample search information, first video information associated with the sample search information, and second video information not associated with the sample search information, wherein the first video information includes first text information and a first video resource, the first text information is used to describe the first video resource, and the second video information includes a second video resource;

[0006] Based on the feature extraction model, obtaining a first text feature of the first text information in the first video information and a first resource feature of the first video resource, and fusing the first text feature with the first resource feature to obtain a first video feature;

[0007] Based on the feature extraction model, obtaining a second resource feature of the second video resource in the second video information, and fusing the first text feature with the second resource feature to obtain a second video feature;

[0008] Based on the matching relationship between the first video feature and the second video feature and the retrieval feature respectively, the feature extraction model is trained, and the feature extraction model is used to obtain the features of any video information.

[0009] In some embodiments, the training of the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature respectively includes:

[0010] Determining a first similarity between the retrieval feature and the first video feature, and a second similarity between the retrieval feature and the second video feature;

[0011] The feature extraction model is trained based on the first similarity and the second similarity.

[0012] In some embodiments, the second video information further includes second text information, and the second text information is used to describe the second video resource. The method of obtaining a second resource feature of the second video resource in the second video information based on the feature extraction model and fusing the first text feature with the second resource feature to obtain a second video feature includes:

[0013] Based on the feature extraction model, obtaining a second text feature of the second text information in the second video information and a second resource feature of the second video resource, fusing the second text feature with the second resource feature to obtain a third video feature of the second video information, and fusing the first text feature with the second resource feature to obtain a second video feature;

[0014] The training of the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature respectively includes:

[0015] The feature extraction model is trained based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

[0016] In some embodiments, the training of the feature extraction model based on the matching relationships between the first video feature, the second video feature, and the third video feature and the search feature respectively includes:

[0017] Determining a first similarity between the retrieval feature and the first video feature, a second similarity between the retrieval feature and the second video feature, and a third similarity between the retrieval feature and the third video feature;

[0018] The feature extraction model is trained based on the first similarity, the second similarity, and the third similarity.

[0019] In some embodiments, the first video resource includes a plurality of video frames, and the process of obtaining the first resource feature of the first video resource based on the feature extraction model includes:

[0020] Determine a key frame among the plurality of video frames, wherein the key frame contains main content described by the first video resource;

[0021] Based on the feature extraction model, feature extraction is performed on the key frame to obtain frame features of the key frame, and based on the frame features of the key frame, the first video feature is determined.

[0022] In some embodiments, determining the first video feature based on the frame feature of the key frame includes:

[0023] When one of the key frames is determined, a frame feature of the key frame is determined as the first video feature;

[0024] When a plurality of key frames are determined, frame features of the plurality of key frames are fused to obtain a fused feature, and the fused feature is determined as the first video feature.

[0025] In some embodiments, determining a key frame among the plurality of video frames comprises:

[0026] Obtaining frame features of each of the video frames;

[0027] Clustering frame features of the plurality of video frames to obtain cluster centers;

[0028] From the multiple video frames, the video frame whose frame feature is closest to the cluster center is determined as the key frame.

[0029] In some embodiments, after training the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature, the method further includes:

[0030] Obtaining a search feature of any search information and a plurality of video information, each of the video information including text information and video resources;

[0031] For each of the video information, based on the feature extraction model, obtain text features of the text information in the video information and resource features of the video resources, and fuse the text features with the resource features to obtain video features of the video information;

[0032] Based on the similarity between the search feature and the video feature of each of the video information, at least one video information with the greatest similarity is determined as the video information associated with the search information.

[0033] According to another aspect of the present disclosure, a processing method based on a feature extraction model is provided, the method comprising:

[0034] Obtaining a search feature of any search information and a plurality of video information, each of the video information including text information and video resources;

[0035] For each of the video information, based on the feature extraction model, obtain text features of the text information in the video information and resource features of the video resources, and fuse the text features with the resource features to obtain video features of the video information;

[0036] Based on the similarity between the search feature and the video feature of each of the video information, determining at least one video information with the greatest similarity as the video information associated with the search information;

[0037] Among them, the feature extraction model is trained based on the matching relationship between the first video feature and the second video feature and the retrieval features of the sample retrieval information respectively, the first video information corresponding to the first video feature is associated with the sample retrieval information and includes first text information and a first video resource, the video information corresponding to the second video feature includes text information associated with the sample retrieval information, but the included video resources are not associated with the sample retrieval information.

[0038] In some embodiments, the method further comprises:

[0039] Acquire the search feature of the sample search information, the first video information associated with the sample search information, and the second video information not associated with the sample search information, wherein the first text information is used to describe the first video resource, and the second video information includes a second video resource;

[0040] Based on the feature extraction model, obtaining a first text feature of the first text information in the first video information and a first resource feature of the first video resource, and fusing the first text feature with the first resource feature to obtain the first video feature;

[0041] Based on the feature extraction model, obtaining a second resource feature of the second video resource in the second video information, and fusing the first text feature with the second resource feature to obtain the second video feature;

[0042] The feature extraction model is trained based on the matching relationships between the first video feature and the second video feature and the retrieval feature respectively.

[0043] In some embodiments, the second video information further includes second text information, and the second text information is used to describe the second video resource. The method of obtaining a second resource feature of the second video resource in the second video information based on the feature extraction model and fusing the first text feature with the second resource feature to obtain a second video feature includes:

[0044] Based on the feature extraction model, obtaining a second text feature of the second text information in the second video information and a second resource feature of the second video resource, fusing the second text feature with the second resource feature to obtain a third video feature of the second video information, and fusing the first text feature with the second resource feature to obtain the second video feature;

[0045] The training of the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature respectively includes:

[0046] The feature extraction model is trained based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

[0047] In some embodiments, the first video resource includes a plurality of video frames, and the process of obtaining the first resource feature of the first video resource based on the feature extraction model includes:

[0048] Determine a key frame among the plurality of video frames, wherein the key frame contains main content described by the first video resource;

[0049] Based on the feature extraction model, feature extraction is performed on the key frame to obtain frame features of the key frame, and based on the frame features of the key frame, the first video feature is determined.

[0050] According to another aspect of the embodiments of the present disclosure, a processing device based on a feature extraction model is provided, the device comprising:

[0051] An acquiring unit is configured to acquire a retrieval feature of sample retrieval information, first video information associated with the sample retrieval information, and second video information not associated with the sample retrieval information, wherein the first video information includes first text information and a first video resource, the first text information is used to describe the first video resource, and the second video information includes a second video resource;

[0052] The acquisition unit is further configured to execute based on the feature extraction model, acquire the first text feature of the first text information in the first video information and the first resource feature of the first video resource, and fuse the first text feature with the first resource feature to obtain the first video feature;

[0053] The acquisition unit is further configured to acquire a second resource feature of the second video resource in the second video information based on the feature extraction model, and fuse the first text feature with the second resource feature to obtain a second video feature;

[0054] The training unit is configured to perform training on the feature extraction model based on the matching relationship between the first video feature and the second video feature and the retrieval feature respectively, and the feature extraction model is used to obtain the features of any video information.

[0055] In some embodiments, the training unit is configured to determine a first similarity between the retrieval feature and the first video feature, and a second similarity between the retrieval feature and the second video feature; and train the feature extraction model based on the first similarity and the second similarity.

[0056] In some embodiments, the second video information further includes second text information, and the second text information is used to describe the second video resource. The acquisition unit is configured to execute based on the feature extraction model, acquire the second text feature of the second text information in the second video information and the second resource feature of the second video resource, fuse the second text feature with the second resource feature to obtain the third video feature of the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature;

[0057] The training unit is configured to perform training on the feature extraction model based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

[0058] In some embodiments, the training unit is configured to determine a first similarity between the retrieval feature and the first video feature, a second similarity between the retrieval feature and the second video feature, and a third similarity between the retrieval feature and the third video feature; and train the feature extraction model based on the first similarity, the second similarity and the third similarity.

[0059] In some embodiments, the first video resource includes a plurality of video frames, and the acquisition unit includes:

[0060] A determination subunit is configured to determine a key frame among the plurality of video frames, wherein the key frame contains a main content described by the first video resource;

[0061] The determination subunit is further configured to perform feature extraction on the key frame based on the feature extraction model to obtain frame features of the key frame, and determine the first video feature based on the frame features of the key frame.

[0062] In some embodiments, the determination subunit is configured to, when one key frame is determined, determine the frame feature of the key frame as the first video feature; when multiple key frames are determined, fuse the frame features of multiple key frames to obtain a fused feature, and determine the fused feature as the first video feature.

[0063] In some embodiments, the determination subunit is configured to execute the acquisition of frame features of each of the video frames; clustering the frame features of multiple video frames to obtain a cluster center; and determining, from the multiple video frames, a video frame whose frame features are closest to the cluster center as the key frame.

[0064] In some embodiments, the apparatus further comprises:

[0065] The acquisition unit is further configured to execute the acquisition of a search feature of any search information and a plurality of video information, each of the video information including text information and video resources;

[0066] The acquisition unit is further configured to execute, for each of the video information, based on the feature extraction model, acquiring text features of the text information in the video information and resource features of the video resources, and fusing the text features with the resource features to obtain video features of the video information;

[0067] The determination unit is further configured to perform a similarity between the search feature and each video feature of the video information, and determine at least one video information with the greatest similarity as the video information associated with the search information.

[0068] According to another aspect of the embodiments of the present disclosure, a processing device based on a feature extraction model is provided, the device comprising:

[0069] An acquisition unit, which acquires a search feature configured to execute acquisition of any search information and a plurality of video information, each of the video information including text information and video resources;

[0070] The acquisition unit is further configured to execute, for each of the video information, based on the feature extraction model, acquiring text features of the text information in the video information and resource features of the video resources, and fusing the text features with the resource features to obtain video features of the video information;

[0071] The determination unit is further configured to perform, based on the similarity between the search feature and the video feature of each of the video information, determining at least one video information with the greatest similarity as the video information associated with the search information;

[0072] Among them, the feature extraction model is trained based on the matching relationship between the first video feature and the second video feature and the retrieval features of the sample retrieval information respectively, the first video information corresponding to the first video feature is associated with the sample retrieval information and includes first text information and a first video resource, the video information corresponding to the second video feature includes text information associated with the sample retrieval information, but the included video resources are not associated with the sample retrieval information.

[0073] In some embodiments, the apparatus further comprises:

[0074] The acquisition unit is further configured to execute acquisition of the retrieval feature of the sample retrieval information, the first video information associated with the sample retrieval information, and the second video information not associated with the sample retrieval information, the first text information is used to describe the first video resource, and the second video information includes a second video resource;

[0075] The acquisition unit is further configured to acquire, based on the feature extraction model, a first text feature of the first text information in the first video information and a first resource feature of the first video resource, and fuse the first text feature with the first resource feature to obtain the first video feature;

[0076] The acquisition unit is further configured to acquire a second resource feature of the second video resource in the second video information based on the feature extraction model, and fuse the first text feature with the second resource feature to obtain the second video feature;

[0077] The training unit is configured to perform training on the feature extraction model based on the matching relationship between the first video feature and the second video feature and the retrieval feature respectively.

[0078] In some embodiments, the second video information further includes second text information, and the second text information is used to describe the second video resource. The acquisition unit is configured to execute based on the feature extraction model, acquire the second text feature of the second text information in the second video information and the second resource feature of the second video resource, fuse the second text feature with the second resource feature to obtain the third video feature of the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature.

[0079] The training unit is configured to perform training on the feature extraction model based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

[0080] In some embodiments, the first video resource includes multiple video frames, and the acquisition unit is configured to determine key frames among the multiple video frames, where the key frames contain the main content described by the first video resource; based on the feature extraction model, perform feature extraction on the key frames to obtain frame features of the key frames, and determine the first video features based on the frame features of the key frames.

[0081] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0082] processor;

[0083] a memory for storing instructions executable by the processor;

[0084] The processor is configured to execute the instructions to implement the processing method based on the feature extraction model described in the above aspect.

[0085] According to another aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the processing method based on the feature extraction model described in the above aspect.

[0086] According to another aspect of an embodiment of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the processing method based on the feature extraction model described in the above aspect.

[0087] The processing method, device, electronic device and storage medium based on the feature extraction model provided by the embodiments of the present disclosure, since the first text information in the first video information is associated with the sample retrieval information, and the second video resource in the second video information is not associated with the sample retrieval information, the first text feature of the first text information and the second resource feature of the second video resource are fused to obtain the second video feature, the first video feature of the first video information is used as the positive sample feature, and the second video feature is used as the negative sample feature, and the feature extraction model is trained based on the retrieval feature, the positive sample feature and the negative sample feature, so as to enrich the features of the trained feature extraction model, take into account the influence of the video information including text information associated with the retrieval information and including video resources not associated with the retrieval information on the feature extraction model, so as to enhance the feature extraction model's recognition ability of the text information and video resources contained in the video information, thereby improving the accuracy of the feature extraction model.

[0088] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0090] Figure 1 It is a schematic diagram showing an implementation environment according to an exemplary embodiment.

[0091] Figure 2 The invention is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment.

[0092] Figure 3 The invention is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment.

[0093] Figure 4 The figure is a flowchart of a training feature extraction model according to an exemplary embodiment.

[0094] Figure 5 The invention is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment.

[0095] Figure 6 The invention is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment.

[0096] Figure 7 The figure is a schematic diagram showing the similarity between retrieved video information and retrieved information according to an exemplary embodiment.

[0097] Figure 8 It is a block diagram of a processing device based on a feature extraction model according to an exemplary embodiment.

[0098] Fig. 9 It is a block diagram of a processing device based on a feature extraction model according to an exemplary embodiment.

[0099] Fig.10 It is a block diagram of a processing device based on a feature extraction model according to an exemplary embodiment.

[0100] Fig.11 It is a block diagram of a processing device based on a feature extraction model according to an exemplary embodiment.

[0101] Fig.12 It is a block diagram of a terminal according to an exemplary embodiment.

[0102] Fig.13 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0103] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0104] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned figure descriptions are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0105] The terms "at least one", "a plurality of", "each" and "any" used in the present disclosure include one, two or more than two, a plurality of includes two or more than two, each refers to each of the corresponding plurality, and any refers to any one of the plurality. For example, the plurality of video information includes 3 video information, and each refers to each of the 3 video information, and any refers to any one of the 3 video information, which can be the first video information, or the second video information, or the third video information.

[0106] It should be noted that the user information involved in this disclosure (including but not limited to user device information, user personal information, etc.) is all information authorized by the user or fully authorized by all parties.

[0107] The processing based on the feature extraction model provided in the embodiments of the present disclosure is performed by an electronic device. In some embodiments, the electronic device is a terminal, for example, the terminal is a mobile phone, a tablet computer, a computer, and other types of terminals. In some embodiments, the electronic device is a server, for example, the server is a server, or a server cluster composed of several servers, or a cloud computing service center.

[0108] In some embodiments, the electronic device is provided as a server. Figure 1 It is a schematic diagram of an implementation environment provided according to an exemplary embodiment. The implementation environment includes: a terminal 101 and a server 102. The terminal 101 can interact with the server 102 through a network connection.

[0109] The server 102 is used to train the feature extraction model and store the feature extraction model. Alternatively, after the feature extraction model is trained based on the server 102, the developer deploys the feature extraction model in other storage locations so that the server 102 can subsequently call the stored feature extraction model.

[0110] In some embodiments, the server 102 also provides information retrieval services. The terminal 101 sends a retrieval request to the server 102 through a network connection with the server 102, and the retrieval request carries retrieval information. The server receives the retrieval request, determines the video information associated with the retrieval information carried in the retrieval request based on the feature extraction model, and sends the video information associated with the retrieval information to the terminal 101. The terminal 101 receives the video information and displays the video information.

[0111] In a possible implementation of the above embodiment, the terminal 101 installs a video application provided by the server 102, and the terminal 101 can implement functions such as video retrieval and video sharing through the target application. The terminal 101 logs in to the video application based on an account, sends a retrieval request to the server 102 based on the video application, and the server 102 receives the retrieval request, determines the video information associated with the retrieval information carried in the retrieval request based on the feature extraction model, and sends the video information associated with the retrieval information to the terminal 101. The terminal 101 receives the video information and displays the video information based on the video application.

[0112] Figure 2 is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment. Figure 2 The method is performed by an electronic device and comprises the following steps:

[0113] 201. An electronic device obtains retrieval features of sample retrieval information, first video information associated with the sample retrieval information, and second video information not associated with the sample retrieval information, wherein the first video information includes first text information and a first video resource, the first text information is used to describe the first video resource, and the second video information includes a second video resource.

[0114] The sample retrieval information is any information, and the retrieval feature is used to characterize the sample retrieval information. The first video resource and the second video resource are videos of any type, for example, the first video resource is a movie, a short video, etc., and the second video resource is a movie, a short video, etc. The first video information is associated with the sample retrieval information, and the second video information is not associated with the sample retrieval information. Then, the first text information and the first video resource in the first video information are both associated with the sample retrieval information, and the second video resource in the second video information is not associated with the sample retrieval information.

[0115] 202. The electronic device obtains a first text feature of the first text information in the first video information and a first resource feature of the first video resource based on a feature extraction model, and fuses the first text feature with the first resource feature to obtain a first video feature.

[0116] Among them, the feature extraction model is used to obtain the features of any video information, and the feature extraction model is the model to be trained. The first text feature is used to characterize the first text information, and the first resource feature is used to characterize the first video resource. In the embodiment of the present disclosure, the first video information is equivalent to the positive sample video information of the sample retrieval information, and the first video feature is equivalent to the positive sample video feature corresponding to the sample retrieval information. Since the first video information includes the first text information and the first video resource, the first text feature of the first text information and the first resource feature of the first video resource are first obtained respectively, and then the first text feature and the first resource feature are fused so that the fused first video feature can characterize the first video information.

[0117] 203. The electronic device obtains a second resource feature of a second video resource in the second video information based on a feature extraction model, and fuses the first text feature with the second resource feature to obtain a second video feature.

[0118] The second resource feature is used to characterize the second video resource.

[0119] Assume that the text information contained in a video information is associated with the search information, but the video resource contained is not associated with the search information. When searching based on the search information, if the video information is regarded as the video information associated with the search, the retrieved video resource will be inaccurate, affecting the accuracy of the search result. Therefore, in order to improve the accuracy of the search result, such video information can be regarded as video information not associated with the search information.

[0120] Accordingly, in the process of training the feature extraction model, the first text information is associated with the sample retrieval information, and the second video resource is not associated with the sample retrieval information. The first text feature of the first text information and the second resource feature of the second video resource are fused to obtain the second video feature. The second video feature is used to represent the video information whose text information is associated with the sample retrieval information but whose video resource is not associated with the sample retrieval information. In addition, the second video feature is used as a negative sample video feature corresponding to the sample retrieval information to indicate that the video information is not associated with the sample retrieval information.

[0121] 204. The electronic device trains a feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature, respectively. The feature extraction model is used to obtain features of any video information.

[0122] The first video feature is equivalent to the positive sample video feature corresponding to the sample retrieval information, and the second video feature is equivalent to the negative sample video feature corresponding to the sample retrieval information. The matching relationship between the first video feature and the retrieval feature, and the matching relationship between the second video feature and the retrieval feature, can reflect the accuracy of the feature extraction model. Based on the matching relationship between the first video feature and the second video feature and the retrieval feature, the feature extraction model is trained to improve the accuracy of the feature extraction model.

[0123] The method provided by the embodiment of the present disclosure, since the first text information in the first video information is associated with the sample retrieval information, and the second video resource in the second video information is not associated with the sample retrieval information, the first text feature of the first text information is merged with the second resource feature of the second video resource to obtain the second video feature, the first video feature of the first video information is used as the positive sample feature, and the second video feature is used as the negative sample feature, and a feature extraction model is trained based on the retrieval feature, the positive sample feature and the negative sample feature, so as to enrich the features of the trained feature extraction model, take into account the influence of video information including text information associated with the retrieval information and including video resources not associated with the retrieval information on the feature extraction model, so as to enhance the feature extraction model's recognition ability of the text information and video resources contained in the video information, thereby improving the accuracy of the feature extraction model.

[0124] In some embodiments, based on the matching relationship between the first video feature and the second video feature and the search feature, respectively, the feature extraction model is trained, including:

[0125] Determining a first similarity between the search feature and the first video feature, and a second similarity between the search feature and the second video feature;

[0126] Based on the first similarity and the second similarity, the feature extraction model is trained.

[0127] Since the first video feature is equivalent to the positive sample video feature of the sample retrieval information, and the second video feature is equivalent to the negative sample video feature of the sample retrieval information, the feature extraction model is trained based on the first similarity and the second similarity to increase the first similarity and decrease the second similarity, thereby improving the accuracy of the feature extraction model.

[0128] In some embodiments, the second video information further includes second text information, the second text information is used to describe the second video resource, based on the feature extraction model, a second resource feature of the second video resource in the second video information is obtained, and the first text feature is merged with the second resource feature to obtain the second video feature, including:

[0129] Based on the feature extraction model, obtain a second text feature of the second text information in the second video information and a second resource feature of the second video resource, fuse the second text feature with the second resource feature to obtain a third video feature of the second video information, and fuse the first text feature with the second resource feature to obtain a second video feature;

[0130] Based on the matching relationship between the first video feature and the second video feature and the search feature, respectively, the feature extraction model is trained, including:

[0131] The feature extraction model is trained based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature, respectively.

[0132] In the disclosed embodiment, the retrieval feature is a feature corresponding to the sample retrieval information, the first video feature is a positive sample feature associated with the sample retrieval information, the second video feature and the third video feature are negative sample features not associated with the retrieval feature, and the second video feature is a negative sample feature generated based on the first video information and the second video information, thereby enriching the features of the trained feature extraction model, taking into account the influence of various negative sample features on the feature extraction model, and training the feature extraction model based on the retrieval feature, the first video feature, the second video feature and the third video feature to enhance the feature extraction model's recognition capability for positive sample video information and negative sample video information, thereby improving the accuracy of the feature extraction model.

[0133] In some embodiments, based on the matching relationship between the first video feature, the second video feature, and the third video feature and the search feature, respectively, the feature extraction model is trained, including:

[0134] Determining a first similarity between the search feature and the first video feature, a second similarity between the search feature and the second video feature, and a third similarity between the search feature and the third video feature;

[0135] The feature extraction model is trained based on the first similarity, the second similarity, and the third similarity.

[0136] Since the first video feature is equivalent to the positive sample video feature of the sample retrieval information, and the second video feature and the third video feature are equivalent to the negative sample video feature of the sample retrieval information, the feature extraction model is trained based on the first similarity, the second similarity and the third similarity to increase the first similarity and reduce the second similarity and the third similarity, thereby improving the accuracy of the feature extraction model.

[0137] In some embodiments, the first video resource includes multiple video frames, and the process of obtaining the first resource feature of the first video resource based on the feature extraction model includes:

[0138] Determine a key frame among the multiple video frames, the key frame including the main content described by the first video resource;

[0139] Based on the feature extraction model, feature extraction is performed on the key frame to obtain frame features of the key frame, and based on the frame features of the key frame, the first video feature is determined.

[0140] Since the first video resource includes multiple video frames, the key frame representing the first video resource in the multiple video frames is determined, and the first video feature is determined based on the frame features of the key frame. In this way, there is no need to obtain the frame features of each video frame in the first video resource, which reduces the amount of calculation, saves resources, and can also ensure the accuracy of the first video feature.

[0141] In some embodiments, determining the first video feature based on the frame feature of the key frame includes:

[0142] When a key frame is determined, a frame feature of the key frame is determined as a first video feature;

[0143] When multiple key frames are determined, frame features of the multiple key frames are fused to obtain a fused feature, and the fused feature is determined as the first video feature.

[0144] Since the first video resource may contain one or more key frames, the first video feature is obtained in different ways based on different situations in which the key frames are determined to be one or more, so as to ensure the accuracy of the obtained video feature.

[0145] In some embodiments, determining a key frame among a plurality of video frames includes:

[0146] Get frame features of each video frame;

[0147] Clustering frame features of multiple video frames to obtain cluster centers;

[0148] From multiple video frames, the video frame whose frame feature is closest to the cluster center is determined as the key frame.

[0149] Since the cluster center is the center of the frame features of multiple video frames, the cluster center can reflect the main content described by the multiple video frames, and the distance between the frame features and the cluster center can reflect the correlation between the corresponding video frame and the main content corresponding to the first video resource. Therefore, the video frame with the closest distance between the frame features and the cluster center is determined as the key frame to ensure that the video frame can represent the first video resource and ensure the accuracy of the selected key frame.

[0150] In some embodiments, after training the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature, the method further includes:

[0151] Obtaining search features of any search information and multiple video information, each video information including text information and video resources;

[0152] For each video information, based on the feature extraction model, the text features of the text information in the video information and the resource features of the video resources are obtained, and the text features are integrated with the resource features to obtain the video features of the video information;

[0153] Based on the similarity between the search feature and the video feature of each video information, at least one video information with the greatest similarity is determined as the video information associated with the search information.

[0154] The method provided by the embodiment of the present disclosure ensures the accuracy of the extracted video features because the feature extraction model has the ability to recognize text information and video resources in the video information. The video information associated with the search information is retrieved based on the search features of the search information and the video features extracted by the feature extraction model, thereby avoiding the influence of the video information that only contains text information associated with the search information, ensuring that the text information and video resources in the retrieved video information are associated with the search information, thereby ensuring the accuracy of the retrieved video information.

[0155] Based on the above Figure 2 In the embodiment shown, the second video information also includes second text information, and based on the feature extraction model, a third video feature for characterizing the second video information is obtained, and the feature extraction model is trained in combination with the third video feature. The training process is detailed in the following embodiment.

[0156] Figure 3 is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment. Figure 3 The method is performed by an electronic device and comprises the following steps:

[0157] 301. An electronic device obtains sample retrieval information, first video information associated with the sample retrieval information, and second video information not associated with the sample retrieval information.

[0158] Among them, the sample retrieval information is arbitrary information. In some embodiments, the sample retrieval information is text information or an image. For example, the sample retrieval information is a query sentence "What is a skateboard" input by the user, or the sample retrieval information is an image input by the user. The first video information includes first text information and a first video resource. The first text information is used to describe the first video resource. In some embodiments, the first text information includes the video theme or video introduction information of the first video resource. For example, the first video resource is a movie, and the first text information includes the movie name of the movie and a brief introduction to the content of the movie. In an embodiment of the present disclosure, the second video information includes second text information and a second video resource. The second text information is used to describe the second video resource. For example, the second text information includes the video theme or video introduction information of the second video resource.

[0159] In the disclosed embodiment, the first video information is associated with the sample retrieval information, and the second video information is not associated with the sample retrieval information, that is, the first video information is equivalent to the positive sample video information of the sample retrieval information, and the second video information is the negative sample video information of the sample retrieval information. The first text information and the first video resource in the first video information are both associated with the sample retrieval information, and the second video resource and the second text information in the second video information are not associated with the sample retrieval information.

[0160] In some embodiments, the first text information in the first video information is associated with the sample search information, which means that the first text information contains the sample search information, or the first text information contains content similar to the sample search information. For example, the sample search information is "skateboard", and the first text information in the first video information is "what is a skateboard".

[0161] In some embodiments, the first video resource in the first video information is associated with the sample retrieval information, which means that the first video resource contains the sample retrieval information, or the first video resource contains the content indicated by the sample retrieval information. For example, the sample retrieval information is "skateboard", the word "skateboard" is displayed in the first video resource, or the first video resource is a video for introducing "skateboard".

[0162] In some embodiments, the second video resource and the second text information in the second video information are not associated with the sample search information, which means that the content contained in the first text information is not associated with the sample search information, and the content contained in the second video resource is not associated with the sample search information. For example, the sample search information is "skateboard", the second text information is "puppies are so cute", and the second video resource is a video for introducing "puppies", that is, the second text information and the second video resource are not associated with the sample search information.

[0163] 302. The electronic device obtains retrieval features of the sample retrieval information based on a feature extraction model.

[0164] The search feature can be expressed in any form, for example, the search feature is expressed in the form of a feature vector.

[0165] In some embodiments, 302 includes: performing feature extraction on the sample retrieval information based on the feature extraction model to obtain retrieval features of the sample retrieval information.

[0166] In some embodiments, the sample retrieval information includes text information, the feature extraction model includes a first extraction sub-model, and 302 includes: based on the first extraction sub-model in the feature extraction model, performing feature extraction on the text information in the sample retrieval information to obtain retrieval features of the sample retrieval information.

[0167] The first extraction submodel is used to extract features of text information, for example, the first extraction submodel is a text encoder. When the sample retrieval information includes text information, the retrieval features of the sample retrieval information are obtained based on the first extraction submodel to ensure that the obtained retrieval features match the sample retrieval information.

[0168] In some embodiments, the sample retrieval information includes an image, the feature extraction model includes a second extraction sub-model, and 302 includes: based on the second extraction sub-model in the feature extraction model, performing feature extraction on the image in the sample retrieval information to obtain retrieval features of the sample retrieval information.

[0169] The second extraction sub-model is used to obtain features of image or video resources, for example, the second extraction sub-model includes a visual encoder. When the sample retrieval information includes an image, the retrieval features of the sample retrieval information are obtained based on the second extraction sub-model to ensure that the obtained retrieval features match the sample retrieval information.

[0170] It should be noted that the embodiment of the present disclosure obtains the retrieval features of the sample retrieval information based on the feature extraction model, and in another embodiment, other methods can be used to obtain the retrieval features of the sample retrieval information. In some embodiments, the retrieval features of the sample retrieval information are obtained based on other models, and the present disclosure does not limit this.

[0171] 303. The electronic device obtains a first text feature of the first text information in the first video information and a first resource feature of the first video resource based on the feature extraction model, and fuses the first text feature with the first resource feature to obtain a first video feature.

[0172] The first text feature, the first resource feature and the first video feature can all be represented in any form. For example, the first text feature, the first resource feature and the first video feature are all represented in the form of feature vectors.

[0173] In some embodiments, the feature extraction model includes a first extraction sub-model, a second extraction sub-model and a fusion sub-model, and 303 includes: based on the first extraction sub-model, performing feature extraction on the first text information to obtain the first text feature; based on the second extraction sub-model, performing feature extraction on the first video resource to obtain the first resource feature; based on the fusion sub-model, fusing the first text feature with the first resource feature to obtain the first video feature.

[0174] Among them, the fusion sub-model is used to fuse the features of text information and video resources belonging to the same video information.

[0175] Since the first extraction sub-model is used to obtain the features of text information, and the second extraction sub-model is used to obtain the features of image or video resources, the first text features and the first resource features are respectively obtained based on the first extraction sub-model and the second extraction sub-model to ensure the accuracy of the first text features and the first resource features. Afterwards, the first text features are combined with the first resource features based on the fusion sub-model so that the fused first video features can represent the content contained in the first resource information, thereby ensuring the accuracy of the first video features.

[0176] In a possible implementation of the above embodiment, the process of obtaining the first video feature based on the fusion sub-model includes: based on the fusion sub-model, performing weighted fusion on the first text feature and the first resource feature to obtain the first video feature.

[0177] For example, if the fusion sub-model is an MSA (Multi-head Self-Attention) model, the weight of the first text feature and the weight of the first resource feature are obtained based on the fusion sub-model, the first text feature and the weight of the first resource feature, and the first text feature and the first resource feature are weighted fused based on the fusion sub-model, the weight of the first text feature and the weight of the first resource feature to obtain the first video feature.

[0178] For another example, the fusion sub-model is MLP (Multilayer Perceptron). After acquiring the first text feature and the first resource feature, the first text feature and the first resource feature are fused based on the fusion sub-model to obtain the first video feature.

[0179] For another example, the fusion sub-model includes a first weight of a text feature and a second weight of a resource feature, and the weight is set by a developer. Based on the fusion sub-model, the first weight and the second weight, the first text feature and the first resource feature are weightedly fused to obtain the first video feature.

[0180] In some embodiments, the first video resource includes multiple video frames, and the process of obtaining the first resource features of the first video resource based on the feature extraction model includes: determining key frames among the multiple video frames, extracting features from the key frames based on the feature extraction model to obtain frame features of the key frames, and determining the first video features based on the frame features of the key frames.

[0181] The key frame includes the main content described by the first video resource, that is, the key frame is a video frame representing the first video resource among the multiple video frames included in the first video resource. For example, the first video resource is a video introducing "skateboard", the main content described by the first video resource is "skateboard", and the key frame in the first video resource is the video frame containing "skateboard". The frame feature of the key frame is used to characterize the key frame, and the frame feature can be represented in any form, for example, the frame feature is represented in the form of a feature vector.

[0182] Since the first video resource includes multiple video frames, the key frame representing the first video resource in the multiple video frames is determined, and the first video feature is determined based on the frame features of the key frame. In this way, there is no need to obtain the frame features of each video frame in the first video resource, which reduces the amount of calculation, saves resources, and can also ensure the accuracy of the first video feature.

[0183] In a possible implementation of the above embodiment, the process of determining the key frames in the multiple video frames includes the following two methods:

[0184] The first method is to obtain the frame features of each video frame, cluster the frame features of multiple video frames, obtain the cluster center, and determine the video frame whose frame feature is closest to the cluster center from the multiple video frames as the key frame.

[0185] Among them, the frame features of each video frame are used to characterize the corresponding video frame. The cluster center is the center of the frame features of multiple video frames, and the cluster center can reflect the main content described by the multiple video frames, that is, the cluster center can reflect the main content expressed by the first video resource. The distance between the frame feature and the cluster center can reflect the correlation between the corresponding video frame and the main content corresponding to the first video resource. The smaller the distance between the frame feature of the video frame and the cluster center, the more relevant the content contained in the video frame is to the main content, and the larger the distance between the frame feature of the video frame and the cluster center, the less relevant the content contained in the video frame is to the main content. Therefore, the video frame whose frame feature is closest to the cluster center is the video frame most relevant to the main content described by the first video resource among the multiple video frames, that is, the video frame can represent the first video resource, and therefore, the video frame is determined as a key frame.

[0186] In some embodiments, multiple cluster centers are determined by clustering, and key frames are determined based on the multiple cluster centers. That is, the first method includes: clustering frame features of multiple video frames to obtain multiple cluster centers, and for each cluster center, from multiple video frames, determining the video frame whose frame features are closest to the cluster center as the key frame to obtain multiple key frames.

[0187] When clustering the frame features of multiple video frames, multiple clusters can be obtained, each cluster includes frame features of multiple video frames, each cluster has a cluster center, and a frame feature that is closest to the cluster center is selected from the multiple frame features included in each cluster, and the video frame corresponding to the frame feature is determined as the first key frame, so that multiple key frames can be obtained.

[0188] Since the contents described by multiple video frames contained in a video resource may be different, clustering is performed using multiple clustering centers to obtain multiple key frames to ensure that the multiple key frames can represent the meaning expressed by the video resource, thereby ensuring the accuracy of the multiple key frames determined, so that the video features of the first video information can be subsequently obtained based on the multiple key frames, thereby ensuring the accuracy of the video features.

[0189] In some embodiments, multiple key frames are determined using a cluster center, that is, in the first method, the process of determining key frames from multiple video frames includes: from the multiple video frames, multiple video frames whose frame features are closest to the cluster center are determined as key frames.

[0190] Since different video frames contain different contents, multiple key frames that can represent the first video resource are selected from multiple video frames to ensure the accuracy of video features subsequently determined based on the multiple key frames.

[0191] The second method: determining the cover video frame of the first video resource as the key frame, or determining the first video frame of the first video resource as the key frame.

[0192] In the embodiment of the present disclosure, the cover video frame of the first video resource, or the first video frame of the first video resource contains the main content described by the first video resource, and therefore, the cover video frame or the first video frame of the first video resource is determined as a key frame.

[0193] In a possible implementation of the above embodiment, the feature extraction model includes a second extraction sub-model, and the process of obtaining the frame features of the key frame based on the feature extraction model includes: based on the second extraction sub-model, performing feature extraction on the key frame to obtain the frame features of the key frame.

[0194] When multiple key frames are determined, feature extraction is performed on each key frame based on the second extraction sub-model to obtain frame features of each key frame.

[0195] In a possible implementation of the above embodiment, the first video feature is determined based on the frame features of the key frame, including: when one key frame is determined, the frame features of the key frame are determined as the first video feature; when multiple key frames are determined, the frame features of the multiple key frames are fused to obtain a fused feature, and the fused feature is determined as the first video feature.

[0196] Since the first video resource may contain one or more key frames, the first video feature is obtained in different ways based on different situations in which the key frames are determined to be one or more, so as to ensure the accuracy of the obtained video feature.

[0197] 304. The electronic device obtains the second text feature of the second text information in the second video information and the second resource feature of the second video resource based on the feature extraction model, fuses the second text feature with the second resource feature to obtain the third video feature of the second video information, and fuses the first text feature with the second resource feature to obtain the second video feature.

[0198] The third video feature is the video feature of the second video information. In the disclosed embodiment, the second video information is equivalent to the negative sample video information of the sample retrieval information, that is, the third video feature of the second video information is equivalent to the negative sample video feature corresponding to the sample retrieval information.

[0199] Since the text information included in the video information may not match the video resource, for example, the text information in the video information is "The puppy is so cute", and the video resource in the video information is a video introducing "skateboard", that is, the text information in the video information does not match the video resource. When searching with the retrieval information, this video information may be retrieved, resulting in the text information in the retrieved video information being associated with the retrieval information, but the video resource in the video information is not associated with the retrieval information, resulting in inaccurate retrieved video information. Therefore, when training the feature extraction model, a second video feature is generated, and the second video feature is used to represent the video information that contains text information that is associated with the sample retrieval information, and the video resource that is not associated with the sample retrieval information, so as to ensure that the subsequent feature extraction model avoids the influence of this video information when performing feature extraction, so as to improve the accuracy of the feature extraction model.

[0200] In the disclosed embodiments, after feature extraction is performed on text information or video resources based on a feature extraction model, the extracted features can be cached so that the cached features can be directly called later without further feature extraction. That is, after feature extraction is performed on the first text information based on the feature extraction model and the first text feature is obtained, the first text feature is cached, and after the second text feature of the second text information in the second video information and the second resource feature of the second video resource are obtained based on the feature extraction model, the second text information and the second resource feature are fused to obtain the third video feature, and the obtained second resource feature is cached so that the cached first text feature can be fused with the second resource feature later to obtain the second video feature.

[0201] In the disclosed embodiment, the process of obtaining the third video feature based on the feature extraction model is the same as the process of obtaining the first video feature based on the feature extraction model, and will not be described in detail here. In addition, the process of obtaining the second video feature based on the first text information and the second resource feature by adopting a fusion method is the same as the process of obtaining the first video feature based on the first text information and the first resource feature by adopting a fusion method, and will not be described in detail here.

[0202] 305. The electronic device trains a feature extraction model based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature, respectively.

[0203] In the disclosed embodiment, the retrieval feature is a feature corresponding to the sample retrieval information, the first video feature is a positive sample feature associated with the sample retrieval information, the second video feature and the third video feature are negative sample features not associated with the retrieval feature, and the second video feature is a negative sample feature generated based on the first video information and the second video information, thereby enriching the features of the trained feature extraction model, taking into account the influence of various negative sample features on the feature extraction model, and training the feature extraction model based on the matching relationship between the first video feature, the second video feature and the third video feature and the retrieval feature, so as to improve the feature extraction model's recognition ability for positive sample video information and negative sample video information, thereby improving the accuracy of the feature extraction model.

[0204] In some embodiments, step 305 includes: determining a first similarity between the retrieval feature and the first video feature, a second similarity between the retrieval feature and the second video feature, and a third similarity between the retrieval feature and the third video feature; and training a feature extraction model based on the first similarity, the second similarity, and the third similarity.

[0205] Among them, the first similarity represents the similarity between the retrieval feature extracted by the feature extraction model and the first video feature. The first similarity can also reflect the association between the sample retrieval information and the first video information. The first similarity is determined based on the extracted retrieval feature and the first video feature determined by the feature extraction model. Since the first video information is associated with the sample retrieval information, the inaccuracy of the feature extraction model can be reflected based on the first similarity. Similarly, the second similarity represents the similarity between the retrieval feature and the second video feature, and the third similarity represents the similarity between the retrieval feature and the third video feature. Since the video information corresponding to the second video feature and the third video feature is not associated with the sample retrieval information, the second similarity and the third similarity can also reflect the inaccuracy of the feature extraction model.

[0206] Since the first video feature is equivalent to the positive sample video feature of the sample retrieval information, and the second video feature and the third video feature are equivalent to the negative sample video feature of the sample retrieval information, the feature extraction model is trained based on the first similarity, the second similarity and the third similarity to increase the first similarity and reduce the second similarity and the third similarity, thereby improving the accuracy of the feature extraction model.

[0207] In a possible implementation of the above embodiment, the first similarity, the second similarity and the third similarity can all be represented by feature distances, for example, determined by using Euclidean distances, etc. Based on the first similarity, the second similarity and the third similarity, the feature extraction model is trained to reduce the distance between the retrieval feature and the first video feature, and to increase the distance between the retrieval feature and the second video feature and the third video feature, so as to improve the accuracy of the feature extraction model.

[0208] In a possible implementation of the above embodiment, the process of training the feature extraction model based on the first similarity, the second similarity and the third similarity includes: determining the sum of the second similarity and the third similarity, determining the loss value based on the ratio between the sum and the first similarity, and training the feature extraction model based on the loss value.

[0209] Wherein, when obtaining the loss value, it can be obtained based on InfoNCE (a loss function). Based on the loss value, the feature extraction model is trained to reduce the ratio, that is, increase the first similarity and reduce the second similarity and the third similarity, thereby improving the accuracy of the feature extraction model.

[0210] In a possible implementation of the above embodiment, the process of training the feature extraction model based on the first similarity, the second similarity and the third similarity includes: determining the sum of the second similarity and the third similarity, determining the loss value based on the difference between the sum and the first similarity, and training the feature extraction model based on the loss value.

[0211] The feature extraction model is trained based on the loss value to reduce the difference, that is, to increase the first similarity and reduce the second similarity and the third similarity, thereby improving the accuracy of the feature extraction model.

[0212] It should be noted that the embodiment of the present disclosure takes the second video information including the second text information and the second video resource as an example, and trains the feature extraction model based on the acquired first video features, second video features and third video features. In another embodiment, the second video information only includes the second video resource, and there is no need to execute steps 304-305. Other methods can be adopted to obtain the second resource features of the second video resource in the second video information based on the feature extraction model, and fuse the first text features with the second resource features to obtain the second video features; and train the feature extraction model based on the retrieval features, the first video features and the second video features.

[0213] In some embodiments, the process of training a feature extraction model based on a retrieval feature, a first video feature, and a second video feature includes: determining a first similarity between the retrieval feature and the first video feature, and a second similarity between the retrieval feature and the second video feature; and training the feature extraction model based on the first similarity and the second similarity.

[0214] Since the first video feature is equivalent to the positive sample video feature of the sample retrieval information, and the second video feature is equivalent to the negative sample video feature of the sample retrieval information, the feature extraction model is trained based on the first similarity and the second similarity to increase the first similarity and decrease the second similarity, thereby improving the accuracy of the feature extraction model.

[0215] In a possible implementation of the above embodiment, the process of training a feature extraction model based on the first similarity and the second similarity includes: determining a loss value based on the ratio between the second similarity and the first similarity, and training the feature extraction model based on the loss value; or determining a loss value based on the difference between the second similarity and the first similarity, and training the feature extraction model based on the loss value.

[0216] The process of determining the loss value based on the first similarity and the second similarity is the same as the process of determining the loss value based on the first similarity, the second similarity and the third similarity, which will not be described again.

[0217] It should be noted that the embodiment of the present disclosure is described by an iterative process of training the feature extraction model, while in another embodiment, the feature extraction model is iterated multiple times according to the above steps 301-305 to improve the accuracy of the feature extraction model. When the iterative rounds of training the feature extraction model reach the round threshold, the training of the feature extraction model is stopped; or, in response to the loss value determined based on the first similarity, the second similarity and the third similarity being less than the loss threshold, the training of the feature extraction model is stopped. The round threshold is an arbitrary value, for example, the target round is 10. The loss threshold is an arbitrary value, for example, the loss threshold is 0.2.

[0218] The method provided by the embodiment of the present disclosure, since the first text information in the first video information is associated with the sample retrieval information, and the second video resource in the second video information is not associated with the sample retrieval information, the first text feature of the first text information is merged with the second resource feature of the second video resource to obtain the second video feature, the first video feature of the first video information is used as the positive sample feature, and the second video feature is used as the negative sample feature, and a feature extraction model is trained based on the retrieval feature, the positive sample feature and the negative sample feature, so as to enrich the features of the trained feature extraction model, take into account the influence of video information including text information associated with the retrieval information and including video resources not associated with the retrieval information on the feature extraction model, so as to enhance the feature extraction model's recognition ability of the text information and video resources contained in the video information, thereby improving the accuracy of the feature extraction model.

[0219] In addition, the feature extraction model is trained with the first video feature, the second video feature and the third video feature, taking into account multiple modalities of the video information, that is, the text information in the video information matches the video resources, or the text information in the video information does not match the video resources, so as to break the modal imbalance and enhance the feature extraction model's ability to recognize the text information and video resources contained in the video information, thereby improving the accuracy of the feature extraction model.

[0220] Furthermore, based on the retrieval feature, the first video feature, the second video feature and the third video feature, a first similarity, a second similarity and a third similarity are obtained. Since the similarity can reflect the inaccuracy of the feature extraction model, the feature extraction model is trained with the obtained similarity to improve the accuracy of the feature extraction model.

[0221] Based on the above Figure 3 The embodiment shown provides a process for training a feature extraction model, such as Figure 4 As shown, the feature extraction model includes a first extraction sub-model, a second extraction sub-model and a fusion sub-model, the first video information 401 includes a first video resource and a first text information, and the second video information 402 includes a second video resource and a second text information. The process includes:

[0222] Based on the first extraction sub-model, feature extraction is performed on the search information to obtain search features.

[0223] The process of obtaining the first video feature of the first video information based on the feature extraction model includes: obtaining the first text feature of the first text information in the first video information based on the first extraction sub-model in the feature extraction model, and caching the first text feature; obtaining the first resource feature of the first video resource in the first video information based on the second extraction sub-model in the feature extraction model, and fusing the first text feature and the first resource feature based on the fusion sub-model in the feature extraction model to obtain the first video feature.

[0224] The process of obtaining the third video feature of the second video information based on the feature extraction model includes: obtaining the second text feature of the second text information in the second video information based on the first extraction sub-model in the feature extraction model; obtaining the second resource feature of the second video resource in the second video information based on the second extraction sub-model in the feature extraction model, caching the second resource feature, and fusing the second text feature and the second resource feature based on the fusion sub-model in the feature extraction model to obtain the third video feature.

[0225] The process of acquiring the second video feature based on the feature extraction model includes: based on the fusion sub-model in the feature extraction model, fusing the cached first text feature and the second resource feature to obtain the second video feature.

[0226] Based on the retrieval feature, the first video feature, the second video feature and the third video feature, the feature extraction model is trained to shorten the distance between the retrieval feature and the first video feature, increase the distance between the retrieval feature and the second video feature, and increase the distance between the retrieval feature and the third video feature, thereby improving the accuracy of the feature extraction model.

[0227] Based on the above Figure 2 or Figure 3 In the embodiment shown, after the feature extraction model is trained, information retrieval can be implemented based on the trained feature extraction model. The information retrieval process is detailed in the following embodiment.

[0228] Figure 5 is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment. Figure 5 The method is performed by an electronic device and comprises the following steps:

[0229] 501. The electronic device obtains a search feature of any search information and a plurality of video information, where each video information includes text information and video resources.

[0230] The search information is the information to be searched, and the video information associated with the search information needs to be searched later. The search feature is used to characterize the search information. The multiple video information is candidate video information, and the video information associated with the search information is searched later from the multiple video information. Each video information includes text information and video resources, and the text information is used to describe the video resources belonging to the same video information.

[0231] 502. The electronic device obtains, for each video information, text features of text information in the video information and resource features of video resources based on a feature extraction model, and fuses the text features with the resource features to obtain video features of the video information.

[0232] The feature extraction model is trained based on the matching relationship between the first video feature and the second video feature and the retrieval feature of the sample retrieval information, the first video information corresponding to the first video feature is associated with the sample retrieval information and includes the first text information and the first video resource, the video information corresponding to the second video feature includes text information associated with the sample retrieval information, but the included video resource is not associated with the sample retrieval information. The first video feature is equivalent to the positive sample video feature corresponding to the sample retrieval information, and the second video feature is equivalent to the negative sample video feature corresponding to the sample retrieval information. Based on the matching relationship between the positive sample video feature and the negative sample video feature and the retrieval feature of the sample retrieval information, the feature extraction model is trained to improve the feature extraction model's ability to recognize the text information and video resources contained in the video information, thereby ensuring the accuracy of the feature extraction model.

[0233] The process of obtaining the video features of the video information is the same as that of the above 303, and will not be repeated here.

[0234] 503. The electronic device determines at least one video information with the greatest similarity as the video information associated with the search information based on the similarity between the search feature and the video feature of each video information.

[0235] Among them, for any video information, the similarity between the video features of the video information and the search features can reflect the degree of association between the video information and the search information. The greater the similarity, the more associated the video information is with the search information. Therefore, from multiple video information, at least one video information with the greatest similarity is selected, and the selected video information is associated with the search information, thereby ensuring the accuracy of the determined video information.

[0236] The method provided by the embodiment of the present disclosure improves the recognition capability of the feature extraction model for text information and video resources contained in the video information, because the feature extraction model is trained based on the matching relationship between the video features of the video information associated with the sample retrieval information and the video features of the video information not associated with the sample retrieval information and the retrieval features of the sample retrieval information. Therefore, the video features of the video information are obtained based on the feature extraction model, thereby ensuring the accuracy of the obtained video features, and the video information associated with the retrieval information is retrieved based on the video features and the retrieval features, thereby avoiding the influence of the video information associated with the retrieval information only by the text information contained, ensuring that the text information and video resources in the retrieved video information are associated with the retrieval information, thereby ensuring the accuracy of the retrieved video information.

[0237] Based on the above Figure 2 or Figure 3 In the embodiment shown, after the feature extraction model is trained, information retrieval can be implemented based on the trained feature extraction model. The information retrieval process is detailed in the following embodiment.

[0238] Figure 6 is a flowchart of a processing method based on a feature extraction model according to an exemplary embodiment. Figure 6 The method is performed by an electronic device and comprises the following steps:

[0239] 601. An electronic device obtains any search information and multiple video information, where each video information includes text information and video resources.

[0240] In some embodiments, the electronic device is provided as a server, and 601 includes: the server receives a retrieval request sent by a terminal, the retrieval request carries the retrieval information, and obtains the multiple video information.

[0241] In the disclosed embodiment, the server provides a retrieval service, and the terminal can interact with the server and send a retrieval request to the server, and the server retrieves retrieval information associated with the retrieval information for the terminal. Therefore, after receiving the retrieval request sent by the terminal, multiple alternative video information is obtained for subsequent retrieval.

[0242] In a possible implementation of the above embodiment, the server stores an information library for storing alternative video information, the information library includes the multiple video information, and the process of obtaining the multiple video information includes: the server obtains the multiple video information from the information library in response to receiving a retrieval request.

[0243] 602. The electronic device obtains retrieval features of retrieval information based on a feature extraction model.

[0244] The process of obtaining the search features is the same as that of step 302 above, and will not be described again here.

[0245] It should be noted that the embodiment of the present disclosure obtains the retrieval features of the retrieval information based on the feature extraction model, and in another embodiment, there is no need to perform step 602, and other methods can be adopted to obtain the retrieval features of the retrieval information. In some embodiments, the retrieval features of the sample retrieval information are obtained based on other models, and the present disclosure does not limit this.

[0246] 603. The electronic device obtains, for each piece of video information, text features of the text information in the video information and resource features of the video resources, and fuses the text features with the resource features to obtain video features of the video information.

[0247] The process of obtaining the video features of the video information is the same as that of the above 303, and will not be repeated here.

[0248] 604. The electronic device determines at least one video information with the greatest similarity as the video information associated with the search information based on the search feature and the similarity between the video features of each video information.

[0249] In some embodiments, the electronic device is provided as a server, and the retrieval information is carried by a retrieval request sent by the terminal. Then, after 309, the method further includes: the server sends video information associated with the retrieval information to the terminal, and the terminal receives the video information and displays the video information.

[0250] For example, the user inputs search information, such as "skateboard", through the terminal, and the terminal sends a search request to the server, which carries the search information. After receiving the search request, the server retrieves the video information associated with the search information according to 601-604 above, and sends the retrieved video information to the terminal. After the terminal receives the video information sent by the server, it displays the video information. The user can view the video information displayed on the terminal through the terminal, thereby realizing information retrieval.

[0251] The method provided by the embodiment of the present disclosure ensures the accuracy of the extracted video features because the feature extraction model has the ability to recognize text information and video resources in the video information. Video information associated with the search information is retrieved based on the video features and search features extracted by the feature extraction model, thereby avoiding the influence of video information that only contains text information associated with the search information, ensuring that the text information and video resources in the retrieved video information are associated with the search information, thereby ensuring the accuracy of the retrieved video information.

[0252] The method provided in the embodiment of the present disclosure is to perform information retrieval based on the trained feature extraction model. Taking "tiger" and "crayfish" as the retrieval information, the retrieval results obtained based on the method provided in the embodiment of the present disclosure and the information retrieval method of the related technology are as follows: Figure 7 shown. Figure 7 The numerical value on each image in represents the similarity between the image and the corresponding search information. By comparison, it can be seen that the similarity obtained by the information retrieval method provided by the embodiment of the present disclosure is more accurate. In addition, based on multiple video information, a PNR (positive negative ratio) test is performed on the method provided by the present disclosure, and based on multiple video information, a PNR test is performed on the method provided by the related art. The PNR of the present disclosure is 2.387, and the PNR of the related art is 2.258. By comparing the PNR, it can be seen that the accuracy of the video information retrieved by the method provided by the embodiment of the present disclosure is higher.

[0253] Figure 8 is a block diagram of a processing device based on a feature extraction model according to an exemplary embodiment. Figure 8 , the device comprises:

[0254] An acquisition unit 801 is configured to execute acquisition of a search feature of sample search information, first video information associated with the sample search information, and second video information not associated with the sample search information, wherein the first video information includes first text information and a first video resource, the first text information is used to describe the first video resource, and the second video information includes a second video resource;

[0255] The acquisition unit 801 is further configured to execute based on the feature extraction model, acquire the first text feature of the first text information in the first video information and the first resource feature of the first video resource, and fuse the first text feature with the first resource feature to obtain the first video feature;

[0256] The acquisition unit 801 is further configured to execute based on the feature extraction model, acquire the second resource feature of the second video resource in the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature;

[0257] The training unit 802 is configured to perform training on a feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature respectively, and the feature extraction model is used to obtain the features of any video information.

[0258] In some embodiments, the training unit 802 is configured to determine a first similarity between the retrieval feature and the first video feature, and a second similarity between the retrieval feature and the second video feature; and train the feature extraction model based on the first similarity and the second similarity.

[0259] In some embodiments, the second video information further includes second text information, and the second text information is used to describe the second video resource. The acquisition unit 801 is configured to execute based on the feature extraction model, obtain the second text feature of the second text information in the second video information and the second resource feature of the second video resource, fuse the second text feature with the second resource feature to obtain the third video feature of the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature;

[0260] The training unit 802 is configured to perform training on the feature extraction model based on the matching relationships between the first video feature, the second video feature and the third video feature and the retrieval feature respectively.

[0261] In some embodiments, the training unit 802 is configured to determine a first similarity between the retrieval feature and the first video feature, a second similarity between the retrieval feature and the second video feature, and a third similarity between the retrieval feature and the third video feature; and train the feature extraction model based on the first similarity, the second similarity and the third similarity.

[0262] In some embodiments, the first video resource includes a plurality of video frames, see Fig. 9 , the acquisition unit 801 includes:

[0263] The determining subunit 8011 is configured to determine a key frame among the plurality of video frames, where the key frame contains a main content described by the first video resource;

[0264] The determination subunit 8011 is further configured to perform feature extraction on the key frame based on the feature extraction model to obtain the frame features of the key frame, and determine the first video feature based on the frame features of the key frame.

[0265] In some embodiments, the determination subunit 8011 is configured to, when one key frame is determined, determine the frame feature of the key frame as the first video feature; when multiple key frames are determined, fuse the frame features of multiple key frames to obtain a fused feature, and determine the fused feature as the first video feature.

[0266] In some embodiments, the determination subunit 8011 is configured to execute obtaining frame features of each video frame; clustering the frame features of multiple video frames to obtain a cluster center; and determining, from multiple video frames, a video frame whose frame features are closest to the cluster center as a key frame.

[0267] In some embodiments, see Fig. 9 The device also includes:

[0268] The acquisition unit 801 is further configured to execute acquisition of a search feature of any search information and a plurality of video information, each video information including text information and video resources;

[0269] The acquisition unit 801 is further configured to execute, for each video information, based on the feature extraction model, acquiring text features of the text information in the video information and resource features of the video resources, and fusing the text features with the resource features to obtain video features of the video information;

[0270] The determination unit 803 is further configured to perform a similarity between the search feature and the video feature of each video information, and determine at least one video information with the greatest similarity as the video information associated with the search information.

[0271] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0272] Fig.10 is a block diagram of a processing device based on a feature extraction model according to an exemplary embodiment. Fig.10 , the device comprises:

[0273] An acquisition unit 1001 acquires a search feature configured to acquire any search information and a plurality of video information, each video information including text information and video resources;

[0274] The acquisition unit 1001 is further configured to execute, for each video information, based on the feature extraction model, acquiring text features of the text information in the video information and resource features of the video resources, and fusing the text features with the resource features to obtain video features of the video information;

[0275] The determination unit 1002 is further configured to perform, based on the similarity between the search feature and the video feature of each video information, determining at least one video information with the greatest similarity as the video information associated with the search information;

[0276] Among them, the feature extraction model is trained based on the matching relationship between the first video feature and the second video feature and the retrieval features of the sample retrieval information respectively. The first video information corresponding to the first video feature is associated with the sample retrieval information and includes the first text information and the first video resource. The text information included in the video information corresponding to the second video feature is associated with the sample retrieval information, but the included video resources are not associated with the sample retrieval information.

[0277] In some embodiments, Fig.11 As shown, the device also includes:

[0278] The acquisition unit 1001 is further configured to execute the acquisition of the retrieval feature of the sample retrieval information, the first video information associated with the sample retrieval information, and the second video information not associated with the sample retrieval information, the first text information is used to describe the first video resource, and the second video information includes the second video resource;

[0279] The acquisition unit 1001 is further configured to execute based on the feature extraction model, acquire the first text feature of the first text information in the first video information and the first resource feature of the first video resource, and fuse the first text feature with the first resource feature to obtain the first video feature;

[0280] The acquisition unit 1001 is further configured to execute based on the feature extraction model, acquire the second resource feature of the second video resource in the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature;

[0281] The training unit 1003 is configured to perform training on the feature extraction model based on the matching relationship between the first video feature and the second video feature and the retrieval feature respectively.

[0282] In some embodiments, the second video information further includes second text information, and the second text information is used to describe the second video resource. The acquisition unit 1001 is configured to execute based on the feature extraction model, obtain the second text feature of the second text information in the second video information and the second resource feature of the second video resource, fuse the second text feature with the second resource feature to obtain the third video feature of the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature;

[0283] The training unit 1003 is configured to perform training on the feature extraction model based on the matching relationships between the first video feature, the second video feature and the third video feature and the retrieval feature respectively.

[0284] In some embodiments, the first video resource includes multiple video frames, and the acquisition unit 1001 is configured to determine the key frames in the multiple video frames, where the key frames contain the main content described by the first video resource; based on the feature extraction model, feature extraction is performed on the key frames to obtain frame features of the key frames, and based on the frame features of the key frames, the first video features are determined.

[0285] In an exemplary embodiment, an electronic device is also provided, the electronic device comprising:

[0286] one or more processors;

[0287] volatile or non-volatile memory for storing one or more processor executable instructions;

[0288] Among them, one or more processors are configured to execute the steps performed by the electronic device in the above-mentioned processing method based on the feature extraction model.

[0289] In some embodiments, the electronic device is provided as a terminal. Fig.12 1 is a block diagram of a terminal 1200 according to an exemplary embodiment. The terminal 1200 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 1200 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0290] The terminal 1200 includes: a processor 1201 and a memory 1202 .

[0291] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0292] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one program code, which is used to be executed by the processor 1201 to implement the processing method based on the feature extraction model provided in the method embodiment of the present disclosure.

[0293] In some embodiments, the terminal 1200 may also optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, a positioning assembly 1208 and a power supply 1209.

[0294] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0295] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0296] The display screen 1205 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. The touch signal can be input to the processor 1201 as a control signal for processing. At this time, the display screen 1205 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1205 can be one, set on the front panel of the terminal 1200; in other embodiments, the display screen 1205 can be at least two, respectively set on different surfaces of the terminal 1200 or in a folding design; in other embodiments, the display screen 1205 can be a flexible display screen, set on a curved surface or a folding surface of the terminal 1200. Even, the display screen 1205 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0297] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0298] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1201 for processing, or input them into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1200. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0299] The positioning component 1208 is used to locate the current geographical location of the terminal 1200 to implement navigation or LBS (Location Based Service). The positioning component 1208 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or Russia's Galileo system.

[0300] The power supply 1209 is used to power various components in the terminal 1200. The power supply 1209 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1209 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0301] In some embodiments, the terminal 1200 further includes one or more sensors 1210 , including but not limited to: an acceleration sensor 1211 , a gyroscope sensor 1212 , a pressure sensor 1213 , a fingerprint sensor 1214 , an optical sensor 1215 , and a proximity sensor 1216 .

[0302] The acceleration sensor 1211 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1200. For example, the acceleration sensor 1211 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1201 can control the display screen 1205 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1211. The acceleration sensor 1211 can also be used to collect motion data of games or users.

[0303] The gyro sensor 1212 can detect the body direction and rotation angle of the terminal 1200, and the gyro sensor 1212 can cooperate with the acceleration sensor 1211 to collect the user's 3D actions on the terminal 1200. The processor 1201 can implement the following functions based on the data collected by the gyro sensor 1212: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0304] The pressure sensor 1213 can be set on the side frame of the terminal 1200 and / or the lower layer of the display screen 1205. When the pressure sensor 1213 is set on the side frame of the terminal 1200, the user's holding signal of the terminal 1200 can be detected, and the processor 1201 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1213. When the pressure sensor 1213 is set on the lower layer of the display screen 1205, the processor 1201 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1205. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0305] The fingerprint sensor 1214 is used to collect the user's fingerprint, and the processor 1201 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1214, or the fingerprint sensor 1214 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1201 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1214 can be set on the front, back, or side of the terminal 1200. When a physical button or a manufacturer logo is set on the terminal 1200, the fingerprint sensor 1214 can be integrated with the physical button or the manufacturer logo.

[0306] The optical sensor 1215 is used to collect the ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the display screen 1205 according to the ambient light intensity collected by the optical sensor 1215. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the display screen 1205 is reduced. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameters of the camera assembly 1206 according to the ambient light intensity collected by the optical sensor 1215.

[0307] The proximity sensor 1216, also called a distance sensor, is disposed on the front panel of the terminal 1200. The proximity sensor 1216 is used to collect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from the screen-on state to the screen-off state; when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from the screen-off state to the screen-on state.

[0308] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the terminal 1200, and the terminal 1200 may include more or fewer components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0309] In some embodiments, the electronic device is provided as a terminal. Fig.13 This is a schematic diagram of the structure of a server according to an exemplary embodiment. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the memory 1302 stores at least one program code, and the at least one program code is loaded and executed by the processor 1301 to implement the methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0310] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, and when the program code in the storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the steps performed by the electronic device in the above-mentioned processing method based on the feature extraction model. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0311] In an exemplary embodiment, a non-temporary computer-readable storage medium is also provided. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps performed by the electronic device in the above-mentioned processing method based on the feature extraction model.

[0312] In an exemplary embodiment, a computer program product is also provided. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the steps executed by the terminal or server in the above-mentioned processing method based on the feature extraction model.

[0313] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0314] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A processing method based on a feature extraction model, characterized in that: The method comprises: Acquire a search feature of sample search information, first video information associated with the sample search information, and second video information not associated with the sample search information, wherein the first video information includes first text information and a first video resource, the first text information is used to describe the first video resource, and the second video information includes a second video resource; Based on the feature extraction model, obtaining a first text feature of the first text information in the first video information and a first resource feature of the first video resource, and fusing the first text feature with the first resource feature to obtain a first video feature; Based on the feature extraction model, obtaining a second resource feature of the second video resource in the second video information, and fusing the first text feature with the second resource feature to obtain a second video feature; Based on the matching relationship between the first video feature and the second video feature and the retrieval feature respectively, the feature extraction model is trained, and the feature extraction model is used to obtain the features of any video information.

2. The method according to claim 1, characterized in that The training of the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature respectively includes: Determining a first similarity between the retrieval feature and the first video feature, and a second similarity between the retrieval feature and the second video feature; The feature extraction model is trained based on the first similarity and the second similarity.

3. The method according to claim 1, characterized in that The second video information further includes second text information, and the second text information is used to describe the second video resource. The second resource feature of the second video resource in the second video information is obtained based on the feature extraction model, and the first text feature is merged with the second resource feature to obtain the second video feature, including: Based on the feature extraction model, obtaining a second text feature of the second text information in the second video information and a second resource feature of the second video resource, fusing the second text feature with the second resource feature to obtain a third video feature of the second video information, and fusing the first text feature with the second resource feature to obtain the second video feature; The training of the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature respectively includes: The feature extraction model is trained based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

4. The method according to claim 3, characterized in that The training of the feature extraction model based on the matching relationships between the first video feature, the second video feature, and the third video feature and the search feature respectively includes: Determining a first similarity between the retrieval feature and the first video feature, a second similarity between the retrieval feature and the second video feature, and a third similarity between the retrieval feature and the third video feature; The feature extraction model is trained based on the first similarity, the second similarity, and the third similarity.

5. The method according to claim 1, characterized in that The first video resource includes a plurality of video frames, and the process of obtaining a first resource feature of the first video resource based on a feature extraction model includes: Determine a key frame among the plurality of video frames, wherein the key frame contains main content described by the first video resource; Based on the feature extraction model, feature extraction is performed on the key frame to obtain frame features of the key frame, and based on the frame features of the key frame, the first resource feature is determined.

6. The method according to claim 5, characterized in that The determining the first resource feature based on the frame feature of the key frame includes: When one of the key frames is determined, determining a frame feature of the key frame as the first resource feature; When a plurality of key frames are determined, frame features of the plurality of key frames are fused to obtain a fused feature, and the fused feature is determined as the first resource feature.

7. The method according to claim 5, characterized in that The determining of the key frames in the plurality of video frames comprises: Obtaining frame features of each of the video frames; Clustering frame features of the plurality of video frames to obtain cluster centers; From the multiple video frames, the video frame whose frame feature is closest to the cluster center is determined as the key frame.

8. The method according to any one of claims 1 to 7, characterized in that: After training the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature, the method further includes: Obtaining a search feature of any search information and a plurality of video information, each of the video information including text information and video resources; For each of the video information, based on the feature extraction model, obtain text features of the text information in the video information and resource features of the video resources, and fuse the text features with the resource features to obtain video features of the video information; Based on the similarity between the search feature and the video feature of each of the video information, at least one video information with the greatest similarity is determined as the video information associated with the search information.

9. A processing method based on a feature extraction model, characterized in that: The method comprises: Obtaining a search feature of any search information and a plurality of video information, each of the video information including text information and video resources; For each of the video information, based on the feature extraction model, obtain text features of the text information in the video information and resource features of the video resources, and fuse the text features with the resource features to obtain video features of the video information; Based on the similarity between the search feature and the video feature of each of the video information, determining at least one video information with the greatest similarity as the video information associated with the search information; Among them, the feature extraction model is trained based on the matching relationship between the first video feature and the second video feature and the retrieval features of the sample retrieval information respectively, the first video information corresponding to the first video feature is associated with the sample retrieval information and includes first text information and a first video resource, the video information corresponding to the second video feature includes text information associated with the sample retrieval information, but the included video resources are not associated with the sample retrieval information.

10. The method according to claim 9, characterized in that The method further comprises: Acquire the search feature of the sample search information, the first video information associated with the sample search information, and the second video information not associated with the sample search information, wherein the first text information is used to describe the first video resource, and the second video information includes a second video resource; Based on the feature extraction model, obtaining a first text feature of the first text information in the first video information and a first resource feature of the first video resource, and fusing the first text feature with the first resource feature to obtain the first video feature; Based on the feature extraction model, obtaining a second resource feature of the second video resource in the second video information, and fusing the first text feature with the second resource feature to obtain the second video feature; The feature extraction model is trained based on the matching relationships between the first video feature and the second video feature and the retrieval feature respectively.

11. The method according to claim 10, characterized in that The second video information further includes second text information, and the second text information is used to describe the second video resource. The method of obtaining a second resource feature of the second video resource in the second video information based on the feature extraction model and fusing the first text feature with the second resource feature to obtain the second video feature includes: Based on the feature extraction model, obtaining a second text feature of the second text information in the second video information and a second resource feature of the second video resource, fusing the second text feature with the second resource feature to obtain a third video feature of the second video information, and fusing the first text feature with the second resource feature to obtain the second video feature; The training of the feature extraction model based on the matching relationship between the first video feature and the second video feature and the search feature respectively includes: The feature extraction model is trained based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

12. The method according to claim 10, characterized in that The first video resource includes a plurality of video frames, and the process of acquiring the first resource feature of the first video resource based on the feature extraction model includes: Determine a key frame among the plurality of video frames, wherein the key frame contains main content described by the first video resource; Based on the feature extraction model, feature extraction is performed on the key frame to obtain frame features of the key frame, and based on the frame features of the key frame, the first resource feature is determined.

13. A processing device based on a feature extraction model, characterized in that: The device comprises: An acquiring unit is configured to acquire a retrieval feature of sample retrieval information, first video information associated with the sample retrieval information, and second video information not associated with the sample retrieval information, wherein the first video information includes first text information and a first video resource, the first text information is used to describe the first video resource, and the second video information includes a second video resource; The acquisition unit is further configured to execute based on the feature extraction model, acquire the first text feature of the first text information in the first video information and the first resource feature of the first video resource, and fuse the first text feature with the first resource feature to obtain the first video feature; The acquisition unit is further configured to acquire a second resource feature of the second video resource in the second video information based on the feature extraction model, and fuse the first text feature with the second resource feature to obtain a second video feature; The training unit is configured to perform training on the feature extraction model based on the matching relationship between the first video feature and the second video feature and the retrieval feature respectively, and the feature extraction model is used to obtain the features of any video information.

14. The device according to claim 13, characterized in that The training unit is configured to determine a first similarity between the retrieval feature and the first video feature, and a second similarity between the retrieval feature and the second video feature; and train the feature extraction model based on the first similarity and the second similarity.

15. The device according to claim 13, characterized in that The second video information further includes second text information, and the second text information is used to describe the second video resource. The acquisition unit is configured to execute based on the feature extraction model, acquire the second text feature of the second text information in the second video information and the second resource feature of the second video resource, fuse the second text feature with the second resource feature to obtain the third video feature of the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature; The training unit is configured to perform training on the feature extraction model based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

16. The device according to claim 15, characterized in that The training unit is configured to determine a first similarity between the retrieval feature and the first video feature, a second similarity between the retrieval feature and the second video feature, and a third similarity between the retrieval feature and the third video feature; and train the feature extraction model based on the first similarity, the second similarity and the third similarity.

17. The device according to claim 13, characterized in that The first video resource includes a plurality of video frames, and the acquisition unit includes: A determination subunit is configured to determine a key frame among the plurality of video frames, wherein the key frame contains a main content described by the first video resource; The determination subunit is further configured to perform feature extraction on the key frame based on the feature extraction model to obtain frame features of the key frame, and determine the first resource feature based on the frame features of the key frame.

18. The device according to claim 17, characterized in that The determining subunit is configured to, when one of the key frames is determined, determine the frame feature of the key frame as the first resource feature; When a plurality of key frames are determined, frame features of the plurality of key frames are fused to obtain a fused feature, and the fused feature is determined as the first resource feature.

19. The device according to claim 17, characterized in that The determination subunit is configured to execute the acquisition of frame features of each of the video frames; clustering the frame features of multiple video frames to obtain a cluster center; and determining, from the multiple video frames, a video frame whose frame features are closest to the cluster center as the key frame.

20. The device according to any one of claims 13 to 19, characterized in that The device also includes: The acquisition unit is further configured to execute the acquisition of a search feature of any search information and a plurality of video information, each of the video information including text information and video resources; The acquisition unit is further configured to execute, for each of the video information, based on the feature extraction model, acquiring text features of the text information in the video information and resource features of the video resources, and fusing the text features with the resource features to obtain video features of the video information; The determination unit is configured to perform a determination based on the similarity between the search feature and each video feature of the video information, and determine at least one video information with the greatest similarity as the video information associated with the search information.

21. A processing device based on a feature extraction model, characterized in that: The device comprises: An acquisition unit is configured to execute acquisition of a search feature of any search information and a plurality of video information, each of the video information includes text information and video resources; The acquisition unit is further configured to execute, for each of the video information, based on the feature extraction model, acquiring text features of the text information in the video information and resource features of the video resources, and fusing the text features with the resource features to obtain video features of the video information; The determination unit is further configured to perform, based on the similarity between the search feature and the video feature of each of the video information, determining at least one video information with the greatest similarity as the video information associated with the search information; Among them, the feature extraction model is trained based on the matching relationship between the first video feature and the second video feature and the retrieval features of the sample retrieval information respectively, the first video information corresponding to the first video feature is associated with the sample retrieval information and includes first text information and a first video resource, the video information corresponding to the second video feature includes text information associated with the sample retrieval information, but the included video resources are not associated with the sample retrieval information.

22. The device according to claim 21, characterized in that The device also includes: The acquisition unit is further configured to execute acquisition of a retrieval feature of the sample retrieval information, the first video information associated with the sample retrieval information, and the second video information not associated with the sample retrieval information, wherein the first text information is used to describe the first video resource, and the second video information includes a second video resource; The acquisition unit is further configured to acquire, based on the feature extraction model, a first text feature of the first text information in the first video information and a first resource feature of the first video resource, and fuse the first text feature with the first resource feature to obtain the first video feature; The acquisition unit is further configured to acquire a second resource feature of the second video resource in the second video information based on the feature extraction model, and fuse the first text feature with the second resource feature to obtain the second video feature; The training unit is configured to perform training on the feature extraction model based on the matching relationship between the first video feature and the second video feature and the retrieval feature respectively.

23. The device according to claim 22, characterized in that The second video information further includes second text information, and the second text information is used to describe the second video resource. The acquisition unit is configured to execute based on the feature extraction model, acquire the second text feature of the second text information in the second video information and the second resource feature of the second video resource, fuse the second text feature with the second resource feature to obtain the third video feature of the second video information, and fuse the first text feature with the second resource feature to obtain the second video feature; The training unit is configured to perform training on the feature extraction model based on the matching relationships between the first video feature, the second video feature, and the third video feature and the retrieval feature respectively.

24. The device according to claim 22, characterized in that The first video resource includes multiple video frames, and the acquisition unit is configured to determine key frames among the multiple video frames, where the key frames contain the main content described by the first video resource; based on the feature extraction model, perform feature extraction on the key frames to obtain frame features of the key frames, and determine the first resource features based on the frame features of the key frames.

25. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the processing method based on the feature extraction model as described in any one of claims 1 to 8; or to implement the processing method based on the feature extraction model as described in any one of claims 9 to 12.

26. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the processing method based on the feature extraction model as described in any one of claims 1 to 8; or, the processing method based on the feature extraction model as described in any one of claims 9 to 12.

27. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the processing method based on the feature extraction model as described in any one of claims 1 to 8 is implemented; or, the processing method based on the feature extraction model as described in any one of claims 9 to 12 is implemented.