Search method, model training method, device, electronic device and medium

Abstract

The present disclosure provides a search method, a model training method, an apparatus, an electronic device and a medium, which relate to the field of artificial intelligence technology, specifically the field of deep learning, image processing and computer vision technology, and can be applied to scenarios such as video processing. The specific implementation scheme is: obtaining local features and semantic features obtained by extracting features from multiple candidate videos respectively, the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content body in the corresponding candidate video belongs; the local features and the semantic features are subjected to feature fusion to obtain fused features. Therefore, in the scenario of searching for videos based on text, the target video that matches the search text is determined from each candidate video based on the semantic features (i.e., deep features) and local features (i.e., shallow features) of each candidate video, which can improve the reliability of the determination result, so that the video search results can meet the actual search needs of the user.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology, specifically deep learning, image processing, and computer vision technology, and can be applied to scenarios such as video processing. In particular, it relates to a search method, a model training method, an apparatus, an electronic device, and a medium. Background Art

[0002] With the rapid development of internet technology and short video applications, online videos, as an important resource carrier, have seen a dramatic increase in volume. For example, the internet generates a massive amount of videos every day. Therefore, it is crucial to search for online videos that match the text input by users to meet their search needs. Summary of the Invention

[0003] The present disclosure provides a search method, device, electronic device, and medium.

[0004] According to one aspect of the present disclosure, a search method is provided, comprising:

[0005] Get the search text and multiple candidate videos;

[0006] Obtaining local features and semantic features obtained by extracting features from each of the plurality of candidate videos, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content of the corresponding candidate video belongs;

[0007] Performing feature fusion on the local features and the semantic features to obtain fused features;

[0008] According to the correlation between the features of the search text and the fusion features of each of the candidate videos, a target video matching the search text is determined from the multiple candidate videos.

[0009] According to another aspect of the present disclosure, a model training method is provided, comprising:

[0010] Acquire a training sample, wherein the training sample includes a sample text and a corresponding plurality of sample videos, wherein the sample text is annotated with correlation annotation information between the plurality of sample videos and the sample text;

[0011] Using a search model to extract features from the multiple sample videos, respectively, to obtain local features and semantic features, and fusing the local features and the semantic features to obtain fused features; wherein the local features are used to characterize at least one of the style and appearance of the corresponding sample videos, and the semantic features are used to characterize the category to which the content of the corresponding sample videos belongs;

[0012] Determining correlation prediction information between the sample text and each of the sample videos based on the features of the sample text and the fusion features of each of the sample videos;

[0013] According to the difference between the correlation annotation information and the correlation prediction information, the model parameters of the search model are adjusted.

[0014] According to another aspect of the present disclosure, there is provided a search device, comprising:

[0015] A first acquisition module is used to acquire a search text and multiple candidate videos;

[0016] A second acquisition module is configured to acquire local features and semantic features obtained by extracting features from the plurality of candidate videos, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content of the corresponding candidate video belongs;

[0017] A fusion module, configured to fuse the local features and the semantic features to obtain fused features;

[0018] The determination module is configured to determine a target video matching the search text from the plurality of candidate videos based on the correlation between the features of the search text and the fusion features of each candidate video.

[0019] According to another aspect of the present disclosure, a model training device is provided, comprising:

[0020] An acquisition module, configured to acquire a training sample, wherein the training sample includes a sample text and a corresponding plurality of sample videos, wherein the sample text is annotated with correlation annotation information between the plurality of sample videos and the sample text;

[0021] a processing module configured to extract features from each of the plurality of sample videos using a search model to obtain local features and semantic features, and to fuse the local features and the semantic features to obtain fused features; wherein the local features are used to characterize at least one of the style and appearance of the corresponding sample videos, and the semantic features are used to characterize the category to which the main content of the corresponding sample videos belongs;

[0022] a determination module, configured to determine correlation prediction information between the sample text and each of the sample videos based on the features of the sample text and the fusion features of each of the sample videos;

[0023] An adjustment module is used to adjust the model parameters of the search model according to the difference between the correlation annotation information and the correlation prediction information.

[0024] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0025] at least one processor; and

[0026] a memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the search method proposed in the above-mentioned first aspect of the present disclosure, or execute the model training method proposed in the above-mentioned other aspect of the present disclosure.

[0028] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium of computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the search method proposed in the above-mentioned first aspect of the present disclosure, or to execute the model training method proposed in the above-mentioned other aspect of the present disclosure.

[0029] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the search method proposed in the above-mentioned first aspect of the present disclosure, or implements the model training method proposed in the above-mentioned other aspect of the present disclosure.

[0030] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0032] Figure 1 A flowchart of the search method provided in the first embodiment of the present disclosure;

[0033] Figure 2 A flowchart of the search method provided in the second embodiment of the present disclosure;

[0034] Figure 3 A flowchart of the search method provided in the third embodiment of the present disclosure;

[0035] Figure 4 A schematic diagram of the structure of a local branch network provided by an embodiment of the present disclosure;

[0036] Figure 5 A flowchart of the search method provided in the fourth embodiment of the present disclosure;

[0037] Figure 6This is a flowchart of the model training method provided in the fifth embodiment of the present disclosure;

[0038] Figure 7 A flowchart of the model training method provided in Example 6 of the present disclosure;

[0039] Figure 8 This is a flow chart of the model training method provided in the seventh embodiment of the present disclosure;

[0040] Figure 9 A flowchart of the model training method provided in the eighth embodiment of the present disclosure;

[0041] Figure 10 A flowchart of the model training method provided in the ninth embodiment of the present disclosure;

[0042] Figure 11 A flowchart of the model training method provided in the tenth embodiment of the present disclosure;

[0043] Figure 12 Schematic diagram of the training process of the search model in the embodiment of the present disclosure;

[0044] Figure 13 A schematic diagram of the structure of a search device provided in the eleventh embodiment of the present disclosure;

[0045] Figure 14 This is a schematic diagram of the structure of the model training device provided in the twelfth embodiment of the present disclosure;

[0046] Figure 15 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0047] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0048] With the continuous enrichment of internet resources, simple text or image retrieval can no longer meet user search needs. Cross-modal retrieval, which allows for cross-modal retrieval across multiple modalities, has become a key research direction and application area in the retrieval field. These modalities can include text, images, video, audio, and other modalities. In recent years, with the rapid growth of short video applications, video has become a crucial resource, encompassing information from multiple modalities. Video retrieval and text-video cross-modal retrieval have become research hotspots.

[0049] Currently, text-video cross-modal retrieval can be achieved based on either retrieval models or search models. To improve the accuracy of video retrieval results, the model can be pre-trained using large-scale data and then fine-tuned for downstream tasks. Furthermore, to better understand video content, the model can be trained by integrating multimodal information.

[0050] That is, fusing multimodal information can expand the feature space that the model can express, and there is a certain degree of information complementarity between different modal information. Fusing multimodal information to train the model can enable the model to obtain features with better generalization ability and stronger discrimination ability.

[0051] In related technologies, the mainstream text-video cross-modal retrieval method is to model the text modality and video modality, using text-video matching pairs for supervision and model training. It is generally divided into two-stream models and single-stream models. Among them, the single-stream model jointly inputs the features of text and video into the model during the input stage, while the two-stream model uses two encoders for modeling. After extracting text features and video features through the two encoders, the features of the two modalities are aligned. The encoder mainly adopts the Transformer structure because the Transformer-based ViT (Vision Transformer) and Swin (Shifted Windows)-Transformer (a Transformer that incorporates the sliding window concept of CNN (Convolutional Neural Network)) have been proven to have the ability to fit large-scale data.

[0052] However, the existing dual-stream model and single-stream model have the following problems:

[0053] First, there is a certain semantic gap between model training and downstream tasks. That is, during model training, contrastive loss is usually used for model optimization. However, in actual text and video retrieval tasks, the focus is not on determining whether two samples are positive or negative pairs, but on recalling all similar samples and correlation scores in the feature library based on the features extracted from the input. When both recall rate and accuracy are required, a more direct standard is needed to constrain model optimization and improve retrieval indicators during application.

[0054] To address the above issues, the present disclosure can improve the retrieval effect by updating the loss function and sample construction method.

[0055] Second, because the model's training goal is to align the final output text features with the video features, during training, the video features become increasingly high-level, closer to the semantic expression of the text. The model also tends to focus on the main subject in the video to extract features, thus losing the concept and style of the video. This approach can easily lead to false recall of many different video clips.

[0056] To address the above issues, the present disclosure may use a local branch network to obtain relatively low-level (or shallow) information in the video, and fuse the shallow features with the video features output by the model.

[0057] The following describes the search method, model training method, device, electronic device and medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart of the search method provided in the first embodiment of the present disclosure.

[0059] like Figure 1 As shown, the search method may include the following steps:

[0060] Step 101: Obtain a search text and multiple candidate videos.

[0061] The execution subject of the embodiment of the present disclosure may be a server.

[0062] In an embodiment of the present disclosure, the search text can be sent from the client to the server, where the client refers to a software program running on an electronic device to provide services to users, such as an application with a search function, such as a browser, video application, short video application, etc.

[0063] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, mobile terminal, etc. The mobile terminal can be, for example, a mobile phone, tablet computer, personal digital assistant, wearable device, vehicle-mounted device, etc., which are hardware devices with various operating systems, touch screens and / or display screens.

[0064] In the embodiment of the present disclosure, the candidate video may be a video stored on the server.

[0065] In the embodiment of the present disclosure, the user can enter a search text on the client side, and the input method of the search text includes but is not limited to touch input (such as sliding, clicking, etc.), keyboard input, voice input, etc. After receiving the search text entered by the user, the client can send the search text to the server.

[0066] Step 102: Obtain local features and semantic features obtained by extracting features from multiple candidate videos, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content body in the corresponding candidate video belongs.

[0067] In the disclosed embodiment, local features and semantic features obtained by feature extraction of each candidate video can be obtained, wherein the local features are used to characterize the style (such as story-telling, funny, educational, emotional, etc.) and / or appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content subject in the corresponding candidate video belongs, wherein the content subject can be a person, an animal, an object, etc., and the category to which the content subject belongs can be a specific person's name, a specific type of animal, a specific type of object, etc.

[0068] As an example, the server can perform feature extraction on each candidate video based on multiple encoding methods to obtain local features and semantic features corresponding to each candidate video. Alternatively, to improve the real-time performance of the search, the server can pre-extract features from each candidate video to obtain local features and semantic features of each candidate video, and store the extracted local features and semantic features in correspondence with each candidate video. In this way, in the present disclosure, the local features and semantic features corresponding to each candidate video can be directly obtained through query.

[0069] Step 103: Fusing the local features and the semantic features to obtain fused features.

[0070] In the embodiment of the present disclosure, for each candidate video, the local features and semantic features of the candidate video may be fused based on feature fusion technology to obtain fused features corresponding to the candidate video.

[0071] In a possible implementation of the embodiment of the present disclosure, the fusion features of each candidate video may be aligned with the features of the search text based on a feature alignment algorithm, so that correlation calculation may be performed based on the aligned features.

[0072] Step 104 : Determine a target video that matches the search text from the plurality of candidate videos based on the correlation between the features of the search text and the fusion features of each candidate video.

[0073] Correlation is used to indicate the degree of association between two features.

[0074] The number of target videos may be at least one.

[0075] In the embodiment of the present disclosure, feature extraction can be performed on the search text to obtain the features of the search text (or called text features), and the correlation between the features of the search text and the fusion features of each candidate video can be calculated, so that the target video that matches the search text can be determined from multiple candidate videos based on the correlation between the features of the search text and the fusion features of each candidate video.

[0076] As an example, the similarity between the features of the search text and the fusion features of each candidate video can be calculated based on a similarity measurement algorithm (such as Pearson correlation coefficient, cosine similarity, etc.) or a distance measurement algorithm (such as Euclidean distance, Manhattan distance, Chebyshev distance and Minkowski distance, etc.), so that the correlation between the features of the search text and the fusion features of each candidate video can be determined based on the similarity between the features of the search text and the fusion features of each candidate video, wherein the similarity and the correlation are positively correlated, that is, the higher the similarity, the higher the correlation between the features of the search text and the fusion features.

[0077] As a possible implementation manner, the correlation may include positive correlation and negative correlation, and according to the correlation, a candidate video that is positively correlated with the search text may be used as a target video.

[0078] As another possible implementation, the correlation may include a correlation score, wherein a higher correlation score indicates a higher degree of association between the two features, and conversely, a lower correlation score indicates a lower degree of association between the two features. Based on the correlation between the features of the search text and the fusion features of each candidate video, the candidate videos may be sorted in descending order of their correlation scores, thereby selecting a set number of candidate videos ranked first and serving as target videos. Alternatively, candidate videos with correlation scores above a set threshold may be selected and served as target videos.

[0079] Optionally, the server may also send video information of the target video to the client, so that the user can watch the target video based on the client.

[0080] As an example, taking the video information as the original video corresponding to the target video, the server can directly send the original video corresponding to the target video to the client, so that the user can directly watch the target video based on the client.

[0081] As another example, assuming that the video information is the storage address of the target video on the server, the server can send the storage address of the target video to the client. Accordingly, after receiving the storage address, the client can access the server through the storage address to obtain the target video, so that the user can watch the target video on the client.

[0082] The search method of the embodiment of the present disclosure obtains local features and semantic features obtained by extracting features from multiple candidate videos respectively, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content body in the corresponding candidate video belongs; the local features and the semantic features are fused to obtain fused features. Therefore, in the scenario of searching for videos based on text, based on the semantic features (i.e., deep features) of each candidate video and the local features (i.e., shallow features) used to characterize the style and / or appearance of the video, the target video that matches the search text is determined from each candidate video, which can improve the reliability of the determination result, so that the video search results can meet the user's actual search needs and improve the user's search experience.

[0083] It should be noted that in the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0084] In order to clearly illustrate how to obtain local features and semantic features of a video in any embodiment of the present disclosure, the present disclosure also proposes a search method.

[0085] Figure 2 This is a flowchart of the search method provided in the second embodiment of the present disclosure.

[0086] like Figure 2 As shown, the search method may include the following steps:

[0087] Step 201: Obtain a search text and multiple candidate videos.

[0088] For explanation of step 201, please refer to the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0089] In step 202 , any one of the multiple candidate videos is used as an input video and input into a video feature extraction network to obtain semantic features of the input video.

[0090] Among them, semantic features are used to represent the category to which the content body in the input video belongs.

[0091] In an embodiment of the present disclosure, any one of multiple candidate videos can be used as an input video, and the input video can be input into a video feature extraction network, so that the video feature extraction network can extract features from the input video to obtain semantic features of the input video.

[0092] Step 203: Input at least one video frame in the input video into an image feature extraction network to obtain semantic features of the at least one video frame in the input video.

[0093] In an embodiment of the present disclosure, at least one video frame in the input video can be input into an image feature extraction network, so that the image feature extraction network can perform feature extraction on the at least one video frame to obtain semantic features of the at least one video frame.

[0094] As an example, the input video may be subjected to frame extraction to obtain at least one video frame. For example, key frames in the input video may be extracted and each key frame may be used as at least one video frame. Alternatively, at least one video frame may be extracted from the input video at a set interval.

[0095] In step 204 , each video frame in the at least one video frame is input into the first local branch network to obtain local features of each video frame.

[0096] The local features are used to characterize at least one of the style and appearance of the input video.

[0097] In an embodiment of the present disclosure, each video frame in at least one video frame in the input video can be input into the first local branch network respectively, so that the first local branch network can extract features from each video frame to obtain local features of each video frame.

[0098] Step 205: Input the input video into the second local branch network to obtain local features of the input video.

[0099] In an embodiment of the present disclosure, the input video may be input into the second local branch network, so that the second local branch network performs feature extraction on the input video to obtain local features of the input video.

[0100] Step 206: Fusing the local features and the semantic features to obtain fused features.

[0101] In the embodiment of the present disclosure, the semantic features and local features of the input video and the semantic features and local features of the video frame may be fused to obtain fused features.

[0102] Step 207 : Determine a target video that matches the search text from the plurality of candidate videos based on the correlation between the features of the search text and the fusion features of each candidate video.

[0103] For explanations of steps 206 to 207 , reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0104] The search method of the embodiment of the present disclosure is based on a feature extraction network of deep learning to extract the semantic features and local features of the video, which can improve the accuracy and reliability of feature extraction. In addition, by extracting the local features of the video and the local features of the video frame through a local branch network, it is possible to effectively extract the shallow features of the video (i.e., local features, the underlying information of the video), so that video search can be performed based on the shallow features (i.e., local features) and deep features (i.e., semantic features, the high-level information of the video), which can improve the accuracy and reliability of the search results to avoid the erroneous recall of videos that are irrelevant to the search text, thereby reducing the user's search experience.

[0105] In order to clearly illustrate how to extract local features of a video frame in any embodiment of the present disclosure, the present disclosure also proposes a search method.

[0106] Figure 3 This is a flowchart of the search method provided in the third embodiment of the present disclosure.

[0107] like Figure 3 As shown, the search method may include the following steps:

[0108] Step 301: Obtain a search text and multiple candidate videos.

[0109] In step 302 , any one of the multiple candidate videos is used as an input video and input into a video feature extraction network to obtain semantic features of the input video.

[0110] Among them, semantic features are used to represent the category to which the content body in the input video belongs.

[0111] Step 303: Input at least one video frame in the input video into an image feature extraction network to obtain semantic features of the at least one video frame in the input video.

[0112] For explanations of steps 301 to 303 , please refer to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0113] Step 304 : For any video frame, generate a first array according to the pixel value of each pixel in multiple color channels.

[0114] The elements in the first array are pixel values, each element having a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs.

[0115] In the embodiment of the present disclosure, the color channels may include, for example, a red R channel, a green G channel, and a blue B channel.

[0116] In an embodiment of the present disclosure, for any one of the at least one video frames mentioned above, a first array can be generated based on the pixel values ​​of each pixel in the any one video frame on multiple color channels, wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs.

[0117] Taking the example that the color channels include three channels of R, G, and B, each element in the first array includes three dimensions, namely W, H, and 3, where W refers to the column position, H refers to the row position, and 3 refers to the color channel.

[0118] Step 305 : Use a sliding window to divide the first array into a plurality of first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays.

[0119] In the embodiment of the present disclosure, the size of the sliding window may be preset.

[0120] In the embodiment of the present disclosure, a sliding window may be used to divide the first array into a plurality of first sub-arrays, wherein overlapping elements exist between adjacent first sub-arrays.

[0121] Step 306 : Use the feature extraction layer of the first local branch network to perform feature extraction on each first sub-array respectively.

[0122] In the embodiment of the present disclosure, the feature extraction layer of the first local branch network may be used to perform feature extraction on each first sub-array to obtain features of each first sub-array.

[0123] As a possible implementation method, the feature extraction layer may include at least one encoding layer, wherein each encoding layer may include MSA (Multi-Head Self-Attention module) and FFN (Feed-Forward Network). The semantic information in the video frame can be fused through MSA and FFN, thereby improving the accuracy and reliability of feature extraction.

[0124] Step 307: Using the window merging layer of the first local branch network, the features of each first sub-array are merged.

[0125] In the embodiment of the present disclosure, the window merging layer of the first local branch network can be used to fuse the features of each first sub-array to obtain fused features.

[0126] As an example, since there are overlapping elements (i.e., overlapping areas) between adjacent first sub-arrays, the features of the adjacent first sub-arrays include features of the same elements. The features of each first sub-array can be spliced ​​based on the features of the overlapping elements between the first sub-arrays to obtain a fused feature.

[0127] As another example, similarity calculation is performed on the features of each first sub-array, and the first sub-arrays are fused according to the calculated similarity to obtain fused features.

[0128] Step 308: Input the fused features into the output layer of the first local branch network to obtain local features of the video frame.

[0129] The local features are used to characterize at least one of the style and appearance of the input video.

[0130] In the embodiment of the present disclosure, the fused features may be input into the output layer of the first local branch network to obtain local features of the video frame.

[0131] In one possible implementation of the disclosed embodiment, the output layer may use a piecewise linear function as the activation function. For example, the activation function used in the output layer may be a RELU function. Because the piecewise linear function does not require exponential calculations, it can reduce computational complexity, thereby improving video search efficiency.

[0132] As an example, the structure of the local branch network is as follows Figure 4 As shown in the example, a first array can be generated according to the pixel values ​​of each pixel in the video frame on multiple color channels, and the first array can be divided by using an overlapping window, that is, the first array can be divided into multiple first sub-arrays by using a sliding window, wherein adjacent first sub-arrays have overlapping elements, such as Figure 4 As shown, subarray 1 includes elements from regions 1, 2, 4, and 5, and subarray 2 includes elements from regions 2, 3, 5, and 6. Subarray 1 and subarray 2 have overlapping elements from regions 2 and 5. For another example, subarray 3 has elements from regions 4, 5, 7, and 8, and subarray 1 and subarray 3 have overlapping elements from regions 4 and 5.

[0133] Afterwards, the feature extraction layer can be used to extract features from each of the first sub-arrays to obtain the features corresponding to each first sub-array. The window merging layer is then used to fuse the features of each first sub-array to obtain fused features. Finally, the fused features are input to the output layer to obtain the local features of the video frame.

[0134] in, Figure 4Taking the example of a feature extraction layer including four encoding layers, Win(window)-MSA refers to feature extraction of a sub-array behind a window.

[0135] Step 309: Input the input video into the second local branch network to obtain local features of the input video.

[0136] Step 310: Fusing local features and semantic features to obtain fused features.

[0137] In the embodiment of the present disclosure, the semantic features and local features of the input video, and the semantic features and local features of the video frames in the input video may be fused to obtain fused features.

[0138] Step 311 : Determine a target video that matches the search text from multiple candidate videos based on the correlation between the features of the search text and the fusion features of each candidate video.

[0139] For explanations of steps 309 to 311 , reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0140] The search method of the embodiment of the present disclosure generates a first array based on the pixel values ​​of each pixel point on multiple color channels for any video frame; uses a sliding window to divide the first array into multiple first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays; uses the feature extraction layer of the first local branch network to extract features from each first sub-array respectively; uses the window merging layer of the first local branch network to fuse the features of each first sub-array; and inputs the fused features into the output layer of the first local branch network to obtain the local features of the video frame. Thus, by using a sliding window to divide the first array to obtain each sub-array and extracting features from each sub-array, it is possible to extract detailed features from the video frame, thereby improving the accuracy of local feature extraction. Moreover, the sliding window is an overlapping or interleaved window, which can avoid the problem of image information loss during the feature extraction process, thereby improving the robustness of local feature extraction.

[0141] In order to clearly illustrate how to extract local features of a video in any embodiment of the present disclosure, the present disclosure also proposes a search method.

[0142] Figure 5 This is a flowchart of the search method provided in the fourth embodiment of the present disclosure.

[0143] like Figure 5 As shown, the search method may include the following steps:

[0144] Step 501: Obtain a search text and multiple candidate videos.

[0145] In step 502 , any one of the multiple candidate videos is used as an input video and input into a video feature extraction network to obtain semantic features of the input video.

[0146] Among them, semantic features are used to represent the category to which the content body in the input video belongs.

[0147] Step 503: Input at least one video frame in the input video into an image feature extraction network to obtain semantic features of the at least one video frame in the input video.

[0148] In step 504 , each video frame in the at least one video frame is input into the first local branch network to obtain local features of each video frame.

[0149] The local features are used to characterize at least one of the style and appearance of the input video.

[0150] For explanations of steps 501 to 504 , reference can be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0151] Step 505: Generate a second array according to each video frame in the input video.

[0152] The elements in the second array are pixel values, each element having a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension for indicating the row position of the pixel point to which it belongs, and a third dimension for indicating the column position of the pixel point to which it belongs.

[0153] In an embodiment of the present disclosure, a second array can be generated based on each video frame in the input video, wherein the elements in the second array are pixel values, and each element has a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension indicating the row position of the pixel point to which it belongs, and a third dimension indicating the column position of the pixel point to which it belongs.

[0154] For example, if the color channels include R, G, and B, the elements in the second array have three dimensions: W, H, and 3T, where W refers to the column position, H refers to the row position, 3 refers to the color channel, and T refers to the number of video frames. For example, if an element belongs to the fifth frame of the input video and is located at the i-th row and j-th column of the fifth frame, the three dimensions of the element are i, j, and 3*5.

[0155] Step 506: Use a sliding window to divide the second array into a plurality of second sub-arrays, wherein there are overlapping elements between adjacent second sub-arrays.

[0156] In the embodiment of the present disclosure, the size of the sliding window may be preset.

[0157] In the embodiment of the present disclosure, a sliding window may be used to divide the second array into a plurality of second sub-arrays, wherein overlapping elements exist between adjacent second sub-arrays.

[0158] Step 507 : Use the feature extraction layer of the second local branch network to perform feature extraction on each second sub-array respectively.

[0159] In the embodiment of the present disclosure, the feature extraction layer of the second local branch network may be used to perform feature extraction on each second sub-array to obtain features of each second sub-array.

[0160] As a possible implementation method, the feature extraction layer may include at least one coding layer, wherein each coding layer may include MSA and FFN. Semantic information in the video can be fused through MSA and FFN, thereby improving the accuracy and reliability of feature extraction.

[0161] Step 508: Using the window merging layer of the second local branch network, the features of each second sub-array are merged.

[0162] In the embodiment of the present disclosure, the window merging layer of the second local branch network can be used to fuse the features of each second sub-array to obtain fused features.

[0163] As an example, since there are overlapping elements (i.e., overlapping areas) between adjacent second sub-arrays, the features of adjacent first sub-arrays include features of the same elements. The features of each second sub-array can be spliced ​​based on the features of the overlapping elements between the second sub-arrays to obtain a fused feature.

[0164] As another example, similarity calculation is performed on the features of each second sub-array, and each first sub-array is fused according to the calculated similarity to obtain fused features.

[0165] Step 509: Input the fused features into the output layer of the second local branch network to obtain local features of the input video.

[0166] In the embodiment of the present disclosure, the fused features may be input into the output layer of the second local branch network to obtain local features of the input video.

[0167] In one possible implementation of the disclosed embodiment, the output layer may use a piecewise linear function as the activation function. For example, the activation function used in the output layer may be a RELU function. Because the piecewise linear function does not require exponential calculations, it can reduce computational complexity, thereby improving video search efficiency.

[0168] Step 510: Fusing local features and semantic features to obtain fused features.

[0169] Step 511 : Determine a target video that matches the search text from multiple candidate videos based on the correlation between the features of the search text and the fusion features of each candidate video.

[0170] For explanations of steps 510 to 511 , reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0171] The search method of the disclosed embodiment uses a sliding window to partition the second array into subarrays, and then performs feature extraction on each subarray. This allows for the extraction of detailed features from the video, thereby improving the accuracy of local feature extraction. Furthermore, the sliding windows are overlapping or interleaved windows, which prevents the loss of video information during the feature extraction process, thereby improving the robustness of local feature extraction.

[0172] The above are various embodiments corresponding to the model application method (ie, the search method). The present disclosure also proposes a model training method.

[0173] Figure 6 This is a flowchart of the model training method provided in Example 5 of the present disclosure.

[0174] like Figure 6 The model training method may include the following steps:

[0175] Step 601: Acquire a training sample, wherein the training sample includes a sample text and a corresponding plurality of sample videos, wherein the sample text is annotated with correlation annotation information between the plurality of sample videos and the sample text.

[0176] In the disclosed embodiment, the number of training samples may be at least one, wherein each training sample may include sample text and multiple sample videos corresponding to the sample text, and the sample text is annotated with correlation annotation information between the multiple sample videos and the sample text. For example, the correlation annotation information may be used to indicate a correlation score or correlation degree between the sample video and the sample text, or for another example, the correlation annotation information may be used to indicate whether the sample video and the sample text are related.

[0177] At a certain moment, when the correlation labeling information indicates that the sample video is correlated with the sample text, or indicates that the correlation score between the sample video and the sample text is high, the sample text and the sample video can be called a positive sample pair. When the correlation labeling information indicates that the sample video is not correlated with the sample text, or indicates that the correlation score between the sample video and the sample text is low, the sample text and the sample video can be called a negative sample pair.

[0178] In the embodiments of the present disclosure, there is no restriction on the method of obtaining training samples. For example, training samples can be obtained from an existing training set, or training samples can be collected online, such as by using web crawler technology to collect training samples online, or training samples can be collected offline, or training samples can be generated manually, etc. The present disclosure does not impose any restrictions on this.

[0179] In step 602 , a search model is used to extract features from a plurality of sample videos to obtain local features and semantic features, and the local features and semantic features are fused to obtain fused features.

[0180] The local features are used to characterize at least one of the style and appearance of the corresponding sample video, and the semantic features are used to characterize the category to which the content of the corresponding sample video belongs.

[0181] In the embodiments of the present disclosure, the explanation of local features and semantic features can be found in the relevant description of any embodiment of the present disclosure and will not be repeated here.

[0182] In the disclosed embodiment, a search model can be used to extract features from multiple sample videos respectively to obtain local features and semantic features corresponding to each sample video, and the local features and semantic features of each sample video are fused to obtain fused features of the corresponding sample video.

[0183] Step 603 : Determine correlation prediction information between the sample text and each sample video based on the features of the sample text and the fusion features of each sample video.

[0184] The relevance prediction information may be used to indicate a relevance score or relevance between the sample text and the sample video, or may be used to indicate whether the sample text and the sample video are relevant.

[0185] In the embodiment of the present disclosure, feature extraction can be performed on the sample text to obtain the features of the sample text (i.e., text features), and the correlation prediction information between the sample text and each sample video can be determined based on the fusion features of the sample text and each sample video.

[0186] As an example, for any sample video among multiple sample videos, the similarity between the fused features of the sample video and the features of the sample text can be calculated, and correlation prediction information between the sample video and the sample text can be determined based on the similarity. For example, the higher the similarity, the higher the correlation score or correlation indicated by the correlation prediction information; conversely, the lower the similarity, the lower the correlation score or correlation indicated by the correlation prediction information.

[0187] Step 604: Adjust the model parameters of the search model according to the difference between the relevance annotation information and the relevance prediction information.

[0188] In the embodiment of the present disclosure, the model parameters in the search model may be adjusted according to the difference between the relevance annotation information and the relevance prediction information.

[0189] As an example, a loss function can be generated based on the difference between the correlation annotation information and the correlation prediction information, wherein the loss function is positively correlated with the difference (i.e., positively correlated), that is, the smaller the difference, the smaller the value of the loss function, and conversely, the larger the difference, the larger the value of the loss function. Therefore, in the present disclosure, the model parameters in the search model can be adjusted according to the loss function to minimize the value of the loss function.

[0190] It should be noted that the above only uses the termination condition of the search model training as an example of minimizing the value of the loss function. In actual application, other termination conditions can also be set. For example, the termination condition can be that the number of training times reaches a set number, or the termination condition can be that the training time reaches a set time, etc. The present disclosure does not impose any restrictions on this.

[0191] The model training method of the embodiment of the present disclosure uses a search model to extract features from multiple sample videos in the training sample respectively to obtain local features and semantic features, and fuses the local features and semantic features to obtain fused features; determines the correlation prediction information between the sample text and each sample video based on the features of the sample text in the training sample and the fused features of each sample video; and adjusts the model parameters of the search model based on the difference between the correlation annotation information and the correlation prediction information of the sample text annotation. Thus, the use of deep learning technology to extract video features can improve the accuracy and reliability of feature extraction. In addition, based on the semantic features of the video (i.e., deep features) and the local features (shallow features) used to characterize the style and / or appearance of the video, the correlation prediction information between the video and the text is determined, which can improve the reliability of the correlation prediction information determination, thereby training the search model based on the reliable correlation prediction information, which can improve the training efficiency of the model.

[0192] In order to clearly illustrate how the search model in the above embodiment extracts local features and semantic features of the sample video, the present disclosure also proposes a model training method.

[0193] Figure 7 This is a flowchart of the model training method provided in Example 6 of the present disclosure.

[0194] Step 701: Obtain training samples, which include sample text and corresponding multiple sample videos.

[0195] The sample text is annotated with correlation annotation information between multiple sample videos and the sample text.

[0196] For explanation of step 701, please refer to the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0197] In step 702 , any one of the multiple sample videos is used as an input video and input into the video feature extraction network of the search model to obtain semantic features of the input video.

[0198] Among them, semantic features are used to represent the category to which the content body in the input video belongs.

[0199] In an embodiment of the present disclosure, any one of the multiple sample videos can be used as an input video, and the input video is input into a video feature extraction network of a search network to obtain semantic features of the input video.

[0200] Step 703: Input at least one video frame in the input video into the image feature extraction network of the search model to obtain semantic features of the at least one video frame in the input video.

[0201] In an embodiment of the present disclosure, at least one video frame in the input video can be input into the image feature extraction network of the search model, so that the image feature extraction network can perform feature extraction on the at least one video frame to obtain semantic features of the at least one video frame.

[0202] As an example, the input video may be subjected to frame extraction to obtain at least one video frame. For example, key frames in the input video may be extracted and each key frame may be used as at least one video frame. Alternatively, at least one video frame may be extracted from the input video at a set interval.

[0203] In step 704 , each video frame in the at least one video frame is input into the first local branch network of the search model to obtain local features of each video frame.

[0204] The local features are used to characterize at least one of the style and appearance of the input video.

[0205] In an embodiment of the present disclosure, each video frame of at least one video frame in the input video can be input into the first local branch network of the search model respectively, so that the first local branch network can extract features of each video frame to obtain local features of each video frame.

[0206] Step 705: Input the input video into the second local branch network of the search model to obtain local features of the input video.

[0207] In an embodiment of the present disclosure, the input video may be input into the second local branch network of the search model, so that the second local branch network performs feature extraction on the input video to obtain local features of the input video.

[0208] Step 706 : Using a search model, the semantic features and local features of the input video and the semantic features and local features of each video frame are fused to obtain fused features of the input video.

[0209] Step 707 : Determine correlation prediction information between the sample text and each sample video based on the features of the sample text and the fusion features of each sample video.

[0210] Step 708: Adjust the model parameters of the search model according to the difference between the relevance annotation information and the relevance prediction information.

[0211] For explanations of steps 706 to 708 , reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0212] The model training method of the embodiment of the present disclosure is based on a deep learning feature extraction network to extract the semantic features and local features of the video, which can improve the accuracy and reliability of feature extraction. In addition, by extracting the local features of the video and the local features of the video frame through a local branch network, it is possible to effectively extract the shallow features of the video (i.e., local features, the underlying information of the video), thereby determining the correlation between the video and the text based on the shallow features (i.e., local features) and deep features (i.e., semantic features, the high-level information of the video), and determining the accuracy and reliability of the results.

[0213] In order to clearly illustrate how local features of a video frame are extracted in any embodiment of the present disclosure, the present disclosure also proposes a model training method.

[0214] Figure 8 This is a flow chart of the model training method provided in Example 7 of the present disclosure.

[0215] like Figure 8 As shown, the model training method may include the following steps:

[0216] Step 801: Obtain training samples, which include sample text and corresponding sample videos.

[0217] The sample text is annotated with correlation annotation information between multiple sample videos and the sample text.

[0218] In step 802 , any one of the multiple sample videos is used as an input video and input into the video feature extraction network of the search model to obtain the semantic features of the input video.

[0219] Among them, semantic features are used to represent the category to which the content body in the input video belongs.

[0220] Step 803: Input at least one video frame in the input video into the image feature extraction network of the search model to obtain semantic features of the at least one video frame in the input video.

[0221] The explanation of steps 801 to 803 can be found in the relevant description of any embodiment of the present disclosure and will not be repeated here.

[0222] Step 804 : For any video frame in the at least one video frame, generate a first array according to the pixel values ​​of each pixel point on multiple color channels.

[0223] The elements in the first array are pixel values, each element having a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs.

[0224] In the embodiment of the present disclosure, the color channels may include, for example, a red R channel, a green G channel, and a blue B channel.

[0225] In an embodiment of the present disclosure, for any one of the at least one video frames mentioned above, a first array can be generated based on the pixel values ​​of each pixel in the any one video frame on multiple color channels, wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs.

[0226] Taking the example that the color channels include three channels of R, G, and B, each element in the first array includes three dimensions, namely W, H, and 3, where W refers to the column position, H refers to the row position, and 3 refers to the color channel.

[0227] Step 805 : Use a sliding window to divide the first array into a plurality of first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays.

[0228] In the embodiment of the present disclosure, the size of the sliding window may be preset.

[0229] In the embodiment of the present disclosure, a sliding window may be used to divide the first array into a plurality of first sub-arrays, wherein overlapping elements exist between adjacent first sub-arrays.

[0230] Step 806 : Use the feature extraction layer of the first local branch network to perform feature extraction on each first sub-array respectively.

[0231] In the embodiment of the present disclosure, the feature extraction layer of the first local branch network in the search model may be used to perform feature extraction on each first sub-array to obtain features of each first sub-array.

[0232] As a possible implementation method, the feature extraction layer may include at least one coding layer, wherein each coding layer may include MSA and FFN. The semantic information in the video frame can be fused by MSA and FFN, thereby improving the accuracy and reliability of feature extraction.

[0233] Step 807: Using the window merging layer of the first local branch network, the features of each first sub-array are merged.

[0234] In the embodiment of the present disclosure, the window merging layer of the first local branch network can be used to fuse the features of each first sub-array to obtain fused features.

[0235] As an example, since there are overlapping elements (i.e., overlapping areas) between adjacent first sub-arrays, the features of the adjacent first sub-arrays include features of the same elements. The features of each first sub-array can be spliced ​​based on the features of the overlapping elements between the first sub-arrays to obtain a fused feature.

[0236] As another example, similarity calculation is performed on the features of each first sub-array, and the first sub-arrays are fused according to the calculated similarity to obtain fused features.

[0237] Step 808: Input the fused features into the output layer of the first local branch network to obtain local features of the video frame.

[0238] The local features are used to characterize at least one of the style and appearance of the input video.

[0239] In the embodiment of the present disclosure, the fused features may be input into the output layer of the first local branch network to obtain local features of the video frame.

[0240] In one possible implementation of the disclosed embodiment, the output layer may use a piecewise linear function as the activation function. For example, the activation function used in the output layer may be a RELU function. Because the piecewise linear function does not require exponential calculations, it can reduce computational complexity, thereby improving video search efficiency.

[0241] Step 809: Input the input video into the second local branch network of the search model to obtain local features of the input video.

[0242] In step 810 , a search model is used to fuse the semantic features and local features of the input video, and the semantic features and local features of each video frame, to obtain fused features of the input video.

[0243] Step 811 : Determine correlation prediction information between the sample text and each sample video based on the features of the sample text and the fusion features of each sample video.

[0244] Step 812: Adjust the model parameters of the search model according to the difference between the relevance annotation information and the relevance prediction information.

[0245] The explanation of steps 809 to 812 can be found in the relevant description of any embodiment of the present disclosure and will not be repeated here.

[0246] The model training method of the disclosed embodiment uses a sliding window to partition a first array into subarrays, and then performs feature extraction on each subarray. This method can extract detailed features from video frames, thereby improving the accuracy of local feature extraction. Furthermore, the sliding windows are overlapping or interleaved windows, which can avoid the loss of image information during the feature extraction process, thereby improving the robustness of local feature extraction.

[0247] In order to clearly illustrate how local features of a video are extracted in any embodiment of the present disclosure, the present disclosure also proposes a model training method.

[0248] Figure 9 This is a flow chart of the model training method provided in Example 8 of the present disclosure.

[0249] like Figure 9 As shown, the model training method may include the following steps:

[0250] Step 901: Obtain training samples, where the training samples include sample text and corresponding multiple sample videos.

[0251] The sample text is annotated with correlation annotation information between multiple sample videos and the sample text.

[0252] In step 902 , any one of the multiple sample videos is used as an input video and input into the video feature extraction network of the search model to obtain the semantic features of the input video.

[0253] Among them, semantic features are used to represent the category to which the content body in the input video belongs.

[0254] Step 903: Input at least one video frame in the input video into the image feature extraction network of the search model to obtain semantic features of the at least one video frame in the input video.

[0255] In step 904 , each video frame in the at least one video frame is input into the first local branch network of the search model to obtain local features of each video frame.

[0256] For explanations of steps 901 to 904 , reference can be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0257] Step 905: Generate a second array according to each video frame in the input video.

[0258] The elements in the second array are pixel values, each element having a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension for indicating the row position of the pixel point to which it belongs, and a third dimension for indicating the column position of the pixel point to which it belongs.

[0259] In an embodiment of the present disclosure, a second array can be generated based on each video frame in the input video, wherein the elements in the second array are pixel values, and each element has a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension indicating the row position of the pixel point to which it belongs, and a third dimension indicating the column position of the pixel point to which it belongs.

[0260] For example, if the color channels include R, G, and B, the elements in the second array have three dimensions: W, H, and 3T, where W refers to the column position, H refers to the row position, 3 refers to the color channel, and T refers to the number of video frames. For example, if an element belongs to the fifth frame of the input video and is located at the i-th row and j-th column of the fifth frame, the three dimensions of the element are i, j, and 3*5.

[0261] Step 906: Use a sliding window to divide the second array into a plurality of second sub-arrays, wherein there are overlapping elements between adjacent second sub-arrays.

[0262] In the embodiment of the present disclosure, the size of the sliding window may be preset.

[0263] In the embodiment of the present disclosure, a sliding window may be used to divide the second array into a plurality of second sub-arrays, wherein overlapping elements exist between adjacent second sub-arrays.

[0264] Step 907 : Use the feature extraction layer of the second local branch network to perform feature extraction on each second sub-array respectively.

[0265] In the embodiment of the present disclosure, the feature extraction layer of the second local branch network in the search model may be used to perform feature extraction on each second sub-array to obtain features of each second sub-array.

[0266] As a possible implementation method, the feature extraction layer may include at least one coding layer, wherein each coding layer may include MSA and FFN. Semantic information in the video can be fused through MSA and FFN, thereby improving the accuracy and reliability of feature extraction.

[0267] Step 908: Using the window merging layer of the second local branch network, the features of each second sub-array are merged.

[0268] In the embodiment of the present disclosure, the window merging layer of the second local branch network can be used to fuse the features of each second sub-array to obtain fused features.

[0269] As an example, since there are overlapping elements (i.e., overlapping areas) between adjacent second sub-arrays, the features of adjacent first sub-arrays include features of the same elements. The features of each second sub-array can be spliced ​​based on the features of the overlapping elements between the second sub-arrays to obtain a fused feature.

[0270] As another example, similarity calculation is performed on the features of each second sub-array, and each first sub-array is fused according to the calculated similarity to obtain fused features.

[0271] Step 909: Input the fused features into the output layer of the second local branch network to obtain local features of the input video.

[0272] The local features are used to characterize at least one of the style and appearance of the input video.

[0273] In the embodiment of the present disclosure, the fused features may be input into the output layer of the second local branch network to obtain local features of the input video.

[0274] In one possible implementation of the disclosed embodiment, the output layer may use a piecewise linear function as the activation function. For example, the activation function used in the output layer may be a RELU function. Because the piecewise linear function does not require exponential calculations, it can reduce computational complexity, thereby improving video search efficiency.

[0275] In step 910 , a search model is used to fuse the semantic features and local features of the input video and the semantic features and local features of each video frame to obtain fused features of the input video.

[0276] In an embodiment of the present disclosure, a fusion network of a search model may be used to fuse the semantic features and local features of an input video, and the semantic features and local features of each video frame, to obtain fused features of the input video.

[0277] Step 911 : Determine correlation prediction information between the sample text and each sample video based on the features of the sample text and the fusion features of each sample video.

[0278] Step 912: Adjust the model parameters of the search model according to the difference between the relevance annotation information and the relevance prediction information.

[0279] The explanation of steps 909 to 912 can be found in the relevant description of any embodiment of the present disclosure and will not be repeated here.

[0280] The model training method of the disclosed embodiment uses a sliding window to partition the second array into subarrays, and then performs feature extraction on each subarray. This method can extract detailed features from the video, thereby improving the accuracy of local feature extraction. Furthermore, the sliding windows are overlapping or interleaved windows, which can avoid the loss of video information during the feature extraction process, thereby improving the robustness of local feature extraction.

[0281] In order to clearly illustrate how the search model is trained in any embodiment of the present disclosure, the present disclosure also proposes a model training method.

[0282] Figure 10 This is a flowchart of the model training method provided in Example 9 of the present disclosure.

[0283] like Figure 10 As shown, the model training method may include the following steps:

[0284] Step 1001: Obtain training samples, where the training samples include sample text and corresponding multiple sample videos.

[0285] The sample text is annotated with correlation annotation information between multiple sample videos and the sample text.

[0286] Step 1002: extract features from multiple sample videos using a search model to obtain local features and semantic features, and fuse the local features and semantic features to obtain fused features.

[0287] The local features are used to characterize at least one of the style and appearance of the corresponding sample video, and the semantic features are used to characterize the category to which the content of the corresponding sample video belongs.

[0288] Step 1003 : Determine correlation prediction information between the sample text and each sample video based on the features of the sample text and the fusion features of each sample video.

[0289] For explanations of steps 1001 to 1003 , please refer to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0290] Step 1004 : Determine the predicted order of the relevance between the plurality of sample videos and the sample texts based on the relevance prediction information.

[0291] In the disclosed embodiments, a predicted ranking of the relevance between multiple sample videos and sample texts can be determined based on the relevance prediction information. For example, when the relevance prediction information indicates the relevance score between the sample videos and the sample text, the sample videos can be sorted in descending order according to their corresponding relevance scores to obtain a predicted ranking.

[0292] Step 1005 : Determine the order of the annotations of the relevance between the plurality of sample videos and the sample texts according to the relevance annotation information.

[0293] In the disclosed embodiments, a ranking of the correlations between multiple sample videos and sample texts can be determined based on the correlation annotation information. For example, when the correlation annotation information indicates the correlation scores between the sample videos and the sample text, the sample videos can be sorted in descending order according to their corresponding correlation scores to obtain the ranking.

[0294] Step 1006 : Generate a first loss function based on the difference between the labeled ranking and the predicted ranking.

[0295] In an embodiment of the present disclosure, a first loss function may be generated based on the difference between the labeled ranking and the predicted ranking. For example, the first loss function may be a Smooth-AP (Smooth-Average Precision) loss function.

[0296] Step 1007: Adjust the model parameters of the search model according to the first loss function.

[0297] In the embodiment of the present disclosure, the model parameters in the search model may be adjusted according to the first loss function to minimize the value of the first loss function.

[0298] It should be noted that the above only uses the termination condition of the search model training as the minimization of the value of the first loss function as an example. In actual application, other termination conditions can also be set. For example, the termination condition can be that the number of training times reaches a set number, or the termination condition can be that the training time reaches a set time, etc. The present disclosure does not impose any restrictions on this.

[0299] The model training method of the disclosed embodiment trains the search model based on the difference between the annotated ranking and the predicted ranking of multiple sample videos. It is understood that when the correlation between the sample text and the sample video is higher, the sample video is ranked higher, and conversely, when the correlation between the sample text and the sample video is lower, the sample video is ranked lower. Training the search model based on the difference between the rankings allows the model to learn the correlation between the text and the video, thereby improving the training effect of the search model, that is, improving the prediction accuracy of the search model.

[0300] In order to clearly illustrate how the search model is trained in any embodiment of the present disclosure, the present disclosure also proposes a model training method.

[0301] Figure 11 This is a flowchart of the model training method provided in the tenth embodiment of the present disclosure.

[0302] like Figure 11 As shown, the model training method may include the following steps:

[0303] Step 1101: Obtain training samples, where the training samples include sample text and corresponding multiple sample videos.

[0304] The sample text is annotated with correlation annotation information between multiple sample videos and the sample text.

[0305] Step 1102 : extract features from multiple sample videos using a search model to obtain local features and semantic features, and fuse the local features and semantic features to obtain fused features.

[0306] The local features are used to characterize at least one of the style and appearance of the corresponding sample video, and the semantic features are used to characterize the category to which the content of the corresponding sample video belongs.

[0307] Step 1103 : Determine correlation prediction information between the sample text and each sample video based on the features of the sample text and the fusion features of each sample video.

[0308] For explanations of steps 1101 to 1103 , please refer to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0309] Step 1104 : Determine labeled relevant videos and labeled non-relevant videos from the plurality of sample videos according to the correlation labeling information.

[0310] In the embodiment of the present disclosure, it is possible to determine to label relevant videos and to label non-relevant videos from a plurality of sample videos according to the relevance labeling information.

[0311] As an example, when the relevance labeling information is used to indicate the relevance score between the sample video and the sample text, the sample video with a relevance score higher than the set score can be labeled as a related video, and the sample video with a relevance score not higher than the set score can be labeled as a non-relevant video.

[0312] As another example, when the relevance labeling information is used to indicate whether a sample video is relevant to a sample text, a sample video relevant to the sample text may be labeled as a relevant video, and a sample video irrelevant to the sample text may be labeled as an irrelevant video.

[0313] Step 1105 : Determine the relevance between the sample text and the annotated relevant videos, and the relevance between the sample text and the annotated non-relevant videos, based on the relevance prediction information.

[0314] In the disclosed embodiment, the relevance between the sample text and the annotated related video may be determined based on the relevance prediction information, and the relevance between the sample text and the annotated non-related video may be determined based on the relevance prediction information.

[0315] As an example, when the correlation prediction information is used to indicate the correlation score between the sample video and the sample text, the correlation between the sample text and the annotated related video is positively correlated with the correlation score between the sample text and the annotated related video (i.e., positively correlated), and the correlation between the sample text and the annotated non-related video is positively correlated with the correlation score between the sample text and the annotated non-related video.

[0316] Step 1106 : Determine a second loss function based on the relevance between the sample text and the annotated relevant video, and the relevance between the sample text and the annotated non-relevant video.

[0317] In the embodiment of the present disclosure, the second loss function may be determined based on the relevance between the sample text and the annotated relevant video, and the relevance between the sample text and the annotated non-relevant video. For example, the second loss function may be a triplet loss function.

[0318] As an example, the second loss function L2 can be as follows:

[0319] L2=max(0,m+d1-d2);

[0320] Where m is the set threshold, d1 is the correlation between the sample text and the annotated relevant videos, and d2 is the correlation between the sample text and the annotated irrelevant videos. The training goal is to narrow the distance between the sample text and the annotated relevant videos (i.e., positive examples) and expand the distance between the sample text and the annotated irrelevant videos (i.e., negative examples).

[0321] Step 1107: Adjust the model parameters of the search model according to the second loss function.

[0322] In the embodiment of the present disclosure, the model parameters in the search model may be adjusted according to the second loss function to minimize the value of the second loss function.

[0323] It should be noted that the above only uses the termination condition of the search model training as the minimization of the value of the second loss function as an example. In actual application, other termination conditions can also be set. For example, the termination condition can be that the number of training times reaches a set number, or the termination condition can be that the training time reaches a set time, etc. The present disclosure does not limit this.

[0324] It should also be noted that the first loss function and the second loss function can be combined to adjust the model parameters of the search model to improve the training effect of the search model, thereby improving the accuracy and reliability of the search model prediction results.

[0325] The model training method of the embodiment of the present disclosure determines a second loss function based on the correlation between the sample text and the annotated relevant videos, as well as the correlation between the sample text and the annotated non-relevant videos, and trains the search model based on the second loss function. This can shorten the distance between the sample text and the annotated relevant videos (i.e., positive examples), and increase the distance between the sample text and the annotated non-relevant videos (i.e., negative examples), so that the model learns the correlation between the text and the positive examples, as well as the non-correlation between the text and the negative examples, thereby improving the accuracy of the search model prediction results.

[0326] As an example, a dual-tower model can be used to encode text and video respectively, wherein a text encoder (such as a BERT (Bidirectional Encoder Representations from Transformers, a bidirectional encoding representation based on machine translation) text model) can be used to encode the text to obtain text features; when encoding the video, the video can first be framed, and a video frame encoder or an image encoder (such as a Swin-Transformer) can be used to perform image encoding on the extracted video frames to obtain semantic features of the video frames. At the same time, a video encoder (such as a video Swin-Transformer) can also be used to encode the video to obtain semantic features of the video. In addition, a local branch network can be added to the video frame encoder and the video encoder to extract shallow features (i.e., local features), and the shallow features and deep features (semantic features of the video frame and semantic features of the video) are fused to obtain fused features of the video, and the fused features are aligned with the text features to calculate the correlation between the text and the video.

[0327] Among them, the design of the local branch network is to have overlapping window attention calculation and spatial attention calculation. The structure of the local branch network can be as follows: Figure 4 shown.

[0328] As an example, Figure 12As shown, the search model can be trained by the following steps:

[0329] 1. Use the encoder in the search model to extract features from the text to obtain text features;

[0330] 2. Use the image encoder (i.e., image feature extraction network) in the search model to extract features from the video frames to obtain semantic features of the video frames;

[0331] 3. Use the first local branch network in the search model to extract features from the video frame to obtain local features of the video frame;

[0332] 4. Use the video encoder (i.e., video feature extraction network) in the search model to extract features from the video and obtain the semantic features of the video;

[0333] 5. Use the second local branch network in the search model to extract features from the video and obtain local features of the video;

[0334] 6. Use the fusion network (such as Fusion Transformer) in the search model to fuse the features in steps 2-5 to obtain the fusion features of the video;

[0335] 7. According to the correlation between the features in step 1 and step 6, the model parameters in the search model are adjusted using Smooth-AP loss Loss and tripletLoss.

[0336] In the present disclosure, Smooth-AP Loss and triplet loss can be used to alternately train the search model. For example, the search model can be trained with Smooth-AP Loss first, and then triplet loss can be used to train the search model after a certain period of training. Among them, triplet loss is a triplet loss, and the three elements are sample text, sample positive example (i.e., labeled relevant video) and sample negative example (i.e., labeled irrelevant video). The optimization goal is to shorten the distance between the sample text and the sample positive example, and to increase the distance between the sample text and the sample negative example. It is a metric loss; Smooth-AP Loss is a loss that directly optimizes the retrieval index AP, and is a ranking loss. The combination of the two loss functions is used to supervise model optimization, so that the trained model can be better applied in downstream retrieval tasks.

[0337] Optionally, the training set can be purified to further improve the training effect of the model. For example, all videos related to the same sample text can be searched for relevance using a preliminary visual feature to obtain at least one cluster. That is, all videos related to the same sample text can be clustered based on the preliminary visual feature to obtain at least one cluster. Afterwards, each cluster can be classified according to its size. Videos with higher relevance generally fall into the largest cluster, and the largest cluster can be used as a positive sample. The remaining small clusters can be screened by sampling, and the positive sample can be assigned to the largest cluster. At the same time, some negative sample examples can be selected. These negative sample examples are good difficult samples.

[0338] In summary, deep search is used to cluster visual features to extract training sets and simultaneously mine high-quality difficult samples. This allows the search model to be trained based on the extracted training sets and difficult samples to improve the model's training performance, thereby increasing its prediction accuracy. Joint training with triplet loss and smooth-AP loss can enhance the model's retrieval performance during cross-modal retrieval. In terms of model structure, a local branch network is designed for both the video frame encoder and the video encoder to extract local features such as the video's style and appearance. These local features are then integrated with the video's semantic features, further enhancing the video search or retrieval performance, and ultimately improving the accuracy of video retrieval results.

[0339] With the above Figures 1 to 5 Corresponding to the search method provided in the embodiment, the present disclosure also provides a search device. Since the search device provided in the embodiment of the present disclosure is consistent with the above Figures 1 to 5 The search method provided in the embodiment corresponds to the embodiment, so the implementation of the search method is also applicable to the search device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0340] Figure 13 This is a structural diagram of the search device provided in Example 11 of the present disclosure.

[0341] like Figure 13 As shown, the search device 1300 may include: a first acquisition module 1301 , a second acquisition module 1302 , a fusion module 1303 and a determination module 1304 .

[0342] The first acquisition module 1301 is used to acquire a search text and multiple candidate videos.

[0343] The second acquisition module 1302 is used to obtain local features and semantic features obtained by extracting features from multiple candidate videos respectively, wherein the local features are used to represent at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to represent the category to which the content body in the corresponding candidate video belongs.

[0344] The fusion module 1303 is used to fuse the local features and the semantic features to obtain fused features.

[0345] The determination module 1304 is configured to determine a target video matching the search text from among multiple candidate videos based on the correlation between the features of the search text and the fusion features of each candidate video.

[0346] In a possible implementation of the embodiment of the present disclosure, the second obtaining module 1302 may include:

[0347] The first input unit is used to take any one of the multiple candidate videos as an input video and input it into a video feature extraction network to obtain semantic features of the input video.

[0348] The second input unit is used to input at least one video frame in the input video into the image feature extraction network to obtain the semantic features of the at least one video frame in the input video.

[0349] The third input unit is used to input each video frame of the at least one video frame into the first local branch network to obtain local features of each video frame.

[0350] The fourth input unit is used to input the input video into the second local branch network to obtain local features of the input video.

[0351] In a possible implementation of the embodiment of the present disclosure, the third input unit is specifically used to: for any video frame, generate a first array according to the pixel values ​​of each pixel on multiple color channels; wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs; use a sliding window to divide the first array into multiple first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays; use the feature extraction layer of the first local branch network to extract features from each first sub-array respectively; use the window merging layer of the first local branch network to fuse the features of each first sub-array; and input the fused features into the output layer of the first local branch network to obtain local features of the video frame.

[0352] In a possible implementation of the embodiment of the present disclosure, the fourth input unit is specifically used to: generate a second array based on each video frame in the input video; wherein the elements in the second array are pixel values, and each element has a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension indicating the row position of the pixel point to which it belongs, and a third dimension indicating the column position of the pixel point to which it belongs; use a sliding window to divide the second array into multiple second sub-arrays, wherein there are overlapping elements between adjacent second sub-arrays; use the feature extraction layer of the second local branch network to extract features from each second sub-array respectively; use the window merging layer of the second local branch network to fuse the features of each second sub-array; and input the fused features into the output layer of the second local branch network to obtain local features of the input video.

[0353] In a possible implementation of the embodiment of the present disclosure, the feature extraction layer includes at least one encoding layer, and the encoding layer includes a multi-head self-attention module MSA and a feedforward network FFN.

[0354] In a possible implementation of the embodiment of the present disclosure, the output layer adopts a piecewise linear function as the activation function.

[0355] The search device of the embodiment of the present disclosure obtains local features and semantic features obtained by extracting features from multiple candidate videos respectively, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content body in the corresponding candidate video belongs; the local features and the semantic features are fused to obtain fused features. Therefore, in the scenario of searching for videos based on text, the target video that matches the search text is determined from each candidate video based on the semantic features (i.e., deep features) of each candidate video and the local features (i.e., shallow features) used to characterize the style and / or appearance of the video. This can improve the reliability of the determination result, thereby enabling the video search results to meet the user's actual search needs and improve the user's search experience.

[0356] With the above Figures 6 to 11 Corresponding to the model training method provided in the embodiment, the present disclosure also provides a model training device. Figures 6 to 11 The model training device provided by the embodiment will not be described in detail in the embodiments of the present disclosure.

[0357] Figure 14 This is a structural diagram of the model training device provided in Example 12 of the present disclosure.

[0358] like Figure 14As shown, the model training device 1400 may include: an acquisition module 1401, a processing module 1402, a determination module 1403 and an adjustment module 1404.

[0359] The acquisition module 1401 is used to acquire training samples, which include sample text and corresponding multiple sample videos, wherein the sample text is annotated with correlation annotation information between the multiple sample videos and the sample text.

[0360] The processing module 1402 is used to use a search model to extract features from multiple sample videos respectively to obtain local features and semantic features, and to fuse the local features and semantic features to obtain fused features; wherein the local features are used to characterize at least one of the style and appearance of the corresponding sample video, and the semantic features are used to characterize the category to which the content body in the corresponding sample video belongs.

[0361] The determination module 1403 is configured to determine correlation prediction information between the sample text and each sample video based on the features of the sample text and the fusion features of each sample video.

[0362] The adjustment module 1404 is configured to adjust model parameters of the search model according to the difference between the relevance annotation information and the relevance prediction information.

[0363] In a possible implementation of the embodiment of the present disclosure, the processing module 1402 may include:

[0364] The first input unit is used to input any one of the multiple sample videos as an input video into the video feature extraction network of the search model to obtain the semantic features of the input video.

[0365] The second input unit is used to input at least one video frame in the input video into the image feature extraction network of the search model to obtain the semantic features of the at least one video frame in the input video.

[0366] The third input unit is used to input each video frame of the at least one video frame into the first local branch network of the search model to obtain local features of each video frame.

[0367] The fourth input unit is used to input the input video into the second local branch network of the search model to obtain local features of the input video.

[0368] In a possible implementation of the embodiment of the present disclosure, the third input unit is specifically used to: for any video frame, generate a first array according to the pixel values ​​of each pixel on multiple color channels; wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs; use a sliding window to divide the first array into multiple first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays; use the feature extraction layer of the first local branch network to extract features from each first sub-array respectively; use the window merging layer of the first local branch network to fuse the features of each first sub-array; and input the fused features into the output layer of the first local branch network to obtain local features of the video frame.

[0369] In a possible implementation of the embodiment of the present disclosure, the fourth input unit is specifically used to: generate a second array based on each video frame in the input video; wherein the elements in the second array are pixel values, and each element has a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension indicating the row position of the pixel point to which it belongs, and a third dimension indicating the column position of the pixel point to which it belongs; use a sliding window to divide the second array into multiple second sub-arrays, wherein there are overlapping elements between adjacent second sub-arrays; use the feature extraction layer of the second local branch network to extract features from each second sub-array respectively; use the window merging layer of the second local branch network to fuse the features of each second sub-array; and input the fused features into the output layer of the second local branch network to obtain local features of the input video.

[0370] In a possible implementation of the embodiment of the present disclosure, the adjustment module 1404 is specifically used to: determine the predicted ranking of the correlation between multiple sample videos and sample texts based on the correlation prediction information; determine the labeled ranking of the correlation between multiple sample videos and sample texts based on the correlation labeling information; generate a first loss function based on the difference between the labeled ranking and the predicted ranking; and adjust the model parameters of the search model based on the first loss function.

[0371] In a possible implementation of the embodiment of the present disclosure, the adjustment module is specifically used to: determine the labeled relevant videos and the labeled non-relevant videos from multiple sample videos based on the correlation labeling information; determine the correlation between the sample text and the labeled relevant videos, as well as the correlation between the sample text and the labeled non-relevant videos based on the correlation prediction information; determine a second loss function based on the correlation between the sample text and the labeled relevant videos, as well as the correlation between the sample text and the labeled non-relevant videos; and adjust the model parameters of the search model based on the second loss function.

[0372] The search device of the embodiment of the present disclosure uses a search model to extract features from multiple sample videos in the training sample respectively to obtain local features and semantic features, and fuses the local features and semantic features to obtain fused features; determines the correlation prediction information between the sample text and each sample video based on the features of the sample text in the training sample and the fused features of each sample video; and adjusts the model parameters of the search model based on the difference between the correlation annotation information and the correlation prediction information annotated by the sample text. Thus, the use of deep learning technology to extract video features can improve the accuracy and reliability of feature extraction. In addition, based on the semantic features of the video (i.e., deep features) and the local features (shallow features) used to characterize the style and / or appearance of the video, the correlation prediction information between the video and the text is determined, which can improve the reliability of the correlation prediction information determination, thereby training the search model based on the reliable correlation prediction information, which can improve the training efficiency of the model.

[0373] In order to implement the above embodiments, the present disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the search method or model training method proposed in any of the above embodiments of the present disclosure.

[0374] In order to implement the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the search method or model training method proposed in any of the above embodiments of the present disclosure.

[0375] In order to implement the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the search method or model training method proposed in any of the above embodiments of the present disclosure.

[0376] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0377] Figure 15A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0378] like Figure 15 As shown, the electronic device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1502 or a computer program loaded from a storage unit 1508 into a RAM (Random Access Memory) 1503. Various programs and data required for the operation of the electronic device 1500 can also be stored in the RAM 1503. The computing unit 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An I / O (Input / Output) interface 1505 is also connected to the bus 1504.

[0379] Multiple components in electronic device 1500 are connected to I / O interface 1505, including: an input unit 1506, such as a keyboard, mouse, etc.; an output unit 1507, such as various types of displays, speakers, etc.; a storage unit 1508, such as a magnetic disk, optical disk, etc.; and a communication unit 1509, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1509 allows electronic device 1500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0380] The computing unit 1501 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1501 performs the various methods and processes described above, such as the above-described search method or model training method. For example, in some embodiments, the above-described search method or model training method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 1508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1500 via the ROM 1502 and / or the communication unit 1509. When the computer program is loaded into the RAM 1503 and executed by the computing unit 1501, one or more steps of the search method or model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured to perform the above-mentioned search method or model training method in any other appropriate manner (for example, by means of firmware).

[0381] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System on Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0382] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0383] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0384] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0385] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0386] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and poor scalability of traditional physical hosts and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0387] It's important to note that artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). This encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0388] According to the technical solution of the embodiment of the present disclosure, local features and semantic features are obtained by extracting features from multiple candidate videos respectively, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content body in the corresponding candidate video belongs; the local features and semantic features are fused to obtain fused features. Thus, in the scenario of searching for videos based on text, the target video that matches the search text is determined from each candidate video based on the semantic features (i.e., deep features) of each candidate video and the local features (i.e., shallow features) used to characterize the style and / or appearance of the video. This can improve the reliability of the determination result, thereby enabling the video search results to meet the user's actual search needs and improve the user's search experience.

[0389] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions proposed in this disclosure can be achieved. This is not limited herein.

[0390] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A search method, comprising: Get the search text and multiple candidate videos; Taking any one of the multiple candidate videos as an input video and inputting it into a video feature extraction network to obtain semantic features of the input video; Inputting at least one video frame in the input video into an image feature extraction network to obtain semantic features of the at least one video frame in the input video; Inputting each video frame of the at least one video frame into a first local branch network to obtain local features of each video frame; Inputting the input video into a second local branch network to obtain local features of the input video, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content body in the corresponding candidate video belongs; fusing the semantic features and local features of the input video with the semantic features and local features of the video frame to obtain a fused feature; According to the correlation between the features of the search text and the fusion features of each of the candidate videos, a target video matching the search text is determined from the multiple candidate videos.

2. The method according to claim 1, wherein Inputting each video frame of the at least one video frame into a first local branch network to obtain local features of each video frame includes: For any of the video frames, a first array is generated based on the pixel values ​​of each pixel on multiple color channels; wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs; Using a sliding window, the first array is divided into a plurality of first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays; Using the feature extraction layer of the first local branch network to perform feature extraction on each of the first sub-arrays; Using the window merging layer of the first local branch network, the features of each of the first sub-arrays are merged; The fused features are input into the output layer of the first local branch network to obtain the local features of the video frame.

3. The method according to claim 1, wherein Inputting the input video into the second local branch network to obtain local features of the input video includes: generating a second array based on each video frame in the input video, wherein the elements in the second array are pixel values, each of the elements having a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension for indicating the row position of the pixel, and a third dimension for indicating the column position of the pixel; Dividing the second array into a plurality of second sub-arrays using a sliding window, wherein there are overlapping elements between adjacent second sub-arrays; Using the feature extraction layer of the second local branch network to perform feature extraction on each of the second sub-arrays; Using the window merging layer of the second local branch network, the features of each of the second sub-arrays are merged; The fused features are input into the output layer of the second local branch network to obtain local features of the input video.

4. The method according to claim 2 or 3, wherein: The feature extraction layer includes at least one encoding layer, and the encoding layer includes a multi-head self-attention module MSA and a feedforward network FFN.

5. The method according to claim 2 or 3, wherein: The output layer uses a piecewise linear function as the activation function.

6. A model training method comprising: Acquire a training sample, wherein the training sample includes a sample text and a corresponding plurality of sample videos, wherein the sample text is annotated with correlation annotation information between the plurality of sample videos and the sample text; Taking any one of the plurality of sample videos as an input video and inputting it into a video feature extraction network of a search model to obtain semantic features of the input video; Inputting at least one video frame in the input video into the image feature extraction network of the search model to obtain semantic features of the at least one video frame in the input video; Inputting each video frame of the at least one video frame into the first local branch network of the search model to obtain local features of each video frame; Inputting the input video into the second local branch network of the search model to obtain local features of the input video, and fusing the semantic features and local features of the input video, and the semantic features and local features of the video frame, to obtain fused features; wherein the local features are used to characterize at least one of the style and appearance of the corresponding sample video, and the semantic features are used to characterize the category to which the content of the corresponding sample video belongs; Determining correlation prediction information between the sample text and each of the sample videos based on the features of the sample text and the fusion features of each of the sample videos; According to the difference between the correlation annotation information and the correlation prediction information, the model parameters of the search model are adjusted.

7. The method according to claim 6, wherein: Inputting each video frame of the at least one video frame into a first local branch network to obtain local features of each video frame includes: For any of the video frames, a first array is generated based on the pixel values ​​of each pixel on multiple color channels; wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs; Using a sliding window, the first array is divided into a plurality of first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays; Using the feature extraction layer of the first local branch network to perform feature extraction on each of the first sub-arrays; Using the window merging layer of the first local branch network, the features of each of the first sub-arrays are merged; The fused features are input into the output layer of the first local branch network to obtain the local features of the video frame.

8. The method according to claim 6, wherein: Inputting the input video into the second local branch network to obtain local features of the input video includes: generating a second array based on each video frame in the input video, wherein the elements in the second array are pixel values, each of the elements having a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension for indicating the row position of the pixel, and a third dimension for indicating the column position of the pixel; Dividing the second array into a plurality of second sub-arrays using a sliding window, wherein there are overlapping elements between adjacent second sub-arrays; Using the feature extraction layer of the second local branch network to perform feature extraction on each of the second sub-arrays; Using the window merging layer of the second local branch network, the features of each of the second sub-arrays are merged; The fused features are input into the output layer of the second local branch network to obtain local features of the input video.

9. The method according to any one of claims 6 to 8, wherein The adjusting the model parameters of the search model according to the difference between the correlation annotation information and the correlation prediction information includes: Determining a predicted ranking of the relevance between the plurality of sample videos and the sample texts based on the relevance prediction information; Determining, based on the correlation annotation information, an annotation ranking of the correlations between the plurality of sample videos and the sample texts; generating a first loss function according to the difference between the labeled ranking and the predicted ranking; Adjust model parameters of the search model according to the first loss function.

10. The method according to any one of claims 6 to 8, wherein: The adjusting the model parameters of the search model according to the difference between the correlation annotation information and the correlation prediction information includes: Determining, from the plurality of sample videos, to label relevant videos and to label non-relevant videos according to the relevance labeling information; Determining, based on the relevance prediction information, the relevance between the sample text and the annotated relevant video, and the relevance between the sample text and the annotated non-relevant video; Determining a second loss function based on the relevance between the sample text and the annotated relevant video, and the relevance between the sample text and the annotated non-relevant video; Adjust model parameters of the search model according to the second loss function.

11. A search device comprising: A first acquisition module is used to acquire a search text and multiple candidate videos; A second acquisition module is configured to acquire local features and semantic features obtained by extracting features from the plurality of candidate videos, wherein the local features are used to characterize at least one of the style and appearance of the corresponding candidate video, and the semantic features are used to characterize the category to which the content of the corresponding candidate video belongs; A fusion module, configured to fuse the local features and the semantic features to obtain fused features; a determination module, configured to determine a target video matching the search text from the plurality of candidate videos based on a correlation between a feature of the search text and a fusion feature of each candidate video; The second acquisition module includes: A first input unit is configured to take any one of the plurality of candidate videos as an input video and input the video into a video feature extraction network to obtain semantic features of the input video; A second input unit is configured to input at least one video frame in the input video into an image feature extraction network to obtain a semantic feature of at least one video frame in the input video; a third input unit, configured to input each video frame of the at least one video frame into the first local branch network, so as to obtain local features of each video frame; a fourth input unit, configured to input the input video into a second local branch network to obtain local features of the input video; The fusion module is specifically used to fuse the semantic features and local features of the input video and the semantic features and local features of the video frame to obtain fused features.

12. The device according to claim 11, wherein The third input unit is specifically used to: For any of the video frames, a first array is generated based on the pixel values ​​of each pixel on multiple color channels; wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs; Using a sliding window, the first array is divided into a plurality of first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays; Using the feature extraction layer of the first local branch network to perform feature extraction on each of the first sub-arrays; Using the window merging layer of the first local branch network, the features of each of the first sub-arrays are merged; The fused features are input into the output layer of the first local branch network to obtain the local features of the video frame.

13. The device according to claim 11, wherein The fourth input unit is specifically used to: generating a second array based on each video frame in the input video, wherein the elements in the second array are pixel values, each of the elements having a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension for indicating the row position of the pixel, and a third dimension for indicating the column position of the pixel; Dividing the second array into a plurality of second sub-arrays using a sliding window, wherein there are overlapping elements between adjacent second sub-arrays; Using the feature extraction layer of the second local branch network to perform feature extraction on each of the second sub-arrays; Using the window merging layer of the second local branch network, the features of each of the second sub-arrays are merged; The fused features are input into the output layer of the second local branch network to obtain local features of the input video.

14. The device according to claim 12 or 13, wherein The feature extraction layer includes at least one encoding layer, and the encoding layer includes a multi-head self-attention module MSA and a feedforward network FFN.

15. The device according to claim 12 or 13, wherein The output layer uses a piecewise linear function as the activation function.

16. A model training device comprising: An acquisition module, configured to acquire a training sample, wherein the training sample includes a sample text and a corresponding plurality of sample videos, wherein the sample text is annotated with correlation annotation information between the plurality of sample videos and the sample text; a processing module configured to extract features from each of the plurality of sample videos using a search model to obtain local features and semantic features, and to fuse the local features and the semantic features to obtain fused features; wherein the local features are used to characterize at least one of the style and appearance of the corresponding sample videos, and the semantic features are used to characterize the category to which the main content of the corresponding sample videos belongs; a determination module, configured to determine correlation prediction information between the sample text and each of the sample videos based on the features of the sample text and the fusion features of each of the sample videos; an adjustment module, configured to adjust model parameters of the search model according to a difference between the correlation annotation information and the correlation prediction information; The processing module includes: A first input unit is configured to input any one of the plurality of sample videos as an input video into a video feature extraction network of the search model to obtain semantic features of the input video; A second input unit is configured to input at least one video frame in the input video into an image feature extraction network of the search model to obtain semantic features of at least one video frame in the input video; a third input unit, configured to input each video frame of the at least one video frame into the first local branch network of the search model, so as to obtain local features of each video frame; a fourth input unit, configured to input the input video into the second local branch network of the search model to obtain local features of the input video; The processing module is further configured to fuse the semantic features and local features of the input video and the semantic features and local features of the video frame to obtain the fused features.

17. The device according to claim 16, wherein The third input unit is specifically used to: For any of the video frames, a first array is generated based on the pixel values ​​of each pixel on multiple color channels; wherein the elements in the first array are pixel values, and each element has a first dimension indicating the color channel to which it belongs, a second dimension indicating the row position of the pixel to which it belongs, and a third dimension indicating the column position of the pixel to which it belongs; Using a sliding window, the first array is divided into a plurality of first sub-arrays, wherein there are overlapping elements between adjacent first sub-arrays; Using the feature extraction layer of the first local branch network to perform feature extraction on each of the first sub-arrays; Using the window merging layer of the first local branch network, the features of each of the first sub-arrays are merged; The fused features are input into the output layer of the first local branch network to obtain the local features of the video frame.

18. The device according to claim 16, wherein The fourth input unit is specifically used to: generating a second array based on each video frame in the input video, wherein the elements in the second array are pixel values, each of the elements having a first dimension for indicating the video frame and color channel to which the pixel value belongs, a second dimension for indicating the row position of the pixel, and a third dimension for indicating the column position of the pixel; Dividing the second array into a plurality of second sub-arrays using a sliding window, wherein there are overlapping elements between adjacent second sub-arrays; Using the feature extraction layer of the second local branch network to perform feature extraction on each of the second sub-arrays; Using the window merging layer of the second local branch network, the features of each of the second sub-arrays are merged; The fused features are input into the output layer of the second local branch network to obtain local features of the input video.

19. The device according to any one of claims 16 to 18, wherein: The adjustment module is specifically used to: Determining a predicted ranking of the relevance between the plurality of sample videos and the sample texts based on the relevance prediction information; Determining, based on the correlation annotation information, an annotation ranking of the correlations between the plurality of sample videos and the sample texts; generating a first loss function according to the difference between the labeled ranking and the predicted ranking; Adjust model parameters of the search model according to the first loss function.

20. The device according to any one of claims 16 to 18, wherein The adjustment module is specifically used to: Determining, from the plurality of sample videos, to label relevant videos and to label non-relevant videos according to the relevance labeling information; Determining, based on the relevance prediction information, the relevance between the sample text and the annotated relevant video, and the relevance between the sample text and the annotated non-relevant video; Determining a second loss function based on the relevance between the sample text and the annotated relevant video, and the relevance between the sample text and the annotated non-relevant video; Adjust model parameters of the search model according to the second loss function.

21. An electronic device, wherein: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 5, or execute the method of any one of claims 6 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 5, or to execute the method according to any one of claims 6 to 10.

23. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 10.

Images