Short video search method and device based on multi-modal and storage medium

Abstract

The short video search method and device based on multiple modes and the storage medium, the method transports the short video object to the multi-modal feature extraction model to extract the multi-modal information feature; the key information of the short video object is extracted, the key video segment of the short video object and the key audio information corresponding to the key video segment are obtained; the key video segment is transported to the first convolutional neural network to extract the video feature, and the image content feature of the key video segment is obtained; the caption position of the key video segment extracts caption text information, the caption text information is first encoded, and the text content feature of the key video segment is obtained; the voice information of the key video segment is transported to the second convolutional neural network for second encoding, and the voice content feature of the key video segment is obtained; the short video reaching the preset similarity threshold is taken as the search result of the short video object and returned to the user. The short video search accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of short video search, and specifically relates to a short video search method, device and storage medium based on multimodality. Background Art

[0002] Short videos are frequently pushed onto various new media platforms, suitable for viewing on the go and in short, casual moments. These videos range from a few seconds to several minutes in length. These videos incorporate topics such as skill sharing, humor and quirkiness, fashion trends, social issues, street interviews, public welfare education, advertising creativity, and commercial customization. Due to their short duration, these videos can be presented as standalone videos or as part of a series.

[0003] At present, there are some search and recommendation schemes for short videos, such as the short video search recommendation method based on voice recognition disclosed in CN113204669A, which imports the voice information corresponding to the short video to be searched by the user into the platform; converts the voice information input by the user into a text format recognizable by the platform; sends the search mode selection instruction to the search interface corresponding to the user to obtain the search mode selected by the user; processes the voice text information corresponding to the user; obtains the information corresponding to each short video on the platform; and matches and filters the processed voice text information with the information corresponding to the short video.

[0004] Another example is the short video search, matching and recommendation method based on deep comparison and analysis of multi-dimensional data disclosed in CN113204709A. The method divides the imported short video into video images, counts the area proportion of each category attribute image in each video image in the short video, analyzes the type attribute image corresponding to each video image in the short video, and counts the number of video images of each category attribute image in the short video, calculates the type attribute image weight ratio coefficient of the short video, and simultaneously obtains the text information in the video voice corresponding to each video image in the short video, obtains the valid words in the video text information of each segment in the short video, compares and analyzes the word frequency of each keyword in the short video, calculates the keyword weight ratio coefficient of the short video, and obtains the video playback time corresponding to each keyword in the short video, calculates the comprehensive matching degree compliance coefficient of the short video, and compares and filters other short videos with the highest matching degree compliance coefficient for recommendation.

[0005] In the existing technology, whether it is searching through voice recognition or converting short videos into pictures and searching for similar videos based on the content of the pictures, only the single information of the short video is used, and the search matching results are not ideal. Many search or recommendation results are not what the user wants, and the search accuracy is low. Summary of the Invention

[0006] To this end, the present invention provides a short video search method, device and storage medium based on multimodality to solve the problem of low accuracy caused by traditional solutions that search and recommend short videos based on single information.

[0007] In order to achieve the above objectives, the present invention provides the following technical solutions: In a first aspect, a short video search method based on multimodality is provided, comprising the following steps:

[0008] (1) obtaining a short video object to be searched, transmitting the short video object to a multimodal feature extraction model, and performing multimodal information feature extraction on the short video object through the multimodal feature extraction model;

[0009] (2) The multimodal information feature extraction of the short video object includes:

[0010] (21) extracting key information from the short video object to obtain a key video segment of the short video object and key audio information corresponding to the key video segment;

[0011] (22) delivering the key video segment to a first convolutional neural network to extract video features to obtain image content features of the key video segment;

[0012] (23) extracting subtitle text information from the subtitle position of the key video segment, performing a first encoding on the subtitle text information, and obtaining text content features of the key video segment;

[0013] (24) transmitting the voice information of the key video segment to a second convolutional neural network for second encoding to obtain the voice content features of the key video segment;

[0014] (25) using the image content feature, the text content feature, and the voice content feature as multimodal information features of the key video segment;

[0015] (3) performing similarity calculation on the multimodal information features of the key video clip and the multimodal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database;

[0016] (4) The short videos that reach the preset similarity threshold are returned to the user as the search results of the short video object.

[0017] As a preferred solution of the short video search method based on multimodality, in the process of obtaining the key video segments in step (21), the key video segments are divided into sequences according to the number of key frames obtained and the preset number of groups;

[0018] In step (22), the first convolutional neural network extracts image content features based on the sequence division results and weight coefficients of the key video clips.

[0019] As a preferred solution of the short video search method based on multimodality, in step (23), the subtitle text information is first encoded through the Transformer model, and the final vector is obtained through the self-attention mechanism using the query vector, the value vector, and the weight matrix. The subtitle text information is first encoded based on the final vector and the hidden layer vector.

[0020] As a preferred solution of the multimodal short video search method, in step (24), the second convolutional neural network extracts speech content features based on the sequence division results and weight coefficients of the key video clips.

[0021] As a preferred solution of the multimodal short video search method, in step (3), a vector dot product is performed on the multimodal information features of the short video object and the multimodal information features of a given short video in the short video database, and then the similarity between the short video object and the given short video in the short video database is obtained based on the modal number of the multimodal information features.

[0022] As a preferred solution of the multimodal short video search method, according to step (2), the multimodal information features of the existing short videos are pre-extracted and stored in the short video database.

[0023] In a second aspect, the present invention provides a short video search device based on multimodality, comprising:

[0024] A short video acquisition module is used to acquire the short video object to be searched and transmit the short video object to the multimodal feature extraction model;

[0025] A multimodal feature extraction module, configured to extract multimodal information features from the short video object using the multimodal feature extraction model;

[0026] The multimodal feature extraction module includes:

[0027] A key information extraction submodule is used to extract key information from the short video object to obtain key video segments of the short video object and key audio information corresponding to the key video segments;

[0028] An image content feature extraction submodule, configured to transmit the key video segment to a first convolutional neural network for extracting video features to obtain image content features of the key video segment;

[0029] a text content feature extraction submodule, configured to extract subtitle text information from the subtitle position of the key video segment, perform a first encoding on the subtitle text information, and obtain text content features of the key video segment;

[0030] a speech content feature extraction submodule, configured to transmit the speech information of the key video segment to a second convolutional neural network for second encoding to obtain speech content features of the key video segment;

[0031] a multimodal information feature combination submodule, configured to use the image content feature, the text content feature, and the voice content feature as the multimodal information feature of the key video segment;

[0032] A similarity statistics module is used to calculate the similarity between the multimodal information features of the key video clip and the multimodal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database;

[0033] The search result returning module is used to return the short videos that meet the preset similarity threshold as the search results of the short video object to the user.

[0034] As a preferred solution of the short video search device based on multimodality, the key information extraction submodule divides the key video segments into sequences according to the number of key frames obtained and the preset number of groups during the acquisition of the key video segments;

[0035] In the image content feature extraction submodule, the first convolutional neural network extracts image content features based on the sequence division results and weight coefficients of the key video clips.

[0036] As a preferred solution of the short video search device based on multimodality, in the text content feature extraction submodule, the subtitle text information is first encoded through the Transformer model, and the query vector, value vector, and weight matrix are used to obtain the final vector through the self-attention mechanism. The subtitle text information is first encoded based on the final vector and the hidden layer vector;

[0037] In the speech content feature extraction submodule, the second convolutional neural network extracts speech content features based on the sequence division results and weight coefficients of the key video clips.

[0038] As a preferred solution of the short video search device based on multimodality, in the similarity statistics module, a vector dot product is performed on the multimodal information features of the short video object and the multimodal information features of a given short video in the short video database, and then the similarity between the short video object and the given short video in the short video database is obtained according to the modal number of the multimodal information features;

[0039] For existing short videos, multimodal information features of the existing short videos are pre-extracted and stored in the short video database according to the key information extraction submodule, image content feature extraction submodule, text content feature extraction submodule, voice content feature extraction submodule and multimodal information feature combination submodule in the multimodal feature extraction module.

[0040] In a third aspect, an electronic device is provided, comprising: a memory and a processor; the processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor; the processor calls the program instructions to execute the multimodal short video search method of the first aspect or any possible implementation thereof.

[0041] In a fourth aspect, a storage medium is provided, in which a program code of a multimodal short video search method is stored, and the program code includes instructions for executing the multimodal short video search method of the first aspect or any possible implementation thereof.

[0042] The present invention has the following advantages: by obtaining a short video object to be searched, the short video object is transmitted to a multimodal feature extraction model, and multimodal information feature extraction is performed on the short video object through the multimodal feature extraction model; the multimodal information feature extraction of the short video object includes: extracting key information of the short video object to obtain a key video segment of the short video object and key audio information corresponding to the key video segment; transmitting the key video segment to a first convolutional neural network to extract video features to obtain image content features of the key video segment; extracting subtitle text information from the subtitle position of the key video segment, performing a first encoding on the subtitle text information to obtain text content features of the key video segment; transmitting the voice information of the key video segment to a second convolutional neural network for second encoding to obtain voice content features of the key video segment; using image content features, text content features, and voice content features as multimodal information features of the key video segment; performing similarity calculation on the multimodal information features of the key video segment and the multimodal information features in a short video database to obtain the similarity between the short video object and each short video in the short video database; and returning short videos that meet a preset similarity threshold as search results of the short video object to the user. The present invention adopts multimodal information extraction technology, integrates the image, subtitle and voice information of short videos, improves the accuracy of short video search, and enables users to obtain more ideal short video search or recommendation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0044] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0045] Figure 1 A schematic diagram of the process flow of a multimodal short video search method provided in Example 1 of the present invention;

[0046] Figure 2 Schematic diagram of a multimodal short video search device provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0047] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0048] Example 1

[0049] See also Figure 1 , embodiment 1 of the present invention provides a short video search method based on multimodality, comprising the following steps:

[0050] S1. Obtain a short video object to be searched, transmit the short video object to a multimodal feature extraction model, and perform multimodal information feature extraction on the short video object through the multimodal feature extraction model;

[0051] S2. Extracting multimodal information features of the short video object includes:

[0052] S21, extracting key information from the short video object to obtain a key video segment of the short video object and key audio information corresponding to the key video segment;

[0053] S22: delivering the key video clip to a first convolutional neural network to extract video features to obtain image content features of the key video clip;

[0054] S23, extracting subtitle text information from the subtitle position of the key video segment, performing a first encoding on the subtitle text information, and obtaining text content features of the key video segment;

[0055] S24: transmitting the voice information of the key video segment to a second convolutional neural network for second encoding to obtain voice content features of the key video segment;

[0056] S25, using the image content feature, the text content feature, and the voice content feature as multimodal information features of the key video segment;

[0057] S3, calculating the similarity between the multimodal information features of the key video clip and the multimodal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database;

[0058] S4. The short videos that reach a preset similarity threshold are returned to the user as search results of the short video object.

[0059] In this embodiment, in step S21, during the acquisition of the key video clips, the key video clips are sequenced according to the number of key frames obtained and the preset number of groups; in step S22, the first convolutional neural network extracts image content features based on the sequence division results and weight coefficients of the key video clips.

[0060] Specifically, the key video clips are fed into the first three-dimensional convolutional neural network to extract the features of the short video objects and obtain the image content features of the short video objects. Assuming the number of key frames obtained is N, and the short video objects are divided into M groups, then A sequence.

[0061] The first convolutional neural network in three dimensions is represented as F img =Conv 3D (W img ,seq), where W img is the weight coefficient, F img represents the extracted image content features, F img It is a (1×dim) vector, where dim represents the vector size.

[0062] In this embodiment, in step S23, the subtitle text information is first encoded through the Transformer model, and the final vector is obtained through the self-attention mechanism using the query vector, the value vector, and the weight matrix. The subtitle text information is first encoded based on the final vector and the hidden layer vector.

[0063] Specifically, the Transformer model itself is an existing technology, first proposed by Google. It uses a self-attention structure to replace the RNN network structure commonly used in NLP tasks. Essentially, it is an encoder-decoder architecture, with the central Transformer component being divided into two parts: the encoding component and the decoding component.

[0064] During the encoding process, let y=(y1,y2…y n ) represents n characters. q=(q1,q2…q n ) indicates that it is through q=Emb q (y) Embedding is performed to obtain the query vector. Then:

[0065] k=(k1,k2…k n )=W k ×q

[0066] v=(v1,v2…v n )=W v ×q

[0067] k, v represent the query vector and value vector respectively, W k ,W v Represent the weight matrix respectively. Then the final vector is obtained through the self-attention mechanism where k T represents the transpose of k. The final text code is F wor =trans(V,hid), where hid represents the hidden layer vector, F wor It is a (1×dim) vector, where dim represents the vector size.

[0068] In this embodiment, in step S3, the second convolutional neural network extracts speech content features based on the sequence division results and weight coefficients of the key video segments. The process of obtaining speech content features is similar to the process of obtaining image content features.

[0069] The speech of the key video clips is fed into the second three-dimensional convolutional neural network to extract the speech features of the short video objects and obtain the speech content features of the short video objects. Assuming the number of key frames obtained is N, and the short video objects are divided into M groups, then A sequence.

[0070] The second convolutional neural network in three dimensions is represented as F vid =Conv 3D (W vid ,seq), where W vid is the weight coefficient, F vid represents the extracted speech content features, F vid It is a (1×dim) vector, where dim represents the vector size.

[0071] In this embodiment, in step S3, a vector dot product is performed on the multimodal information features of the short video object and the multimodal information features of a given short video in the short video database. Then, the similarity between the short video object and the given short video in the short video database is obtained based on the modal number of the multimodal information features. For existing short videos, the multimodal information features of the existing short videos are pre-extracted according to step S2 and stored in the short video database.

[0072] Specifically, the image content features, text content features, and voice content features obtained are used as the final multimodal information features of the short video object. The multimodal information features are expressed as F all =(F img ,F wor ,F vid ).

[0073] Then, the obtained multi-modal information features are compared with the multi-modal information features in the short video database to calculate the similarity between the short video object and each short video in the short video database. The similarity calculation is as follows:

[0074]

[0075] The “·” symbol represents the dot product of vectors. is the multimodal information of a short video in the short video database, Indicates the multimodal information of the short video object that needs to be searched.

[0076] By setting the similarity threshold, the first few short videos with the highest scores are returned to the user as the final search results. Assuming that there are Num videos in the short video database, the final Score is all =(s1,s2,s3…s num ), the final return result is Top k (Score all ), where Top k It means taking the k with the highest scores.

[0077] In this embodiment, during the establishment of the short video database, the existing short videos are subjected to the multimodal feature extraction process of step S2 to extract multimodal information, and the multimodal information is saved. The database is represented as:

[0078]

[0079] in, Indicates the multimodal information of the num-th video, and num obtained in step S25 indicates the total number of short videos.

[0080] In summary, the present invention obtains a short video object to be searched, transmits the short video object to a multimodal feature extraction model, and extracts multimodal information features of the short video object through the multimodal feature extraction model; the multimodal information feature extraction of the short video object includes: extracting key information of the short video object to obtain a key video clip of the short video object and key audio information corresponding to the key video clip; transmitting the key video clip to a first convolutional neural network to extract video features to obtain image content features of the key video clip; extracting subtitle text information from the subtitle position of the key video clip, performing a first encoding on the subtitle text information to obtain text content features of the key video clip; transmitting the voice information of the key video clip to a second convolutional neural network for a second encoding to obtain voice content features of the key video clip; using image content features, text content features, and voice content features as multimodal information features of the key video clip; performing similarity calculation on the multimodal information features of the key video clip and the multimodal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database; and returning short videos that meet a preset similarity threshold as search results of the short video object to the user. The present invention adopts multimodal information extraction technology, integrates the image, subtitle and voice information of short videos, improves the accuracy of short video search, and enables users to obtain more ideal short video search or recommendation results.

[0081] Example 2

[0082] See also Figure 2 , Embodiment 2 of the present invention further provides a short video search device based on multimodality, comprising:

[0083] The short video acquisition module 1 is used to acquire the short video object to be searched and transmit the short video object to the multimodal feature extraction model;

[0084] A multimodal feature extraction module 2, configured to extract multimodal information features from the short video object using the multimodal feature extraction model;

[0085] The multimodal feature extraction module 2 includes:

[0086] A key information extraction submodule 21 is configured to extract key information from the short video object to obtain key video segments of the short video object and key audio information corresponding to the key video segments;

[0087] An image content feature extraction submodule 22 is configured to transmit the key video segment to a first convolutional neural network for extracting video features to obtain image content features of the key video segment;

[0088] The text content feature extraction submodule 23 is configured to extract subtitle text information from the subtitle position of the key video segment, perform a first encoding on the subtitle text information, and obtain text content features of the key video segment;

[0089] A speech content feature extraction submodule 24 is configured to transmit the speech information of the key video segment to a second convolutional neural network for second encoding to obtain speech content features of the key video segment;

[0090] a multimodal information feature combination submodule 25 for using the image content feature, the text content feature, and the voice content feature as the multimodal information feature of the key video segment;

[0091] A similarity statistics module 3 is used to calculate the similarity between the multimodal information features of the key video clip and the multimodal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database;

[0092] The search result returning module 4 is configured to return the short videos that have reached a preset similarity threshold as the search results of the short video object to the user.

[0093] In this embodiment, during the acquisition of the key video segments by the key information extraction submodule 21, the key video segments are divided into sequences according to the number of key frames obtained and the preset number of groups;

[0094] In the image content feature extraction submodule 22, the first convolutional neural network extracts image content features according to the sequence division results and weight coefficients of the key video clips.

[0095] In this embodiment, the text content feature extraction submodule 23 performs a first encoding on the subtitle text information through a Transformer model, obtains a final vector using the query vector, the value vector, and the weight matrix through a self-attention mechanism, and performs a first encoding on the subtitle text information based on the final vector and the hidden layer vector.

[0096] In the speech content feature extraction submodule 24, the second convolutional neural network extracts speech content features according to the sequence division results and weight coefficients of the key video segments.

[0097] In this embodiment, in the similarity statistics module 3, a vector dot product is performed on the multimodal information features of the short video object and the multimodal information features of a given short video in the short video database, and then the similarity between the short video object and the given short video in the short video database is obtained according to the modal number of the multimodal information features;

[0098] In this embodiment, the multimodal information features of the existing short videos are pre-extracted and stored in the short video database according to the key information extraction submodule 21, image content feature extraction submodule 22, text content feature extraction submodule 23, voice content feature extraction submodule 24 and multimodal information feature combination submodule 25 in the multimodal feature extraction module 2.

[0099] It should be noted that the information interaction, execution process, etc. between the modules / sub-modules of the above-mentioned device are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and no further details will be given here.

[0100] Example 3

[0101] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code for a multimodal short video search method is stored. The program code includes instructions for executing the multimodal short video search method of embodiment 1 or any possible implementation thereof.

[0102] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0103] Example 4

[0104] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0105] The processor and the memory communicate with each other through a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the multimodal short video search method of Example 1 or any possible implementation thereof.

[0106] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in a memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0107] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0108] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0109] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A short video search method based on multimodality, characterized in that: The following steps are involved: (1) obtaining a short video object to be searched, transmitting the short video object to a multimodal feature extraction model, and performing multimodal information feature extraction on the short video object through the multimodal feature extraction model; (2) The multimodal information feature extraction of the short video object includes: (21) extracting key information from the short video object to obtain a key video segment of the short video object and key audio information corresponding to the key video segment; (22) delivering the key video segment to a first convolutional neural network to extract video features to obtain image content features of the key video segment; (23) extracting subtitle text information from the subtitle position of the key video segment, performing a first encoding on the subtitle text information, and obtaining text content features of the key video segment; (24) transmitting the voice information of the key video segment to a second convolutional neural network for second encoding to obtain the voice content features of the key video segment; (25) using the image content feature, the text content feature, and the voice content feature as multimodal information features of the key video segment; (3) performing similarity calculation on the multimodal information features of the key video clip and the multimodal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database; (4) returning the short videos that meet the preset similarity threshold as the search results of the short video object to the user; The key video clips are fed into the first three-dimensional convolutional neural network to extract the features of the short video objects and obtain the image content features of the short video objects. Assuming the number of key frames obtained is N, the short video objects are divided into M groups, then sequences; The first convolutional neural network in three dimensions is represented as F img =Conv 3D (W img ,seq), where W img is the weight coefficient, F img represents the extracted image content features, F img Is a (1×dim) vector, dim represents the vector size; During the encoding process, let y=(y1,y2…y n ) represents n characters; q=(q1,q2…q n ) indicates that it is through q=Emb q (y) Embbeding is performed to obtain the query vector; then: k=(k1,k2…k n )=In k ×q v=(v1,v2…v n )=W v ×q k, v represent the query vector and value vector respectively, W k ,W v Represent the weight matrices respectively; then the final vector is obtained through the self-attention mechanism where k T represents the transpose of k; the final text code is F wor =trans(V,hid), where hid represents the hidden layer vector, F wor is a (1×dim) vector; The second convolutional neural network in three dimensions is represented as F vid =Conv 3D (W vid ,seq), where W vid is the weight coefficient, F vid represents the extracted speech content features, F vid Is a (1×dim) vector, dim represents the vector size; The final image content features, text content features, and voice content features are used as the final multimodal information features of the short video object. The multimodal information features are expressed as F all =(F img ,F wor ,F vid ); The obtained multi-modal information features are similarly calculated with the multi-modal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database. The similarity is calculated as follows: The "·" symbol represents the dot product of vectors. is the multimodal information of a short video in the short video database, Indicates the multimodal information of the short video object that needs to be searched.

2. The multimodal short video search method according to claim 1, characterized in that: In the process of obtaining the key video segments in step (21), the key video segments are divided into sequences according to the number of key frames obtained and the preset number of groups; In step (22), the first convolutional neural network extracts image content features based on the sequence division results and weight coefficients of the key video clips.

3. The multimodal short video search method according to claim 2, characterized in that: In step (23), the subtitle text information is first encoded through the Transformer model, and the final vector is obtained through the self-attention mechanism using the query vector, value vector, and weight matrix. The subtitle text information is first encoded based on the final vector and the hidden layer vector.

4. The multimodal short video search method according to claim 3, characterized in that: In step (24), the second convolutional neural network extracts speech content features based on the sequence division results and weight coefficients of the key video clips.

5. The multimodal short video search method according to claim 4, characterized in that: In step (3), a vector dot product is performed on the multimodal information features of the short video object and the multimodal information features of a given short video in the short video database, and then the similarity between the short video object and the given short video in the short video database is obtained based on the modal data of the multimodal information features.

6. The multimodal short video search method according to claim 1, characterized in that: According to step (2), multimodal information features of the existing short video are pre-extracted and stored in the short video database.

7. A multimodal short video search device, comprising the multimodal short video search method according to any one of claims 1 to 6, characterized in that: include: A short video acquisition module is used to acquire the short video object to be searched and transmit the short video object to the multimodal feature extraction model; A multimodal feature extraction module, configured to extract multimodal information features from the short video object using the multimodal feature extraction model; The multimodal feature extraction module includes: A key information extraction submodule is used to extract key information from the short video object to obtain key video segments of the short video object and key audio information corresponding to the key video segments; An image content feature extraction submodule, configured to transmit the key video segment to a first convolutional neural network for extracting video features to obtain image content features of the key video segment; a text content feature extraction submodule, configured to extract subtitle text information from the subtitle position of the key video segment, perform a first encoding on the subtitle text information, and obtain text content features of the key video segment; a speech content feature extraction submodule, configured to transmit the speech information of the key video segment to a second convolutional neural network for second encoding to obtain speech content features of the key video segment; a multimodal information feature combination submodule, configured to use the image content feature, the text content feature, and the voice content feature as the multimodal information feature of the key video segment; A similarity statistics module is used to calculate the similarity between the multimodal information features of the key video clip and the multimodal information features in the short video database to obtain the similarity between the short video object and each short video in the short video database; The search result returning module is used to return the short videos that meet the preset similarity threshold as the search results of the short video object to the user.

8. The multimodal short video search device according to claim 7, characterized in that: During the acquisition of the key video segments by the key information extraction submodule, the key video segments are divided into sequences according to the number of key frames obtained and the preset number of groups; In the image content feature extraction submodule, the first convolutional neural network extracts image content features based on the sequence division results and weight coefficients of the key video segments; In the text content feature extraction submodule, the subtitle text information is first encoded using a Transformer model, a final vector is obtained by using a query vector, a value vector, and a weight matrix through a self-attention mechanism, and the subtitle text information is first encoded based on the final vector and the hidden layer vector; In the speech content feature extraction submodule, the second convolutional neural network extracts speech content features based on the sequence division results and weight coefficients of the key video clips.

9. The multimodal short video search device according to claim 8, characterized in that: In the similarity statistics module, a vector dot product is performed on the multimodal information features of the short video object and the multimodal information features of a given short video in the short video database, and then the similarity between the short video object and the given short video in the short video database is obtained based on the modal data of the multimodal information features; For existing short videos, multimodal information features of the existing short videos are pre-extracted and stored in the short video database according to the key information extraction submodule, image content feature extraction submodule, text content feature extraction submodule, voice content feature extraction submodule and multimodal information feature combination submodule in the multimodal feature extraction module.

10. A storage medium, characterized in that: The storage medium stores a program code of a short video search method based on multimodality, and the program code includes instructions for executing the short video search method based on multimodality as described in any one of claims 1 to 6.

Images