Multimodal-based video retrieval method and device, electronic equipment and storage medium

Abstract

The application relates to the field of intelligent decision-making and the field of digital medicine, and discloses a video retrieval method and device based on multiple modes, electronic equipment and a storage medium, the method comprising the following steps: querying multiple mode attributes of a to-be-retrieved video, selecting a multiple mode network, and extracting multiple mode features in the to-be-retrieved video; extracting a feature sequence, constructing a fixed cluster group of the feature sequence, performing mean pooling on the fixed cluster group to obtain a pooled cluster group, performing fixed-length alignment on the feature sequence according to the pooled cluster group to obtain an aligned sequence; extracting a feature dimension, aligning the dimension length to obtain an aligned dimension, and determining aligned multiple mode features of the multiple mode features according to the aligned sequence and the aligned dimension; performing feature splicing and feature fusion; performing feature vector coding on a retrieval text, calculating a cosine similarity, and determining a video retrieval result of the retrieval text according to the cosine similarity. The application can realize automatic indexing and retrieval of videos and improve the comprehensiveness of the entire video retrieval process.

Description

Technical Field

[0001] This invention relates to the fields of intelligent decision-making and digital healthcare, and in particular to a multimodal video retrieval method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Multimodal video retrieval refers to the process of extracting multimodal features contained in dynamic videos, classifying videos based on these multimodal features, and matching videos with input search text to achieve the process of retrieving videos from input text.

[0003] Currently, with the rise of video search technology, people can find desired medical videos, health videos, and other medical content by inputting a simple medical-related text description or a few keywords. This method supports functions such as disease-aided diagnosis and health management. Traditional medical video search methods rely on manual tagging of videos, assigning appropriate titles or descriptions based on the content. In today's era of social media, this information is usually added by video creators such as doctors and hospitals when uploading videos. However, these title tags or medical-related descriptions sometimes fail to fully summarize the medical content of the video. Since even a minor illness may involve a wide range of medical knowledge, sometimes the medical information in these titles or tags is even missing, such as when the video creator did not add descriptive information, or when the medical information in the video comes from other sources such as the internet. This search method, which relies on "hard tags," cannot accurately complete the search task. Therefore, because medical video classification relies on manual tagging, video search is not comprehensive enough. Summary of the Invention

[0004] This invention provides a multimodal video retrieval method, device, electronic device, and computer-readable storage medium, the main purpose of which is to realize automated indexing and retrieval of medical videos and improve the comprehensiveness of the entire medical video retrieval process.

[0005] To achieve the above objectives, the present invention provides a multimodal video retrieval method, comprising:

[0006] Obtain the video to be retrieved, query the multimodal attributes of the video to be retrieved, select the multimodal network corresponding to the multimodal attributes, and use the multimodal network to extract the multimodal features in the video to be retrieved;

[0007] Extract the feature sequences of the multimodal features, construct a fixed cluster of the feature sequences, perform mean pooling on the fixed clusters to obtain a pooled cluster, and perform fixed-length alignment on the feature sequences based on the pooled clusters to obtain an aligned sequence;

[0008] The feature dimensions of the multimodal features are extracted, and the dimension lengths of the feature dimensions are aligned using a pre-built multilayer perceptron to obtain the aligned dimensions.

[0009] Based on the alignment sequence and the alignment dimension, the aligned multimodal features in the multimodal features are determined. Multiple features in the aligned multimodal features are concatenated to obtain concatenated multimodal features. The concatenated multimodal features are then fused using a fully connected visual network to obtain fused multimodal features.

[0010] The search text is obtained, and the search text is encoded using feature vectors to obtain an encoded text vector. The cosine similarity between the fused multimodal features and the encoded text vector is calculated. Based on the cosine similarity, the video search result of the search text is determined.

[0011] Optionally, querying the multimodal attributes of the video to be retrieved includes:

[0012] The video to be retrieved is played back to obtain the video to be played.

[0013] Perform an integrity check on the video elements of the playing video;

[0014] When the integrity check of the video elements in the playing video is successful, the multimodal attributes of the video to be retrieved are determined;

[0015] When the integrity check of the video elements in the playing video fails, the failed portion of the video is extracted.

[0016] Based on the failed portion of the video, the multimodal attributes of the video to be retrieved are determined.

[0017] Optionally, selecting the multimodal network corresponding to the multimodal attribute includes:

[0018] The purpose of identifying the features of the multimodal attributes;

[0019] Obtain the neural network structure and query the network objective of the neural network structure;

[0020] Match the network objective with the feature objective;

[0021] When the network objective matches the feature objective, the multimodal network corresponding to the multimodal attribute is determined.

[0022] Optionally, aligning the dimension length of the feature dimension using a pre-built multilayer perceptron to obtain the aligned dimension includes:

[0023] Configure the target length of the feature dimension;

[0024] Based on the target length and the feature dimension, select multilayer neurons in the pre-constructed multilayer perceptron;

[0025] The alignment dimension is obtained by performing a dimension length alignment operation on the feature dimension using the multilayer neurons.

[0026] Optionally, the step of using a fully connected visual network to fuse the stitched multimodal features to obtain fused multimodal features includes:

[0027] The concatenated multimodal features are fused using the following formula to obtain the column-fused features:

[0028] U *, = *, +2σ(W1LayerNorm() *, )

[0029] Among them, U *, X represents the column fusion feature. *, Let represent the set of column vectors of the feature vector set in the concatenated multimodal features, i represent the column of the feature vector set in the concatenated multimodal features, W1 and W2 represent the weights of the preceding and following NLP layers in the fully connected visual network when performing feature column fusion, σ represents the bias in the fully connected visual network, and LayerNorm represents the algorithm for normalizing the concatenated multimodal features. LayerNorm() *, This represents the process of standardizing the set of column vectors of the feature vector set in the concatenated multimodal features, where X represents the set of feature vectors in the concatenated multimodal features;

[0030] The column-to-row fusion features are obtained by performing feature-row fusion on the column-to-row fusion features using the following formula:

[0031] Y j,* = j,* +4σ(W3LayerNorm() j,* )

[0032] Among them, Y j,* U represents the fused multimodal features. j,* W3 and W4 represent the row feature vectors in the column fusion features, respectively, representing the weights of the preceding and following NLP layers in the fully connected visual network during feature row fusion. σ represents the bias in the fully connected visual network, and LayerNorm represents the algorithm for standardizing the concatenated multimodal features. LayerNorm() j,* This represents the process of standardizing the row feature vector in the column fusion feature, where U represents the column fusion feature.

[0033] Optionally, the step of encoding the retrieved text using feature vectors to obtain an encoded text vector includes:

[0034] Determine the text category of the retrieved text;

[0035] Construct the initial encoding for the text category;

[0036] Match the searched text with the text category;

[0037] When the searched text successfully matches the text category, the searched text is encoded using feature vectors according to the initial encoding to obtain an encoded text vector.

[0038] Optionally, calculating the cosine similarity between the fused multimodal features and the encoded text vector includes:

[0039] The cosine similarity between the fused multimodal features and the encoded text vector is calculated using the following formula.

[0040]

[0041] Among them, W uv The cosine similarity between the fused multimodal features and the encoded text vector is represented by N(u), where N(u) represents the feature corresponding to u in the set N of the fused multimodal features, and M(v) represents the text vector corresponding to v in the set M of the encoded text vectors.

[0042] To address the above problems, the present invention also provides a multimodal video retrieval device, the device comprising:

[0043] The feature extraction module is used to acquire the video to be retrieved, query the multimodal attributes of the video to be retrieved, select the multimodal network corresponding to the multimodal attributes, and use the multimodal network to extract the multimodal features in the video to be retrieved.

[0044] A fixed-length alignment module is used to extract the feature sequence of the multimodal features, construct a fixed cluster of the feature sequence, perform mean pooling on the fixed cluster to obtain a pooled cluster, and perform fixed-length alignment on the feature sequence based on the pooled cluster to obtain an aligned sequence.

[0045] The dimension alignment module is used to extract the feature dimensions of the multimodal features and align the dimension lengths of the feature dimensions using a pre-built multilayer perceptron to obtain the aligned dimensions.

[0046] The feature fusion module is used to determine the aligned multimodal features in the multimodal features according to the alignment sequence and the alignment dimension, perform feature concatenation on multiple features in the aligned multimodal features to obtain concatenated multimodal features, and perform feature fusion on the concatenated multimodal features using a fully connected visual network to obtain fused multimodal features.

[0047] The retrieval determination module is used to acquire retrieval text, encode the retrieval text using feature vectors to obtain encoded text vectors, calculate the cosine similarity between the fused multimodal features and the encoded text vectors, and determine the video retrieval result of the retrieval text based on the cosine similarity.

[0048] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0049] At least one processor; and,

[0050] A memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores a computer program that can be executed by the at least one processor to implement the multimodal video retrieval method described above.

[0052] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the multimodal video retrieval method described above.

[0053] As can be seen, this embodiment of the invention acquires the video to be retrieved for tag assignment of medical videos uploaded to the platform. Furthermore, this embodiment queries the multimodal attributes of the video to be retrieved to extract key medical features from multiple dimensions, improving the comprehensiveness of the medical information analysis of the video. Further, this embodiment selects the multimodal network corresponding to the multimodal attributes to extract medical features from the video using different types of feature extraction networks, reducing the simplification of medical video tags and improving the comprehensiveness of medical video analysis. Finally, this embodiment utilizes the multimodal network to extract the video to be retrieved. The invention utilizes multimodal features to extract key medical information from the video to be retrieved from multiple feature perspectives, ensuring comprehensive analysis of the medical information carried in the video. In this embodiment, the feature sequences of the multimodal features are extracted to uniformly adjust the length of different medical feature sequences generated from different video durations, ensuring the uniformity of the medical feature data. Furthermore, this embodiment constructs fixed clusters of the feature sequences to assign different categories of medical feature vectors to different clusters, i.e., one cluster contains one category of medical feature vectors, ensuring the classification processing of medical feature vectors. Further, this embodiment applies a mean to the fixed clusters. Pooling is used to determine the average length of the feature sequence, thereby controlling the length of the feature sequence. Further, in this embodiment, the feature sequence is aligned to a fixed length based on the pooling clusters to adjust sequences of different lengths into sequences of uniform length, ensuring that subsequent analysis of the sequence is not affected by messy medical data and improving the efficiency of medical data analysis. This embodiment extracts the feature dimensions of the multimodal features to adjust irregular dimension lengths. Further, this embodiment uses a pre-built multilayer perceptron to align the dimension lengths of the feature dimensions to correct inconsistent dimension lengths and improve the standardization of medical data. This embodiment uses root... Based on the alignment sequence and the alignment dimension, aligned multimodal features are determined in the multimodal features to recombine the formatted medical features, ensuring the orderliness of medical data. Furthermore, this embodiment of the invention performs feature concatenation on multiple medical features in the aligned multimodal features to combine medical feature vectors with other medical feature vectors, ensuring subsequent fusion processing. Further, this embodiment of the invention utilizes a fully connected visual network to perform feature fusion on the concatenated multimodal features, fusing the extracted multimodal features to obtain more implicit medical features, improving the comprehensiveness of medical feature extraction, and ensuring the comprehensiveness of medical video retrieval. This embodiment of the invention obtains the retrieval text...This invention aims to match medical keyword text input by a user to search for videos with their corresponding medical videos. Furthermore, this embodiment encodes the search text using feature vectors to convert it into a medical feature vector form, facilitating matching with the feature vectors of medical videos. Further, this embodiment calculates the cosine similarity between the fused multimodal features and the encoded text vector to identify medical videos similar to the search text. By utilizing multi-dimensional features, the comprehensiveness of the search is improved. Furthermore, this embodiment determines the video search results based on the cosine similarity, and uses the calculated results to determine the medical video search results, achieving artificial intelligence automation and improving the efficiency of medical video retrieval. Therefore, the multimodal video retrieval method, device, electronic device, and computer-readable storage medium proposed in this embodiment can achieve automated indexing and retrieval of medical videos, improving the comprehensiveness of the entire medical video retrieval process. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a multimodal video retrieval method according to an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of a module of a multimodal video retrieval device provided in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the internal structure of an electronic device that implements a multimodal video retrieval method according to an embodiment of the present invention;

[0057] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0058] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0059] This invention provides a multimodal video retrieval method. The execution entity of the multimodal video retrieval method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this invention: a server, a terminal, etc. In other words, the multimodal video retrieval method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0060] Reference Figure 1 The diagram shown is a flowchart illustrating a multimodal video retrieval method according to an embodiment of the present invention. In this embodiment, the multimodal video retrieval method includes the following steps S1-S5:

[0061] S1. Obtain the video to be retrieved, query the multimodal attributes of the video to be retrieved, select the multimodal network corresponding to the multimodal attributes, and use the multimodal network to extract the multimodal features in the video to be retrieved.

[0062] This invention provides an embodiment of the method for acquiring videos to be retrieved, which are then used to assign tags to medical videos uploaded to the platform. The videos to be retrieved refer to videos uploaded by users to the platform, and these videos are generated based on different business scenarios. For example, in the digital healthcare industry, the videos to be retrieved may be medical videos, popular science videos, promotional videos for therapies, medical payment videos, etc.; in the financial business scenario, the videos to be retrieved may be fund popular science videos, stock price analysis videos, bank card application videos, etc.

[0063] Furthermore, this embodiment of the invention queries the multimodal attributes of the video to be retrieved to extract key medical features from the video from multiple dimensions, thereby improving the comprehensiveness of the medical information analysis of the video. The multimodal attributes refer to the analytical dimensions of the video to be retrieved, including object category information, scene information, dynamic temporal information, audio, and subtitles.

[0064] In one embodiment of the present invention, querying the multimodal attributes of the video to be retrieved includes: playing the video to be retrieved to obtain a playing video; performing an integrity check on the video elements of the playing video; determining the multimodal attributes of the video to be retrieved when the integrity check of the video elements of the playing video is successful; extracting the video failure portion of the video integrity check when the integrity check of the video elements of the playing video fails; and determining the multimodal attributes of the video to be retrieved based on the video failure portion.

[0065] For example, when playing the video to be retrieved, it is possible to query whether the video elements of the video to be retrieved are complete, such as whether the sound of the video to be retrieved is missing. If the sound is not missing, then the sound of the video to be retrieved is complete enough, and the audio information is used as the multimodal attribute; if the sound is missing, it means that the sound of the video is not complete enough, and the missing sound is not used as the multimodal attribute of the video.

[0066] Furthermore, this embodiment of the invention selects a multimodal network corresponding to the multimodal attribute to extract medical features from the video using different types of feature extraction networks. This reduces the simplification of medical video labels and improves the comprehensiveness of medical video analysis. The multimodal network refers to a feature extraction network used to extract video features, also known as an "expert." This includes: 1. For static image information, a ResNet network trained on ImageNet can be used as an expert to extract body category information; 2. A SENet network pre-trained on Place365 can be used to extract scene information; 3. For dynamic temporal information, models such as I3D can be used to extract video temporal features; 4. For audio, a VGGish network can be used to sample audio and extract features; 5. For subtitles, pre-trained models such as BERT and GPT are used to encode text features.

[0067] In one embodiment of the present invention, selecting the multimodal network corresponding to the multimodal attribute includes: identifying the feature purpose of the multimodal attribute; obtaining a neural network structure and querying the network purpose of the neural network structure; matching the network purpose with the feature purpose; and determining the multimodal network corresponding to the multimodal attribute when the network purpose and the feature purpose are successfully matched.

[0068] The feature objective refers to the purpose of extracting features corresponding to the multimodal attributes. For example, when the multimodal attribute is an object category, the feature objective is to extract foreground region features from the video. The network objective refers to the feature extraction objective of existing neural network structures. For example, the objective of the ResNet network is to extract foreground region features from the video.

[0069] Furthermore, this embodiment of the invention utilizes the multimodal network to extract multimodal features from the video to be retrieved, thereby obtaining key medical information attached to the video from multiple feature perspectives and ensuring comprehensive analysis of the medical information carried in the video. The multimodal features refer to feature vectors corresponding to the multimodal attributes, including features such as object category information, video scene information, dynamic temporal information, audio, and subtitles in the video.

[0070] In one embodiment of the present invention, the step of extracting multimodal features from the video to be retrieved using the multimodal network includes: querying the network structure of the multimodal network; identifying the feature extraction step of the network structure; and using the feature extraction step to extract multimodal features from the video to be retrieved.

[0071] For example, if the multimodal network is a ResNet network, the network structure of the ResNet network includes a convolutional layer, a pooling layer, a series of residual structures, and finally an average pooling downsampling layer and a fully connected layer to obtain the final output. If the network structure is the above-mentioned convolutional layer, and the feature extraction step of the convolutional layer is determined to be a convolution on the size of the image, then feature extraction can be achieved using the feature extraction step. For example, if the input image is 224*224*3, then after a 7*7, 64 convolution in the convolutional layer, with stride=2 and padding=3, the output size is (224-7+6) / 2+1=112.5 rounded down to 112, so the output is 112*112*64.

[0072] S2. Extract the feature sequences of the multimodal features, construct a fixed cluster of the feature sequences, perform mean pooling on the fixed clusters to obtain a pooled cluster, and perform fixed-length alignment on the feature sequences based on the pooled clusters to obtain an aligned sequence.

[0073] This invention extracts feature sequences from the multimodal features to uniformly adjust the lengths of different medical feature sequences generated from different medical video durations, ensuring the consistency of medical feature data. Here, the feature sequence refers to the length of the feature vector sequence of the multimodal features, or it can be understood as the length of the feature vector corresponding to a certain dimension among multiple dimensions of a extracted modality feature.

[0074] In one embodiment of the present invention, the step of extracting the feature sequence of the multimodal features includes: obtaining the feature vector of the multimodal features, extracting vector data from the feature vector; identifying the vector sequence of the vector data, and using the vector sequence as the feature sequence of the multimodal features.

[0075] For example, when the feature vector of the multimodal feature is obtained as a one-dimensional array, the data in the array is extracted, and the extracted data with sequential relationships are constructed into a sequence to obtain the vector sequence.

[0076] Furthermore, this embodiment of the invention constructs fixed clusters of the feature sequences to classify different categories of medical feature vectors into different clusters; that is, each cluster contains medical feature vectors of one category, ensuring the classification processing of medical feature vectors. Here, the fixed cluster refers to a cluster containing vector data from a specific category of feature vectors.

[0077] In one embodiment of the present invention, constructing a fixed cluster of the feature sequence includes: querying the feature category of the feature sequence; determining a fixed category of the feature sequence based on the feature category; and determining a fixed cluster of the feature sequence based on the fixed category.

[0078] For example, the feature category of the feature sequence can be queried by querying the number of output feature vectors. If the number of the feature vectors is 5, then the feature category is determined to be 5. At the same time, it is determined that there are 5 categories of feature sequences. Then the fixed category of the feature sequence is 5, and finally the corresponding fixed clusters are 5.

[0079] Furthermore, in this embodiment of the invention, mean pooling is performed on the fixed cluster to determine the average length of the feature sequence, thereby achieving length control of the feature sequence.

[0080] In one embodiment of the present invention, the fixed cluster is subjected to mean pooling using the following formula to obtain a pooled cluster:

[0081]

[0082] Among them, g p Let p be the pooling cluster of the fixed cluster in the pooling cluster, E be the total number of feature sequences in the fixed cluster p, q be the q feature sequence in the fixed cluster p, and g be the pooling cluster of the fixed cluster p. pq This represents the length of the feature sequence q in the fixed cluster p.

[0083] Furthermore, in this embodiment of the invention, the feature sequences are aligned to a fixed length according to the pooling clusters, so as to adjust sequences of different lengths into sequences of uniform length, ensuring that subsequent analysis of the sequences will not be affected by messy medical data, thereby improving the efficiency of medical data analysis.

[0084] In one embodiment of the present invention, the step of performing fixed-length alignment on the feature sequence according to the pooling cluster to obtain an aligned sequence includes: extracting the cluster length in the pooling cluster; and using the cluster length as the length of the feature sequence to obtain the aligned sequence.

[0085] The cluster length refers to the average length of the feature sequences calculated in each pooling cluster.

[0086] S3. Extract the feature dimensions of the multimodal features, and align the dimension lengths of the feature dimensions using a pre-built multilayer perceptron to obtain the aligned dimensions.

[0087] This invention extracts the feature dimensions of the multimodal features to adjust irregular dimension lengths. Here, the feature dimension refers to different categories of features within a specific modality of the multimodal features, i.e., features of different dimensions.

[0088] In one embodiment of the present invention, extracting the feature dimension of the multimodal features includes: querying the feature category of the feature sequence in the multimodal features; and identifying the feature dimension of the feature sequence according to the feature category.

[0089] For example, if the feature categories of the feature sequence x(0,1) and the feature sequence y(1,0) are A and B, then the feature sequences are combined according to the feature categories to obtain (x,y) as the feature dimension.

[0090] Furthermore, this embodiment of the invention utilizes a pre-built multilayer perceptron to align the dimensional lengths of the feature dimensions, thereby correcting inconsistent dimensional lengths and improving the standardization of medical data. The pre-built multilayer perceptron refers to an artificial neural network, where the layers are fully connected, with the bottom layer being the input layer, the middle layers being hidden layers, and the final layer being the output layer.

[0091] In one embodiment of the present invention, aligning the dimension length of the feature dimension using a pre-built multilayer perceptron to obtain the aligned dimension includes: configuring a target length for the feature dimension; selecting multilayer neurons in the pre-built multilayer perceptron based on the target length and the feature dimension; and performing a dimension length alignment operation on the feature dimension using the multilayer neurons to obtain the aligned dimension.

[0092] For example, the multilayer perceptron consists of an input layer, a hidden layer, and an output layer. Each layer consists of multiple neurons, and the neurons in each layer are connected to the neurons in the next layer. This changes the length of the feature dimension. For example, if the length of the feature dimension is 3-dimensional and the length of the target dimension is 4-dimensional, then the neurons in the first layer should be 3 neurons, and the neurons in the second layer should be 4 neurons. The neurons in the first layer and the neurons in the second layer are connected through fully connected layers.

[0093] S4. Based on the alignment sequence and the alignment dimension, determine the aligned multimodal features in the multimodal features, perform feature concatenation on multiple features in the aligned multimodal features to obtain concatenated multimodal features, and use a fully connected visual network to perform feature fusion on the concatenated multimodal features to obtain fused multimodal features.

[0094] This invention, through embodiments thereof, determines aligned multimodal features within the multimodal features based on the alignment sequence and the alignment dimension. This alignment is used to recombine formatted medical features, ensuring the orderliness of medical data. The aligned multimodal features refer to the feature vector formed by the alignment sequence and the alignment dimension after processing.

[0095] In one embodiment of the present invention, the step of determining the alignment multimodal feature in the multimodal feature according to the alignment sequence and the alignment dimension is achieved by concatenating the alignment sequence and the alignment dimension to form the alignment multimodal feature.

[0096] Furthermore, in this embodiment of the invention, multiple medical features in the aligned multimodal features are spliced ​​together to combine the medical feature vector with other medical feature vectors, thereby ensuring subsequent fusion processing.

[0097] In one embodiment of the present invention, the step of concatenating multiple features in the aligned multimodal features to obtain concatenated multimodal features is achieved by combining multiple features in the aligned multimodal features sequentially. For example, concatenating feature vector A (1, 0, 0, 0) with feature vector B (0, 1, 0, 0) can be achieved using (1, 0, 0, 0, 0, 1, 0, 0).

[0098] Furthermore, this embodiment of the invention utilizes a fully connected visual network to perform feature fusion on the stitched multimodal features, thereby fusing the extracted multimodal features to obtain more implicit medical features, improving the comprehensiveness of medical feature extraction, and ensuring the comprehensiveness of medical video retrieval.

[0099] In one embodiment of the present invention, the step of using a fully connected visual network to perform feature fusion on the stitched multimodal features to obtain fused multimodal features includes: performing feature column fusion on the stitched multimodal features using the following formula to obtain column fused features:

[0100] U *, = *, +2σ(W1LayerNorm() *, )

[0101] Among them, U *, X represents the column fusion feature. *, Let represent the set of column vectors of the feature vector set in the concatenated multimodal features, i represent the column of the feature vector set in the concatenated multimodal features, W1 and W2 represent the weights of the preceding and following NLP layers in the fully connected visual network when performing feature column fusion, σ represents the bias in the fully connected visual network, and LayerNorm represents the algorithm for normalizing the concatenated multimodal features. LayerNorm() *, This represents the process of standardizing the set of column vectors of the feature vector set in the concatenated multimodal features, where X represents the set of feature vectors in the concatenated multimodal features;

[0102] The column-to-row fusion features are obtained by performing feature-row fusion on the column-to-row fusion features using the following formula:

[0103] Y j,* = j,* +4σ(W3LayerNorm() j,* )

[0104] Among them, Y j,* U represents the fused multimodal features. j,* W3 and W4 represent the row feature vectors in the column fusion features, respectively, representing the weights of the preceding and following NLP layers in the fully connected visual network during feature row fusion. σ represents the bias in the fully connected visual network, and LayerNorm represents the algorithm for standardizing the concatenated multimodal features. LayerNorm() j,* This represents the process of standardizing the row feature vector in the column fusion feature, where U represents the column fusion feature.

[0105] S5. Obtain the search text, encode the search text using feature vectors to obtain an encoded text vector, calculate the cosine similarity between the fused multimodal features and the encoded text vector, and determine the video search result of the search text based on the cosine similarity.

[0106] This invention provides an embodiment of a method for obtaining search text to match medical keywords entered by a user as a search term with their corresponding medical videos. The search text refers to the text entered into a search engine to search for videos.

[0107] Furthermore, in this embodiment of the invention, the retrieved text is encoded using feature vectors to convert it into a medical feature vector form, which facilitates matching with the feature vectors of medical videos.

[0108] In one embodiment of the present invention, the step of performing feature vector encoding on the retrieved text to obtain an encoded text vector includes: determining the text category of the retrieved text; constructing an initial encoding for the text category; matching the retrieved text with the text category; and when the retrieved text successfully matches the text category, performing feature vector encoding on the retrieved text according to the initial encoding to obtain an encoded text vector.

[0109] For example, when the search text is "potato and tomato", it is determined that the search text is of Chinese character type and has no symbols. The text category of the search text is determined to be all Chinese characters. The initial encoding of the text category is to sort all Chinese characters and encode them as 0. The constructed initial encoding is used as the first row. The search text is divided into multiple single characters and used as columns. When the search text "bean" in the second row matches the Chinese character "bean" in the third column, the encoding of the second row is "001000...". The encoding vector of the search text can be obtained in this way.

[0110] Furthermore, this embodiment of the invention calculates the cosine similarity between the fused multimodal features and the encoded text vector to determine medical videos similar to the search text. By utilizing multi-dimensional features for retrieval, the comprehensiveness of the retrieval is improved. The cosine similarity refers to the proportional similarity of two variables in various directions (attributes). It is achieved by using the cosine of the angle between two vectors in vector space as a measure of the difference between two individuals. The closer the value is to 1, the closer the angle is to 0°, meaning the two vectors are more similar.

[0111] In one embodiment of the present invention, the cosine similarity between the fused multimodal features and the encoded text vector is calculated using the following formula, including:

[0112]

[0113] Among them, W uvThe cosine similarity between the fused multimodal features and the encoded text vector is represented by N(u), where N(u) represents the feature corresponding to u in the set N of the fused multimodal features, and M(v) represents the text vector corresponding to v in the set M of the encoded text vectors.

[0114] Furthermore, in this embodiment of the invention, the video retrieval result of the searched text is determined based on the cosine similarity, which is used to determine the medical video retrieval result based on the calculation result, thereby realizing artificial intelligence automation and improving the efficiency of medical video retrieval.

[0115] In one embodiment of the present invention, determining the video retrieval result of the searched text based on the cosine similarity includes: configuring a similarity threshold for the cosine similarity; determining whether the cosine similarity meets the similarity threshold; when the cosine similarity meets the similarity threshold, using the video to be retrieved corresponding to the cosine similarity as the video retrieval result of the searched text; when the cosine similarity does not meet the similarity threshold, the video retrieval result of the searched text is a retrieval failure.

[0116] The cosine similarity can be set to 0.8, or it can be set according to the actual situation.

[0117] As can be seen, this embodiment of the invention acquires the video to be retrieved for tag assignment of medical videos uploaded to the platform. Furthermore, this embodiment queries the multimodal attributes of the video to be retrieved to extract key medical features from multiple dimensions, improving the comprehensiveness of the medical information analysis of the video. Further, this embodiment selects the multimodal network corresponding to the multimodal attributes to extract medical features from the video using different types of feature extraction networks, reducing the simplification of medical video tags and improving the comprehensiveness of medical video analysis. Finally, this embodiment utilizes the multimodal network to extract the video to be retrieved. The invention utilizes multimodal features to extract key medical information from the video to be retrieved from multiple feature perspectives, ensuring comprehensive analysis of the medical information carried in the video. In this embodiment, the feature sequences of the multimodal features are extracted to uniformly adjust the length of different medical feature sequences generated from different video durations, ensuring the uniformity of the medical feature data. Furthermore, this embodiment constructs fixed clusters of the feature sequences to assign different categories of medical feature vectors to different clusters, i.e., one cluster contains one category of medical feature vectors, ensuring the classification processing of medical feature vectors. Further, this embodiment applies a mean to the fixed clusters. Pooling is used to determine the average length of the feature sequence, thereby controlling the length of the feature sequence. Further, in this embodiment, the feature sequence is aligned to a fixed length based on the pooling clusters to adjust sequences of different lengths into sequences of uniform length, ensuring that subsequent analysis of the sequence is not affected by messy medical data and improving the efficiency of medical data analysis. This embodiment extracts the feature dimensions of the multimodal features to adjust irregular dimension lengths. Further, this embodiment uses a pre-built multilayer perceptron to align the dimension lengths of the feature dimensions to correct inconsistent dimension lengths and improve the standardization of medical data. This embodiment uses root... Based on the alignment sequence and the alignment dimension, aligned multimodal features are determined in the multimodal features to recombine the formatted medical features, ensuring the orderliness of medical data. Furthermore, this embodiment of the invention performs feature concatenation on multiple medical features in the aligned multimodal features to combine medical feature vectors with other medical feature vectors, ensuring subsequent fusion processing. Further, this embodiment of the invention utilizes a fully connected visual network to perform feature fusion on the concatenated multimodal features, fusing the extracted multimodal features to obtain more implicit medical features, improving the comprehensiveness of medical feature extraction, and ensuring the comprehensiveness of medical video retrieval. This embodiment of the invention obtains the retrieval text...This invention aims to match medical keyword text input by a user with corresponding medical videos. Furthermore, this embodiment encodes the search text using feature vectors, converting it into a medical feature vector form for easier matching with the feature vectors of medical videos. Further, this embodiment calculates the cosine similarity between the fused multimodal features and the encoded text vector to identify medical videos similar to the search text. By utilizing multi-dimensional features, the comprehensiveness of the search is improved. Furthermore, this embodiment determines the video search results based on the cosine similarity, and then uses the calculated results to determine the medical video search results, achieving artificial intelligence automation and improving the efficiency of medical video retrieval. Therefore, this multimodal video retrieval method can achieve automated indexing and retrieval of medical videos, improving the comprehensiveness of the entire medical video retrieval process.

[0118] like Figure 2 The diagram shown is a functional block diagram of the multimodal video retrieval device of the present invention.

[0119] The multimodal video retrieval device 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the multimodal video retrieval device may include a feature extraction module 101, a fixed-length alignment module 102, a dimension alignment module 103, a feature fusion module 104, and a retrieval determination module 105. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0120] In this embodiment, the functions of each module / unit are as follows:

[0121] The feature extraction module 101 is used to acquire the video to be retrieved, query the multimodal attributes of the video to be retrieved, select the multimodal network corresponding to the multimodal attributes, and use the multimodal network to extract the multimodal features in the video to be retrieved.

[0122] The fixed-length alignment module 102 is used to extract the feature sequence of the multimodal features, construct a fixed cluster of the feature sequence, perform mean pooling on the fixed cluster to obtain a pooled cluster, and perform fixed-length alignment on the feature sequence based on the pooled cluster to obtain an aligned sequence.

[0123] The dimension alignment module 103 is used to extract the feature dimensions of the multimodal features and align the dimension lengths of the feature dimensions using a pre-built multilayer perceptron to obtain the aligned dimensions.

[0124] The feature fusion module 104 is used to determine the aligned multimodal features in the multimodal features according to the alignment sequence and the alignment dimension, perform feature concatenation on multiple features in the aligned multimodal features to obtain concatenated multimodal features, and perform feature fusion on the concatenated multimodal features using a fully connected visual network to obtain fused multimodal features.

[0125] The retrieval determination module 105 is used to acquire the retrieval text, encode the retrieval text using feature vectors to obtain an encoded text vector, calculate the cosine similarity between the fused multimodal features and the encoded text vector, and determine the video retrieval result of the retrieval text based on the cosine similarity.

[0126] In detail, the modules in the multimodal video retrieval device 100 described in this embodiment of the invention employ the same methods as described above. Figure 1 The method uses the same techniques as the multimodal video retrieval method described in the previous section and can produce the same technical effects, so it will not be repeated here.

[0127] like Figure 3 The diagram shown is a structural schematic of an electronic device 1 that implements the multimodal video retrieval method of the present invention.

[0128] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a multimodal video retrieval program.

[0129] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a multimodal video retrieval program) and calls data stored in the memory 11 to perform various functions and process data of the electronic device 1.

[0130] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for a multimodal video retrieval program, but also to temporarily store data that has been output or will be output.

[0131] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0132] The communication interface 13 is used for communication between the aforementioned electronic device 1 and other devices, including a network interface and an employee interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish a communication connection between the electronic device 1 and other electronic devices 1. The employee interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual employee interface.

[0133] Figure 3 Only electronic device 1 with components is shown; those skilled in the art will understand that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0134] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0135] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in terms of the scope of the patent invention.

[0136] The multimodal video retrieval program stored in the memory 11 of the electronic device 1 is a combination of multiple computer programs. When run in the processor 10, it can achieve the following:

[0137] Obtain the video to be retrieved, query the multimodal attributes of the video to be retrieved, select the multimodal network corresponding to the multimodal attributes, and use the multimodal network to extract the multimodal features in the video to be retrieved;

[0138] Extract the feature sequences of the multimodal features, construct a fixed cluster of the feature sequences, perform mean pooling on the fixed clusters to obtain a pooled cluster, and perform fixed-length alignment on the feature sequences based on the pooled clusters to obtain an aligned sequence;

[0139] The feature dimensions of the multimodal features are extracted, and the dimension lengths of the feature dimensions are aligned using a pre-built multilayer perceptron to obtain the aligned dimensions.

[0140] Based on the alignment sequence and the alignment dimension, the aligned multimodal features in the multimodal features are determined. Multiple features in the aligned multimodal features are concatenated to obtain concatenated multimodal features. The concatenated multimodal features are then fused using a fully connected visual network to obtain fused multimodal features.

[0141] The search text is obtained, and the search text is encoded using feature vectors to obtain an encoded text vector. The cosine similarity between the fused multimodal features and the encoded text vector is calculated. Based on the cosine similarity, the video search result of the search text is determined.

[0142] Specifically, the processor 10's implementation method of the above-mentioned computer program can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0143] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0144] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device 1, can perform the following:

[0145] Obtain the video to be retrieved, query the multimodal attributes of the video to be retrieved, select the multimodal network corresponding to the multimodal attributes, and use the multimodal network to extract the multimodal features in the video to be retrieved;

[0146] Extract the feature sequences of the multimodal features, construct a fixed cluster of the feature sequences, perform mean pooling on the fixed clusters to obtain a pooled cluster, and perform fixed-length alignment on the feature sequences based on the pooled clusters to obtain an aligned sequence;

[0147] The feature dimensions of the multimodal features are extracted, and the dimension lengths of the feature dimensions are aligned using a pre-built multilayer perceptron to obtain the aligned dimensions.

[0148] Based on the alignment sequence and the alignment dimension, the aligned multimodal features in the multimodal features are determined. Multiple features in the aligned multimodal features are concatenated to obtain concatenated multimodal features. The concatenated multimodal features are then fused using a fully connected visual network to obtain fused multimodal features.

[0149] The search text is obtained, and the search text is encoded using feature vectors to obtain an encoded text vector. The cosine similarity between the fused multimodal features and the encoded text vector is calculated. Based on the cosine similarity, the video search result of the search text is determined.

[0150] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0151] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0153] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0154] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0155] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0156] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for multi-modal based video retrieval, the method comprising: The method comprises: acquiring a video to be retrieved, querying a multi-modal attribute of the video to be retrieved, selecting a multi-modal network corresponding to the multi-modal attribute, and extracting a multi-modal feature in the video to be retrieved by using the multi-modal network; extracting a feature sequence of the multi-modal feature, constructing a fixed cluster group of the feature sequence, performing mean pooling on the fixed cluster group to obtain a pooled cluster group, performing fixed-length alignment on the feature sequence according to the pooled cluster group to obtain an aligned sequence; extracting a feature dimension of the multi-modal feature, and aligning the dimension length of the feature dimension by using a pre-constructed multi-layer perceptron to obtain an aligned dimension; determining an aligned multi-modal feature in the multi-modal feature according to the aligned sequence and the aligned dimension, performing feature splicing on a plurality of features in the aligned multi-modal feature to obtain a spliced multi-modal feature, and performing feature fusion on the spliced multi-modal feature by using a fully connected visual network to obtain a fused multi-modal feature; acquiring a retrieval text, performing feature vector encoding on the retrieval text to obtain an encoded text vector, calculating a cosine similarity between the fused multi-modal feature and the encoded text vector, and determining a video retrieval result of the retrieval text according to the cosine similarity. 2.The multi-modal based video retrieval method of claim 1, wherein, The querying of the multi-modal attribute of the video to be retrieved comprises: playing the video to be retrieved to obtain a played video; performing integrity inspection on a video element of the played video; when the integrity inspection on the video element of the played video is successful, determining the multi-modal attribute of the video to be retrieved; when the integrity inspection on the video element of the played video fails, extracting a video failure part of the video element whose integrity inspection fails; determining the multi-modal attribute of the video to be retrieved according to the video failure part. 3.The multi-modal based video retrieval method of claim 1, wherein, The selecting of the multi-modal network corresponding to the multi-modal attribute comprises: identifying a feature purpose of the multi-modal attribute; acquiring a neural network structure, querying a network purpose of the neural network structure; matching the network purpose with the feature purpose; when the network purpose and the feature purpose are matched successfully, determining the multi-modal network corresponding to the multi-modal attribute. 4.The multi-modal based video retrieval method of claim 1, wherein, The aligning of the dimension length of the feature dimension by using the pre-constructed multi-layer perceptron to obtain the aligned dimension comprises: configuring a target length of the feature dimension; selecting a plurality of layers of neurons in the pre-constructed multi-layer perceptron according to the target length and the feature dimension; performing a dimension length alignment operation on the feature dimension by using the plurality of layers of neurons to obtain the aligned dimension. 5.The multi-modal based video retrieval method of claim 1, wherein, The feature fusion on the spliced multi-modal feature by using the fully connected visual network to obtain the fused multi-modal feature comprises: performing feature column fusion on the spliced multi-modal feature by using the following formula to obtain column fused features: wherein, denotes the column fusion feature of the column fusion feature, denotes a set of column vectors of the set of feature vectors in the concatenated multimodal feature, i denotes a column of the set of feature vectors in the concatenated multimodal feature, and denote the weights of the NLPs in the fully connected visual network before and after the layer when performing column fusion of features, denotes the bias in the fully connected visual network, denotes an algorithm for normalizing the concatenated multimodal feature, denotes a process for normalizing the set of column vectors of the set of feature vectors in the concatenated multimodal feature, denotes the set of feature vectors in the concatenated multimodal feature; performing feature row fusion on the column fused features by using the following formula to obtain the fused multi-modal feature: wherein, denotes the fusion multi-modal feature, denotes a row feature vector in the column fusion feature, and denote the weights of the front and back layers of NLP in the full connection visual network when fusing the feature row, respectively, denotes the bias in the full connection visual network, denotes an algorithm for normalizing the spliced multi-modal feature, denotes a process for normalizing the row feature vector in the column fusion feature, denotes the column fusion feature. 6.The multi-modal based video retrieval method of claim 1, wherein, The performing of feature vector encoding on the retrieval text to obtain an encoded text vector comprises: determining a text category of the retrieval text; constructing an initial encoding of the text category; matching the search text with the text category; when the search text is successfully matched with the text category, performing feature vector coding on the search text according to the initial coding to obtain a coded text vector.

7. The multi-modal based video retrieval method according to any one of claims 1 to 5, wherein, the calculating the cosine similarity between the fused multi-modal feature and the coded text vector comprises: the calculating the cosine similarity between the fused multi-modal feature and the coded text vector comprises wherein, denotes a cosine similarity between the fused multi-modal feature and the encoded text vector, denotes a set of fused multi-modal features in the fused multi-modal feature corresponding feature, denotes a set of encoded text vectors in the encoded text vector corresponding text vector. 8.A multi-modal based video retrieval apparatus, characterized by, The device comprises: a feature extraction module configured to acquire a to-be-searched video, query multi-modal attributes of the to-be-searched video, select a multi-modal network corresponding to the multi-modal attributes, and extract multi-modal features in the to-be-searched video by using the multi-modal network; a fixed-length alignment module configured to extract a feature sequence of the multi-modal features, construct a fixed cluster group of the feature sequence, perform mean pooling on the fixed cluster group to obtain a pooled cluster group, perform fixed-length alignment on the feature sequence according to the pooled cluster group to obtain an aligned sequence; a dimension alignment module configured to extract a feature dimension of the multi-modal features, and align the dimension length of the feature dimension by using a pre-constructed multi-layer perception machine to obtain an aligned dimension; a feature fusion module configured to determine aligned multi-modal features in the multi-modal features according to the aligned sequence and the aligned dimension, perform feature splicing on a plurality of features in the aligned multi-modal features to obtain spliced multi-modal features, and perform feature fusion on the spliced multi-modal features by using a fully connected visual network to obtain fused multi-modal features; a search determination module configured to acquire a search text, perform feature vector coding on the search text to obtain a coded text vector, calculate the cosine similarity between the fused multi-modal features and the coded text vector, and determine a video search result of the search text according to the cosine similarity.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the multi-modal based video search method according to any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the multi-modal based video search method according to any one of claims 1 to 7.

Images