Video retrieval method and device, electronic equipment and computer readable storage medium

Abstract

The present application relates to artificial intelligence technology, and discloses a video retrieval method, comprising: segmenting each preset video file through a lens to obtain a first video segment set, sequentially performing semantic segmentation on each first video segment to obtain a second video segment set corresponding to the video file, extracting a video segment feature of each second video segment by using a pre-trained CLIP+LSTM model, fusing all video segment features to obtain a video feature of the corresponding video file, receiving a text to be retrieved, extracting a text feature of the text to be retrieved by using the pre-trained CLIP+LSTM model, sequentially calculating a feature similarity between the text feature and a video feature of each preset video file, and selecting a video file corresponding to a feature similarity satisfying a preset similarity condition as a target video file. The present application also proposes a video retrieval device, equipment and medium. The present application can improve the accuracy of medical video retrieval in the field of intelligent medical treatment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a video retrieval method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] With the development of the Internet and video technology, more and more intelligent medical platforms publish medical popular science videos on the Internet and push medical popular science knowledge to users in a visual and intuitive manner. As the number of medical popular science videos increases, how to meet the needs of users to accurately and quickly retrieve the videos they need is a key problem for major medical platforms.

[0003] Common video retrieval methods in the industry include:

[0004] First, match the user input query text with the video text title;

[0005] Second, extract the video tags, and then match the query text with the video tags;

[0006] Third, use ASR (Automatic Speech Recognition) or OCR (Optical Character Recognition) technology to identify the video's corresponding text information, and then match the query text with the identified video text information.

[0007] The above methods are essentially a match between text (query text) and text (video tags, video title, video text information), i.e., a match between data in the same representation space. This approach often loses information in the medical video image and picture itself, so the accuracy of the above video retrieval needs to be improved. SUMMARY

[0008] The present application provides a video retrieval method, device, electronic equipment and computer readable storage medium, which aims to improve the accuracy of medical video retrieval in the field of intelligent medicine.

[0009] To achieve the above purpose, the present application provides a video retrieval method, comprising:

[0010] Segment each preset video file by lens to obtain a first video segment set corresponding to each preset video file;

[0011] Sequentially perform semantic segmentation on each first video segment in the first video segment set to obtain a second video segment set of the corresponding video file;

[0012] extracting video segment features of each second video segment in the second video segment set in sequence by using the pre-trained CLIP+LSTM model, and fusing all the video segment features to obtain video features of the corresponding video file;

[0013] receiving a text to be searched, extracting text features of the text to be searched by using the pre-trained CLIP+LSTM model, sequentially calculating feature similarities between the text features and video features of each preset video file, and selecting a video file corresponding to a feature similarity meeting a preset similarity condition as a target video file.

[0014] Optionally, the semantic segmentation of each first video segment in the first video segment set to obtain the second video segment set of the corresponding video file comprises:

[0015] recognizing the text of each first video segment and segmenting the text of each first video segment;

[0016] performing sentence vector conversion on each segment of each first video segment to obtain a sentence vector set of the corresponding first video segment;

[0017] calculating the adjacent window similarity and the skip window similarity between each two sentence vectors in the sentence vector set to obtain corresponding vector similarities, and dividing the segments corresponding to the vector similarities meeting a preset similarity threshold into one second video segment;

[0018] collecting all the second video segments to obtain the second video segment set of the corresponding preset video file.

[0019] Optionally, the sentence vector conversion on each segment of each first video segment to obtain a sentence vector set of the corresponding first video segment comprises:

[0020] segmenting each segment in sequence and performing word vector conversion on each word;

[0021] adding the word vectors corresponding to each segment to obtain a word vector matrix of each segment;

[0022] performing a pooling operation on each word vector matrix to obtain a sentence vector corresponding to each segment.

[0023] Optionally, the calculation of the adjacent window similarity and the skip window similarity between each two sentence vectors in the sentence vector set to obtain corresponding vector similarities, and the division of the segments corresponding to the vector similarities meeting a preset similarity threshold into one second video segment comprise:

[0024] Step A, taking the first sentence vector in the sentence vector set as a starting point;

[0025] Step B: calculating the adjacent window similarity between the starting point and the sentence vectors adjacent to the starting point, and determining whether the adjacent window similarity is greater than a preset similarity threshold;

[0026] When the adjacent window similarity is greater than the preset semantic similarity threshold, step C is performed, and the starting point and the sentence vector adjacent to the starting point are used as a temporary video segment;

[0027] Step C1: removing the sentence vectors in the temporary video segment from the sentence vector set, and determining whether the sentence vector set after removing the vectors is empty;

[0028] When the sentence vector set after the vector is removed is empty, executing C11, dividing the temporary video segment into a second video segment, and jumping to step E1;

[0029] If the sentence vector set after the vector is removed is not empty, executing C12, taking the first sentence vector in the sentence vector set as a starting point, calculating the adjacent window similarity and the jump window similarity between the starting point and the vector in the temporary video segment, performing weighted averaging on the adjacent window similarity and the jump window similarity to obtain a vector similarity, and determining whether the vector similarity is greater than a preset similarity threshold;

[0030] When the vector similarity is greater than the preset similarity threshold, executing C121, adding the starting point to the temporary video segment, and returning to step C1;

[0031] When the vector similarity is not greater than the preset similarity threshold, executing C122, dividing the temporary video segment into a second video segment, removing the vector corresponding to the second video segment from the vector set, and jumping to step E;

[0032] When the adjacent window similarity is not greater than the preset similarity threshold, executing step D, dividing the starting point into a second video segment, and removing the starting point from the sentence vector set;

[0033] Step E: Determine whether the sentence vector set after removing the vector is empty;

[0034] When the sentence vector set after removing the vector is empty, executing step E1, collecting the second video segments to obtain the second video segment set;

[0035] When the sentence vector set after removing the vector is not empty, return to the above step A.

[0036] Optionally, the pre-trained CLIP+LSTM model is used to sequentially extract video segment features of each second video segment in the second video segment set, and video features of a corresponding video file are obtained by fusing all the video segment features, including:

[0037] In time sequence, video frame extraction is sequentially performed on each second video segment to obtain a video frame set of each second video segment.

[0038] The CLIP part of the pre-trained CLIP+LSTM model is used to sequentially extract frame feature vectors of each video frame in the video frame set.

[0039] The LSTM part of the pre-trained CLIP+LSTM model is used to perform convolution operation on all frame feature vectors of each second video segment to obtain video segment features of a corresponding second video segment.

[0040] Pool operation is performed on all video segment features corresponding to the preset video file to obtain video features of the preset video file.

[0041] Optionally, the pre-trained CLIP+LSTM model is used to extract text features of the text to be searched, including:

[0042] Tokenization is performed on the text to be searched to obtain one or more search tokens, and a word vector of each search token is obtained.

[0043] The CLIP part of the pre-trained CLIP+LSTM model is used to splice the word vector of each search token to obtain a text vector matrix.

[0044] A search token is sequentially selected as a target token, and a key value of the target token is calculated according to the word vector of the target token and the text vector matrix.

[0045] According to the order from large to small of the key value, a preset number of search tokens are selected as feature tokens.

[0046] The word vector of the feature token is spliced to obtain the text features of the text to be searched.

[0047] Optionally, the key value of the target token is calculated according to the word vector of the target token and the text vector matrix, including:

[0048] The key value of the target token is calculated by using the following key value algorithm:

[0049]

[0050] Wherein, K is the key value, |W| is the text vector matrix, T is the matrix transpose symbol, || is the modulus symbol, is the word vector of the target word segmentation.

[0051] To solve the above problems, the application also provides a video retrieval device, the device comprises:

[0052] The lens segmentation module is configured to perform segmentation on each preset video file according to a lens to obtain a first video segment set corresponding to each preset video file.

[0053] The semantic segmentation module is configured to sequentially perform semantic segmentation on each first video segment in the first video segment set to obtain a second video segment set of the corresponding video file.

[0054] The video feature extraction module is configured to sequentially extract video segment features of each second video segment in the second video segment set by using a pre-trained CLIP+LSTM model, and fuse all the video segment features to obtain a video feature of the corresponding video file.

[0055] The text and video feature comparison module is configured to receive a text to be retrieved, extract a text feature of the text to be retrieved by using the pre-trained CLIP+LSTM model, sequentially calculate a feature similarity between the text feature and a video feature of each preset video file, and select a video file corresponding to a feature similarity satisfying a preset similarity condition as a target video file.

[0056] To solve the above problems, the application also provides a computer readable storage medium, the computer readable storage medium stores at least one computer program, the at least one computer program is executed by a processor in an electronic device to realize the video retrieval method described above.

[0057] The embodiment of the application realizes accurate refinement of the video file by sequentially performing lens segmentation and semantic segmentation on the preset video file, extracts the video segment features of the refined second video segment, obtains the video feature of the final video file based on the video segment features, and the video feature reflects the image features of the video file itself. Finally, by calculating the feature similarity between the text feature of the text to be retrieved and the video feature of the video file, the video file satisfying the preset similarity condition is selected as the target video file. This retrieval method is based on the video feature of the video file itself for retrieval and comparison, thereby improving the accuracy of video file retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The flowchart of the video retrieval method provided by an embodiment of the application is shown in the figure;

[0059] Figure 2 A detailed implementation flowchart of one of the steps in the video retrieval method provided by an embodiment of the present application is shown in the figure.

[0060] Figure 3 A detailed implementation flowchart of another of the steps in the video retrieval method provided by an embodiment of the present application is shown in the figure.

[0061] Figure 4 A detailed implementation flowchart of another of the steps in the video retrieval method provided by an embodiment of the present application is shown in the figure.

[0062] Figure 5 A functional module diagram of the video retrieval device provided by an embodiment of the present application is shown in the figure.

[0063] Figure 6 A structural schematic diagram of an electronic device for implementing the video retrieval method provided by an embodiment of the present application is shown in the figure.

[0064] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0066] Embodiments of the present application provide a video retrieval method. The execution subject of the video retrieval method includes but is not limited to at least one of electronic devices such as a server and a terminal, which can be configured to execute the method provided by the embodiments of the present application. In other words, the video retrieval method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0067] Referring to Figure 1 A flowchart of the video retrieval method provided by an embodiment of the present application is shown in the figure.

[0068] In the present embodiment, the video retrieval method includes:

[0069] S1, performing a segmentation operation on each preset video file through a lens to obtain a first video segment set corresponding to each preset video file;

[0070] In the embodiment of the present application, the smart medical platform is taken as an example for illustration. The smart medical platform provides medical aid and medical knowledge popularization to ordinary users by maintaining and publishing a series of medical popular science videos. The preset video file refers to the medical video maintained by the smart medical platform, such as common disease prevention knowledge video, family medical aid common sense video, public health protection video, medical hot event video, etc.

[0071] It can be understood that generally one video text is composed of one or more groups of shots, each shot represents a group of independent meanings, and each preset video file is segmented according to shots, so that one preset video file is divided into a plurality of video segments, and each video segment contains image information of one shot. In this way, the characteristics of each preset video file itself can be better obtained subsequently.

[0072] In the embodiment of the present application, the shot segmentation tool disclosed in OPENCV can be used to segment each preset video file.

[0073] S2, sequentially performing semantic segmentation on each first video segment in the first video segment set to obtain a second video segment set of the corresponding video file;

[0074] It can be understood that if a shot contains rich semantic information, for example, a long shot, each first video segment can be further subdivided to ensure that the semantic of the second video segment obtained by final cutting is more pure and the granularity is not too large, which is beneficial to improving the accuracy of subsequent extraction of video features of the corresponding video file based on the finally cut video segment.

[0075] In detail, referring to Figure 2 As shown in the figure, the S2 comprises:

[0076] S21, recognizing the text of each first video segment and performing sentence segmentation on the text of each first video segment;

[0077] S22, performing sentence vector conversion on each sentence of each first video segment to obtain a sentence vector set of the corresponding first video segment;

[0078] S23, calculating the adjacent window similarity and the skip window similarity between each two sentence vectors in the sentence vector set to obtain corresponding vector similarities, and dividing the sentence corresponding to the vector similarity satisfying the preset similarity threshold into one second video segment;

[0079] S24, collecting all the second video segments to obtain a second video segment set of the corresponding preset video file.

[0080] In the embodiment of the present application, the ASR technology can be used to obtain the video segment text of each first video segment, and after the video segment text is segmented, the semantic segmentation of the first video segment is performed in units of the segmented sentences.

[0081] In another optional embodiment of the present application, the acoustic model can be used to perform speech recognition based on the voice information corresponding to the first video segment to obtain the video segment text corresponding to the first video segment. The acoustic model models the pronunciation of each word to establish a database containing multiple words and the standard pronunciation of each word. The pronunciation of the user at each time is obtained by collecting the pronunciation of the user in the first video segment at each time in the voice information, and then the pronunciation is matched with the words in the database containing multiple words and the standard pronunciation of each word to achieve speech recognition of the voice information and obtain the video segment text.

[0082] In an optional embodiment of the present application, the sentence vector conversion of each segmented sentence can be performed by the following method:

[0083] The segmented words are sequentially obtained for each segmented sentence, and the word vector conversion is performed on each segmented word;

[0084] The word vectors corresponding to each segmented sentence are added to obtain the word vector matrix of each segmented sentence.

[0085] The pooling operation is performed on each word vector matrix to obtain the sentence vector corresponding to each segmented sentence.

[0086] In the embodiment of the present application, the segmented words can be obtained by using a preset standard dictionary for the segmented sentence processing, and the standard dictionary contains multiple standard segmented words. The segmented words can also be obtained by using a segmented word tool, for example, jieba segmentation.

[0087] In the embodiment of the present application, the word vector conversion of each segmented word can be performed by using a model having a word vector conversion function, such as a word2vec model or an NLP (Natural Language Processing) model.

[0088] In an optional embodiment of the present application, the k-max pooling method can be used to perform the pooling operation on the word vector matrix, and the value of k can be predefined, for example, the value of k can be 35%. Since the number of segmented words contained in each segmented sentence is different, preferably, the number of segmented words in each segmented sentence can be rounded up by 35%, and the first K maximum values in each pooling block are taken, for example, if the segmented sentence 1 contains only one segmented word, then 1*35% is rounded up to 1, and if the segmented sentence 2 contains five segmented words, then 5*35% is rounded up to 2.

[0089] In detail, referring to Figure 3 The method comprises the following steps of:

[0090] Step A, taking the first sentence vector in the sentence vector set as a starting point;

[0091] Step B, calculating the adjacent window similarity between the starting point and the sentence vector adjacent to the starting point, and determining whether the adjacent window similarity is greater than a preset similarity threshold;

[0092] When the adjacent window similarity is greater than the preset semantic similarity threshold, step C is performed, in which the starting point and the sentence vector adjacent to the starting point are taken as a temporary video segment;

[0093] Step C1, removing the sentence vector in the temporary video segment from the sentence vector set, and determining whether the sentence vector set after the removal is empty;

[0094] When the sentence vector set after the removal is empty, C11 is performed, in which the temporary video segment is divided into a second video segment, and step E1 is jumped to;

[0095] When the sentence vector set after the removal is not empty, C12 is performed, in which the first sentence vector in the sentence vector set is taken as a starting point, the adjacent window similarity and the skip window similarity between the starting point and the vector in the temporary video segment are calculated, the vector similarity is obtained by weightedly averaging the adjacent window similarity and the skip window similarity, and it is determined whether the vector similarity is greater than a preset similarity threshold;

[0096] When the vector similarity is greater than the preset similarity threshold, C121 is performed, in which the starting point is added to the temporary video segment, and step C1 is returned to;

[0097] When the vector similarity is not greater than the preset similarity threshold, C122 is performed, in which the temporary video segment is divided into a second video segment, the vector corresponding to the second video segment is removed from the vector set, and step E is jumped to;

[0098] When the adjacent window similarity is not greater than the preset similarity threshold, step D is performed, in which the starting point is divided into a second video segment, and the starting point is removed from the sentence vector set;

[0099] Step E, determining whether the sentence vector set after the removal is empty;

[0100] When the sentence vector set after the elimination of the vector is empty, then step E1 is performed, the second video segments are collected to obtain the second video segment set;

[0101] When the sentence vector set after the elimination of the vector is not empty, then step A is returned.

[0102] In the embodiment of the application, the adjacent window similarity and the skip window similarity are similarities between two sentence vectors, i.e., two adjacent window similarities and two skip window similarities. For example, a first video segment text includes sentence vectors S1, S2, S3, S4 and S5, there is an adjacent window similarity between S1 and S2, a skip window similarity between S1 and S3, an adjacent window similarity between S2 and S3, a skip window similarity between S2 and S4, and a skip window similarity between S3 and S5.

[0103] In the embodiment of the application, a pre-trained MLP (Multilayer Perceptron) model can be used to calculate the adjacent window similarity or the skip window similarity between each two sentence vectors.

[0104] In the embodiment of the application, different weights can be assigned to the adjacent window similarity and the skip window similarity in advance, and finally the two are weighted and averaged to obtain the vector similarity.

[0105] In the embodiment of the application, the preset similarity threshold can be set according to actual business conditions.

[0106] Exemplarily, assuming that a first video segment includes four sub-sentences, the corresponding sentence vectors are S1, S2, S3 and S4, and the first video segment is subjected to semantic segmentation, the following several division results can be obtained:

[0107] The first division result includes S1, S2+S3 and S4 three second video segments, the adjacent window similarity between S1 and S2 is less than the preset similarity threshold, so the sub-sentence where S1 is located is divided into an independent second video segment; the adjacent window similarity between S2 and S3 is greater than the preset similarity threshold, and the vector similarity corresponding to the adjacent window similarity between S2 and S3 and the skip window similarity between S2 and S4 is less than the preset similarity threshold, so S2 and S3 are divided into an independent second video segment, and S4 is divided into an independent video segment;

[0108] The second division result includes three second video segments S1+S2, S3 and S4, the adjacent window similarity between S1 and S2 is greater than the preset similarity threshold, the adjacent window similarity between S2 and S3 and the vector similarity corresponding to the jump window similarity between S1 and S3 are less than the preset similarity threshold, S1 and S2 are divided into an independent second video segment, the adjacent window similarity between S3 and S4 is less than the preset similarity threshold, the sentence where S3 is located is divided into an independent second video segment, and the sentence corresponding to S4 is divided into an independent second video segment.

[0109] The third division result includes two video segments S1+S2+S3 and S4, the adjacent window similarity between S1 and S2 is greater than the preset similarity threshold, the adjacent window similarity between S2 and S3 and the vector similarity corresponding to the jump window similarity between S1 and S3 are greater than the preset similarity threshold, the adjacent window similarity between S3 and S4 and the vector similarity corresponding to the jump window similarity between S2 and S4 are less than the preset similarity threshold, S4 is an independent second video segment, and S1, S2 and S3 are an independent second video segment.

[0110] It should be noted that the above is only an example, and there can be multiple division results for S1, S2, S3 and S4.

[0111] S3, a pre-trained CLIP+LSTM model is used to extract video segment features of each second video segment in the second video segment set in turn, and video features of a corresponding video file are obtained by fusing all the video segment features.

[0112] In the embodiment of the application, the pre-trained CLIP+LSTM model includes a CLIP part (Contrastive Language-Image Pre-training) and an LSTM part (Long Short-Term Memory, long short-term memory network).

[0113] In detail, the pre-trained CLIP+LSTM model is used to extract video segment features of each second video segment in the second video segment set in turn, and video features of a corresponding video file are obtained by fusing all the video segment features, which includes:

[0114] According to the time sequence, video frame extraction is performed on each second video segment in turn to obtain a video frame set of each second video segment.

[0115] The CLIP part in the pre-trained CLIP+LSTM model is used to extract frame feature vectors of each video frame in the video frame set in turn.

[0116] performing convolution operation on all frame feature vectors of each second video segment by using the LSTM part in the pre-trained CLIP+LSTM model to obtain a video segment feature corresponding to the second video segment;

[0117] performing pooling operation on all video segment features corresponding to the preset video file to obtain a video feature of the preset video file.

[0118] In an optional embodiment of the present application, four video frames are extracted per second from each second video segment according to the chronological order to form a video frame set of the second video segment.

[0119] In the embodiments of the present application, video frames of each second video segment are sequentially extracted in units of the second video segment, a video segment feature corresponding to each video frame is extracted based on the frame feature vector of the video frame by using the pre-trained CLIP+LSTM model, and finally a video feature corresponding to the video file is obtained based on the video segment features of all second video segments, wherein the video feature reflects the image and picture features of the video itself.

[0120] S4, receiving a to-be-retrieved text, extracting a text feature of the to-be-retrieved text by using the pre-trained CLIP+LSTM model, sequentially calculating feature similarities between the text feature and video features of each preset video file, and selecting a video file corresponding to a feature similarity satisfying a preset similarity condition as a target video file.

[0121] In the embodiments of the present application, the same model, i.e., the pre-trained CLIP+LSTM model, can be used to extract the text feature of the to-be-retrieved text. In this way, the text feature of the to-be-retrieved text and the video feature of the video file can be mapped to the same representation space, which facilitates subsequent comparison and calculation of the two.

[0122] In detail, referring to FIG. 2, Figure 4 extracting the text feature of the to-be-retrieved text by using the pre-trained CLIP+LSTM model, comprises:

[0123] S41, performing word segmentation on the to-be-retrieved text to obtain one or more retrieval words, and obtaining a word vector of each retrieval word;

[0124] S42, performing splicing on the word vector of each retrieval word by using the CLIP part in the pre-trained CLIP+LSTM model to obtain a text vector matrix;

[0125] S43, sequentially selecting one retrieval word as a target word, and calculating a key value of the target word according to the word vector of the target word and the text vector matrix.

[0126] S44, selecting a preset number of retrieval words as feature words in the order of the key value from large to small;

[0127] S45, concatenating the word vectors of the feature words to obtain the text feature of the text to be retrieved.

[0128] In detail, since the text to be retrieved contains a large number of retrieval words, not every retrieval word is a feature of the text to be retrieved, therefore, the plurality of retrieval words need to be screened, the embodiment of the application selects one retrieval word from the plurality of retrieval words as a target word, calculates the key value of the target word according to the word vector of the target word and the text vector matrix, and screens the feature word representative of the text to be retrieved according to the key value, so as to obtain the text feature of the text to be retrieved.

[0129] Specifically, the key value of the target word is calculated according to the word vector of the target word and the text vector matrix, comprising:

[0130] The key value of the target word is calculated by using the following key value algorithm:

[0131]

[0132] Wherein, K is the key value, |W| is the text vector matrix, T is the matrix transpose symbol, || is the modulus symbol, is the word vector of the target word.

[0133] In the embodiment of the application, a preset number of retrieval words are selected from the plurality of retrieval words as feature words in the order of the key value of each retrieval word from large to small.

[0134] For example, the plurality of retrieval words include retrieval word A, retrieval word B and retrieval word C, wherein the key value of retrieval word A is 80, the key value of retrieval word B is 70, and the key value of retrieval word C is 30, if the preset number is 2, then retrieval word A and retrieval word B are selected as feature words in the order of the key value from large to small, and the word vectors of the retrieval word A and the retrieval word B are concatenated to obtain the text feature of the text to be retrieved.

[0135] In the embodiment of the application, the feature similarity can be obtained by calculating the cosine similarity between the text feature and the video feature of each video file.

[0136] Further, the cosine similarity between the text feature and the video feature of each second video segment can be calculated, and the second video segment closest to the text feature can be matched by using the calculated cosine similarity, thereby further improving the accuracy of video retrieval.

[0137] The embodiment of the present application realizes accurate refinement of the video file by sequentially performing lens segmentation and semantic segmentation on the preset video file, extracts the video segment feature of the refined second video segment, obtains the final video feature of the video file based on the video segment feature, and finally selects the video file meeting the preset similarity condition as the target video file by calculating the feature similarity between the text feature of the text to be retrieved and the video feature of the video file. This retrieval method is based on the video feature of the video file itself for retrieval comparison, thereby improving the accuracy of video file retrieval.

[0138] As shown in Figure 5 is a functional module diagram of a video retrieval device provided by an embodiment of the present application.

[0139] The video retrieval device 100 can be installed in an electronic device. According to the implemented functions, the video retrieval device 100 includes a lens segmentation module 101, a semantic segmentation module 102, a video feature extraction module 103, and a text and video feature comparison module 104. The modules of the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and are stored in the memory of the electronic device.

[0140] In the present embodiment, the functions of each module / unit are as follows:

[0141] The lens segmentation module 101 is configured to perform segmentation on each preset video file by lens to obtain a first video segment set corresponding to each preset video file;

[0142] The semantic segmentation module 102 is configured to sequentially perform semantic segmentation on each first video segment in the first video segment set to obtain a second video segment set of the corresponding video file;

[0143] The video feature extraction module 103 is configured to use a pre-trained CLIP+LSTM model to sequentially extract the video segment feature of each second video segment in the second video segment set, and fuse all the video segment features to obtain the video feature of the corresponding video file;

[0144] The text and video feature comparison module 104 is configured to receive a text to be retrieved, extract a text feature of the text to be retrieved by using the pre-trained CLIP+LSTM model, sequentially calculate a feature similarity between the text feature and a video feature of each of the preset video files, and select a video file corresponding to a feature similarity satisfying a preset similarity condition as a target video file.

[0145] In detail, each module in the video retrieval device 100 in the embodiment of the present application adopts the same technical means as the video retrieval method described in the above Figures 1 to 4 , and can produce the same technical effects, which will not be described here again.

[0146] As shown in Figure 6 , it is a structural schematic diagram of an electronic device for implementing a video retrieval method according to an embodiment of the present application.

[0147] The electronic device 1 can include a processor 10, a memory 11 and a bus, and can further include a computer program, such as a video retrieval program, stored in the memory 11 and executable on the processor 10.

[0148] The memory 11 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, for example, a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, for example, a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 can include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 can be used not only to store application software and various data installed in the electronic device 1, such as the code of the video retrieval program, but also to temporarily store data that has been output or will be output.

[0149] The processor 10 can be composed of integrated circuits in some embodiments, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, combinations of various control chips, etc. The processor 10 is the control unit of the electronic device, which connects various components of the entire electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (such as a video retrieval program, etc.), and calls data stored in the memory 11 to perform various functions and process data of the electronic device 1.

[0150] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 11, the at least one processor 10, etc.

[0151] Figure 6 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 6 The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0152] For example, although not shown, the electronic device 1 can also include a power supply (such as a battery) for powering various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so as to realize functions such as charge management, discharge management, and power consumption management through the power management device. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, etc. The electronic device 1 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0153] Further, the electronic device 1 can also include a network interface, which can optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually used to establish a communication connection between the electronic device 1 and other electronic devices.

[0154] Optionally, the electronic device 1 can also include a user interface, which can be a display, an input unit such as a keyboard, and optionally a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

[0155] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by the structure.

[0156] The video retrieval program stored in the memory 11 in the electronic device 1 is a combination of a plurality of instructions, which, when executed in the processor 10, can achieve:

[0157] Perform segmentation on each preset video file through a lens to obtain a first video segment set corresponding to each preset video file;

[0158] Perform semantic segmentation on each first video segment in the first video segment set in turn to obtain a second video segment set of the corresponding video file;

[0159] Using a pre-trained CLIP+LSTM model, extract the video segment features of each second video segment in the second video segment set in turn, and fuse all the video segment features to obtain the video features of the corresponding video file;

[0160] Receive a text to be retrieved, extract the text features of the text to be retrieved using the pre-trained CLIP+LSTM model, and calculate the feature similarity between the text features and the video features of each preset video file in turn, and select the video file corresponding to the feature similarity that meets the preset similarity condition as the target video file.

[0161] Further, the modules / units integrated in the electronic device 1, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM).

[0162] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can realize the following steps when executed by a processor of an electronic device:

[0163] Perform segmentation on each preset video file through a lens to obtain a first video segment set corresponding to each preset video file;

[0164] Perform semantic segmentation on each first video segment in the first video segment set in sequence to obtain a second video segment set of the corresponding video file;

[0165] Extract video segment features of each second video segment in the second video segment set in sequence by using a pre-trained CLIP+LSTM model, and fuse all the video segment features to obtain video features of the corresponding video file;

[0166] Receive a text to be searched, extract text features of the text to be searched by using the pre-trained CLIP+LSTM model, calculate feature similarities between the text features and video features of each preset video file in sequence, and select a video file corresponding to a feature similarity meeting a preset similarity condition as a target video file.

[0167] In addition, each function module in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software function modules.

[0168] It is obvious for those skilled in the art that the application is not limited to the details of the above exemplary embodiments, and the application can be realized in other specific forms without departing from the spirit or essential characteristics of the application.

[0169] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the application. Any additional reference signs in the claims should not be regarded as limiting the claims involved.

[0170] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0171] The embodiments of the present application can acquire and process relevant data based on holographic projection technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results.

[0172] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video retrieval method, characterized in that: The method comprises: Performing a segmentation operation on each preset video file through a lens to obtain a first video segment set corresponding to each preset video file; Sequentially performing semantic segmentation on each first video segment in the first video segment set to obtain a second video segment set corresponding to the video file; Using a pre-trained CLIP+LSTM model, sequentially extracting video segment features of each second video segment in the second video segment set, and fusing all the video segment features to obtain video features of the corresponding video file; Receiving a text to be retrieved, extracting text features of the text to be retrieved using the pre-trained CLIP+LSTM model, sequentially calculating feature similarities between the text features and video features of each of the preset video files, and selecting a video file corresponding to the feature similarity that meets a preset similarity condition as a target video file; Among them, the semantic segmentation is performed on each first video segment in the first video segment set in turn to obtain a second video segment set corresponding to the video file, including: identifying the text of each first video segment and dividing the text of each first video segment into sentences; performing sentence vector conversion on each sentence of each first video segment to obtain a sentence vector set corresponding to the first video segment; calculating the adjacent window similarity and the jump window similarity between every two sentence vectors in the sentence vector set to obtain the corresponding vector similarity, and dividing the sentences corresponding to the vector similarity that meets the preset similarity threshold into a second video segment; and collecting all the second video segments to obtain a second video segment set corresponding to the preset video file.

2. The video retrieval method according to claim 1, wherein: The performing sentence vector conversion on each sentence of each first video segment to obtain a sentence vector set corresponding to the first video segment includes: Segment each of the sentences in turn, and convert each of the segmented words into a word vector; Add the word vectors corresponding to each of the clauses to obtain a word vector matrix for each of the clauses; A pooling operation is performed on each of the word vector matrices to obtain a sentence vector corresponding to each of the clauses.

3. The video retrieval method according to claim 1, wherein: The step of calculating the adjacent window similarity and the skip window similarity between each two sentence vectors in the sentence vector set to obtain corresponding vector similarities, and dividing sentences corresponding to vector similarities that meet a preset similarity threshold into a second video segment includes: Step A: taking the first sentence vector in the sentence vector set as the starting point; Step B: calculating the adjacent window similarity between the starting point and the sentence vectors adjacent to the starting point, and determining whether the adjacent window similarity is greater than a preset similarity threshold; When the adjacent window similarity is greater than the preset semantic similarity threshold, step C is performed, and the starting point and the sentence vector adjacent to the starting point are used as a temporary video segment; Step C1: removing the sentence vectors in the temporary video segment from the sentence vector set, and determining whether the sentence vector set after removing the vectors is empty; When the sentence vector set after the vector is removed is empty, executing C11, dividing the temporary video segment into a second video segment, and jumping to step E1; If the sentence vector set after the vector is removed is not empty, executing C12, taking the first sentence vector in the sentence vector set as a starting point, calculating the adjacent window similarity and the jump window similarity between the starting point and the vector in the temporary video segment, performing weighted averaging on the adjacent window similarity and the jump window similarity to obtain a vector similarity, and determining whether the vector similarity is greater than a preset similarity threshold; When the vector similarity is greater than the preset similarity threshold, executing C121, adding the starting point to the temporary video segment, and returning to step C1; When the vector similarity is not greater than the preset similarity threshold, executing C122, dividing the temporary video segment into a second video segment, removing the vector corresponding to the second video segment from the vector set, and jumping to step E; When the adjacent window similarity is not greater than the preset similarity threshold, executing step D, dividing the starting point into a second video segment, and removing the starting point from the sentence vector set; Step E: Determine whether the sentence vector set after removing the vector is empty; When the sentence vector set after removing the vector is empty, executing step E1, collecting the second video segments to obtain the second video segment set; When the sentence vector set after removing the vector is not empty, return to the above step A.

4. The video retrieval method according to claim 1, wherein: The method of sequentially extracting the video segment features of each second video segment in the second video segment set by using the pre-trained CLIP+LSTM model and fusing all the video segment features to obtain the video features of the corresponding video file includes: Extracting video frames from each of the second video segments in sequence according to time to obtain a video frame set for each of the second video segments; Using the CLIP part of the pre-trained CLIP+LSTM model, sequentially extracting a frame feature vector of each video frame in the video frame set; Using the LSTM part of the pre-trained CLIP+LSTM model, a convolution operation is performed on all frame feature vectors of each second video segment to obtain a video segment feature corresponding to the second video segment; A pooling operation is performed on all video segment features corresponding to the preset video file to obtain video features of the preset video file.

5. The video retrieval method according to claim 1, wherein: The method of extracting text features of the text to be retrieved by using the pre-trained CLIP+LSTM model includes: Segmenting the text to be searched to obtain one or more search segments, and obtaining a word vector for each search segment; Using the CLIP part of the pre-trained CLIP+LSTM model, the word vectors of each search word are concatenated to obtain a text vector matrix; Selecting a search word as a target word in turn, and calculating the key value of the target word according to the word vector of the target word and the text vector matrix; Selecting a preset number of search terms as feature terms in descending order of the key values; The word vectors of the feature segmentations are concatenated to obtain the text features of the text to be retrieved.

6. The video retrieval method according to claim 5, wherein: The step of calculating the key value of the target word segmentation according to the word vector of the target word segmentation and the text vector matrix includes: The key value of the target word is calculated using the following key value algorithm: in, is the key value, is the text vector matrix, is the matrix transpose symbol, To find the modulo symbol, The word vector of the target word.

7. A video retrieval device for implementing the video retrieval method according to any one of claims 1 to 6, characterized in that: The device comprises: a shot segmentation module, configured to perform a segmentation operation on each preset video file by shot, to obtain a first video segment set corresponding to each preset video file; a semantic segmentation module, configured to sequentially perform semantic segmentation on each first video segment in the first video segment set to obtain a second video segment set corresponding to the video file; a video feature extraction module, configured to sequentially extract video segment features of each second video segment in the second video segment set using a pre-trained CLIP+LSTM model, and fuse all the video segment features to obtain video features of the corresponding video file; The text and video feature comparison module is used to receive the text to be retrieved, use the pre-trained CLIP+LSTM model to extract the text features of the text to be retrieved, calculate the feature similarity between the text features and the video features of each preset video file in turn, and select the video file corresponding to the feature similarity that meets the preset similarity conditions as the target video file.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the video retrieval method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video retrieval method according to any one of claims 1 to 6 is implemented.

Images