Video clip retrieval method and device, computer equipment and storage medium
The Q-Former model integrates video and text features and uses loop gating to calculate the similarity value, which solves the problem of inaccurate positioning in traditional video clip search methods and achieves more accurate video clip retrieval.
Patent Information
- Application Number
- CN202411679581.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-03-14
AI Technical Summary
Traditional video clip search methods have a large correlation between frames, resulting in inaccurate positioning of video clips, which in turn leads to inaccurate search results.
The Q-Former model is used to feature fusion of the video feature vector of the video clip and the text feature vector of the query text to obtain the second video feature vector of the video clip. Then, the loop output vector of the video clip is calculated using loop gating, and the similarity value of the video clip and the query text is calculated to determine the target video clip.
By integrating the multimodal characteristics of text and video, the correspondence between text and video is purified, which significantly improves the accuracy of video clip retrieval, allowing users to retrieve relevant video clips more accurately.
Smart Images
Figure CN119622030A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of artificial intelligence, financial technology and medical health technology, and in particular to a video clip retrieval method, device, computer equipment and storage medium. Background Art
[0002] With the explosive growth of video content, how to quickly retrieve video clips has become crucial. For example, in the field of financial technology, how to quickly and accurately retrieve explanation videos of different insurances; in the field of medical health, how to quickly and accurately retrieve diagnosis videos of different diseases, etc.
[0003] Traditional video clip retrieval is done by adding text descriptions to frames in the video and matching the retrieval text with the frame text to obtain retrieval results. However, this method results in inaccurate positioning of video clips due to the high correlation between frames, and conflicts will occur when retrieving similar images, resulting in inaccurate retrieval. Summary of the invention
[0004] The present invention provides a video segment retrieval method, device, computer equipment and storage medium to solve the technical problem that inaccurate video segment positioning leads to inaccurate video retrieval results.
[0005] In a first aspect, a video clip retrieval method is provided, comprising: determining first video feature vectors of N video clips and a first text feature vector of a query text, where N is a positive integer; inputting the first video feature vectors and the first text feature vectors of the N video clips into a query transformer Q-Former model, respectively, to obtain second video feature vectors of the N video clips; based on the second video feature vectors and the first text feature vectors of the N video clips, respectively calculating cyclic output vectors of the N video clips by adopting a cyclic gating method; determining N similarity values between the cyclic output vectors of the N video clips and the first text feature vector; and determining a target video clip corresponding to the query text based on the similarity values of the N video clips.
[0006] In a second aspect, a video clip retrieval device based on artificial intelligence is provided, including: a determination module, used to determine the first video feature vectors of N video clips and the first text feature vector of a query text; a calculation module, used to input the first video feature vectors and the first text feature vectors of the N video clips into a query transformer Q-Former model, respectively, to obtain the second video feature vectors of the N video clips; the calculation module is also used to calculate the cyclic output vectors of the N video clips respectively based on the second video feature vectors and the first text feature vectors of the N video clips, using a cyclic gating method; the determination module is also used to determine N similarity values between the cyclic output vectors of the N video clips and the first text feature vector; and the query module is used to determine the target video clip corresponding to the query text based on the similarity values of the N video clips.
[0007] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned video clip retrieval method when executing the computer program.
[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video clip retrieval method are implemented.
[0009] In the scheme implemented by the above-mentioned video clip retrieval method, device, computer equipment and storage medium, the first video feature vector and the first text feature vector of N video clips are feature fused through the Q-Former model to obtain the second video feature vector of the N video clips; then, based on the second video feature vector and the first text feature vector, the cyclic output vector of the N videos is calculated by cyclic gating; based on the similarity between the cyclic output vector and the first text feature vector, only the target video clip of the query text is determined. In the present application, the features of text and video are fused through the Q-Former model, so that the output feature vector takes into account the multimodal features of text and video, and further the correspondence between text and video is purified through cyclic gating, so that the user can retrieve the corresponding video clip more accurately after entering the text. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0011] Figure 1is a schematic diagram of an application environment of a video clip retrieval method according to an embodiment of the present invention;
[0012] Figure 2 is a schematic diagram of a process of a video clip retrieval method according to an embodiment of the present invention;
[0013] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;
[0014] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S20;
[0015] Figure 5 yes Figure 4 A schematic flow chart of a specific implementation of step S30;
[0016] Figure 6 yes Figure 5 A schematic flow chart of a specific implementation of steps S32 and S33;
[0017] Figure 7 yes Figure 2 A schematic flow chart of a specific implementation of step S50;
[0018] Figure 8 is a structural diagram of a video clip retrieval device in one embodiment of the present invention;
[0019] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;
[0020] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] The video clip retrieval method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can receive the query text input by the user through the client, and determine the first text feature vector of the query text according to the query text, input the first video feature vector and the first text feature vector of the N video clips into the query transformer Q-Former model, and obtain the second video feature vector of the N video clips; based on the second video feature vector and the first text feature vector of the N video clips, the cyclic output vector of the N video clips is calculated respectively by cyclic gating; the N similarity values between the cyclic output vector of the N video clips and the first text feature vector are determined; based on the similarity value and the similarity threshold of the N video clips, the target video clip corresponding to the query text is determined. In the present application, the features of text and video are integrated through the Q-Former model, so that the output feature vector takes into account the multimodal features of text and video, and further purifies the correspondence between text and video through cyclic gating, so that the user can retrieve the corresponding video clip more accurately after entering the query text. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.
[0023] See also Figure 2 As shown, Figure 2 A schematic flow chart of a video clip retrieval method provided by an embodiment of the present invention includes the following steps:
[0024] S10: Determine the first video feature vectors of N video clips and the first text feature vector of the query text, where N is a positive integer.
[0025] The video clip retrieval method provided by the present invention can be applied to retrieval websites or intelligent retrieval customer service or intelligent assistants and other intelligent retrieval engines in various application scenarios. The intelligent retrieval engine is usually implemented through a server, which can receive the query text input by the user in real time. For example, in the field of insurance applications, users usually need to understand the specific content of a certain type of insurance. At this time, they can use the intelligent retrieval engine to retrieve video clips related to this type of insurance; for another example, in the medical field, users want to understand the specific use of a certain type of medicine. They can use the intelligent retrieval engine to retrieve video clips related to the instructions for use of this type of medicine; for another example, various morning meeting videos, consultation videos, learning videos, and training videos, users only need to enter the query text to quickly retrieve the relevant video clips, greatly improving work efficiency.
[0026] For example, the query text may be "Ping An Ping An Fu product introduction". After receiving the query text, the intelligent search engine needs to determine the first text feature vector of the query text and the first video feature vector of the video clip. The above-mentioned Ping An Ping An Fu refers to the name of an insurance product.
[0027] Specifically, after receiving the query text, the intelligent retrieval engine first performs text segmentation on the query text, and then inputs the segmented text into the Bidirectional Encoder Representations from Transformers (BERT) model. The BERT model outputs a feature vector representing the feature information of the query text, which is also the first text feature vector.
[0028] Then, the first video feature vectors of the N video clips are determined. Figure 3 As shown, determining the first video feature vectors of N video clips in step S10 includes the following steps:
[0029] S11: extracting frames from the original video at every preset time period to obtain N groups of images containing a preset number of frames, where one group of images corresponds to one video segment.
[0030] S12: Input N groups of images into the ViT model respectively to obtain a feature vector of each frame image in each group of images.
[0031] S13: Determine first video feature vectors of N video clips based on the feature vector of each frame of the N groups of images.
[0032] For steps S11-S13, a preset number of frames are extracted from the original video at preset time intervals (for example, 5 frames are extracted every half second or 10 frames are extracted every second, etc., and the 5 frames of images correspond to a half-second video segment. The embodiment of the present application is explained below by taking 5 frames of images for half a second as an example). The 5 frames of images are input into a Vision Transformer (ViT) model, and the ViT model outputs a feature vector for each frame of the image. The feature vectors of the 5 frames of images are subjected to a maximum pooling operation to obtain a first video feature vector with each half second as a video segment.
[0033] It is worth noting that the ViT model in the embodiment of the present application is different from the traditional ViT model. The traditional ViT model divides a single frame image into patches of equal size, tiles them from left to right and from top to bottom, and adds a positional embedding feature vector after tiling, that is, the position information encoding of the patch in the picture is added. In the embodiment of the present application, in addition to positional embedding, a frame embedding feature vector is also added. That is, after inputting N groups of images into the ViT model in step S12, the positional embedding feature vector and frame embedding feature vector of each frame image in each group of images are determined. Among them, the frame embedding feature vector is determined based on the relative time and absolute time of each group of images. The relative time represents the frame order of the current frame image in the current group, and the absolute time represents the frame order of the current group of images in the original video. Then, based on the positional embedding feature vector and frame embedding feature vector of each frame image, the feature vector of each frame image is obtained.
[0034] It should be understood that the reason why half a second is used as a video segment is that the granularity of half a second is more precise than 1 second in expressing video segment retrieval, thereby making the retrieval more accurate.
[0035] After the first video feature vectors of the N video clips are obtained in step S13, the vector dimension is D, which can be represented as an N*D matrix. The N*D matrix is further linearly transformed and multiplied with a D*D learnable parameter matrix, and the output matrix size remains unchanged, which is still N*D.
[0036] S20: Inputting the first video feature vectors and the first text feature vectors of the N video clips into the Q-Former model respectively to obtain the second video feature vectors of the N video clips.
[0037] The video feature vector of the video clip and the text feature vector of the query text are input into the Q-Former model, and the cross-attention mechanism of the Q-Former model makes the text features focus on the video features, so as to achieve the alignment between the text features and the video features, thereby achieving the effect of retrieving videos through text. Figure 4 As shown, a specific feature alignment solution is provided. In S20, the second video feature vectors of N video clips are obtained, which specifically includes the following steps S21-S22:
[0038] S21: Calculate the cross-attention weight of each video clip based on the first video feature vector and the first text feature vector of each video clip.
[0039] S22: Determine a second video feature vector for each video segment based on the cross-attention weight of each video segment and the first video feature vector.
[0040] In steps S21-S22, the first text feature vector is used as Q in the Q-Former model, and the first video feature vector of the video clip is used as K and V in the Q-Former model. The cross-attention weight is calculated based on Q, K, and V, and the cross-attention weight is multiplied by the first video feature vector of the video clip to obtain the second feature vector of the video clip.
[0041] S30: Based on the second video feature vectors and the first text feature vectors of the N video clips, the cyclic output vectors of the N video clips are respectively calculated by adopting a cyclic gating method.
[0042] Among them, recurrent gating is an effective variant of recurrent neural networks, which solves the gradient vanishing and gradient exploding problems by introducing a gating mechanism, thereby effectively processing long-term dependencies in sequences and reducing unnecessary information, thereby improving the performance and efficiency of recurrent neural networks.
[0043] In some embodiments of the present application, Figure 5 As shown, a specific cyclic gating scheme is provided. In S30, based on the second video feature vectors and the first text feature vectors of the N video clips, the cyclic output vectors of the N video clips are calculated respectively by using the cyclic gating method, which specifically includes the following steps S31-S33:
[0044] S31: Map the first text feature vector to the same dimension as the first video feature vector to obtain a second text feature vector.
[0045] S32: Based on the second video feature vector and the second text feature vector of each video segment, an input gating vector, a gating vector of a forget gate, and a current input memory of each video segment are calculated.
[0046] S33: Based on the second video feature vector of each video clip, the input gating vector, the gating vector of the forget gate and the current input memory, a cyclic output vector of each video clip is obtained.
[0047] In steps S31-S33, since the dimension of the first text feature vector output by the BERT model may be different from that of the video feature vector, the first text feature vector is first converted into a second text feature vector of the same dimension as the video feature vector, i.e., 1*D. The input gating vector, the gating vector of the forget gate, and the current input memory are further calculated based on the second video feature vector and the second text feature vector of the video clip, and finally a cyclic output vector of a cycle is obtained based on the second video feature vector, the input gating vector, the gating vector of the forget gate, and the current input memory.
[0048] Specifically, the second video feature vector is taken as H, the second text feature vector is taken as X, and the input gating is calculated:
[0049] I t =σ(X t W xi +H t-1 W hi +b i )
[0050] Among them, W is the learnable parameter matrix, b is the bias term, and X and H are multiplied by different parameter matrices W respectively, and then the bias term is added and input into the sigmoid function to obtain the input gating vector I.
[0051] Furthermore, the second video feature vector is used as H, the second text feature vector is used as X, but different W and b are used, and the sigmoid function is input to calculate the gating vector F of the forget gate:
[0052] F t =σ(X t W xf +H t-1 W hf +b f )
[0053] Furthermore, the second video feature vector is used as H, the second text feature vector is used as X, but different W and b are used, and the tanh function is input to calculate the current input memory C:
[0054] C~ t =tanh(X t W xc +H t-1 W hc +b c )
[0055] Finally, a loop output vector H' of a loop is calculated: H'=H*F+I*C.
[0056] In some embodiments, the process of calculating the loop output vector H' is looped k times to obtain the final target loop output vector. Figure 6 As shown, step S32 can be specifically implemented as step S321, and step S33 can be specifically implemented as step S331 and step S332:
[0057] S321: Based on the second video feature vector and the second text feature vector of each video clip, k parameter matrices and bias items are selected to obtain k input gating vectors, k gating vectors of forget gates and k current input memories of each video clip.
[0058] S331: Determine k output vectors of each video segment based on the second video feature vector of each video segment, k input gating vectors, gating vectors of k forget gates, and k current input memories.
[0059] S332: Select a target output vector for each video segment from the k output vectors of each video segment.
[0060] In steps S321-S332, the second video feature vector and the second text feature vector of each video clip are looped k times through cyclic gating, so that the part of the query text corresponding to the video clip gradually converges, the correspondence between the query text and the video clip is strengthened, and finally N target loop output vectors are obtained.
[0061] S40: Determine N similarity values between the loop output vectors of the N video segments and the first text feature vector.
[0062] In some embodiments, the loop output vectors of N video clips are respectively concatenated with the first text feature vector to obtain N concatenated feature vectors, and the N concatenated feature vectors are input into a two-layer MLP layer, and a similarity value is outputted through a sigmoid function.
[0063] The loop output vector used for splicing is a target loop output vector output after each video clip is looped K times.
[0064] S50: Determine a target video segment corresponding to the query text based on the similarity values of the N video segments.
[0065] In some embodiments, based on the similarity values of the N video segments, a video segment with the greatest similarity is selected as the target video segment.
[0066] In other embodiments, by setting a similarity threshold, video clips with a similarity greater than the similarity threshold are screened out, and then the target video clip is determined from the screened video clips based on a preset rule. Figure 7As shown, a method for determining a target video segment is shown, that is, in step S50, based on the similarity values of N video segments, determining the target video segment corresponding to the query text, specifically comprising the following steps:
[0067] S51: Determine M video segments among the N video segments whose similarity values are greater than a similarity threshold.
[0068] S52: Calculate the number of consecutive frames in the M video segments and the average similarity of the M video segments, and calculate the decision value.
[0069] S53: The video segment with the largest decision value is used as the target video segment.
[0070] In steps S51-S53, M video clips with similarity values greater than the similarity threshold are screened out from the N video clips, the number of consecutive frames F of each video clip in the M video clips and the average similarity value S of the M video clips are calculated, and finally O=W1*F+W2*S is calculated, and the video clip with the maximum value of O is selected as the target video clip.
[0071] It is worth noting that W1 and W2 are manually set weight values, satisfying W1 + W2 = 1. The larger W1 is, the video segment is biased towards the video duration, and the larger W2 is, the video segment is biased towards the similarity with the query text.
[0072] It can be seen that in the above scheme, the features of text and video are integrated through the Q-Former model, so that the output feature vector takes into account the multimodal features of text and video, and further purifies the correspondence between text and video through recurrent gating, so that users can retrieve the corresponding video clips more accurately after entering text.
[0073] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0074] In one embodiment, a video segment retrieval device is provided, which corresponds to the video segment retrieval method in the above embodiment. Figure 8 As shown, the video clip retrieval device includes a determination module 101, a calculation module 102 and a query module 103. The functional modules are described in detail as follows:
[0075] A determination module 101, configured to determine first video feature vectors of N video clips and a first text feature vector of a query text;
[0076] A calculation module 102, configured to input the first video feature vectors and the first text feature vectors of the N video clips into a query transformer Q-Former model respectively to obtain second video feature vectors of the N video clips;
[0077] The calculation module 102 is further used to calculate the cyclic output vectors of the N video clips respectively based on the second video feature vectors and the first text feature vectors of the N video clips in a cyclic gating manner;
[0078] The determination module 101 is further used to determine N similarity values between the cyclic output vectors of the N video clips and the first text feature vector;
[0079] The query module 103 is used to determine the target video segment corresponding to the query text based on the similarity values and similarity thresholds of the N video segments.
[0080] In one embodiment, the determination module 101 is specifically used to extract frames from the original video at every preset time period to obtain N groups of images containing a preset number of frames, where one group of images corresponds to one video clip; the N groups of images are respectively input into the ViT model to obtain a feature vector of each frame image in each group of images; based on the feature vector of each frame image in the N groups of images, the first video feature vector of the N video clips is determined.
[0081] In one embodiment, the determination module 101 is specifically used to determine the position embedding feature vector and the frame embedding feature vector of each frame image in each group of images, wherein the frame embedding feature vector is determined based on the relative time and the absolute time of each group of images, wherein the relative time represents the frame order of the current frame image in the current group, and the absolute time represents the frame order of the current group of images in the original video; based on the position embedding feature vector and the frame embedding feature vector of each frame image, a feature vector of each frame image is obtained.
[0082] In one embodiment, the calculation module 102 is specifically used to calculate the cross-attention weight of each video clip based on the first video feature vector and the first text feature vector of each video clip; based on the cross-attention weight of each video clip and the first video feature vector, determine the second video feature vector of each video clip.
[0083] In one embodiment, the calculation module 102 is specifically used to map the first text feature vector to the same dimension as the first video feature vector to obtain a second text feature vector; based on the second video feature vector and the second text feature vector of each video clip, calculate the input gating vector, the gating vector of the forget gate, and the current input memory of each video clip; based on the second video feature vector, the input gating vector, the gating vector of the forget gate, and the current input memory of each video clip, obtain the loop output vector of each video clip.
[0084] In one embodiment, the calculation module 102 is specifically used to select k parameter matrices and bias items based on the second video feature vector and the second text feature vector of each video clip to obtain k input gating vectors, k gating vectors of forget gates and k current input memories of each video clip; determine k loop output vectors of each video clip based on the second video feature vector, k input gating vectors, k gating vectors of forget gates and k current input memories of each video clip; select the target loop output vector of each video clip from the k loop output vectors of each video clip.
[0085] In one embodiment, the query module 103 is specifically used to determine M video segments among N video segments whose similarity values are greater than a similarity threshold; calculate the number of consecutive frames in the M video segments and the average similarity of the M video segments, and calculate a decision value; and use the video segment with the largest decision value as the target video segment.
[0086] The present invention provides a video clip retrieval device, which integrates the features of text and video through a Q-Former model, so that the output feature vector takes into account the multimodal features of text and video, and further purifies the correspondence between text and video through recurrent gating, so that users can retrieve the corresponding video clip more accurately after inputting text.
[0087] The specific definition of the video clip retrieval device can be found in the definition of the video clip retrieval method above, which will not be repeated here. Each module in the above video clip retrieval device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0088] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a video clip retrieval processing method server side.
[0089] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a video clip retrieval method
[0090] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:
[0091] Determine the first video feature vectors of N video clips and the first text feature vector of the query text, where N is a positive integer;
[0092] Inputting the first video feature vectors and the first text feature vectors of the N video clips into a query transformer Q-Former model respectively to obtain second video feature vectors of the N video clips;
[0093] Based on the second video feature vectors and the first text feature vectors of the N video clips, respectively calculating the cyclic output vectors of the N video clips in a cyclic gating manner;
[0094] Determining N similarity values between the loop output vectors of the N video clips and the first text feature vector;
[0095] Based on the similarity values and similarity thresholds of the N video segments, a target video segment corresponding to the query text is determined.
[0096] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0097] Determine the first video feature vectors of N video clips and the first text feature vector of the query text, where N is a positive integer;
[0098] Inputting the first video feature vectors and the first text feature vectors of the N video clips into a query transformer Q-Former model respectively to obtain second video feature vectors of the N video clips;
[0099] Based on the second video feature vectors and the first text feature vectors of the N video clips, respectively calculating the cyclic output vectors of the N video clips in a cyclic gating manner;
[0100] Determining N similarity values between the loop output vectors of the N video clips and the first text feature vector;
[0101] Based on the similarity values and similarity thresholds of the N video segments, a target video segment corresponding to the query text is determined.
[0102] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0103] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0104] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0105] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A video clip retrieval method, characterized in that: include: Determine the first video feature vectors of N video clips and the first text feature vector of the query text, where N is a positive integer; Inputting the first video feature vectors and the first text feature vectors of the N video clips into a query transformer Q-Former model respectively to obtain second video feature vectors of the N video clips; Based on the second video feature vectors and the first text feature vectors of the N video clips, respectively calculating the cyclic output vectors of the N video clips in a cyclic gating manner; Determining N similarity values between the loop output vectors of the N video clips and the first text feature vector; Based on the similarity values of the N video segments, a target video segment corresponding to the query text is determined.
2. The method according to claim 1, characterized in that The determining of the first video feature vectors of the N video clips includes: Extract frames from the original video at every preset time period to obtain N groups of images containing a preset number of frames, where one group of images corresponds to one video clip; Input N groups of images into the ViT model respectively, and obtain the feature vector of each frame in each group of images; Based on the feature vector of each frame of the N groups of images, first video feature vectors of the N video clips are determined.
3. The method according to claim 2, characterized in that The N groups of images are input into the ViT model respectively to obtain the feature vector of each frame image in each group of images, including: Determine a position embedding feature vector and a frame embedding feature vector of each frame image in each group of images, wherein the frame embedding feature vector is determined based on a relative time and an absolute time of each group of images, wherein the relative time represents a frame sequence of a current frame image in the current group, and the absolute time represents a frame sequence of the current group of images in the original video; Based on the position embedding feature vector and the frame embedding feature vector of each frame image, a feature vector of each frame image is obtained.
4. The method according to claim 1, characterized in that The first video feature vectors and the first text feature vectors of the N video clips are respectively input into a query transformer Q-Former model to obtain second video feature vectors of the N video clips, including: Calculating a cross-attention weight for each video segment based on a first video feature vector and a first text feature vector for each video segment; Based on the cross-attention weight of each video segment and the first video feature vector, a second video feature vector of each video segment is determined.
5. The method according to claim 1, characterized in that The method of calculating the cyclic output vectors of the N video clips respectively based on the second video feature vector and the first text feature vector of the N video clips by using a cyclic gating method includes: Mapping the first text feature vector to the same dimension as the first video feature vector to obtain a second text feature vector; Based on the second video feature vector and the second text feature vector of each video clip, calculating the input gating vector, the gating vector of the forget gate, and the current input memory of each video clip; Based on the second video feature vector of each video clip, the input gating vector, the gating vector of the forget gate and the current input memory, a cyclic output vector of each video clip is obtained.
6. The method according to claim 5, characterized in that The step of calculating the input gating vector, the gating vector of the forget gate, and the current input memory of each video segment based on the second video feature vector and the second text feature vector of each video segment comprises: Based on the second video feature vector and the second text feature vector of each video clip, k parameter matrices and bias items are selected to obtain k input gating vectors, k gating vectors of forget gates, and k current input memories of each video clip; The step of obtaining a cyclic output vector for each video segment based on the second video feature vector of each video segment, the input gating vector, the gating vector of the forget gate, and the current input memory includes: Determine k cyclic output vectors for each video segment based on the second video feature vector of each video segment, the k input gating vectors, the gating vectors of the k forget gates, and the k current input memories; A target cyclic output vector for each video segment is selected from the k cyclic output vectors for each video segment.
7. The method according to any one of claims 1 to 6, characterized in that: The step of determining the target video segment corresponding to the query text based on the similarity values of the N video segments includes: Determine M video segments whose similarity values are greater than a similarity threshold among the N video segments; Calculate the number of consecutive frames in the M video clips and the average similarity of the M video clips, and calculate the decision value; The video segment with the largest decision value is taken as the target video segment.
8. A video clip retrieval device, characterized in that: include: A determination module, used to determine first video feature vectors of N video clips and a first text feature vector of a query text; A calculation module, used for inputting the first video feature vectors and the first text feature vectors of the N video clips into a query transformer Q-Former model respectively to obtain second video feature vectors of the N video clips; The calculation module is further used to calculate the cyclic output vectors of the N video clips respectively based on the second video feature vectors and the first text feature vectors of the N video clips in a cyclic gating manner; The determination module is further used to determine N similarity values between the cyclic output vectors of the N video clips and the first text feature vector; The query module is used to determine the target video segment corresponding to the query text based on the similarity values of the N video segments.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the video clip retrieval method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the video clip retrieval method according to any one of claims 1 to 7 are implemented.