Video retrieval method and device, computer equipment and storage medium
By combining multimodal feature representation of text and video features in the video retrieval model, the problem of low accuracy of existing text video retrieval methods is solved, and more efficient video retrieval is achieved.
Patent Information
- Application Number
- CN202411543277.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-18
AI Technical Summary
The existing text video retrieval method has low retrieval accuracy and requires manual filtering to obtain the required video clips.
By inputting the search text into the text encoder of the video search model, the text feature representation is obtained; inputting the video data into the video encoder of the video search model, the multimodal feature representation is obtained; then compute the feature similarity and select the largest video data as the target Video data.
It improves the accuracy of video retrieval, reduces the need for manual screening, and achieves more accurate video data retrieval.
Smart Images
Figure CN119474459A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of multimedia information processing, natural language processing and machine learning technology, and in particular to a video retrieval method, device, computer equipment and storage medium. Background Art
[0002] Text video retrieval technology refers to finding the best matching video clip from a large amount of video data in the video library by inputting a text description (keywords or short sentences). This technology is widely used in public safety, traffic management, emergency response and other aspects in the construction of smart cities.
[0003] For example, among a large amount of video data captured by a camera installed at an intersection where a traffic accident occurred, by inputting text descriptions such as "vehicle collision" or "traffic accident", the large amount of video data captured by the camera is retrieved to obtain a video clip of the traffic accident.
[0004] However, current text video retrieval technology usually uses the image features of video frames in video clips as video features for video retrieval. Video retrieval based on a single video feature may retrieve too many video clips, and manual further screening of the video clips is required to obtain the required video clips.
[0005] Therefore, the existing text video retrieval methods have low retrieval accuracy. Summary of the invention
[0006] The embodiments of the present invention provide a video retrieval method, apparatus, computer equipment and storage medium to solve the problem of low retrieval accuracy of existing text video retrieval methods.
[0007] A video retrieval method, the method comprising: Get the video data from the retrieved text and video library; Inputting the search text into the text encoder of the video retrieval model to obtain a text feature representation of the search text; Inputting the video data into the video encoder of the video retrieval model to obtain a multimodal feature representation of each of the video data; Calculating feature similarities between the text feature representation and each of the multimodal feature representations respectively; The video data with the largest feature similarity is selected as the retrieved target video data.
[0008] In the above method, optionally, the step of inputting the video data into a video encoder of the video retrieval model to obtain a multimodal feature representation of each video data includes: Performing multimodal feature extraction on the video data to obtain first video features of each of the video data in multiple dimensions; Mapping the first video features of multiple dimensions into target feature representations of the same dimensions through a fully connected layer; Performing concatenation processing on the target feature representation to obtain a multimodal feature vector; The multimodal feature vector is converted into a multimodal feature representation through a conversion model.
[0009] In the above method, optionally, the multi-dimensional first video features include at least two or more features among image features, voice features, action features and streamer features.
[0010] In the above method, optionally, the image feature is obtained by: Filtering a first number of video frame images from the video data as target video images; The target video image is subjected to feature extraction by a feature extraction model to obtain the image features.
[0011] In the above method, optionally, the video retrieval model is trained in the following manner: Obtaining training video samples and training text samples annotated with each of the training video samples; Inputting the training video sample into the video encoder to obtain training video features of the training video sample in multiple dimensions, and performing feature fusion on the training video features in multiple dimensions to obtain fused training representation; Inputting the training text sample into the text encoder to obtain a training text representation of the training text sample; The contrast loss value of the training text feature representation and the fused training representation is calculated to determine that the video retrieval model has completed training when the contrast loss value reaches a preset training condition.
[0012] In the above method, optionally, the training video sample is obtained by: Get the original video data; According to the content difference between consecutive video frames in the original video data, the video data is cut into a plurality of the training video samples.
[0013] In the above method, optionally, the training text sample is obtained by: Performing multimodal feature extraction on the training video sample to obtain second video features of the training video sample in multiple dimensions; Performing feature fusion on the multi-dimensional second video features through a fully connected layer to obtain context features of the training video sample; Inputting the context features into a text annotation model to obtain preliminary annotated text; The video text is annotated again manually based on the preliminary annotated text to obtain the training text sample.
[0014] A video retrieval device, comprising: A data acquisition module is used to acquire video data from the search text and video library; A first representation extraction module, used for inputting the video sample into the text encoder of the video retrieval model to obtain a text feature representation of the retrieved text; A second representation extraction module, inputting the video data into a video encoder of the video retrieval model to obtain a multimodal feature representation of each of the video data; A similarity calculation module, used to calculate the feature similarity between the text feature representation and each of the multimodal feature representations; The video data determination module is used to select the video data with the greatest feature similarity as the retrieved target video data.
[0015] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-mentioned video retrieval methods when executing the computer program.
[0016] A computer-readable storage medium stores a computer program, wherein the computer program implements any of the above-mentioned video retrieval methods when executed by a processor.
[0017] The above-mentioned video retrieval method, apparatus, computer equipment and storage medium obtain the text feature representation of the retrieval text by inputting the retrieval text into the text encoder of the video retrieval model; inputting the video data into the video encoder of the video retrieval model to obtain the multimodal feature representation of each video data; then, respectively calculating the feature similarity between the text feature representation and each multimodal feature representation, and selecting the video data with the largest feature similarity as the retrieved target video data. It can be seen that the present invention retrieves the video data by extracting the multimodal feature representation of the video data through the video encoder, which can achieve the purpose of improving the video retrieval accuracy compared to retrieving the video data through a single image feature in the video data. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0019] Figure 1 is a flow chart of an implementation of a video retrieval method in one embodiment of the present invention; Figure 2 is a partial implementation flow chart of a video retrieval method in one embodiment of the present invention; Figure 3 is a partial implementation flow chart of a video retrieval method in one embodiment of the present invention; Figure 4 is a partial implementation flow chart of a video retrieval method in one embodiment of the present invention; Figure 5 is a partial implementation flow chart of a video retrieval method in one embodiment of the present invention; Figure 6 is a partial implementation flow chart of a video retrieval method in one embodiment of the present invention; Figure 7 is a structural schematic diagram of a video retrieval device in one embodiment of the present invention; Figure 8 It is a structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0021] The present invention discloses a video retrieval method, apparatus, computer equipment and storage medium, which obtains the text feature representation of the retrieval text by inputting the retrieval text into the text encoder of the video retrieval model; and obtains the multimodal feature representation of each video data by inputting the video data into the video encoder of the video retrieval model; and then respectively calculates the feature similarity between the text feature representation and each multimodal feature representation, and selects the video data with the largest feature similarity as the retrieved target video data. It can be seen that the present invention retrieves the video data by extracting the multimodal feature representation of the video data through the video encoder, which can achieve the purpose of improving the video retrieval accuracy compared to retrieving the video data through a single image feature in the video data. The following is an explanation through specific embodiments.
[0022] In one embodiment, if Figure 1 FIG. 1 is a flowchart of a video retrieval method disclosed in an embodiment of the present invention. The method is applicable to electronic devices with data processing capabilities, such as mobile phones, laptops, tablet computers, and servers. The method in this embodiment includes the following steps: S101: Acquire the video data in the search text and video library.
[0023] The search text is a description text input by the user, indicating that the user expects to retrieve video data corresponding to the description text, and the video library is used to store a large amount of video data, which can be a large amount of complete videos or a large amount of video clips cut from the complete video data. When the search text input by the user is received, the video data in the video library is extracted to perform the subsequent step of retrieving the video data based on the search text.
[0024] For example, taking the video data captured by a city camera as an example, when the user enters the search text "vehicle collision video", the video data stored in the video library corresponding to the camera is obtained, and these video clips are retrieved based on the search text "vehicle collision video". In this way, it is possible to obtain the search text and the video data in the video library.
[0025] S102: Input the search text into the text encoder of the video retrieval model to obtain a text feature representation of the search text.
[0026] The text encoder is used to extract features from the retrieved text to obtain text feature representation of the retrieved text.
[0027] In a specific implementation, the text encoder in this embodiment includes but is not limited to a CLIP text encoder, an encoder based on RNN or CNN, etc. The retrieved text is input into the CLIP text encoder, the RNN-based encoder or the CNN-based encoder to obtain a text feature representation corresponding to the retrieved text.
[0028] S103: Input the video data into the video encoder of the video retrieval model to obtain a multimodal feature representation of each video data.
[0029] It should be understood that there is a large amount of video data in the video library. To search the large amount of video data in the video library, it is necessary to extract the multimodal feature representation of each video data separately.
[0030] In a specific implementation, in this embodiment, feature representations of each video data can be extracted from multiple dimensions, and then the feature representations of multiple dimensions are fused to obtain a multimodal feature representation of each video data.
[0031] In one embodiment, the above step S103 can be implemented in the following manner: Figure 2 As shown: S201: Perform multimodal feature extraction on video data to obtain first video features of each video data in multiple dimensions.
[0032] Among them, performing multimodal feature extraction on video data refers to performing feature extraction on video data from multiple dimensions to obtain a first video feature of each video data in each dimension.
[0033] In a specific implementation, the multiple dimensions in this embodiment include but are not limited to two or more dimensions of image dimension, voice dimension, action dimension and light streaming dimension. That is to say, in this embodiment, feature extraction can be performed on video data from two or more dimensions of image dimension, voice dimension, action dimension and light streaming dimension respectively to obtain the first video features of each video data in multiple dimensions.
[0034] S202: Mapping first video features of multiple dimensions into target feature representations of the same dimensions through a fully connected layer.
[0035] It should be understood that to fuse first video features of different dimensions, it is first necessary to map the first video features of different dimensions to the same dimension, that is, map the first video features of multiple dimensions to target feature representations of the same dimension, and then perform feature fusion on each target feature representation.
[0036] In a specific implementation, in this embodiment, the dimensions of the first video features of multiple dimensions can be transformed through one or more fully connected layers to respectively map the first video features of different dimensions to target feature representations of the same dimension.
[0037] S203: Concatenate the target feature representation to obtain a multimodal feature vector.
[0038] In a specific implementation, in this embodiment, different target feature representations can be concatenated one by one to obtain a multimodal feature vector.
[0039] Among them, the concatenated multimodal feature vector can be shown as follows: (len_1+len_2…+len_n)* dim Among them, len_1, len_2, len_n indicate that there are n target feature representations, and dim indicates the dimension of the multimodal feature vector.
[0040] S204: Convert the multimodal feature vector into a multimodal feature representation through a conversion model.
[0041] In a specific implementation, in this embodiment, the multimodal features can be input into the transformer model to obtain the final multimodal feature representation through the multi-layer self-attention mechanism in the transformer model.
[0042] S104: Calculate the feature similarity between the text feature representation and each multimodal feature representation respectively.
[0043] In a specific implementation, in this embodiment, the cosine similarity between the text feature representation and the multimodal feature representation can be used as the feature similarity between the text feature representation and the multimodal feature representation.
[0044] Input the text feature representation and multimodal feature representation into the following formulas respectively: in, represents the text feature representation, represents multimodal feature representation, Indicates feature similarity.
[0045] S105: Select the video data with the largest feature similarity as the retrieved target video data.
[0046] In a specific implementation, in this embodiment, a video data with the greatest feature similarity can be selected as the retrieved target video data, or, in this embodiment, multiple video data with the greatest feature similarity can be selected as the retrieved target video data. In this embodiment, there is no limit on the number of retrieved target video data.
[0047] For example, when there are multiple video data corresponding to the same feature similarity, and the feature similarity is the maximum value, it is determined that the multiple video data are all target video data.
[0048] For another example, when the feature similarities corresponding to each video data are different, three video data with the largest feature similarities are selected as target video data in descending order of feature similarities.
[0049] In summary, the present invention discloses a video retrieval method, which obtains a text feature representation of the retrieval text by inputting the retrieval text into the text encoder of the video retrieval model; inputting the video data into the video encoder of the video retrieval model to obtain a multimodal feature representation of each video data; then, respectively calculating the feature similarity between the text feature representation and each multimodal feature representation, and selecting the video data with the largest feature similarity as the retrieved target video data. It can be seen that the present invention retrieves the video data by extracting the multimodal feature representation of the video data through the video encoder, which can achieve the purpose of improving the video retrieval accuracy compared to retrieving the video data through a single image feature in the video data.
[0050] based on Figure 1 In a specific implementation, the multi-dimensional first video feature in this embodiment includes at least two or more features of image features, voice features, action features and streamer features.
[0051] In one embodiment, the image features in this embodiment are obtained by the following method: Figure 3 As shown: S301: Filter out a first number of video frame images from video data as target video images.
[0052] It should be noted that the durations of different video data may be different. The longer the duration of the video data, the more video frame images it contains, and the shorter the duration of the video data, the fewer video frame images it contains. In order to ensure that the extracted image features are in the same dimension, in this embodiment, the number of video frames of the video data is obtained from each video data as the target video image. Among them, the first number can be set according to actual needs, such as 50, 60, etc., and is not limited in this implementation.
[0053] In a specific implementation, in this embodiment, a first number of video frame images can be extracted from each video data as target video images by random sampling, or, in this embodiment, a first number of video frame images can be extracted from each video data as target video images by average sampling, thereby screening out a first number of video frame images from the video data as target video images.
[0054] For example, taking the video data containing 180 frames of video frame images as an example, one frame of video frame images is randomly extracted from the 180 frames of video frame images, and then one frame of video frame images is randomly collected from the remaining 179 frames of video frame images, and so on, until 60 frames of video frame images are extracted from the 180 frames of video frame images, and the 60 frames of video frame images are used as the target video images.
[0055] For another example, taking the case where the video data includes 180 video frame images, one video frame image is randomly extracted from every three consecutive video frame images in the 180 video frame images as the target video image.
[0056] It should be noted that the above is only a partial implementation method of selecting a first number of video frame images from video data as target video images in this embodiment, and this embodiment does not limit the specific implementation method of selecting a first number of video frame images from video data as target video images.
[0057] S302: Extract features of the target video image through a feature extraction model to obtain image features.
[0058] Each target video image is input into the feature extraction model respectively to obtain the frame-level features corresponding to each target video image, and the frame-level features are spliced to obtain the image features of the video data.
[0059] In a specific implementation, the feature extraction model in this embodiment includes but is not limited to a convolutional neural network (CNN), a self-editing device, etc. Each target video image is input into the convolutional neural network to obtain the frame-level features corresponding to each target video image, and then the frame-level features are spliced into the image features of the video data. In this way, the feature extraction model can be used to extract features of the target video image to obtain image features.
[0060] In one embodiment, the speech features in this embodiment can be obtained in the following manner: first, separate the audio data from the video data, and then extract the speech features of the audio data, wherein the speech features include but are not limited to one or more of time domain features, frequency domain features, and time-frequency domain features.
[0061] In one embodiment, the motion features of the target video image in this embodiment include, but are not limited to, extracting the motion features of the target video image by methods such as optical flow method, background difference method, and frame difference method.
[0062] In one embodiment, the optical flow features of the target video image in this embodiment include but are not limited to being extracted by any one of the Lucas-Kanade method, the Horn-Schunck method and the Farneback method.
[0063] based on Figure 1 In the specific implementation, the video retrieval model in this embodiment can be trained in the following way: Figure 4 As shown: S401: Acquire training video samples and training text samples annotated with each training video sample.
[0064] In one embodiment, the training video samples in this embodiment can be obtained by the following method: Figure 5 As shown: S501: Acquire original video data.
[0065] The original video data may be a complete video recorded by a video recording device within a certain period of time, such as a complete video recorded by a surveillance camera within 24 hours.
[0066] S502: Cut the video data into a plurality of training video samples according to content differences between consecutive video frame images in the original video data.
[0067] In a specific implementation, in this embodiment, the content difference between consecutive video frame images in the original video data can be obtained by any one of the image difference method (Difference-based) or the histogram thresholding method (Histogram Thresholding), and then the original video data is divided into multiple video segments, namely, training video data.
[0068] For example, in the continuous video frame images in the original video data, there is a difference between whether there are pedestrians or not, from no pedestrians entering the field of view of the surveillance camera to two consecutive video frame images in which pedestrians enter the field of view of the surveillance camera. Based on these two video frame images, the original video data is divided into two video segments, namely, training video data.
[0069] S402: Input the training video sample into the video encoder to obtain the training video features of the training video sample in multiple dimensions, and perform feature fusion on the training video features in multiple dimensions to obtain a fused training representation.
[0070] Multimodal features are extracted from training video samples through a video encoder to obtain training video features in multiple dimensions for each training video sample. The training video features in multiple dimensions are then mapped to training feature representations of the same dimension through a fully connected layer. The training feature representations are concatenated to obtain a fused training representation. Finally, the multimodal feature vector is converted into a multimodal feature representation through a conversion model.
[0071] S403: Input the training text sample into the text encoder to obtain a training text representation of the training text sample.
[0072] The text encoder is used to extract features from the training text to obtain text feature representations of the training text.
[0073] In a specific implementation, the text encoder in this embodiment includes but is not limited to a CLIP text encoder, an encoder based on RNN or CNN, etc. The training text is input into the CLIP text encoder, the RNN-based encoder or the CNN-based encoder to obtain a text feature representation corresponding to the training text.
[0074] S404: Calculate the contrast loss value of the training text feature representation and the fusion training representation, so as to determine that the video retrieval model has completed training when the contrast loss value reaches a preset training condition.
[0075] In one embodiment, the training condition in this embodiment may be that when the contrast loss value converges, it is determined that the video retrieval model has completed training.
[0076] In another embodiment, the training condition in this embodiment may be that when the contrast loss value reaches below a preset loss threshold, it is determined that the video retrieval model has completed training.
[0077] In summary, in this embodiment, the video retrieval model is trained through contrast learning, so that the model has better stability, which is beneficial to improving the training effect of the video retrieval model and improving the retrieval accuracy of the video retrieval model.
[0078] based on Figure 4 In the specific implementation, the training text samples in this embodiment can be obtained by the following method: Figure 6 As shown: S601: Perform multi-dimensional feature extraction on training video samples to obtain second video features of the training video samples in multiple dimensions.
[0079] The multimodal feature extraction of the training video samples refers to extracting features of the training video samples from multiple dimensions to obtain a second video feature of each training video sample in each dimension.
[0080] In a specific implementation, the multiple dimensions in this embodiment include but are not limited to two or more dimensions among image dimension, voice dimension, action dimension and light streaming dimension. That is to say, in this embodiment, feature extraction can be performed on training video samples from two or more dimensions among image dimension, voice dimension, action dimension and light streaming dimension respectively to obtain second video features of each training video sample in multiple dimensions.
[0081] S602: Perform feature fusion on multi-dimensional second video features through a fully connected layer to obtain context features of training video samples.
[0082] It should be understood that in order to fuse second video features of different dimensions, it is first necessary to map the second video features of different dimensions to the same dimension, that is, map the second video features of multiple dimensions to training feature representations of the same dimension, and then perform feature fusion on each training feature representation to obtain the contextual features of the training video samples.
[0083] S603: Input the context features into the text annotation model to obtain preliminary annotated text.
[0084] The context features are input into the text annotation model to obtain preliminary annotation text, and the training video samples are annotated by the preliminary annotation text.
[0085] In a specific implementation, the text annotation model in this embodiment includes but is not limited to a Video Caption model. The context features are input into the Video Caption model to obtain preliminary annotation text, and the training video samples are annotated using the preliminary annotation text.
[0086] S604: manually annotating the video text again based on the preliminary annotated text to obtain a training text sample.
[0087] It is understandable that the preliminary annotated text obtained through the text annotation model may contain standard errors, or the annotation may not be accurate enough. Therefore, the preliminary annotated text can be modified manually to finally obtain a training text sample.
[0088] In summary, in this embodiment, the training video text is annotated by both model annotation and manual annotation, which not only solves the problem of inaccurate model annotation, but also makes up for the problem of low efficiency of manual annotation.
[0089] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0090] In one embodiment, a video retrieval device is provided, which corresponds one-to-one to the video retrieval method in the above embodiment. Figure 7 As shown, the video retrieval device includes a data acquisition module, a first representation extraction module, a second representation extraction module, a similarity calculation module and a video data determination module. The functional modules are described in detail as follows: The data acquisition module 701 is used to acquire the video data in the search text and video library; A first representation extraction module 702, used to input the video sample into the text encoder of the video retrieval model to obtain the text feature representation of the retrieved text; The second representation extraction module 703 inputs the video data into a video encoder of a video retrieval model to obtain a multimodal feature representation of each video data; A similarity calculation module 704 is used to calculate the feature similarity between the text feature representation and each multimodal feature representation; The video data determination module 705 is used to select the video data with the greatest feature similarity as the retrieved target video data.
[0091] In summary, the video retrieval device in this embodiment obtains the text feature representation of the retrieval text by inputting the retrieval text into the text encoder of the video retrieval model; inputs the video data into the video encoder of the video retrieval model to obtain the multimodal feature representation of each video data; then, the feature similarity between the text feature representation and each multimodal feature representation is calculated respectively, and the video data with the largest feature similarity is selected as the retrieved target video data. It can be seen that the present invention retrieves the video data by extracting the multimodal feature representation of the video data through the video encoder, which can achieve the purpose of improving the video retrieval accuracy compared to retrieving the video data through a single image feature in the video data.
[0092] In one implementation, the second representation extraction module 703 may be used to: Perform multimodal feature extraction on the video data to obtain first video features of each video data in multiple dimensions; Mapping the first video features of multiple dimensions into target feature representations of the same dimensions through a fully connected layer; Concatenate the target feature representation to obtain a multimodal feature vector; The multimodal feature vector is converted into a multimodal feature representation through a conversion model.
[0093] In one implementation, the multi-dimensional first video feature includes at least two or more features of an image feature, a voice feature, an action feature, and a streamer feature.
[0094] In one implementation, the image features are obtained by: Filtering a first number of video frame images from the video data as target video images; The feature extraction model is used to extract features from the target video image to obtain image features.
[0095] In one implementation, the video retrieval model is trained in the following manner: Obtain training video samples and training text samples annotated with each training video sample; Inputting the training video sample into the video encoder to obtain the training video features of the training video sample in multiple dimensions, and performing feature fusion on the training video features in multiple dimensions to obtain fused training representation; Input the training text sample into the text encoder to obtain the training text representation of the training text sample; The contrast loss value of the training text feature representation and the fusion training representation is calculated to determine that the video retrieval model has completed training when the contrast loss value reaches a preset training condition.
[0096] In one implementation, the training video samples are obtained by: Get the original video data; According to the content difference between consecutive video frames in the original video data, the video data is cut into multiple training video samples.
[0097] In one implementation, the training text sample is obtained by: Performing multimodal feature extraction on the training video samples to obtain second video features of the training video samples in multiple dimensions; The multi-dimensional second video features are fused through a fully connected layer to obtain context features of the training video samples; Input the context features into the text annotation model to obtain preliminary annotated text; The video text is manually annotated again based on the preliminary annotated text to obtain training text samples.
[0098] For the specific definition of the video retrieval device, please refer to the definition of the video retrieval method above, which will not be repeated here. Each module in the above video retrieval device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0099] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video retrieval method is implemented.
[0100] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved: Get the video data from the retrieved text and video library; Input the retrieved text into the text encoder of the video retrieval model to obtain the text feature representation of the retrieved text; Input the video data into the video encoder of the video retrieval model to obtain a multimodal feature representation of each video data; Calculate the feature similarity between the text feature representation and each multimodal feature representation respectively; The video data with the largest feature similarity is selected as the retrieved target video data.
[0101] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video retrieval method in the above embodiment is implemented.
[0102] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0103] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0104] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A video retrieval method, characterized in that: The method comprises: Get the video data from the retrieved text and video library; Inputting the search text into the text encoder of the video retrieval model to obtain a text feature representation of the search text; Inputting the video data into the video encoder of the video retrieval model to obtain a multimodal feature representation of each of the video data; Calculating feature similarities between the text feature representation and each of the multimodal feature representations respectively; The video data with the largest feature similarity is selected as the retrieved target video data.
2. The video retrieval method according to claim 1, characterized in that: The step of inputting the video data into the video encoder of the video retrieval model to obtain a multimodal feature representation of each of the video data comprises: Performing multimodal feature extraction on the video data to obtain first video features of each of the video data in multiple dimensions; Mapping the first video features of multiple dimensions into target feature representations of the same dimensions through a fully connected layer; Performing concatenation processing on the target feature representation to obtain a multimodal feature vector; The multimodal feature vector is converted into a multimodal feature representation through a conversion model.
3. The video retrieval method according to claim 2, characterized in that: The multi-dimensional first video features include at least two or more features of image features, voice features, action features and light stream features.
4. The video retrieval method according to claim 3, characterized in that: The image features are obtained in the following way: Filtering a first number of video frame images from the video data as target video images; The target video image is subjected to feature extraction by a feature extraction model to obtain the image features.
5. The video retrieval method according to claim 1, characterized in that: The video retrieval model is trained in the following way: Obtaining training video samples and training text samples annotated with each of the training video samples; Inputting the training video sample into the video encoder to obtain training video features of the training video sample in multiple dimensions, and performing feature fusion on the training video features in multiple dimensions to obtain fused training representation; Inputting the training text sample into the text encoder to obtain a training text representation of the training text sample; The contrast loss value of the training text feature representation and the fused training representation is calculated to determine that the video retrieval model has completed training when the contrast loss value reaches a preset training condition.
6. The video retrieval method according to claim 5, characterized in that: The training video samples are obtained in the following manner: Get the original video data; According to the content difference between consecutive video frames in the original video data, the video data is cut into a plurality of the training video samples.
7. The video retrieval method according to claim 5, characterized in that: The training text sample is obtained in the following manner: Performing multimodal feature extraction on the training video sample to obtain second video features of the training video sample in multiple dimensions; Performing feature fusion on the multi-dimensional second video features through a fully connected layer to obtain context features of the training video sample; Inputting the context features into a text annotation model to obtain preliminary annotated text; The video text is annotated again manually based on the preliminary annotated text to obtain the training text sample.
8. A video retrieval device, characterized in that: include: A data acquisition module is used to acquire video data from the search text and video library; A first representation extraction module, used for inputting the video sample into the text encoder of the video retrieval model to obtain a text feature representation of the retrieved text; A second representation extraction module, inputting the video data into a video encoder of the video retrieval model to obtain a multimodal feature representation of each of the video data; A similarity calculation module, used to calculate the feature similarity between the text feature representation and each of the multimodal feature representations; The video data determination module is used to select the video data with the greatest feature similarity as the retrieved target video data.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the video retrieval method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video retrieval method according to any one of claims 1 to 7 is implemented.