Video content retrieval method and device
By segmenting the video scenes and identifying the scene categories of video clips using the CLIP model, the problem of unstable video content retrieval in the prior art is solved, and efficient and accurate video content retrieval is achieved.
Patent Information
- Application Number
- CN202410995255.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
Existing video content retrieval technology is difficult to accurately understand the complex semantics and abstract concepts in video content, and is affected by factors such as lighting changes, noise, and occlusion, resulting in unstable search results, limiting the scope of application and effect.
By segmenting the searched videos, a consistent video clip is generated, and a CLIP-based image processing model is used to identify the scene category of the video clip based on the search statement, and the target video clip matching the search statement is determined.
It improves the accuracy and speed of video content retrieval, has strong robustness and scalability, and can adapt to video data sets of different sizes and complex retrieval needs.
Smart Images

Figure CN118939834A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of Internet technology, and in particular, to a video content retrieval method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] Video content retrieval is a field that has emerged with the rapid growth of video materials and the continuous development of information retrieval technology. With the vigorous development of the Internet and various video sharing platforms, the massive amount of unstructured video clips has made how to effectively organize, manage and retrieve video information a key issue in the field of information retrieval. For intelligent editing, the rapid retrieval of specific content can also play a key role in editing the wonderful content of the video. Traditional text- and keyword-based methods face challenges in video retrieval because the richness and dynamics of video content are difficult to accurately express with a single text description.
[0003] Since video content may contain complex semantics and abstract concepts, existing technologies often cannot accurately understand these contents, thus affecting the accuracy of retrieval results. In addition, video content is affected by many factors, such as lighting changes, noise, occlusion, and dynamic changes in the picture. These factors may cause the features extracted from the video to be not robust enough, affecting the stability of the retrieval results. These limitations affect the application scope and effect of video content retrieval technology, and a faster and more effective solution is needed.
[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention
[0005] The embodiments of the present application provide a video content retrieval method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.
[0006] One aspect of an embodiment of the present application provides a video content retrieval method, the method comprising:
[0007] Get the video to be retrieved and the search sentence;
[0008] Segmenting the video to be retrieved into multiple video clips with different scenes;
[0009] Identify each video clip according to the search statement to obtain a scene category corresponding to each video clip, wherein the scene category is used to indicate a semantic relationship between the video clip and the search statement;
[0010] According to the scene category corresponding to each video segment, a target video segment matching the search statement is determined from the multiple video segments.
[0011] Optionally, identifying each video clip according to the search statement to obtain a scene category corresponding to each video clip includes:
[0012] Get CLIP-based image processing models;
[0013] Through the CLIP-based image processing model, each video clip is identified according to the search statement to obtain the scene category corresponding to each video clip.
[0014] Optionally, identifying each video clip according to the search statement to obtain a scene category corresponding to each video clip includes:
[0015] Extracting text features corresponding to the search sentence from the search sentence, and determining a plurality of picture content categories according to the text features;
[0016] Each video segment is identified according to the multiple picture content categories to obtain a scene category corresponding to each video segment.
[0017] Optionally, identifying each video clip according to the multiple picture content categories to obtain a scene category corresponding to each video clip includes:
[0018] For each video segment, extract frames from the video segment to obtain multiple frames of images;
[0019] Extracting features from each frame of image to obtain image features, and determining the similarity between the frame of image and each picture content category according to the image features;
[0020] Determining a target content category corresponding to the frame image according to a similarity between the frame image and each screen content category;
[0021] The scene category corresponding to the video segment is determined according to the target content category corresponding to all the frame images in the video segment.
[0022] Optionally, determining the target content category corresponding to the frame image according to the similarity between the frame image and each screen content category includes:
[0023] According to the similarity between the frame image and each screen content text, the screen content category with the highest similarity is determined as the target content category corresponding to the frame image.
[0024] Optionally, determining the scene category corresponding to the video clip according to the target content category corresponding to all the frame images in the video clip includes:
[0025] determining one or more positive categories from the plurality of picture content categories;
[0026] Determining the number of frame images belonging to the positive category according to the target content category corresponding to all frame images in the video clip;
[0027] The scene category corresponding to the video clip is determined according to the number of frame images belonging to the positive category.
[0028] Optionally, the step of performing scene segmentation on the video to be retrieved to obtain a plurality of video clips with different scenes includes:
[0029] Extracting features from each frame of the video to be retrieved to obtain image feature information;
[0030] Determine a feature difference between two adjacent frame images according to the image feature information;
[0031] According to the feature difference between the two adjacent frame images, the video to be retrieved is segmented into scenes to obtain a plurality of video clips with different scenes.
[0032] Another aspect of an embodiment of the present application provides a video content retrieval device, the device comprising:
[0033] A search statement acquisition module is used to acquire the video to be searched and the search statement;
[0034] A scene segmentation module, used for performing scene segmentation on the video to be retrieved to obtain multiple video clips with different scenes;
[0035] A scene category determination module, used to identify each video segment according to the search sentence, and obtain a scene category corresponding to each video segment, wherein the scene category is used to indicate a semantic relationship between the video segment and the search sentence;
[0036] The target video segment determination module is used to determine a target video segment matching the search statement from the multiple video segments according to the scene category corresponding to each video segment.
[0037] Another aspect of an embodiment of the present application provides a computer device, including:
[0038] at least one processor; and
[0039] a memory communicatively coupled to the at least one processor;
[0040] Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video content retrieval method as described above.
[0041] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the video content retrieval method described above is implemented.
[0042] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the video content retrieval method described above when executed by a processor.
[0043] The embodiment of the present application adopts the above technical solution, which may include the following advantages: by performing scene cutting before analyzing the video content, the content theme of each generated video clip is consistent, which greatly facilitates the subsequent content analysis work and can save a lot of computing power. When analyzing the video content, the scene category corresponding to the video clip is obtained by identifying the video clip according to the search statement. The scene category can indicate the semantic relationship between the video clip and the search statement, so that the video clip related to the search content can be effectively extracted from the video according to the scene category, which not only improves the speed of video content retrieval, but also improves the accuracy of video content retrieval. It has strong robustness and scalability and can adapt to video data sets of different sizes and complex retrieval requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0045] Figure 1 The following schematically shows an operating environment diagram of the video content retrieval method according to the first embodiment of the present application;
[0046] Figure 2 A flowchart of a video content retrieval method according to Embodiment 1 of the present application is schematically shown;
[0047] Figure 3 A modular block diagram of a video content retrieval method according to an embodiment of the present application is schematically shown;
[0048] Figure 4 A block diagram schematically shows a video content retrieval device according to the second embodiment of the present application; and
[0049] Figure 5 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0051] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0052] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.
[0053] First, the following terms are explained:
[0054] Intelligent editing: refers to the process of automatically analyzing and editing audio and video materials using artificial intelligence technologies such as computer vision, speech recognition and deep learning. This technology can identify the characters, scenes, and shots in the video, and automatically determine the editing order and timeline based on these recognition results, thereby realizing an automated editing process.
[0055] Video content retrieval: By using computer vision, speech processing, natural language processing and other technologies, the video content retrieval system can automatically analyze video data, extract key features, and perform efficient search and matching based on these features. Users can submit retrieval requests to the system by entering keywords, descriptions or samples, and the system will quickly locate video clips or related information that match the request and extract them.
[0056] CLIP: A language-image pre-training model based on contrastive learning. The core idea of this model is to map text and images into a shared vector space through large-scale contrastive learning, so that the model can understand and establish the semantic relationship between images and text.
[0057] Scene segmentation: is an algorithm used to identify scene changes in videos. This algorithm automatically detects the location of scene changes by analyzing the video content, especially the feature changes between video frames.
[0058] Zero-sample learning: Its core idea is to enable AI systems to recognize and process data categories that have never been seen before, that is, the trained classifier can not only recognize the existing data categories in the training set, but also effectively distinguish data from unseen categories. This feature enables AI systems to have the ability to transfer knowledge, so that they can process new categories without any training data, which is very consistent with the characteristics of the wide variety of data categories in the real world.
[0059] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below:
[0060] Video content retrieval is a field that has emerged with the rapid growth of video materials and the continuous development of information retrieval technology. With the vigorous development of the Internet and various video sharing platforms, the massive amount of unstructured video clips has made how to effectively organize, manage and retrieve video information a key issue in the field of information retrieval. For intelligent editing, the rapid retrieval of specific content can also play a key role in editing the wonderful content of the video. Traditional text- and keyword-based methods face challenges in video retrieval because the richness and dynamics of video content are difficult to accurately express with a single text description.
[0061] Currently, video content retrieval mainly uses the following methods:
[0062] 1. Key frame extraction: A key frame is a key image frame that describes the content of a shot and can reflect the main content of the shot. By extracting key frames, the amount of video data processing can be greatly reduced while retaining the main information of the shot. There are many methods for selecting key frames, such as frame averaging and histogram averaging.
[0063] 2. Image feature extraction: Feature extraction is a key step in video content retrieval, which can extract the underlying physical features of image content, such as color histogram, image contour features, etc. These features can be used to describe the visual attributes of video content and provide a basis for subsequent content matching and retrieval.
[0064] 3. Similarity measurement: In video content retrieval, similarity measurement is a key method used to measure the similarity between the query video and the videos in the database. Commonly used similarity measurement methods include distance measurement, correlation calculation, correlation coefficient calculation, etc. By converting video content into feature vectors, the distance or similarity between points can be calculated in the feature space, thereby realizing rapid retrieval of videos.
[0065] In addition, with the development of deep learning and artificial intelligence technology, more and more methods are being applied to video content retrieval, such as feature extraction and classification based on deep learning, video content understanding based on temporal information, etc. These methods can more accurately capture the semantic information of video content and improve the accuracy and efficiency of retrieval.
[0066] However, existing video content retrieval technologies still face some challenges. Although these technologies can extract image features and perform similarity measurements, they still have limitations in semantic understanding. Video content may contain complex semantics and abstract concepts, and existing technologies often cannot accurately understand these contents, which affects the accuracy of retrieval results. In addition, video content is affected by many factors, such as lighting changes, noise, occlusion, and dynamic changes in the picture. These factors may cause the extracted features to be not robust enough, affecting the stability of the retrieval results. In addition, deep learning-based methods usually require a lot of computing resources and time to train, and the model may need to be retrained when retrieval involves different categories of video content. These limitations affect the application scope and effect of video content retrieval technology, and it is necessary to find a faster and more effective solution.
[0067] To this end, the embodiment of the present application provides a technical solution for video content retrieval. In the technical solution, by obtaining a video to be retrieved and a search statement; performing scene segmentation on the video to be retrieved to obtain multiple video clips with different scenes; identifying each video clip according to the search statement to obtain the scene category corresponding to each video clip, wherein the scene category is used to indicate the semantic relationship between the video clip and the search statement; and determining the target video clip matching the search statement from the multiple video clips according to the scene category corresponding to each video clip. By performing scene segmentation before analyzing the video content, the content theme in each generated video clip is consistent, which greatly facilitates the subsequent content analysis work and can save a lot of computing power. Because it is possible to directly search for each scene instead of searching the entire video frame by frame, this not only improves the speed of video content retrieval, but also improves the accuracy of retrieval, because the consistency of the content theme of each video clip makes it possible to locate the target content more accurately. When analyzing the video content, the scene category corresponding to the video clip is obtained by identifying the video clip according to the search statement, and the scene category can indicate the semantic relationship between the video clip and the search statement, so that the scene clip related to the search content can be effectively extracted from the video. These scene clips not only highly match the search keywords, but also have rich semantic information, which provides strong support for the analysis and utilization of subsequent video intelligent editing. At the same time, this strategy is also highly robust and scalable, and can adapt to video data sets of different sizes and complex search requirements. See below for details.
[0068] Finally, for ease of understanding, an exemplary operating environment is provided below.
[0069] like Figure 1 As shown, the environment diagram includes a service platform 2, a network 4, and a client 6, wherein:
[0070] The service platform 2 may be composed of a single or multiple computing devices. The multiple computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., operating system, dedicated application, server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device.
[0071] The service platform 2 may be configured to communicate with the client 6 or the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, and the like, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.
[0072] The service platform 2 can provide storage, reading, writing, querying, deleting and other services, such as providing video content retrieval services for clients.
[0073] Client 6 can be a Windows, Android TM ) or iOS operating systems, such as smart phones, tablet devices, laptop computers, virtual reality devices, gaming devices, set-top boxes, car terminals, and smart TVs. Based on the above operating systems, various application programs, such as programs for video content retrieval, can be run.
[0074] The client 6 may provide / configure a user access page for manipulating the service platform 2 or uploading an object, etc.
[0075] It should be noted that the above devices are exemplary, and the number and type of devices are adjustable in different scenarios or according to different needs.
[0076] The following uses the service platform 2 as the execution subject to introduce the technical solution of the present application through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments described here.
[0077] Embodiment 1
[0078] Figure 2 The flowchart of the video content retrieval method according to the first embodiment of the present application is schematically shown.
[0079] like Figure 2 As shown, the video content retrieval method may include steps S202 to S208, wherein:
[0080] Step S202, obtaining the video to be searched and the search sentence;
[0081] Step S204, performing scene segmentation on the video to be retrieved to obtain a plurality of video clips with different scenes;
[0082] Step S206, identifying each video clip according to the search statement to obtain a scene category corresponding to each video clip, wherein the scene category is used to indicate a semantic relationship between the video clip and the search statement;
[0083] Step S208: determining a target video segment matching the search statement from the multiple video segments according to the scene category corresponding to each video segment.
[0084] The video content retrieval method provided in this embodiment is as follows: by performing scene segmentation before identifying the video content, the content theme generated in each video clip is consistent, which greatly facilitates the subsequent content analysis work and can save a lot of computing power. When analyzing the video content, the scene category corresponding to the video clip is obtained by identifying the video clip according to the search sentence. The scene category can indicate the semantic relationship between the video clip and the search sentence, so that the video clip related to the search content can be effectively extracted from the video according to the scene category, which not only improves the speed of video content retrieval, but also improves the accuracy of video content retrieval, has strong robustness and scalability, and can adapt to video data sets of different sizes and complex retrieval requirements.
[0085] The following combination Figure 2 , each step in steps S202 to S208 and other optional steps are described in detail.
[0086] Step S202 , obtain the video to be retrieved and the search sentence;
[0087] The video to be retrieved may be a video loaded from a cloud server, or may be a video obtained from a local storage unit of a user terminal, and this embodiment does not impose any specific limitation on this.
[0088] The search sentence can be a text entered by the user for the user to search, such as a search sentence of "fight scene" or "two people fighting", etc. The video content search platform can provide a video content search interface for the user, through which the user can select or upload a video to be searched and enter a search sentence for video content search.
[0089] Step S204 , performing scene segmentation on the video to be retrieved to obtain a plurality of video clips with different scenes;
[0090] In this embodiment, the video to be retrieved can be segmented into scenes to obtain multiple video clips with different scenes. After the scene segmentation, the video content or theme in each generated video clip is basically consistent, which facilitates the subsequent video content analysis work.
[0091] Step S206 , identifying each video clip according to the search statement, and obtaining a scene category corresponding to each video clip, wherein the scene category conforms to the semantic representation of the search statement;
[0092] In this embodiment, each video clip may be identified according to the search sentence to obtain a scene category corresponding to each video clip. The scene category is used to indicate a semantic relationship between the video clip and the search sentence.
[0093] In an optional embodiment of the present application, step S206 may include the following steps:
[0094] Acquire a CLIP-based image processing model; identify each video clip according to the search statement using the CLIP-based image processing model to obtain a scene category corresponding to each video clip.
[0095] In the scene content judgment stage, we use the CLIP model to understand the semantics of each scene segment. The CLIP model, full name Contrastive Language-Image Pre-training, is a visual text alignment model based on the Transformer architecture. It achieves deep understanding and efficient alignment of image and text content through pre-training of a large-scale corpus. In the scene content judgment stage, we first convert the key frames extracted from each scene segment into image inputs, and then use the CLIP model to perform in-depth analysis of these images. The CLIP model maps the visual information in the image into a shared semantic space through its unique visual text alignment mechanism, and associates it with the text description of the picture content we preset. For example, if we need to retrieve a fight scene, we now need to calculate the similarity between the picture features extracted by the CLIP model and the text features composed of the two texts "fighting scene" and "non-fighting scene", and then we can determine whether the picture belongs to the "fighting scene". In this way, a video retrieval problem is cleverly transformed into a simple classification problem. Of course, in the actual operation process, our label setting is not so simple. In order to improve the reliability of the algorithm, we will also set more redundant labels to calculate the similarity together. This prevents the network from mistakenly selecting a category when there is no clear discriminant feature specified in the picture. For example, redundant labels such as "natural scenery scene", "normal scene", and "pure color scene" will be added to the fighting scene.
[0096] This embodiment proposes an innovative video content retrieval technology based on the CLIP model. Based on the pre-trained CLIP model, this technology can directly learn rich semantic information from the original video content. The advantage of this technology lies in its zero-sample learning property, that is, it can be directly applied to video content retrieval without explicit training for a specific task. Compared with traditional methods that require a large amount of labeled data and retraining models, this technology is more plug-and-play.
[0097] Another advantage of this technology is its ability to understand the semantics of videos. Through pre-training, the CLIP model has learned rich semantic representations, which enables it to better understand the semantic information in video content. This makes it perform better when processing complex semantics and abstract concepts, and can understand and retrieve video content more accurately than traditional methods.
[0098] In general, CLIP-based video content retrieval technology has the characteristics of zero-shot learning, strong semantic understanding ability, and plug-and-play. These advantages enable it to more effectively cope with the limitations of existing technologies and bring new possibilities and opportunities to the field of video content retrieval.
[0099] Step S208 , according to the scene category corresponding to each video clip, determining a target video clip matching the search statement from the multiple video clips.
[0100] In this embodiment, after analyzing and obtaining the scene category corresponding to each video clip, the target video clip that matches the search statement can be determined from multiple video clips according to the scene category corresponding to each video clip, where the target video clip can include one or more and can be set according to the actual search effect requirements, and the embodiment of the present application does not impose specific restrictions on this.
[0101] In an optional embodiment of the present application, identifying each video clip according to the search statement to obtain a scene category corresponding to each video clip includes:
[0102] Extracting text features corresponding to the search sentence from the search sentence, and determining multiple picture content categories based on the text features; identifying each video clip based on the multiple picture content categories to obtain a scene category corresponding to each video clip.
[0103] In this embodiment, text features corresponding to the search sentence are extracted from the search sentence, and multiple picture content categories are determined based on the text features. Then, each video clip is identified based on the multiple picture content categories to obtain the scene category corresponding to each video clip.
[0104] In an optional embodiment of the present application, identifying each video clip according to the multiple picture content categories to obtain the scene category corresponding to each video clip may include the following steps S11-S14:
[0105] Step S11, for each video segment, extracting frames from the video segment to obtain multiple frames of images;
[0106] Step S12, extracting features from each frame image to obtain image features, and determining the similarity between the frame image and each picture content category according to the image features;
[0107] Step S13, determining the target content category corresponding to the frame image according to the similarity between the frame image and each picture content category;
[0108] Step S14: determining the scene category corresponding to the video clip according to the target content category corresponding to all the frame images in the video clip.
[0109] In this embodiment, when analyzing the video content, for each video segment, multiple frames of images are obtained by extracting frames from the video segment, wherein the frames can be extracted at a preset interval, such as a preset interval of 1 second or 2 seconds, etc., which is not specifically limited in this embodiment. Image features are obtained by extracting features from each frame of image, and the similarity between the frame image and each screen content category is determined based on the image features. Specifically, the frame image and the multiple screen content categories are input into the Clip model, and the Clip model is used to analyze and output the similarity between the frame image and each screen content category.
[0110] Finally, the target content category corresponding to the frame image can be determined based on the similarity between the frame image and each screen content category, and the scene category corresponding to the video segment can be determined based on the target content categories corresponding to all the frame images in the video segment.
[0111] In an optional embodiment of the present application, step S13 may include the following steps:
[0112] According to the similarity between the frame image and each screen content text, the screen content category with the highest similarity is determined as the target content category corresponding to the frame image.
[0113] In this embodiment, the similarity between the frame image and each screen content text may be sorted or voted to determine the screen content category with the highest similarity, and the screen content category with the highest similarity may be used as the target content category corresponding to the frame image.
[0114] In an optional embodiment of the present application, step S14 may include the following steps:
[0115] Determine one or more positive categories from the multiple picture content categories; determine the number of frame images belonging to the positive category based on the target content category corresponding to all the frame images in the video clip; and determine the scene category corresponding to the video clip based on the number of frame images belonging to the positive category.
[0116] In this embodiment, one or more positive categories are determined from multiple picture content categories, and then the number of frame images belonging to the positive category is counted according to the target content category corresponding to all the frame images in the video clip, and the scene category corresponding to the video clip is determined according to the number of frame images belonging to the positive category. Specifically, a ratio can be set, such as 50% or 60%, etc. When the ratio of the number of frame images belonging to the positive category in all the extracted frames exceeds the set ratio, it is considered that the scene category corresponding to the video clip is a positive category, such as a "fighting scene category"; when the ratio of the number of frame images belonging to the positive category in all the extracted frames does not reach the set ratio, it is considered that the scene category corresponding to the video clip is a negative category, such as a "non-fighting scene category".
[0117] In an optional embodiment of the present application, the step of performing scene segmentation on the video to be retrieved to obtain a plurality of video clips with different scenes includes:
[0118] Feature extraction is performed on each frame image in the video to be retrieved to obtain image feature information; feature differences between two adjacent frame images are determined based on the image feature information; and based on the feature differences between the two adjacent frame images, scene segmentation is performed on the video to be retrieved to obtain multiple video clips with different scenes.
[0119] Scene segmentation in a video is a key step in content analysis because it can divide a video into different logical sections, each of which is a scene, and the subject matter of the lens content within it is usually consistent. This scene segmentation not only helps to understand the overall structure of the video, but also significantly improves efficiency when performing content retrieval. In this embodiment, the feature difference between two adjacent frame images in the video to be retrieved can be determined, and then the scene segmentation of the video to be retrieved can be performed based on the feature difference between the two adjacent frame images to obtain multiple video clips with different scenes.
[0120] As an example, the Scene Cut algorithm can be used to segment the scene of the video to be retrieved. The Scene Cut algorithm is a tool specifically used to achieve this goal. The algorithm analyzes the video content frame by frame and uses image processing technology and machine learning algorithms to identify the scene switching points in the video. These switching points are usually determined based on significant changes in features such as color, texture, and motion between video frames. During the execution of the algorithm, features are first extracted for each frame of the video. These features include color histograms, edge information, motion vectors, etc. Then, the algorithm compares the feature differences between adjacent frames. If the difference exceeds a certain set threshold, it is considered that a scene switch has occurred here. In this way, the Scene Cut algorithm can aggregate each frame of the video according to the scene to form independent scene segments. The content theme within each scene is consistent, which greatly facilitates subsequent content analysis.
[0121] like Figure 3 A modular block diagram of a video content retrieval method is shown. Figure 3 In the example, the search phrase "fighting scene" is used. By performing scene cut on the video, multiple clips with different scenes are obtained. Frames are extracted for each clip and then input into the Clip model for zero shot classification. For example, category 1: the scene is a wonderful fighting scene, category 2: the scene is not a wonderful fighting scene. Then, based on the classification results, strategic clips are performed to determine the scene category corresponding to each video clip, such as fighting or non-fighting.
[0122] Embodiment 2
[0123] Figure 4 The block diagram of the video content retrieval device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 4 As shown, the video content retrieval device 400 may include: a retrieval statement acquisition module 410, a scene segmentation module 420, a scene category determination module 430 and a target video segment determination module 440, wherein:
[0124] A search statement acquisition module 410 is used to acquire a video to be searched and a search statement;
[0125] A scene segmentation module 420, configured to perform scene segmentation on the video to be retrieved to obtain a plurality of video clips with different scenes;
[0126] A scene category determination module 430, configured to identify each video segment according to the search statement to obtain a scene category corresponding to each video segment, wherein the scene category is used to indicate a semantic relationship between the video segment and the search statement;
[0127] The target video segment determination module 440 is configured to determine a target video segment matching the search statement from the multiple video segments according to the scene category corresponding to each video segment.
[0128] In an optional embodiment of the present application, the scene category determination module 430 is specifically used to:
[0129] Acquire a CLIP-based image processing model; identify each video clip according to the search statement using the CLIP-based image processing model to obtain a scene category corresponding to each video clip.
[0130] In an optional embodiment of the present application, the scene category determination module 430 includes:
[0131] A picture content category determination submodule, used to extract text features corresponding to the search sentence from the search sentence, and determine multiple picture content categories according to the text features;
[0132] The scene category determination submodule is used to identify each video segment according to the multiple picture content categories to obtain the scene category corresponding to each video segment.
[0133] In an optional embodiment of the present application, the scene category determination submodule includes:
[0134] A frame extraction unit, used for extracting frames from each video segment to obtain multiple frames of images;
[0135] A similarity determination unit, configured to extract features from each frame image to obtain image features, and determine the similarity between the frame image and each picture content category according to the image features;
[0136] a target content category determination unit, configured to determine a target content category corresponding to the frame image according to a similarity between the frame image and each picture content category;
[0137] The scene category determination unit is used to determine the scene category corresponding to the video segment according to the target content category corresponding to all the frame images in the video segment.
[0138] In an optional embodiment of the present application, the target content category determination unit is specifically configured to:
[0139] According to the similarity between the frame image and each screen content text, the screen content category with the highest similarity is determined as the target content category corresponding to the frame image.
[0140] In an optional embodiment of the present application, the scene category determination unit is specifically used to:
[0141] determining one or more positive categories from the plurality of picture content categories;
[0142] Determining the number of frame images belonging to the positive category according to the target content category corresponding to all frame images in the video clip;
[0143] The scene category corresponding to the video clip is determined according to the number of frame images belonging to the positive category.
[0144] In an optional embodiment of the present application, the scene segmentation module 420 includes:
[0145] An image feature extraction submodule is used to extract features from each frame of the image in the video to be retrieved to obtain image feature information;
[0146] A frame image difference determination submodule, used to determine a feature difference between two adjacent frame images according to the image feature information;
[0147] The scene segmentation submodule is used to perform scene segmentation on the video to be retrieved according to the feature difference between the two adjacent frame images to obtain multiple video clips with different scenes.
[0148] Embodiment 3
[0149] Figure 5 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the video content retrieval method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 5 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:
[0150] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the video content retrieval method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.
[0151] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.
[0152] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.
[0153] It should be pointed out that Figure 5 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.
[0154] In this embodiment, the video content retrieval method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.
[0155] Embodiment 4
[0156] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video content retrieval method in the embodiment are implemented.
[0157] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video content retrieval method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.
[0158] Embodiment 5
[0159] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.
[0160] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.
[0161] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A video content retrieval method, characterized in that: The method comprises: Get the video to be retrieved and the search sentence; Performing scene segmentation on the video to be retrieved to obtain a plurality of video clips with different scenes; Identify each video clip according to the search statement to obtain a scene category corresponding to each video clip, wherein the scene category is used to indicate a semantic relationship between the video clip and the search statement; According to the scene category corresponding to each video segment, a target video segment matching the search statement is determined from the multiple video segments.
2. The video content retrieval method according to claim 1, characterized in that: The step of identifying each video clip according to the search statement to obtain a scene category corresponding to each video clip includes: Get CLIP-based image processing models; Through the CLIP-based image processing model, each video clip is identified according to the search statement to obtain the scene category corresponding to each video clip.
3. The video content retrieval method according to claim 1 or 2, characterized in that: The step of identifying each video clip according to the search statement to obtain a scene category corresponding to each video clip includes: Extracting text features corresponding to the search sentence from the search sentence, and determining a plurality of picture content categories according to the text features; Each video segment is identified according to the multiple picture content categories to obtain a scene category corresponding to each video segment.
4. The video content retrieval method according to claim 3, characterized in that: The step of identifying each video clip according to the multiple picture content categories to obtain a scene category corresponding to each video clip includes: For each video segment, extract frames from the video segment to obtain multiple frames of images; Extracting features from each frame of image to obtain image features, and determining the similarity between the frame of image and each picture content category according to the image features; Determining a target content category corresponding to the frame image according to a similarity between the frame image and each screen content category; The scene category corresponding to the video segment is determined according to the target content category corresponding to all the frame images in the video segment.
5. The video content retrieval method according to claim 4, characterized in that: The determining the target content category corresponding to the frame image according to the similarity between the frame image and each screen content category includes: According to the similarity between the frame image and each screen content text, the screen content category with the highest similarity is determined as the target content category corresponding to the frame image.
6. The video content retrieval method according to claim 4, characterized in that: The step of determining the scene category corresponding to the video clip according to the target content category corresponding to all the frame images in the video clip includes: determining one or more positive categories from the plurality of picture content categories; Determining the number of frame images belonging to the positive category according to the target content category corresponding to all frame images in the video clip; The scene category corresponding to the video clip is determined according to the number of frame images belonging to the positive category.
7. The video content retrieval method according to claim 1, characterized in that: The step of performing scene segmentation on the video to be retrieved to obtain a plurality of video clips with different scenes includes: Extracting features from each frame of the video to be retrieved to obtain image feature information; Determine a feature difference between two adjacent frame images according to the image feature information; According to the feature difference between the two adjacent frame images, the video to be retrieved is segmented into scenes to obtain a plurality of video clips with different scenes.
8. A video content retrieval device, characterized in that: The device comprises: A search statement acquisition module is used to acquire the video to be searched and the search statement; A scene segmentation module, used to perform scene segmentation on the video to be retrieved to obtain multiple video clips with different scenes; A scene category determination module, used to identify each video segment according to the search sentence, and obtain a scene category corresponding to each video segment, wherein the scene category is used to indicate a semantic relationship between the video segment and the search sentence; The target video segment determination module is used to determine a target video segment matching the search statement from the multiple video segments according to the scene category corresponding to each video segment.
9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video content retrieval method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the video content retrieval method according to any one of claims 1 to 7 is implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the video content retrieval method according to claims 1 to 7 are implemented.
Citation Information
Patent Citations
Video classification method, device, electronic device and computer-readable storage medium
CN109376268A
Video classification method, equipment and medium
CN112749685A
Target type video detection method and device, equipment and storage medium
CN112800919A
Method and device for extracting video theme text, equipment and storage medium
CN113395578A
Video scene segmentation method, device and equipment and computer readable storage medium
CN114283351A
Cited By
Text video retrieval method and device, equipment and storage medium
CN120892601A
