Video content retrieval method and device

Abstract

The invention provides a video content retrieval method and device.The method comprises the steps that all video streams in a video library are traversed, a first frame sequence of the video streams is obtained based on a first frame rate, and if the pixel difference value between adjacent video frames in the first frame sequence is larger than a first threshold value, the video streams are retrieved; if yes, determining a video clip corresponding to the adjacent video frame as a candidate clip, and obtaining a key frame in the candidate clip based on a second frame rate; determining each granularity video corresponding to the video stream, and determining structured data corresponding to each granularity video according to a pre-trained processing model; and matching the user retrieval information with the structured data corresponding to each granularity video of each video stream in the video library, and determining target video content according to a matching result. According to the method provided by the invention, the accuracy of video retrieval can be improved.

Description

Technical Field

[0001] This specification relates to the field of video retrieval technology, and in particular to a video content retrieval method and apparatus. Background Technology

[0002] Currently, mainstream video retrieval methods mainly rely on simple text matching of metadata such as video titles, tags, and descriptions, or on manually categorizing content through tagging. Although some methods have introduced image recognition technology, which can extract static objects or scene information from videos, they cannot "understand" the semantic information of videos. At the same time, they cannot support cross-modal retrieval capabilities such as "image search for video".

[0003] Therefore, how to improve the retrieval capability of video content based on the semantic information of the video and support multiple retrieval input methods such as text, image, and tags, and achieve accurate retrieval of video content is an urgent problem to be solved in this application. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this specification provides a video content retrieval method and apparatus that can achieve accurate retrieval of video content.

[0005] According to a first aspect of the embodiments of this specification, a video content retrieval method is provided, the method comprising: The video streams in the video library are traversed, and the first frame sequence of the video stream is obtained based on the first frame rate. If the pixel difference between adjacent video frames in the first frame sequence is greater than the first threshold, the video segment corresponding to the adjacent video frame is determined as a candidate segment, and the key frame in the candidate segment is obtained based on the second frame rate; wherein, the second frame rate is greater than the first frame rate. The process involves determining the granularity of the video stream and then using a pre-trained processing model to determine the structured data corresponding to each granularity of the video stream. Each granularity of the video stream includes the entire video stream, the keyframes of the video stream, and each sub-video stream within the video stream. The structured data includes multi-dimensional semantic vectors and textual description information. The user search information is matched with the structured data corresponding to each granularity video of each video stream in the video library, and the target video content is determined based on the matching results; wherein, the user search information includes at least one of text information, image information, tag information and video information.

[0006] According to a second aspect of the embodiments of this specification, a video content retrieval device is provided, comprising: The acquisition module is used to traverse each video stream in the video library, acquire the first frame sequence of the video stream based on a first frame rate, and if the pixel difference between adjacent video frames in the first frame sequence is greater than a first threshold, then the video segment corresponding to the adjacent video frame is determined as a candidate segment, and key frames in the candidate segment are acquired based on a second frame rate; wherein, the second frame rate is greater than the first frame rate. The determination module is used to determine the granularity videos corresponding to the video stream, and to determine the structured data corresponding to each granularity video according to the pre-trained processing model; each granularity video includes the entire video stream, the keyframes of the video stream, and each sub-video stream in the video stream, and the structured data includes multi-dimensional semantic vectors and text description information; The matching module is used to match user search information with the structured data corresponding to each granularity video of each video stream in the video library, and determine the target video content based on the matching results; wherein, the user search information includes at least one of text information, image information, tag information and video information.

[0007] In one exemplary embodiment of this specification, the acquisition module is specifically used for: The second frame sequence corresponding to the candidate segment is obtained based on the second frame rate; Semantic features are extracted from each video frame in the second frame sequence to obtain the semantic feature vector corresponding to each video frame. The scene switching points in the second frame sequence are determined based on the semantic feature vector corresponding to each video frame. The candidate segment is divided into multiple sub-segments based on different scene switching points, and the keyframes in each sub-segment are determined.

[0008] In one exemplary embodiment of this specification, the pre-trained processing model includes a pre-trained multimodal video understanding model and a pre-trained multimodal vector model, and the determining module is specifically used for: Based on the pre-trained multimodal video understanding model, multi-dimensional features corresponding to at least one dimension of information among the subject object, scene information, environment information and action information corresponding to each granularity video are determined, and the dimensional features corresponding to each granularity video are processed to obtain the text description information corresponding to each granularity video. Based on the pre-trained multimodal vector model, the text description information corresponding to each granularity video is encoded to obtain the multidimensional semantic vector corresponding to each granularity video.

[0009] In one exemplary embodiment of this specification, the determining module is specifically used for: For each granular video, a preset thought chain corresponding to the granular video is determined, and the associated features corresponding to the preset thought chain in the multi-dimensional features of the granular video are obtained. The associated features are logically processed based on the preset thought chain to obtain the text description information in the structured data; wherein, the text description information includes at least one of title, summary and tag.

[0010] In one exemplary embodiment of this specification, the matching module is specifically used for: If the user search information contains text information, image information, or video information, then the feature vector corresponding to the user search information is determined based on the feature vectors corresponding to the text information, image information, or video information, respectively. For each video stream in the video library, the weighted sum of the similarities between the feature vector and the multidimensional semantic vectors in the structured data corresponding to each granularity video in the video stream is determined as the first similarity. Based on the order of the first similarity from largest to smallest, a preset number of video streams are determined as the first video stream; If the user search information contains tag information, then the tag information is matched with the text description information in the structured data corresponding to the video stream, and the matched video stream is determined as the second video stream; The intersection of the first video stream and the second video stream is determined as the target video stream; The target video content is determined based on the matching degree between each granularity video in the target video stream and the feature vector and the tag information.

[0011] In one exemplary embodiment of this specification, the matching module is specifically used for: In each granular video corresponding to the target video stream, the granular video with a similarity greater than a similarity threshold with the feature vector is determined as the first target content, and the granular video with a matching degree greater than a matching degree threshold with the tag information is determined as the second target content; The intersection of the first target content and the second target content is the target video content.

[0012] In one exemplary embodiment of this specification, the video content retrieval device further includes a feedback module, the feedback module being configured to: Obtain user feedback on the target video content, determine the content to be optimized in the target video content based on the feedback, and update the pre-trained processing model based on the content to be optimized.

[0013] According to a third aspect of the embodiments of this specification, an apparatus is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of any of the above-described video retrieval content methods.

[0014] The technical solutions provided in the embodiments of this specification may include the following beneficial effects: First, in the embodiments of this specification, by obtaining the first frame sequence of the video stream based on the first frame rate and determining the candidate segments with significant changes in the video based on the pixel difference values ​​between adjacent video frames, redundant calculations are reduced compared to the method of processing the entire video; then, a higher second frame rate is used to extract key frames from the candidate segments. By first coarsely screening and then finely extracting, the processing efficiency of key frame extraction is improved while ensuring that the key frames can accurately capture the core content of the candidate segments.

[0015] Secondly, in the embodiments of this specification, the video stream and keyframes are processed by a pre-trained processing model, and each granularity of the video stream, such as the video stream, keyframes, and sub-video streams, is converted into corresponding structured data. This enables the acquisition of multi-dimensional semantic vectors and textual description information in the video, providing a data foundation for accurate retrieval.

[0016] Third, in the embodiments of this specification, by matching the user's search information with the structured data of each granularity of the video stream to determine the target video content, accurate retrieval of video content can be achieved. At the same time, the user's search information may include at least one of text information, image information, tag information and video information, so that regardless of whether the search is based on text, image, tag or video, the target video content can be retrieved efficiently, thus improving the video content retrieval capability.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0019] Figure 1 This is a flowchart illustrating a video content retrieval method according to an exemplary embodiment of this specification; Figure 2 This is a flowchart illustrating a two-stage keyframe extraction process according to an exemplary embodiment of this specification; Figure 3 This is a flowchart illustrating, according to an exemplary embodiment, the determination of structured data corresponding to each granularity of video. Figure 4This is a flowchart illustrating, according to an exemplary embodiment, how a label is determined through a preset thought process. Figure 5 This is a flowchart illustrating another video content retrieval method according to an exemplary embodiment of this specification; Figure 6 This is a closed-loop schematic diagram of the retrieval feedback illustrated in this specification according to an exemplary embodiment; Figure 7 This is a flowchart illustrating a video content retrieval system according to an exemplary embodiment. Figure 8 This is a block diagram of a video content retrieval device illustrated in this specification according to an exemplary embodiment; Figure 9 This is a hardware structure diagram of a computer device containing the document processing apparatus in the embodiments of this specification. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0021] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0023] The embodiments described in this specification will now be described in detail.

[0024] like Figure 1 As shown, Figure 1This is a flowchart illustrating a video content retrieval method according to an exemplary embodiment, comprising the following steps: Step 101: Traverse each video stream in the video library, obtain the first frame sequence of the video stream based on the first frame rate, and if the pixel difference between adjacent video frames in the first frame sequence is greater than the first threshold, determine the video segments corresponding to the adjacent video frames as candidate segments, and obtain the key frames in the candidate segments based on the second frame rate.

[0025] Step 102: Determine the granularity of the video stream and determine the structured data corresponding to each granularity of the video stream based on the pre-trained processing model.

[0026] Step 103: Match the user's search information with the structured data corresponding to each granularity of video in each video stream in the video library, and determine the target video content based on the matching results.

[0027] In step 101, each video stream in the video library can be traversed. For the currently traversed video stream, such as... Figure 2 As shown, Figure 2 This specification illustrates a two-stage keyframe extraction flowchart based on an exemplary embodiment, which can extract video frames from the video stream starting from the beginning frame of the video stream based on a first frame rate to obtain a first frame sequence.

[0028] For every two adjacent video frames in the first frame sequence, their pixel difference value can be calculated. As an example, the degree of difference can be determined by calculating the difference in grayscale values ​​of corresponding pixels in adjacent video frames. This first threshold can be determined according to the actual situation, and this application does not limit it.

[0029] When the pixel difference between adjacent video frames exceeds the first threshold, a significant image change can be determined between the two video frames. The time interval of these adjacent video frames can then be further determined, and video segments within that time interval can be identified as candidate segments. After obtaining multiple candidate segments, those with overlapping time intervals can be merged.

[0030] After identifying candidate segments, they can be further extracted based on a second frame rate to obtain a second frame sequence. This second frame rate is higher than the first frame rate. This allows for a coarse screening of the entire video stream based on the first frame rate, filtering out a large number of static or low-variation invalid segments. Then, the candidate segments are finely screened based on the second frame rate to capture video frames more densely within the candidate segments. This improves processing efficiency, reduces unnecessary computation, and ensures the accuracy of keyframe extraction, achieving a balance between efficiency and accuracy.

[0031] After determining the second frame sequence, keyframes can be identified based on this sequence and designated as keyframes in the video stream. This involves extracting semantic features from each video frame in the second frame sequence to obtain a corresponding semantic feature vector. Scene transition points in the second frame sequence are then determined based on these semantic feature vectors. For example, determining scene transition points in the second frame sequence can be achieved by: processing each video frame in the second frame sequence using a pre-trained scene extraction model to capture scene information within each frame and converting each frame into a fixed-dimensional semantic feature vector; calculating the cosine similarity between the semantic feature vectors of every two adjacent video frames; if the cosine similarity is less than a preset transition point threshold, a scene transition is determined between the adjacent video frames. The time point of the preceding video frame in the adjacent video frames can be identified as the scene transition point, and the time point of the following video frame in the adjacent video frames can be identified as the starting point of the next scene. By determining scene transition points based on semantic feature vectors, accurate segmentation of candidate segments is achieved, providing an accurate data foundation for subsequent keyframe selection.

[0032] After obtaining the scene transition point, the candidate segment can be divided into multiple sub-segments based on this point, and keyframes in each sub-segment can be determined. The method for determining keyframes in a sub-segment can be as follows: The average semantic feature vector of all video frames contained in the sub-segment can be determined. Then, the video frames in the sub-segment can be traversed, and the cosine similarity between the semantic feature vector of each video frame and this average value can be calculated. The video frame with the highest cosine similarity is selected as the keyframe, ensuring that the obtained keyframe represents the semantic features of the sub-segment. If the sub-segment length exceeds a preset duration threshold, such as 5 seconds, the visual saliency features of the video frames can be further combined, such as extracting the proportion of the foreground subject, to determine the visual saliency value. From the top N (e.g., top 3) video frames ranked by cosine similarity, the frame with the highest visual saliency value is selected as the final keyframe, ensuring that the keyframe represents the semantics of the sub-segment and has clear visual recognizability.

[0033] In step 102, the video stream can be divided into multiple sub-video streams according to the shots or preset duration. The entire video stream, the keyframes corresponding to the video stream, and each video stream in the video stream can be determined as video at each granularity.

[0034] Figure 3A flowchart illustrating the determination of structured data corresponding to videos at each granularity is shown. After obtaining the videos at each granularity, at least one dimension of information—the subject object, scene information, environmental information, and action information—can be acquired based on a pre-trained multimodal video understanding model, and features corresponding to each dimension can be obtained. Pre-defined associated features can be acquired, and the features corresponding to each dimension can be validated based on these features. If the validation passes, the dimensional features corresponding to each dimension are fused to obtain the multi-dimensional features corresponding to the granular video. By validating and fusing the multi-dimensional features, invalid or erroneous features can be filtered out, ensuring the reliability of the textual description information. The features corresponding to the main object can include the main object category and main object attributes, the features corresponding to the scene information can include the scene category and scene attributes, the features corresponding to the environment information can include weather conditions and light intensity, and the features corresponding to the action information can include the action category. The preset association features can be based on the dimensional feature logical matching rules constructed from the knowledge graph. These preset association rules are generated through training on a large amount of labeled data and can be used to verify the features corresponding to different dimensional information, as well as whether the features conform to objective logic or common sense association, to ensure that the extracted multi-dimensional features are consistent in semantics and scene. For example, if the feature of the main object is a truck and the environment information is a road, then it conforms to the logical rules and the verification passes. Multi-dimensional features can be obtained by concatenating the dimensional features corresponding to each dimensional information according to the time order.

[0035] After obtaining the multi-dimensional features corresponding to each granularity of video, a pre-trained multimodal video understanding model can be used to process these features to obtain textual descriptions for each granularity of video. For each granularity of video, a pre-defined thought chain can be determined based on cue word engineering. The associated features corresponding to this pre-defined thought chain within the multi-dimensional features of that granularity of video are then obtained. These associated features are logically processed based on the pre-defined thought chain to obtain textual descriptions from the structured data. This pre-defined thought chain consists of a series of sub-questions that conform to cognitive logic, such as "Who are the people in the video?", "What are they doing?", "Where is the scene?", and "What is the outcome of the event?". By obtaining the associated features corresponding to this pre-defined thought chain within the multi-dimensional features, and guiding the pre-trained multimodal video understanding model to reason and analyze these associated features, the textual descriptions can be logically consistent, resulting in high-quality, logically sound textual descriptions. This textual description information can include at least one of the following: title, summary, and tags. The text description information for each sub-video stream can include a summary and tags; the text description information for each video stream can include a title, summary, and tags; the text description information for each keyframe can include a summary and tags. The summary is a general overview of the core content corresponding to each granularity of video, and the tags are the keywords involved in each granularity of video. As an example, Figure 4A flowchart illustrating the process of determining labels through a pre-defined thought process is shown. Figure 4 To generate hazard identification labels for vehicles during driving, one can first identify the problem to be solved in the prompt word project, determine the task decomposition in the prompt word project based on the problem, determine the preset thinking chain based on each of the decomposed tasks, determine the associated features corresponding to the preset thinking chain, perform logical processing on the associated features based on the preset thinking chain, and determine the obtained label based on the result of the logical processing.

[0036] The text description information corresponding to each granularity of video can be encoded using a pre-trained multimodal vector model to obtain multidimensional semantic vectors for each granularity of video. Specifically, the text encoder in the pre-trained multimodal vector model converts the text description information into a sequence of tokens that the model can recognize. Then, a multi-layer Transformer structure is used to extract features and perform semantic mapping on the token sequence, ultimately outputting the feature vector corresponding to the text description information. After obtaining the multidimensional semantic vectors for each granularity of video, the multidimensional semantic vectors can be normalized to map them to the same dimensional space.

[0037] After obtaining the text description information and multidimensional semantic vectors corresponding to each granularity of video, these can be stored in a database. Tags from the text description information and their corresponding video identifiers can be stored in a tag library, summaries from the text description information and their corresponding video identifiers can be stored in a summary library, titles from the text description information and their corresponding video identifiers can be stored in a title library, and multidimensional semantic vectors and their corresponding video identifiers can be stored in a vector library. This facilitates subsequent retrieval of structured data from the database using the video identifiers, further enhancing the matching of user search information. A pre-trained processing model accurately extracts multidimensional features from each granularity of video and transforms them into structured text description information and multidimensional semantic vectors, ensuring comprehensive feature extraction and improving the accuracy of representing the content of each granularity of video.

[0038] In step 103, the user's search information may include at least one of text information, image information, tag information, and video information. By being compatible with search information such as text, images, tags, and videos, the versatility of the search can be improved.

[0039] If the user's search information includes text, image, or video information, the feature vector corresponding to the user's search information can be determined based on the feature vectors corresponding to the text, image, or video information, respectively. For each video stream in the video library, the weighted sum of the similarities between the feature vector and the multidimensional semantic vectors in the structured data corresponding to each granularity of the video stream can be determined as the first similarity. Based on the first similarity in descending order, a preset number of video streams can be determined as the first video stream. If the user's search information includes tag information, the tag information can be matched with the text description information in the structured data corresponding to the video stream, and the matched video stream can be determined as the second video stream. The intersection of the first and second video streams can be determined as the target video stream. After obtaining the feature vectors corresponding to text, image, or video information, each feature vector can be mapped to a vector space of the same dimension. Then, the feature vectors corresponding to the text, image, or video information are fused. The feature fusion method can be based on weighted fusion according to the weights corresponding to the text, image, or video information, or it can be fused through an attention mechanism to obtain the feature vectors. The weights corresponding to each granularity of video in the video stream can be preset according to the importance of each granularity of video. Similarity can be determined by calculating the cosine similarity between vectors; the higher the cosine similarity value, the more similar the vectors. The preset number can be set according to the actual application scenario, such as setting it to 5, that is, taking the top 5 video streams with the highest first similarity as the first video stream. The tags in the tag information corresponding to the user's search information can be standardized to ensure consistency with the tag descriptions in the structured data.

[0040] After obtaining the target video stream, the target video content can be determined based on the matching degree between each granular video in the target video stream and the feature vector and tag information. The specific method for determining the target video content is as follows: among the granular videos corresponding to the target video stream, granular videos with a similarity greater than a similarity threshold with the feature vector can be identified as the first target content, and granular videos with a matching degree greater than a matching degree threshold with the tag information can be identified as the second target content. The similarity threshold for each target video stream can be determined based on the first similarity between each target video stream and the user's search information. For example, the variance between each first similarity between each target video stream and the user's search information can be calculated, and the sum of the first similarity and the variance for each target video stream can be used as the similarity threshold. Furthermore, the number of overlapping tags between the tags corresponding to the user's search information and the text description information corresponding to the video stream can be determined, and the ratio of the number of overlapping tags to the total number of tags corresponding to the user's search information can be used as the matching degree. As an example, a basic threshold can be determined based on positive feedback data from historical searches. If the number of tags corresponding to the user's search information is less than a preset first threshold (e.g., 2), the basic threshold can be reduced by the preset number threshold to obtain a matching threshold. If the number of tags corresponding to the user's search information is greater than a preset second threshold (e.g., 5), the basic threshold can be increased by the preset number threshold to obtain a matching threshold. By filtering the target video stream and then further filtering the target video streams that meet the conditions, the accuracy of the search results can be guaranteed, and the target video content that matches the user's search information can be accurately retrieved.

[0041] like Figure 5 As shown, Figure 5 This is a flowchart illustrating another video content retrieval method according to an exemplary embodiment of this specification, including the following steps: Step 101: Traverse each video stream in the video library, obtain the first frame sequence of the video stream based on the first frame rate, and if the pixel difference between adjacent video frames in the first frame sequence is greater than the first threshold, determine the video segments corresponding to the adjacent video frames as candidate segments, and obtain the key frames in the candidate segments based on the second frame rate.

[0042] Step 102: Determine the granularity of the video stream and determine the structured data corresponding to each granularity of the video stream based on the pre-trained processing model.

[0043] Step 103: Match the user's search information with the structured data corresponding to each granularity of video in each video stream in the video library, and determine the target video content based on the matching results.

[0044] Step 504: Obtain user feedback on the target video content, determine the content to be optimized in the target video content based on the feedback, and update the pre-trained processing model based on the content to be optimized.

[0045] In step 504, as Figure 6 As shown, Figure 6 This is a closed-loop schematic diagram of retrieval feedback illustrated in this specification according to an exemplary embodiment. User feedback information on the target video content can be obtained. This feedback information may include positive feedback information and negative feedback information. The positive feedback information indicates video content in the target video content that matches the user's search requirements, while the negative feedback information indicates video content in the target video content that does not match the user's search requirements. Content in the target video content that needs optimization can be determined based on the negative feedback information.

[0046] After receiving user feedback, access logs can be recorded, and the quality of the feedback can be evaluated. If the feedback is high-value, the pre-trained processing model can be updated based on the corresponding content to be optimized. The structured data corresponding to the content to be optimized can be identified, and this structured data, along with the user's search information, can be used to supplement the training data. This supplementary data is then input into the multimodal video understanding model and multimodal vector model within the pre-trained processing model for fine-tuning. During fine-tuning, the encoder parameters can be adjusted to allow the model to relearn the association between the content to be optimized and the search information, reducing computational bias related to the content to be optimized. If the content to be optimized suffers from label matching bias—for example, if the labels of a video content are actually related to the user's search labels but the model fails to recognize them—the labels of the video content and the search labels can be combined to form associated samples, which are then added to the model's training sample set. This retrains the model's parameters in the label matching process, allowing the model to learn the association rules for such labels. When updating the pre-trained processing model, incremental training can be used, i.e., small-batch training is performed only on the sample data corresponding to the content to be optimized, avoiding excessive resource consumption caused by full retraining. During training, a smaller learning rate can be set, such as reducing the original learning rate by 10%, to ensure that the model retains its original correct matching ability while correcting biases. After the model is updated, the optimization effect can be verified through new video retrieval. If the proportion of negative feedback in user feedback on similar searches decreases to a preset threshold, such as below 5%, it indicates that the model optimization is effective. The updated model can be automatically deployed online through a unified model gateway to replace the old model, and the improvements in the updated model can be output, or the user can be prompted to provide new feedback. If the desired effect is not achieved, the above steps can be repeated to continue to determine the content to be optimized based on new feedback and iteratively update the model. If the feedback is not high-value feedback, there is no need to update the model, and the normal feedback should be responded to, i.e., informing the user that the feedback has been received, indicating that the user is aware of it, or prompting the user to provide new feedback. Among these, feedback that is specific, relevant to model adjustment, and actionable can be identified as high-value feedback, while feedback that cannot determine the content to be optimized, is irrelevant to model processing, or has no practical actionable effect can be identified as low-value feedback.

[0047] By using the above methods, the matching deviation of the processing model can be corrected based on actual feedback information, making the updated processing model more in line with the user's search needs and improving the matching accuracy of target video content in the search process.

[0048] Corresponding to the embodiments of the foregoing methods, this specification also provides an embodiment of a video content retrieval system. For example... Figure 7 As shown, Figure 7The specification illustrates a flowchart of a video content retrieval system according to an exemplary embodiment. Users can perform video searches by inputting user search information through a unified application interface (API). This user search information can include at least one of text, image, tag, and video information, enabling text search, image search, tag search, and video search. Based on the unified API, a pre-trained processing model can be invoked through a multi-model adaptation module to perform video understanding on videos of various granularities in the database, obtaining structured data corresponding to text descriptions and multi-dimensional semantic vectors for each granularity of video. The gateway receives user video search requests and records access logs during the video search process. These access logs include user feedback on the video search, which can be fed back through an online feedback module. If the feedback is high-value, the pre-trained processing model can be updated based on it. Simultaneously, the database module can provide prompt word templates, example libraries, and domain knowledge during model training, thereby assisting in model training or updates. This video content retrieval system forms a complete closed loop, from inputting video retrieval needs to data processing, model support, and feedback optimization. Each module collaborates to achieve accurate and intelligent video content retrieval and iterative optimization, improving the effectiveness of video retrieval. The deployment of the video content retrieval system provided in this manual can also evolve in stages: from initial offline batch processing based on Transformers, to a mid-term approach using a VLLM inference engine to achieve efficient online services, and finally to professional and highly available service governance through a unified model gateway.

[0049] like Figure 8 As shown, Figure 8 This is a block diagram illustrating a video content retrieval device according to an exemplary embodiment of this specification. The video content retrieval device includes an acquisition module 810, a determination module 820, and a matching module 830, wherein: The acquisition module 810 is used to traverse each video stream in the video library, acquire the first frame sequence of the video stream based on the first frame rate, and if the pixel difference between adjacent video frames in the first frame sequence is greater than the first threshold, then the video segments corresponding to the adjacent video frames are determined as candidate segments, and key frames in the candidate segments are acquired based on the second frame rate; wherein, the second frame rate is greater than the first frame rate. The determination module 820 is used to determine the granularity of the video stream and determine the structured data corresponding to each granularity of the video stream based on the pre-trained processing model. Each granularity of the video stream includes the whole video stream, the key frames of the video stream, and each sub-video stream in the video stream. The structured data includes multi-dimensional semantic vectors and text description information. The matching module 830 is used to match the user's search information with the structured data corresponding to each granularity video of each video stream in the video library, and determine the target video content based on the matching results; wherein, the user's search information includes at least one of text information, image information, tag information and video information.

[0050] In this embodiment, the device further includes a feedback module, which is used to: obtain user feedback information on the target video content, determine the content to be optimized in the target video content based on the feedback information, and update the pre-trained processing model according to the content to be optimized.

[0051] The acquisition module 810 is also used to: acquire the second frame sequence corresponding to the candidate segment based on the second frame rate; extract semantic features from each video frame in the second frame sequence to obtain the semantic feature vector corresponding to each video frame; determine the scene switching point in the second frame sequence based on the semantic feature vector corresponding to each video frame; divide the candidate segment into multiple sub-segments based on different scene switching points; and determine the key frame in each sub-segment.

[0052] The determination module 820 is also used to: determine the multi-dimensional features corresponding to at least one dimension of information among the subject object, scene information, environment information and action information corresponding to each granularity video based on the pre-trained multimodal video understanding model, and process the dimensional features corresponding to each granularity video to obtain the text description information corresponding to each granularity video; and encode the text description information corresponding to each granularity video based on the pre-trained multimodal vector model to obtain the multi-dimensional semantic vector corresponding to each granularity video.

[0053] The determination module 820 is also used to: for each granular video, obtain at least one dimension of information from the main object, scene information, environment information and action information of the granular video, and obtain the features corresponding to each dimension of information respectively; verify the features corresponding to each dimension of information respectively based on the preset associated features; if the verification passes, fuse the dimensional features corresponding to each dimension of information to obtain the multi-dimensional features corresponding to the granular video.

[0054] The determination module 820 is also used to: determine the preset thought chain corresponding to each granular video, obtain the associated features corresponding to the preset thought chain in the multi-dimensional features of the granular video, and perform logical processing on the associated features based on the preset thought chain to obtain text description information in the structured data; wherein, the text description information includes at least one of title, summary and tag.

[0055] The matching module 830 is further configured to: if the user's search information contains text information, image information, or video information, determine the feature vector corresponding to the user's search information based on the feature vectors corresponding to the text information, image information, or video information respectively; for each video stream in the video library, determine the first similarity by weighted sum of the similarities between the feature vector and the multidimensional semantic vectors in the structured data corresponding to each granularity video in the video stream; determine a preset number of video streams as the first video stream based on the order of the first similarity from largest to smallest; if the user's search information contains tag information, match the tag information with the text description information in the structured data corresponding to the video stream, and determine the matched video stream as the second video stream; determine the intersection of the first video stream and the second video stream as the target video stream; and determine the target video content based on the matching degree between each granularity video in the target video stream and the feature vector and tag information respectively.

[0056] The matching module 830 is further configured to: in each granular video corresponding to the target video stream, determine the granular video with a similarity greater than a similarity threshold with the feature vector as the first target content, and determine the granular video with a matching degree greater than a matching degree threshold with the tag information as the second target content; and determine the intersection of the first target content and the second target content as the target video content.

[0057] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of the apparatus and the terminal to which it is applied.

[0058] The embodiments of the video content retrieval device described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor responsible for video content retrieval loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 9 The diagram shown is a hardware structure diagram of a computer device containing the video content retrieval device as described in this specification, except... Figure 9 In addition to the processor 910, memory 930, network interface 920, and non-volatile memory 940 shown, the server or electronic device where the video content retrieval device 931 is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0059] Accordingly, this specification also provides a video content retrieval device, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the above-described video content retrieval method.

[0060] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0061] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0062] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0063] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0064] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0065] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A video content retrieval method, characterized in that, include: The video streams in the video library are traversed, and the first frame sequence of the video stream is obtained based on the first frame rate. If the pixel difference between adjacent video frames in the first frame sequence is greater than the first threshold, the video segment corresponding to the adjacent video frame is determined as a candidate segment, and the key frame in the candidate segment is obtained based on the second frame rate; wherein, the second frame rate is greater than the first frame rate. The process involves determining the granularity of the video stream and then using a pre-trained processing model to determine the structured data corresponding to each granularity of the video stream. Each granularity of the video stream includes the entire video stream, the keyframes of the video stream, and each sub-video stream within the video stream. The structured data includes multi-dimensional semantic vectors and textual description information. The user search information is matched with the structured data corresponding to each granularity video of each video stream in the video library, and the target video content is determined based on the matching results; wherein, the user search information includes at least one of text information, image information, tag information and video information.

2. The method according to claim 1, characterized in that, The step of obtaining keyframes from the candidate segments based on the second frame rate includes: The second frame sequence corresponding to the candidate segment is obtained based on the second frame rate; Semantic features are extracted from each video frame in the second frame sequence to obtain the semantic feature vector corresponding to each video frame. The scene switching points in the second frame sequence are determined based on the semantic feature vector corresponding to each video frame. The candidate segment is divided into multiple sub-segments based on different scene switching points, and the keyframes in each sub-segment are determined.

3. The method according to claim 1, characterized in that, The pre-trained processing model includes a pre-trained multimodal video understanding model and a pre-trained multimodal vector model. The step of determining the structured data corresponding to each granularity of video based on the pre-trained processing model includes: Based on the pre-trained multimodal video understanding model, multi-dimensional features corresponding to at least one dimension of information among the subject object, scene information, environment information and action information corresponding to each granularity video are determined, and the dimensional features corresponding to each granularity video are processed to obtain the text description information corresponding to each granularity video. Based on the pre-trained multimodal vector model, the text description information corresponding to each granularity video is encoded to obtain the multidimensional semantic vector corresponding to each granularity video.

4. The method according to claim 3, characterized in that, The pre-trained multimodal video understanding model determines the dimensional features corresponding to at least one of the following dimensions: subject object, scene information, environmental information, and action information, for each granularity of video: For each granular video, at least one dimension of information is obtained from the main object, scene information, environment information, and action information of the granular video, and the features corresponding to each dimension of information are obtained respectively; Based on preset association features, the features corresponding to each dimension of information are verified separately; if the verification passes, the dimensional features corresponding to each dimension of information are fused to obtain the multi-dimensional features corresponding to the granular video.

5. The method according to claim 3, characterized in that, The process of processing the multi-dimensional features corresponding to each granularity of video to obtain the text description information corresponding to each granularity of video includes: For each granular video, a preset thought chain corresponding to the granular video is determined, and the associated features corresponding to the preset thought chain in the multi-dimensional features of the granular video are obtained. The associated features are logically processed based on the preset thought chain to obtain the text description information in the structured data; wherein, the text description information includes at least one of title, summary and tag.

6. The method according to claim 1, characterized in that, The step of matching user search information with structured data corresponding to each granularity of video streams in the video library, and determining the target video content based on the matching results, includes: If the user search information contains text information, image information, or video information, then the feature vector corresponding to the user search information is determined based on the feature vectors corresponding to the text information, image information, or video information, respectively. For each video stream in the video library, the weighted sum of the similarities between the feature vector and the multidimensional semantic vectors in the structured data corresponding to each granularity video in the video stream is determined as the first similarity. Based on the order of the first similarity from largest to smallest, a preset number of video streams are determined as the first video stream; If the user search information contains tag information, then the tag information is matched with the text description information in the structured data corresponding to the video stream, and the matched video stream is determined as the second video stream; The intersection of the first video stream and the second video stream is determined as the target video stream; The target video content is determined based on the matching degree between each granularity video in the target video stream and the feature vector and the tag information.

7. The method according to claim 6, characterized in that, The step of determining the target video content based on the matching degree between each granularity video in the target video stream and the feature vector and the tag information includes: In each granular video corresponding to the target video stream, the granular video with a similarity greater than a similarity threshold with the feature vector is determined as the first target content, and the granular video with a matching degree greater than a matching degree threshold with the tag information is determined as the second target content; The intersection of the first target content and the second target content is the target video content.

8. The method according to claim 1, characterized in that, The method further includes: Obtain user feedback on the target video content, and determine the content to be optimized in the target video content based on the feedback information; The pre-trained processing model is updated based on the content to be optimized.

9. A video content retrieval device, the device comprising: The acquisition module is used to traverse each video stream in the video library, acquire the first frame sequence of the video stream based on a first frame rate, and if the pixel difference between adjacent video frames in the first frame sequence is greater than a first threshold, then the video segment corresponding to the adjacent video frame is determined as a candidate segment, and key frames in the candidate segment are acquired based on a second frame rate; wherein, the second frame rate is greater than the first frame rate. The determination module is used to determine the granularity videos corresponding to the video stream, and to determine the structured data corresponding to each granularity video according to the pre-trained processing model; each granularity video includes the entire video stream, the keyframes of the video stream, and each sub-video stream in the video stream, and the structured data includes multi-dimensional semantic vectors and text description information; The matching module is used to match user search information with the structured data corresponding to each granularity video of each video stream in the video library, and determine the target video content based on the matching results; wherein, the user search information includes at least one of text information, image information, tag information and video information.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 7.

Images