Video clip retrieval method and device, electronic equipment and readable storage medium

Abstract

The invention provides a video clip retrieval method and device, electronic equipment and a readable storage medium, and the method comprises the steps: carrying out the scene segmentation, motion judgment, aesthetic scoring and textbox recognition of an original video based on a plurality of data processing operators, and obtaining a multi-segmentation video; generating a vector database corresponding to the multi-segmented video according to the semantic analysis result of the multi-segmented video; vector retrieval is carried out in the vector database according to the user demand instruction, and a user-defined open source data set is constructed and comprises a plurality of target video clips. According to the method, through multi-dimensional automatic screening and semantic vector retrieval, efficient and automatic generation of a high-quality open source video data set customized as required is realized, the automation level and practicability of data construction are remarkably improved, the specific requirements of different fields and application scenes for data are met, and the method is suitable for large-scale popularization and application. And more accurate and more targeted data support is provided for model training.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video clip retrieval method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, high-quality training data has become a core driver for improving model performance. Currently, the number of open-source datasets continues to grow, but researchers and developers still face significant challenges in accurately acquiring high-quality data that meets specific needs from this vast resource pool. At the same time, industry model training is accelerating its shift from general domains to vertical fields such as healthcare, finance, and manufacturing, placing new demands on training data that are more vertical, refined, and personalized.

[0003] However, existing data preparation processes rely heavily on manual operations, typically requiring the collection of datasets across multiple platforms and the sequential execution of tedious steps such as downloading, summarizing, filtering, deduplication, and quality assessment. This is not only inefficient and costly, but also makes it difficult to guarantee data consistency and applicability, severely restricting the efficient development and iteration of large-scale models in vertical domains. Summary of the Invention

[0004] This invention provides a video clip retrieval method, apparatus, electronic device, and readable storage medium to overcome the shortcomings of existing technologies in efficiently and accurately acquiring high-quality training data that meets the refined and personalized needs of vertical domains. It enables the efficient construction of highly domain-aligned personalized datasets, meets the specific data requirements of different domains and application scenarios, and provides more accurate and targeted data support for model training.

[0005] On one hand, the present invention provides a video segment retrieval method, comprising: performing scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain multi-segmented videos; generating a vector database corresponding to the multi-segmented videos based on the semantic parsing results of the multi-segmented videos; performing vector retrieval in the vector database according to user requirements to construct a custom open-source dataset, wherein the custom open-source dataset includes multiple target video segments.

[0006] Furthermore, the process of performing scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain a multi-segmented video includes: based on the scene segmentation operator, dividing the original video into multiple video segments according to the content change scores between adjacent frames in the original video; each video segment corresponds to a visual scene; based on the motion determination operator, calculating the global average value of the optical flow vector amplitude between adjacent frames in each video segment, and filtering out video segments with insufficient motion based on the global average value; based on the aesthetic scoring operator, evaluating the human preference score, naturalness, and distortion degree of each video segment, and filtering out video segments with insufficient aesthetics based on the human preference score, naturalness, and distortion degree; based on the text box recognition operator, performing text box detection on each video segment, and filtering out video segments with excessive text content based on the text box detection results; wherein, the multi-segmented video is one or more video segments retained after motion determination, aesthetic scoring, and text box recognition.

[0007] Furthermore, the step of generating a vector database corresponding to the multi-segmented video based on the semantic parsing results of the multi-segmented video includes: extracting keyframes from the multi-segmented video and concatenating them sequentially according to the time sequence to obtain a long image to be parsed; performing semantic parsing on the long image to be parsed from the dimensions of video content, visual perspective, and style to obtain the semantic parsing results; converting the semantic parsing results into vector form and storing them in an initially empty database to obtain the vector database; wherein, there is a one-to-one mapping relationship between the semantic parsing results in vector form and the multi-segmented video.

[0008] Furthermore, the step of performing vector retrieval in the vector database according to user demand instructions to construct a custom open-source dataset includes: converting the user demand instructions into vector form to obtain a query vector; calculating the vector similarity between the query vector and multiple vectors in the vector database; determining the vectors with vector similarity higher than a preset similarity threshold as target vectors; and outputting the target video segment corresponding to the target vector to obtain the custom open-source dataset.

[0009] Further, the step of outputting the target video segment corresponding to the target vector to obtain the custom open-source dataset includes: performing quantitative evaluation on the target video segment corresponding to the target vector; if the quantitative evaluation result is qualified, storing the target video segment corresponding to the target vector into the initially empty custom open-source dataset; if the quantitative evaluation result is unqualified, re-optimizing the retrieval based on the quantitative evaluation result.

[0010] Furthermore, the original video is either a user-owned private video or an open-source video.

[0011] Secondly, the present invention also provides a video segment retrieval device, comprising: a video processing module, used to perform scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain multi-segmented videos; a vector database generation module, used to generate a vector database corresponding to the multi-segmented videos based on the semantic parsing results of the multi-segmented videos; and a target video segment retrieval module, used to perform vector retrieval in the vector database according to user requirements to construct a custom open-source dataset, wherein the custom open-source dataset includes multiple target video segments.

[0012] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video segment retrieval method as described above.

[0013] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video segment retrieval method as described above.

[0014] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video segment retrieval method as described above.

[0015] The video segment retrieval method provided by this invention performs scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video using multiple data processing operators to obtain multi-segmented videos. Based on the semantic parsing results of the multi-segmented videos, a vector database corresponding to each video segment is generated. Then, vector retrieval is performed in the vector database according to user requirements to construct a custom open-source dataset, which includes multiple target video segments. This method achieves efficient automatic generation of high-quality, on-demand customized open-source video datasets through multi-dimensional automatic filtering and semantic vector retrieval, significantly improving the automation level and practicality of data construction. It meets the specific data needs of different fields and application scenarios, providing more accurate and targeted data support for model training. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1This is a flowchart illustrating the video segment retrieval method provided in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the integrated platform for the video segment retrieval method provided in this embodiment of the invention.

[0019] Figure 3 This is a schematic diagram of the video clip retrieval device provided in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] It's important to note that with the rapid development of artificial intelligence, data has become increasingly crucial as a core driving force. Currently, the number of open-source datasets is growing rapidly; however, researchers and developers face new challenges in accurately locating and matching their needs within massive datasets and acquiring high-quality data resources. Simultaneously, industry model training is accelerating its expansion from general domains to vertical domains, leading to a trend towards more vertical, refined, and personalized training data requirements. However, traditional model training relies on manually collecting multiple datasets, followed by a series of cumbersome processes such as downloading, summarizing, filtering, deduplication, and quality checks. This is not only time-consuming and labor-intensive but also severely restricts model development efficiency.

[0023] In view of this, the present invention proposes a video segment retrieval method, specifically, Figure 1 A flowchart illustrating the video segment retrieval method provided in an embodiment of the present invention is shown.

[0024] like Figure 1 As shown, the method includes: S110, performing scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain multi-segmented videos; S120, generating a vector database corresponding to the multi-segmented videos based on the semantic parsing results of the multi-segmented videos; S130, performing vector retrieval in the vector database according to user requirements to construct a custom open-source dataset, wherein the custom open-source dataset includes multiple target video segments.

[0025] It should be noted that the automatic generation of custom open-source datasets provided in this embodiment of the invention can be integrated into a Data Cube platform. The Data Cube platform is a comprehensive big data platform that integrates data collection, integration, analysis, visualization, and intelligent decision support.

[0026] The following will provide a detailed description of steps S110-S130 and related steps.

[0027] S110 performs scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on a variety of data processing operators to obtain multi-segmented videos.

[0028] As is easily understood, this step performs multi-dimensional intelligent analysis on the original video (private video provided by the user or open-source video provided through public channels), thereby automatically segmenting the original video into multiple semantically meaningful video segments, i.e., multi-segmented video. This process is completed collaboratively by a set of preset data processing operators. Each data processing operator is a small intelligent module that performs a specific analysis task, and each operator is responsible for a type of video processing task.

[0029] In one specific embodiment, the data processing operators include a scene segmentation operator, a motion determination operator, an aesthetic scoring operator, and a text box recognition operator.

[0030] The scene segmentation operator is used to analyze the original video, identify scene boundary points in the original video (such as when the camera switches or the content of the screen changes significantly), and then divide the original video into several continuous video segments, each video segment corresponding to a relatively complete visual scene.

[0031] The motion detection operator is used to perform dynamic content analysis on each video segment to determine whether there is obvious object movement or camera movement (such as people walking, vehicles driving, camera panning, etc.) in the video segment, thereby filtering out video segments with insufficient motion.

[0032] The aesthetic scoring operator is used to evaluate the visual appeal of each video segment. Based on a learning model of human aesthetic preferences, it comprehensively considers factors such as composition, color harmony, clarity, and exposure to score each video segment, thereby filtering out video segments with aesthetic deficiencies.

[0033] Text box recognition operators are used to detect whether each video clip contains visible text areas (such as subtitles), thereby filtering out video clips with excessive text content.

[0034] After collaborative processing by various data processing operators, a multi-segmented video can be obtained, which includes video segments corresponding to multiple visual scenes, and then step S120 is executed.

[0035] S120, Based on the semantic parsing results of the multi-segmented video, generate a vector database corresponding to the multi-segmented video.

[0036] Specifically, pre-trained multimodal artificial intelligence models (such as the Qwen2.5-VL-Instruct model) are used to perform semantic analysis on keyframes or short temporal segments of multi-segmented videos to obtain semantic parsing results. The semantic parsing results are a set of natural language-style labels or structured semantic descriptions, including but not limited to information such as objects, actions, scene categories, emotional atmosphere, and visual perspective in the video.

[0037] After obtaining the semantic parsing results, to facilitate rapid comparison and retrieval by the system, the results are converted into high-dimensional numerical vectors and indexed, resulting in a vector database corresponding to the multi-segmented videos. In the vector database, the high-dimensional numerical vectors corresponding to multi-segmented videos with similar semantics are spatially closer, while the high-dimensional numerical vectors corresponding to multi-segmented videos with significant semantic differences are spatially farther apart. The high-dimensional numerical vectors in the vector database are bound to their corresponding multi-segmented videos for quick retrieval.

[0038] This step S120 converts the originally unstructured video content into a structured, computable, and searchable vector form, enabling the system to quickly locate matching vectors and multi-segmented videos based on the user's natural language commands, thereby achieving the automated construction of custom open-source datasets.

[0039] Based on the step of generating a vector database corresponding to the multi-segmented video according to the semantic parsing results of the multi-segmented video, step S130 is further executed.

[0040] S130: Perform vector retrieval in the vector database according to user requirements, and construct a custom open-source dataset, which includes multiple target video clips.

[0041] Specifically, users input their data requirements and user requests via natural language, such as "I need a visually appealing, text-free, slow-moving forest sunrise video" or "Collect all short videos that include dialogue and are set in an indoor office."

[0042] After receiving the user's request instruction, the user's request instruction is converted into a high-dimensional semantic vector as a query vector. This query vector represents the target location of the video content that the user expects in the semantic space.

[0043] Subsequently, vector similarity retrieval is performed in the vector database. Specifically, the vector similarity between the query vector and all video vectors in the database is calculated (using metrics such as cosine similarity or Euclidean distance), and the vectors are sorted from highest to lowest similarity. The most matching target video segments are then returned, forming a custom open-source dataset. This custom open-source dataset can be used to train artificial intelligence models.

[0044] In this embodiment, the original video is segmented into scenes, determined by motion, scored aesthetically, and identified by various data processing operators to obtain multi-segmented videos. A vector database corresponding to each segmented video is then generated based on the semantic parsing results. Vector retrieval is performed in the vector database according to user requirements to construct a custom open-source dataset, which includes multiple target video segments. This method achieves efficient automatic generation of high-quality, on-demand customized open-source video datasets through multi-dimensional automatic filtering and semantic vector retrieval. This significantly improves the automation level and practicality of data construction, meets the specific data needs of different fields and application scenarios, and provides more accurate and targeted data support for model training.

[0045] Based on the above embodiments, the processing procedure of the original video in step S110 will be described in detail below.

[0046] The original video is segmented into multiple segments based on various data processing operators, including scene segmentation, motion determination, aesthetic scoring, and text box recognition. The process involves: using a scene segmentation operator to divide the original video into multiple video segments based on the content change scores between adjacent frames; each video segment corresponds to a visual scene; using a motion determination operator to calculate the global average of the optical flow vector amplitude between adjacent frames in each video segment, and filtering out video segments with insufficient motion based on this global average; using an aesthetic scoring operator to evaluate the human preference score, naturalness, and distortion level of each video segment, and filtering out video segments with insufficient aesthetics based on these scores; and using a text box recognition operator to perform text box detection on each video segment, and filtering out video segments with excessive text content based on the text box detection results. Each segmented video consists of one or more video segments retained after motion determination, aesthetic scoring, and text box recognition.

[0047] As is easily understood, this embodiment performs multi-dimensional intelligent analysis on the original video provided by the user and automatically segments it into multiple video segments with specific semantic value and structured attributes, i.e., multi-segmented video. This process is implemented through the data processing module in the DataCube platform, which is built on the Ray framework and supports efficient processing of petabyte-level data.

[0048] Scene segmentation is achieved using scene segmentation operators. Scene segmentation aims to identify and segment different visual scenes in the original video. Specifically, it utilizes the ContentDetector algorithm from the PySceneDetect toolkit to determine segmentation points based on the content change score between adjacent frames in the original video. The content change score is a quantitative indicator that measures the difference in image features between two frames, such as the degree of change in color distribution and edge information. When the content change score exceeds a preset threshold, this point is considered a scene boundary, and the original video is segmented into two video segments at this point.

[0049] For example, in an original video that transitions from indoors to outdoors, the significant differences in indoor and outdoor lighting conditions and background elements cause the content change score to rise sharply, triggering a segmentation operation. Ultimately, each segmented video clip represents a relatively independent and continuous visual scene.

[0050] Motion determination is implemented using a motion determination operator. Motion determination is used to evaluate the dynamic characteristics of objects or cameras in each video segment. Specifically, optical flow estimation techniques are used to calculate the optical flow vector amplitude of each pixel between adjacent frames in the video segment, and then the global average of these optical flow vector amplitudes is taken. To eliminate the influence of resolution differences, the global average is further normalized by dividing it by the length of the shorter side of the current frame image. If the normalized global average is lower than a set threshold, the video segment is considered to lack sufficient motion information and may not be suitable for certain application scenarios (such as action recognition training), and therefore is filtered out.

[0051] For example, in a landscape time-lapse video, although the time span is long, the normalized global average value is low because the main subject in the video is almost still, and it will be marked as "insufficient motion" and excluded.

[0052] Aesthetic scoring is implemented using an aesthetic scoring operator. The aesthetic score focuses on the visual aesthetic quality of each video segment. Specifically, the MUSIQ model is used to evaluate how well video frames in a video segment conform to human preferences, while the BRISQUE model is used to evaluate the naturalness and distortion of video frames. The former emphasizes capturing subjective aesthetic factors such as composition and color matching, while the latter focuses on the technical quality of the image, such as sharpness and noise levels.

[0053] For each video segment, an overall aesthetic score is generated by comprehensively considering the human preference score, naturalness, and distortion level of its keyframes. If the aesthetic score of a video segment does not meet the preset standard (e.g., the total score is below a certain threshold), it is considered to be of substandard aesthetic quality and is filtered out. The aesthetic score helps ensure that the video segments in the custom-generated dataset have high visual appeal and technical quality.

[0054] Text box recognition is implemented using a text box recognition operator. Text box recognition is primarily used to detect excessive text interference in video clips. Specifically, the PaddleOCR multilingual model is used for text box region detection, but it does not recognize the actual text content. The text box recognition operator can locate the positions where text may exist in the video frame and determine whether text is present. For example, subtitles, advertising slogans, road signs, etc., in the video will be marked as "text-containing regions".

[0055] Based on the text box detection results, if the text area in a video clip occupies too large a proportion (exceeding the set ratio), the video clip is considered to contain too much text interference and is not suitable for certain applications (such as image classification tasks that require a clean background), and is therefore filtered out.

[0056] It should be noted that the preset threshold, set threshold, preset standard, and set ratio in this embodiment can be set according to actual needs, and are not specifically limited here.

[0057] After steps such as scene segmentation, motion determination, aesthetic scoring, and text box recognition, the remaining video segments constitute the final multi-segment video. The multi-segment video not only includes the original video content, but can also use the indicator data (such as scene number, motion intensity level, aesthetic score, whether it contains text regions and their positions) obtained from the steps of scene segmentation, motion determination, aesthetic scoring, and text box recognition as additional attribute information.

[0058] Furthermore, to ensure the efficiency of the entire data processing flow, this embodiment employs the Ray framework for distributed computing. The Ray framework supports large-scale parallel computing, fully utilizing cluster resources and improving data processing efficiency. Specific measures include: 1) Parallel processing: dividing data into multiple small blocks and distributing them to different computing nodes for parallel processing; 2) Load balancing: dynamically adjusting the workload of each computing node to ensure full resource utilization; 3) Fault tolerance mechanism: a robust fault tolerance mechanism is designed to ensure the data processing task continues smoothly even if some nodes fail.

[0059] In summary, step S110, through a series of automated and intelligent data processing methods, achieved in-depth analysis and optimized screening of the original video, laying a solid foundation for the subsequent construction of a high-quality open-source dataset.

[0060] Based on the above embodiments, the following will further describe in detail the process of constructing the vector database in step S120.

[0061] Based on the semantic parsing results of the multi-segmented video, a vector database corresponding to the multi-segmented video is generated, including: extracting keyframes from the multi-segmented video and concatenating them sequentially according to the time sequence to obtain a long image to be parsed; performing semantic parsing on the long image to be parsed from the dimensions of video content, visual perspective, and style to obtain semantic parsing results; converting the semantic parsing results into vector form and storing them in an initially empty database to obtain a vector database; wherein, there is a one-to-one mapping relationship between the semantic parsing results in vector form and the multi-segmented video.

[0062] The process is straightforward: First, frames are extracted from the multi-segmented video at preset time intervals to ensure accurate extraction of representative keyframes from each segment. The preset time intervals can be flexibly set based on video length and content complexity, such as extracting a frame every second or every few seconds. Next, the extracted keyframes are stitched together sequentially in chronological order to form a complete long image, preserving key information from the multi-segmented video (such as character movements and scene changes).

[0063] Next, the advanced Qwen2.5-VL-Instruct model was used to perform in-depth analysis of the stitched long image to be parsed. The Qwen2.5-VL-Instruct model has excellent semantic parsing capabilities and can perform fine-grained parsing of image content from three main dimensions.

[0064] The first dimension is the content level: the model can identify and understand the key information conveyed by the image, including but not limited to the actions and postures of people, detailed elements in the scene (such as buildings and natural landscapes), and the characteristics of various objects (such as color and shape). For example, in an outdoor activity video, the model can identify the scene of "a group of people having a picnic by the lake".

[0065] The second dimension is visual perspective: the model can deeply analyze the shooting angle of an image (such as overhead or low-angle shots), composition (such as the rule of thirds or central symmetry), and the location of the focal point, revealing the photographer's unique perspective and intention during the creative process. For example, it can determine that a certain image was shot from a low angle to enhance the subject's sense of majesty.

[0066] The third dimension is style: the model can accurately identify and describe the artistic style presented in the image, whether it is the characteristics of different art movements such as realism, romanticism or modernism, which helps to distinguish different types of video content, such as documentaries and art short films.

[0067] Through this multi-dimensional, fine-grained semantic parsing, this embodiment can more comprehensively understand and describe video content, thereby generating detailed and accurate semantic parsing results.

[0068] After obtaining the semantic parsing results, they are converted into high-dimensional numerical vectors. This process can be aided by a pre-trained deep learning model, which can map complex semantic information into a fixed-length array of numbers. Vectors corresponding to video clips with similar content are spatially closer, while vectors corresponding to video clips with different content are farther apart.

[0069] Subsequently, these vectors, along with the original video ID, timestamp range, and other metadata (such as the scene number, motion intensity level, aesthetic score, whether it contains text regions and their locations, etc. mentioned in the above embodiments), are stored in an initially empty database to obtain a vector database.

[0070] The vector database is implemented based on the Milvus distributed database, supporting highly available, highly scalable, and high-performance vector data storage and retrieval services. Each Milvus node runs an instance responsible for handling vector data storage and retrieval tasks. Nodes communicate and synchronize data through a distributed coordination mechanism (such as ETCD) to ensure system stability and data consistency. ETCD, as a distributed coordination service, manages the metadata and status information of Milvus nodes. The cluster adopts a multi-node deployment to guarantee high availability and data consistency.

[0071] This embodiment extracts keyframes and stitches them into a long graph. It then uses the Qwen2.5-VL-Instruct model to perform multi-dimensional semantic parsing, converts the parsing results into vector form, and stores these vector data in a distributed vector database built on Milvus. This realizes the transformation from unstructured video content to structured, computable, and searchable vector data, laying the foundation for the automated construction of subsequent custom open-source datasets.

[0072] Based on the above embodiments, the process of constructing a custom open-source dataset in step S130 will be described in detail below.

[0073] Based on user requests, vector retrieval is performed in the vector database to construct a custom open-source dataset. This includes: converting user requests into vector form to obtain query vectors; calculating the vector similarity between the query vector and multiple vectors in the vector database; identifying vectors with similarity higher than a preset similarity threshold as target vectors; and outputting the target video segments corresponding to the target vectors to obtain the custom open-source dataset.

[0074] Output the target video segments corresponding to the target vectors to obtain a custom open-source dataset, including: quantitatively evaluating the target video segments corresponding to the target vectors; if the quantitative evaluation results are satisfactory, storing the target video segments corresponding to the target vectors into the initially empty custom open-source dataset; if the quantitative evaluation results are unsatisfactory, re-optimizing the retrieval based on the quantitative evaluation results.

[0075] The process is straightforward: first, it receives user requests in natural language, such as "finding a visually appealing forest sunrise video without text interference." Then, the Qwen3-embedding model transforms the user request into a fixed-length numerical vector, the query vector. This query vector represents the location of the desired content in the semantic space, facilitating subsequent comparison with video vectors in the database.

[0076] Then, HNSW (Hierarchical Navigable Small World) is used as the retrieval engine to quickly find several video vectors most similar to the query vector in the vector database. HNSW is an efficient approximate nearest neighbor search algorithm that can quickly locate similar vectors in large-scale datasets, greatly improving retrieval speed and accuracy.

[0077] For each candidate video vector, calculate its vector similarity to the query vector, using metrics such as cosine similarity or Euclidean distance. If the vector similarity of a candidate video vector is higher than a preset similarity threshold, it is marked as the target vector, and the video segment corresponding to the target vector is considered the content that best meets the user's needs, i.e., the target video segment.

[0078] For each candidate vector, the system calculates its similarity to the query vector (usually using metrics such as cosine similarity or Euclidean distance). If the similarity of a candidate vector is higher than a preset threshold, it is marked as a target vector. The video segments corresponding to these target vectors are considered to be the content that best meets the user's needs, i.e., the target video segments.

[0079] To further ensure data quality, this embodiment performs a quantitative evaluation of the target video segment corresponding to each target vector. This process involves multiple dimensions, such as whether the aesthetic score meets the requirements and whether there is excessive text interference. If all quantitative indicators of a video segment are qualified, it is added to the initially empty custom open-source dataset; otherwise, the retrieval strategy is adjusted according to the specific results of the quantitative evaluation, and a more precise retrieval is performed again until a video segment that meets all conditions is found.

[0080] To further improve the quality of search results and user experience, this embodiment also introduces a re-ranking module. This module uses the Qwen3-reranker model to perform secondary analysis and evaluation on the retrieved target vectors. Qwen3-reranker can deeply understand the underlying features of each video segment and re-rank them according to user preferences. For example, for video segments of different styles on the same topic, the model can adjust the order in the results list based on the user's specific preferences (such as a preference for realistic styles over abstract styles). The re-ranking process is deployed in a multi-machine, multi-GPU environment using the VLLM framework, ensuring high-performance response even under a large number of concurrent requests.

[0081] After the above processing, the final output is a custom open-source dataset consisting of multiple carefully selected video clips.

[0082] In some other embodiments, Figure 2 A schematic diagram of the integrated platform for the video segment retrieval method provided in an embodiment of the present invention is shown.

[0083] like Figure 2 As shown, the platform includes a data processing module, a video understanding module, a data retrieval module, and a data evaluation module. The data processing module, based on Ray, implements an automated pipeline for processing petabyte-scale data. Specific data processing operators include scene segmentation, motion scoring, aesthetic scoring, and OCR. The video understanding module deeply integrates the CLIP Understanding engine to perform semantic analysis on multimodal samples from multiple dimensions such as content, behavior, perspective, and style, constructing detailed data profiles for each sample. The data retrieval module adopts a Hybrid Retrieval system to achieve millisecond-level accurate retrieval of cross-modal features. The data evaluation module evaluates the dataset construction effect through Data Evaluation, ensuring the generation of personalized, high-quality datasets.

[0084] The video clip retrieval method provided in this embodiment of the invention has the following four advantages.

[0085] Firstly, it improves data retrieval and processing efficiency. Specifically: 1) Sub-second response: By inputting data requests in natural language, the platform can provide feedback and generate datasets in as little as a second, greatly shortening the time from request submission to data acquisition. Compared to traditional manual retrieval and processing workflows, efficiency is significantly improved; 2) Eliminating cumbersome processes: It eliminates a series of cumbersome processes such as manual retrieval, downloading, summarizing, filtering, deduplication, and quality inspection, greatly reducing manpower and time costs, enabling users to obtain and use data more efficiently, and improving the overall efficiency of model development.

[0086] Secondly, improve data quality and relevance. Specifically: 1) Deep semantic parsing: Deeply integrate the CLIPUnderstanding engine to perform semantic parsing on multimodal samples from multiple dimensions such as ontology, behavior, perspective, and style, constructing detailed data profiles for each sample to ensure that the retrieved data is highly relevant to user needs and provides high-quality materials for model training; 2) Precise recall: Adopt the Hybrid Retrieval system to achieve millisecond-level precise recall of cross-modal features, further improving the accuracy and usability of data and avoiding data bias and mismatch problems that may occur in traditional methods.

[0087] Thirdly, reducing the cost of data use. Specifically, this involves the continuously growing data volume: currently, a base of 100 million data samples has been accumulated, and the number continues to grow, providing users with abundant data resources. Users do not need to collect and organize large amounts of data themselves, further reducing the cost of data acquisition.

[0088] Fourthly, enhance the personalization and domain relevance of datasets. Specifically, this involves personalized dataset construction: relying on a personalized dataset construction engine, it can match target scenarios with one click and efficiently build personalized datasets with high domain relevance, meeting the specific data needs of different fields and application scenarios, such as AI-assisted athlete training and game scene generation, providing more accurate and targeted data support for model training.

[0089] Corresponding to the video segment retrieval methods described in the above embodiments, the present invention also proposes a video segment retrieval device. Specifically, Figure 3 A schematic diagram of the structure of the video clip retrieval device provided in an embodiment of the present invention is shown.

[0090] like Figure 3 As shown, the device includes: a video processing module 310, used to perform scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain multi-segmented videos; a vector database generation module 320, used to generate a vector database corresponding to the multi-segmented videos based on the semantic parsing results of the multi-segmented videos; and a target video segment retrieval module 330, used to perform vector retrieval in the vector database according to user requirements to construct a custom open-source dataset, wherein the custom open-source dataset includes multiple target video segments.

[0091] In this embodiment, the video processing module 310 performs scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video using various data processing operators to obtain multi-segmented videos. The vector database generation module 320 generates a vector database corresponding to the multi-segmented videos based on the semantic parsing results. Then, the target video segment retrieval module 330 performs vector retrieval in the vector database according to user requirements, constructing a custom open-source dataset that includes multiple target video segments. This device achieves efficient automatic generation of high-quality, on-demand customized open-source video datasets through multi-dimensional automatic filtering and semantic vector retrieval, significantly improving the automation level and practicality of data construction, meeting the specific data needs of different fields and application scenarios, and providing more accurate and targeted data support for model training.

[0092] It should be noted that the video segment retrieval device provided in this embodiment of the invention can be referred to in correspondence with the video segment retrieval methods described in the above embodiments, and will not be repeated here.

[0093] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a video segment retrieval method. This method includes: performing scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain multi-segmented videos; generating a vector database corresponding to the multi-segmented videos based on the semantic parsing results of the multi-segmented videos; and performing vector retrieval in the vector database according to user requirements to construct a custom open-source dataset, which includes multiple target video segments.

[0094] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video segment retrieval method provided by the above methods. The method includes: performing scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain multi-segmented videos; generating a vector database corresponding to the multi-segmented videos based on the semantic parsing results of the multi-segmented videos; and performing vector retrieval in the vector database according to user requirements to construct a custom open-source dataset, which includes multiple target video segments.

[0096] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the video segment retrieval method provided by the above methods. The method includes: performing scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on multiple data processing operators to obtain multi-segmented videos; generating a vector database corresponding to the multi-segmented videos based on the semantic parsing results of the multi-segmented videos; and performing vector retrieval in the vector database according to user requirements to construct a custom open-source dataset, wherein the custom open-source dataset includes multiple target video segments.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video clip retrieval method, characterized in that, include: The original video is segmented into scenes, determined by motion, scored aesthetically, and recognized by a variety of data processing operators to obtain multi-segmented videos. Based on the semantic parsing results of the multi-segmented video, a vector database corresponding to the multi-segmented video is generated; Based on user requirements, vector retrieval is performed in the vector database to construct a custom open-source dataset, which includes multiple target video clips.

2. The video segment retrieval method according to claim 1, characterized in that, The process involves scene segmentation, motion determination, aesthetic scoring, and text box recognition of the original video using multiple data processing operators, resulting in multi-segmented videos, including: Based on the scene segmentation operator, the original video is segmented into multiple video segments according to the content change scores between adjacent frames in the original video; each video segment corresponds to a visual scene. Based on the motion determination operator, the global average value of the optical flow vector amplitude between adjacent frames in each video segment is calculated, and video segments with insufficient motion are filtered out based on the global average value. Based on the aesthetic scoring operator, the human preference score, naturalness and distortion of each video segment are evaluated, and video segments with insufficient aesthetics are filtered out according to the human preference score, naturalness and distortion. Based on the text box recognition operator, text box detection is performed on each video segment, and video segments with excessive text content are filtered out based on the text box detection results. The multi-segmented video refers to one or more video segments that have been retained after motion determination, aesthetic scoring, and text box recognition.

3. The video segment retrieval method according to claim 1, characterized in that, The step of generating a vector database corresponding to the multi-segmented video based on the semantic parsing results of the multi-segmented video includes: Key frames are extracted from the multi-segmented video and stitched together sequentially according to time sequence to obtain the long image to be analyzed. The semantic analysis of the long image to be analyzed is performed from the dimensions of video content, visual perspective, and style to obtain the semantic analysis result. The semantic parsing results are converted into vector form and stored in an initially empty database to obtain the vector database; wherein, there is a one-to-one mapping relationship between the semantic parsing results in vector form and the multi-segmented video.

4. The video segment retrieval method according to claim 1, characterized in that, The process of performing vector retrieval in the vector database according to user requirements and constructing a custom open-source dataset includes: The user request instructions are converted into vector form to obtain the query vector; Calculate the vector similarity between the query vector and multiple vectors in the vector database; Vectors with a similarity higher than a preset similarity threshold are identified as target vectors; Output the target video segment corresponding to the target vector to obtain the custom open-source dataset.

5. The video segment retrieval method according to claim 4, characterized in that, The process of outputting the target video segment corresponding to the target vector yields the custom open-source dataset, including: The target video segment corresponding to the target vector is quantitatively evaluated; If the quantitative evaluation results are satisfactory, the target video segment corresponding to the target vector will be stored in the initially empty custom open-source dataset. If the quantitative evaluation result is unsatisfactory, the search will be re-optimized based on the quantitative evaluation result.

6. The video segment retrieval method according to any one of claims 1-5, characterized in that, The original video is either a user-owned private video or an open-source video.

7. A video clip retrieval device, characterized in that, include: The video processing module is used to perform scene segmentation, motion determination, aesthetic scoring, and text box recognition on the original video based on a variety of data processing operators, resulting in multi-segmented videos. The vector database generation module is used to generate a vector database corresponding to the multi-segmented video based on the semantic parsing results of the multi-segmented video. The target video segment retrieval module is used to perform vector retrieval in the vector database according to user requirements and construct a custom open-source dataset, which includes multiple target video segments.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the video segment retrieval method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video segment retrieval method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video segment retrieval method as described in any one of claims 1 to 6.

Images