Method for copying and classifying index of film and television materials based on intelligent voice search

Abstract

The application discloses a film and television material copying and classification index establishing method based on intelligent voice retrieval, which comprises the following steps: starting an analysis thread in parallel when a copying engine transmits a video file, extracting container metadata and auxiliary file information, combining a file name regular analysis to complete structured metadata set generation; performing content analysis on the copied file, cutting scenes and extracting representative frames for classification through interframe difference detection, synchronously performing voice recognition and voiceprint extraction on an audio track to form content analysis features; aggregating multi-dimensional features to construct a label system and establish a hybrid index library; receiving user voice queries, generating multi-dimensional query features through streaming recognition and entity extraction, combining voiceprint verification and intent expansion, searching in the hybrid index in stages, weighting and sorting to push results, supporting secondary retrieval triggered by supplementary inquiries, realizing film and television material copying and classification, and providing material retrieval precision positioning assistance for users.

Description

Technical Field

[0001] This invention relates to the field of film and television material management technology, and in particular to a method for copying and classifying film and television materials based on intelligent voice retrieval. Background Technology

[0002] Film and television footage management is a core aspect of film and television production, advertising creativity, and short video adaptation. Post-production staff need to quickly locate target clips from massive amounts of original footage and categorize them. Currently, the industry standard workflow involves copying footage from memory cards to a workstation's solid-state drive. After the copy is complete, post-production assistants or script supervisors manually categorize and label the footage according to the script supervisor's sheet. Once labeled, the footage is imported into media asset management software to create an index before retrieval and editing can begin. In this process, the copying, categorization, and retrieval steps are completely sequential. The copying process itself does not generate any index data, and all footage categorization and labeling rely on manual work, resulting in a significant time gap between the end of filming and the availability of the footage. Regarding retrieval methods, existing systems primarily rely on filename searches, metadata field filtering, or keyword matching after converting audio tracks to text using automatic speech recognition technology.

[0003] However, filename searches require users to have a precise understanding of the naming rules for the footage, and metadata filtering can only cover limited dimensions such as camera model and shooting date, failing to support the retrieval needs for scene content and dialogue semantics. When users only have a vague impression of the footage, such as remembering only a character's voice characteristics, a general shooting scene, or some dialogue fragments, existing systems lack effective fuzzy search and candidate guidance capabilities, often requiring users to repeatedly try and fail or browse the entire document to locate the target. These shortcomings make it difficult for existing technologies to meet the actual needs of rapid location and efficient management of footage in film and television post-production work. There is an urgent need for a method that can simultaneously complete the establishment of a classification index during the copying process and support users to combine precise searching with fuzzy guidance using natural language. Summary of the Invention

[0004] This invention overcomes the shortcomings of the prior art and provides a method for copying and classifying film and television materials based on intelligent voice retrieval.

[0005] To achieve the above objectives, the first aspect of this invention provides a method for copying and classifying film and television materials based on intelligent voice retrieval, comprising: S102, while the copy engine is transferring the video file from the source medium to the target storage, a parallel parsing thread is started to extract metadata from the video container header and auxiliary files in the same directory, and information is completed simultaneously through filename regular expression parsing to generate a structured meta dataset produced in parallel with the copy operation. S104: Perform content analysis on the copied file, segment the scene based on the inter-frame difference detection of the video track and extract representative frames, classify the scene using the extracted representative frames, and perform speech activity detection, streaming speech recognition and voiceprint feature extraction on the audio track to finally obtain the file content parsing features. S106, the structured meta-dataset and the file content parsing features are aggregated and a multi-dimensional label system is constructed. Based on the multi-dimensional label system, a compressed bitmap index, a text inverted index, a semantic hash index, and a voiceprint local sensitive hash index are constructed respectively to obtain a hybrid index library. S108: The user's voice query is received and converted into text by streaming speech recognition. The structured query conditions are extracted using the named entity recognition model, and the fuzzy description is mapped into terms and expanded with intent. At the same time, the voiceprint features of the query speech are extracted from the query speech, and finally integrated to generate multi-dimensional query features. S110, based on the multi-dimensional query features, a phased retrieval is performed in the hybrid index library to obtain a candidate file set. The candidate files are calculated with a weighted score of tag matching degree and text relevance, and then sorted and pushed. If the user's expected material is not found, a second retrieval is performed through information supplementation query to generate a material auxiliary positioning list for push.

[0006] In this solution, S102 specifically includes: When the copy engine transfers video files from the source medium to the target storage, it intercepts the file header data block of each video file through a metadata parsing thread that runs in parallel with the data transport pipeline. The data is then distributed to the corresponding parser according to the container format to extract the timecode track, shooting date, encoding format, resolution and device identifier, and generate the container header metadata field. The container header metadata field is stored in an in-memory hash table with the file identifier as the key. The auxiliary files in the same directory of the source medium are scanned synchronously. The field record file is established with tabs to separate column index mapping and the volume number, scene number, shot number and start time code are extracted row by row. The project file is traversed in a streaming manner to extract metadata key-value pairs. The start timecode extracted from the scene log file is aligned with the timecode track extracted from the container header using a nearest neighbor search. When the difference is less than a preset threshold, the scene number and shot number of the corresponding scene log entry are associated with the memory hash table entry of the same file identifier. For each file name, a pre-defined set of regular expression templates is applied for pattern matching to extract the camera position identifier, session number, lens number, and number of shots. The file name parsing results are then merged with the existing metadata fields in the memory hash table using a weighted voting method. Existing values ​​are retained and cross-validated with the file name parsing results, while missing fields are directly added. After parsing the container header, aligning the field record files, and completing the filenames using regular expressions, a structured metadata dataset is generated based on the set of metadata key-value pairs corresponding to each file identifier in the memory hash table.

[0007] In this solution, S104 specifically includes: The content analysis of the files already copied to the disk is performed, video frame extraction and decoding are performed with a preset sparse sampling rate, each frame is converted to HSV space and the normalized histogram of hue components is calculated, the Bartholomew's coefficient between the histograms of adjacent sampled frames is calculated as the inter-frame difference measure, and the mean and standard deviation of the inter-frame difference values ​​are calculated through a sliding window. A dynamic threshold is generated based on the mean and standard deviation of the inter-frame difference values. The current inter-frame difference value is compared with the dynamic threshold. When the threshold is exceeded, it is determined that a scene switch has been detected and the start and end positions of the scene are marked. For each scene segment, the frame corresponding to the midpoint of time is taken as a candidate representative frame, and the Laplacian variance of the candidate representative frame is calculated as a sharpness index. If the sharpness index of the candidate representative frame is less than the preset sharpness threshold, then the search continues frame by frame from the candidate representative frame to the beginning and end of the scene segment until a frame with a Laplace variance exceeding the preset sharpness threshold is found as the qualified representative frame of the current scene. Qualified representative frames are fed into a lightweight convolutional neural network scene classifier. The scene classifier is composed of stacked inverted residual structures. Each inverted residual block contains pointwise dimensionality-increasing convolution, depthwise separable convolution, compression activation module and pointwise dimensionality-reducing convolution. After global average pooling, the fully connected layer outputs the scene category labels. The audio stream is extracted from the video file. Mel frequency cepstral coefficients, first-order differences, and second-order differences are extracted with preset frame length and frame shift. After being concatenated into an acoustic feature vector sequence, it is fed into a speech activity detector based on a bidirectional long short-term memory network. After median filtering and double threshold hysteresis comparison, the speech segments are merged and speech fragments are removed to obtain a set of speech segments. The acoustic feature vector sequence of each speech segment is fed into the streaming speech recognition model. The encoder consists of a stack of multi-layer convolutional enhanced self-attention blocks. Each layer contains a feedforward module, a multi-head self-attention module, and a convolutional module. The decoder jointly connects temporal classification and attention mechanisms to output dialogue text with word-level timestamps. The acoustic feature vector sequence of each speech segment is fed into the voiceprint embedding model. The voiceprint embedding model uses an extended convolutional residual network as its backbone, stacks multiple extended convolutional layers, and uses attention statistical pooling to weight and aggregate frame-level features into voiceprint embedding vectors. Finally, it combines scene type labels and dialogue text to generate file content parsing features.

[0008] In this solution, S106 specifically includes: Using the file identifier as the primary key, the structured meta-dataset is associated and merged with the file content parsing features to construct a multi-dimensional label vector. The multi-dimensional label vector includes date label, scene label, lens label, device source label, and scene type label. For each discrete value in the multidimensional label vector, a compressed bitmap instance is created. The bitmap is divided into buckets according to the high bits of the file identifier. The low bits of the bucket are stored in an array or bitmap based on the data density. All files are traversed and the identifier is set into the corresponding label bitmap to complete the construction of the compressed bitmap index. The content keywords in the keyword tag list of each file are used as dictionary entries. The dictionary is organized in a B-tree structure. Each entry points to a skip list arranged in ascending order of file identifier. The skip list nodes record the frequency of the content words in the corresponding file, thus obtaining the text inverted index. For each file's complete dialogue text, a fixed-length information fingerprint is calculated. The hash values ​​of each word in the text are weighted and accumulated bit by bit according to the word weight, and then a binary code is generated by symbol determination. The code is divided into multiple short code segments. Each short code segment is used as a key to establish an inverted mapping and store the file identifier and the complete binary code to obtain a semantic hash index. A set of random projection vectors following a standard normal distribution is pre-generated and divided into several groups. The voiceprint embedding vector is calculated and the dot product operation is performed with each projection vector in the group to obtain a binary code. The binary code is used as the bucket identifier to store the file identifier and the voiceprint embedding vector into the corresponding bucket and to construct a voiceprint local sensitive hash index. The compressed bitmap index, text inverted index, semantic hash index, and voiceprint local sensitive hash index are integrated to generate a hybrid index library.

[0009] In this solution, S108 specifically includes: Collect the voice search commands of the target user, perform acoustic preprocessing on the voice search commands, extract the Mel frequency cepstral coefficients, first-order difference and second-order difference from the voice search commands, concatenate them into an acoustic feature vector sequence and send it into a pre-constructed streaming speech recognition model for analysis; The encoder of the streaming speech recognition model is processed by multi-layer convolutional enhanced self-attention blocks, and the decoder is jointly connected with temporal classification and attention mechanisms to output the query text sequence. At the same time, the acoustic feature vector sequence is fed into the voiceprint embedding model in parallel, and aggregated into query speech voiceprint features through extended convolutional residual network and attention statistical pooling. The query text sequence is concatenated with character embeddings and pre-trained word embeddings from the film and television field and then fed into the named entity recognition model. The context is modeled in both forward and reverse time sequences through a bidirectional long short-term memory network. The conditional random field layer is combined with the label transition matrix to perform global optimal decoding and output the preliminary query conditions. Term mapping is performed on each entity in the initial query conditions. Based on the query condition set after term mapping, intent expansion is performed. Missing or broad dimensions are marked as wildcards. Scenario conditions that match the higher-level labels are expanded along the lower-level thesaurus. Undetermined role reference conditions are marked as not being filtered, forming a query label structure consisting of a set of definite labels and a set of fuzzy labels. The query tag structure, query text sequence, and temporarily stored query voiceprint features are integrated into a multidimensional query feature.

[0010] In this solution, S110 specifically includes: Based on multidimensional query feature extraction, a label set, a fuzzy label set, and a query text sequence are determined. The compressed bitmap instance corresponding to each label value in the determined label set is retrieved from the compressed bitmap index of the hybrid index library and the intersection operation is performed to obtain the basic candidate bitmap. If the determined label set is empty, the basic candidate bitmap is initialized as the full set bitmap. Subsequently, the fuzzy label set is processed. A union operation is performed on the compressed bitmaps corresponding to all candidate label values ​​within the value range of each fuzzy label in the fuzzy label set to generate a fuzzy candidate bitmap. The basic candidate bitmap is then intersected with the fuzzy candidate bitmap to obtain the label filtering candidate bitmap. After segmenting the query text sequence and removing stop words, a list of query keywords is obtained. For each keyword in the query keyword list, the skip list corresponding to each keyword is pulled from the text inverted index of the hybrid index library. When there are multiple query keywords, a multi-way merge intersection operation is performed on multiple skip lists to obtain a set of exact matching file identifiers. Simultaneously, the information fingerprint of the query text is calculated, and file identifiers whose Hamming distance meets the preset threshold are retrieved and filtered through semantic hash index to form a fuzzy matching set. The exact matching set and the fuzzy matching set are then combined and their intersection is performed with the label filtering candidate bitmap to obtain the final candidate file set. The ratio of the number of tags that successfully match the determined tag set and the fuzzy tag set in the multidimensional tag vector of the target candidate file to the total number of tags in the query conditions is used as the tag matching score. The text relevance score is obtained by weighted summation based on the term frequency and inverse document frequency of the query keywords in the file. The tag matching degree and text relevance are weighted and summed according to preset weights to obtain a comprehensive score, which is then sorted in descending order. The sorted file list, along with the corresponding comprehensive score, scene type tag, date tag, and representative frame thumbnail, is assembled into a search results page and pushed to the user interface for display.

[0011] In this solution, S110 further includes: If the user does not find the desired material after browsing the search results and providing feedback through the interface, the information supplementation inquiry process is initiated. Expired or broad dimensions are extracted from the current query tag structure to generate supplementary questions targeting the shooting time period, equipment model, scene environment, and character characteristics. At the same time, the user is asked whether they authorize the use of the target user's voiceprint for matching and retrieval. After receiving user feedback, the user's supplementary information is converted into supplementary label conditions through term mapping and then merged with the original valid conditions. Bitmap pruning and text matching recall are then re-executed to generate a supplementary candidate file set. If the user authorizes voiceprint matching, the voiceprint features of the query speech are extracted from the multidimensional query features. The corresponding hash bucket is located using the voiceprint local sensitive hash index. The voiceprint embedding vector and associated file identifier stored in the bucket are extracted. The cosine similarity is calculated for each one, and the file identifiers with the highest similarity are selected as the voiceprint matching candidate set. The supplementary candidate file set and the voiceprint matching candidate set are combined, deduplicated, and the comprehensive score is recalculated and sorted to generate a material-assisted positioning list for secondary push.

[0012] Compared to existing technologies, this solution deeply integrates the material copying operation with the construction of a multi-dimensional classification index, breaking the inherent pattern of sequential execution of copying, classification, and retrieval in traditional workflows. It proposes a "copy-and-classify" technical concept, simultaneously launching a metadata parsing thread and a content analysis pipeline while the copy engine transmits video files from the source medium to the target storage. This ensures that by the time the file copy is completed, its corresponding multi-dimensional tags and hybrid index are already ready, allowing users to directly perform retrieval without any waiting time, significantly compressing the time window from shooting material to usable status.

[0013] At the index construction level, this solution organically integrates the previously isolated metadata index and content feature index, constructing a multi-dimensional tag system covering date tags, scene tags, shot tags, equipment source tags, scene type tags, pseudo-speaker tags, and content keyword tags. Based on this, a hybrid index library is built, expanding the material profile from a single file attribute description to a comprehensive multi-dimensional depiction including shooting equipment information, script supervisor information, scene type, dialogue content, and voiceprint identity. Regarding the retrieval mechanism, addressing the limitations of existing technologies in voice retrieval methods and their lack of support for fuzzy queries, this solution designs a multi-dimensional query parsing process based on natural language understanding. This process automatically identifies film industry-specific terminology from the user's voice description and performs terminology mapping and intent expansion on fuzzy descriptions, achieving an organic combination of precise search and fuzzy guidance. Furthermore, when users are dissatisfied with the initial search results, this solution introduces a deep retrieval mechanism involving supplementary information queries and voiceprint matching, further improving the recall and accuracy of material location. This provides a fully automated and flexible technical solution for film and television post-production material management, from copying to retrieval. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the drawings used in the embodiments or examples will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained according to these drawings without creative effort.

[0015] Figure 1 A flowchart of a method for copying and classifying film and television materials based on intelligent voice retrieval, provided in an embodiment of the present invention; Figure 2 A flowchart of a film and television material retrieval method based on a hybrid index library is provided in one embodiment of the present invention; Figure 3 A flowchart of a film and television material-assisted localization method based on depth interrogation is provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0016] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0018] Figure 1 A flowchart of a method for copying and classifying film and television materials based on intelligent voice retrieval, provided in an embodiment of the present invention; like Figure 1 As shown, this invention provides a flowchart of a method for copying and classifying film and television materials based on intelligent voice retrieval, including: S102, while the copy engine is transferring the video file from the source medium to the target storage, a parallel parsing thread is started to extract metadata from the video container header and auxiliary files in the same directory, and information is completed simultaneously through filename regular expression parsing to generate a structured meta dataset produced in parallel with the copy operation. S104: Perform content analysis on the copied file, segment the scene based on the inter-frame difference detection of the video track and extract representative frames, classify the scene using the extracted representative frames, and perform speech activity detection, streaming speech recognition and voiceprint feature extraction on the audio track to finally obtain the file content parsing features. S106, the structured meta-dataset and the file content parsing features are aggregated and a multi-dimensional label system is constructed. Based on the multi-dimensional label system, a compressed bitmap index, a text inverted index, a semantic hash index, and a voiceprint local sensitive hash index are constructed respectively to obtain a hybrid index library. S108: The user's voice query is received and converted into text by streaming speech recognition. The structured query conditions are extracted using the named entity recognition model, and the fuzzy description is mapped into terms and expanded with intent. At the same time, the voiceprint features of the query speech are extracted from the query speech, and finally integrated to generate multi-dimensional query features. S110, based on the multi-dimensional query features, a phased retrieval is performed in the hybrid index library to obtain a candidate file set. The candidate files are calculated with a weighted score of tag matching degree and text relevance, and then sorted and pushed. If the user's expected material is not found, a second retrieval is performed through information supplementation query to generate a material auxiliary positioning list for push.

[0019] Furthermore, in a preferred embodiment of the present invention, step S102 specifically includes: When the copy engine transfers video files from the source medium to the target storage, it intercepts the file header data block of each video file through a metadata parsing thread that runs in parallel with the data transport pipeline. The data is then distributed to the corresponding parser according to the container format to extract the timecode track, shooting date, encoding format, resolution and device identifier, and generate the container header metadata field. The container header metadata field is stored in an in-memory hash table with the file identifier as the key. The auxiliary files in the same directory of the source medium are scanned synchronously. The field record file is established with tabs to separate column index mapping and the volume number, scene number, shot number and start time code are extracted row by row. The project file is traversed in a streaming manner to extract metadata key-value pairs. The start timecode extracted from the scene log file is aligned with the timecode track extracted from the container header using a nearest neighbor search. When the difference is less than a preset threshold, the scene number and shot number of the corresponding scene log entry are associated with the memory hash table entry of the same file identifier. For each file name, a pre-defined set of regular expression templates is applied for pattern matching to extract the camera position identifier, session number, lens number, and number of shots. The file name parsing results are then merged with the existing metadata fields in the memory hash table using a weighted voting method. Existing values ​​are retained and cross-validated with the file name parsing results, while missing fields are directly added. After parsing the container header, aligning the field record files, and completing the filenames using regular expressions, a structured metadata dataset is generated based on the set of metadata key-value pairs corresponding to each file identifier in the memory hash table.

[0020] It's important to note that on a film production set, a single camera might generate hundreds of clips per day. The metadata for each clip is distributed across the file container header, the script log file, and the filename string, each independent and with a different format. Post-production staff typically need to manually review the script log, check the timecodes line by line, and then match the shooting parameters with the files—a time-consuming and error-prone process. To address this pain point, a metadata parsing thread was designed to run parallel to the data transfer pipeline, allowing metadata extraction and file copying to be completed simultaneously, eliminating the need for manual post-copying processing.

[0021] Specifically, the parsing thread first intercepts the header data block of each video file and assigns it to the corresponding parser for processing based on the container format. For MXF format, the parser traverses the partition packets according to the SMPTE 377M standard using KL encoding, extracting the timecode track from the source packet and extracting the shooting date, encoding format, resolution, camera model, and lens model from the system metadata packet. For MOV format, the parser locates the user data box under the moov atom, reconstructs the shooting timestamp and device identifier from it, and simultaneously traverses the track header to obtain the resolution and frame rate. The extracted fields are stored in an in-memory hash table using the file identifier as the key, so that each file has a preliminary metadata record before the copy is completed, providing a foundation for subsequent completion. At the same time, auxiliary files in the same directory as the source media are scanned. For ALE format scene log files, the parser reads the first line using tabs as delimiters to establish a column index mapping, and then extracts the volume number, scene number, shot number, and start timecode line by line to form a structured scene log entry object. For project files such as FCPXML, the parser traverses the element tree in a streaming manner, locates the resource elements, and extracts the metadata key-value pairs. These auxiliary files are often entered synchronously by the script supervisor on set, containing rich shooting organization information and serving as an important supplementary source of metadata for the container header.

[0022] Aligning the timecode in the scene log file with the timecode track in the container header is crucial. Since manual clapperboard recording may deviate from the equipment clock, this solution employs a nearest neighbor search strategy. It searches for the position with the smallest difference on the container timecode track based on the starting timecode of the scene log entry. When the difference is less than a preset threshold, it is determined to be the same segment, and the scene number and shot number are associated with the corresponding file identifier. This automatic alignment eliminates the tedious manual matching of each entry and prevents mismatches through threshold constraints. The filenames also contain structured information. A pre-defined regular expression template is then used for pattern matching. The parsed results are merged with existing fields in the hash table using a weighted voting method. Existing values ​​are retained and cross-validated using the filename parsing results; missing fields are directly added. Through the progressive integration of container header parsing, scene log file alignment, and filename regular expression completion, a structured metadata dataset that evolves with the copying progress is obtained during the copying process.

[0023] Furthermore, in a preferred embodiment of the present invention, step S104 specifically includes: The content analysis of the files already copied to the disk is performed, video frame extraction and decoding are performed with a preset sparse sampling rate, each frame is converted to HSV space and the normalized histogram of hue components is calculated, the Bartholomew's coefficient between the histograms of adjacent sampled frames is calculated as the inter-frame difference measure, and the mean and standard deviation of the inter-frame difference values ​​are calculated through a sliding window. A dynamic threshold is generated based on the mean and standard deviation of the inter-frame difference values. The current inter-frame difference value is compared with the dynamic threshold. When the threshold is exceeded, it is determined that a scene switch has been detected and the start and end positions of the scene are marked. For each scene segment, the frame corresponding to the midpoint of time is taken as a candidate representative frame, and the Laplacian variance of the candidate representative frame is calculated as a sharpness index. If the sharpness index of the candidate representative frame is less than the preset sharpness threshold, then the search continues frame by frame from the candidate representative frame to the beginning and end of the scene segment until a frame with a Laplace variance exceeding the preset sharpness threshold is found as the qualified representative frame of the current scene. Qualified representative frames are fed into a lightweight convolutional neural network scene classifier. The scene classifier is composed of stacked inverted residual structures. Each inverted residual block contains pointwise dimensionality-increasing convolution, depthwise separable convolution, compression activation module and pointwise dimensionality-reducing convolution. After global average pooling, the fully connected layer outputs the scene category labels. The audio stream is extracted from the video file. Mel frequency cepstral coefficients, first-order differences, and second-order differences are extracted with preset frame length and frame shift. After being concatenated into an acoustic feature vector sequence, it is fed into a speech activity detector based on a bidirectional long short-term memory network. After median filtering and double threshold hysteresis comparison, the speech segments are merged and speech fragments are removed to obtain a set of speech segments. The acoustic feature vector sequence of each speech segment is fed into the streaming speech recognition model. The encoder consists of a stack of multi-layer convolutional enhanced self-attention blocks. Each layer contains a feedforward module, a multi-head self-attention module, and a convolutional module. The decoder jointly connects temporal classification and attention mechanisms to output dialogue text with word-level timestamps. The acoustic feature vector sequence of each speech segment is fed into the voiceprint embedding model. The voiceprint embedding model uses an extended convolutional residual network as its backbone, stacks multiple extended convolutional layers, and uses attention statistical pooling to weight and aggregate frame-level features into voiceprint embedding vectors. Finally, it combines scene type labels and dialogue text to generate file content parsing features.

[0024] It should be noted that in film and television post-production, when faced with a large amount of footage, post-production staff typically need to play through each video clip to determine the scene type and key content points. This process is extremely time-consuming and relies heavily on personal experience. This invention addresses this efficiency bottleneck by automatically initiating a content analysis pipeline after the files are copied to disk. This allows the visual and audio features of the footage to be fully extracted without manual intervention, filling the functional gap of the traditional copying process that only copies without analyzing.

[0025] Specifically, the primary task of video track analysis is scene segmentation, which involves automatically dividing a complete video into several shot segments. Frame-by-frame decoding is performed using a preset sparse sampling rate, significantly reducing computational load while maintaining detection accuracy. Each frame is converted to the HSV color space, and a normalized histogram of hue components is calculated. The Baumkuchen coefficient between the histograms of adjacent frames is calculated as a measure of inter-frame difference. The mean and standard deviation of the inter-frame difference values ​​are calculated in real-time using a sliding window, dynamically generating a judgment threshold that adapts to the content. When the inter-frame difference exceeds this dynamic threshold, a scene change is determined, and the start and end positions are marked. This effectively balances the detection needs of action scenes with drastic changes in composition and scenes with gradual changes in lighting, avoiding insensitivity to editing rhythm. For each scene segment, the frame corresponding to the midpoint of time is taken as a candidate representative frame, and its Laplacian variance is calculated as a sharpness index. If the candidate representative frame is of poor quality due to motion blur or focus errors, a frame-by-frame search is performed from that frame towards both ends of the segment until a qualified representative frame with satisfactory sharpness is found.

[0026] The scene classifier, representing the frames fed in, is constructed using stacked inverse residual structures. Each inverse residual block sequentially contains pointwise increasing-dimensionality convolutions, depthwise separable convolutions, a compression activation module, and pointwise decreasing-dimensionality convolutions. Pointwise increasing-dimensionality convolutions map features to higher dimensions to enrich representation capabilities; depthwise separable convolutions significantly reduce computation while maintaining accuracy; the compression activation module adaptively enhances useful features by explicitly modeling inter-channel dependencies; and finally, after global average pooling, a fully connected layer outputs the scene category label. This allows post-production staff to quickly understand the scene attributes of each clip without manually dragging a progress bar.

[0027] In audio track analysis, after extracting the audio stream, Mel-frequency cepstral coefficients and their first and second-order differences are calculated using a preset frame length and frame shift. These are then concatenated into an acoustic feature vector sequence and fed into a speech activity detector. After median filtering and double-threshold hysteresis comparison, the speech segments are merged, and short noise fragments are removed to obtain a valid set of speech segments. The speech segments are then processed sequentially by a streaming speech recognition model and a voiceprint embedding model, outputting dialogue text with word-level timestamps and fixed-dimensional voiceprint embedding vectors, respectively. The streaming speech recognition model enables the system to output recognition results before the speech input has finished, achieving low-latency processing. The voiceprint embedding vectors generate pseudo-speaker labels through unsupervised clustering, providing a foundation for subsequent role-based material retrieval. The video and audio analysis results are finally merged with scene type labels to form file content parsing features, constituting a complete material profile from visual to auditory perspectives. This provides multi-dimensional feature support for users to perform accurate retrieval and fuzzy guidance through voice descriptions.

[0028] Furthermore, in a preferred embodiment of the present invention, S106 specifically includes: Using the file identifier as the primary key, the structured meta-dataset is associated and merged with the file content parsing features to construct a multi-dimensional label vector. The multi-dimensional label vector includes date label, scene label, lens label, device source label, and scene type label. For each discrete value in the multidimensional label vector, a compressed bitmap instance is created. The bitmap is divided into buckets according to the high bits of the file identifier. The low bits of the bucket are stored in an array or bitmap based on the data density. All files are traversed and the identifier is set into the corresponding label bitmap to complete the construction of the compressed bitmap index. The content keywords in the keyword tag list of each file are used as dictionary entries. The dictionary is organized in a B-tree structure. Each entry points to a skip list arranged in ascending order of file identifier. The skip list nodes record the frequency of the content words in the corresponding file, thus obtaining the text inverted index. For each file's complete dialogue text, a fixed-length information fingerprint is calculated. The hash values ​​of each word in the text are weighted and accumulated bit by bit according to the word weight, and then a binary code is generated by symbol determination. The code is divided into multiple short code segments. Each short code segment is used as a key to establish an inverted mapping and store the file identifier and the complete binary code to obtain a semantic hash index. A set of random projection vectors following a standard normal distribution is pre-generated and divided into several groups. The voiceprint embedding vector is calculated and the dot product operation is performed with each projection vector in the group to obtain a binary code. The binary code is used as the bucket identifier to store the file identifier and the voiceprint embedding vector into the corresponding bucket and to construct a voiceprint local sensitive hash index. The compressed bitmap index, text inverted index, semantic hash index, and voiceprint local sensitive hash index are integrated to generate a hybrid index library.

[0029] It should be noted that in film and television material management scenarios, single-dimensional indexes often fail to meet diverse retrieval needs. For example, relying solely on filenames cannot search by scene type, and relying solely on shooting dates cannot pinpoint specific dialogue content. This solution merges the structured meta-dataset produced in the preceding steps with the file content parsing features, using the file identifier as the primary key, to construct a multi-dimensional tag vector containing date tags, scene tags, shot tags, equipment source tags, and scene type tags. This ensures that each file simultaneously carries a complete attribute description from three sources: the file header, the script supervisor's note, and content analysis, providing a unified data foundation for multi-dimensional cross-retrieval. Specifically, four complementary index structures are constructed in parallel to address different retrieval modes. The compressed bitmap index handles efficient filtering of discrete tags. Within the bitmap, it is bucketed according to the high-order bits of the file identifier. Within each bucket, the low-order bits are adaptively stored using either an ordered short integer array or a bitmap, based on data density, significantly reducing memory usage while maintaining the speed of set operations. The inverted text index is used for keyword retrieval of dialogue content. The dictionary is organized in a B-tree structure to support prefix queries. Each entry points to a skip list arranged in ascending order by file identifier. Skip list nodes record word frequency information. For multi-keyword queries, intersection is quickly found through multi-way merge of the skip list. The semantic hash index is for fuzzy text matching. A fixed-length information fingerprint is calculated for the complete dialogue text of each file. The hash values ​​of each word are weighted and accumulated bit by bit according to word weight, and a binary code is generated after symbol determination. The code is divided into multiple short codes and an inverted index is established, so that only short codes need to be compared to quickly filter out semantically similar candidate files. The voiceprint local sensitive hash index is for voiceprint matching needs. A high-dimensional voiceprint embedding vector is mapped to a low-dimensional binary code through a pre-generated random projection vector. Voiceprint vectors within the same bucket have high consistency in cosine similarity. During the query, it is not necessary to traverse all files to locate voiceprint-similar materials in sublinear time. The four types of indexes work together to support the joint execution of multiple query modes such as precise tag filtering, keyword retrieval, fuzzy text search, and voiceprint similarity matching.

[0030] Furthermore, in a preferred embodiment of the present invention, step S108 specifically includes: Collect the voice search commands of the target user, perform acoustic preprocessing on the voice search commands, extract the Mel frequency cepstral coefficients, first-order difference and second-order difference from the voice search commands, concatenate them into an acoustic feature vector sequence and send it into a pre-constructed streaming speech recognition model for analysis; The encoder of the streaming speech recognition model is processed by multi-layer convolutional enhanced self-attention blocks, and the decoder is jointly connected with temporal classification and attention mechanisms to output the query text sequence. At the same time, the acoustic feature vector sequence is fed into the voiceprint embedding model in parallel, and aggregated into query speech voiceprint features through extended convolutional residual network and attention statistical pooling. The query text sequence is concatenated with character embeddings and pre-trained word embeddings from the film and television field and then fed into the named entity recognition model. The context is modeled in both forward and reverse time sequences through a bidirectional long short-term memory network. The conditional random field layer is combined with the label transition matrix to perform global optimal decoding and output the preliminary query conditions. Term mapping is performed on each entity in the initial query conditions. Based on the query condition set after term mapping, intent expansion is performed. Missing or broad dimensions are marked as wildcards. Scenario conditions that match the higher-level labels are expanded along the lower-level thesaurus. Undetermined role reference conditions are marked as not being filtered, forming a query label structure consisting of a set of definite labels and a set of fuzzy labels. The query tag structure, query text sequence, and temporarily stored query voiceprint features are integrated into a multidimensional query feature.

[0031] It's important to note that in post-production film and television retrieval scenarios, users often describe remembered footage clips using colloquial natural language, such as "Can you find the night scene shot in that alleyway last week with camera B?" This query method incorporates multi-dimensional information such as date, device, scene, and shooting number, but the expression differs from the standardized tags stored in the system. This solution addresses the semantic gap between colloquial queries and structured indexes by designing a complete parsing pipeline from speech signals to multi-dimensional query features.

[0032] Specifically, the Mel-frequency cepstral coefficients and their first and second differences are first extracted from the voice retrieval command. The first difference reflects the instantaneous change trend of acoustic features, and the second difference characterizes the acceleration of change. The concatenation of these three features allows the feature vector to simultaneously contain static spectral envelope and dynamic temporal information. After being concatenated into an acoustic feature vector sequence, it is fed into a pre-constructed streaming speech recognition model. This model adopts the same architecture as the source code, with the encoder's multi-layer convolutional enhancement self-attention blocks extracting high-level semantic representations. The decoder jointly connects temporal classification and attention mechanisms to output the probability distribution of word segments frame by frame, and generates the query text sequence through beam search decoding. Simultaneously with speech recognition, the acoustic feature vector sequence is fed into the speaker embedding model in parallel. The receptive field is expanded layer by layer through an extended convolutional residual network, and then the frame-level features are weighted and aggregated through attention statistical pooling. Compared with simple average pooling, attention statistical pooling can automatically learn which frames contribute more to speaker recognition, generating more discriminative query speech speaker features and temporarily storing them. This parallel processing architecture of "recognition plus voiceprint" allows for the simultaneous output of both text and voiceprint query features in a single acoustic preprocessing step, avoiding redundant calculations.

[0033] Then, named entity recognition is performed based on the query text sequence. Character embeddings and pre-trained word embeddings from the film and television domain are concatenated as model inputs. Character embeddings capture morphological features at the word formation level, while domain word embeddings provide semantic priors for film and television terms. A bidirectional long short-term memory network performs forward and backward temporal modeling of the context. A conditional random field layer combines the label transition matrix learned from the training data for globally optimal decoding, outputting entities such as scene number, shot number, date, equipment code, scene description words, and character references. After decoding, the terminology mapping stage converts colloquial expressions into standardized labels. For example, "B-machine" is converted into a prefix of the equipment source label through an alias mapping table, and "alley" is expanded into a candidate label set such as "outdoor scene" through a film and television domain terminology synonym graph. The intent expansion stage checks the completeness of each dimension, marking missing or overly broad dimensions as wildcards, refining scene conditions that match higher-level labels downwards along the synonym graph, and temporarily suspending filtering for unmatched character reference conditions. Finally, a query label structure consisting of two parts, a definite label set and a fuzzy label set, is formed. This two-layer structure balances the pruning capability of precise criteria with the recall coverage of fuzzy criteria, effectively solving the problem of the search scope being too narrow or too wide when users have unclear memories.

[0034] Figure 2 A flowchart of a film and television material retrieval method based on a hybrid index library is provided in one embodiment of the present invention; like Figure 2 As shown, the present invention provides a flowchart of a film and television material retrieval method based on a hybrid index library: Based on multidimensional query feature extraction, a label set, a fuzzy label set, and a query text sequence are determined. The compressed bitmap instance corresponding to each label value in the determined label set is retrieved from the compressed bitmap index of the hybrid index library and the intersection operation is performed to obtain the basic candidate bitmap. If the determined label set is empty, the basic candidate bitmap is initialized as the full set bitmap. Subsequently, the fuzzy label set is processed. A union operation is performed on the compressed bitmaps corresponding to all candidate label values ​​within the value range of each fuzzy label in the fuzzy label set to generate a fuzzy candidate bitmap. The basic candidate bitmap is then intersected with the fuzzy candidate bitmap to obtain the label filtering candidate bitmap. After segmenting the query text sequence and removing stop words, a list of query keywords is obtained. For each keyword in the query keyword list, the skip list corresponding to each keyword is pulled from the text inverted index of the hybrid index library. When there are multiple query keywords, a multi-way merge intersection operation is performed on multiple skip lists to obtain a set of exact matching file identifiers. Simultaneously, the information fingerprint of the query text is calculated, and file identifiers whose Hamming distance meets the preset threshold are retrieved and filtered through semantic hash index to form a fuzzy matching set. The exact matching set and the fuzzy matching set are then combined and their intersection is performed with the label filtering candidate bitmap to obtain the final candidate file set. The ratio of the number of tags that successfully match the determined tag set and the fuzzy tag set in the multidimensional tag vector of the target candidate file to the total number of tags in the query conditions is used as the tag matching score. The text relevance score is obtained by weighted summation based on the term frequency and inverse document frequency of the query keywords in the file. The tag matching degree and text relevance are weighted and summed according to preset weights to obtain a comprehensive score, which is then sorted in descending order. The sorted file list, along with the corresponding comprehensive score, scene type tag, date tag, and representative frame thumbnail, is assembled into a search results page and pushed to the user interface for display.

[0035] It's important to note that in film and television post-production, users' retrieval needs when facing a media library often present two typical forms: one is a clear target, where users can accurately state the scene number or specific date; the other is a vague memory, where users can only describe a general impression of the scene or fragmented dialogue. Traditional single-index structures struggle to simultaneously address both scenarios, either missing relevant results due to overly strict conditions or returning a large amount of irrelevant material due to overly broad matching. This solution, through a hierarchical design of defined tag sets and fuzzy tag sets in multi-dimensional query features, combined with the collaborative operation of multiple indexes in a hybrid index library, achieves a phased retrieval strategy from precise pruning to fuzzy recall.

[0036] Specifically, the first step in the retrieval process is to perform tag filtering using a compressed bitmap index. A defined tag set is extracted from the multidimensional query features. For each tag value, the corresponding compressed bitmap instance is retrieved from the compressed bitmap index, and an intersection operation is performed. Compared to traversing file records one by one, the bitmap intersection operation requires only a few instruction cycles on the CPU, instantly compressing the candidate size from the entire database to a smaller range. If the defined tag set is empty, it indicates that the user's query lacks explicit precise conditions. In this case, the system initializes the basic candidate bitmap to a full set bitmap, ensuring that no candidates are missed due to a lack of precise conditions. Subsequently, the fuzzy tag set is processed. A union operation is performed on the compressed bitmaps corresponding to candidate tag values ​​within its range to generate a fuzzy candidate bitmap. This fuzzy candidate bitmap is then intersected with the basic candidate bitmap to obtain the tag-filtering candidate bitmap. This "precise intersection first, fuzzy union supplement" strategy ensures that precise conditions dominate the filtering direction while effectively expanding the recall coverage using fuzzy conditions.

[0037] In the text matching phase, the query text sequence is segmented and stop words are removed to obtain a list of query keywords. Skip lists corresponding to each keyword are retrieved from the inverted text index. When multiple query keywords exist, a multi-way merge intersection strategy is employed, using the shortest skip list as the driving force to perform binary search in the remaining lists, resulting in a set of exact matching file identifiers containing all keywords, avoiding the high overhead of comparing the entire list one by one. Simultaneously, the information fingerprint of the query text is calculated and fuzzy recall is performed using a semantic hash index. The core advantage of the semantic hash index lies in transforming high-dimensional text similarity calculation into a Hamming distance comparison of fixed-length binary codes. File identifiers with a Hamming distance not exceeding a preset threshold are selected to form a fuzzy matching set, thus tolerating differences in colloquial expressions such as synonym substitution and word order adjustments. The union of the exact matching set and the fuzzy matching set is then intersected with the tag-filtered candidate bitmap to obtain the final candidate file set. At this point, the candidate file set has converged from the full database size to a range suitable for detailed calculation of each file.

[0038] During the scoring phase, two scores are calculated: tag matching score and text relevance score. Tag matching score is measured by the ratio of the number of successfully matched tags to the total number of tags in the query conditions. Text relevance is obtained by weighted summation of the term frequency and inverse document frequency (IVF) of the query keywords in the document. The IVF component assigns higher weight to keywords that are rare in the entire database, making the matching results more consistent with the user's actual search intent. The two scores are weighted and summed according to preset weights and then sorted in descending order, assembled into a search results page that includes a comprehensive score, scene type tags, date tags, and representative frame thumbnails. The display of representative frame thumbnails allows users to quickly determine the scene content without opening the video, further improving search efficiency.

[0039] Figure 3 A flowchart of a film and television material-assisted localization method based on depth interrogation is provided in an embodiment of the present invention; like Figure 3 As shown, the present invention provides a flowchart of a method for assisted localization of film and television materials based on depth interrogation, comprising: If the user does not find the desired material after browsing the search results and providing feedback through the interface, the information supplementation inquiry process is initiated. Expired or broad dimensions are extracted from the current query tag structure to generate supplementary questions targeting the shooting time period, equipment model, scene environment, and character characteristics. At the same time, the user is asked whether they authorize the use of the target user's voiceprint for matching and retrieval. After receiving user feedback, the user's supplementary information is converted into supplementary label conditions through term mapping and then merged with the original valid conditions. Bitmap pruning and text matching recall are then re-executed to generate a supplementary candidate file set. If the user authorizes voiceprint matching, the voiceprint features of the query speech are extracted from the multidimensional query features. The corresponding hash bucket is located using the voiceprint local sensitive hash index. The voiceprint embedding vector and associated file identifier stored in the bucket are extracted. The cosine similarity is calculated for each one, and the file identifiers with the highest similarity are selected as the voiceprint matching candidate set. The supplementary candidate file set and the voiceprint matching candidate set are combined, deduplicated, and the comprehensive score is recalculated and sorted to generate a material-assisted positioning list for secondary push.

[0040] It's important to note that in real-world film and television material retrieval scenarios, a user's initial voice query may be based on fragmented memory or limited information, and the initial search results may not accurately match the target. Traditional retrieval systems typically only require users to re-enter their search criteria when they don't find the desired material, lacking the ability to reuse existing search information and provide targeted supplementary guidance. This solution addresses this issue by automatically initiating a supplementary information query process when the initial search fails to meet expectations, transforming a passive, one-time query into an interactive, progressive retrieval process.

[0041] Specifically, the system first automatically analyzes the reasons for the initial search's failure to find a match from the current query tag structure, identifying which dimensions are invalid or too broad. For example, if a user initially describes "looking for a night scene shot," the first round might return hundreds of results, indicating that the scene condition is too broad and needs further constraint. Based on this analysis, targeted supplementary questions are generated to guide the user to recall information dimensions that can further narrow down the search, such as the shooting time period, equipment model, scene environment, and character characteristics. Simultaneously, the system asks the user if they authorize the use of their own voiceprint for matching. This design considers a common scenario in film production: cinematographers, directors, or script supervisors often appear in behind-the-scenes footage or are captured on camera, and their voiceprint features may match the speaker on a certain audio track in the material library. Similarly, the person querying the material might have participated in the recording themselves, such as recording the lead actor. Since the voiceprint has already been embedded into a vector and temporarily stored in the multi-dimensional query feature object during the initial processing of the query audio, it can be directly retrieved without reprocessing the audio, avoiding redundant computation overhead.

[0042] After receiving user feedback, the user-added text information is converted into supplementary label conditions through term mapping, merged with the valid conditions in the original query label structure, and the bitmap pruning and text matching recall process is re-executed to generate a supplementary candidate file set. This path uses the newly added label constraints to further narrow the scope of the first round of candidates, which is a secondary precise filtering based on semantic labels. If the user also authorizes voiceprint matching, the temporarily stored query speech voiceprint features are extracted from the multi-dimensional query features, and the corresponding hash bucket is located through the voiceprint local sensitive hash index. Since the voiceprint local sensitive hash index has already grouped segments with similar acoustic features into the same bucket during construction, it is not necessary to traverse the entire database of voiceprint vectors during retrieval. It is only necessary to extract the voiceprint embedding vectors and associated file identifiers stored in the bucket, calculate the cosine similarity of each, and then select the several file identifiers with the highest similarity as the voiceprint matching candidate set. Finally, the supplementary candidate file set and the voiceprint matching candidate set are combined and deduplicated, the comprehensive score is recalculated and sorted, and a material auxiliary positioning list is generated for secondary push. This dual-path complementary strategy of "tag-supplemented retrieval plus voiceprint parallel matching" enables users to gradually approach the target material through multiple rounds of interaction, even if their memory is fuzzy, thus significantly improving retrieval robustness and response speed.

[0043] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for copying and classifying film and television materials based on intelligent voice retrieval, characterized in that, include: S102, while the copy engine is transferring the video file from the source medium to the target storage, a parallel parsing thread is started to extract metadata from the video container header and auxiliary files in the same directory, and information is completed simultaneously through filename regular expression parsing to generate a structured meta dataset produced in parallel with the copy operation. S104: Perform content analysis on the copied file, segment the scene based on the inter-frame difference detection of the video track and extract representative frames, classify the scene using the extracted representative frames, and perform speech activity detection, streaming speech recognition and voiceprint feature extraction on the audio track to finally obtain the file content parsing features. S106, the structured meta-dataset and the file content parsing features are aggregated and a multi-dimensional label system is constructed. Based on the multi-dimensional label system, a compressed bitmap index, a text inverted index, a semantic hash index, and a voiceprint local sensitive hash index are constructed respectively to obtain a hybrid index library. S108: The user's voice query is received and converted into text by streaming speech recognition. The structured query conditions are extracted using the named entity recognition model, and the fuzzy description is mapped into terms and expanded with intent. At the same time, the voiceprint features of the query speech are extracted from the query speech, and finally integrated to generate multi-dimensional query features. S110, based on the multi-dimensional query features, a phased retrieval is performed in the hybrid index library to obtain a candidate file set. The candidate files are calculated with a weighted score of tag matching degree and text relevance, and then sorted and pushed. If the user's expected material is not found, a second retrieval is performed through information supplementation query to generate a material auxiliary positioning list for push.

2. The method for copying and classifying film and television materials based on intelligent voice retrieval according to claim 1, characterized in that, S102 specifically includes: When the copy engine transfers video files from the source medium to the target storage, it intercepts the file header data block of each video file through a metadata parsing thread that runs in parallel with the data transport pipeline. The data is then distributed to the corresponding parser according to the container format to extract the timecode track, shooting date, encoding format, resolution and device identifier, and generate the container header metadata field. The container header metadata field is stored in an in-memory hash table with the file identifier as the key. The auxiliary files in the same directory of the source medium are scanned synchronously. The field record file is established with tabs to separate column index mapping and the volume number, scene number, shot number and start time code are extracted row by row. The project file is traversed in a streaming manner to extract metadata key-value pairs. The start timecode extracted from the scene log file is aligned with the timecode track extracted from the container header using a nearest neighbor search. When the difference is less than a preset threshold, the scene number and shot number of the corresponding scene log entry are associated with the memory hash table entry of the same file identifier. For each file name, a pre-defined set of regular expression templates is applied for pattern matching to extract the camera position identifier, session number, lens number, and number of shots. The file name parsing results are then merged with the existing metadata fields in the memory hash table using a weighted voting method. Existing values ​​are retained and cross-validated with the file name parsing results, while missing fields are directly added. After parsing the container header, aligning the field record files, and completing the filenames using regular expressions, a structured metadata dataset is generated based on the set of metadata key-value pairs corresponding to each file identifier in the memory hash table.

3. The method for copying and classifying film and television materials based on intelligent voice retrieval according to claim 1, characterized in that, S104 specifically includes: The content analysis of the files already copied to the disk is performed, video frame extraction and decoding are performed with a preset sparse sampling rate, each frame is converted to HSV space and the normalized histogram of hue components is calculated, the Bartholomew's coefficient between the histograms of adjacent sampled frames is calculated as the inter-frame difference measure, and the mean and standard deviation of the inter-frame difference values ​​are calculated through a sliding window. A dynamic threshold is generated based on the mean and standard deviation of the inter-frame difference values. The current inter-frame difference value is compared with the dynamic threshold. When the threshold is exceeded, it is determined that a scene switch has been detected and the start and end positions of the scene are marked. For each scene segment, the frame corresponding to the midpoint of time is taken as a candidate representative frame, and the Laplacian variance of the candidate representative frame is calculated as a sharpness index. If the sharpness index of the candidate representative frame is less than the preset sharpness threshold, then the search continues frame by frame from the candidate representative frame to the beginning and end of the scene segment until a frame with a Laplace variance exceeding the preset sharpness threshold is found as the qualified representative frame of the current scene. Qualified representative frames are fed into a lightweight convolutional neural network scene classifier. The scene classifier is composed of stacked inverted residual structures. Each inverted residual block contains pointwise dimensionality-increasing convolution, depthwise separable convolution, compression activation module and pointwise dimensionality-reducing convolution. After global average pooling, the fully connected layer outputs the scene category labels. The audio stream is extracted from the video file. Mel frequency cepstral coefficients, first-order differences, and second-order differences are extracted with preset frame length and frame shift. After being concatenated into an acoustic feature vector sequence, it is fed into a speech activity detector based on a bidirectional long short-term memory network. After median filtering and double threshold hysteresis comparison, the speech segments are merged and speech fragments are removed to obtain a set of speech segments. The acoustic feature vector sequence of each speech segment is fed into the streaming speech recognition model. The encoder consists of a stack of multi-layer convolutional enhanced self-attention blocks. Each layer contains a feedforward module, a multi-head self-attention module, and a convolutional module. The decoder jointly connects temporal classification and attention mechanisms to output dialogue text with word-level timestamps. The acoustic feature vector sequence of each speech segment is fed into the voiceprint embedding model. The voiceprint embedding model uses an extended convolutional residual network as its backbone, stacks multiple extended convolutional layers, and uses attention statistical pooling to weight and aggregate frame-level features into voiceprint embedding vectors. Finally, it combines scene type labels and dialogue text to generate file content parsing features.

4. The method for copying and classifying film and television materials based on intelligent voice retrieval according to claim 1, characterized in that, S106 specifically includes: Using the file identifier as the primary key, the structured meta-dataset is associated and merged with the file content parsing features to construct a multi-dimensional label vector. The multi-dimensional label vector includes date label, scene label, lens label, device source label, and scene type label. For each discrete value in the multidimensional label vector, a compressed bitmap instance is created. The bitmap is divided into buckets according to the high bits of the file identifier. The low bits of the bucket are stored in an array or bitmap based on the data density. All files are traversed and the identifier is set into the corresponding label bitmap to complete the construction of the compressed bitmap index. The content keywords in the keyword tag list of each file are used as dictionary entries. The dictionary is organized in a B-tree structure. Each entry points to a skip list arranged in ascending order of file identifier. The skip list nodes record the frequency of the content words in the corresponding file, thus obtaining the text inverted index. For each file's complete dialogue text, a fixed-length information fingerprint is calculated. The hash values ​​of each word in the text are weighted and accumulated bit by bit according to the word weight, and then a binary code is generated by symbol determination. The code is divided into multiple short code segments. Each short code segment is used as a key to establish an inverted mapping and store the file identifier and the complete binary code to obtain a semantic hash index. A set of random projection vectors following a standard normal distribution is pre-generated and divided into several groups. The voiceprint embedding vector is calculated and the dot product operation is performed with each projection vector in the group to obtain a binary code. The binary code is used as the bucket identifier to store the file identifier and the voiceprint embedding vector into the corresponding bucket and to construct a voiceprint local sensitive hash index. The compressed bitmap index, text inverted index, semantic hash index, and voiceprint local sensitive hash index are integrated to generate a hybrid index library.

5. The method for copying and classifying film and television materials based on intelligent voice retrieval according to claim 1, characterized in that, S108 specifically includes: Collect the voice search commands of the target user, perform acoustic preprocessing on the voice search commands, extract the Mel frequency cepstral coefficients, first-order difference and second-order difference from the voice search commands, concatenate them into an acoustic feature vector sequence and send it into a pre-constructed streaming speech recognition model for analysis; The encoder of the streaming speech recognition model is processed by multi-layer convolutional enhanced self-attention blocks, and the decoder is jointly connected with temporal classification and attention mechanisms to output the query text sequence. At the same time, the acoustic feature vector sequence is fed into the voiceprint embedding model in parallel, and aggregated into query speech voiceprint features through extended convolutional residual network and attention statistical pooling. The query text sequence is concatenated with character embeddings and pre-trained word embeddings from the film and television field and then fed into the named entity recognition model. The context is modeled in both forward and reverse time sequences through a bidirectional long short-term memory network. The conditional random field layer is combined with the label transition matrix to perform global optimal decoding and output the preliminary query conditions. Term mapping is performed on each entity in the initial query conditions. Based on the query condition set after term mapping, intent expansion is performed. Missing or broad dimensions are marked as wildcards. Scenario conditions that match the higher-level labels are expanded along the lower-level thesaurus. Undetermined role reference conditions are marked as not being filtered, forming a query label structure consisting of a set of definite labels and a set of fuzzy labels. The query tag structure, query text sequence, and temporarily stored query voiceprint features are integrated into a multidimensional query feature.

6. The method for copying and classifying film and television materials based on intelligent voice retrieval according to claim 1, characterized in that, S110 specifically includes: Based on multidimensional query feature extraction, a label set, a fuzzy label set, and a query text sequence are determined. The compressed bitmap instance corresponding to each label value in the determined label set is retrieved from the compressed bitmap index of the hybrid index library and the intersection operation is performed to obtain the basic candidate bitmap. If the determined label set is empty, the basic candidate bitmap is initialized as the full set bitmap. Subsequently, the fuzzy label set is processed. A union operation is performed on the compressed bitmaps corresponding to all candidate label values ​​within the value range of each fuzzy label in the fuzzy label set to generate a fuzzy candidate bitmap. The basic candidate bitmap is then intersected with the fuzzy candidate bitmap to obtain the label filtering candidate bitmap. After segmenting the query text sequence and removing stop words, a list of query keywords is obtained. For each keyword in the query keyword list, the skip list corresponding to each keyword is pulled from the text inverted index of the hybrid index library. When there are multiple query keywords, a multi-way merge intersection operation is performed on multiple skip lists to obtain a set of exact matching file identifiers. Simultaneously, the information fingerprint of the query text is calculated, and file identifiers whose Hamming distance meets the preset threshold are retrieved and filtered through semantic hash index to form a fuzzy matching set. The exact matching set and the fuzzy matching set are then combined and their intersection is performed with the label filtering candidate bitmap to obtain the final candidate file set. The ratio of the number of tags that successfully match the determined tag set and the fuzzy tag set in the multidimensional tag vector of the target candidate file to the total number of tags in the query conditions is used as the tag matching score. The text relevance score is obtained by weighted summation based on the term frequency and inverse document frequency of the query keywords in the file. The tag matching degree and text relevance are weighted and summed according to preset weights to obtain a comprehensive score, which is then sorted in descending order. The sorted file list, along with the corresponding comprehensive score, scene type tag, date tag, and representative frame thumbnail, is assembled into a search results page and pushed to the user interface for display.

7. The method for copying and classifying film and television materials based on intelligent voice retrieval according to claim 1, characterized in that, The S110 further includes: If the user does not find the desired material after browsing the search results and providing feedback through the interface, the information supplementation inquiry process is initiated. Expired or broad dimensions are extracted from the current query tag structure to generate supplementary questions targeting the shooting time period, equipment model, scene environment, and character characteristics. At the same time, the user is asked whether they authorize the use of the target user's voiceprint for matching and retrieval. After receiving user feedback, the user's supplementary information is converted into supplementary label conditions through term mapping and then merged with the original valid conditions. Bitmap pruning and text matching recall are then re-executed to generate a supplementary candidate file set. If the user authorizes voiceprint matching, the voiceprint features of the query speech are extracted from the multidimensional query features. The corresponding hash bucket is located using the voiceprint local sensitive hash index. The voiceprint embedding vector and associated file identifier stored in the bucket are extracted. The cosine similarity is calculated for each one, and the file identifiers with the highest similarity are selected as the voiceprint matching candidate set. The supplementary candidate file set and the voiceprint matching candidate set are combined, deduplicated, and the comprehensive score is recalculated and sorted to generate a material-assisted positioning list for secondary push.

Images