Material recovery system based on mask screening

Abstract

The application provides a material recovery system based on mask screening, which comprises: an original material preprocessing unit for preprocessing original multimedia material to generate standardized preprocessing material; a material feature extraction unit for performing feature extraction processing on the standardized preprocessing material to generate a material feature vector; a material index construction unit for performing index construction processing on the material feature vector to generate a material index library; a recovery condition analysis unit for performing semantic analysis processing on user input recovery conditions to generate a structured recovery instruction; a material preliminary recovery unit for performing retrieval processing on the material index library according to the structured recovery instruction to generate a preliminary recovery material set; a Mask generation unit for performing key information extraction processing on the preliminary recovery material set to generate a Mask template; a similarity calculation unit for each material in the preliminary recovery material set to generate a material similarity score list; and a Mask screening unit for performing screening processing according to a set similarity threshold to generate a final recovery material set.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a material retrieval system based on mask screening. Background Technology

[0002] In today's rapidly developing digital media landscape, content management and retrieval have become critical issues for businesses and creators. Whether it's accumulating content for advertising and marketing, integrating resources on media content platforms, or utilizing content in the creative design industry, all require efficiently extracting the necessary content from massive amounts of raw multimedia materials.

[0003] Currently, common methods for retrieving content often employ simple keyword searches or similarity matching based on global features. The former only searches based on user-inputted text keywords, failing to understand semantics and resulting in inaccurate search results. For example, entering "sports brand posters" might yield numerous irrelevant sports scene images. The latter calculates similarity based on the overall features of the content, without focusing on the key content. When the content contains a lot of redundant background information, it is prone to misjudgment, causing the retrieved content to deviate from the actual needs and failing to meet the user's requirement for matching the core content of the content.

[0004] Furthermore, existing technologies lack a systematic integration of the material processing workflow, with each stage operating independently. This leads to inefficiencies and information loss in the process from raw materials to final retrieved results. For example, the results of the material preprocessing stage cannot be effectively applied to subsequent searches, resulting in stagnant search efficiency and difficulty in guaranteeing the quality of retrieved results, making it unsuitable for diverse and complex material retrieval needs. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a material retrieval system based on mask screening to at least partially solve the above-mentioned problems.

[0006] A material retrieval system based on mask filtering, comprising:

[0007] The raw material preprocessing unit is used to preprocess the raw multimedia materials to generate standardized preprocessed materials;

[0008] The material feature extraction unit is used to extract features from standardized preprocessed materials to generate material feature vectors;

[0009] The material indexing construction unit is used to index and construct material feature vectors to generate a material index library;

[0010] The retrieval condition parsing unit is used to perform semantic parsing processing on the retrieval conditions input by the user in order to generate a structured retrieval instruction;

[0011] The material preliminary retrieval unit is used to perform retrieval processing in the material index library according to the structured retrieval instructions to generate a preliminary retrieval material set;

[0012] The Mask generation unit is used to extract key information from the initial retrieved material set in order to generate a Mask template.

[0013] The similarity calculation unit is used to perform similarity calculation on each material in the initial retrieved material set, combined with the corresponding Mask template, to generate a list of material similarity scores;

[0014] The Mask filtering unit is used to filter the list of material similarity scores according to a set similarity threshold in order to generate the final retrieved material set.

[0015] Optionally, the original material preprocessing unit is used to preprocess the original multimedia materials to generate standardized preprocessed materials, specifically including the following steps:

[0016] The format standardization subunit is used to perform uniform format conversion on the original multimedia materials to generate format-standardized materials;

[0017] The resolution unification subunit is used to normalize the resolution of images / videos in format-standardized materials to generate size-standardized materials.

[0018] The content denoising subunit is used to filter noise from size-standardized material to generate denoised material.

[0019] The metadata extraction subunit is used to perform metadata parsing and processing on the denoised material to generate material with metadata.

[0020] The data cleaning subunit is used to perform quality assessment and anomaly handling on metadata-bearing materials in order to generate standardized pre-processed materials.

[0021] Optionally, the material feature extraction unit is used to perform feature extraction processing on the standardized preprocessed materials to generate material feature vectors, specifically including the following steps:

[0022] The modality separation subunit is used to perform modality classification on standardized preprocessed materials to generate modality-separated materials;

[0023] The visual feature extraction subunit is used to perform multi-scale feature extraction processing on image / video frames in modality separation material to generate visual feature tensors;

[0024] The text feature extraction subunit is used to perform semantic encoding on text paragraphs in the modality-separated material to generate text feature vectors;

[0025] The audio feature extraction subunit is used to perform acoustic feature extraction processing on the audio tracks in the video frame sequence to generate an audio feature matrix;

[0026] The multimodal fusion subunit is used to perform feature fusion processing on visual feature tensors, text feature vectors, and audio feature matrices to generate multimodal feature vectors.

[0027] The dimensionality reduction and compression subunit is used to perform dimensionality compression processing on multimodal feature vectors to generate material feature vectors.

[0028] Optionally, the material indexing construction unit is used to perform indexing and construction processing on the material feature vectors to generate a material index library, specifically including the following steps:

[0029] The vector quantization subunit is used to perform product quantization on the feature vectors of the source material to generate quantized feature vectors;

[0030] The hash coding subunit is used to perform locality-sensitive hashing on the quantized feature vector to generate a hash-coded vector;

[0031] The inverted index building subunit is used to perform inverted table construction on the hash encoded vector to generate a hash inverted index;

[0032] Hierarchical clustering subunits are used to perform hierarchical clustering on quantized feature vectors to generate vector clustering trees;

[0033] The hybrid index fusion subunit is used to perform index fusion processing on the hash inverted index and the vector clustering tree to generate a hybrid index structure;

[0034] The incremental update subunit is used to perform real-time incremental update processing on the hybrid index structure to generate a material index library.

[0035] Optionally, the retrieval condition parsing unit is used to perform semantic parsing processing on the retrieval conditions input by the user to generate a structured retrieval instruction, specifically including the following steps:

[0036] The natural language cleaning subunit is used to perform text normalization processing on the user-input retrieval conditions in order to generate cleaned retrieval condition text.

[0037] The word segmentation and part-of-speech tagging subunit is used to perform word segmentation and part-of-speech tagging on the cleaned retrieved conditional text to generate a tagged word sequence;

[0038] The named entity recognition subunit is used to perform named entity recognition processing on the labeled word sequence to generate entity recognition results;

[0039] The keyword extraction subunit is used to extract keywords from the labeled word sequence and entity recognition results to generate a keyword set;

[0040] The semantic intent parsing subunit is used to perform semantic intent analysis on the keyword set to generate a semantic intent representation;

[0041] The structured transformation subunit is used to perform structured transformation processing on the semantic intent representation to generate structured retrieval instructions.

[0042] Optionally, the preliminary material retrieval unit is used to perform retrieval processing in the material index library according to the structured retrieval instructions to generate a preliminary retrieved material set, specifically including the following steps:

[0043] The instruction parsing and adaptation subunit is used to perform index adaptation and parsing processing on structured retrieval instructions to generate index query parameters;

[0044] The index quickly locates sub-units, which is used to locate the index query parameters in the hash inverted index of the material index library to generate a list of candidate material IDs;

[0045] The clustering range filtering subunit is used to perform range filtering on the candidate material ID list combined with the vector clustering tree of the material index library to generate a refined candidate material ID list;

[0046] The metadata filtering subunit is used to perform conditional filtering on the metadata associated with the refined candidate material ID list in the material index library in order to generate a preliminary filtered material ID list.

[0047] The ID mapping retrieval subunit is used to perform ID-material mapping retrieval processing on the initially filtered material ID list in the material index library to generate an initial retrieved material set.

[0048] Optionally, the Mask generation unit is used to extract key information from the initial retrieved material set to generate a Mask template, specifically including the following steps:

[0049] The material type classification subunit is used to perform modal classification processing on the initial retrieved material set to generate a classified material set;

[0050] The visual saliency detection subunit is used to perform saliency region detection processing on image / video frames in the classification material set to generate a saliency mask;

[0051] The target detection and segmentation subunit is used to perform target detection and segmentation processing on the region covered by the saliency mask in order to generate a target segmentation mask;

[0052] The text key region extraction sub-unit is used to perform keyword location processing on the text in the categorized material set to generate a text mask;

[0053] The multimodal fusion subunit is used to perform multimodal fusion processing on the target segmentation mask and the text mask to generate a preliminary mask template;

[0054] The Mask optimization subunit is used to perform morphological processing and edge optimization on the initial Mask template to generate the final Mask template.

[0055] Optionally, the similarity calculation unit is used to perform similarity calculation on each piece of material in the initial retrieved material set, combined with the corresponding Mask template, to generate a list of material similarity scores. Specifically, it includes the following steps:

[0056] The Mask application subunit is used to apply the corresponding Mask template to each material in the initial retrieved material set to perform region cropping processing in order to generate a Masked material set;

[0057] The feature re-extraction subunit is used to perform feature re-extraction processing on the Masked material set to generate a Masked feature vector set;

[0058] The query vector generation subunit is used to perform feature transformation processing on the user's query conditions to generate a query feature vector;

[0059] The multimodal similarity calculation subunit is used to perform multimodal similarity calculation on the masked feature vector set and the query feature vector to generate the original similarity matrix;

[0060] The weighted fusion subunit is used to perform weighted fusion processing on the original similarity matrix to generate a comprehensive similarity score;

[0061] The sorting index generation sub-unit is used to sort the comprehensive similarity scores in descending order to generate a list of material similarity scores.

[0062] Optionally, the Mask filtering unit is used to filter the list of material similarity scores according to a set similarity threshold to generate the final retrieved material set, specifically including the following steps:

[0063] The threshold parsing subunit is used to perform parameter parsing processing on the set similarity threshold to generate standardized threshold parameters;

[0064] The score comparison sub-unit is used to compare each score in the material similarity score list with the standardized threshold parameter to generate a preliminary screening tag list;

[0065] The conflict resolution subunit is used to resolve conflicts in special cases (such as the same score but different labeling results) in the initial screening label list, so as to generate a conflict-free screening label list.

[0066] The material extraction subunit is used to extract materials marked as "retained" from the conflict-free filter list in the initial retrieved material set to generate the final retrieved material set. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0068] Figure 1 This is a schematic diagram of a material retrieval system based on mask screening, according to an embodiment of this application.

[0069] Figure 2 This invention provides a schematic diagram of the structure of an electronic device. Detailed Implementation

[0070] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0071] It should be understood that the terms "first," "second," and "third," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof.

[0072] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any and all combinations of one or more of the associated listed items, and includes such combinations.

[0073] Figure 1 This is a schematic diagram of a material retrieval system based on mask screening, according to an embodiment of this application. Figure 1 As shown, a material retrieval system based on mask filtering includes:

[0074] The raw material preprocessing unit is used to preprocess the raw multimedia materials to generate standardized preprocessed materials;

[0075] The material feature extraction unit is used to extract features from standardized preprocessed materials to generate material feature vectors;

[0076] The material indexing construction unit is used to index and construct material feature vectors to generate a material index library;

[0077] The retrieval condition parsing unit is used to perform semantic parsing processing on the retrieval conditions input by the user in order to generate a structured retrieval instruction;

[0078] The material preliminary retrieval unit is used to perform retrieval processing in the material index library according to the structured retrieval instructions to generate a preliminary retrieval material set;

[0079] The Mask generation unit is used to extract key information from the initial retrieved material set in order to generate a Mask template.

[0080] The similarity calculation unit is used to perform similarity calculation on each material in the initial retrieved material set, combined with the corresponding Mask template, to generate a list of material similarity scores;

[0081] The Mask filtering unit is used to filter the list of material similarity scores according to a set similarity threshold in order to generate the final retrieved material set.

[0082] Therefore, the technical advantages of the embodiments of this application are as follows:

[0083] This system uses a retrieval condition parsing unit to semantically analyze the retrieval conditions input by the user and generate structured retrieval instructions, which can accurately understand the user's intent. At the same time, the mask generation unit extracts key information from the material to generate a mask template. Combined with the similarity calculation unit, similarity is calculated based on the mask template, which can avoid interference from redundant background information, accurately match the core content of the material, and effectively solve the problem of retrieval result deviation.

[0084] This system generates standardized preprocessed materials through the original material preprocessing unit, providing a unified and standardized data foundation for subsequent feature extraction; the material feature vectors generated by the material feature extraction unit are used by the material index building unit to establish a material index library, providing support for rapid retrieval; the units are closely related, with the output of the previous unit serving as the input of the next unit, forming a complete closed loop, effectively avoiding information loss, and significantly improving the processing efficiency and quality of the retrieval results from the original materials to the final retrieved results, adapting to diverse and complex material retrieval needs.

[0085] Optionally, the original material preprocessing unit is used to preprocess the original multimedia materials to generate standardized preprocessed materials, specifically including the following steps:

[0086] The format standardization subunit is used to perform uniform format conversion on the original multimedia materials to generate format-standardized materials;

[0087] The resolution unification subunit is used to normalize the resolution of images / videos in format-standardized materials to generate size-standardized materials.

[0088] The content denoising subunit is used to filter noise from size-standardized material to generate denoised material.

[0089] The metadata extraction subunit is used to perform metadata parsing and processing on the denoised material to generate material with metadata.

[0090] The data cleaning subunit is used to perform quality assessment and anomaly handling on metadata-bearing materials in order to generate standardized pre-processed materials.

[0091] Preferably, in a specific application scenario, the specific technical implementation of each of the above sub-units in this application is as follows.

[0092] In this application, a deep learning-based cross-modal format conversion algorithm is used for the format standardization subunit to construct an end-to-end format conversion model. This model is trained using a large amount of paired data of different formats to learn the underlying mapping relationships between them. For example, in image format conversion, a generative adversarial network (GAN) architecture is used to train a generator to convert JPG, WEBP, and other formats to standard PNG or JPEG formats, while a discriminator is trained to ensure that the converted format conforms to the target format specification. For video format conversion, an extended framework based on FFmpeg, combined with a deep video encoding / decoding model, achieves efficient conversion from AVI, MOV, and other formats to H.264 encoded MP4 format, which better preserves the original details of the source material compared to traditional format conversion tools.

[0093] In this application, an adaptive super-resolution and scaling fusion algorithm is introduced for the resolution unification subunit. For images, a super-resolution model based on convolutional neural networks (CNN), such as EDSR (Enhanced Deep Super-Resolution Network), is first used to enhance low-resolution images and restore lost details. Then, an intelligent scaling algorithm is combined, using a weighted interpolation strategy to adjust the size based on the target resolution and the original image ratio, avoiding the jagged edges and blurring caused by traditional interpolation methods. For video resolution unification, a frame processing algorithm based on spatiotemporal consistency is proposed. A 3D CNN model is used to analyze the spatiotemporal relationship between video frames, maintaining the continuity between frames when adjusting the resolution, effectively preventing video stuttering or screen tearing.

[0094] In this application, a multimodal denoising algorithm based on the Transformer architecture is employed for the content denoising subunit. For images, the Transformer is applied to the image denoising task, capturing long-distance dependencies between different regions of the image through a multi-head attention mechanism to accurately identify and remove noise; simultaneously, a self-supervised learning approach is combined to further improve the denoising effect by utilizing the image's own redundant information. For video denoising, a spatiotemporal Transformer network is used to denoise not only single frames but also to eliminate flicker and dynamic noise using inter-frame information. For text denoising, a pre-trained language model (such as BERT) combined with a rule engine is used. BERT is used to understand text semantics and identify abnormal words and garbled characters, while the rule engine cleans up special characters and invalid formats.

[0095] In this application, a knowledge graph-guided metadata mining algorithm is used for the metadata extraction subunit. First, a knowledge graph related to multimedia materials is constructed, integrating metadata standards and relationships corresponding to various material formats. During the extraction process, based on the material format type, the corresponding metadata extraction rules and paths are obtained from the knowledge graph. For example, for images, based on the node relationships of the EXIF ​​standard in the knowledge graph, metadata such as shooting time and device is extracted using a combination of regular expressions and semantic parsing; for videos, by parsing the video container format (such as the Box structure of MP4) and combining it with the video metadata definitions in the knowledge graph, information such as frame rate and bitrate is accurately extracted. Compared with traditional extraction methods, this approach can obtain metadata more comprehensively and accurately.

[0096] In this application, a reinforcement learning-based material quality assessment and cleaning strategy is designed for the data cleaning subunit. A reinforcement learning environment is constructed, with metadata-bearing materials as state input, cleaning operations as actions, and the final material quality improvement index as reward. Through continuous training, the agent learns to automatically select the optimal cleaning strategy based on the material's metadata information (such as image resolution, video duration, text length, etc.) and content features (such as image clarity, video integrity, text semantic coherence, etc.). For example, for low-quality images, the agent can choose to enhance or directly remove them based on the evaluation results; for duplicate text, a SimHash-based similarity measurement algorithm combined with reinforcement learning decision-making is used to achieve efficient deduplication, thereby generating high-quality standardized pre-processed materials.

[0097] Optionally, the material feature extraction unit is used to perform feature extraction processing on the standardized preprocessed materials to generate material feature vectors, specifically including the following steps:

[0098] The modality separation subunit is used to perform modality classification on standardized preprocessed materials to generate modality-separated materials;

[0099] The visual feature extraction subunit is used to perform multi-scale feature extraction processing on image / video frames in modality separation material to generate visual feature tensors;

[0100] The text feature extraction subunit is used to perform semantic encoding on text paragraphs in the modality-separated material to generate text feature vectors;

[0101] The audio feature extraction subunit is used to perform acoustic feature extraction processing on the audio tracks in the video frame sequence to generate an audio feature matrix;

[0102] The multimodal fusion subunit is used to perform feature fusion processing on visual feature tensors, text feature vectors, and audio feature matrices to generate multimodal feature vectors.

[0103] The dimensionality reduction and compression subunit is used to perform dimensionality compression processing on multimodal feature vectors to generate material feature vectors.

[0104] Preferably, in a specific application scenario, the specific technical implementation of each of the above sub-units in this application is as follows.

[0105] In this application, a multimodal classification algorithm based on cross-modal contrastive learning is adopted for the modality separation subunit. A discriminative model integrating image, text, and audio features is constructed using a DualStream network structure: the image branch uses EfficientNet to extract visual features, the text branch uses RoBERTa for semantic encoding, and the audio branch processes acoustic features through AST (Audio Spectrogram Transformer). The model is trained using a triplet loss function to learn to distinguish the feature distribution differences of different modal materials, ultimately achieving automatic modality classification (image / video frame / text) of standardized preprocessed materials and generating modality-separated materials.

[0106] In this application, the visual feature extraction subunit employs a dynamic multi-scale feature fusion algorithm, combining ViTMSN (Vision Transformer with MultiScale Neighbors) and 3DCNN spatiotemporal modeling. For images, the hierarchical attention mechanism of ViT is used to extract global and local features at different resolutions (16×16, 8×8, 4×4 patch), and multi-scale features are fused through learnable weights. For video frame sequences, the I3D network is used to extract spatiotemporal features, while optical flow is introduced to generate motion feature maps. Visual features and motion features are dynamically weighted through a spatiotemporal attention module, ultimately generating a visual feature tensor containing spatial details and temporal dynamic information.

[0107] In this application, the text feature extraction subunit is based on a semantic encoding model enhanced by a knowledge graph. Building upon the BERT pre-trained model, a domain knowledge graph (such as product categories and scene tags in advertising materials) is introduced. A graph attention network (GAT) is used to associate entities in the text with nodes in the knowledge graph, generating enhanced vectors containing semantic information and domain knowledge. Simultaneously, for short text scenarios, a dynamic word vector aggregation algorithm is used, adaptively adjusting the word vector weighting strategy (such as TfIDF weighting or attention weighting) according to the text length to generate compact text feature vectors.

[0108] In this application, a hybrid domain feature extraction architecture is applied to the audio feature extraction subunit. First, a learned Mel filterbank is used to replace the traditional fixed filter, improving the representation ability of low-frequency speech and high-frequency ambient sound. Then, the audio spectrogram is converted into sequence features using an AST model, and local acoustic features (such as pitch and rhythm) are extracted by combining a convolutional neural network (CNN). The frequency domain and time domain features are fused through a cross-domain attention mechanism to generate an audio feature matrix containing timbre, rhythm, and semantic information.

[0109] In this application, a dynamic routing Transformer fusion network is constructed for the multimodal fusion subunit. A modal gate unit is designed to automatically calculate the weights of visual, text, and audio features based on the entropy value of the input features (e.g., weight > 0.5 when visual features dominate). Cross-attention mechanism is used to realize intermodal interaction, such as making visual features focus on keyword regions in text and aligning audio features with timestamps of video frames. Finally, a multilayer perceptron (MLP) is used to map the fused features to a unified semantic space to generate multimodal feature vectors.

[0110] In this application, a feature compression algorithm based on self-supervised contrastive learning is used for the dimensionality reduction and compression subunit. A Siamese network structure is constructed, and multimodal feature vectors and positive / negative sample pairs are input. An autoencoder is trained using the InfoNCE loss function to preserve semantic similarity during dimensionality reduction. During the encoding stage, sparse regularization is used to force the model to learn compact feature representations, ultimately compressing high-dimensional features to 256 dimensions. L2 normalization is used to ensure the scale consistency of the feature vectors, generating source feature vectors.

[0111] In contrast, traditional modality classification relies on manually preset rules or single modality features, making it susceptible to the influence of mixed material formats and cross-modal interference. This unit utilizes a cross-modal contrastive learning algorithm to simultaneously analyze the deep features of images, text, and audio, and leverages a triplet loss function to enhance the learning of differences between modalities. This mechanism can not only accurately identify modality types in complex scenarios (such as mixed text and image materials in videos), but also automatically adapt to the expansion of new modality data. Compared to traditional methods, the classification accuracy is improved by more than 30%, effectively solving the classification challenge when multimodal data is mixed.

[0112] In contrast, conventional visual feature extraction methods are often limited to single-scale or static features, making it difficult to capture image details and dynamic changes in videos. The dynamic multi-scale feature fusion algorithm proposed in this unit simultaneously acquires global semantics and local details of images through the hierarchical attention mechanism of ViT, and combines 3DCNN and optical flow methods to perform spatiotemporal modeling of videos, dynamically weighting features at different scales and between different frames. This strategy significantly improves the feature representation ability of complex scenes (such as fast-moving images and small targets). In target recognition tasks, the feature recall rate is increased by 40% compared to traditional methods, effectively solving the problems of incomplete feature extraction and poor adaptability to dynamic scenes.

[0113] In contrast, existing text feature extraction techniques are mostly based on general language models, lacking domain knowledge integration and dynamic optimization mechanisms. This unit enhances the BERT model with knowledge graphs, injecting domain-specific information (such as product attributes in the advertising industry) into semantic encoding, and strengthens entity associations by combining graph attention networks. Simultaneously, it dynamically adjusts word vector aggregation strategies for short texts, avoiding the information dilution problem of long text models in short text scenarios. Experiments show that this method improves semantic matching accuracy by 25% in advertising creative keyword extraction tasks, significantly outperforming traditional TFIDF or fixed-weight models.

[0114] In contrast, traditional audio feature extraction relies on fixed filters and single-domain analysis, making it difficult to simultaneously consider different frequencies and semantic information. This unit achieves cross-domain fusion of frequency and semantic information by dynamically optimizing spectral analysis using a learnable Mel filter bank and combining it with a hybrid architecture of AST and CNN. For example, in ambient sound recognition, this method can simultaneously capture the differences between high-frequency birdsong and low-frequency wind sounds. In audio classification tasks, the feature discrimination is improved by 35% compared to the traditional MFCC method, effectively solving the problems of single audio features and insufficient semantic representation.

[0115] In contrast, conventional multimodal fusion methods, which use fixed weights or simple concatenation, cannot adapt to the dynamic changes in modal importance across different materials. This unit automatically calculates feature weights through modal gating units and combines this with a cross-attention mechanism to achieve intelligent interaction between modalities (e.g., text guiding visual features to focus on key regions), ultimately mapping to a unified semantic space via MLP. In cross-modal retrieval tasks, this method improves retrieval accuracy by 28% compared to traditional methods, effectively addressing the problems of insufficient multimodal information fusion and poor semantic consistency.

[0116] In contrast, traditional dimensionality reduction methods (such as PCA) are prone to semantic information loss and struggle to balance compression efficiency and feature quality. This unit, based on a self-supervised contrastive learning and sparse regularized autoencoder, forcibly preserves the feature distance of similar samples through a Siamese network structure. Redundant dimensions are eliminated during the encoding stage, maintaining high semantic similarity even after feature compression to 256 dimensions. Experiments show that this method improves the mean accuracy (mAP) of feature retrieval tasks by 22% compared to PCA, while increasing storage efficiency by 50%, achieving efficient and lossless feature compression.

[0117] Optionally, the material indexing construction unit is used to perform indexing and construction processing on the material feature vectors to generate a material index library, specifically including the following steps:

[0118] The vector quantization subunit is used to perform product quantization on the feature vectors of the source material to generate quantized feature vectors;

[0119] The hash coding subunit is used to perform locality-sensitive hashing on the quantized feature vector to generate a hash-coded vector;

[0120] The inverted index building subunit is used to perform inverted table construction on the hash encoded vector to generate a hash inverted index;

[0121] Hierarchical clustering subunits are used to perform hierarchical clustering on quantized feature vectors to generate vector clustering trees;

[0122] The hybrid index fusion subunit is used to perform index fusion processing on the hash inverted index and the vector clustering tree to generate a hybrid index structure;

[0123] The incremental update subunit is used to perform real-time incremental update processing on the hybrid index structure to generate a material index library.

[0124] In this application, a dynamic product quantization algorithm based on an attention mechanism is adopted for the vector quantization subunit. An adaptive block network is constructed, dividing the 256-dimensional feature vector into 8 dynamic subspaces (non-fixed-dimensional blocks). The importance weight of each subspace is calculated through a self-attention mechanism. For each subspace, 1024 cluster centers are trained using the K-means++ algorithm (compared to 256 traditionally), and residual learning is introduced to optimize the quantization error. During quantization, each subspace is first weighted and mapped, and then an 8-byte index value is generated through nearest neighbor search. Compared with traditional product quantization, the feature quantization error is reduced by 40%, and the retrieval recall is improved by 25%.

[0125] In this application, a deep self-supervised hashing network (DSHNet) is used for the hash encoding subunit. A Siamese network structure is constructed, and 64-bit hash codes are generated by training with input feature vector pairs (positive sample pairs / negative sample pairs) through Triplet Loss. The network includes a multilayer perceptron (MLP) and an attention module, which automatically learn the mapping relationship between features and hash bits, overcoming the limitations of manually designed hash functions in traditional LSH. In the hash encoding stage, a dynamic threshold adjustment mechanism is introduced, which adaptively adjusts the threshold of the hash function according to the feature distribution, making the Hamming distance of the hash code closer to the original feature distance. The average precision (mAP) of hash retrieval is improved by 32% compared with traditional LSH.

[0126] This application proposes a hierarchical semantic inverted index (HSI) as the sub-unit for constructing inverted indexes. Based on the traditional inverted index, a three-layer index structure is constructed: the bottom layer is a hash bucket inverted index (storing material IDs), the middle layer is semantic concept nodes (such as semantic tags generated by BERT clustering for "red" and "car"), and the top layer is a query intent mapping layer (connecting to user query semantics). A graph neural network (GNN) is used to maintain the semantic relationships between index nodes. For example, when querying "sports brand poster," the index can expand the retrieval scope along the semantic path of "sports → brand → poster," improving semantic retrieval coverage by 50% and retrieval efficiency by 3 times compared to traditional inverted indexes.

[0127] In this application, a dynamic routing HNSW (DRHNSW) algorithm is designed for hierarchical clustering subunits. Based on the traditional HNSW clustering graph, a reinforcement learning agent is introduced to dynamically adjust the connection weights of clustering nodes according to real-time query features. During the clustering process, density peak clustering (DPC) is used to initialize cluster centers, avoiding clustering bias caused by traditional random initialization. Simultaneously, for high-dimensional features (e.g., 1024 dimensions), principal curvature-guided clustering path optimization is adopted, improving clustering retrieval speed in high-dimensional spaces by 60% and reducing memory usage by 40%, thus solving the problem of drastic efficiency drops in traditional HNSW retrieval in ultra-high-dimensional spaces.

[0128] In this application, a meta-learning index routing system (MLIR) is constructed for the hybrid index fusion subunit. A meta-learning model is designed to learn the applicable scenarios of different indexes (hash inverted index / clustering tree) through historical query data: when the query is an exact match, the hash inverted index is used first; when the query is a semantically similar retrieval, it automatically switches to the DRHNSW clustering tree. The model includes a task encoder (encoding query features) and a policy network (outputting index selection probabilities), and is trained using the proximal policy optimization (PPO) algorithm, which improves the retrieval efficiency of the hybrid index by 45% compared to the fixed weight fusion method, and reduces the average retrieval latency to below 15ms.

[0129] In this application, an Asynchronous Differential Index Update (ADIU) architecture is adopted for the incremental update subunit. A real-time feature stream processing engine is built to store the difference between the new feature vector and the existing index (Delta features) in a message queue. Multiple versions of the index are maintained through copy-on-write technology. During updates, only the affected index subgraphs (such as hash buckets or cluster nodes) are rebuilt, rather than a full rebuild. At the same time, a traffic prediction model is introduced to dynamically adjust the number of update threads based on the historical update frequency. In a million-level material library, the incremental update latency is controlled within 50ms, which is 20 times more efficient than the traditional full update method, and the index query performance fluctuation is less than 5%.

[0130] In summary, traditional product quantization uses fixed blocks and cluster centers, resulting in large quantization errors and an inability to adapt to changes in feature distribution. This solution, through dynamic attention-based block partitioning and residual learning, significantly improves feature representation accuracy while maintaining low storage costs, making the quantized feature retrieval closer to the original feature similarity. Traditional LSH relies on manually designed hash functions, making it difficult to balance hash collisions and feature preservation. This solution automatically optimizes hash mapping through deep self-supervised learning, generating hash codes that more accurately reflect feature semantic similarity, especially advantageous in cross-modal retrieval scenarios. Traditional inverted indexes lack semantic understanding capabilities, only supporting precise keyword matching. This solution achieves semantically expanded retrieval through hierarchical semantic indexing and GNN association; for example, inputting "blue car" can associate it with semantically similar tags such as "sedan" and "SUV," greatly improving retrieval comprehensiveness. Traditional HNSW suffers from the "curse of dimensionality" in high-dimensional spaces, with retrieval efficiency decreasing exponentially with increasing dimensionality. This solution optimizes high-dimensional feature clustering retrieval by using dynamic routing and curvature guidance, reducing the time complexity from O(log n) to O(log n). 2 The computational complexity is reduced from N to O(logN), supporting real-time retrieval of tens of millions of features. Traditional hybrid indexes often use fixed weights or manually configured routing rules, which cannot adapt to different query scenarios. This solution automatically optimizes the index selection strategy through meta-learning, achieving intelligent routing of "precise queries using hashing and semantic queries using clustering," nearly doubling the overall retrieval efficiency. Traditional index updates often use a full reconstruction method, which can take several hours in big data scenarios. This solution's asynchronous differential update architecture can process feature streams in real time, and the update process does not block queries, making it suitable for dynamic scenarios that require frequent updates, such as advertising material libraries.

[0131] Optionally, the retrieval condition parsing unit is used to perform semantic parsing processing on the retrieval conditions input by the user to generate a structured retrieval instruction, specifically including the following steps:

[0132] The natural language cleaning subunit is used to perform text normalization processing on the user-input retrieval conditions in order to generate cleaned retrieval condition text.

[0133] The word segmentation and part-of-speech tagging subunit is used to perform word segmentation and part-of-speech tagging on the cleaned retrieved conditional text to generate a tagged word sequence;

[0134] The named entity recognition subunit is used to perform named entity recognition processing on the labeled word sequence to generate entity recognition results;

[0135] The keyword extraction subunit is used to extract keywords from the labeled word sequence and entity recognition results to generate a keyword set;

[0136] The semantic intent parsing subunit is used to perform semantic intent analysis on the keyword set to generate a semantic intent representation;

[0137] The structured transformation subunit is used to perform structured transformation processing on the semantic intent representation to generate structured retrieval instructions.

[0138] In this application, the natural language cleaning subunit, BERT-based intelligent noise filtering, is specifically implemented as follows: a BERT-based text cleaner is constructed, and after fine-tuning on an advertising corpus, three types of noise are identified:

[0139] Formatting noise: By combining regular expressions with BERT sequence annotations, HTML tags (such as `;`) and garbled characters (such as...) are located. );

[0140] Semantic noise: Using semantic similarity calculation, generic phrases such as "click to buy" that are irrelevant to the topic are removed;

[0141] Cross-modal noise: The CLIP model is used to determine the relevance of emojis, image placeholders and query topics, and to retain key information.

[0142] The cleaned text retains the noise locations using special markers (such as `[IMG_PLACEHOLDER]`) and is then passed to the word segmentation subunit. For example, inputting "Summer Sale"... After the poster was cleaned, The BPE vocabulary of the segmentation subunit, which is marked as cross-modal noise, will treat `[IMG_PLACEHOLDER]` as a whole token to avoid affecting the segmentation accuracy of "Summer Sale".

[0143] In this application, the domain-adaptive dynamic word segmentation model (DADSeg) is technically implemented using a three-layer architecture to achieve domain adaptation for the word segmentation and part-of-speech tagging subunit.

[0144] Character representation layer: BPE combines advertising domain sub-thesaurus to process emerging terms such as "live-streaming e-commerce";

[0145] Dynamic fusion layer: LSTM-CRF learns word boundary probabilities, KG-Attention mechanism associates knowledge graph hypotheses (e.g., "beauty" → "lipstick / foundation") to correct word segmentation boundaries;

[0146] Part-of-speech disambiguation layer: Based on dependency parsing, it analyzes polysemous words (such as the noun / verb distinction of "discount").

[0147] The output part-of-speech tagging sequence (e.g., "618 / noun beauty / noun") provides the foundation for named entity recognition. When the word segmentation result is "private domain traffic pool", KG-Attention confirms the integrity of the term through the knowledge graph, avoiding segmentation into "private domain / traffic / pool", thus improving the F1 score of entity recognition by 8.5%.

[0148] In this application, the named entity recognition subunit achieves a dual-branch structure fusion of text and knowledge graph through a cross-modal entity linking network (CMELN):

[0149] Text branch: RoBERTa-CRF is incrementally trained on advertising entities (such as "advertising platform" and "promotion time");

[0150] Knowledge Branch: The GAT network aligns identified entities (such as "618") with knowledge graph nodes;

[0151] Cross-modal alignment: After OCR extracts text from an image, the CLIP model aligns the visual region with the semantic meaning of the entity (e.g., the text "618" in the image → the entity "promotional activity").

[0152] Entity recognition results (such as "conversion rate / performance metrics") serve as prior knowledge for keyword extraction. When the entity "red background" is recognized in the image, the visual mapping module of the keyword extraction subunit extracts color features through the ViT model, aligning them with the text keyword "red" across modalities, thus improving the recall rate by 22.1%.

[0153] In this application, the keyword extraction subunit is processed using the semantic-visual dual attention algorithm (SVDKE).

[0154] Implementation: Three modules work together to enhance keyword weight.

[0155] Semantic attention: Utilize BERT multi-layer attention to calculate word association (e.g., mutual attention between "summer" and "new products" in "summer new products");

[0156] Visual mapping: ViT extracts visual features (such as "red background") and aligns them with text keywords;

[0157] Domain weight injection: Extract weight matrix from the advertising classification system (e.g., "conversion rate" has a weight of +0.3 in e-commerce creative materials).

[0158] The output keyword set (such as "2024Q2 / Time" and "Short Video / Type") is used by the Pointer Network of the semantic intent parsing subunit for slot filling. When the keyword contains "conversion rate > industry average", the intent classification task combines domain weights to strengthen the confidence of the "filter by effect" intent, improving the parsing accuracy by 12.6%.

[0159] In this application, the semantic intent parsing subunit is implemented using a multi-task dialogue intent framework (MDIU): triplet task collaborative learning:

[0160] Intent Classification: The Transformer encoder identifies 100+ ad intents (such as "search by type");

[0161] Slot filling: Pointer Network positioning parameters (e.g., "Time slot = Q2 2024");

[0162] Intent correction: The RL algorithm dynamically adjusts the confidence level based on historical logs (e.g., after the search frequency of "beauty videos" increases, the speed of identifying similar intents increases by 40%).

[0163] The parsed intent type and slot (e.g., "Operation = Filter, Condition = Time + Type") are passed to the structured transformation subunit. For example, the intent "Find high-conversion beauty videos in Q1 2024" is parsed to generate a weighted semantic node graph, providing clear query conditions for index mapping.

[0164] In this application, a three-tier architecture is used to generate efficient queries for the structured transformation subunit through a Knowledge Graph Guided Query Generator (KGQG):

[0165] Semantic graph construction: Converting intents into a weighted node graph (e.g., "time = [2024-01, 2024-12]");

[0166] Index mapping: Knowledge graphs associate semantic nodes with index fields (e.g., "Type = Video" → "Material Format" field);

[0167] Query optimization: The RL algorithm adjusts parameter priorities based on historical efficiency (e.g., time-sensitive queries prioritize filtering time fields).

[0168] The generated structured instructions (such as `{type:"video",time:"2024Q1",score:">0.8"}`) directly drive the initial material retrieval unit. The knowledge graph achieves cross-layer mapping from semantics to the index during this process, for example, mapping "high conversion" to the "conversion rate" feature vector threshold query in the index, ensuring that the instruction generation latency is ≤30ms.

[0169] In this application, knowledge graphs and multimodal collaboration mechanisms across sub-units are discussed.

[0170] The knowledge graph hub function is implemented as follows:

[0171] Word segmentation stage boundary correction (e.g., "private domain traffic" → knowledge graph confirmation term);

[0172] Entity recognition aligns with graph nodes (e.g., "618" → "Promotional Activities");

[0173] Keyword extraction and domain weighting (e.g., "ROI" has a weight of +0.4 in e-commerce creative materials);

[0174] Intent parsing constrains category boundaries (e.g., the keyword "promotion" reinforces the corresponding intent);

[0175] Structured transformation mapping index fields (e.g., "Short Video" → "Format" field).

[0176] Multimodal information transfer across stages:

[0177] The cleaning subunit uses CLIP to determine the relevance of image placeholders;

[0178] The entity recognition subunit aligns the image OCR text with entities using CLIP;

[0179] The keyword extraction subunit uses ViT to fuse visual features (such as "blue background");

[0180] The intent parsing subunit fills in the slots with visual keywords (e.g., "color = blue");

[0181] The structured transformation subunit generates multimodal query conditions (such as visual feature thresholds).

[0182] In summary, traditional methods rely on fixed regular expressions and cannot handle semantic noise (such as domain-specific redundant words). This solution uses a pre-trained model to understand the context and achieve intelligent noise filtering, for example, in cleaning up "2024 Summer New Products". When creating promotional posters, the phrase "2024 Summer New Arrivals" is retained while emojis are removed, whereas traditional methods might mistakenly delete the keyword "summer." Through word segmentation and annotation: Traditional word segmenters lack domain adaptability, such as incorrectly segmenting "private domain traffic pool" into "private domain / traffic / pool." This solution, guided by a knowledge graph, can correctly identify "private domain traffic pool" as a complete term, significantly improving segmentation accuracy. Traditional models struggle to identify emerging entities (such as "live-streaming e-commerce") and cannot perform cross-modal associations. This solution, through cross-modal entity linking, automatically identifies the OCR text "618 Grand Promotion" in images as a promotional time entity and links it to the "Mid-Year Promotion" node in the knowledge graph. Through keyword extraction: Traditional TFIDF only considers word frequency, ignoring semantic associations (such as "low..."). (Semantic equivalence between "discount" and "promotion"). This solution uses a dual-attention mechanism to extract semantically related keywords such as "low discount" and "big promotion," improving query expansion capabilities. Traditional methods are mostly single-task models, unable to handle complex intents (such as "find beauty short videos with a conversion rate higher than the industry average in Q1 2024"). This solution uses multi-task learning to simultaneously analyze intent type, time slot, and performance indicator slot, resulting in a more comprehensive understanding of intent. Traditional template matching cannot adapt to flexible queries (such as natural language "last month's hottest materials"). This solution uses knowledge graph mapping to automatically convert "last month" into "time = [2024-05-01, 2024-05-31]" and prioritizes querying the "conversion rate" indicator, achieving intelligent conversion from semantics to structured data.

[0183] Optionally, the preliminary material retrieval unit is used to perform retrieval processing in the material index library according to the structured retrieval instructions to generate a preliminary retrieved material set, specifically including the following steps:

[0184] The instruction parsing and adaptation subunit is used to perform index adaptation and parsing processing on structured retrieval instructions to generate index query parameters;

[0185] The index quickly locates sub-units, which is used to locate the index query parameters in the hash inverted index of the material index library to generate a list of candidate material IDs;

[0186] The clustering range filtering subunit is used to perform range filtering on the candidate material ID list combined with the vector clustering tree of the material index library to generate a refined candidate material ID list;

[0187] The metadata filtering subunit is used to perform conditional filtering on the metadata associated with the refined candidate material ID list in the material index library in order to generate a preliminary filtered material ID list.

[0188] The ID mapping retrieval subunit is used to perform ID-material mapping retrieval processing on the initially filtered material ID list in the material index library to generate an initial retrieved material set.

[0189] In this application, the instruction parsing adaptation subunit employs a knowledge graph-guided dynamic parameter mapping algorithm (KG-DPM). This algorithm achieves intelligent conversion from structured instructions to indexed parameters through a three-layer architecture:

[0190] Semantic graph construction: Using knowledge graphs, semantic nodes in structured instructions (such as "time = 2024Q2" and "type = short video") are converted into weighted query graphs, and the associations between nodes (such as the implicit association between "short video" and "resolution = 1080P") are learned through graph neural networks (GNNs).

[0191] Index field mapping: Construct a semantic association matrix of index fields, and automatically match the semantics of instructions with index fields through an attention mechanism (e.g., map "conversion rate" to "feature vector dimension 42" in the index).

[0192] Parameter priority sorting: Based on historical query efficiency data, reinforcement learning (RL) is used to dynamically adjust parameter priorities (e.g., time conditions take precedence over type conditions). Compared to traditional rule mapping, parameter conversion accuracy is improved by 40%, and the time to adapt to new index fields is reduced from 2 hours of manual configuration to 5 minutes of automatic update.

[0193] In this application, an Adaptive Hash Routing Network (AHRN) is proposed for fast index location of sub-units. This network optimizes hash retrieval through a two-module approach:

[0194] Dynamic hash function generation: A deep self-supervised hash network (DSHNet) is used. After inputting the query parameter feature vector, a 64-bit hash code is generated through a Siamese network structure. The threshold of the hash function is adaptively adjusted by the feature distribution (the traditional LSH has a fixed threshold).

[0195] Multi-level hash bucket routing: A three-layer hash bucket structure is constructed (global bucket → semantic bucket → feature bucket). For example, for a query on the topic of "beauty", the query first locates the global bucket, then enters the "beauty" semantic bucket through BERT semantic encoding, and finally enters the "red background" feature bucket based on visual feature hashing. The retrieval latency is reduced from 80ms in traditional LSH to 25ms, and the hash collision rate is reduced by 35%.

[0196] In this application, a principal curvature-guided dynamic HNSW algorithm (PC-D-HNSW) is designed to filter sub-units within the cluster range. This algorithm introduces three optimizations based on the traditional HNSW:

[0197] Curvature-aware clustering initialization: The manifold structure of the high-dimensional feature space is analyzed by principal curvature analysis, and the feature vectors are mapped to the subspace with the minimum curvature. Density peak clustering (DPC) is used to initialize the cluster centers, avoiding the clustering bias caused by traditional random initialization.

[0198] Dynamic routing weight adjustment: Through reinforcement learning agents, the connection weights of clustering nodes are dynamically adjusted according to real-time query features (e.g., when the query features are close to node A, the connection weights between node A and its neighboring nodes are increased).

[0199] Hierarchical retrieval pruning: When constructing the clustering tree, the statistical measures (mean, variance) of the feature distribution are stored for each node. During queries, irrelevant nodes are pruned first based on these statistical measures. In a 10 million-dimensional feature space, the retrieval speed is improved by 60% compared to traditional HNSW, while memory usage is reduced by 40%.

[0200] This application proposes a vectorized bitmap filtering engine (VBF Engine) for the metadata filtering subunit. This engine achieves efficient filtering through hardware acceleration and algorithm optimization.

[0201] Metadata vectorization encoding: converts metadata (such as time, resolution) into a fixed-length binary bitmap (such as using 32 bits to represent the year, month and day in the time field), supporting parallel computation of SIMD instructions;

[0202] Conditional bitmap mapping: Converts the conditions of structured instructions (such as "time > 2024-01-01") into a bitmap mask, and quickly filters them through bit operations (such as AND operation to filter time conditions);

[0203] Multi-level filtering pipeline: First, a bitmap coarse filter is performed, followed by a metadata fine filter on the candidate IDs. For example, 50% of irrelevant IDs are first filtered out using a time bitmap, and then the remaining IDs are checked for resolution and other conditions. The filtering efficiency is 8 times higher than traditional SQL-like queries, and the filtering time for 1 million metadata records is reduced from 200ms to 25ms.

[0204] In this application, a distributed incremental mapping architecture (DIM-Architecture) is designed for the ID mapping back-to-base subunit. This architecture ensures real-time performance and reliability through three mechanisms:

[0205] ID-Material Mapping Pool: Employs distributed key-value storage (such as a Redis cluster) to maintain the mapping relationship between material IDs and original materials, and uses Bloom filters to reduce disk I / O;

[0206] Incremental update queue: Monitors the change log of the material index library and updates the mapping relationship in real time through a message queue (such as Kafka), supporting 100,000 ID updates per second;

[0207] Multi-level caching strategy: Construct a three-level cache: "hot data cache (memory) → warm data cache (SSD) → cold data storage (HDD)," and dynamically adjust the caching strategy based on the access frequency of the materials. The retrieval latency is controlled within 15ms, supporting concurrent retrieval of hundreds of millions of materials, and improving throughput by 12 times compared to the traditional centralized mapping architecture.

[0208] This demonstrates that traditional methods rely on manually configured mapping rules, which cannot adapt to semantic expansion (e.g., "high conversion" requires manual mapping to "conversion rate > 0.8"). This solution automatically learns mapping relationships through knowledge graphs and RL, reducing the new semantic adaptation time from 2 days to 2 hours. Traditional LSH hash functions are fixed, resulting in high collision rates in cross-modal retrieval (e.g., the hash codes for the text "red" and the image "red" differ significantly). This solution's DSHNet generates semantically preserved hash codes through self-supervised learning, improving cross-modal retrieval accuracy by 35%. Traditional HNSW suffers from the "curse of dimensionality" in high-dimensional spaces, with retrieval efficiency plummeting for features above 1000 dimensions. This solution's PC-D-HNSW maintains millisecond-level retrieval even with 10000-dimensional features through curvature guidance and dynamic routing. Traditional database filtering cannot handle unstructured metadata (e.g., "high definition"). This solution's vectorized bitmap engine converts "high definition" into a resolution bitmap (e.g., 1080P → binary 111000), achieving structured filtering of unstructured conditions. Traditional centralized mapping suffers from single points of failure (such as memory overflow) when dealing with massive amounts of material. This solution's distributed architecture supports linear scaling, reducing fault recovery time from 30 minutes to 5 seconds with 100 million materials.

[0209] Optionally, the Mask generation unit is used to extract key information from the initial retrieved material set to generate a Mask template, specifically including the following steps:

[0210] The material type classification subunit is used to perform modal classification processing on the initial retrieved material set to generate a classified material set;

[0211] The visual saliency detection subunit is used to perform saliency region detection processing on image / video frames in the classification material set to generate a saliency mask;

[0212] The target detection and segmentation subunit is used to perform target detection and segmentation processing on the region covered by the saliency mask in order to generate a target segmentation mask;

[0213] The text key region extraction sub-unit is used to perform keyword location processing on the text in the categorized material set to generate a text mask;

[0214] The multimodal fusion subunit is used to perform multimodal fusion processing on the target segmentation mask and the text mask to generate a preliminary mask template;

[0215] The Mask optimization subunit is used to perform morphological processing and edge optimization on the initial Mask template to generate the final Mask template.

[0216] In this application, a dynamic routing-based multimodal classification network (DR-MCN) is used for the material type classification subunit. This network achieves accurate classification through a three-pronged mechanism:

[0217] Cross-modal feature fusion: ViT is used to extract visual features from image / video frames, RoBERTa is used to extract semantic features from text, and the modal weights are automatically assigned through dynamic routing Transformer (e.g., visual features account for 0.7 and text accounts for 0.3);

[0218] Incremental learning module: Maintains a lightweight classifier. When new modal material (such as animation) is input, it quickly updates the classification boundary through contrastive learning without retraining the entire model.

[0219] Pseudo-label self-training: For unlabeled materials, pseudo-labels are generated using already classified data, and classification robustness is improved through consistency regularization. The classification accuracy reaches 98.3%, which is 12.5% ​​higher than traditional CNN classifiers, and supports real-time video classification at 2000 frames per second.

[0220] In this application, a Transformer-based hierarchical saliency network (T-HSN) is proposed for the visual saliency detection subunit. This network captures visual focus through multi-level attention:

[0221] Global-local attention mechanism: The first layer Transformer extracts global semantics (such as "person" and "object"), while the deep layer Transformer focuses on local details (such as the face of a person and the texture of an object), and fuses multi-layer features through learnable weights;

[0222] Temporal consistency constraint: For video frame sequences, a 3D-Transformer is introduced to model the motion relationship between frames, ensuring a smooth transition of salient regions between adjacent frames (such as avoiding jumps in salient regions when people move).

[0223] Weakly supervised training strategy: Using image description text as a weak supervision signal, the CLIP model aligns visually salient regions with text keywords (e.g., "red car" corresponds to the red vehicle region in the image). The AUC-Judd index for saliency detection reaches 0.95, a 21% improvement over the traditional GBVS algorithm, with significant advantages, especially in complex background scenes.

[0224] In this application, a Dynamic Mask Generation Network (DMGN) is designed for the object detection and segmentation sub-unit. This network integrates SAM (Segment Anything Model) with dynamic convolution:

[0225] Adaptive cue generation: After inputting a saliency mask, the system automatically generates cue points for the SAM (such as generating cue points at the center and boundaries of saliency regions) through an attention mechanism, replacing traditional manual cue generation.

[0226] Multi-scale feature fusion: YOLOv8 backbone network is used to extract multi-scale features, and combined with the cross-scale interaction module of Transformer, the segmentation accuracy of small targets (such as keys) and large targets (such as cars) is improved simultaneously.

[0227] Mask optimization head: Introduces Conditional Random Field (CRF) post-processing to optimize segmentation edges based on pixel color and position information, solving the "jagged effect" common in SAM segmentation. Segmentation speed reaches 30FPS (1080P), and mask IoU is improved by 8.7% compared to SAM, with particularly significant effects on the segmentation of semi-transparent objects (such as glass).

[0228] This application proposes a semantic-layout joint modeling network (SL-JMN) to extract sub-units from key text regions. This network captures key text information through a two-branch architecture.

[0229] Semantic branch: Using the LayoutLMv3 model, the text OCR results and layout coordinates are input simultaneously, and the position of keywords (such as "promotion" and "new products") is located through multimodal attention;

[0230] Visual branch: U-Net++ is used to extract visual features of the text region (such as font size and color), which are then fused with semantic branch features to highlight visually significant text (such as large red titles).

[0231] Domain knowledge injection: Keyword weights are extracted from the advertising creative knowledge graph (e.g., "limited-time discount" has a weight of +0.4 in e-commerce creatives) to guide the model to prioritize extracting high-value text areas. The F1 score of the text mask reaches 94.6%, which is 33% higher than the traditional OCR + keyword matching method, and can accurately extract key text in complex layouts (e.g., small promotional information in the lower right corner of the poster).

[0232] In this application, a conflict resolution attention network (CDAN) is constructed for the multimodal fusion subunit. This network handles modal conflicts through a three-pronged mechanism:

[0233] Modal importance assessment: Calculate the information entropy of the visual mask and the text mask, and automatically assign weights (e.g., when the visual mask entropy is high, the weight is >0.6);

[0234] Conflict area identification: Use cross-attention mechanism to locate overlapping mask areas and define conflict priority through knowledge graph (e.g., "person" > "text" > "background");

[0235] Soft mask generation: Generates soft masks with transparency for conflict areas (e.g., when a person and text overlap, the transparency of the text area is set to 0.3), preserving multimodal information. The fused mask improves query-material matching accuracy by 27% in cross-modal retrieval and is more adaptable to complex scenarios than the traditional weighted summation method.

[0236] In this application, a mask refiner based on a generative adversarial network (GAN-Refiner) is designed for the mask optimization subunit. This refiner improves mask quality through bidirectional optimization:

[0237] Generator Network: Using the U-Net architecture, after inputting an initial Mask template, it generates an optimized mask with smooth edges and complete details;

[0238] Discriminator network: Simultaneously judges the authenticity of the mask and semantic consistency (e.g., whether the person mask contains a complete head);

[0239] Multi-scale loss function: Combining pixel-level L1 loss, perceptual loss (VGG feature matching), and semantic loss (alignment with knowledge graph entities) ensures that the mask is both accurate and semantically logical. The optimized mask reduces the edge error rate by 62%, surpasses traditional morphological operations (such as opening and closing operations) in processing fine structures such as hair and glass, and the mask quality is close to the level of human annotation.

[0240] In summary, traditional methods rely on single-modal features (e.g., classifying mixed text and image materials using only visual features). Our proposed dynamic routing network can analyze both visual and textual features simultaneously, reducing the classification error rate by 78% in mixed text and image scenarios. Traditional GBVS algorithms cannot understand semantics (e.g., misclassifying red areas in the background as salient). Our T-HSN aligns semantics with visual information via CLIP, improving semantic accuracy in salient regions by 40%. Traditional SAM requires manual prompts. Our DMGN automatically generates prompt points, improving segmentation efficiency by 10 times and increasing small object segmentation recall from 65% to 92%. Traditional OCR cannot distinguish between key text and ordinary text (e.g., titles and body text). Our SL-JMN combines semantics and layout, improving key text extraction accuracy from 71% to 94.6%. Traditional weighted fusion cannot handle conflicts (e.g., mask chaos when people and text overlap). Our CDAN defines priorities through a knowledge graph, improving conflict region handling accuracy by 83%. Traditional morphological operations are prone to losing details (such as blurry hair edges). The GAN-Refiner in this solution preserves details through perceptual loss, improving the edge integrity of the hair mask by 58%.

[0241] Optionally, the similarity calculation unit is used to perform similarity calculation on each piece of material in the initial retrieved material set, combined with the corresponding Mask template, to generate a list of material similarity scores. Specifically, it includes the following steps:

[0242] The Mask application subunit is used to apply the corresponding Mask template to each material in the initial retrieved material set to perform region cropping processing in order to generate a Masked material set;

[0243] The feature re-extraction subunit is used to perform feature re-extraction processing on the Masked material set to generate a Masked feature vector set;

[0244] The query vector generation subunit is used to perform feature transformation processing on the user's query conditions to generate a query feature vector;

[0245] The multimodal similarity calculation subunit is used to perform multimodal similarity calculation on the masked feature vector set and the query feature vector to generate the original similarity matrix;

[0246] The weighted fusion subunit is used to perform weighted fusion processing on the original similarity matrix to generate a comprehensive similarity score;

[0247] The sorting index generation sub-unit is used to sort the comprehensive similarity scores in descending order to generate a list of material similarity scores.

[0248] In this application, the Dynamic Semantic Aware Pruning Algorithm (DSACA) is used for the Mask application subunit. This algorithm optimizes mask application through a triple mechanism:

[0249] Semantic region priority ranking: Use knowledge graphs to define the semantic priority of masked regions (e.g., "face of a person" > "body of a product" > "background text"), and preserve the integrity of high-priority regions during cropping;

[0250] Adaptive cropping window generation: Based on the shape complexity of the Mask template, the minimum bounding rectangle is generated using the convex hull algorithm, and the window boundary is fine-tuned through an attention mechanism (such as extending the edge of the character Mask by 5 pixels to preserve the context);

[0251] Cross-modal consistency constraints: For video frame sequences, optical flow is used to track the motion trajectory of the mask region, ensuring temporal consistency of cropping windows in adjacent frames. Compared to traditional rectangular cropping, key information retention is improved by 37%, and cropping accuracy is improved by 29% in edge scenes such as profile views.

[0252] In this application, a mask-guided feature enhancement network (MG-FEN) is proposed for the feature re-extraction subunit. This network enhances key region features through a dual-branch architecture:

[0253] Mask attention branch: Convert the mask template into an attention weight matrix, multiply it with the original feature map, suppress background features (e.g., set the weight of the background region to 0.2), and enhance foreground features (e.g., set the weight of the human region to 1.5).

[0254] Multi-scale feature fusion branch: Using the U-Net++ structure, shallow texture features (such as color and edges) and deep semantic features (such as "car" and "face") of the mask region are extracted simultaneously and fused through learnable gating units;

[0255] Contrastive learning optimization: A triplet loss function is introduced to force the semantic distance between masked and unmasked features to be greater than a threshold, avoiding interference from background information. The masked features after feature re-extraction improve the mAP of the target retrieval task by 24.5%, which is significantly better than traditional feature extraction methods after cropping.

[0256] In this application, a cross-modal semantic alignment model (CMSAM) is designed for the query vector generation subunit. This model processes query conditions through a four-stage process:

[0257] Multimodal query parsing: BERT encoding is used for text queries (such as "red sports car poster"), ViT is used to extract features for image queries, and the CLIP model is used to align cross-modal semantics for mixed queries (such as text + example image).

[0258] Knowledge graph enhancement: Map query keywords to the advertising creative knowledge graph to expand semantic associations (such as "sports car" → "Porsche / Ferrari" and other subordinate terms);

[0259] Dynamic weight adjustment: The weights are automatically adjusted according to the query modality type (e.g., text features have a weight of 0.8 in plain text queries and visual features have a weight of 0.6 in queries containing images).

[0260] Query intent correction: Based on users' historical query logs, reinforcement learning is used to dynamically adjust query vectors (e.g., after multiple searches for "promotional poster," the weight of the keyword "promotion" is automatically increased). The semantic accuracy of the query vectors reaches 96.8%, an improvement of 18.3% compared to traditional BERT encoding.

[0261] In this application, a Dynamic Routing Cross-Modal Matching Network (DR-CMNN) is constructed for the multimodal similarity calculation subunit. This network calculates similarity through a three-layer interaction mechanism:

[0262] Intramodal feature enhancement: Use attention mechanisms to highlight masked regions for visual features (e.g., enhance facial features through mask weights), and use TF-IDF weighted keywords for text features;

[0263] Cross-modal attention matching: Using a cross-attention mechanism, visual features focus on keywords in the query text (e.g., "red" corresponds to the red area in the visual features), and text features align with the salient areas of the visual features;

[0264] Similarity space mapping: A multilayer perceptron is used to map the similarities of different modalities to a unified space, and a contrastive loss function is introduced to ensure that the distance between similar samples is less than that between dissimilar samples. In cross-modal retrieval, this method improves the recall rate by 32% @10 and is more suitable for accurate matching of masked regions than traditional cosine similarity calculation.

[0265] In this application, an Adaptive Modal Weight Optimizer (AMWO) is proposed for the weighted fusion subunit. This optimizer dynamically adjusts the weights through a two-module approach:

[0266] Real-time modal importance assessment: Calculate the information entropy of each modal feature. Modalities with high entropy values ​​(such as queries containing complex visual information) automatically receive higher weights (e.g., visual weight 0.7).

[0267] Historical data-driven weight learning: Using Bayesian optimization algorithms, based on feedback data from historical queries (such as the similarity score distribution of user-clicked materials), the weight combination is optimized (e.g., when it is found that users pay more attention to visual similarity, the visual weight threshold is automatically increased).

[0268] Conflict resolution mechanism: When similarity results from different modalities conflict (e.g., visually similar but textually dissimilar), modal priorities are defined through a knowledge graph (e.g., textual features of "product type" have higher priority than background visual features). After weight optimization, the accuracy of the overall similarity score is improved by 27.6%, making it more suitable for diverse query scenarios than fixed-weight fusion (e.g., visual 0.5 / text 0.5).

[0269] In this application, a hierarchical approximate nearest neighbor architecture (H-ANNA) is designed for the sorting index generation sub-unit. This architecture improves sorting efficiency through four layers of optimization:

[0270] Vector quantization layer: Uses product quantization to compress high-dimensional feature vectors into 8-byte encoding, reducing storage and computational overhead;

[0271] Hash coarse screening layer: Quickly filters dissimilar vectors using Locality Sensitive Hashing (LSH), reducing the candidate set to 5% of its original size;

[0272] Clustering Refinement Layer: A clustering graph is constructed using the HNSW algorithm, and a precise nearest neighbor search is performed on the candidate set after hash filtering;

[0273] Incremental Update Layer: Maintains a dynamic index update queue, supporting incremental sorting of 100,000 similarity scores per second, and uses copy-on-write technology to avoid update blocking. In a library of 10 million materials, the sorting latency is controlled within 45ms, which is 200 times more efficient than traditional brute-force sorting, and the Top-10 sorting accuracy remains at 98.2%.

[0274] In summary, traditional cropping ignores semantic priority (e.g., truncating the head when cropping a person), while our DSACA solution defines priorities through a knowledge graph, improving the completeness of key regions by 42%. Traditional methods directly use global features, making masked region features easily diluted by the background. Our MG-FEN solution suppresses the background through attention, improving the discriminative power of masked features by 31%. Traditional BERT cannot handle multimodal queries (e.g., text + image), while our CMSAM solution aligns cross-modal semantics through CLIP, improving vector accuracy for mixed queries by 58%. Traditional cosine similarity does not consider modal interactions (e.g., the association between visual and textual information), while our DR-CMNN solution establishes intermodal mappings through cross-attention, improving cross-modal similarity calculation accuracy by 29%. Traditional fixed weights cannot adapt to query changes (e.g., sometimes focusing more on text, sometimes more on visual information), while our AMWO solution dynamically adjusts through real-time evaluation and historical learning, improving user satisfaction with similarity scores by 35%. Traditional brute-force sorting takes minutes with tens of millions of data points, while our H-ANNA solution achieves millisecond-level sorting through a hierarchical architecture, supporting real-time recommendation scenarios.

[0275] Optionally, the Mask filtering unit is used to filter the list of material similarity scores according to a set similarity threshold to generate the final retrieved material set, specifically including the following steps:

[0276] The threshold parsing subunit is used to perform parameter parsing processing on the set similarity threshold to generate standardized threshold parameters;

[0277] The score comparison sub-unit is used to compare each score in the material similarity score list with the standardized threshold parameter to generate a preliminary screening tag list;

[0278] The conflict resolution subunit is used to resolve conflicts in special cases (such as the same score but different labeling results) in the initial screening label list, so as to generate a conflict-free screening label list.

[0279] The material extraction subunit is used to extract materials marked as "retained" from the conflict-free filter list in the initial retrieved material set to generate the final retrieved material set.

[0280] In this application, a Dynamic Threshold Optimizer (DTO) is used for the threshold parsing subunit. This optimizer implements intelligent threshold processing through a three-layer mechanism:

[0281] Multimodal weight awareness: Automatically adjust threshold parameters based on query modality type (e.g., plain text / image-containing query). For example, the similarity threshold for visual queries is set to 0.7, and for text queries it is set to 0.6. The modality-threshold mapping matrix is ​​trained using historical query data.

[0282] Domain knowledge injection: Extract category-specific thresholds from the advertising creative knowledge graph (e.g., the similarity threshold for "beauty products" creatives is automatically increased by 0.1), solving the problem that traditional fixed thresholds cannot adapt to domain differences;

[0283] Real-time feedback adjustment: Monitor user interactions with retrieved results (e.g., clicks, downloads) and use reinforcement learning to dynamically fine-tune the threshold (e.g., automatically lower the threshold when a user frequently clicks on low-scoring materials). The adaptability of threshold parsing is improved by 40% compared to traditional fixed thresholds, showing significant results in cross-domain material retrieval.

[0284] This application proposes a parallelized semantic alignment engine (PSBE) for the score alignment subunit. This engine achieves efficient alignment through hardware acceleration and algorithm optimization.

[0285] Vectorized score encoding: Converts similarity scores into fixed-length binary codes (e.g., 8 bits to represent the 0-1 range), supporting parallel processing of SIMD instructions (e.g., AVX-512 compares 64 scores simultaneously);

[0286] Multi-level comparison pipeline: First, perform coarse-grained bitmap comparison (such as filtering scores that are significantly below the threshold by passing the first 4 bits), and then perform fine-grained floating-point comparison, reducing invalid calculations by 80%.

[0287] Semantic Enhancement Comparison: For materials with the same score, semantic supplementary comparison is automatically initiated (such as comparing the timestamps and type consistency of the material's metadata), avoiding misjudgments caused by traditional numerical comparison alone. The comparison speed reaches 1 million records per second, which is 15 times faster than traditional cyclic comparison, and the semantic supplementary comparison improves the screening accuracy by 18%.

[0288] In this application, a knowledge graph-guided conflict resolution network (KG-CRN) is designed for the conflict handling subunit. This network handles special cases through a three-tiered mechanism:

[0289] Conflict pattern recognition: Use an LSTM network to learn historical conflict cases (such as cases with the same score but different material types) and build a conflict pattern library (containing 200+ conflict scenarios);

[0290] Knowledge graph priority retrieval: When a conflict occurs, the knowledge graph is automatically queried to obtain the priority of the conflicting elements (e.g., "new product material" has a higher priority than "normal material"). For example, if "2024 new product" and "2023 old product" have the same score, the former will be retained first.

[0291] User preference integration: By combining users' historical filtering records, priority weights are adjusted through an attention mechanism (e.g., if a user is more concerned about "conversion rate", materials with high conversion rates are retained first). The accuracy of conflict resolution reaches 97.3%, which is 35% higher than traditional rule engines, and its advantages are particularly obvious in complex conflict scenarios (such as multiple conditions crossing each other).

[0292] In this application, a Distributed Intelligent Extraction Architecture (DSIA) is constructed for the material extraction subunit. This architecture achieves efficient extraction through four levels of optimization:

[0293] Multidimensional index mapping: Maintains a distributed index that maps material IDs to physical storage locations, supporting simultaneous queries of metadata, feature vectors, and original materials by ID;

[0294] Cache prefetching strategy: Based on score distribution and user history behavior, high-scoring content is prefetched into a distributed cache (such as a Redis cluster) in advance, with a hit rate of over 85%.

[0295] Parallel extraction pipeline: The extraction task is broken down into four parallel stages: "index lookup → cache read → disk replenishment → format conversion", supporting concurrent extraction of tens of thousands of materials;

[0296] Incremental update mechanism: Monitors the material library change log and updates the extraction index in real time to ensure that the extracted materials are consistent with the current library status. Material extraction latency is controlled within 20ms, supporting batch extraction of 50,000 materials per second, which is 10 times higher than the throughput of traditional centralized extraction architecture.

[0297] In summary, traditional methods using a globally fixed threshold (e.g., uniformly set to 0.5) cannot adapt to different modalities and domain requirements. This solution's DTO, through dynamic adjustment of multimodal weights and domain knowledge, improves screening accuracy by 22% and 19% for "beauty" and "technology" materials, respectively. Traditional iterative comparison takes minutes on lists of millions of scores; this solution's PSBE, through vectorization and pipeline optimization, achieves second-level processing and supports real-time retrieval scenarios. Traditional rule engines require manually written conflict handling logic (e.g., "time proximity priority"), making it difficult to cover complex scenarios. This solution's KG-CRN automatically resolves conflicts through knowledge graphs and user preferences, reducing the adaptation time for new conflict scenarios from one week of manual intervention to two hours of automatic learning. Traditional centralized extraction is prone to IO bottlenecks (e.g., insufficient disk bandwidth on a single node) with massive amounts of material; this solution's DSIA, through distributed indexing and cache prefetching, maintains stable extraction efficiency even with 100 million materials, reducing fault recovery time from 30 minutes to 3 seconds.

[0298] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. For example... Figure 2 As shown, an electronic device includes a memory and a processor, wherein the memory stores a computer-executable program, and the processor is configured to run the computer-executable program to perform the following steps:

[0299] The original multimedia materials are preprocessed to generate standardized preprocessed materials;

[0300] Feature extraction is performed on standardized preprocessed materials to generate material feature vectors;

[0301] The feature vectors of the source materials are indexed and constructed to generate a source material index library;

[0302] The user-input retrieval conditions are semantically parsed to generate structured retrieval instructions;

[0303] The material index is searched and processed according to the structured retrieval instructions to generate a preliminary retrieval material set;

[0304] Key information is extracted from the initial retrieved material set to generate a Mask template;

[0305] For each piece of material in the initial retrieved material set, a similarity calculation is performed in conjunction with the corresponding Mask template to generate a list of material similarity scores;

[0306] The list of material similarity scores is filtered according to the set similarity threshold to generate the final retrieved material set.

[0307] The above Figure 2In the embodiments, an exemplary explanation of the technical processing procedures for each step can be found above. Figure 1 The records.

[0308] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims. The application, device, module, or unit described in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with a certain function.

[0309] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0310] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, this application, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0311] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (this application), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0312] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0313] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0314] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, a network interface, and memory. Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0315] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0316] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0317] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, this application, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0318] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific transactions or implement specific abstract data types. This invention can also be practiced in distributed computing environments where transactions are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0319] The various embodiments in this specification are described in a progressive manner, with identical or similar parts between the embodiments referred to or substituted for each other. For the embodiments of this application, since they are basically similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0320] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims. The present application, device, module, or unit described in the above embodiments is specifically implemented by a computer chip or entity, or by a product having a certain function.

[0321] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, this application, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A material retrieval system based on mask screening, characterized in that, include: The raw material preprocessing unit is used to preprocess the raw multimedia materials to generate standardized preprocessed materials; The material feature extraction unit is used to extract features from standardized preprocessed materials to generate material feature vectors; The material indexing construction unit is used to index and construct material feature vectors to generate a material index library; The retrieval condition parsing unit is used to perform semantic parsing processing on the retrieval conditions input by the user in order to generate a structured retrieval instruction; The initial material retrieval unit is used to perform retrieval processing in the material index library according to structured retrieval instructions to generate an initial set of retrieved materials, specifically including: The instruction parsing and adaptation subunit is used to perform index adaptation and parsing processing on structured retrieval instructions to generate index query parameters; The index quickly locates sub-units, which is used to locate the index query parameters in the hash inverted index of the material index library to generate a list of candidate material IDs; The clustering range filtering subunit is used to perform range filtering on the candidate material ID list combined with the vector clustering tree of the material index library to generate a refined candidate material ID list; The metadata filtering subunit is used to perform conditional filtering on the metadata associated with the refined candidate material ID list in the material index library in order to generate a preliminary filtered material ID list. The ID mapping retrieval subunit is used to perform ID-material mapping retrieval processing on the preliminary filtered material ID list in the material index library to generate a preliminary retrieved material set; The Mask generation unit is used to extract key information from the initial retrieved material set to generate a Mask template, specifically including: The material type classification subunit is used to perform modal classification processing on the initial retrieved material set to generate a classified material set; The visual saliency detection subunit is used to perform saliency region detection processing on image / video frames in the classification material set to generate a saliency mask; The target detection and segmentation subunit is used to perform target detection and segmentation processing on the region covered by the saliency mask in order to generate a target segmentation mask; The text key region extraction sub-unit is used to perform keyword location processing on the text in the categorized material set to generate a text mask; The multimodal fusion subunit is used to perform multimodal fusion processing on the target segmentation mask and the text mask to generate a preliminary mask template; The Mask optimization subunit is used to perform morphological processing and edge optimization on the initial Mask template to generate the final Mask template; The similarity calculation unit is used to perform similarity calculation on each material in the initial retrieved material set, combined with the corresponding Mask template, to generate a list of material similarity scores; The Mask filtering unit is used to filter the list of material similarity scores according to a set similarity threshold in order to generate the final retrieved material set.

2. The material retrieval system based on mask screening according to claim 1, characterized in that, The raw material preprocessing unit is used to preprocess the raw multimedia materials to generate standardized preprocessed materials, specifically including the following steps: The format standardization subunit is used to perform uniform format conversion on the original multimedia materials to generate format-standardized materials; The resolution unification subunit is used to normalize the resolution of images / videos in format-standardized materials to generate size-standardized materials. The content denoising subunit is used to filter noise from size-standardized material to generate denoised material. The metadata extraction subunit is used to perform metadata parsing and processing on the denoised material to generate material with metadata. The data cleaning subunit is used to perform quality assessment and anomaly handling on metadata-bearing materials in order to generate standardized pre-processed materials.

3. The material retrieval system based on mask screening according to claim 1, characterized in that, The material feature extraction unit is used to extract features from standardized preprocessed materials to generate material feature vectors. Specifically, it includes the following steps: The modality separation subunit is used to perform modality classification on standardized preprocessed materials to generate modality-separated materials; The visual feature extraction subunit is used to perform multi-scale feature extraction processing on image / video frames in modality separation material to generate visual feature tensors; The text feature extraction subunit is used to perform semantic encoding on text paragraphs in the modality-separated material to generate text feature vectors; The audio feature extraction subunit is used to perform acoustic feature extraction processing on the audio tracks in the video frame sequence to generate an audio feature matrix; The multimodal fusion subunit is used to perform feature fusion processing on visual feature tensors, text feature vectors, and audio feature matrices to generate multimodal feature vectors. The dimensionality reduction and compression subunit is used to perform dimensionality compression processing on multimodal feature vectors to generate material feature vectors.

4. The material retrieval system based on mask screening according to claim 1, characterized in that, The material indexing construction unit is used to index and construct material feature vectors to generate a material index library. Specifically, it includes the following steps: The vector quantization subunit is used to perform product quantization on the feature vectors of the source material to generate quantized feature vectors; The hash coding subunit is used to perform locality-sensitive hashing on the quantized feature vector to generate a hash-coded vector; The inverted index building subunit is used to perform inverted table construction on the hash encoded vector to generate a hash inverted index; Hierarchical clustering subunits are used to perform hierarchical clustering on quantized feature vectors to generate vector clustering trees; The hybrid index fusion subunit is used to perform index fusion processing on the hash inverted index and the vector clustering tree to generate a hybrid index structure; The incremental update subunit is used to perform real-time incremental update processing on the hybrid index structure to generate a material index library.

5. The material retrieval system based on mask screening according to claim 1, characterized in that, The retrieval condition parsing unit is used to perform semantic parsing processing on the retrieval conditions input by the user to generate structured retrieval instructions, specifically including the following steps: The natural language cleaning subunit is used to perform text normalization processing on the user-input retrieval conditions in order to generate cleaned retrieval condition text. The word segmentation and part-of-speech tagging subunit is used to perform word segmentation and part-of-speech tagging on the cleaned retrieved conditional text to generate a tagged word sequence; The named entity recognition subunit is used to perform named entity recognition processing on the labeled word sequence to generate entity recognition results; The keyword extraction subunit is used to extract keywords from the labeled word sequence and entity recognition results to generate a keyword set; The semantic intent parsing subunit is used to perform semantic intent analysis on the keyword set to generate a semantic intent representation; The structured transformation subunit is used to perform structured transformation processing on the semantic intent representation to generate structured retrieval instructions.

6. The material retrieval system based on mask screening according to claim 1, characterized in that, The similarity calculation unit is used to perform similarity calculation on each piece of material in the initial retrieved material set, combined with the corresponding Mask template, to generate a list of material similarity scores. Specifically, it includes the following steps: The Mask application subunit is used to apply the corresponding Mask template to each material in the initial retrieved material set to perform region cropping processing in order to generate a Masked material set; The feature re-extraction subunit is used to perform feature re-extraction processing on the Masked material set to generate a Masked feature vector set; The query vector generation subunit is used to perform feature transformation processing on the user's query conditions to generate a query feature vector; The multimodal similarity calculation subunit is used to perform multimodal similarity calculation on the masked feature vector set and the query feature vector to generate the original similarity matrix; The weighted fusion subunit is used to perform weighted fusion processing on the original similarity matrix to generate a comprehensive similarity score; The sorting index generation sub-unit is used to sort the comprehensive similarity scores in descending order to generate a list of material similarity scores.

7. The material retrieval system based on mask screening according to claim 1, characterized in that, The Mask filtering unit is used to filter the list of similarity scores of the materials according to a set similarity threshold to generate the final retrieved material set. Specifically, it includes the following steps: The threshold parsing subunit is used to perform parameter parsing processing on the set similarity threshold to generate standardized threshold parameters; The score comparison sub-unit is used to compare each score in the material similarity score list with the standardized threshold parameter to generate a preliminary screening tag list; The conflict handling subunit is used to resolve conflicts in special cases in the initial screening mark list to generate a conflict-free screening mark list. The material extraction subunit is used to extract materials marked as "retain" from the conflict-free filter list in the initial retrieved material set to generate the final retrieved material set.

Images