A multimedia hierarchical feature indexing method and system

Abstract

The application discloses a multimedia hierarchical feature index method and system based on multi-source knowledge base retrieval, relates to the field of information retrieval, and comprises the following steps: for each multimedia object in a multimedia data set, K kinds of modal feature vectors are extracted, and an independent basic hierarchical index tree is constructed; a cross-modal contradiction matrix is constructed; a contradiction discriminator is called to calculate a contradiction score; when a multimodal query containing multiple query modal feature vectors is received, retrieval is performed in the corresponding basic hierarchical index tree; the retrieval paths are compared, and path similarity is calculated; if the path similarity is greater than or equal to a preset path similarity threshold, the result is returned after fusion; if the path similarity is lower than the preset threshold, cross-modal retrieval divergence is determined; an arbitration module is activated; a subnode is selected for in-depth retrieval, and a backtracking stack is reserved; if the quality is lower than a preset quality threshold, backtracking is performed; a suboptimal path is selected for retrieval; and finally, a retrieval result is obtained. The application solves the problems of low hierarchical retrieval efficiency and low retrieval accuracy in the prior art.

Description

Technical Field

[0001] This invention relates to the field of information retrieval, and specifically to a multimedia hierarchical feature indexing method and system. Background Technology

[0002] Existing multimodal retrieval methods typically fuse feature vectors from different modalities into a single space or construct a joint index structure, using a single distance metric for similarity matching. The feature space distributions of different modalities are heterogeneous, and forced fusion or joint indexing introduces metric bias, leading to reduced retrieval accuracy. Furthermore, the inability to effectively perceive semantic contradictions between different modalities within the same multimedia object results in severely distorted fusion retrieval results. Simultaneously, the lack of awareness of cross-modal contradictions leads to low retrieval efficiency. Summary of the Invention

[0003] This application provides a multimedia hierarchical feature indexing method and system, which addresses the problems of low efficiency and low accuracy in hierarchical retrieval in existing technologies.

[0004] In view of the above problems, this application provides a multimedia hierarchical feature indexing method and system.

[0005] Firstly, this application provides a multimedia hierarchical feature indexing method, the method comprising: For each multimedia object in the multimedia dataset, extract K modal feature vectors and construct an independent basic hierarchical index tree for each modal feature vector; Construct a cross-modal contradiction matrix within each leaf node; When inserting a multimedia object, the contradiction discriminator is called to calculate the contradiction score of all modal feature vector pairs and fill it into the cross-modal contradiction matrix; When a multimodal query containing feature vectors of multiple query modalities is received, it is retrieved from the corresponding basic hierarchical index tree. Compare the retrieval paths of different basic hierarchical index trees and calculate the path similarity; When the path similarity is greater than or equal to the preset path similarity threshold, the retrieval results of each basic hierarchical index tree are merged and returned. When the value is below a preset threshold, a cross-modal retrieval divergence is determined; the arbitration module is activated to output the arbitration path weight vector; the child node is selected for in-depth retrieval based on the arbitration path weight vector, and the backtracking stack is retained. If the quality of the retrieval result is lower than the preset quality threshold, the divergence point is backtracked to select the suboptimal path to continue the retrieval until the final retrieval result is obtained.

[0006] Secondly, the present invention provides a multimedia hierarchical feature indexing system, the system comprising: The index tree building module is used to extract K modal feature vectors for each multimedia object in the multimedia dataset and build an independent basic hierarchical index tree for each modal feature vector; The contradiction matrix construction module is used to construct a cross-modal contradiction matrix within each leaf node; The contradiction score calculation module is used to call the contradiction discriminator to calculate the contradiction score of all modal feature vector pairs when inserting multimedia objects, and fill it into the cross-modal contradiction matrix; The multimodal query module is used to retrieve data from the corresponding basic hierarchical index tree when a multimodal query containing feature vectors of multiple query modes is received. The path similarity calculation module is used to compare the retrieval paths of different basic hierarchical index trees and calculate the path similarity. The retrieval result fusion module is used to fuse the retrieval results of each basic hierarchical index tree and return them when the path similarity is greater than or equal to a preset path similarity threshold. The retrieval result acquisition module is used to determine that a cross-modal retrieval divergence has occurred when the result is below a preset threshold; the arbitration module is activated to output an arbitration path weight vector; the sub-nodes are selected for in-depth retrieval based on the arbitration path weight vector, and a backtracking stack is retained. If the quality of the retrieval result is lower than a preset quality threshold, the system backtracks to the divergence point to select a suboptimal path to continue retrieval until the final retrieval result is obtained.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: First, this application extracts K modal feature vectors for each object in the multimedia dataset and constructs independent basic hierarchical index trees for each, avoiding metric bias and information loss, improving indexing and retrieval efficiency, and preserving the original semantic information for subsequent cross-modal conflict detection. Second, a cross-modal conflict matrix is ​​constructed in each leaf node, and a conflict discriminator is called to fill in the conflict score during insertion, quantifying the semantic inconsistency between modalities into indexable structured data, ensuring the real-time performance of online queries. Third, when a multimodal query is received, it independently searches in the corresponding basic hierarchical index trees and compares the retrieval paths and calculation paths of different trees. Similarity is improved to enhance retrieval efficiency and interpretability. Furthermore, when the path similarity is not lower than a preset threshold, the retrieval results of each tree are merged and returned, providing a fast response channel for conflict-free queries. Further, when the path similarity is lower than the threshold and cross-modal retrieval divergence is determined, the arbitration module is activated to output the arbitration path weight vector. Based on this, the child node is selected for in-depth retrieval and the backtracking stack is retained. When the quality of the retrieval result does not meet the preset quality threshold, the suboptimal path is selected to continue iterative retrieval until the final result is obtained. This realizes dynamic adjustment of the retrieval path, effectively avoids the accumulation of erroneous paths, and improves the retrieval robustness and result quality in complex and contradictory scenarios. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a multimedia hierarchical feature indexing method provided in this application; Figure 2 This is a schematic diagram of the structure of a multimedia hierarchical feature indexing system provided in this application.

[0009] In the attached diagram, the components represented by each number are as follows: The module includes: index tree construction module 11, contradiction matrix construction module 12, contradiction score calculation module 13, multimodal query module 14, path similarity calculation module 15, retrieval result fusion module 16, and retrieval result acquisition module 17. Detailed Implementation

[0010] This application provides a multimedia hierarchical feature indexing method, which specifically addresses the problems of low efficiency and low accuracy in hierarchical retrieval in existing technologies.

[0011] The present invention will now be described in detail with reference to the accompanying drawings.

[0012] Example 1, as Figure 1 As shown, this application provides a multimedia hierarchical feature indexing method, the method comprising: S10: For each multimedia object in the multimedia dataset, extract K modal feature vectors and construct an independent basic hierarchical index tree for each modal feature vector; In this embodiment, a multimedia object is a digital content unit containing at least two different types of information streams, including video clips, image and text pairs, and audio and video streams; a modal feature vector is a numerical vector representation extracted from a certain information channel of the multimedia object that can characterize the semantic content of that channel; and a basic hierarchical index tree is a tree-shaped data structure constructed for a single modal feature vector space, used to efficiently perform nearest neighbor or range retrieval.

[0013] Specifically, using feature extractors pre-trained for different modalities, K modal feature vectors are extracted from multimedia objects, where K is an integer greater than or equal to 2, including visual feature vectors, audio feature vectors, and text feature vectors. Subsequently, a dedicated basic hierarchical index tree is independently constructed for each modal feature vector.

[0014] For each modality of a multimedia object, a separate tree-structured index is constructed. The root node of each tree represents the entire feature space, intermediate nodes recursively divide the feature space into several sub-regions through clustering or spatial partitioning, and leaf nodes store pointers to specific multimedia object identifiers. For example, a visual index tree is constructed for the visual modality; and another audio index tree is constructed for the audio modality. The index trees for different modalities are structurally independent, but they collaborate during retrieval through path divergence detection and arbitration modules.

[0015] The basic hierarchical index tree can be implemented using existing algorithms such as hierarchical clustering trees, approximate nearest neighbor trees, or navigable small world graphs.

[0016] Step S10 in the method provided in this embodiment of the invention includes: A pre-trained feature extractor, the feature extractor comprising a convolutional neural network for extracting visual features, an audio embedding network for extracting audio features, and a language model for extracting text features; Based on the feature extractor, for each multimedia object, the visual features of the visual frame sequence are extracted as the first modality feature vector. Based on the feature extractor, the audio embedding features of the audio track are extracted as the second modality feature vector; Based on the feature extractor, the text embedding features of the associated text are extracted as the third modality feature vector; The first modal feature vector, the second modal feature vector, and the third modal feature vector are unified into the K modal feature vectors; Wherein, K is an integer greater than or equal to 2, and the modal feature vector includes at least two of the following: visual feature vector, audio feature vector, and text feature vector; the leaf node of each of the basic hierarchical index trees stores a pointer to the corresponding multimedia object identifier.

[0017] In this embodiment, firstly, the audio embedding network is a neural network specifically designed to process audio signals, which can extract semantic embeddings of audio content using convolutional or recurrent structures; the language model is a pre-trained model based on the Transformer architecture, which can encode text sentences or phrases into context-aware vector representations and extract semantic features of text modalities.

[0018] Pre-load and instantiate pre-trained feature extractors for different modalities. A convolutional neural network pre-trained on ImageNet is selected as the visual feature extractor, with an output layer dimension of 2048; a pre-trained convolutional or recurrent structure is selected as the audio embedding network, outputting 128-dimensional audio embedding vectors; and a pre-trained language model is selected as the text feature extractor, outputting 768-dimensional text embedding vectors. The extractor parameters remain unchanged during subsequent indexing to ensure the stability and reproducibility of the feature representations. The pre-training step is typically completed during system initialization, and the feature extractors are loaded into memory or GPU memory for subsequent use on each multimedia object.

[0019] For example, the visual feature extractor uses a convolutional neural network to construct an initial convolutional layer, a max pooling layer, a pooling layer, and a fully connected output layer.

[0020] The initial convolutional layer consists of a 7×7 convolutional kernel with a stride of 2 and 64 output channels, followed by batch normalization and ReLU activation. The max pooling layer is 3×3 in size with a stride of 2. Then, four residual block groups are sequentially connected, containing 3, 4, 6, and 3 residual blocks respectively, with output channels of 256, 512, 1024, and 2048. Each residual block consists of two 1×1 convolutions and one 3×3 convolution, using batch normalization and ReLU activation. Finally, a global average pooling layer and a fully connected output layer are connected, with an output dimension of 2048 and no activation function.

[0021] The model was trained with an initial learning rate of 0.1, using a momentum SGD optimizer, a cross-entropy loss function, a batch size of 256, and 90 iterations. The learning rate was reduced to 0.1 at 30 and 60 epochs. Forward propagation was used with a 224×224 image as input, extracting features through linear transformations and non-linear activations at each layer. Backpropagation was used to calculate and update the gradient of the loss with respect to the weights of each layer. The training stopped when the loss on the validation set did not decrease for five consecutive epochs. For example, given an image of a cat, the network output a 2048-dimensional feature vector.

[0022] The audio embedding network is constructed using a convolutional or recurrent structure. The network contains six convolutional layers: the first layer has a 3×3 kernel, 64 output channels, a stride of 1, padding of 1, and ReLU activation; the second layer is the same as the first layer; then, the max pooling size is 2×2 with a stride of 2; the third and fourth layers both have a 3×3 kernel, 128 output channels, and ReLU activation; then, max pooling is performed again; the fifth and sixth layers have a 3×3 kernel, 256 output channels, and ReLU activation; then, max pooling is performed a third time; then, it is flattened into a fully connected layer with an output of 4096, using the ReLU activation function; finally, the output layer is a 128-dimensional embedding with no activation function.

[0023] Training used an initial learning rate of 0.001, the Adam optimizer, and a contrastive loss function to ensure that embeddings of similar audio classes are close together and those of dissimilar classes are far apart. The batch size was 128, and the training run was 50 iterations. Forward propagation used the Mel-spectrum as input to extract audio event features; backpropagation used gradient descent to update parameters. Early stopping was determined when the average precision on the validation set did not improve for three consecutive iterations. The validation set was the AudioSet validation set, with an mAP of 0.85 or higher.

[0024] The text language model is built upon a Sentence-BERT architecture with a 12-layer Transformer, 768-dimensional hidden layers, 12 attention heads, a 3072-dimensional feedforward network, and GELU activation. Pooling layers are added to the BERT architecture to output 768-dimensional sentence vectors.

[0025] Training was performed using natural language inference and semantic text similarity datasets, with an initial learning rate of 2e. -5 The AdamW optimizer is used with a weight decay of 0.01 and a triplet loss function. The batch size is 32, and the iteration is performed for 3 rounds. Forward propagation transforms the input text into word segments, which are then processed through an embedding layer and multiple Transformer encodings to obtain a context representation, followed by pooling to obtain sentence vectors. Backpropagation calculates and updates the gradients. The iteration stops when the Spearman correlation coefficient on the validation set does not increase for two consecutive rounds and reaches above 0.85.

[0026] Secondly, for video-based multimedia objects, a visual frame sequence refers to a set of keyframes or sampled frames arranged in chronological order. Uniform sampling or adaptive sampling based on scene transition detection is typically used to reduce computational cost while maintaining content representativeness. Visual features are depth feature vectors extracted from each frame image via a CNN. For objects containing multiple frames, features from multiple frames can be aggregated into a global visual feature vector using average pooling, max pooling, or attention-weighted methods.

[0027] The first modality feature vector is defined as the visual modality among K modalities. Its corresponding feature vector has a fixed dimension and is used for subsequent index construction and contradiction calculation.

[0028] Specifically, for each multimedia object in the multimedia dataset, this method calls the aforementioned pre-trained convolutional neural network feature extractor to perform visual feature extraction operations.

[0029] First, visual frame sequences are extracted from the multimedia object. For example, for a 10-second video at 30fps, 10 keyframes are obtained using a uniform sampling strategy of 1 frame per second. Then, each frame is preprocessed (scaled and normalized) and input into a convolutional neural network model. The output of the penultimate layer is extracted, resulting in 10 2048-dimensional feature vectors. These feature vectors are then aggregated along the temporal dimension, for example, using average pooling to calculate the arithmetic mean of each dimension, ultimately yielding a 2048-dimensional global visual feature vector. If the multimedia object itself is a static image, single-frame features can be extracted directly; the feature vector will serve as the unique index key for that object in the visual hierarchical index tree.

[0030] Furthermore, an audio track is a continuous sound signal contained within a multimedia object, typically stored in pulse code modulation format. For objects without audio, an all-zero vector can be generated or the object can be directly marked as empty.

[0031] Audio embedding features are dense vectors extracted from the original waveform or spectrogram by an audio embedding network. They are usually low in dimension but have rich acoustic semantics and can represent timbre, pitch, ambient sound, speech content, etc. The second modality feature vector defines the audio modality as the second modality, and its dimension is consistent with the output dimension of the audio embedding network.

[0032] Specifically, for the same multimedia object, after extracting visual features, the embedding features of the audio track are extracted. First, the audio stream is demultiplexed from the multimedia object container and decoded into raw PCM waveform data. Then, the waveform undergoes preprocessing such as pre-emphasis, framing, and windowing to generate a Mel spectrogram or directly input the raw waveform into the audio embedding network. A pre-trained model receives a 96×64 Mel spectrogram as input and outputs a 128-dimensional embedding vector. The audio is segmented into several segments, each segment generating a vector. Average pooling is performed on all segment vectors to obtain a 128-dimensional global audio embedding feature vector, which serves as the second modality feature vector for the multimedia object.

[0033] Furthermore, associated text is text metadata associated with multimedia objects in any form, including but not limited to titles, tags, descriptions, subtitles, user comments, text within the screen recognized by OCR, and speech-to-text text recognized by ASR; text embedding features are natural language text mapped into fixed-length semantic vectors through a pre-trained language model; third-modality feature vectors define the text modality as the third modality, and their dimension depends on the selected language model.

[0034] Specifically, for the same multimedia object, the embedding features of its associated text are extracted. First, all available associated text sources for the multimedia object are collected. For example, for online videos, the title, uploader tags, video description, automatically generated subtitles, and text transcribed from the audio via speech recognition are obtained. The text fragments are then concatenated and cleaned and truncated as necessary. Next, the processed text is input into a pre-trained language model. For example, using the Sentence-BERT model, a sentence is input, and the model outputs a semantic vector. If multiple discontinuous text fragments exist, each fragment can be encoded independently and averaged. The resulting fixed-dimensional vector is the third-modality feature vector of the multimedia object.

[0035] Furthermore, after extracting the feature vectors for the first, second, and third modalities respectively, a unique identifier is assigned to the multimedia object. Then, multimodal feature records are created and stored in association to obtain K types of modal feature vectors containing visual feature vectors, audio feature vectors, and text feature vectors.

[0036] Where K is an integer greater than or equal to 2, and the modal feature vector includes at least two of the following: visual feature vector, audio feature vector, and text feature vector; the leaf nodes of each basic hierarchical index tree store pointers to the corresponding multimedia object identifiers.

[0037] A pointer is a reference to the identifier of a corresponding multimedia object. When a search hits a leaf node, the pointer is used to locate the storage location of the multimedia object, retrieve the complete multimedia data content, and return it to the user. The leaf node storage consists of a list or set maintained internally by each leaf node of the basic hierarchical index tree. The elements in this list are pointers to multimedia object identifiers. When a search reaches a leaf node, these pointers are returned.

[0038] Specifically, when constructing each basic hierarchical index tree, it is stipulated that each leaf node does not directly store the visual feature vector of the multimedia object, but instead stores a list of pointers. When a multimedia object is inserted into the visual index tree, the visual feature vector determines the leaf node it should fall into. After determining the target leaf node, a pointer to the unique identifier of the multimedia object is added to the pointer list of that leaf node. Similarly, for the audio index tree and the text index tree, the leaf nodes also store pointers to the same objects, but the same pointer will appear in different leaf nodes of different modal index trees, so that each modal index tree independently maintains its pointer list without interference.

[0039] In this embodiment, by using dedicated feature extractors pre-trained for each modality, high-quality extraction of visual, audio, and text feature vectors is ensured, avoiding semantic distortion caused by forced fusion. By constructing a cross-modal inconsistency matrix within each leaf node and calling the inconsistency discriminator to fill in the inconsistency score during insertion, the semantic inconsistency between modalities is quantified into indexable structured information, providing a direct basis for subsequent arbitration. By comparing the retrieval path similarity of different basic hierarchical index trees, cross-modal retrieval disagreement is determined with extremely low computational overhead, solving the retrieval failure problem under modal inconsistency and improving the robustness of the system. Modal confidence is used to avoid interference from noisy modalities on the retrieval results.

[0040] S20: Construct a cross-modal contradiction matrix within each leaf node; In this embodiment, a leaf node is the smallest structural unit in the basic hierarchical index tree that does not branch downwards and directly stores data pointers. A leaf node typically corresponds to a locally dense region in the feature space, containing several multimedia objects with similar feature vectors. The cross-modal inconsistency matrix is ​​a K×K symmetric data structure stored in each leaf node, whose matrix elements quantify the degree of semantic inconsistency between different modal feature vector pairs of the same multimedia object. The rows and columns of the matrix correspond to modal types.

[0041] Specifically, after the initial construction of all basic hierarchical index trees is completed, a logically related cross-modal conflict matrix is ​​created within each leaf node of each index tree for each multimedia object. When a multimedia object is inserted into a leaf node of a modality index tree through its feature vector, a K×K conflict matrix is ​​reserved or constructed within the storage area of ​​that leaf node A. The matrix stores a conflict score matrix to be filled. The dimension of the matrix is ​​equal to the total number of modalities K. For example, if K=3, the matrix is ​​a 3×3 symmetric matrix, with diagonal elements being meaningless or always 0, and off-diagonal elements used to record the degree of conflict between corresponding modality pairs.

[0042] Step S20 of the method provided in this embodiment of the invention, after constructing the cross-modal contradiction matrix within each leaf node, further includes: The cross-modal contradiction matrix is ​​a K×K symmetric matrix, where the element in the i-th row and j-th column represents the degree of contradiction between the i-th modal feature vector and the j-th modal feature vector of the multimedia object; For each pair of modal types, a discriminant is pre-trained. The discriminant is used to take two modal feature vectors of any multimedia object as input and output a discriminant score between 0 and 1. The higher the discriminant score, the more inconsistent the two modal feature vectors are. In each intermediate node, maintain the mean matrix of the cross-modal conflict matrix of all multimedia objects within the node, as well as the variance of the conflict score within the node; When the variance exceeds a preset variance threshold, it is determined that there are semantically inconsistent multimedia objects within the node, triggering the node splitting rule for contradiction perception.

[0043] In this embodiment, the cross-modal contradiction matrix is ​​first determined to be a K×K symmetric matrix. Assume the multimedia object has K modalities; for example, when K=3, modality index 1 is visual, 2 is audio, and 3 is text. The diagonal elements of the matrix, having no actual semantic meaning, can be always set to 0 or ignored.

[0044] Because the contradictions are symmetrical, only the upper or lower triangular portion needs to be maintained during actual storage. The specific value of each element is calculated by the contradiction discriminator in subsequent steps. The matrix is ​​created and populated when an object is inserted, and then stored associatively in the leaf nodes of the underlying hierarchical index tree to which the object belongs. When the object is moved to a different leaf node, the matrix migrates along with the object pointer.

[0045] Secondly, the contradiction discriminator is a binary classification or regression model, usually implemented as a deep neural network.

[0046] Specifically, the contradiction discriminator needs to be trained before inserting multimedia objects and calculating contradiction scores.

[0047] Specifically, for each pair of modalities, a large number of multimodal data samples labeled with the degree of contradiction are collected. Each sample contains two modal feature vectors of the same multimedia object, and a true contradiction score, where 0 indicates consistency and 1 indicates contradiction. A neural network is trained, whose input is the two modal feature vectors of a multimedia object, such as visual feature vectors and text feature vectors, and whose output is a contradiction score between 0 and 1. The closer the score is to 0, the more consistent the two modalities are semantically; the closer the score is to 1, the more contradictory the two modalities are semantically.

[0048] The loss function uses mean squared error or binary cross-entropy. After training, the discriminator's parameters are fixed. During subsequent online insertion, when calculating the visual-audio inconsistency score for a multimedia object, the visual and audio feature vectors of the object are obtained and input into the visual-audio inconsistency discriminator. After forward propagation, the output score is obtained. The score is directly used as the element value at the corresponding position in the cross-modal inconsistency matrix. Similarly, for other modal pairs, their respective discriminators are invoked.

[0049] Furthermore, intermediate nodes are located at the middle level of the basic hierarchical index tree, containing several child nodes. An intermediate node represents a region in the feature space, containing a collection of all multimedia objects within that region.

[0050] The mean matrix is ​​a K×K matrix obtained by taking the element-wise arithmetic mean of the cross-modal conflict matrices of all multimedia objects within a node. For example, if there are N objects within a node, and each object has a conflict matrix, then a certain value of the mean matrix is ​​the sum divided by the number of objects in the node. The mean matrix reflects the average degree of conflict in the region represented by the node.

[0051] During the construction and maintenance of each basic hierarchical index tree, the mean matrix of the cross-modal conflict matrix of all multimedia objects within the node and the variance of the node conflict score are stored at each intermediate node.

[0052] Preferably, independent variances are calculated for each modality pair, such as the variance of visual-audio inconsistencies, the variance of visual-text inconsistencies, and the variance of audio-text inconsistencies.

[0053] When a new multimedia object is inserted into the index tree, the contradiction matrix is ​​calculated. The object is assigned to a leaf node, and all ancestor intermediate nodes are updated upwards along the path: the contradiction matrix of the new object is added to the calculation of the mean matrix, the counts and sums are updated, and the variance is also updated. When an object is deleted from a node, the statistics are also updated.

[0054] Ultimately, the preset variance threshold is a hyperparameter set by the system according to application requirements, such as 0.1 or 0.2, used to determine whether the dispersion of the contradiction scores within a node has reached a level that affects retrieval efficiency or accuracy.

[0055] Semantically inconsistent multimedia objects are those within a node that differ significantly from other objects in their cross-modal contradictory characteristics. For example, if most objects in a node have consistent audio and video, but a few objects have contradictory audio and video, then these few objects are considered semantically inconsistent. The node splitting rule for contradiction perception is different from the traditional rule based on feature distance. It considers both feature space distance and contradiction matrix similarity, and groups objects with similar contradictory characteristics into the same child node.

[0056] Specifically, during the dynamic maintenance of the index tree, for each intermediate node, the maintained variance of the contradiction score is compared with a preset variance threshold. When the variance exceeds the preset threshold, it indicates that the multimedia objects within the node have significantly differentiated in terms of cross-modal contradiction characteristics. Therefore, a split operation is performed on the node, re-dividing its multimedia objects into multiple new child nodes, reducing the contradiction variance within each new child node to below the threshold. Simultaneously, the contradiction-aware node splitting rule is triggered.

[0057] For example, the variance of the inconsistency score within a node is as high as 0.40, far exceeding the preset threshold of 0.10. This triggers a splitting rule, assigning the nature documentary to child node A and the conflicting video to child node B.

[0058] In step S20 of the method provided in this embodiment of the invention, the node splitting rule for conflict perception includes: When splitting a node, calculate the contradiction matrix similarity between every two multimedia objects within the node; The similarity of the contradiction matrix is ​​used as a weighting factor in the distance metric, and together with the feature distance between the main modality feature vectors, it participates in the clustering partitioning. The similarity of the contradiction matrix is ​​used as a partitioning criterion alongside the feature distance of the main modality feature vector, and multimedia objects with high contradiction matrix similarity are preferentially partitioned into the same child node.

[0059] In this embodiment, firstly, when splitting a node, the similarity of the contradiction matrix between every two multimedia objects within the node is calculated. Splitting a node occurs during the index tree maintenance process when the statistical characteristics of an intermediate node exceed a preset threshold, causing all multimedia objects under that node to be reassigned to two or more new child nodes. After the split, the original node becomes an internal node, and the newly generated child nodes become its child nodes.

[0060] Specifically, the similarity of the contradiction matrices between all pairs of multimedia objects within a node is calculated. Suppose the node to be split contains N multimedia objects, each of which has its cross-modal contradiction matrix stored. For each pair, the two contradiction matrices are flattened into K²-dimensional vectors, and then the cosine similarity between the two vectors is calculated. The cosine similarity is the ratio of the dot product of the two vectors to the product of their moduli, typically between [0,1]. A larger value indicates a more similar contradiction patterns between the two objects.

[0061] Secondly, distance metrics are functions used in clustering algorithms to measure the degree of difference between two samples. Traditional distance metrics are based solely on the Euclidean or Manhattan distance of feature vectors; weighting factors are multiplicative or additive weight coefficients that transform the similarity of the contradiction matrix through a certain transformation, and are used to adjust the contribution ratio of feature distance to the final clustering distance.

[0062] The dominant modality feature vector is the feature vector of a multimedia object within the modality corresponding to the underlying hierarchical index tree. For example, in the splitting of a visual index tree, the dominant modality is the visual modality, and the dominant modality feature vector is the visual feature vector. Clustering partitioning divides the N objects within a node into k subsets, and each subset becomes a new child node.

[0063] After obtaining the contradiction matrix similarity between every two multimedia objects within the node to be split, a clustering operation is performed. The contradiction matrix similarity is used as a weighting factor, and together with the feature distance between the principal modality feature vectors, it determines the final clustering distance.

[0064] Specifically, for any two objects, the composite distance D is defined as α×d1+β×d2, where d1 and d2 are the Euclidean or cosine distance between the principal modal feature vectors, and α and β are preset weight coefficients. The weight coefficients can be set as the historical average or mode of the preset weight coefficients corresponding to each modal feature vector in the historical data, for example, α=0.6, β=0.4, which is used to balance the importance of the feature space and the contradiction space.

[0065] Using k-means or hierarchical clustering algorithms, with composite distance D as the distance metric, all objects within a node are divided into a predetermined number of clusters. During the clustering process, cluster centers are calculated separately in the feature space and the contradiction space, iterating until convergence.

[0066] Ultimately, the similarity of the contradiction matrix is ​​used as a partitioning criterion alongside the feature distance of the main modality feature vector, prioritizing the partitioning of multimedia objects with high contradiction matrix similarity into the same child node.

[0067] Specifically, the mainstream approach to traditional multimodal hierarchical indexing is to use the feature distance of the main modality feature vector as the basis for partitioning. For example, an index tree is constructed based solely on visual feature clustering, with other modality features attached as supplementary information to the leaf nodes.

[0068] Pre-clustering is performed on all objects within a node based on the contradiction matrix similarity to form an initial coarse-grained group, ensuring that multimedia object pairs with contradiction similarity higher than a certain threshold are placed in the same candidate group. Then, while maintaining hard constraints, fine-grained clustering is performed based on the distance between the main modality feature vectors to finally determine the child node partitioning.

[0069] In this embodiment, the contradiction matrix similarity between every two multimedia objects within a node is used as a partitioning criterion alongside the main modality feature distance, aligning the index tree structure with contradiction patterns. Objects with high contradiction matrix similarity are preferentially grouped into the same child node, effectively reducing the variance of contradiction scores within each child node after splitting, providing a more reliable basis for the retrieval process. After splitting, objects with different contradiction patterns are separated into different index branches. When a multimodal query is received, the arbitration module can quickly identify the branch matching the query intent, reducing the number of nodes that need to be accessed and lowering retrieval latency. Finally, the contradiction matrix similarity is used as a weighting factor along with the feature distance to participate in clustering, adjusting the weight coefficients according to the application scenario to enhance the dynamic adaptive capability of the index, enabling the index tree to continuously self-optimize and maintain long-term retrieval performance.

[0070] S30: When inserting a multimedia object, call the contradiction discriminator to calculate the contradiction score of all modal feature vector pairs and fill it into the cross-modal contradiction matrix; In this embodiment, the contradiction discriminator is a pre-trained discrimination model, typically implemented as a binary classification or regression neural network. Its input consists of feature vectors from any two modalities of the same multimedia object, and its output is a scalar value ranging from [0,1], i.e., the contradiction score. A higher score indicates a greater semantic inconsistency or conflict between the two modalities.

[0071] The contradiction score is a quantitative semantic distance metric, distinct from the Euclidean distance in the feature space, measuring the degree of semantic opposition or unusual consistency between modalities; filling involves assigning the calculated contradiction score to the corresponding symmetric position in the cross-modal contradiction matrix.

[0072] Specifically, when a new multimedia object needs to be inserted into the indexing system, its K modal feature vectors are first extracted. Then, contradiction calculation and insertion are performed. Specifically, all modal feature vector pairs of the multimedia object are traversed. For each modal type pair, the modal feature vectors corresponding to the multimedia object are input into a pre-trained contradiction discriminator for that modal pair. After forward propagation calculation, the discriminator outputs a contradiction score. After calculating all modal pairs, the cross-modal contradiction matrix of the multimedia object is constructed. Afterward, each modal feature vector is inserted into its corresponding basic hierarchical index tree, and a pointer or copy of the contradiction matrix is ​​associated and stored in all leaf nodes of the object.

[0073] In this embodiment, the contradiction score is read during the query process, avoiding the invocation of a complex discriminator and enabling high-concurrency multimodal retrieval; the offline calculation of the contradiction score is optimized by using batch data, improving the computational efficiency of the discriminator and providing key input features for the subsequent arbitration module.

[0074] S40: When a multimodal query containing feature vectors of multiple query modalities is received, the corresponding basic hierarchical index tree is searched respectively; In this embodiment, a multimodal query is a query example in which a user-submitted query request contains at least two modalities; retrieval is the process of performing a nearest neighbor search or range search in the index tree, usually starting from the root node, recursively selecting the child node path most similar to the query feature vector, eventually reaching the leaf node, and returning the multimedia object pointer stored therein as a candidate result.

[0075] Specifically, upon receiving a multimodal query request, the feature vectors of each query modality are first parsed out. Each query modality is then directed to its dedicated underlying hierarchical index tree—that is, input into a pre-built visual underlying hierarchical index tree—and independently performs a standard retrieval process from root to leaf, resulting in a set of candidate leaf nodes and their corresponding retrieval paths. The retrieval processes are completely parallel and do not interfere with each other, each leveraging the efficiency of its own single-modal index tree to quickly narrow down the search scope. Each index tree returns a candidate result that is a pointer to a multimedia object, along with path information.

[0076] In this embodiment, an index structure optimized for modality is used to achieve theoretically optimal retrieval complexity and improve retrieval efficiency; fusion retrieval loses path information, providing original data for cross-modal path comparison; the degree of path difference reflects the consistency of modality distribution in the index space, providing a basis for judgment in subsequent steps.

[0077] S50: Compare the retrieval paths of different basic hierarchical index trees and calculate the path similarity; In this embodiment of the application, the retrieval path is a sequence of nodes that are passed sequentially from the root node in the tree index structure according to the node selection strategy until one or more leaf nodes are reached; the path is an ordered sequence of nodes; the path similarity is a measure of the degree of overlap or structural consistency between two or more retrieval paths, and the calculation method can be the ratio of the longest common subsequence length of the path node sequence to the maximum path length, or a more complex variant of tree edit distance.

[0078] Specifically, after obtaining the path information generated by independent retrieval of each basic hierarchical index tree, the retrieval path is obtained for each pair of query modalities. Then, the path similarity between the two node sequences is calculated, which is the length of the longest common subsequence of the two sequences. The path similarity is then obtained by dividing the length of the longest common subsequence by the maximum length of the two original sequences.

[0079] Step S50 in the method provided in this embodiment of the invention includes: Record the node sequence from the root node to the candidate leaf node in the corresponding basic hierarchical index tree for each modality feature vector; Calculate the length of the longest common subsequence between node sequences of any two modalities; The path similarity is obtained by dividing the length of the longest common subsequence by the maximum length of the two node sequences.

[0080] In this embodiment, firstly, when a multimodal query is received, for each modal feature vector contained in the query, a retrieval is performed in the corresponding basic hierarchical index tree. During the retrieval process, the complete node sequence traversed from the root node to the candidate leaf node is simultaneously recorded.

[0081] Taking a visual index tree as an example, retrieval begins at the root node. The distance between the query vector and the center of the regions of each child node under the root node is calculated. The child node with the smallest distance is selected as the next hop, and its identifier is appended to the node sequence list. This process is repeated, passing through intermediate nodes sequentially until a candidate leaf node is reached. At this point, a node sequence is recorded. If the retrieval strategy allows multiple candidate paths, a node sequence can be recorded for each path separately. For other modalities, the same recording operation is performed in parallel in their respective index trees to obtain their respective node sequences. The length of the sequence may vary depending on the depth of the tree, but all sequences begin at the root node of their respective tree and end at a candidate leaf node.

[0082] Secondly, the longest common subsequence is the longest sequence among two given sequences that appear in the same order but are not required to be consecutive; the node sequence is an ordered list of node identifiers obtained by recording.

[0083] After obtaining the node sequences of each modality, for each pair of modalities that need to be compared, calculate the length of the longest common subsequence between their node sequences.

[0084] Specifically, the LCS algorithm is invoked, taking each modality pair as input and outputting an integer representing the length of the longest common node sequence in both sequences, maintaining relative order. When calculating the LCS, since the starting nodes V0 and V1 appear in both sequences in the same order, while V2 is absent in the audio sequence and A2 and A3 are absent in the visual sequence, the LCS is [V0, V1] with a length of 2. The node identifiers in the LCS are completely identical; a match is considered complete only when nodes in both sequences point to the same physical node in the index tree. Because the index trees for different modalities are constructed independently, node identifiers are usually different. However, if the two index trees were constructed using the same spatial partitioning strategy and have consistent data distribution, it is possible for node identifiers to be identical or semantically corresponding.

[0085] Finally, after obtaining the length of the longest common subsequence (LCS) of the two modal node sequences, the lengths of the two node sequences are obtained separately, and their maximum values ​​are calculated. Then, the path similarity is calculated: Path Similarity = Longest Common Subsequence Length / Maximum Value. For example, if the visual sequence length is 6, the audio sequence length is 5, and the LCS length is 4, then the path similarity = 4 / 6 ≈ 0.67. If the path similarity = 1, it indicates that the paths are completely identical. If the path similarity is very low, it indicates that the paths are severely divergent.

[0086] In this embodiment, the length of the longest common subsequence and the normalized similarity are calculated by recording the node sequences naturally generated during the retrieval process to meet the real-time requirements; a complete record query is performed through the node sequences, and the LCS length can reflect the degree of alignment from coarse-grained to fine-grained, which has a stronger discriminative ability than comparing only leaf nodes; the path similarity calculation eliminates the influence of the difference in depth of different modal index trees and avoids the waste of calculation on incorrect paths.

[0087] S60: When the path similarity is greater than or equal to the preset path similarity threshold, the retrieval results of each basic hierarchical index tree are merged and returned; In this embodiment of the application, the preset path similarity threshold is a hyperparameter pre-set by the system or the user. This threshold is used to determine whether the retrieval paths of each modality are semantically consistent enough that it can be considered that there is no cross-modal conflict.

[0088] Fusion is the process of combining a set of candidate results obtained from multiple independent modal searches into a unified list of results. Fusion methods can include weighted averaging, voting, or ranking-based mutual information maximization, among others.

[0089] Specifically, after calculating the path similarity between the retrieval paths of each modality, the path similarity is compared with a preset threshold. When the path similarity is greater than or equal to the threshold, it indicates that the retrieval paths of each modality highly overlap in the index tree, suggesting that the multimodal features of the query have good semantic consistency. In this case, there is no need for complex conflict arbitration, and the process proceeds directly to the result fusion stage.

[0090] This involves retrieving the candidate multimedia object sets returned by each basic hierarchical index tree and then performing a fusion operation. The objects are then sorted according to the weighted sum of their distance scores across modalities. Finally, the fused result is returned to the user as the query response.

[0091] In this embodiment, the consistency of query modalities is checked to avoid time-consuming operations and improve response speed; the diversity and accuracy of fusion results are improved through path similarity; and the threshold is dynamically adjusted according to the fault tolerance requirements of the application scenario to flexibly control the sensitivity of entering arbitration.

[0092] S70: When the value is below a preset threshold, a cross-modal retrieval divergence is determined; the arbitration module is activated to output the arbitration path weight vector; the child node is selected for in-depth retrieval according to the arbitration path weight vector, and the backtracking stack is retained. If the quality of the retrieval result is lower than the preset quality threshold, the divergence point is backtracked to select the suboptimal path to continue the retrieval until the final retrieval result is obtained.

[0093] In this embodiment, cross-modal retrieval divergence occurs when the retrieval paths of index trees of different modalities significantly fork due to semantic contradictions or noise between modalities in the query, resulting in inconsistent or conflicting candidate result sets. The arbitration module is a lightweight neural network or reinforcement learning agent. Its input includes the multimodal context of the current node, and its output is an arbitration path weight vector. Each component of the vector corresponds to a child node, representing the priority or probability of selecting that child node to continue the retrieval.

[0094] A backtracking stack is a last-in-first-out (LIFO) data structure used to record alternative paths not selected at branch points during the retrieval process, along with their corresponding arbitration weights. When the current path's retrieval result is unsatisfactory, the top element of the stack can be popped, backtracking to the previous branch point, and a suboptimal path can be used to continue the retrieval. A preset quality threshold is an indicator used to measure whether the retrieval results meet the requirements.

[0095] Specifically, when the calculated path similarity is lower than a preset threshold, a cross-modal retrieval divergence is determined. At this point, the arbitration and backtracking retrieval process begins. First, the arbitration module is activated. The arbitration module takes the intermediate state of the multimodal query at the current node as input, including the feature vectors of each modality query, the aggregated statistics of the cross-modal contradiction matrices of the multimedia objects contained in each child node, and the confidence level of each modality feature vector. After calculation, the arbitration module outputs an arbitration path weight vector, selects the child node with the highest weight as the next hop based on the weights, and pushes the path information of the current node and the remaining child nodes into the backtracking stack for storage. Then, the process is repeated in the child node. When a leaf node is reached and a candidate result is obtained, the quality of the result is evaluated. If the quality is lower than a preset threshold, backtracking is triggered, and the process iterates until a result meeting the quality standard is found or the backtracking stack is empty.

[0096] In step S70 of the method provided in this embodiment of the invention, the arbitration module employs a lightweight neural network, and the training method of the lightweight neural network includes: Construct a reinforcement learning environment where the state is the intermediate result of multimodal retrieval at the current node, the action is to select the child node index to visit next, and the reward is the weighted sum of the accuracy of the final retrieval result and the number of nodes visited. The lightweight neural network is trained using historical multimodal query logs so that the arbitration path weight vector output by the lightweight neural network can maximize the cumulative reward when cross-modal retrieval divergence occurs. The inputs to the arbitration module include the feature vectors of each query modality in the multimodal query, the candidate set of child nodes of each modality at the current node, the cross-modal contradiction matrix stored in the candidate multimedia object, and the confidence of each modality feature vector. The confidence of each modality feature vector is generated by the corresponding feature extractor and calculated based on the signal-to-noise ratio of the feature extractor output layer. The signal-to-noise ratio is positively correlated with the confidence. The confidence is used to reduce the influence of low-confidence modalities in the calculation of the arbitration path weight vector.

[0097] In this embodiment, firstly, the reinforcement learning environment is a machine learning framework in which an agent learns the optimal policy through interaction with the environment. The environment receives the agent's actions and returns new states and reward signals; the state is the intermediate result of multimodal retrieval at the current node, specifically including: the identifier of the current node and the mean and variance of its stored cross-modal conflict matrix, the feature vectors of each modality query, and the path information from the root node to the current node.

[0098] An action is the selection of the next child node index to be visited. For example, if the current node has m child nodes, the action space size is m, and each action corresponds to selecting one child node for in-depth retrieval. The reward is a scalar signal fed back by the environment after the agent performs an action, which is used to evaluate the quality of the action.

[0099] Specifically, a reinforcement learning environment simulating the real retrieval process is constructed, which encapsulates the structure of the basic hierarchical index tree, the contradiction statistics of each node, and the cross-modal contradiction matrix of candidate multimedia objects.

[0100] The state of the environment is represented as a feature vector. When the agent selects the index of a child node, the environment updates the current node to the selected child node and returns the new state. If the selected child node is a leaf node, the environment terminates the current round of retrieval, calculates the relevance of all candidate objects on the path from the root to the leaf node to the query, and counts the total number of nodes visited during the entire retrieval process. Then, the reward value is calculated according to a preset formula: R = w1 × relevance of the final returned result to the query - w2 × total number of nodes visited during the retrieval, where w1 and w2 are the weight coefficients of the relevance of the final returned result to the query and the total number of nodes visited during the retrieval, respectively, which can be determined according to their influence on the reward value.

[0101] During the retrieval process, the current node may be the root node, an intermediate node, or a leaf node, depending on the progress of the retrieval. The reward function encourages the agent to minimize the number of node visits while maintaining high accuracy, thus achieving a balance between efficiency and precision.

[0102] Secondly, regarding the model structure, the lightweight neural network adopts a three-layer fully connected architecture. The number of nodes in the input layer equals the dimension of the state vector, specifically including: feature vectors of each query modality, flattened cross-modal conflict mean matrix of each child node under the current node, confidence of each modality, and meta-features such as node depth and number of objects, with a total input dimension of approximately 400 dimensions. Following this are two hidden layers. The first hidden layer contains 128 neurons and uses the ReLU activation function to extract high-order interaction features; the second hidden layer contains 64 neurons and also uses the ReLU activation function to further compress the feature representation. Finally, the number of nodes in the output layer equals the number of child nodes m of the current node, uses the Softmax activation function, and outputs the arbitration path weight vector, i.e., the probability distribution of each child node being selected.

[0103] During training, a batch of query samples is sampled from the logs. For each query, it is input into the reinforcement learning environment, and the current lightweight neural network acts as the agent to perform the retrieval process. At each step of the retrieval process, the network receives the current state and outputs an arbitration path weight vector. The environment selects an action based on this weight vector and ultimately receives a reward. A policy gradient method is used to calculate the gradient of the reward with respect to the network parameters, and an optimizer is used to update the network parameters so that the network tends to output weight vectors that bring higher rewards. The training process is iterative. If the reward does not improve after 10 consecutive rounds of validation, training stops and the optimal model parameters are restored. The iteration stops when the maximum number of iterations (5000 rounds) is reached, or when an early stopping condition is triggered.

[0104] After training, the model is tested using an independent validation set containing various typical multimodal queries. For each validation sample, the matching degree between the model's output arbitration path weight vector and the actual optimal path, as well as the accuracy and number of node visits in the final retrieval result, are recorded.

[0105] When cross-modal retrieval disagreements occur during actual retrieval, the state vector of the current node is input into the network, and the network outputs an arbitration path weight vector through forward propagation, based on which a child node is selected for further retrieval. Since the network is lightweight, the impact of the decision-making process on retrieval latency is negligible.

[0106] Among them, the candidate multimedia objects are all multimedia objects stored under the current node during the retrieval process; the child node candidate set is the set of all child nodes that need to be evaluated in the next layer when the current node is retrieved in the basic hierarchical index tree.

[0107] Specifically, at each arbitration decision, i.e. after a disagreement occurs, it is necessary to select child nodes at the current node, collect the feature vectors of each query modality, the features of the candidate child node set of each modality at the current node, the aggregated statistics of the cross-modal conflict matrix stored in the candidate multimedia objects, and the confidence level of each modality feature vector. Among them, the features of the candidate child node set of each modality at the current node are obtained by flattening the cross-modal conflict mean matrix stored in the child node for each child node and using the difference between it and the mean matrix of its parent node as features.

[0108] The aggregate statistics of the cross-modal contradiction matrix stored in the candidate multimedia object storage are the contradiction statistics within the child nodes; the confidence of each modal feature vector is the signal-to-noise ratio calculated by each feature extractor before its output layer. For example, for a convolutional neural network, the variance of the activation values ​​of the feature map of the last convolutional layer can be taken as the signal, and the prediction uncertainty can be taken as noise. The signal-to-noise ratio is calculated as the ratio of the variance of the effective signal of the feature vector to the variance of the noise. Then, the SNR is mapped to the [0,1] interval by the sigmoid function as the confidence level, where high confidence indicates that the feature is reliable and low confidence indicates that the feature is unreliable.

[0109] Before feeding the confidence score into the lightweight neural network, the query modality feature vector is modulated, compressing the feature vector of the low-confidence modality to near zero, thus reducing its contribution to the network. The modulated feature vector is then concatenated with other inputs and fed into the neural network. The network outputs an arbitration path weight vector, where each component corresponds to the selection weight of a child node. Because the influence of the low-confidence modality is weakened, the network will rely more on the features of the high-confidence modality and conflicting statistical information to make decisions.

[0110] For example, a low-resolution image has a visual confidence of 0.3, while a clear audio recording of a call has an audio confidence of 0.9. The visual confidence after modulation by the arbitration module is close to zero, while the audio features remain clear.

[0111] Step S70 in the method provided in this embodiment of the invention further includes: Receive a user-specified contradiction query mode, wherein the contradiction query mode includes at least one of a visual and text contradiction mode, an audio and visual contradiction mode, and an audio and text contradiction mode; Based on the contradiction query pattern, determine the corresponding contradiction matrix template; Filter leaf nodes from all leaf nodes where the similarity between the cross-modal contradiction matrix and the contradiction matrix template is greater than a preset contradiction similarity threshold; The multimedia object corresponding to the multimedia object identifier pointed to by the leaf node is returned as the contradiction search result.

[0112] In this embodiment, firstly, the contradiction query mode is the user-specified retrieval intent, the target of which is an object in which there is a significant semantic conflict or inconsistency between modal pairs of a specific multimedia object; the visual and text contradiction mode is to require that there is a semantic contradiction between the visual features and the associated text features in the returned multimedia object; the audio and visual contradiction mode is to require that the audio content and visual content in the returned object are inconsistent; and the audio and text contradiction mode is to require that the audio content and text description in the returned object conflict.

[0113] Specifically, the system receives the contradictory query mode specified by the user through a query interface, which can be a checkbox or drop-down menu in a graphical interface. First, it parses the user input, then identifies the type of contradiction the user intends to retrieve. If the user selects both a visual and text contradictory mode and an audio and text contradictory mode, it returns an object that satisfies both contradictory types, records the user's contradictory query intent, and converts it into parameters that can be processed in subsequent steps. If the user does not specify any contradictory query mode, it processes the query as a regular multimodal query; if at least one contradictory mode is specified, it proceeds to the contradictory query stage.

[0114] Secondly, based on the aforementioned contradiction query pattern, the corresponding contradiction matrix template is determined. The contradiction matrix template is a K×K symmetric matrix template that theoretically represents the idealized characteristics of a specific contradiction pattern. The element values ​​in the template reflect the numerical distribution that the expected cross-modal contradiction matrix of a multimedia object should have for a certain contradiction query pattern.

[0115] Specifically, upon receiving a user's contradictory query pattern, a corresponding contradictory matrix template is determined based on the pattern. This template is a K×K symmetric matrix with the same dimensions as the cross-modal contradictory matrix of the multimedia object, where K represents the total number of modalities. Each element in the template represents the expected degree of contradiction for the corresponding modality pair. For explicitly specified contradictory modal pairs, the value at the corresponding position in the template is set to a high value, indicating the expectation of retrieving objects with high contradictory scores for that modality pair. For unspecified modal pairs, the value at the corresponding position in the template can be set to 0.5 or 0.

[0116] Furthermore, among all leaf nodes, leaf nodes whose similarity to the cross-modal contradiction matrix and the contradiction matrix template is greater than a preset contradiction similarity threshold are selected. This preset contradiction similarity threshold is a hyperparameter set by the system or the user, with a value range of (0,1). Only leaf nodes with a similarity greater than the preset contradiction similarity threshold are considered to meet the contradiction query conditions. The specific threshold can be customized by the user according to the actual scenario, for example, set to 0.7.

[0117] Specifically, iterate through all leaf nodes. For each leaf node, obtain the cross-modal contradiction matrix of each multimedia object stored within the node. Calculate the similarity between the cross-modal contradiction matrix and the contradiction matrix template. Since a leaf node may contain multiple objects, the maximum, average, or maximum similarity between the contradiction matrices of all objects within the node and the template can be used as the representative similarity of that leaf node.

[0118] If the similarity is greater than a preset contradiction similarity threshold, the leaf node is marked as a candidate node, and the multimedia object identifier it points to is added to the candidate result set. The maximum similarity between the contradiction matrix of all objects within an intermediate node and common templates can be maintained in advance, allowing for quick skipping of subtrees that do not meet the conditions during traversal.

[0119] Finally, the multimedia object corresponding to the multimedia object identifier pointed to by the leaf node is returned as the contradiction search result.

[0120] Specifically, all multimedia object identifiers are extracted from the leaf nodes, removing those that might appear in different leaf nodes of multiple modal index trees, since a contradiction query only needs to return once. Then, the corresponding multimedia object is retrieved from the multimedia data store based on the multimedia object identifier. Finally, the multimedia objects are sorted in descending order of similarity to the contradiction matrix template to form a contradiction search results list, which is returned to the user. If the number of results exceeds the preset maximum, only the top N most similar results are returned. Users can further browse or perform secondary filtering based on the returned results.

[0121] Step S70 in the method provided in this embodiment of the invention further includes: When a new multimedia object is inserted or the modal feature vector of an existing multimedia object is updated, the cross-modal contradiction matrix of the multimedia object is recalculated. Update the mean matrix and variance of each parent node upwards along the underlying hierarchical index tree where the multimedia object is located; If the variance of a parent node exceeds a preset variance threshold, a local reconstruction of the subtree rooted at the corresponding parent node is triggered, and the subtree is reorganized according to the node splitting rules based on contradiction awareness.

[0122] In this embodiment of the application, firstly, when a new object is detected to be inserted or a certain modal feature vector of an existing object is updated, it is ensured that the cross-modal contradiction matrix of the object is consistent with its latest modal feature vector.

[0123] Specifically, for newly inserted multimedia objects, all K modal feature vectors are extracted. Then, the contradiction discriminator for each modality pair is called sequentially to calculate the contradiction score and fill it into the newly constructed cross-modal contradiction matrix. For feature vector updates of existing objects, the record of the object in unified storage is located, and the updated modality index is identified. Subsequently, only all modality pairs involving that modality are recalculated: that is, each modality ≠ updating the modality index. The updated modality pair is used as input to call the contradiction discriminator to obtain a new contradiction score, and the corresponding two symmetrical positions in the contradiction matrix are updated. The contradiction scores between other modality pairs that are not involved in the update remain unchanged.

[0124] Secondly, update the mean matrix and variance of each parent node upwards along the basic hierarchical index tree where the multimedia object is located.

[0125] The effects of a multimedia object are propagated upwards to its ancestor nodes in all underlying hierarchical index trees to which it belongs.

[0126] Specifically, for each index tree, the leaf node containing the multimedia object is located first. Within the leaf node, the old value of the object's contradiction matrix is ​​stored, or the matrix is ​​newly added. The internal records of that leaf node are updated: for insertion operations, the contradiction matrix of the new object is added to the node statistics; for update operations, the old matrix is ​​replaced with the new matrix, and the statistics are adjusted. Then, the process traverses upwards along the parent pointer: for each ancestor node, including the direct parent of the leaf node, grandparent nodes, and up to the root node, the mean matrix and variance it maintains are updated. Updates employ online algorithms, such as maintaining the total number of objects within a node, the cumulative sum of the elements of the contradiction matrix, and the squares of each element.

[0127] New mean = (old mean × total number of objects in the maintenance node + new matrix) / (total number of objects in the maintenance node + 1); during updates, new mean = (old mean × total number of objects in the maintenance node - old matrix + new matrix) / total number of objects in the maintenance node. Variance is similarly calculated using cumulative sums and sums of squares. Since updating each ancestor node involves only a constant number of matrix additions and divisions, the computational complexity is extremely low. Insertion operations are performed in parallel on all K index trees.

[0128] For example, suppose there is an intermediate node N in the visual index tree, which originally contains 100 objects with a visual-audio inconsistency mean of 0.2 and a variance of 0.01, indicating that the objects within the node are consistent. A multimedia object, originally under the node, has its audio features updated, causing its visual-audio score in the inconsistency matrix to change from 0.1 to 0.9. Updating along the path: the leaf nodes are updated first, then the parent nodes, and the new mean becomes (0.2 × 100 - 0.1 + 0.9) / 100 = 0.208.

[0129] Finally, if the variance of a parent node exceeds a preset variance threshold, a local reconstruction of the subtree rooted at the corresponding parent node is triggered, and the subtree is reorganized according to the node splitting rules based on contradiction awareness.

[0130] Specifically, when the variance of a parent node exceeds a preset variance threshold, it indicates that the set of multimedia objects covered by the parent node shows significant differentiation in the degree of cross-modal contradiction, that is, there are objects with high degree of contradiction and objects with low degree of contradiction within the node.

[0131] At this point, a local reconstruction of the subtree rooted at the parent node is triggered. The specific process is as follows: extract all child nodes under the parent node and obtain the cross-modal contradiction matrix of each multimedia object; use the similarity of the contradiction matrix as the clustering basis and combine the main modality feature distance to re-divide the clusters; create new child nodes for each new cluster; update the mean matrix and variance statistics of the parent node.

[0132] Following the conflict-aware node splitting rules, objects are re-clustered into multiple new child nodes. After clustering, the newly generated subtrees replace the original subtrees without reconstructing the entire basic hierarchical index tree. The child node pointers of the parent node are updated, and the storage space occupied by the replaced old nodes is released. After the local reconstruction is completed, the conflict variance of the parent node and its descendant nodes will fall below the preset variance threshold again. The local reconstruction does not involve the rest of the index tree, avoiding the high computational cost of global reconstruction.

[0133] Step S70 in the method provided in this embodiment of the invention further includes: Monitor the access frequency of each intermediate node in the underlying hierarchical index tree; When the access frequency of a certain intermediate node within a preset time window is lower than the first threshold, the intermediate node and its corresponding child nodes are merged into a leaf node to shorten the depth of the index tree. When the access frequency of a certain leaf node is higher than the second threshold, a split operation is performed on the leaf node, and the multimedia objects under the leaf node are clustered into multiple new child nodes according to the cross-modal contradiction matrix similarity, thereby increasing the depth of the index tree. The first threshold is less than the second threshold.

[0134] In this embodiment of the application, the access frequency of each intermediate node in each basic hierarchical index tree is continuously monitored.

[0135] Specifically, whenever a multimodal query is executed and the retrieval path passes through an intermediate node, an atomic increment operation is performed on the node's access counter. A sliding window mechanism is used to count the number of accesses within the window, ensuring that the frequency reflects recent access hotspots. Each intermediate node maintains two variables: the number of accesses within the current window and the last reset time. The counter is decayed or reset at fixed time intervals or when the window slides. The monitoring results are stored in the node metadata for subsequent decision-making.

[0136] Secondly, the first threshold is a percentage threshold that is dynamically adjusted based on the total query volume and is a preset low access frequency threshold used to determine whether an intermediate node is considered low-access. The preset time window is flexibly configured based on the dynamic characteristics of the multimedia dataset, the timeliness requirements of the query load, and the system resource overhead, for example, 24 hours. The merging operation is only triggered when the access frequency within the window is lower than the first threshold.

[0137] Specifically, for each intermediate node in each basic hierarchical index tree (excluding the root node), its access frequency within a preset time window is obtained and compared with a first threshold. When the access frequency is lower than the first threshold, it indicates that the feature area covered by the node is rarely hit by user queries, belonging to a low-access area, and triggering a merge operation.

[0138] During the merging operation, the identifiers and eigenvectors of all multimedia objects within the original intermediate node's subtree are collected. This information is then placed into a new leaf node. The new leaf node becomes the direct child of the original intermediate node's parent node, replacing the entire original subtree. The memory and storage resources occupied by the original intermediate node and all its child nodes are then released, thus merging the intermediate node and all its child nodes into a new leaf node. Simultaneously, the statistical information of the new leaf node's cross-modal inconsistency matrix, such as the mean matrix and variance, is recalculated, and the statistical information of the parent node, such as the total number of objects, is updated upwards.

[0139] For example, in the audio index tree, an intermediate node N represents the polar animal call region, containing two child nodes: N1 (polar bear call) and N2 (penguin call), each with a small number of audio objects. The first threshold determines that node N is unpopular, and it is merged with N1 and N2 into a leaf node Nˈ. After merging, the previous path: root node → N → N1 → leaf is transformed into the current path: root node → Nˈ.

[0140] Furthermore, the second threshold is a high access frequency threshold preset by the system, and its value is greater than the first threshold. For example, if the first threshold is 10 times / day, the second threshold could be 1000 times / day. The second threshold is used to determine whether a leaf node is a high-access node.

[0141] Simultaneously, the access frequency of each leaf node is monitored. When the access frequency of a leaf node exceeds a second threshold within a preset time window, it indicates that the feature area covered by the leaf node is a hotspot for user queries, and a large number of search requests are hitting the node. Leaf nodes typically contain a large number of multimedia objects, and their cross-modal contradictory characteristics may differ, thus triggering a split operation.

[0142] Specifically, for leaf nodes, firstly, the cross-modal contradiction matrix and principal modality feature vector of all multimedia objects within the leaf node are collected; according to the contradiction-aware splitting rule, the objects are clustered into multiple subsets; then, a new leaf node is created for each subset to store object pointers; at the same time, a new intermediate node is created, with the new leaf node as its child node; the position of the original leaf node is replaced by the new intermediate node, and the pointer of the parent node is updated.

[0143] After the split, the original leaf nodes disappear, resulting in one intermediate node and multiple new leaf nodes, increasing the index tree depth by 1. Because the number of objects within the new leaf nodes decreases and the inconsistency variance is reduced, subsequent queries targeting hotspot areas can more quickly locate the correct child nodes, improving the concurrent processing capability and accuracy of retrieval.

[0144] Ultimately, the first threshold is less than the second threshold, where the relationship between the first threshold and the second threshold is that the first threshold is less than the second threshold. This creates a hysteresis interval between low-frequency access merging and high-frequency access splitting, preventing the node state from changing frequently due to small fluctuations in access frequency.

[0145] In this embodiment, a lightweight neural network is used as the arbitration module, and modal confidence calculated by the signal-to-noise ratio of the feature extractor output layer is introduced. When cross-modal retrieval disagreements occur, the optimal path selection strategy is adaptively learned, effectively reducing the interference of noisy modalities on decision-making. Secondly, by receiving the user-specified contradictory query pattern, the pre-stored cross-modal contradictory matrix and template similarity are used for filtering, achieving efficient retrieval for specific cross-modal contradictory types and providing direct technical support. Thirdly, when a new object is inserted or a feature is updated, the contradictory matrix is ​​recalculated, and the mean matrix and variance of the parent node are incrementally updated upwards along the tree path. Then, when the variance exceeds the threshold, a local subtree reconstruction rooted at the corresponding parent node is triggered, ensuring that the index structure can respond to the dynamic evolution of data content in real time. Finally, the access frequency of intermediate nodes and leaf nodes is monitored, so that the depth of the index tree can adaptively match the access hotspot distribution of user queries, improving the response speed of hot queries while ensuring compact storage.

[0146] The embodiments of this application, through the specific implementation methods described above, achieve the following technical effects: In this embodiment, firstly, dedicated feature extractors pre-trained for each modality are used to ensure high-quality extraction of visual, audio, and text feature vectors, avoiding semantic distortion caused by forced fusion. By constructing a cross-modal inconsistency matrix in each leaf node and calling the inconsistency discriminator to fill in the inconsistency score during insertion, the semantic inconsistency between modalities is quantified into indexable structured information, providing a direct basis for subsequent arbitration. By comparing the retrieval path similarity of different basic hierarchical index trees, cross-modal retrieval disagreement is determined with extremely low computational overhead, solving the retrieval failure problem under modal inconsistency and improving the robustness of the system. Modal confidence is used to avoid interference from noisy modalities on the retrieval results.

[0147] Secondly, the similarity of the contradiction matrix between every two multimedia objects within a node is calculated and used as a partitioning criterion alongside the main modality feature distance, aligning the index tree structure with contradiction patterns. Objects with high contradiction matrix similarity are prioritized for partitioning into the same child node, effectively reducing the variance of contradiction scores within each child node after splitting, providing a more reliable basis for the retrieval process. After splitting, objects with different contradiction patterns are separated into different index branches. When a multimodal query is received, the arbitration module can quickly identify the branch matching the query intent, reducing the number of nodes that need to be accessed and lowering retrieval latency. Finally, the contradiction matrix similarity is used as a weighting factor along with the feature distance for clustering, and the weight coefficients are adjusted according to the application scenario to enhance the dynamic adaptive capability of the index, enabling the index tree to continuously self-optimize and maintain long-term retrieval performance.

[0148] Furthermore, the contradiction score is read during the query process, avoiding the need to call a complex discriminator, which makes high-concurrency multimodal retrieval possible; the offline calculation of contradiction score is optimized by using batch data, improving the computational efficiency of the discriminator and providing key input features for the subsequent arbitration module.

[0149] Furthermore, by utilizing an index structure optimized for modality, theoretically optimal retrieval complexity is achieved, thereby improving retrieval efficiency; fusion retrieval may lose path information, but provides raw data for cross-modal path comparison; the degree of path difference reflects the consistency of modality distribution in the index space, providing a basis for judgment in subsequent steps.

[0150] Furthermore, by recording the node sequences naturally generated during the retrieval process, the length of the longest common subsequence and the normalized similarity are calculated to meet real-time requirements; a complete record query is performed through the node sequences, and the LCS length can reflect the degree of alignment from coarse-grained to fine-grained, which has a stronger discriminative ability than comparing only leaf nodes; by calculating path similarity, the influence of the difference in depth between different modal index trees is eliminated, avoiding computational waste on incorrect paths.

[0151] Furthermore, query modality consistency avoids time-consuming operations and improves response speed; path similarity enhances the diversity and accuracy of fusion results; and thresholds are dynamically adjusted according to the fault tolerance requirements of the application scenario to flexibly control the sensitivity of arbitration.

[0152] Finally, a lightweight neural network is used as the arbitration module, and modal confidence calculated from the signal-to-noise ratio of the feature extractor output layer is introduced. When cross-modal retrieval disagreements occur, the optimal path selection strategy is adaptively learned, effectively reducing the interference of noisy modalities on the decision. Secondly, by receiving user-specified contradictory query patterns, the pre-stored cross-modal contradictory matrix and template similarity filtering are used to achieve efficient retrieval for specific cross-modal contradictory types, providing direct technical support. Thirdly, when a new object is inserted or a feature is updated, the contradictory matrix is ​​recalculated, and the mean matrix and variance of the parent node are incrementally updated upwards along the tree path. Then, when the variance exceeds the threshold, a local subtree reconstruction rooted at the corresponding parent node is triggered, ensuring that the index structure can respond to the dynamic evolution of data content in real time. Finally, the access frequency of intermediate nodes and leaf nodes is monitored, so that the index tree depth can adaptively match the access hotspot distribution of user queries, improving the response speed of hot queries while ensuring compact storage.

[0153] Example 2, as Figure 2 As shown, based on the same inventive concept as the multimedia hierarchical feature indexing method provided in Embodiment 1, this embodiment of the invention also provides a multimedia hierarchical feature indexing system, the system comprising: The index tree construction module 11 is used to extract K modal feature vectors for each multimedia object in the multimedia dataset and to construct an independent basic hierarchical index tree for each modal feature vector. The contradiction matrix construction module 12 is used to construct a cross-modal contradiction matrix within each leaf node; The contradiction score calculation module 13 is used to call the contradiction discriminator to calculate the contradiction score of all modal feature vector pairs when inserting multimedia objects, and fill it into the cross-modal contradiction matrix; The multimodal query module 14 is used to retrieve data from the corresponding basic hierarchical index tree when a multimodal query containing feature vectors of multiple query modes is received. The path similarity calculation module 15 is used to compare the retrieval paths of different basic hierarchical index trees and calculate the path similarity. The retrieval result fusion module 16 is used to fuse the retrieval results of each basic hierarchical index tree and return them when the path similarity is greater than or equal to a preset path similarity threshold. The retrieval result acquisition module 17 is used to determine that a cross-modal retrieval divergence has occurred when the result is below a preset threshold; activate the arbitration module to output the arbitration path weight vector; select a child node for in-depth retrieval based on the arbitration path weight vector, and retain the backtracking stack. If the quality of the retrieval result is lower than the preset quality threshold, backtrack to the divergence point to select the suboptimal path to continue retrieval until the final retrieval result is obtained.

[0154] In one embodiment, the index tree building module 11 is used for: A pre-trained feature extractor, the feature extractor comprising a convolutional neural network for extracting visual features, an audio embedding network for extracting audio features, and a language model for extracting text features; Based on the feature extractor, for each multimedia object, the visual features of the visual frame sequence are extracted as the first modality feature vector. Based on the feature extractor, the audio embedding features of the audio track are extracted as the second modality feature vector; Based on the feature extractor, the text embedding features of the associated text are extracted as the third modality feature vector; The first modal feature vector, the second modal feature vector, and the third modal feature vector are unified into the K modal feature vectors; Wherein, K is an integer greater than or equal to 2, and the modal feature vector includes at least two of the following: visual feature vector, audio feature vector, and text feature vector; the leaf node of each of the basic hierarchical index trees stores a pointer to the corresponding multimedia object identifier.

[0155] In one embodiment, the contradiction matrix construction module 12 is used for: The cross-modal contradiction matrix is ​​a K×K symmetric matrix, where the element in the i-th row and j-th column represents the degree of contradiction between the i-th modal feature vector and the j-th modal feature vector of the multimedia object; For each pair of modal types, a discriminant is pre-trained. The discriminant is used to take two modal feature vectors of any multimedia object as input and output a discriminant score between 0 and 1. The higher the discriminant score, the more inconsistent the two modal feature vectors are. In each intermediate node, maintain the mean matrix of the cross-modal conflict matrix of all multimedia objects within the node, as well as the variance of the conflict score within the node; When the variance exceeds a preset variance threshold, it is determined that there are semantically inconsistent multimedia objects within the node, triggering the node splitting rule for contradiction perception.

[0156] The node splitting rules for conflict perception include: When splitting a node, calculate the contradiction matrix similarity between every two multimedia objects within the node; The similarity of the contradiction matrix is ​​used as a weighting factor in the distance metric, and together with the feature distance between the main modality feature vectors, it participates in the clustering partitioning. The similarity of the contradiction matrix is ​​used as a partitioning criterion alongside the feature distance of the main modality feature vector, and multimedia objects with high contradiction matrix similarity are preferentially partitioned into the same child node.

[0157] In one embodiment, the path similarity calculation module 15 is used for: Record the node sequence from the root node to the candidate leaf node in the corresponding basic hierarchical index tree for each modality feature vector; Calculate the length of the longest common subsequence between node sequences of any two modalities; The path similarity is obtained by dividing the length of the longest common subsequence by the maximum length of the two node sequences.

[0158] In one embodiment, the retrieval result acquisition module 17 is used for: Construct a reinforcement learning environment where the state is the intermediate result of multimodal retrieval at the current node, the action is to select the child node index to visit next, and the reward is the weighted sum of the accuracy of the final retrieval result and the number of nodes visited. The lightweight neural network is trained using historical multimodal query logs so that the arbitration path weight vector output by the lightweight neural network can maximize the cumulative reward when cross-modal retrieval divergence occurs. The inputs to the arbitration module include the feature vectors of each query modality in the multimodal query, the candidate set of child nodes of each modality at the current node, the cross-modal contradiction matrix stored in the candidate multimedia object, and the confidence of each modality feature vector. The confidence of each modality feature vector is generated by the corresponding feature extractor and calculated based on the signal-to-noise ratio of the feature extractor output layer. The signal-to-noise ratio is positively correlated with the confidence. The confidence is used to reduce the influence of low-confidence modalities in the calculation of the arbitration path weight vector.

[0159] In one embodiment, the retrieval result acquisition module 17 is further configured to: Receive a user-specified contradiction query mode, wherein the contradiction query mode includes at least one of a visual and text contradiction mode, an audio and visual contradiction mode, and an audio and text contradiction mode; Based on the contradiction query pattern, determine the corresponding contradiction matrix template; Filter leaf nodes from all leaf nodes where the similarity between the cross-modal contradiction matrix and the contradiction matrix template is greater than a preset contradiction similarity threshold; The multimedia object corresponding to the multimedia object identifier pointed to by the leaf node is returned as the contradiction search result.

[0160] In one embodiment, the retrieval result acquisition module 17 is further configured to: When a new multimedia object is inserted or the modal feature vector of an existing multimedia object is updated, the cross-modal contradiction matrix of the multimedia object is recalculated. Update the mean matrix and variance of each parent node upwards along the underlying hierarchical index tree where the multimedia object is located; If the variance of a parent node exceeds a preset variance threshold, a local reconstruction of the subtree rooted at the corresponding parent node is triggered, and the subtree is reorganized according to the node splitting rules based on contradiction awareness.

[0161] In one embodiment, the retrieval result acquisition module 17 is further configured to: Monitor the access frequency of each intermediate node in the underlying hierarchical index tree; When the access frequency of a certain intermediate node within a preset time window is lower than the first threshold, the intermediate node and its corresponding child nodes are merged into a leaf node to shorten the depth of the index tree. When the access frequency of a certain leaf node is higher than the second threshold, a split operation is performed on the leaf node, and the multimedia objects under the leaf node are clustered into multiple new child nodes according to the cross-modal contradiction matrix similarity, thereby increasing the depth of the index tree. The first threshold is less than the second threshold.

[0162] Compared to existing technologies, this application first uses dedicated feature extractors pre-trained for each modality to ensure high-quality extraction of visual, audio, and text feature vectors, avoiding semantic distortion caused by forced fusion. By constructing a cross-modal inconsistency matrix in each leaf node and calling the inconsistency discriminator to fill in the inconsistency score during insertion, the semantic inconsistency between modalities is quantified into indexable structured information, providing a direct basis for subsequent arbitration. By comparing the retrieval path similarity of different basic hierarchical index trees, cross-modal retrieval disagreement is determined with extremely low computational cost, solving the retrieval failure problem under modal inconsistency and improving the robustness of the system. Modal confidence is used to avoid interference from noisy modalities on the retrieval results.

[0163] Secondly, the similarity of the contradiction matrix between every two multimedia objects within a node is calculated and used as a partitioning criterion alongside the main modality feature distance, aligning the index tree structure with contradiction patterns. Objects with high contradiction matrix similarity are prioritized for partitioning into the same child node, effectively reducing the variance of contradiction scores within each child node after splitting, providing a more reliable basis for the retrieval process. After splitting, objects with different contradiction patterns are separated into different index branches. When a multimodal query is received, the arbitration module can quickly identify the branch matching the query intent, reducing the number of nodes that need to be accessed and lowering retrieval latency. Finally, the contradiction matrix similarity is used as a weighting factor along with the feature distance for clustering, and the weight coefficients are adjusted according to the application scenario to enhance the dynamic adaptive capability of the index, enabling the index tree to continuously self-optimize and maintain long-term retrieval performance.

[0164] Furthermore, the contradiction score is read during the query process, avoiding the need to call a complex discriminator, which makes high-concurrency multimodal retrieval possible; the offline calculation of contradiction score is optimized by using batch data, improving the computational efficiency of the discriminator and providing key input features for the subsequent arbitration module.

[0165] Furthermore, by utilizing an index structure optimized for modality, theoretically optimal retrieval complexity is achieved, thereby improving retrieval efficiency; fusion retrieval may lose path information, but provides raw data for cross-modal path comparison; the degree of path difference reflects the consistency of modality distribution in the index space, providing a basis for judgment in subsequent steps.

[0166] Furthermore, by recording the node sequences naturally generated during the retrieval process, the length of the longest common subsequence and the normalized similarity are calculated to meet real-time requirements; a complete record query is performed through the node sequences, and the LCS length can reflect the degree of alignment from coarse-grained to fine-grained, which has a stronger discriminative ability than comparing only leaf nodes; by calculating path similarity, the influence of the difference in depth between different modal index trees is eliminated, avoiding computational waste on incorrect paths.

[0167] Furthermore, query modality consistency avoids time-consuming operations and improves response speed; path similarity enhances the diversity and accuracy of fusion results; and thresholds are dynamically adjusted according to the fault tolerance requirements of the application scenario to flexibly control the sensitivity of arbitration.

[0168] Finally, a lightweight neural network is used as the arbitration module, and modal confidence calculated from the signal-to-noise ratio of the feature extractor output layer is introduced. When cross-modal retrieval disagreements occur, the optimal path selection strategy is adaptively learned, effectively reducing the interference of noisy modalities on the decision. Secondly, by receiving user-specified contradictory query patterns, the pre-stored cross-modal contradictory matrix and template similarity filtering are used to achieve efficient retrieval for specific cross-modal contradictory types, providing direct technical support. Thirdly, when a new object is inserted or a feature is updated, the contradictory matrix is ​​recalculated, and the mean matrix and variance of the parent node are incrementally updated upwards along the tree path. Then, when the variance exceeds the threshold, a local subtree reconstruction rooted at the corresponding parent node is triggered, ensuring that the index structure can respond to the dynamic evolution of data content in real time. Finally, the access frequency of intermediate nodes and leaf nodes is monitored, so that the index tree depth can adaptively match the access hotspot distribution of user queries, improving the response speed of hot queries while ensuring compact storage.

Claims

1. A multimedia hierarchical feature indexing method, characterized in that, include: For each multimedia object in the multimedia dataset, extract K modal feature vectors and construct an independent basic hierarchical index tree for each modal feature vector; Construct a cross-modal contradiction matrix within each leaf node; When inserting a multimedia object, the contradiction discriminator is called to calculate the contradiction score of all modal feature vector pairs and fill it into the cross-modal contradiction matrix; When a multimodal query containing feature vectors of multiple query modalities is received, it is retrieved from the corresponding basic hierarchical index tree. Compare the retrieval paths of different basic hierarchical index trees and calculate the path similarity; When the path similarity is greater than or equal to the preset path similarity threshold, the retrieval results of each basic hierarchical index tree are merged and returned. A cross-modal retrieval divergence is determined when the value is below a preset threshold. The arbitration module is activated to output the arbitration path weight vector; based on the arbitration path weight vector, a child node is selected for in-depth retrieval, and a backtracking stack is retained. If the quality of the retrieval result is lower than a preset quality threshold, the process backtracks to the branch point to select the suboptimal path and continues retrieval until the final retrieval result is obtained.

2. The method as described in claim 1, characterized in that, For each multimedia object in the multimedia dataset, extract K modal feature vectors, and construct an independent basic hierarchical index tree for each modal feature vector, including: A pre-trained feature extractor, the feature extractor comprising a convolutional neural network for extracting visual features, an audio embedding network for extracting audio features, and a language model for extracting text features; Based on the feature extractor, for each multimedia object, the visual features of the visual frame sequence are extracted as the first modality feature vector. Based on the feature extractor, the audio embedding features of the audio track are extracted as the second modality feature vector; Based on the feature extractor, the text embedding features of the associated text are extracted as the third modality feature vector; The first modal feature vector, the second modal feature vector, and the third modal feature vector are unified into the K modal feature vectors; Wherein, K is an integer greater than or equal to 2, and the modal feature vector includes at least two of the following: visual feature vector, audio feature vector, and text feature vector; the leaf node of each of the basic hierarchical index trees stores a pointer to the corresponding multimedia object identifier.

3. The method as described in claim 1, characterized in that, After constructing the cross-modal contradiction matrix within each leaf node, the following steps are also included: The cross-modal contradiction matrix is ​​a K×K symmetric matrix, where the element in the i-th row and j-th column represents the degree of contradiction between the i-th modal feature vector and the j-th modal feature vector of the multimedia object; For each pair of modal types, a discriminant is pre-trained. The discriminant is used to take two modal feature vectors of any multimedia object as input and output a discriminant score between 0 and 1. The higher the discriminant score, the more inconsistent the two modal feature vectors are. In each intermediate node, maintain the mean matrix of the cross-modal conflict matrix of all multimedia objects within the node, as well as the variance of the conflict score within the node; When the variance exceeds a preset variance threshold, it is determined that there are semantically inconsistent multimedia objects within the node, triggering the node splitting rule for contradiction perception.

4. The method as described in claim 3, characterized in that, The node splitting rules for conflict detection include: When splitting a node, calculate the contradiction matrix similarity between every two multimedia objects within the node; The similarity of the contradiction matrix is ​​used as a weighting factor in the distance metric, and together with the feature distance between the main modality feature vectors, it participates in the clustering partitioning. The similarity of the contradiction matrix is ​​used as a partitioning criterion alongside the feature distance of the main modality feature vector, and multimedia objects with high contradiction matrix similarity are preferentially partitioned into the same child node.

5. The method as described in claim 1, characterized in that, Compare the retrieval paths of different basic hierarchical index trees and calculate path similarity, including: Record the node sequence from the root node to the candidate leaf node in the corresponding basic hierarchical index tree for each modality feature vector; Calculate the length of the longest common subsequence between node sequences of any two modalities; The path similarity is obtained by dividing the length of the longest common subsequence by the maximum length of the two node sequences.

6. The method as described in claim 1, characterized in that, The arbitration module employs a lightweight neural network, and the training method for the lightweight neural network includes: Construct a reinforcement learning environment where the state is the intermediate result of multimodal retrieval at the current node, the action is to select the child node index to visit next, and the reward is the weighted sum of the accuracy of the final retrieval result and the number of nodes visited. The lightweight neural network is trained using historical multimodal query logs so that the arbitration path weight vector output by the lightweight neural network can maximize the cumulative reward when cross-modal retrieval divergence occurs. The inputs to the arbitration module include the feature vectors of each query modality in the multimodal query, the candidate set of child nodes of each modality at the current node, the cross-modal contradiction matrix stored in the candidate multimedia object, and the confidence of each modality feature vector. The confidence of each modality feature vector is generated by the corresponding feature extractor and calculated based on the signal-to-noise ratio of the feature extractor output layer. The signal-to-noise ratio is positively correlated with the confidence. The confidence is used to reduce the influence of low-confidence modalities in the calculation of the arbitration path weight vector.

7. The method as described in claim 1, characterized in that, Also includes: Receive a user-specified contradiction query mode, wherein the contradiction query mode includes at least one of a visual and text contradiction mode, an audio and visual contradiction mode, and an audio and text contradiction mode; Based on the contradiction query pattern, determine the corresponding contradiction matrix template; Filter leaf nodes from all leaf nodes where the similarity between the cross-modal contradiction matrix and the contradiction matrix template is greater than a preset contradiction similarity threshold; The multimedia object corresponding to the multimedia object identifier pointed to by the leaf node is returned as the contradiction search result.

8. The method as described in claim 1, characterized in that, Also includes: When a new multimedia object is inserted or the modal feature vector of an existing multimedia object is updated, the cross-modal contradiction matrix of the multimedia object is recalculated. Update the mean matrix and variance of each parent node upwards along the basic hierarchical index tree where the multimedia object is located; If the variance of a parent node exceeds a preset variance threshold, a local reconstruction of the subtree rooted at the corresponding parent node is triggered, and the subtree is reorganized according to the node splitting rules based on contradiction awareness.

9. The method as described in claim 1, characterized in that, Also includes: Monitor the access frequency of each intermediate node in the underlying hierarchical index tree; When the access frequency of a certain intermediate node within a preset time window is lower than the first threshold, the intermediate node and its corresponding child nodes are merged into a leaf node to shorten the depth of the index tree. When the access frequency of a certain leaf node is higher than the second threshold, a split operation is performed on the leaf node, and the multimedia objects under the leaf node are clustered into multiple new child nodes according to the cross-modal contradiction matrix similarity, thereby increasing the depth of the index tree. The first threshold is less than the second threshold.

10. A multimedia hierarchical feature indexing system based on multi-source knowledge base retrieval, characterized in that, A multimedia hierarchical feature indexing method based on multi-source knowledge base retrieval as described in any one of claims 1-9, the system comprising: The index tree building module is used to extract K modal feature vectors for each multimedia object in the multimedia dataset and build an independent basic hierarchical index tree for each modal feature vector; The contradiction matrix construction module is used to construct a cross-modal contradiction matrix within each leaf node; The contradiction score calculation module is used to call the contradiction discriminator to calculate the contradiction score of all modal feature vector pairs when inserting multimedia objects, and fill it into the cross-modal contradiction matrix; The multimodal query module is used to retrieve data from the corresponding basic hierarchical index tree when a multimodal query containing feature vectors of multiple query modes is received. The path similarity calculation module is used to compare the retrieval paths of different basic hierarchical index trees and calculate the path similarity. The retrieval result fusion module is used to fuse the retrieval results of each basic hierarchical index tree and return them when the path similarity is greater than or equal to a preset path similarity threshold. The retrieval result acquisition module is used to determine that a cross-modal retrieval divergence has occurred when the result is below a preset threshold; the arbitration module is activated to output an arbitration path weight vector; the sub-nodes are selected for in-depth retrieval based on the arbitration path weight vector, and a backtracking stack is retained. If the quality of the retrieval result is lower than a preset quality threshold, the system backtracks to the divergence point to select a suboptimal path to continue retrieval until the final retrieval result is obtained.

Images