Multi-modal art material intelligent retrieval method and device, equipment and medium

Abstract

The invention relates to a multi-mode art material intelligent retrieval method and device, equipment and a medium. The method comprises the following steps: performing off-line feature extraction on an art material library image and constructing a multi-level style feature index; obtaining query data containing at least two modes of an image, a text and a sketch, and mapping each mode data to a unified style semantic space to generate a pseudo style attribute vector and a corresponding deterministic score; performing dynamic weight calculation and weighted fusion based on the vector and score of each modal to form a combined style query vector; finally, a retrieval result list is generated through similarity calculation and sorting of the combined style query vectors and the feature indexes. By adopting the method, fine-grained alignment and dynamic weight fusion of multi-modal query on the aspect of art styles can be realized, and the accuracy of art material style retrieval under complex mixed intentions is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of art material retrieval technology, and in particular relates to a multimodal intelligent retrieval method, device, equipment and medium for art materials. Background Technology

[0002] With the rapid development of industries such as digital content creation, game art, and graphic design, the need for efficient and accurate retrieval of massive amounts of art assets is becoming increasingly urgent. Traditional art asset management relies heavily on manually labeled keywords or simple category tags, and its retrieval efficiency and accuracy are no longer sufficient to meet the complex needs of modern creation for specific styles, compositions, and details.

[0003] Currently, art resource retrieval technologies mainly encompass text-based keyword retrieval, content-based image retrieval (CBIR), and early multimodal retrieval methods that combine both. Text-based keyword retrieval relies on metadata tags for the resources, but these tags often fail to capture subtle artistic style features. Content-based image retrieval matches images by calculating their visual features (such as color, texture, and shape), but its ability to understand high-level abstract styles like "cyberpunk" or "ink painting" is limited. Some existing multimodal retrieval methods attempt to align text descriptions with image features in a shared semantic space to achieve cross-modal retrieval.

[0004] However, existing technologies have significant limitations when handling complex, mixed-intent queries from users. When users simultaneously provide multiple modalities of query input, such as reference images, descriptive text, and sketches outlining compositions, existing methods lack effective mechanisms to deeply understand and integrate the fine-grained semantics carried by these heterogeneous information, particularly those related to art style. Specifically, this manifests as: difficulty in mapping different modal query information to a unified semantic space capable of accurately depicting the details of art style; and an inability to dynamically adjust the contribution weights of each modal input in the final retrieval decision based on the credibility and complementarity of the input content itself. This results in unsatisfactory style relevance in the search results, failing to meet the precise needs of professional art creation. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, equipment, and medium for intelligent retrieval of multimodal art materials that can achieve fine-grained semantic intelligent retrieval of multimodal art materials, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a multimodal intelligent retrieval method for art materials, including:

[0007] S1. Obtain image data from the art resource library, perform offline feature extraction on the image data, and generate a multi-level style feature vector set corresponding to each image in the art resource library;

[0008] S2. Construct an index based on a multi-level style feature vector set to generate a multi-level style feature index;

[0009] S3. Obtain material query data, perform feature mapping on each modality data in the material query data, and generate pseudo-style attribute vectors and deterministic scores for each modality in the style semantic space; the material query data includes image modality data, text modality data, and sketch modality data;

[0010] S4. Calculate dynamic weights based on the pseudo-style attribute vectors and deterministic scores of each modality to generate dynamic fusion weights for each modality.

[0011] S5. Based on dynamic fusion weights, the pseudo-style attribute vectors of each modality are weighted and fused to generate a joint style query vector;

[0012] S6. Calculate the similarity between the joint style query vector and the multi-level style feature index to generate comprehensive similarity information between each source image and the joint style query vector;

[0013] S7. Based on comprehensive similarity information, sort the results from high to low similarity and generate a list of search results.

[0014] In one embodiment, S1 includes:

[0015] S11. For each image in the art resource library, feature extraction is performed using a pre-trained deep neural network to generate a global feature map corresponding to each image.

[0016] S12. Input the global feature map into the style attribute decoupling network, and perform attribute prediction through multiple parallel attribute prediction heads to generate attribute values ​​for each predefined style dimension.

[0017] S13. Concatenate the attribute values ​​output by all attribute prediction heads to generate a global style vector set for the image.

[0018] S14. Based on the global feature maps corresponding to each image, feature extraction is performed through a local style encoder to generate a set of local style feature vectors for the image.

[0019] S15. Based on the global feature maps corresponding to each image, feature extraction is performed through a region style encoder to generate a set of region style features for the image.

[0020] S16. Integrate the global style vector set, the local style feature vector set, and the regional style feature set into a multi-level style feature vector set for the image.

[0021] In one embodiment, S2 includes:

[0022] S21. Construct the first vector index for the global style vector set;

[0023] S22. Construct a second vector index for the set of regional style features;

[0024] S23. Construct a third feature index for the local style feature map;

[0025] S24. Combine the first vector index, the second vector index, and the third feature index to generate a multi-level style feature index.

[0026] In one embodiment, S3 includes:

[0027] S31. When the material query data is image-type modal data, feature extraction is performed on the image-type modal data through a feature extraction network to generate the first intermediate feature;

[0028] S32. Perform style semantic space mapping on the first intermediate feature to generate a pseudo-style attribute vector, and set the deterministic score of the image class modality data to a fixed value.

[0029] S33. When the material query data is text modal data, the text modal data is processed by a text encoder to extract features and generate text semantic features.

[0030] S34. The semantic features of the text are mapped through a text style attribute mapping network to generate a pseudo-style attribute vector of the text. The confidence score of the generated text is used as the deterministic score of the text modal data through confidence evaluation.

[0031] S35. When the material query data is sketch modal data, the sketch modal data is processed by a sketch encoder to extract features and generate a sketch structure feature vector.

[0032] S36. Project the sketch structure feature vectors using a preset projection matrix to generate sketch pseudo-style attribute vectors, and calculate the deterministic score of sketch modal data based on the modulus of the sketch structure feature vectors.

[0033] In one embodiment, prior to S36, the following is also included:

[0034] S361. Construct training sample triplets; the training sample triplets include anchor sketch modal data, positive example image data with similar structure and style to the anchor sketch modal data, and negative example image data with similar structure but mismatched style to the anchor sketch modal data.

[0035] S362. Based on anchor point sketch modal data, positive example image data, and negative example image data, the projection matrix is ​​optimized and trained by minimizing the triplet loss function to obtain the optimized projection matrix parameters; the triplet loss function is:

[0036]

[0037] in, This is the pseudo-style attribute vector of the sketch obtained by projecting the anchor sketch modal data. This is the global style vector extracted from positive example image data. This is the global style vector extracted from negative example image data. Let cosine distance function, These are boundary parameters;

[0038] S363. Construct a preset projection matrix based on the optimized projection matrix parameters.

[0039] In one embodiment, S4 includes:

[0040] S41. Based on the deterministic scores of each modality, the initial weights are calculated using a normalization function to generate the initial basic weights for each modality.

[0041] S42. Calculate the intermodal difference measure based on the pseudostyle attribute vectors of each modality, and generate the complementarity coefficient matrix;

[0042] S43. Based on the initial base weights and complementarity coefficient matrix, the fusion weights are calculated using a weight adjustment function to generate dynamic fusion weights for each modality; the calculation method for dynamic fusion weights is as follows:

[0043]

[0044] in, For dynamic weight fusion, For modality The initial base weights, As a balance factor, For modality With mode The complementarity coefficient between them For modality The initial base weights, This is the normalization factor.

[0045] In one embodiment, S42 includes:

[0046] S421. Calculate the similarity between pseudo-style attribute vectors of any two different modalities and generate the cosine distance between modalities.

[0047] S422. Based on the cosine distance between modes, perform coefficient mapping to generate the corresponding complementarity coefficients; the calculation method for the complementarity coefficients is as follows:

[0048]

[0049] in, The complementarity coefficient, For modality With mode The cosine distance between them For scale parameters;

[0050] S423. Combine the complementarity coefficients between all mode pairs into a complementarity coefficient matrix.

[0051] Secondly, this application also provides a multimodal intelligent retrieval device for art materials, including:

[0052] The offline feature extraction module is used to acquire image data from the art resource library, perform offline feature extraction on the image data, and generate a multi-level style feature vector set corresponding to each image in the art resource library;

[0053] The index building module is used to build an index based on a multi-level style feature vector set and generate a multi-level style feature index.

[0054] The multimodal feature mapping module is used to acquire material query data, perform feature mapping on each modality data in the material query data, and generate pseudo-style attribute vectors and deterministic scores for each modality in the style semantic space; the material query data includes image modality data, text modality data, and sketch modality data;

[0055] The dynamic weight calculation module is used to perform dynamic weight calculation based on the pseudo-style attribute vector and deterministic score of each modality, and generate dynamic fusion weights corresponding to each modality.

[0056] The joint style fusion module is used to perform weighted fusion of pseudo-style attribute vectors of each modality based on dynamic fusion weights to generate a joint style query vector;

[0057] The similarity calculation module is used to calculate the similarity based on the joint style query vector and the multi-level style feature index, and generate comprehensive similarity information between each source image and the joint style query vector.

[0058] The search and ranking module is used to sort the results from highest to lowest similarity based on comprehensive similarity information and generate a list of search results.

[0059] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the multimodal art material intelligent retrieval method as described in the first aspect.

[0060] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal art material intelligent retrieval method as described in the first aspect.

[0061] The aforementioned intelligent retrieval method, apparatus, device, and medium for multimodal art materials acquires image data from an art material library and performs offline feature extraction and index construction to establish a fine-grained style feature foundation for retrieval. During retrieval, it acquires material query data containing at least two modalities, including images, text, and sketches. By mapping each modal data to a unified style semantic space, it generates corresponding pseudo-style attribute vectors and a deterministic score to evaluate their credibility. Then, based on the vectors and scores of each modality, it performs dynamic weight calculation and weighted fusion to form a joint style query vector that comprehensively reflects the user's complex intent. Finally, by calculating the similarity and ranking of this joint vector with a pre-constructed multi-level style feature index, it outputs a list of retrieval results. This method achieves fine-grained alignment of multimodal queries by constructing a unified style semantic space and introduces a dynamic fusion mechanism based on credibility evaluation, effectively improving the retrieval accuracy of art materials in the art style dimension and its adaptability to complex mixed query intents. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating a multimodal intelligent retrieval method for art materials provided by the present invention.

[0064] Figure 2 A flowchart illustrating a method for generating a multi-level style feature vector set in an optional embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of the structure of a multimodal art material intelligent retrieval device provided by the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] In one embodiment, such as Figure 1As shown, a multimodal intelligent retrieval method for art materials is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0068] S1. Obtain image data from the art resource library, perform offline feature extraction on the image data, and generate a multi-level style feature vector set corresponding to each image in the art resource library.

[0069] Optionally, mature pre-trained deep neural networks (such as ResNet50, Residual Network 50, or Residual Network 50) are used to encode features for each image, outputting a global feature map containing global semantic information of the image. Predefined style dimensions cover fine-grained attributes such as color tendency, brushstroke type, composition structure, and art style. The style attribute decoupling network uses multiple parallel fully connected layers as attribute prediction heads to extract and quantize information corresponding to the style dimension in the global feature map, outputting attribute values ​​for each dimension and concatenating them to form a global style vector set. The local style encoder uses a sliding window mechanism to traverse the global feature map, extracting feature vectors from local image regions (such as texture details and local color distribution) to form a local style feature vector set. The region style encoder uses object detection algorithms (such as Faster R-CNN, FasterRegion-based Convolutional Neural Networks) to segment semantic regions in the image, such as the main object and background regions, and then encodes the features of each region to generate a region style feature set. Finally, the three feature sets are integrated into a complete multi-level style feature vector set.

[0070] S2. Construct an index based on a multi-level style feature vector set to generate a multi-level style feature index.

[0071] Optionally, to address the high-dimensionality of the global style vector set, a KD-tree (K-Dimensional Tree) or a ball tree is used to construct the first vector index, enabling rapid matching at the global style level. After grouping the regional style feature set by semantic region type, an inverted index combined with vector quantization techniques (such as Product Quantization, PQ) is used to construct the second vector index, supporting accurate retrieval by regional style. The local style feature vector set is dimensionality-reduced and encoded using a feature hashing algorithm (such as Locality-Sensitive Hashing, LSH) to construct the third feature index, improving the efficiency of local detail matching. Furthermore, the first, second, and third feature indices are linked and bound using unique image identifiers, ensuring that each level of index can be synchronously invoked for collaborative queries during retrieval, forming a structurally complete multi-level style feature index.

[0072] S3. Obtain material query data, perform feature mapping on each modality data in the material query data, and generate pseudo-style attribute vectors and deterministic scores for each modality in the style semantic space; the material query data includes image modality data, text modality data and sketch modality data.

[0073] Optionally, during the feature mapping process of the material query data, image modal data is directly input into a pre-trained deep neural network for feature extraction, generating a first intermediate feature that is homologous to the global feature map structure. This first intermediate feature is then mapped to a unified style semantic space through a linear projection layer, resulting in a pseudo-style attribute vector. Since image data directly carries style information, its deterministic score is preset to a fixed value reflecting the stability of feature extraction. Text modal data is first segmented and semantically encoded using a BERT (Bidirectional Encoder Representations from Transformers) model, generating text semantic features containing text style description information. Then, a text style attribute mapping network composed of fully connected layers and activation functions (such as ReLU) maps the text semantic features to a text pseudo-style attribute vector. Simultaneously, a softmax function is used to evaluate the confidence of the mapping result, and the output probability value is the deterministic score of the text modal data. Sketch modal data is processed by a lightweight convolutional neural network (CNN) to extract structural features such as line structure and composition ratio, generating sketch structural feature vectors. These vectors are then projected onto the style semantic space using a preset projection matrix to obtain sketch pseudo-style attribute vectors. The deterministic score is obtained by normalizing the magnitude of the sketch structural feature vectors, reflecting the clarity and completeness of the sketch structural features.

[0074] S4. Calculate dynamic weights based on the pseudo-style attribute vectors and deterministic scores of each modality to generate dynamic fusion weights for each modality.

[0075] Optionally, the dynamic weight calculation first normalizes the deterministic scores of each mode using the sigmoid function: The deterministic scores are mapped to the [0,1] interval to obtain the initial basic weights for each modality. Intermodal difference is measured by calculating the cosine distance between any two pseudostyle attribute vectors of different modalities. The formula for calculating the cosine distance is: ,in , Modal With mode The pseudo-style attribute vector, where ・ represents the vector dot product. Let L2 norm represent the vector. Based on cosine distance, it is expressed by the formula: Complementarity coefficients are generated, where λ is a scale parameter used to control the decay rate of the coefficients with distance. The complementarity coefficients of all mode pairs are combined to form a complementarity coefficient matrix. The dynamic fusion weights are calculated using the following formula: Where η is the balance factor, used to adjust the degree of complementarity, and Z is the normalization factor, calculated as follows: This ensures that the sum of the dynamic fusion weights of all modalities is 1.

[0076] S5. Based on dynamic fusion weights, the pseudo-style attribute vectors of each modality are weighted and fused to generate a joint style query vector.

[0077] Optionally, the weighted fusion process directly adopts a linear weighted summation method, multiplying the pseudo-style attribute vector of each modality element-wise with the corresponding dynamic fusion weight, and then summing all the product results to generate a joint style query vector. The calculation formula is as follows: Where Q is the joint style query vector, For modality Dynamic fusion weights, For modality The pseudo-style attribute vector is generated. By dynamically adjusting the fusion weights, modalities with high deterministic scores and strong complementarity occupy higher weights in the joint query vector, fully integrating the effective style information of each modality and avoiding the impact of information bias from a single modality on the search results.

[0078] S6. Calculate the similarity between the joint style query vector and the multi-level style feature index to generate comprehensive similarity information between each source image and the joint style query vector.

[0079] Optionally, the joint style query vector is compared with the global style vector, regional style vector, and local style feature vector in the multi-level style feature index using cosine similarity calculation to obtain global similarity, regional similarity, and local similarity. Fixed weights are assigned based on the importance of each feature level, such as a global similarity weight of 0.5, a regional similarity weight of 0.3, and a local similarity weight of 0.2. The comprehensive similarity information between each source image and the joint style query vector is obtained by weighted summation, and the calculation formula is as follows: ,in, To assess overall similarity, , , These are the weights for global, regional, and local similarity, respectively. , , These are the corresponding similarity values. Cosine similarity is calculated using the formula: , where F is the feature vector of the image in the material library, and Q is the joint style query vector.

[0080] S7. Based on comprehensive similarity information, sort the results from high to low similarity and generate a list of search results.

[0081] Optionally, the comprehensive similarity information and corresponding image identifiers of all source images are collected. The source images are then sorted in descending order of comprehensive similarity value. For source images with the same comprehensive similarity value, they are sorted in reverse order of their entry time to ensure consistency of results. After sorting, a search result list is generated, containing image identifiers, comprehensive similarity values, and style matching key point descriptions. The style matching key point descriptions are automatically generated based on the matching dimension of the joint style query vector and the source image feature vector, to help users quickly determine the style fit of the source images.

[0082] The aforementioned intelligent retrieval method for multimodal art materials lays the foundation for efficient retrieval by extracting multi-level style features offline and constructing an index. It performs precise feature mapping and deterministic evaluation on multimodal query data, generates a joint query vector that comprehensively reflects the user's style needs by combining a dynamic weight fusion strategy, and then achieves deep fusion of multimodal information and fine-grained style matching through multi-level index collaborative similarity calculation and ranking. This effectively solves the problems of existing methods being unable to handle mixed intent queries and insufficient style fit, significantly improves the accuracy and professionalism of art material retrieval, and meets the precise needs of professional creation scenarios.

[0083] In one embodiment, such as Figure 2 As shown, S1 includes:

[0084] S11. For each image in the art resource library, perform feature extraction using a pre-trained deep neural network to generate a global feature map corresponding to each image.

[0085] Optionally, a deep neural network pre-trained on the ImageNet dataset can be used as the base model for feature extraction. For example, ResNet (Residual Network) or EfficientNet can be selected. First, the input image is standardized preprocessed, mapping pixel values ​​to a preset range and adjusting them to the input size required by the model. Then, low-level visual features and high-level semantic features of the image are gradually extracted through convolutional layers, batch normalization layers, and activation functions (such as ReLU). Finally, a global feature map with fixed dimensions is output through a global average pooling layer. This global feature map integrates the overall color distribution, texture features, main structure, and abstract semantic information of the image, providing a foundation for subsequent style attribute decoupling and multi-scale feature extraction. It is mainly based on the hierarchical feature extraction mechanism of deep neural networks, where lower-level networks capture basic features such as edges and textures, while higher-level networks aggregate to form global features with semantic expressive capabilities.

[0086] S12. Input the global feature map into the style attribute decoupling network, and perform attribute prediction through multiple parallel attribute prediction heads to generate attribute values ​​for each predefined style dimension.

[0087] Optionally, the style attribute decoupling network is built on a convolutional neural network architecture. Its core lies in separating feature information from different style dimensions through an attention mechanism. Multiple parallel attribute prediction heads employ a stacked structure of fully connected layers, with each prediction head corresponding to a set of predefined style dimensions. These predefined style dimensions need to be clearly defined based on the needs of the art creation scenario, specifically including color style dimensions (e.g., warm tones, cool tones, contrasting color combinations), brushstroke style dimensions (e.g., delicate brushstrokes, bold brushstrokes, ink wash brushstrokes), composition style dimensions (e.g., symmetrical composition, divergent composition, layered composition), and art style dimensions (e.g., cyberpunk, traditional Chinese style, impressionism). After the global feature map is input into this convolutional neural network, the attention mechanism focuses on the feature regions corresponding to each style dimension. Each attribute prediction head quantizes and encodes the target features through fully connected layers, outputting continuous value attributes for each predefined style dimension, thus achieving decoupling and precise characterization of style attributes.

[0088] S13. Concatenate the attribute values ​​output by all attribute prediction heads to generate a global style vector set for the image.

[0089] Optionally, following a predefined style dimension sorting order, the attribute values ​​output by all attribute prediction heads are concatenated into vectors. Specifically, the single-style-dimensional attribute values ​​output by each attribute prediction head constitute a one-dimensional vector, which is then concatenated sequentially in the order of color style, brushstroke style, composition style, and art movement to form a global style vector set whose dimension is equal to the sum of the lengths of all style-dimensional attribute vectors. Each element in this set corresponds to a complete style-dimensional feature representation, and each component of the vector quantifies and reflects the specific features of the image under that style dimension. For example, the components of the color style vector correspond to the proportion weight of different hues, and the components of the brushstroke style vector correspond to the distribution characteristics of various brushstrokes. This concatenation method achieves a comprehensive quantitative representation of the global style of the image.

[0090] S14. Based on the global feature maps corresponding to each image, feature extraction is performed through a local style encoder to generate a set of local style feature vectors for the image.

[0091] Optionally, the local style encoder employs an architecture combining a sliding window and convolutional layers. First, the size and stride of the sliding window are adaptively adjusted according to the image dimensions, ensuring the window completely covers the image without redundant overlap. Then, the image sub-block corresponding to each sliding window is input into a feature extraction unit composed of convolutional and pooling layers to extract local texture details, local color transitions, and small structural features from the sub-block. Each image sub-block, after processing by the feature extraction unit, outputs a fixed-dimensional local feature vector. All local feature vectors corresponding to sub-blocks are arranged in order of their spatial position in the image, forming a set of local style feature vectors. This sliding window mechanism enables refined feature capture of local image regions, compensating for the shortcomings of global feature maps in expressing detailed style, such as accurately extracting fine-grained style information like corner decorative textures and local color gradients.

[0092] S15. Based on the global feature maps corresponding to each image, feature extraction is performed through a region style encoder to generate a set of region style features for the image.

[0093] Optionally, an instance-based semantic segmentation algorithm is used to process the original image corresponding to the global feature map. For example, Mask R-CNN (Mask Region-based Convolutional Neural Networks) is selected. Predefined semantic region categories include subject regions, background regions, decorative element regions, and core visual regions. This algorithm automatically segments the pixel range of each semantic region in the image and generates a mask. Based on the generated mask, feature sub-maps corresponding to each semantic region are cropped from the global feature map. Then, convolution and pooling operations, consistent with global feature extraction, are applied to each feature sub-map to extract the overall style features of each region, generating a region style feature vector corresponding to each semantic region. The feature vectors of all semantic regions are combined in a predefined category order to form a region style feature set. This process focuses on the style differences between different semantic regions. For example, the style of the subject region may differ significantly from the style of the background region. Region-level feature extraction achieves a structured representation of the image style.

[0094] S16. Integrate the global style vector set, the local style feature vector set, and the regional style feature set into a multi-level style feature vector set for the image.

[0095] Optionally, the global style vector set, local style feature vector set, and regional style feature set are structurally integrated according to feature hierarchy. Specifically, each set is assigned a unique feature identifier to distinguish the feature type. The three types of feature sets are associated and stored in key-value pairs, where the key is the feature type identifier (global, local, or regional), and the value is the corresponding feature vector set. The integrated multi-level style feature vector set fully preserves the feature information of the image at the three levels of global overall style, local detail style, and regional structured style, and maintains semantic consistency and relevance between the sets. This facilitates the use of corresponding features for index construction and similarity calculation for different retrieval needs, achieving full-dimensional style feature coverage from macro to micro.

[0096] In the above embodiments, global features are extracted through a pre-trained deep neural network, and the style attribute decoupling network separates and quantifies the attributes of each style dimension. Then, fine-grained and structured style features are supplemented by local and regional encoders, and finally integrated into a multi-level style feature vector set. This comprehensively and accurately depicts the style information of the image, effectively solving the problem that traditional feature extraction is difficult to take into account both global and local, overall and structural aspects. It provides a fine-grained and multi-level style feature foundation for subsequent index construction and multimodal retrieval, and significantly improves the style matching accuracy of art material retrieval.

[0097] In one embodiment, S2 includes:

[0098] S21. Construct the first vector index for the global style vector set.

[0099] Optionally, considering the high-dimensional and dense nature of the global style vector set, L2 normalization is first performed on all vectors to eliminate scale differences between different vectors and ensure consistency in subsequent similarity calculations. A KD-tree (K-Dimensional Tree) is used to construct the index. Its technical principle is to recursively select the median as the split point in each dimension of the vector, gradually dividing the high-dimensional space into multiple non-overlapping subspaces, each corresponding to a node in the KD-tree. During construction, the subspace is cyclically split according to the vector dimensions until the number of vectors contained in each leaf node reaches a preset threshold. Simultaneously, the subspace boundary range and corresponding vector list are stored for each node. If the extremely high vector dimension causes a decrease in KD-tree retrieval efficiency, it can be replaced with a ball tree. Ball trees use hyperspheres as the basic splitting unit, which is more suitable for the clustering distribution characteristics of high-dimensional data. The resulting first vector index can support fast approximate nearest neighbor queries at the global style level, significantly reducing the computational cost of full vector comparison.

[0100] S22. Construct a second vector index for the set of regional style features.

[0101] Optionally, the regional style feature set is first grouped according to predefined semantic region categories. An inverted index is then constructed based on this grouping result. The key of the inverted index is the semantic region category identifier, and the value is the regional style feature vector of all images within that category and the corresponding image's unique identifier. For the vector set under each region category, Product Quantization (PQ) technology is used to optimize retrieval performance. The high-dimensional regional style vector is decomposed into multiple low-dimensional sub-vectors. A dedicated codebook is constructed for each sub-vector using the K-means clustering algorithm. The original vector is then encoded into a codebook index sequence corresponding to each sub-vector, forming a compact quantized vector representation. The inverted index and the quantized vector index are combined to construct a second vector index. This index supports quickly filtering out candidate image sets that meet the criteria by region category, and then performing precise similarity calculations within the candidate set using quantized vectors, achieving efficient and accurate retrieval in the regional style dimension.

[0102] S23. Construct a third feature index for the local style feature map.

[0103] Optionally, given the large number and focus on fine-grained details of the local style feature vector set, a third feature index is constructed using Locality-Sensitive Hashing (LSH) technology. The core principle is to design a series of hash functions that map similar local feature vectors in high-dimensional space to the same hash bucket with a high probability, while mapping dissimilar vectors to the same hash bucket has an extremely low probability. Specifically, a random projection hash function is chosen, projecting local feature vectors onto a low-dimensional binary space using multiple randomly generated projection matrices to generate fixed-length hash codes, each corresponding to a hash bucket. The hash codes of all local feature vectors, along with their corresponding image identifiers and spatial location information within the image, are stored in a hash table to form the third feature index. This index can quickly locate candidate sets of similar local features using hash codes, avoiding a complete comparison of all local vectors and significantly improving the efficiency of local detail style matching.

[0104] S24. Combine the first vector index, the second vector index, and the third feature index to generate a multi-level style feature index.

[0105] Optionally, an index association table is constructed using the image's unique identifier as the core association criterion. Each record contains the image's unique identifier and pointers to the first vector index item, the second vector index item, and the third feature index item. Specifically, the first vector index item pointer points to the node position corresponding to the global style vector of the image in a KD-tree or sphere tree; the second vector index item pointer points to the entries corresponding to the style vectors of each semantic region of the image in the inverted index; and the third feature index item pointer points to the hash bucket positions corresponding to all local feature vectors of the image in the hash table. This association table achieves logical binding of the three indexes, generating a multi-level style feature index. This ensures that during the retrieval process, global, regional, and local index queries can be triggered simultaneously based on the joint style query vector. The image's unique identifier is then used to integrate the retrieval results from each level, balancing retrieval efficiency with the comprehensiveness of multi-dimensional style matching.

[0106] In the above embodiments, adaptive indexing techniques are adopted for the different characteristics of the three types of style features: global, regional, and local. A multi-level style feature index with a complete structure is formed by associating with unique image identifiers. This enables fast retrieval across all dimensions, from overall style to local details and from structured regions to fine-grained elements. It effectively solves the problem that traditional single indexes cannot balance the efficiency of high-dimensional feature retrieval with the accuracy of multi-dimensional matching. It provides efficient and comprehensive index support for multimodal retrieval, ensuring the speed of the retrieval process and the accuracy of the results.

[0107] In one embodiment, S3 includes:

[0108] S31. When the material query data is image-type modal data, feature extraction is performed on the image-type modal data through a feature extraction network to generate the first intermediate feature.

[0109] Optionally, a pre-trained deep neural network, derived from the global feature extraction in S1, is used as the feature extraction network. First, preprocessing operations consistent with those in the source library are performed on the image-type modal data, including pixel value standardization and resizing to fit the network input requirements. Low-level visual features and high-level semantic features of the image are extracted layer by layer through convolutional layers, batch normalization layers, and activation functions. A first intermediate feature with fixed dimensions is output through a global average pooling layer. This feature fully preserves the style-related information of the query image, including global attributes such as overall color distribution, brushstroke features, and compositional structure, as well as key information such as local texture details. This provides a high-quality feature foundation for style semantic space mapping, ensuring consistency with the feature extraction logic of the source library images and guaranteeing the comparability of style features across images.

[0110] S32. Perform style semantic space mapping on the first intermediate feature to generate a pseudo-style attribute vector, and set the deterministic score of the image class modality data to a fixed value.

[0111] Optionally, the style semantic space is a pre-defined high-dimensional vector space, to which the style feature vectors of all images in the resource library are mapped to achieve a unified metric. A trained linear projection layer is used to map the first intermediate features. The parameters of this projection layer are optimized through supervised training of image features and style attribute labels in the resource library, which can accurately project the first intermediate features into the style semantic space, generating pseudo-style attribute vectors with the same dimension as the global style vectors in the resource library. Since image-type modal data directly and intuitively carries style information, the feature extraction process has high stability and reliability. Therefore, its deterministic score is set to a fixed value. This fixed value is determined based on the feature extraction experimental results of a large number of image data and is used to reflect the inherent credibility of image-type data in style expression.

[0112] S33. When the material query data is text modal data, the text modal data is used to extract features through a text encoder to generate text semantic features.

[0113] Optionally, the text encoder uses a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model. First, the text modal data is preprocessed, including word segmentation, adding special markers (such as [CLS] and [SEP]), and generating word embedding vectors, which integrate lexical semantics and positional information. The model's multi-head self-attention mechanism captures the dependencies between words in the text, mining the semantic associations between style keywords such as "cyberpunk" and "ink wash style" and their modifiers. Then, a feedforward neural network performs a non-linear transformation on the attention-weighted features, finally outputting the vector corresponding to the [CLS] marker as the text semantic feature. This feature comprehensively covers the stylistic intent and semantic information of the text description.

[0114] S34. The semantic features of the text are mapped through a text style attribute mapping network to generate a pseudo-style attribute vector of the text. The confidence score of the generated text is then used as the deterministic score of the text modality data through confidence evaluation.

[0115] Optionally, the text style attribute mapping network consists of multiple fully connected layers and activation functions (such as the ReLU function). Its network parameters are obtained through pairing text style description data with corresponding image style feature vectors during training. This transforms text semantic features into vector representations in the style semantic space, generating text pseudo-style attribute vectors with the same dimension as the image pseudo-style attribute vectors. Confidence evaluation is achieved using the softmax function, which normalizes the probability of the text style attribute mapping network's output. The formula is: ,in Let i be the i-th component of the output vector of the mapping network. The probability value output is the text confidence score, which is the sum of the exponents of all components. This score reflects the reliability of the match between the text description and the style attribute vector, and is directly used as the deterministic score of the text modality data.

[0116] S35. When the material query data is sketch modal data, the sketch modal data is used to extract features through a sketch encoder to generate a sketch structure feature vector.

[0117] Optionally, the sketch encoder employs a lightweight convolutional neural network architecture. First, it performs binarization preprocessing on the sketch modality data, setting line areas as foreground and blank areas as background, then adjusts them to a uniform size to fit the encoder input. Multiple convolutional layers extract the line structure features of the sketch, including key information such as line direction, curvature, intersection distribution, and line segment length ratios. Zero padding is used during convolution to maintain the feature map size, and pooling layers reduce the feature dimensionality while preserving key structural information. Finally, a fixed-dimensional sketch structure feature vector is output through a fully connected layer. This sketch structure feature vector focuses on the sketch's geometric structure and compositional logic, ignoring non-style-related detail noise, thus providing a structured feature foundation for style attribute mapping.

[0118] S36. Project the sketch structure feature vectors using a preset projection matrix to generate sketch pseudo-style attribute vectors, and calculate the deterministic score of sketch modal data based on the modulus of the sketch structure feature vectors.

[0119] Optionally, the preset projection matrix is ​​a parameter matrix obtained through optimization training using a triplet loss function. Its dimensions match the dimensions of the sketch structure feature vector and the style semantic space, enabling linear projection of the sketch structure feature vector onto the style semantic space to generate a pseudo-style attribute vector for the sketch, thus realizing the transformation of sketch structure information into style features. The determination score is calculated by first solving for the L2 modulus of the sketch structure feature vector, using the following formula: ,in The i-th component of the sketch structure feature vector is then mapped to the [0,1] interval using the sigmoid function to obtain the deterministic score. The larger the modulus, the clearer the sketch structure and the more explicit the style expression, and the higher the corresponding deterministic score. Conversely, the smaller the modulus, the lower the deterministic score, thus accurately quantifying the reliability of the style expression of the sketch modal data.

[0120] In the above embodiments, images, text, and drafts are targeted. Figure 3Based on the characteristics of modal data, appropriate feature extraction and style semantic mapping strategies are adopted to generate pseudo-style attribute vectors in a unified style semantic space. The reliability of each modal data is quantified by calculating differentiated deterministic scores, which realizes the standardization and accurate representation of multimodal query information. This effectively solves the problem of the difficulty in uniformly measuring heterogeneous modal information and improves the adaptability and accuracy of multimodal art material retrieval.

[0121] In one embodiment, prior to S36, the following is also included:

[0122] S361. Construct training sample triplets; the training sample triplets include anchor sketch modal data, positive example image data with similar structure and style to the anchor sketch modal data, and negative example image data with similar structure but mismatched style to the anchor sketch modal data.

[0123] Optionally, sketches covering different composition types and line complexities are randomly selected from a massive sketch dataset as anchor sketch modal data to ensure sample diversity and representativeness. Structural similarity is determined based on the cosine similarity of the sketch's structural feature vectors. By calculating the similarity between the structural feature vectors of candidate sketches and anchor sketches, image data corresponding to sketches with similarity higher than a preset threshold are selected as the structural similarity candidate set. Style matching is determined by comparing the global style vector of the image data with the target style vector corresponding to the anchor sketch. The expected style of the anchor sketch is determined by manual annotation. In the structural similarity candidate set, images with a global style vector similarity higher than a preset threshold are selected as positive example image data. After excluding positive example images from the structural similarity candidate set, images with a global style vector similarity lower than a preset threshold are selected as negative example image data. Finally, a training sample triplet containing an anchor, a positive example, and a negative example is formed for each group.

[0124] S362. Based on anchor point sketch modal data, positive example image data, and negative example image data, the projection matrix is ​​optimized and trained by minimizing the triplet loss function to obtain the optimized projection matrix parameters; the triplet loss function is:

[0125]

[0126] in, This is the pseudo-style attribute vector of the sketch obtained by projecting the anchor sketch modal data. This is the global style vector extracted from positive example image data. This is the global style vector extracted from negative example image data. Let cosine distance function, These are boundary parameters.

[0127] Optionally, the projection matrix is ​​first initialized as a matrix with a random normal distribution, where the number of rows equals the dimension of the sketch structure feature vector and the number of columns equals the dimension of the style semantic space. Anchor sketch modal data is input into the sketch encoder to generate sketch structure feature vectors, which are then projected through the current projection matrix to obtain the sketch pseudo-style attribute vector. Simultaneously, extract the global style vector from the positive example image data. Global style vectors of negative example image data Cosine distance function The calculation formula is: Where a and b are input vectors, and ・ denotes the vector dot product. Represents the L2 norm of a vector. Boundary parameters. This is used to control the interval between positive and negative examples, ensuring that the distance between the anchor point and the positive example is small enough, and the distance from the negative example is large enough after training. A gradient descent optimizer (such as the Adam optimizer, or Adaptive Moment Estimation) is used to calculate the gradient of the loss function with respect to the projection matrix parameters. The projection matrix parameters are updated through backpropagation. This process is iterated until the loss function value converges to a stable range, resulting in the optimized projection matrix parameters.

[0128] S363. Construct a preset projection matrix based on the optimized projection matrix parameters.

[0129] Optionally, the converged projection matrix parameters after optimized training are stored in a binary file format to preserve parameter accuracy. When loading the stored parameters to construct the preset projection matrix, it is necessary to ensure that the matrix dimensions are consistent with those during training, i.e., the number of rows matches the dimension of the sketch structure feature vector, and the number of columns matches the dimension of the style semantic space. Further, the constructed projection matrix is ​​validated using a small number of validation sample triples. The distance relationship between the pseudo-style attribute vector of the anchor sketch in the validation set after projection and the global style vectors of positive and negative examples is calculated to confirm that the distance to positive examples is less than the distance to negative examples and the interval is greater than the boundary parameters. The requirement is to ensure that the projection matrix can accurately map the sketch structural features to the style semantic space.

[0130] In the above embodiments, by constructing training sample triples with similar structures and clear style differences, the projection matrix is ​​optimized using the triple loss function. This enables the optimized projection matrix to accurately map the sketch structure features to the style semantic space, effectively aligning the style representation of sketches with that of images. This solves the problem of inconsistent style semantics between sketch modalities and other modalities, providing key technical support for the effective retrieval of sketch modal data and improving the accuracy and reliability of sketch queries in multimodal retrieval.

[0131] In one embodiment, S4 includes:

[0132] S41. Based on the deterministic scores of each modality, the initial weights are calculated using a normalization function to generate the initial basic weights for each modality.

[0133] Optionally, the softmax normalization function is used to process the deterministic scores of each mode to calculate the initial base weights. The core principle of the softmax function is to amplify the differences between scores through exponential operations, and then perform normalization to make the sum of the initial base weights of all modes equal to 1. The specific calculation method is as follows: ,in For modality The initial base weights, For modality Deterministic scores, The sum of the exponential scores for all modal deterministic scores. This process first collects data from image classes, text classes, and grass classes. Figure 3 The deterministic scores of each modality are substituted into the above formula, converted to non-negative values ​​through exponential operations, and then weighted proportionally so that modalities with higher deterministic scores receive higher initial base weights, objectively reflecting the credibility of each modality's style expression.

[0134] S42. Calculate the intermodal difference measure based on the pseudostyle attribute vectors of each modality, and generate the complementarity coefficient matrix.

[0135] Optionally, the pseudo-style attribute vectors of all modalities are traversed, and a difference metric is calculated for any two vectors of different modalities (modality m and modality n). The difference metric uses the cosine distance function, and its calculation formula is as follows: ,in , Modal With mode The pseudo-style attribute vector, where ・ represents the vector dot product. L2 norm and cosine distance of a vector The value range is [0,1]. A larger value indicates a greater difference in style expression between the two modalities and stronger complementarity. Based on the cosine distance using the formula: Generate complementarity coefficients, where λ is a scale parameter used to control the rate at which the coefficients decay with distance. The value range is (0,1]. The larger The smaller, (1- The larger the value, the stronger the complementarity. The complementarity coefficients for all mode pairs (m,n) are calculated. Arranged sequentially by rows and columns, they form a two-dimensional complementarity coefficient matrix, where the number of rows and columns is equal to the number of modal types.

[0136] S43. Based on the initial base weights and complementarity coefficient matrix, the fusion weights are calculated using a weight adjustment function to generate dynamic fusion weights for each modality; the calculation method for dynamic fusion weights is as follows:

[0137]

[0138] in, For dynamic weight fusion, For modality The initial base weights, As a balance factor, For modality With mode The complementarity coefficient between them For modality The initial base weights, This is the normalization factor.

[0139] Optionally, the calculation of dynamic fusion weights first clarifies the definition and function of each parameter: the balance factor η is used to adjust the influence of complementarity on the weights, ensuring a balance between its own credibility and intermodal complementarity; the normalization factor Z is the sum of the numerators of all modes, used to ensure that the final sum of all dynamic fusion weights is 1. The specific calculation process is as follows: for each mode m, first calculate the (1- ) of all other modes n (n≠m). )and The sum of the products of, i.e. The sum of these products reflects the complementary contributions of other modes to mode m; after multiplying this contribution value by the balance factor η, it is combined with the initial base weights of mode m itself. Add them together to obtain the weight numerator of that mode; then calculate the sum of the numerators of all modes to obtain the normalization factor: Finally, the numerator of mode m is divided by Z to obtain the dynamic fusion weight of mode m. This process integrates the credibility of a modality itself with the complementarity between modalities to achieve dynamic adjustment of weights, so that modalities with high credibility and high complementarity occupy more reasonable weights in the joint query vector.

[0140] In the above embodiments, the deterministic scores are first converted into initial basic weights through a normalization function, and then a complementarity coefficient matrix is ​​generated through an intermodal difference measure. Finally, the two are combined and a dynamic fusion weight is calculated through a weight adjustment function. This achieves adaptive weight allocation based on the credibility and complementarity of each modality, effectively avoiding the limitations of single weight allocation, making multimodal information fusion more reasonable and targeted, providing accurate weight support for the generation of joint style query vectors, and thus improving the style fit of the search results.

[0141] In one embodiment, S42 includes:

[0142] S421. Calculate the similarity between pseudo-style attribute vectors of any two different modalities and generate the cosine distance between the modalities.

[0143] Optionally, all pseudo-style attribute vector combinations of different modalities are automatically traversed first to ensure coverage of all non-repeating modality pairs, such as image-text, image-sketch, and text-sketch, avoiding omissions or duplicate calculations. For each target modality pair, i.e., modality m and modality n, the dot product of the two vectors is first calculated, which reflects the directional consistency of the vectors in the style semantic space; then, the L2 norm of the two vectors is calculated separately to normalize the dot product result and eliminate the influence of vector scaling. The formula for calculating cosine similarity is: ,in , Let be the pseudo-style attribute vectors for modal m and modal n, respectively, and · denote the dot product operation. Represents the L2 norm. The intermodal cosine distance is expressed through... The value range is [0,1]. The larger the distance, the more significant the difference in style semantic expression between the two modalities.

[0144] S422. Based on the cosine distance between modes, perform coefficient mapping to generate the corresponding complementarity coefficients; the calculation method for the complementarity coefficients is as follows:

[0145]

[0146] in, The complementarity coefficient, For modality With mode The cosine distance between them This is the scale parameter.

[0147] Optionally, the scaling parameter λ is a preset positive number used to adjust the sensitivity of the complementarity coefficient to changes in the intermodal cosine distance. A larger value of λ indicates a more significant impact of small changes in the cosine distance on the complementarity coefficient, and vice versa. During calculation, the intermodal cosine distance is first... A squaring operation is performed to enhance the representation of distance difference. The result is then multiplied by -λ, and the result is mapped to the (0,1] interval using the exponential function exp(・), finally yielding the complementarity coefficient. The core of this mapping logic lies in the fact that when... When the difference between the two modes is large (significant difference and strong complementarity), The value is smaller, corresponding to (1- A larger value can highlight the complementary contribution of the mode pair in subsequent weight calculations; when When the size is small (the two modalities are similar in expression and have weak complementarity), The value is close to 1, (1- The value is relatively small to avoid over-computing the contributions of similar modes and to ensure the rationality and smoothness of the complementarity metric.

[0148] S423. Combine the complementarity coefficients between all mode pairs into a complementarity coefficient matrix.

[0149] Optionally, a fixed sorting rule for the modalities is first defined, such as image type, text, and sketch. Based on this sorting, the row and column indices of the complementarity coefficient matrix are determined. The dimension of the matrix is ​​equal to the number of modal types involved in the retrieval; for example, three modal types correspond to a 3×3 matrix. When constructing the matrix, the row index corresponds to modality m, and the column index corresponds to modality n. Only the complementarity coefficients corresponding to modality pairs where m ≠ n are included. Fill in the corresponding positions in the matrix. No coefficients need to be filled in at positions of the same mode (m=n), as only the complementarity between different modes is considered. The accuracy of coefficient filling is automatically verified during the construction process to ensure the accuracy of each mode pair. Each matrix row and column index corresponds one-to-one with the matrix index, ultimately forming a well-structured and clearly indexed complementary coefficient matrix.

[0150] In the above embodiments, the differences in semantic styles of different modalities are accurately measured by cosine distance, and then transformed into quantified complementarity coefficients through exponential mapping. These coefficients are then combined into a structured matrix, which systematically and accurately quantifies the degree of complementarity between multiple modalities. This provides a reliable quantitative basis for dynamic weight allocation, effectively solves the problem of difficulty in accurately characterizing complementarity in multimodal information fusion, ensures the rationality and pertinence of subsequent weight adjustments, and thus improves the overall accuracy of multimodal retrieval.

[0151] In the aforementioned intelligent retrieval method, device, equipment, and medium for multimodal art materials, offline multi-level style feature extraction (including global, local, and regional levels) is first performed on images in the art material database, and an associated index is constructed. Then, for multimodal query data of images, text, and sketches, pseudo-style attribute vectors and deterministic scores in a unified style semantic space are generated through feature mapping. Based on the credibility and complementarity of each modality, fusion weights are dynamically calculated, and a weighted joint query vector is generated and matched and sorted with the multi-level index to form the retrieval results. Existing technologies have difficulty integrating multimodal heterogeneous fine-grained style semantics and cannot dynamically adjust weights. This technical solution achieves deep integration of multimodal information and fine-grained style matching through a unified semantic space mapping and dynamic weight fusion mechanism, effectively improving the style fit and accuracy of art material retrieval and meeting the precise needs of professional creation.

[0152] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0153] Based on the same inventive concept, this application also provides an intelligent retrieval device for implementing the aforementioned intelligent retrieval method for multimodal art materials. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the intelligent retrieval device for multimodal art materials provided below can be found in the limitations of the intelligent retrieval method for multimodal art materials described above, and will not be repeated here.

[0154] In one exemplary embodiment, such as Figure 3 As shown, a multimodal art resource intelligent retrieval device 10 is provided, comprising:

[0155] The offline feature extraction module 11 is used to acquire image data from the art material library, perform offline feature extraction on the image data, and generate a multi-level style feature vector set corresponding to each image in the art material library;

[0156] Index building module 12 is used to build an index based on a multi-level style feature vector set and generate a multi-level style feature index.

[0157] The multimodal feature mapping module 13 is used to acquire material query data, perform feature mapping on each modality data in the material query data, and generate pseudo-style attribute vectors and deterministic scores for each modality in the style semantic space; the material query data includes image modality data, text modality data and sketch modality data;

[0158] The dynamic weight calculation module 14 is used to perform dynamic weight calculation based on the pseudo-style attribute vector and deterministic score of each modality, and generate dynamic fusion weights corresponding to each modality.

[0159] The joint style fusion module 15 is used to perform weighted fusion of the pseudo-style attribute vectors of each modality based on dynamic fusion weights to generate a joint style query vector.

[0160] The similarity calculation module 16 is used to calculate the similarity based on the joint style query vector and the multi-level style feature index, and generate comprehensive similarity information between each material image and the joint style query vector.

[0161] The retrieval and sorting module 17 is used to sort the results from high to low based on the comprehensive similarity information and generate a list of retrieval results.

[0162] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the multimodal art material intelligent retrieval method as described above.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal art material intelligent retrieval method as described above.

[0164] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0165] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A multi-modal art material intelligent retrieval method, characterized in that, The method comprises: S1, acquiring image data in an art material library, performing offline feature extraction on the image data, and generating a multi-level style feature vector set corresponding to each image in the art material library; S2, indexing based on the multi-level style feature vector set to generate a multi-level style feature index; S3, acquiring material query data, performing feature mapping on each modality data in the material query data to generate a pseudo-style attribute vector and a certainty score of each modality in a style semantic space; the material query data includes image class modality data, text modality data, and sketch modality data; S4, dynamically calculating the weight based on the pseudo-style attribute vector and the certainty score of each modality to generate a dynamic fusion weight corresponding to each modality; S5, weighting and fusing the pseudo-style attribute vector of each modality based on the dynamic fusion weight to generate a joint style query vector; S6, calculating the similarity between the joint style query vector and the multi-level style feature index to generate comprehensive similarity information of each material image and the joint style query vector; S7, based on the comprehensive similarity information, sorting from high to low according to the similarity to generate a search result list.

2. The method of claim 1, wherein, The S1 comprises: S11, for each image in the art material library, performing feature extraction through a pre-trained deep neural network to generate a global feature map corresponding to each image; S12, inputting the global feature map into a style attribute decoupling network to perform attribute prediction through multiple parallel attribute prediction heads to generate attribute values of each predefined style dimension; S13, concatenating the attribute values output by all the attribute prediction heads to generate a global style vector set of the image; S14, based on the global feature map corresponding to each image, performing feature extraction through a local style encoder to generate a local style feature vector set of the image; S15, based on the global feature map corresponding to each image, performing feature extraction through a regional style encoder to generate a regional style feature set of the image; S16, integrating the global style vector set, the local style feature vector set, and the regional style feature set into a multi-level style feature vector set of the image.

3. The method of claim 2, wherein, The S2 comprises: S21, constructing a first vector index for the global style vector set; S22, constructing a second vector index for the regional style feature set; S23, constructing a third feature index for the local style feature map; S24, associating the first vector index, the second vector index, and the third feature index to generate the multi-level style feature index.

4. The method of claim 1, wherein, The S3 comprises: S31, when the material query data is image class modality data, performing feature extraction on the image class modality data through a feature extraction network to generate a first intermediate feature; S32, performing style semantic space mapping on the first intermediate feature to generate a pseudo-style attribute vector, and setting the certainty score of the image class modality data to a fixed value; S33, when the material query data is text modal data, performing feature extraction on the text modal data through a text encoder to generate text semantic features; S34, performing mapping on the text semantic features through a text style attribute mapping network to generate a text pseudo-style attribute vector, and taking a text confidence as a certainty score of the text modal data through confidence evaluation; S35, when the material query data is sketch modal data, performing feature extraction on the sketch modal data through a sketch encoder to generate a sketch structure feature vector; S36, performing projection transformation on the sketch structure feature vector through a preset projection matrix to generate a sketch pseudo-style attribute vector, and calculating a certainty score of the sketch modal data based on a modulus of the sketch structure feature vector.

5. The method of claim 4, wherein, Before the S36, further comprising: S361, constructing a training sample triple; the training sample triple includes anchor sketch modal data, positive example image data similar in structure and matching in style to the anchor sketch modal data, and negative example image data similar in structure but not matching in style to the anchor sketch modal data; S362, based on the anchor sketch modal data, the positive example image data, and the negative example image data, optimizing and training the projection matrix by minimizing a triple loss function to obtain optimized projection matrix parameters; the triple loss function is: wherein, is a sketch pseudo-style attribute vector obtained after projection of the anchor sketch modality data, is a global style vector extracted from the positive example image data, is a global style vector extracted from the negative example image data, is a cosine distance function, is a boundary parameter; S363, constructing the preset projection matrix based on the optimized projection matrix parameters.

6. The method of claim 1, wherein, The S4 comprises: S41, based on the certainty scores of each modality, performing initial weight calculation through a normalization function to generate initial basic weights of each modality; S42, performing inter-modal difference measurement calculation according to the pseudo-style attribute vectors of each modality to generate a complementarity coefficient matrix; S43, based on the initial basic weights and the complementarity coefficient matrix, performing fusion weight calculation through a weight adjustment function to generate dynamic fusion weights corresponding to the modalities; the calculation method of the dynamic fusion weights is: wherein, is a dynamic fusion weight, is an initial base weight for modality , is a balancing factor, is an initial base weight for modality , is a complementarity coefficient between modalities , is an initial base weight for modality , is a normalization factor.

7. The method of claim 1, wherein, The S42 comprises: S421, performing similarity calculation on the pseudo-style attribute vectors of any two different modalities to generate inter-modal cosine distances; S422, based on the inter-modal cosine distances, performing coefficient mapping to generate corresponding complementarity coefficients; the calculation method of the complementarity coefficients is: in, The complementarity coefficient, For modality With mode The cosine distance between them For scale parameters; S423, combining the complementarity coefficients between all modalities into the complementarity coefficient matrix.

8. A multi-modal art material intelligent retrieval device, characterized in that, The device comprises: An offline feature extraction module configured to acquire image data in an art material library, perform offline feature extraction on the image data, and generate a multi-level style feature vector set corresponding to each image in the art material library; An index construction module configured to perform index construction based on the multi-level style feature vector set to generate a multi-level style feature index; A multi-modal feature mapping module configured to acquire material query data, perform feature mapping on each modality data in the material query data, and generate pseudo-style attribute vectors and certainty scores of each modality in a style semantic space; the material query data includes image modal data, text modal data, and sketch modal data. a dynamic weight calculation module, configured to perform dynamic weight calculation based on the pseudo style attribute vectors of the modalities and the certainty scores, to generate dynamic fusion weights corresponding to the modalities; a joint style fusion module, configured to perform weighted fusion on the pseudo style attribute vectors of the modalities based on the dynamic fusion weights, to generate a joint style query vector; a similarity calculation module, configured to perform similarity calculation on the joint style query vector and the multi-level style feature index, to generate comprehensive similarity information of each material image and the joint style query vector; a retrieval and sorting module, configured to perform sorting according to similarity from high to low based on the comprehensive similarity information, to generate a retrieval result list. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 7.

Images