Multi-dimensional data intelligent retrieval matching method and system for graphic and text features

Abstract

The invention provides an intelligent retrieval matching method and system for multidimensional data of image-text features, and relates to the technical field of icon image retrieval. Comprising the following steps: extracting image features, text content and semantic features of an icon image by using a convolutional neural network, an image segmentation attention mechanism network, a converter optical character recognition model and a bidirectional semantic understanding model, constructing the extracted features into heterogeneous feature tensors, and performing singular value decomposition to obtain icon feature fingerprint vectors; and constructing a multi-level index based on locality sensitive hashing, realizing rapid retrieval, calculating visual, text and semantic similarities in combination with a deep metric learning model, weighting according to variances and discrimination coefficients of similarity features to obtain a comprehensive similarity score, and outputting a retrieval result with the highest similarity.

Description

Technical Field

[0001] The present invention relates to the technical field of icon image retrieval, and in particular to a multi-dimensional data intelligent retrieval and matching method and system for graphic and text features. Background Art

[0002] Icon retrieval is a key link in icon registration and management. The purpose is to find registered icons similar to the to-be-retrieved icon in the icon database to avoid icon infringement and duplicate registration. With the continuous increase in the number of icon applications, traditional retrieval methods based on text or image features are facing increasing challenges;

[0003] Existing icon retrieval technologies mainly rely on single-modal feature matching, such as retrieving based on features such as the shape, color, and texture of images or text-based literal descriptions. When dealing with complex icons, such as icons containing a combination of graphics and text, the retrieval accuracy and efficiency are relatively low. In addition, traditional retrieval methods are difficult to effectively handle icon deformation, noise, and occlusion, resulting in a decrease in the accuracy of retrieval results. At the same time, there is also a lack of semantic understanding ability, unable to accurately capture the meaning and connotation of icons, and it is difficult to distinguish between similar icons;

[0004] Therefore, there is an urgent need for a solution to solve the problems existing in the prior art. Summary of the Invention

[0005] Embodiments of the present invention provide a multi-dimensional data intelligent retrieval and matching method and system for graphic and text features, which can at least solve some problems existing in the prior art.

[0006] In the first aspect of the embodiments of the present invention, a multi-dimensional data intelligent retrieval and matching method for graphic and text features is provided, including:

[0007] Obtain the to-be-retrieved icon image and add it to a convolutional neural network for feature extraction. Combine multiple convolutional layers and pooling layers to extract a first feature map. Based on the first feature map, input the to-be-retrieved icon image into an image segmentation attention mechanism network. Determine the spatial dependence relationship of the image region through the spatial attention mechanism. Combine the channel attention module to generate an attention weight map and output the graphic region and the text region. Add the text region to the transducer optical character recognition model, convert it into a feature sequence through the encoder, and perform character-by-character decoding through the multi-head attention mechanism in the decoder to obtain the text content. Extract features from the text content through a bidirectional semantic understanding model to obtain a text semantic feature vector. Add the graphic region to the vision transformer network, perform cross-layer transfer of features based on the dense connection structure, and combine the residual connection and the feature pyramid structure for feature fusion to obtain an image feature vector;

[0008] Construct the image feature vector, the text content, and the text semantic feature vector into a heterogeneous feature tensor. Perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and factor matrices. Perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector. Construct a locality-sensitive hashing index based on the icon feature fingerprint vector. Calculate optimal hash function parameters in combination with the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density. Encode the icon feature fingerprint vector in combination with position-sensitive encoding to obtain an encoded fingerprint vector. Construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer, and add the icon feature fingerprint vector to the fine-grained index layer to obtain an icon feature multi-dimensional index database;

[0009] Receive a retrieval request and obtain a target icon image. Perform feature extraction and encoding on the target icon image to generate a target feature fingerprint vector and perform a quick search in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. Calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer. Determine the feature similarity based on the feature distance and select the icon feature fingerprint vectors with a feature similarity greater than a pre-set similarity threshold to generate a similarity candidate set, which is added to a deep metric learning model. Calculate the relative distance relationship through a triplet loss function, and constrain the feature space distribution in combination with a contrastive loss function to obtain a metric feature representation and calculate the visual similarity, text similarity, and semantic similarity. Determine the weight parameters by determining the variance and discriminant coefficient of each similarity feature, and determine the comprehensive similarity score through weighted summation. Sort in descending order and select the top 3 retrieval results as the similar retrieval results for output.

[0010] In an alternative embodiment,

[0011] Obtain a to-be-retrieved icon image and add it to a convolutional neural network for feature extraction. Combine multiple convolutional layers and pooling layers to extract a first feature map. Based on the first feature map, input the to-be-retrieved icon image into an image segmentation attention mechanism network. Determine the spatial dependence relationship of the image region through a spatial attention mechanism, generate an attention weight map in combination with a channel attention module, and output a graphic region and a text region. Add the text region to a transducer optical character recognition model, convert it into a feature sequence through an encoder, and perform character-by-character decoding through a multi-head attention mechanism in a decoder to obtain text content. Perform feature extraction on the text content through a bidirectional semantic understanding model to obtain a text semantic feature vector. Add the graphic region to a vision transformer network, perform cross-layer feature transfer based on a dense connection structure, and perform feature fusion in combination with a residual connection and a feature pyramid structure to obtain an image feature vector including:

[0012] Obtain the icon image to be retrieved and add it to a convolutional neural network, which is composed of multiple convolutional layers and pooling layers connected in sequence. The size of the convolutional kernel in each convolutional layer decreases gradually from front to back, and the number of convolutional kernels increases gradually from front to back. Feature extraction is performed on the icon image to be retrieved through the convolutional layer and the pooling layer to obtain a first feature map;

[0013] Based on the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network. The image segmentation attention mechanism network performs average pooling and max pooling on the first feature map in the channel dimension through a spatial attention mechanism to obtain a first feature map and a second feature map. After concatenating the first feature map and the second feature map in the channel dimension and performing convolutional processing, determine the spatial dependence relationship of the image region. Perform global average pooling and max pooling on the first feature map in the spatial dimension through a channel attention module to obtain a first feature vector and a second feature vector. Input the first feature vector and the second feature vector into a fully connected network to generate an attention weight map, and perform weighted processing on the first feature map based on the attention weight map to output a graphic region and a text region;

[0014] Add the text region to a transducer optical character recognition model. The encoder of the transducer optical character recognition model divides the text region into image patches and then performs linear projection to convert it into a feature sequence. The feature sequence undergoes multi-head attention processing to obtain encoded features, and the encoded features are input into the decoder. The decoder performs character-by-character decoding in an autoregressive manner through a multi-head attention mechanism to obtain the text content;

[0015] Input the text content into a bidirectional semantic understanding model. The bidirectional semantic understanding model uses a multi-layer stacked transducer encoder to perform bidirectional feature extraction on the text content through a multi-head attention mechanism, and combines residual connections to transmit information between layers to obtain a text semantic feature vector;

[0016] Add the graphic region to a vision transformer network. After the vision transformer network performs image patch division and linear transformation on the graphic region, it performs cross-layer feature transfer based on a dense connection structure, constructs a feature pyramid at different scales to extract multi-scale features, and combines residual connections to perform feature fusion on the multi-scale features to obtain an image feature vector.

[0017] In an alternative embodiment,

[0018] Adding the graphic region to a vision transformer network, where the vision transformer network performs image patch division and linear transformation on the graphic region, and then performs cross-layer feature transfer based on a dense connection structure, constructs a feature pyramid at different scales to extract multi-scale features, and combines residual connections to perform feature fusion on the multi-scale features to obtain an image feature vector includes:

[0019] Add the said graphic area to a pre - set transformer network, and perform size normalization processing through bilinear interpolation to obtain a normalized graphic area;

[0020] Divide the normalized graphic area into a sequence of image patches in a regular grid pattern. Convert the pixel matrix of each image patch into an image patch feature vector through a linear projection layer, assign a learnable position encoding vector to each image patch position, add the position encoding vector to the image patch feature vector to obtain a feature sequence integrating position information, and establish the spatial relationship between different image patches;

[0021] Construct a multi - layer transformer encoder network. The multi - layer transformer encoder network adopts a dense connection structure. Concatenate the feature maps output by each layer in the channel dimension as the input of the next layer, adjust the number of feature channels through a convolutional layer, and perform feature transfer between different levels to obtain a transferred feature map;

[0022] Based on the transferred feature map, construct a feature pyramid structure. Obtain large - scale feature maps and small - scale feature maps by performing pooling operations at different scales on the input features. The large - scale feature maps retain the image detail information, and the small - scale feature maps obtain the large - range receptive field information. Upsample the small - scale feature maps through a transposed convolution operation to unify the spatial resolutions of the large - scale feature maps and the upsampled small - scale feature maps;

[0023] Concatenate the feature maps of different scales with unified resolutions in the channel dimension to obtain multi - scale features. Adaptively fuse the multi - scale features through a convolutional layer, and adopt a residual connection mechanism to directly superimpose the input features of the current layer onto the output features to obtain a fused output feature;

[0024] Perform a global pooling operation on the fused output feature to compress the spatial dimension into a low - dimensional feature vector. Process the low - dimensional feature vector through a multi - layer non - linear transformation network. Introduce non - linear expression ability through an activation function after each transformation, and output an image feature vector containing local details and global semantics.

[0025] In an alternative embodiment,

[0026] Construct the image feature vector, the text content, and the text semantic feature vector into a heterogeneous feature tensor, perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and factor matrices, perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector, construct a locality-sensitive hashing index based on the icon feature fingerprint vector, calculate optimal hashing function parameters based on the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density, encode the icon feature fingerprint vector with position-sensitive encoding to obtain an encoded fingerprint vector, construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer, add the icon feature fingerprint vector to the fine-grained index layer, and obtain an icon feature multi-dimensional index database including:

[0027] Preprocess the image feature vector, the text content, and the text semantic feature vector. Perform normalization on the image feature vector to eliminate the scale difference between dimensions, perform word segmentation and encoding conversion on the text content to convert it into a fixed-length feature representation, perform normalization on the text semantic feature vector to make its distribution range consistent with other features, and organize the preprocessed features into a third-order heterogeneous feature tensor;

[0028] Perform matrix expansion on the heterogeneous feature tensor along the sample dimension, the feature type dimension, and the eigenvalue dimension respectively. Perform singular value decomposition on each expanded matrix to extract the main components, and recombine the main components according to a preset rank parameter to obtain a core tensor and factor matrices;

[0029] Calculate the correlation coefficients between different dimensions in the core tensor to construct a correlation coefficient matrix, perform eigenvalue decomposition on the correlation coefficient matrix to extract the principal components, calculate the contribution rate of each principal component, and select the combination of principal components with a contribution rate exceeding a preset threshold to form an icon feature fingerprint vector;

[0030] Divide the feature space into multiple regions, count the number of samples in each region to obtain the regional sample density distribution, determine the number of locality-sensitive hashing functions according to the regional sample density distribution, and perform fine-grained division on the regions with a sample density greater than the first density threshold by increasing the number of hashing functions, and perform coarse-grained division on the regions with a sample density less than the second density threshold by reducing the number of hashing functions;

[0031] Calculate the variance of each dimension of the icon feature fingerprint vector, sort the dimension importance according to the variance value, assign 16-bit binary encoding to the top 5 dimensions with the highest importance, assign 8-bit binary encoding to the dimensions after the fifth place, determine the binary encoding mapping relationship through eigenvalue interval division, and convert the icon feature fingerprint vector into an encoded fingerprint vector;

[0032] Construct a multi-level index structure. In the coarse-grained index layer, segment the encoded fingerprint vectors by 32-bit lengths. Calculate the 32-bit hash value for each segment as the index key to establish a hash table. The hash table points to buckets. When the hash values of the encoded fingerprint vectors are within the Hamming distance threshold of 8, store their indices in the same bucket. In the fine-grained index layer, construct the icon feature fingerprint vectors into a tree structure. Each node of the tree represents the numerical range of the feature vectors in the current dimension, obtaining an icon feature multi-dimensional index database.

[0033] In an alternative implementation,

[0034] Perform matrix unfolding on the heterogeneous feature tensor along the sample dimension, feature type dimension, and feature value dimension respectively. Perform singular value decomposition on each unfolded matrix to extract the main components. Recombine the main components according to a pre-set rank parameter to obtain a core tensor and factor matrices, including:

[0035] Construct the image features, text content, and text semantic features of the icon data into a third-order heterogeneous feature tensor. The third-order heterogeneous feature tensor includes a sample dimension, a feature type dimension, and a feature value dimension;

[0036] Unfold the third-order heterogeneous feature tensor along the sample dimension to obtain a first unfolded matrix. The number of rows of the first unfolded matrix is the number of samples, and the number of columns is the product of the feature type dimension and the feature value dimension. Unfold it along the feature type dimension to obtain a second unfolded matrix. The number of rows of the second unfolded matrix is the number of feature types, and the number of columns is the product of the sample dimension and the feature value dimension. Unfold it along the feature value dimension to obtain a third unfolded matrix. The number of rows of the third unfolded matrix is the total sum of the feature value dimension, and the number of columns is the product of the sample dimension and the feature type dimension;

[0037] Divide the first unfolded matrix, the second unfolded matrix, and the third unfolded matrix into sub-block matrices of equal size. The size of each sub-block matrix is one-tenth of the corresponding unfolded matrix;

[0038] Perform iterative singular value decomposition on each sub-block matrix. Randomly initialize a unit vector, multiply the unit vector by the sub-block matrix and normalize it to obtain an updated vector. Repeat the multiplication and normalization operations until the updated vector converges to obtain a first pair of singular vectors. Subtract the rank-one matrix constructed from the first pair of singular vectors from the sub-block matrix to obtain a remaining matrix. Repeat the multiplication and normalization operations on the remaining matrix to obtain the next pair of singular vectors;

[0039] Add L2 norm constraints to the singular vectors during the singular value decomposition process, and perform iterative optimization using a dynamic learning rate. The initial value of the dynamic learning rate is 0.1, and it is adaptively adjusted according to the change of the reconstruction error;

[0040] Select the largest singular value and its corresponding left and right singular vectors from the singular value decomposition results of each sub-block matrix according to the pre-set rank parameter, fuse the singular values and singular vectors screened by each sub-block matrix through a collaborative filtering algorithm, form a sample dimension factor matrix with the left singular vectors, and form a feature type dimension factor matrix and an eigenvalue dimension factor matrix with the right singular vectors;

[0041] Adopt a progressive strategy to construct an initial small-scale core tensor, calculate the reconstruction error, and gradually increase the scale of the core tensor when the reconstruction error is greater than the preset threshold and has not reached the maximum scale limit, and repeat the increase until the maximum scale limit is reached to obtain the core tensor.

[0042] In an alternative embodiment,

[0043] Receive a retrieval request and obtain a target icon image, perform feature extraction and encoding on the target icon image, generate a target feature fingerprint vector and perform a quick retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors, calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determine the feature similarity based on the feature distance and select the icon feature fingerprint vectors with the feature similarity greater than the pre-set similarity threshold to generate a similarity candidate set, add them to the deep metric learning model, calculate the relative distance relationship through the triplet loss function, combine the contrast loss function to constrain the feature space distribution, obtain the metric feature representation and calculate the visual similarity, text similarity and semantic similarity, determine the weight parameter by determining the variance and discriminant coefficient of each similarity feature, determine the comprehensive similarity score through weighted summation, sort in descending order and select the top 3 retrieval results as the similar retrieval results for output, including:

[0044] Receive a retrieval request and obtain a target icon image, perform feature extraction and normalization preprocessing on the target icon image, extract an image feature vector through a multi-layer convolutional network, extract a text vector and a semantic feature vector through a pre-trained language model, and splice the image feature vector, the text vector and the semantic feature vector to form a target feature fingerprint vector;

[0045] Segment the target feature fingerprint vector into multiple feature segments according to a preset length, and perform binarization processing on the feature values in each feature segment to obtain a binary code, where the binarization processing includes recording the feature values greater than the mean of the feature segment as a first identifier and recording the feature values less than the mean of the feature segment as a second identifier;

[0046] In the coarse-grained index layer, use locality-sensitive hashing to map the binary code to a hash value, look up the encoded fingerprint vector in the hash table whose distance from the hash value is less than a preset threshold, add the encoded fingerprint vector to the initial candidate set. If the number of samples in the storage area of the hash table exceeds the preset quantity threshold, increase the number of hash functions in the current storage area for region subdivision;

[0047] Traverse the encoded fingerprint vectors in the initial candidate set in the feature tree of the fine-grained index layer. The internal nodes of the feature tree represent regions in the feature space, and the leaf nodes store the icon feature fingerprint vectors. Calculate the feature distance between the target feature fingerprint vector and the center point of the node, select the child node with the closest feature distance to continue traversing until reaching the leaf node, and generate a similarity candidate set from the icon feature fingerprint vectors corresponding to the leaf nodes whose feature distance is less than the preset similarity threshold;

[0048] Add the similarity candidate set to the deep metric learning model. Among them, the deep metric learning model includes a multi-layer fully connected encoder and a metric network based on the attention mechanism. Select the sample with the closest distance to the target feature fingerprint vector from the similarity candidate set as the positive sample, the sample with the farthest distance as the negative sample, and use the target feature fingerprint vector as the anchor sample. Combine the anchor sample, the positive sample, and the negative sample to construct a triplet;

[0049] Calculate the relative distance relationship for the triplet through the triplet loss function, and combine the contrast loss function to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance from the negative sample. Calculate the feature representation similarity between icon pairs respectively, compare the similarity with the preset target similarity threshold, and screen the icon pairs to form a feature representation set;

[0050] Obtain the metric feature representation based on the deep metric learning model, calculate the visual similarity, text similarity, and semantic similarity between the icon feature fingerprint vectors in the feature representation set and the target feature fingerprint vector. Determine the weight parameters based on the variance and discriminant coefficient of each similarity feature in the training set, determine the comprehensive similarity score through weighted summation, sort the comprehensive similarity scores in descending order, and select the top 3 retrieval results as the similar retrieval results for output.

[0051] In an optional implementation manner,

[0052] Calculate the relative distance relationship for the triple through the triple loss function, and combine the contrast loss function to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance from the negative sample. Calculate the feature representation similarity between icon pairs respectively, and compare the similarity with a preset target similarity threshold to screen the icon pairs to form a feature representation set, including:

[0053] Receive a triple composed of a target feature fingerprint vector, a positive sample feature fingerprint vector, and a negative sample feature fingerprint vector, and input the triple into a deep metric learning model. The deep metric learning model includes a multi-layer fully connected encoder network and a metric network based on an attention mechanism. The encoder network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence. The output dimensions of the first fully connected layer, the second fully connected layer, and the third fully connected layer decrease in sequence. After each fully connected layer, a non-linear activation layer and a batch normalization layer are connected;

[0054] Calculate the Euclidean distance between the target feature fingerprint vector and the positive sample feature fingerprint vector through the metric network to obtain the positive sample distance, calculate the Euclidean distance between the target feature fingerprint vector and the negative sample feature fingerprint vector to obtain the negative sample distance, and compare the difference between the positive sample distance and the negative sample distance with a preset interval parameter;

[0055] Calculate the cosine similarity of the feature representation for the icon pairs in the triple. Set a first target similarity for icon pairs of the same category and a second target similarity for icon pairs of different categories. The first target similarity is greater than the second target similarity, and perform feature constraint based on the difference between the cosine similarity of the feature representation and the corresponding target similarity;

[0056] Calculate the loss value of the triple, select a preset number of triples with the largest loss value as hard samples, and use the hard samples to update the parameters of the encoder network and the metric network;

[0057] Perform feature mapping on the feature fingerprint vectors of the icon pairs in the similarity candidate set through the encoder network to obtain mapped features, calculate the cosine similarity between the mapped features through the metric network, and save the icon pairs with the cosine similarity greater than the preset similarity threshold to the feature representation set.

[0058] In the second aspect of the embodiments of the present invention, a multi-dimensional data intelligent retrieval and matching system for graphic and text features is provided, including:

[0059] The first unit is used to obtain the icon image to be retrieved, add it to a convolutional neural network for feature extraction, combine multiple convolutional layers and pooling layers to extract the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network based on the first feature map, determine the spatial dependence relationship of the image region through the spatial attention mechanism, generate an attention weight map in combination with the channel attention module, and output the graphic region and the text region. Add the text region to the transformer optical character recognition model, convert it into a feature sequence through the encoder, and perform character-by-character decoding through the multi-head attention mechanism in the decoder to obtain the text content. Extract the features of the text content through the bidirectional semantic understanding model to obtain the text semantic feature vector. Add the graphic region to the vision transformer network, perform cross-layer feature transfer based on the dense connection structure, and perform feature fusion in combination with the residual connection and the feature pyramid structure to obtain the image feature vector;

[0060] The second unit is used to construct a heterogeneous feature tensor from the image feature vector, the text content, and the text semantic feature vector, perform singular value decomposition on the heterogeneous feature tensor to obtain the core tensor and the factor matrix, perform correlation analysis decomposition on the core tensor to obtain the icon feature fingerprint vector, construct a locality-sensitive hashing index based on the icon feature fingerprint vector, calculate the optimal hash function parameters in combination with the distribution density of the icon feature fingerprint vector, and adjust the hash storage unit based on the distribution density. Encode the icon feature fingerprint vector in combination with the position-sensitive encoding to obtain the encoded fingerprint vector, construct a multi-level index structure, add the encoded fingerprint vector to the coarse-grained index layer, and add the icon feature fingerprint vector to the fine-grained index layer to obtain the icon feature multi-dimensional index database;

[0061] The third unit is used to receive a retrieval request and obtain the target icon image, perform feature extraction and encoding on the target icon image, generate a target feature fingerprint vector, and perform a quick retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. Calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determine the feature similarity based on the feature distance, and select the icon feature fingerprint vectors with the feature similarity greater than the pre-set similarity threshold to generate a similarity candidate set, add it to the deep metric learning model, calculate the relative distance relationship through the triplet loss function, and constrain the feature space distribution in combination with the contrastive loss function to obtain the metric feature representation, and calculate the visual similarity, text similarity, and semantic similarity. Determine the weight parameters by determining the variance and discriminant coefficient of each similarity feature, and determine the comprehensive similarity score through weighted summation, sort in descending order, and select the top 3 retrieval results as the similar retrieval results for output.

[0062] In the third aspect of the embodiments of the present invention,

[0063] Provided is an electronic device, comprising:

[0064] a processor;

[0065] a memory for storing instructions executable by the processor;

[0066] wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0067] In a fourth aspect of the embodiments of the present invention,

[0068] a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0069] In the present invention, by combining technologies such as image segmentation attention mechanism, transformer optical character recognition, and bidirectional semantic understanding model, the graphic features, text content, and semantic information of icon images can be more accurately extracted, and a multi-dimensional heterogeneous feature tensor is constructed. The local sensitive hashing technology is used to construct a multi-level index structure, and the position-sensitive coding and distribution density adaptive adjustment of hash storage units are combined to achieve fast retrieval of massive icon data. The deep metric learning model is used to re-rank the candidate set, and more discriminative feature representations are learned through the triplet loss function and the contrastive loss function, and comprehensive evaluation is performed by combining visual similarity, text similarity, and semantic similarity. Finally, the retrieval result with the highest similarity is output, ensuring the reliability of the retrieval result. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a schematic flow chart of a method for intelligent retrieval and matching of multi-dimensional data of graphic and text features in an embodiment of the present invention;

[0071] Figure 2 is a schematic structural diagram of a system for intelligent retrieval and matching of multi-dimensional data of graphic and text features in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0073] The technical solution of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0074] Figure 1 It is a schematic flowchart of a multi-dimensional data intelligent retrieval and matching method for graphic and text features in an embodiment of the present invention. As Figure 1 shown, the method includes:

[0075] S1. Obtain the icon image to be retrieved and add it to the convolutional neural network for feature extraction. Combine multiple convolutional layers and pooling layers to extract the first feature map. Based on the first feature map, input the icon image to be retrieved into the image segmentation attention mechanism network. Determine the spatial dependence relationship of the image region through the spatial attention mechanism, generate an attention weight map in combination with the channel attention module, and output the graphic region and the text region. Add the text region to the transformer optical character recognition model, convert it into a feature sequence through the encoder, and perform character-by-character decoding through the multi-head attention mechanism in the decoder to obtain the text content. Extract the features of the text content through the bidirectional semantic understanding model to obtain the text semantic feature vector. Add the graphic region to the vision transformer network, perform cross-layer transfer of features based on the dense connection structure, and perform feature fusion in combination with the residual connection and the feature pyramid structure to obtain the image feature vector;

[0076] The image segmentation attention mechanism network is a neural network that introduces the attention mechanism in the image segmentation task. By emphasizing the important regions in the image, it dynamically adjusts the focus of the network to improve the segmentation accuracy, especially in complex backgrounds or detail processing. The transformer optical character recognition model is a model that uses the transformer architecture for optical character recognition, and uses the self-attention mechanism to handle the dependency relationship of the character sequence. The bidirectional semantic understanding model is a model that can consider context information simultaneously, usually based on a bidirectional neural network structure, and can capture the dependency relationship before and after words or sentences, and is widely used in natural language processing tasks. The vision transformer network is a vision processing network based on the transformer model, which uses the self-attention mechanism to model the global information of each part of the image, can handle long-range dependencies, and is especially suitable for processing large-scale images and complex vision tasks. The dense connection structure means that each layer in the network is connected to all the previous layers, which improves the reusability of features and the efficiency of the gradient flow, and reduces the problem of gradient disappearance in network training.

[0077] In an alternative embodiment,

[0078] Obtain the icon image to be retrieved and add it to a convolutional neural network for feature extraction. Combine multiple convolutional layers and pooling layers to extract the first feature map. Based on the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network. Determine the spatial dependence relationship of the image region through the spatial attention mechanism. Combine the channel attention module to generate an attention weight map and output the graphic region and the text region. Add the text region to the transformer optical character recognition model, convert it into a feature sequence through the encoder, and perform character-by-character decoding through the multi-head attention mechanism in the decoder to obtain the text content. Extract features from the text content through a bidirectional semantic understanding model to obtain a text semantic feature vector. Add the graphic region to the vision transformer network, perform cross-layer feature transfer based on the dense connection structure, and combine the residual connection and the feature pyramid structure for feature fusion to obtain an image feature vector, including:

[0079] Obtain the icon image to be retrieved and add it to a convolutional neural network, which is composed of multiple convolutional layers and pooling layers connected in sequence. The convolutional kernel size in each convolutional layer decreases gradually from front to back, and the number of convolutional kernels increases gradually from front to back. Extract the first feature map from the icon image to be retrieved through the convolutional layer and the pooling layer;

[0080] Based on the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network. The image segmentation attention mechanism network performs average pooling and max pooling on the first feature map in the channel dimension through the spatial attention mechanism to obtain the first feature map and the second feature map. After concatenating the first feature map and the second feature map in the channel dimension and performing convolutional processing, determine the spatial dependence relationship of the image region. Perform global average pooling and max pooling on the first feature map in the spatial dimension through the channel attention module to obtain the first feature vector and the second feature vector. Input the first feature vector and the second feature vector into a fully connected network to generate an attention weight map. Based on the attention weight map, perform weighted processing on the first feature map and output the graphic region and the text region;

[0081] Add the text region to the transformer optical character recognition model. The encoder of the transformer optical character recognition model divides the text region into image patches and performs linear projection to convert it into a feature sequence. The feature sequence undergoes multi-head attention processing to obtain encoded features. The encoded features are input into the decoder, and the decoder performs character-by-character decoding in an autoregressive manner through the multi-head attention mechanism to obtain the text content;

[0082] Input the text content into a bidirectional semantic understanding model. The bidirectional semantic understanding model uses a multi-layer stacked transformer encoder to perform bidirectional feature extraction on the text content through the multi-head attention mechanism, and combines the residual connection to transfer information between layers to obtain a text semantic feature vector;

[0083] Add the said graphic region to a vision transformer network. After the vision transformer network performs image patch division and linear transformation on the graphic region, it conducts cross-layer feature transfer based on a dense connection structure, constructs a feature pyramid at different scales to extract multi-scale features, and combines residual connections to perform feature fusion on the multi-scale features to obtain an image feature vector.

[0084] The bidirectional feature extraction refers to extracting features from the front and back information of the input sequence simultaneously through a bidirectional network structure (such as a bidirectional LSTM or a bidirectional transformer), which has wide applications in fields such as natural language processing and speech recognition and can improve the ability to capture context information. The character-by-character decoding is a method of decoding at the character level and is often used in optical character recognition tasks. By decoding each character one by one instead of processing the entire sentence or paragraph at the same time, the recognition accuracy and controllability can be improved.

[0085] Obtain the icon image to be retrieved. For example, the user uploads an icon image to be retrieved, with the format of PNG and the size of 256x256 pixels.

[0086] Input the icon image to be retrieved into a pre-trained convolutional neural network (CNN). The convolutional neural network contains multiple convolutional layers and pooling layers. For example, the convolutional neural network contains 4 convolutional layers and 3 pooling layers. The first convolutional layer uses 64 convolutional kernels with a size of 7x7, the second convolutional layer uses 128 convolutional kernels with a size of 5x5, the third convolutional layer uses 256 convolutional kernels with a size of 3x3, and the fourth convolutional layer uses 512 convolutional kernels with a size of 3x3. All pooling layers adopt max pooling with a size of 2x2. The convolutional layers and pooling layers are alternately connected, and a pooling layer is connected after each convolutional layer. After the processing of the convolutional layers and pooling layers, a first feature map is extracted, with the size of 16x16x512.

[0087] Input the first feature map and the original 256x256 icon image into the image segmentation attention mechanism network. The image segmentation attention mechanism network includes a spatial attention mechanism and a channel attention module. The spatial attention mechanism performs average pooling and max pooling on the first feature map respectively in the channel dimension to obtain two 16x16x1 feature maps. After concatenating these two feature maps in the channel dimension and processing them through a 3x3 convolutional layer, a 16x16x1 spatial attention map is obtained, which reflects the spatial dependence relationship of the image regions. The channel attention module performs global average pooling and max pooling on the first feature map respectively in the spatial dimension to obtain two 1x1x512 feature vectors. These two feature vectors are respectively input into two fully connected layers with ReLU activation functions, and then the output results are added and passed through a Sigmoid activation function to generate a 1x1x512 channel attention weight map. Multiply the spatial attention map and the channel attention weight map, and then multiply the result with the first feature map to obtain a weighted feature map. Finally, through a segmentation network, such as U-Net, process the weighted feature map to output the segmentation results of the graphic region and the text region, such as two 256x256 binary images, where the pixels with a value of 1 represent the corresponding regions.

[0088] Input the text region image into the Transformer optical character recognition model. The encoder of this model divides the text region image into 16x16 image patches, and linearly projects each image patch into a feature sequence. After processing this feature sequence through the multi-head attention mechanism, encoded features are obtained. The encoded features are input into the decoder, and the decoder decodes character by character in an autoregressive manner through the multi-head attention mechanism to obtain the text content. For example, the recognition result is "Example Icon".

[0089] Input the text content into a bidirectional semantic understanding model, such as BERT. The bidirectional semantic understanding model uses a multi-layer stacked Transformer encoder to perform bidirectional feature extraction on the text content through the multi-head attention mechanism, and combines residual connections to transfer information between layers to obtain a text semantic feature vector, such as a 768-dimensional vector.

[0090] Input the graphic region image into the Vision Transformer network (ViT). This network divides the graphic region image into 16x16 image patches, performs a linear transformation, transfers features across layers based on a dense connection structure, constructs a feature pyramid at different scales to extract multi-scale features, and combines residual connections to perform feature fusion on the multi-scale features to obtain an image feature vector, such as a 1024-dimensional vector.

[0091] In this embodiment, by combining image features and text semantic features for retrieval, similar icons can be more accurately matched, the misjudgment rate can be reduced, the text content and graphic details in the icons can be recognized, fine-grained retrieval based on text, graphics, or a combination of both is supported, and the diverse retrieval needs of users can be met. Even when there are situations such as deformation, noise, or occlusion in the icon image, this method can still effectively extract features and perform matching, improving the robustness of the retrieval.

[0092] In an alternative embodiment,

[0093] Add the graphic region to the vision transformer network. After the vision transformer network performs image patch division and linear transformation on the graphic region, cross-layer feature transfer is performed based on a dense connection structure, and multi-scale features are extracted by constructing a feature pyramid at different scales. Feature fusion of the multi-scale features is performed by combining residual connections to obtain an image feature vector, including:

[0094] Add the graphic region to a pre-set transformer network, and perform size normalization processing through bilinear interpolation to obtain a normalized graphic region;

[0095] Divide the normalized graphic region into a sequence of image patches in a regular grid pattern, convert the pixel matrix of each image patch into an image patch feature vector through a linear projection layer, assign a learnable position encoding vector to each image patch position, add the position encoding vector to the image patch feature vector to obtain a feature sequence integrating position information, and establish the spatial relationship between different image patches;

[0096] Construct a multi-layer transformer encoder network. The multi-layer transformer encoder network adopts a dense connection structure. The feature maps output by each layer are concatenated in the channel dimension and used as the input of the next layer. The number of feature channels is adjusted through a convolutional layer, and feature transfer is performed between different levels to obtain a transferred feature map;

[0097] Based on the transferred feature map, construct a feature pyramid structure. The input feature is subjected to pooling operations at different scales to obtain a large-scale feature map and a small-scale feature map. The large-scale feature map retains the image detail information, and the small-scale feature map obtains the large-range receptive field information. The small-scale feature map is upsampled through a deconvolution operation to unify the spatial resolution of the large-scale feature map and the upsampled small-scale feature map;

[0098] Concatenate the feature maps of different scales with unified resolution in the channel dimension to obtain multi-scale features. The multi-scale features are adaptively fused through a convolutional layer, and the residual connection mechanism is used to directly superimpose the current layer input feature onto the output feature to obtain a fused output feature;

[0099] Perform a global pooling operation on the fused output features to compress the spatial dimension into a low-dimensional feature vector, and process the low-dimensional feature vector through a multi-layer non-linear transformation network. After each transformation, introduce non-linear expression ability through an activation function, and output an image feature vector containing local details and global semantics.

[0100] The pixel matrix refers to the matrix composed of the numerical values of each pixel in an image. Usually, the pixel values of an image are represented in the form of a two-dimensional array, which is widely used in fields such as image processing and computer vision. The transposed convolution operation is an operation in a convolutional neural network, which is used to upsample an image through the inverse process of the convolution operation to restore the spatial resolution of the image, and is often used in generative adversarial networks and image generation tasks.

[0101] Obtain the graphic area to be processed and add it to a pre-trained vision transformer network. This vision transformer network has been trained on a large-scale dataset and has powerful image feature extraction capabilities. To adapt to the input size of the network, use the bilinear interpolation method to perform size normalization processing on the graphic area. For example, scale a 1024x768 graphic area to 224x224.

[0102] Divide the normalized graphic area into a sequence of regularly gridded image patches. For example, divide a 224x224 graphic area into 14x14 image patches of size 16x16. The pixel matrix of each image patch is converted into an image patch feature vector through a linear projection layer. At the same time, assign a learnable position encoding vector to each image patch position, which is used to record the position information of the image patch in the original graphic area. Add the position encoding vector to the corresponding image patch feature vector to obtain a feature sequence that fuses position information, thereby establishing the spatial relationship between different image patches. Assume that the linear projection layer converts a 16x16 image patch into a 768-dimensional feature vector, and the position encoding vector is also 768-dimensional, then the result after addition is still a 768-dimensional feature vector.

[0103] Construct a multi-layer transformer encoder network. This network adopts a dense connection structure, and splices the feature maps output by each layer in the channel dimension as the input of the next layer. For example, the first-layer transformer encoder outputs a feature map with 768 channels, and the second layer outputs a feature map with 1024 channels. Then splice these two feature maps in the channel dimension to obtain a feature map with 1792 channels as the input of the third layer. To control the feature dimension, add a convolutional layer after each layer to adjust the number of feature channels. For example, adjust 1792 channels to 768 channels. This dense connection structure can transfer features between different levels, enabling the network to learn richer feature representations.

[0104] After the multi-layer transformer encoder network, a feature pyramid structure is constructed. Pooling operations of different scales are performed on the feature maps output by the network. For example, using max pooling, large-scale feature maps and small-scale feature maps are obtained. The large-scale feature maps retain the detailed information of the image. For example, a 2x2 pooling kernel is used; the small-scale feature maps obtain information on a large receptive field. For example, a 4x4 pooling kernel is used. The small-scale feature maps are upsampled through deconvolution operations to unify their spatial resolution with that of the large-scale feature maps. For example, the feature map obtained after 4x4 pooling is upsampled by a factor of 2.

[0105] The feature maps of different scales with unified resolution are concatenated in the channel dimension to obtain multi-scale features. For example, the feature maps obtained by 2x2 pooling and 4x4 pooling followed by upsampling are concatenated together. Adaptive fusion of the multi-scale features is performed through a convolutional layer. In order to retain the original feature information, a residual connection mechanism is adopted, and the input features of the current layer are directly added to the output features to obtain the fused output features.

[0106] A global pooling operation is performed on the fused output features. For example, global average pooling is used to compress the spatial dimension into a low-dimensional feature vector. The low-dimensional feature vector is processed through a multi-layer non-linear transformation network. After each transformation, an activation function, such as the ReLU function, is used to introduce non-linear expression ability. Finally, an image feature vector containing local details and global semantics is output.

[0107] In this embodiment, by combining the vision transformer network and the dense connection structure, richer image features can be extracted, improving the expression ability of the model. The feature pyramid structure can effectively fuse feature information of different scales, enabling the model to take into account both local details and global semantics. The introduction of the residual connection mechanism can enhance the robustness of the model and improve the generalization ability of the model.

[0108] S2. The image feature vector, the text content, and the text semantic feature vector are constructed into a heterogeneous feature tensor. Singular value decomposition is performed on the heterogeneous feature tensor to obtain a core tensor and factor matrices. Correlation analysis decomposition is performed on the core tensor to obtain an icon feature fingerprint vector. A locality-sensitive hashing index is constructed based on the icon feature fingerprint vector. The optimal hashing function parameters are calculated by combining the distribution density of the icon feature fingerprint vector, and the hash storage unit is adjusted based on the distribution density. The icon feature fingerprint vector is encoded by combining position-sensitive coding to obtain an encoded fingerprint vector. A multi-level index structure is constructed and the encoded fingerprint vector is added to the coarse-grained index layer, and the icon feature fingerprint vector is added to the fine-grained index layer to obtain an icon feature multi-dimensional index database;

[0109] The heterogeneous feature tensor is a tensor used to represent complex multi-dimensional data, usually containing different types of feature data (such as numerical, categorical, etc.), and is used to handle problems with multiple features and heterogeneous data sources. It is commonly used in multi-modal learning and recommendation systems. The core tensor is a low-dimensional tensor that represents the main features of the data, while the factor matrices correspond to the representations of each dimension in the tensor. The core tensor reconstructs the original tensor by combining with the factor matrices. The optimal hash function is a function that maps data to a low-dimensional space through a hashing operation, with the goal of making similar data points as close as possible in the hashed space, thereby improving the efficiency of data storage and retrieval. It is commonly used in applications such as approximate nearest neighbor search. The hash storage unit is a data storage structure used to store data points after being mapped by the hash function, which reduces storage space and accelerates data retrieval, and is commonly used in efficient data storage and query. The position-sensitive encoding is an encoding method used to represent the position features in data. By considering the relative positions of data points in space, the model can capture the information dependencies in the spatial or temporal dimensions, and is widely used in fields such as natural language processing and image processing. The coarse-grained index layer is a method for large-scale data indexing. By coarsely partitioning and indexing the data, it improves the retrieval efficiency and is commonly used in databases and information retrieval systems. The fine-grained index layer is a further refinement based on the coarse-grained index, providing a more precise index to improve the query speed, and is commonly used in multi-level index structures, especially when high-precision queries are required.

[0110] In an alternative embodiment,

[0111] Construct the heterogeneous feature tensor from the image feature vector, the text content, and the text semantic feature vector. Perform singular value decomposition on the heterogeneous feature tensor to obtain the core tensor and factor matrices. Perform correlation analysis decomposition on the core tensor to obtain the icon feature fingerprint vector. Construct a locality-sensitive hashing index based on the icon feature fingerprint vector. Calculate the optimal hash function parameters by combining the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density. Encode the icon feature fingerprint vector by combining position-sensitive encoding to obtain the encoded fingerprint vector. Construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer, and add the icon feature fingerprint vector to the fine-grained index layer to obtain an icon feature multi-dimensional index database including:

[0112] Preprocess the image feature vector, the text content, and the text semantic feature vector. Perform standardization on the image feature vector to eliminate the scale difference between dimensions, perform word segmentation and encoding conversion on the text content to convert it into a fixed-length feature representation, perform normalization on the text semantic feature vector to make its distribution range consistent with other features, and organize the preprocessed features into a third-order heterogeneous feature tensor;

[0113] Perform matrix expansion on the heterogeneous feature tensor along the sample dimension, the feature type dimension, and the eigenvalue dimension respectively. Perform singular value decomposition on each expanded matrix to extract the main components, and recombine the main components according to the preset rank parameter to obtain the core tensor and the factor matrix;

[0114] Calculate the correlation coefficients between different dimensions in the core tensor to construct a correlation coefficient matrix, perform eigenvalue decomposition on the correlation coefficient matrix to extract the principal components, calculate the contribution rate of each principal component, and select the combination of principal components with a contribution rate exceeding the preset threshold to form an icon feature fingerprint vector;

[0115] Divide the feature space into multiple regions, count the number of samples in each region to obtain the regional sample density distribution, determine the number of local sensitive hashing functions according to the regional sample density distribution, and perform fine-grained division on the regions with a sample density greater than the first density threshold by increasing the number of hashing functions, and perform coarse-grained division on the regions with a sample density less than the second density threshold by reducing the number of hashing functions;

[0116] Calculate the variance of each dimension of the icon feature fingerprint vector, sort the dimension importance according to the variance value, assign 16-bit binary encoding to the top 5 dimensions in terms of importance, assign 8-bit binary encoding to the dimensions after the fifth place, determine the binary encoding mapping relationship through eigenvalue interval division, and convert the icon feature fingerprint vector into an encoded fingerprint vector;

[0117] Construct a multi-level index structure. In the coarse-grained index layer, segment the encoded fingerprint vector according to a 32-bit length, calculate a 32-bit hash value for each segment as an index key to establish a hash table, and the hash table points to a storage bucket. When the hash value of the encoded fingerprint vector is within the Hamming distance threshold of 8, store its index in the same storage bucket. In the fine-grained index layer, construct the icon feature fingerprint vector into a tree structure, and each node of the tree represents the numerical interval of the feature vector in the current dimension, to obtain an icon feature multi-dimensional index database.

[0118] The correlation coefficient matrix is a matrix representing the linear correlation relationships among multiple variables. The elements in the matrix represent the correlation coefficients between each pair of variables and are commonly used in statistics and machine learning to analyze the dependence relationships among variables. The locality-sensitive hashing function is a method that maps similar data points to the same bucket through hashing. Its goal is to map similar data points to similar hash values to accelerate approximate nearest neighbor search. The Hamming distance threshold is a metric used to measure the difference between two binary strings of the same length and is commonly used in string comparison and data retrieval. The smaller the Hamming distance, the more similar the two strings are.

[0119] Extract and preprocess the feature of the icon image data. Extract the visual feature vector of the image. For example, use a convolutional neural network to extract the depth feature of the image. At the same time, extract the text content of the icon and use natural language processing techniques to extract the semantic feature vector of the text. For example, use the BERT model to obtain word vectors. Standardize the image feature vector. For example, use Z-score standardization to eliminate the scale differences between different dimensions. Segment and encode the text content. For example, use Word2Vec to convert the text into a fixed-length feature representation. Normalize the text semantic feature vector. For example, use MinMaxScaler to make its distribution range consistent with other features. Organize the preprocessed image feature vector, text content feature, and text semantic feature vector into a third-order heterogeneous feature tensor. For example, assume there are 1000 icon samples, the dimension of the image feature vector is 512, the dimension of the text content feature is 200, and the dimension of the text semantic feature vector is 768. Then a third-order tensor of 1000x3x(512 + 200 + 768) can be constructed.

[0120] Perform Tucker decomposition on the heterogeneous feature tensor. Unfold the third-order tensor into matrices along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Perform singular value decomposition on each unfolded matrix and extract the main components. For example, set the rank parameters as (100, 3, 500), then a core tensor of 100x3x500 and three factor matrices can be obtained.

[0121] Perform correlation analysis decomposition on the core tensor to obtain the icon feature fingerprint vector. Calculate the correlation coefficients between different dimensions in the core tensor and construct a correlation coefficient matrix. Perform eigenvalue decomposition on the correlation coefficient matrix and extract the main components. Calculate the contribution rate of each main component. For example, set the threshold as 85%, and select the combination of main components whose contribution rates exceed the threshold to form the icon feature fingerprint vector. Assume the first 100 main components are selected, then the dimension of the icon feature fingerprint vector is 100.

[0122] Construct a locality-sensitive hashing index based on the icon feature fingerprint vector. Divide the feature space into multiple regions. For example, if each dimension is divided into 10 intervals, the feature space can be divided into 10 to the 100th power of regions. Count the number of samples in each region to obtain the regional sample density distribution. Determine the number of locality-sensitive hashing functions according to the regional sample density distribution. For example, set the first density threshold to 100 and the second density threshold to 10. Regions with a sample density greater than 100 are finely divided by increasing the number of hashing functions, such as using 5 hashing functions; regions with a sample density less than 10 are coarsely divided by reducing the number of hashing functions, such as using 2 hashing functions; other regions use 3 hashing functions.

[0123] Encode the icon feature fingerprint vector in combination with position-sensitive encoding. Calculate the variance of each dimension of the icon feature fingerprint vector, and sort the dimension importance according to the variance value. Assign 16-bit binary encoding to the top 5 dimensions in terms of importance, and assign 8-bit binary encoding to the dimensions after the fifth place. Determine the binary encoding mapping relationship through eigenvalue interval division, and convert the icon feature fingerprint vector into an encoded fingerprint vector. For example, assume that a certain dimension has the largest variance, and its value range is from 0 to 1. Then 0 to 0.0625 can be mapped to 0000000000000000, 0.0625 to 0.125 can be mapped to 0000000000000001, and so on.

[0124] Construct a multi-level index structure. In the coarse-grained index layer, segment the encoded fingerprint vector according to a 32-bit length, and calculate a 32-bit hash value for each segment as the index key to establish a hash table. The hash table points to a bucket. When the hash value of the encoded fingerprint vector is within the Hamming distance threshold of 8, its index is stored in the same bucket. In the fine-grained index layer, construct the icon feature fingerprint vector into a tree structure. Each node of the tree represents the numerical interval of the feature vector in the current dimension. For example, the root node represents the first dimension, and its child nodes represent different value intervals of this dimension, and so on. Finally, the leaf nodes store pointers to specific icon samples. Thus, the construction of the icon feature multi-dimensional index database is completed.

[0125] In this embodiment, by combining image features, text content, and semantic features, and using Tucker decomposition and correlation analysis to extract the icon feature fingerprint vector, the icon information can be more comprehensively expressed, thereby improving the retrieval accuracy. The design of locality-sensitive hashing and the multi-level index structure can effectively reduce the retrieval space and speed up the retrieval speed. At the same time, position-sensitive encoding can further improve the retrieval efficiency, can process large-scale icon data, and supports the fusion of multiple feature types, having good scalability.

[0126] In an alternative embodiment,

[0127] Perform matrix unfolding on the heterogeneous feature tensor along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Perform singular value decomposition on each unfolded matrix to extract the main components, and recombine the main components according to the preset rank parameter to obtain the core tensor and factor matrices, including:

[0128] Construct a third-order heterogeneous feature tensor from the image features, text content, and text semantic features of the icon data. The third-order heterogeneous feature tensor includes a sample dimension, a feature type dimension, and an eigenvalue dimension;

[0129] Unfold the third-order heterogeneous feature tensor along the sample dimension to obtain a first unfolded matrix. The number of rows of the first unfolded matrix is the number of samples, and the number of columns is the product of the feature type dimension and the eigenvalue dimension. Unfold it along the feature type dimension to obtain a second unfolded matrix. The number of rows of the second unfolded matrix is the number of feature types, and the number of columns is the product of the sample dimension and the eigenvalue dimension. Unfold it along the eigenvalue dimension to obtain a third unfolded matrix. The number of rows of the third unfolded matrix is the sum of the eigenvalue dimensions, and the number of columns is the product of the sample dimension and the feature type dimension;

[0130] Divide the first unfolded matrix, the second unfolded matrix, and the third unfolded matrix into sub-block matrices of equal size. The size of each sub-block matrix is one-tenth of the corresponding unfolded matrix;

[0131] Perform iterative singular value decomposition on each sub-block matrix. Randomly initialize a unit vector, multiply the unit vector by the sub-block matrix and normalize it to obtain an updated vector. Repeat the multiplication and normalization operations until the updated vector converges to obtain a first pair of singular vectors. Subtract the rank-one matrix constructed from the first pair of singular vectors from the sub-block matrix to obtain a remaining matrix, and repeat the multiplication and normalization operations on the remaining matrix to obtain the next pair of singular vectors;

[0132] Add an L2 norm constraint to the singular vectors during the singular value decomposition process, and perform iterative optimization using a dynamic learning rate. The initial value of the dynamic learning rate is 0.1, and it is adaptively adjusted according to the change of the reconstruction error;

[0133] Select the largest singular value and its corresponding left and right singular vectors from the singular value decomposition results of each sub-block matrix according to the preset rank parameter. Fuse the singular values and singular vectors selected from each sub-block matrix through a collaborative filtering algorithm. Form the sample dimension factor matrix from the left singular vectors, and form the feature type dimension factor matrix and the eigenvalue dimension factor matrix from the right singular vectors;

[0134] Construct an initial small-scale core tensor using a progressive strategy, calculate the reconstruction error. When the reconstruction error is greater than the preset threshold and has not reached the maximum scale limit, gradually increase the scale of the core tensor, and repeat the increase until the maximum scale limit is reached to obtain the core tensor.

[0135] The sub-block matrix refers to a partial sub-matrix in a matrix, usually obtained by partitioning the matrix. Each sub-block matrix contains a part of the data in the matrix and is often used in matrix decomposition and solving optimization problems. The rank-one matrix refers to a matrix with a rank of 1, that is, the matrix can be expressed as the outer product of two vectors. Rank-one matrices are often used in tasks such as data dimensionality reduction and feature representation. The collaborative filtering algorithm is an algorithm commonly used in recommendation systems. By analyzing the similarity between users or the similarity between items, it predicts the content that users may be interested in based on historical behavior or preferences and is widely used in fields such as e-commerce and movie recommendations.

[0136] Collect and preprocess icon data. Obtain icon data from the icon database, including icon images, text content, and text semantic features. Standardize the icon images, such as unifying the size and color space conversion. Perform preprocessing operations on the text content, such as word segmentation and stop word removal. Use a pre-trained word vector model to extract the semantic features of the text content.

[0137] Construct a third-order heterogeneous feature tensor. Organize the preprocessed icon data into a third-order tensor. The three dimensions of the tensor represent samples, feature types, and feature values respectively. For example, the image feature vector, text content feature vector, and semantic feature vector of a certain icon are filled into the corresponding positions of the tensor. Suppose there are 1000 icon samples, the image feature dimension is 512, the text content feature dimension is 256, and the semantic feature dimension is 768, then a tensor of 1000x3x(512 + 256 + 768) can be constructed.

[0138] Perform matrix unfolding and singular value decomposition on the tensor. Unfold the constructed third-order tensor into three matrices along the sample dimension, feature type dimension, and feature value dimension respectively. For example, when unfolding along the sample dimension, a matrix with 1000 rows and 3*(512 + 256 + 768) columns is obtained. Divide each unfolded matrix into sub-block matrices of equal size, and the size of each sub-block matrix is one-tenth of the corresponding unfolded matrix. Perform iterative singular value decomposition operations on each sub-block matrix to extract the main components. During the iterative singular value decomposition process, a dynamic learning rate is used for optimization. The initial learning rate is set to 0.1 and is adaptively adjusted according to the reconstruction error. At the same time, an L2 norm constraint is added to the singular vectors to prevent overfitting.

[0139] During the singular value decomposition process, important singular values and singular vectors are selected. According to a preset rank parameter, a certain number of the largest singular values and their corresponding left and right singular vectors are selected from the singular value decomposition results of each sub-block matrix. For example, if the rank parameter is set to 10, then the first 10 largest singular values and their corresponding singular vectors are selected. The singular values and singular vectors screened from each sub-block matrix are fused using a collaborative filtering algorithm. The left singular vectors are composed into a sample dimension factor matrix, and the right singular vectors are composed into a feature type dimension factor matrix and an eigenvalue dimension factor matrix.

[0140] An incremental strategy is adopted to construct the core tensor. An initial small-scale core tensor is constructed, and the reconstruction error is calculated. If the reconstruction error is greater than a preset threshold and the core tensor has not reached the maximum scale limit, the scale of the core tensor is gradually increased until the preset maximum scale limit is reached or the reconstruction error is less than the threshold.

[0141] Exemplarily, assume that a singular value decomposition is performed on a 100x3x1536 sub-block matrix, and the rank parameter is set to 10. After singular value decomposition and screening, 10 singular values and their corresponding 10 left singular vectors (100-dimensional) and 10 right singular vectors (4608-dimensional) are obtained. These 10 left singular vectors are composed into a 100x10 sample factor matrix. Similar operations and fusions are performed on the singular values and singular vectors of all sub-block matrices, and finally, a complete core tensor and factor matrix are obtained.

[0142] In this embodiment, by fusing multi-modal features, icon information can be more comprehensively described, thereby improving the precision and recall rate of icon retrieval. Multi-modal feature fusion can better capture the fine-grained features of icons, thereby improving the accuracy of icon classification. Through tensor decomposition technology, the data dimension can be effectively reduced, the computational complexity can be reduced, and thus the efficiency of icon analysis can be improved.

[0143] S3. Receive the retrieval request and obtain the target icon image, perform feature extraction and encoding on the target icon image, generate the target feature fingerprint vector, and perform a fast retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. Calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer. Determine the feature similarity based on the feature distance, and select the icon feature fingerprint vectors with feature similarity greater than the pre-set similarity threshold to generate a similarity candidate set, which is added to the deep metric learning model. Calculate the relative distance relationship through the triplet loss function, combine the contrast loss function to constrain the feature space distribution, obtain the metric feature representation, calculate the visual similarity, text similarity, and semantic similarity, determine the weight parameters by determining the variance and discriminant coefficient of each similarity feature, determine the comprehensive similarity score through weighted summation, sort in descending order, and select the top 3 retrieval results as the similar retrieval results for output.

[0144] The fast retrieval is a technology for quickly finding and matching target items in a database. The deep metric learning model is a model that uses a deep neural network for metric learning. Its purpose is to learn a mapping function that maps the input data to an embedding space, so that the distance between similar data points is small, and the distance between dissimilar data points is large. The triplet loss function is a commonly used loss function in deep metric learning, which is used to optimize the embedding space of the model.

[0145] In an alternative embodiment,

[0146] Receive the retrieval request and obtain the target icon image, perform feature extraction and encoding on the target icon image, generate the target feature fingerprint vector, and perform a fast retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. Calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer. Determine the feature similarity based on the feature distance, and select the icon feature fingerprint vectors with feature similarity greater than the pre-set similarity threshold to generate a similarity candidate set, which is added to the deep metric learning model. Calculate the relative distance relationship through the triplet loss function, combine the contrast loss function to constrain the feature space distribution, obtain the metric feature representation, calculate the visual similarity, text similarity, and semantic similarity, determine the weight parameters by determining the variance and discriminant coefficient of each similarity feature, determine the comprehensive similarity score through weighted summation, sort in descending order, and select the top 3 retrieval results as the similar retrieval results for output includes:

[0147] Receive a retrieval request and obtain a target icon image, perform feature extraction and normalization preprocessing on the target icon image, extract an image feature vector through a multi-layer convolutional network, extract a text vector and a semantic feature vector through a pre-trained language model, and splice the image feature vector, the text vector, and the semantic feature vector to form a target feature fingerprint vector;

[0148] Segment the target feature fingerprint vector into multiple feature segments according to a preset length, and perform binarization processing on the feature values in each feature segment to obtain binary codes. Among them, the binarization processing includes recording the feature values greater than the mean of the feature segment as the first identifier and the feature values less than the mean of the feature segment as the second identifier;

[0149] Use locality-sensitive hashing in the coarse-grained index layer to map the binary code to a hash value, search for the encoded fingerprint vector with a distance less than a preset threshold from the hash value in the hash table, add the encoded fingerprint vector to the initial candidate set. If the number of samples in the storage area of the hash table exceeds the preset quantity threshold, increase the number of hash functions in the current storage area for region subdivision;

[0150] Traverse the encoded fingerprint vectors in the initial candidate set in the feature tree of the fine-grained index layer. The internal nodes of the feature tree represent regions in the feature space, and the leaf nodes store icon feature fingerprint vectors. Calculate the feature distance between the target feature fingerprint vector and the center point of the node, select the child node with the closest feature distance to continue traversing until the leaf node, and generate a similarity candidate set for the icon feature fingerprint vectors corresponding to the leaf nodes with a feature distance less than the preset similarity threshold;

[0151] Add the similarity candidate set to the deep metric learning model. The deep metric learning model includes a multi-layer fully connected encoder and an attention mechanism-based metric network. Select the sample with the closest distance to the target feature fingerprint vector from the similarity candidate set as the positive sample, the sample with the farthest distance as the negative sample, use the target feature fingerprint vector as the anchor sample, and combine the anchor sample, the positive sample, and the negative sample to construct a triplet;

[0152] Calculate the relative distance relationship for the triplet through the triplet loss function, combine the contrastive loss function to constrain the feature space distribution, make the distance between the target feature fingerprint vector and the positive sample less than the distance from the negative sample, and calculate the feature representation similarity between icon pairs respectively. Compare the similarity with the preset target similarity threshold, and screen the icon pairs to form a feature representation set;

[0153] Based on the depth metric learning model, a metric feature representation is obtained. The visual similarity, text similarity, and semantic similarity are calculated between the icon feature fingerprint vectors in the feature representation set and the target feature fingerprint vector. The weight parameters are determined based on the variance and discriminant coefficient of each similarity feature in the training set, and the comprehensive similarity score is determined through weighted summation. The comprehensive similarity scores are sorted in descending order, and the top 3 retrieval results are selected as the similar retrieval results for output.

[0154] The binarization process is a process of converting input data (such as images, texts, etc.) into a binary format, which is usually used for feature encoding, image compression, or reducing the storage requirements of data. The locality-sensitive hashing is an algorithm for fast retrieval of high-dimensional data. The core idea is to construct a hash function so that similar data points are mapped to the same hash bucket. The leaf node is the terminal node in a tree structure (such as a decision tree, random forest, etc.), which represents the final classification or regression result of the tree. The anchor sample in metric learning refers to one of the samples used to define the similarity metric, and is usually a reference point for comparison with other samples.

[0155] Receive a retrieval request and obtain the target icon image. For example, the user uploads an icon image containing graphics and text as the retrieval target.

[0156] Perform feature extraction and normalization preprocessing on the target icon image. Adjust the obtained icon image to a unified size, such as 224x224 pixels, and use a multi-layer convolutional network (such as ResNet50) to extract the image feature vector. Assume that the dimension of the extracted image feature vector is 2048. At the same time, use a pre-trained language model (such as BERT) to extract the text vector and semantic feature vector corresponding to the icon text. Assume that the dimension of the text vector is 768 and the dimension of the semantic feature vector is 512. Finally, concatenate the image feature vector, text vector, and semantic feature vector to form the target feature fingerprint vector, with a dimension of 2048 + 768 + 512 = 3328.

[0157] The target feature fingerprint vector is segmented into multiple feature segments according to a preset length. For example, the 3328-dimensional feature vector is segmented according to a length of 512, resulting in 6 feature segments, and the remaining 256-dimensional feature segment is processed separately. The feature values in each feature segment are binarized to obtain binary codes. Specifically, calculate the mean value of each feature segment, and mark the feature values greater than the mean value as "1" and the feature values less than the mean value as "0". For example, if the mean value of a certain feature segment is 0.5 and the feature values of this feature segment are [0.6, 0.3, 0.7, 0.2], then the corresponding binary code is [1, 0, 1, 0].

[0158] In the coarse-grained index layer, local sensitive hashing is used to map binary codes to hash values. For example, the SimHash algorithm is used to map the binary codes of each feature segment to 64-bit hash values. Look up the encoded fingerprint vectors in the hash table whose distance from the target hash value is less than a preset threshold. For example, set the threshold to 3, look up the encoded fingerprint vectors with a Hamming distance less than or equal to 3, and add these encoded fingerprint vectors to the initial candidate set. If the number of samples in the storage area of the hash table exceeds a preset quantity threshold (e.g., 1000), then increase the number of hash functions in the current storage area (e.g., from 3 to 4) for regional subdivision to improve the retrieval efficiency.

[0159] Traverse the encoded fingerprint vectors in the initial candidate set in the feature tree of the fine-grained index layer. The internal nodes of the feature tree represent regions in the feature space, and the leaf nodes store the icon feature fingerprint vectors. Calculate the feature distance (e.g., Euclidean distance) between the target feature fingerprint vector and the center point of the node. Select the child node with the closest feature distance to continue traversing until reaching the leaf node. Generate a similarity candidate set for the icon feature fingerprint vectors corresponding to the leaf nodes whose feature distance is less than a preset similarity threshold (e.g., 0.8).

[0160] Add the similarity candidate set to the deep metric learning model. The deep metric learning model includes a multi-layer fully connected encoder and a metric network based on the attention mechanism. Select the sample with the closest distance to the target feature fingerprint vector from the similarity candidate set as the positive sample, and the sample with the farthest distance as the negative sample. Use the target feature fingerprint vector as the anchor sample, and combine the anchor sample, positive sample, and negative sample to construct a triplet. For example, if the target icon image has the closest distance to icon image A and the farthest distance to icon image B in the candidate set, then construct the triplet (target icon, A icon, B icon).

[0161] Calculate the relative distance relationship for the triplet through the triplet loss function, and combine the contrastive loss function to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance from the negative sample, and calculate the feature representation similarity between icon pairs respectively. Compare the similarity with the preset target similarity threshold, and screen the icon pairs to form a feature representation set.

[0162] Based on the metric feature representation obtained from the deep metric learning model, calculate the visual similarity, text similarity, and semantic similarity between the icon feature fingerprint vectors and the target feature fingerprint vectors in the feature representation set. For example, use cosine similarity to calculate the visual similarity, text similarity, and semantic similarity. Determine the weight parameters based on the variance and discriminant coefficient of each similarity feature in the training set. For example, if the variance of the visual similarity is large and the discriminant coefficient is high, a larger weight is assigned. Determine the comprehensive similarity score through weighted summation. Sort the comprehensive similarity scores in descending order, and select the top 3 retrieval results as the similar retrieval results for output.

[0163] In this embodiment, through the two-level retrieval architecture of coarse-grained indexing and fine-grained indexing, the candidate set is quickly screened, the retrieval range is narrowed, and the retrieval efficiency is improved. The deep metric learning model is used to learn more discriminative feature representations, combined with multi-dimensional similarity metrics of vision, text, and semantics to improve the retrieval accuracy. The target icon image is preprocessed by normalization, and combined with binarization processing and local sensitive hashing algorithm, which enhances the robustness of the retrieval system and has strong resistance to interference factors such as image noise and deformation.

[0164] In an alternative embodiment,

[0165] Calculate the relative distance relationship for the triple through the triple loss function, and combine the contrast loss function to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance from the negative sample, and calculate the feature representation similarity between icon pairs respectively. Compare the similarity with a preset target similarity threshold, and screen the icon pairs to form a feature representation set, including:

[0166] Receive a triple composed of a target feature fingerprint vector, a positive sample feature fingerprint vector, and a negative sample feature fingerprint vector, and input the triple into the deep metric learning model. The deep metric learning model includes a multi-layer fully connected encoder network and a metric network based on the attention mechanism. The encoder network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence. The output dimensions of the first fully connected layer, the second fully connected layer, and the third fully connected layer decrease in sequence. After each fully connected layer, a non-linear activation layer and a batch normalization layer are connected;

[0167] Calculate the Euclidean distance between the target feature fingerprint vector and the positive sample feature fingerprint vector through the metric network to obtain the positive sample distance, calculate the Euclidean distance between the target feature fingerprint vector and the negative sample feature fingerprint vector to obtain the negative sample distance, and compare the difference between the positive sample distance and the negative sample distance with a preset interval parameter;

[0168] Calculating the cosine similarity of the feature representation for the icon pairs in the triplet, setting a first target similarity for the icon pairs of the same category, and setting a second target similarity for the icon pairs of different categories, wherein the first target similarity is greater than the second target similarity, and performing feature constraint based on the difference between the cosine similarity of the feature representation and the corresponding target similarity;

[0169] Calculating loss values of the triplets, selecting a preset number of triplets with the largest loss values as difficult samples, and using the difficult samples for updating parameters of the encoder network and the metric network;

[0170] The feature fingerprint vectors of the icon pairs in the similarity candidate set are feature mapped through the encoder network to obtain mapping features, the cosine similarity between the mapping features is calculated through the metric network, and the icon pairs whose cosine similarity is greater than a preset similarity threshold are saved in the feature representation set.

[0171] Positive samples are samples that are similar to anchor samples. In metric learning, positive samples should have feature representations similar to those of anchor samples. Difficult samples are samples that are difficult to classify or separate for model training and are typically used to enhance the model's learning capabilities. In deep learning, difficult samples are reinforced through specific sampling strategies or weighting methods to improve the model's robustness and generalization capabilities.

[0172] Prepare an icon image dataset for model training. This dataset contains a large number of icon images, each of which is labeled with the category to which it belongs. For example, we collected 10,000 icon images, covering 1,000 different icon categories, with each category containing 10 images. Randomly extract the target icon image, positive sample icon images of the same category, and negative sample icon images of different categories from the dataset to form a triplet for model training. For example, randomly select an "apple" icon image as the target image, another "apple" icon image as the positive sample, and a "banana" icon image as the negative sample to form a triplet.

[0173] A deep metric learning model is constructed, consisting of an encoder network and a metric network. The encoder network converts the input icon image into a feature vector. ResNet50 is used as an example of an encoder network, which contains multiple convolutional and pooling layers to extract image features. The metric network calculates the distance or similarity between two feature vectors. A metric network based on an attention mechanism is used here to better capture subtle differences between feature vectors. Assuming the feature vector output by the encoder network has a dimension of 512, the attention mechanism calculates the weights of the features of different dimensions between two feature vectors to obtain a weighted feature representation, and then calculates the Euclidean distance between them.

[0174] The model is trained using triplet loss and contrastive loss functions. The prepared triplets are input into the deep metric learning model, and the encoder network extracts the feature vectors of the target image, positive samples, and negative samples respectively. The metric network calculates the distance between the target feature vector and the positive sample feature vector (positive sample distance), and the distance between the target feature vector and the negative sample feature vector (negative sample distance). The goal of the triplet loss is to make the positive sample distance less than the negative sample distance, with at least a preset margin, such as 0.2, between them. At the same time, the cosine similarity of the icon pairs in the triplet is calculated and compared with the preset target similarity. For example, for icon pairs of the same category, the target similarity is set to 0.9; for icon pairs of different categories, the target similarity is set to 0.1. The feature constraints are imposed by minimizing the similarity difference, further improving the performance of the model. During training, for example, 128 triplets with the largest loss values are selected as hard samples for updating the model parameters, accelerating the convergence speed of the model.

[0175] Icon image retrieval is performed. The icon image to be retrieved is input into the trained deep metric learning model to obtain its feature vector. This feature vector is compared with the feature vectors of all icon images in the database, and their cosine similarities are calculated. The icon images with similarities greater than a preset threshold, such as 0.8, are screened out to form the final retrieval result.

[0176] In this embodiment, through deep metric learning, the model can learn more discriminative feature representations of icon images, thereby improving the retrieval accuracy and more effectively identifying similar icons. Training with triplet loss and contrastive loss makes the model more robust to subtle changes in images, such as illumination, angle, deformation, etc. Even if there are certain differences in the icon images, their similarities can be accurately judged. By pre-computing the feature vectors of all icon images in the database, the retrieval process only needs to calculate the similarities between the image to be retrieved and the images in the database, greatly improving the retrieval efficiency and enabling similar icons to be quickly found from a large amount of data.

[0177] Figure 2 It is a schematic structural diagram of the multi-dimensional data intelligent retrieval and matching system for the text and image features of the embodiment of the present invention, as Figure 2 shown, the system includes:

[0178] The first unit is used to obtain the icon image to be retrieved, add it to a convolutional neural network for feature extraction, combine multiple convolutional layers and pooling layers to extract the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network based on the first feature map, determine the spatial dependence relationship of the image region through the spatial attention mechanism, generate an attention weight map in combination with a channel attention module, and output a graphic region and a text region. Add the text region to a transducer optical character recognition model, convert it into a feature sequence through an encoder, and perform character-by-character decoding through a multi-head attention mechanism in a decoder to obtain text content. Perform feature extraction on the text content through a bidirectional semantic understanding model to obtain a text semantic feature vector. Add the graphic region to a vision transformer network, perform cross-layer feature transfer based on a dense connection structure, and perform feature fusion in combination with a residual connection and a feature pyramid structure to obtain an image feature vector;

[0179] The second unit is used to construct a heterogeneous feature tensor from the image feature vector, the text content, and the text semantic feature vector, perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and factor matrices, perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector, construct a locality-sensitive hashing index based on the icon feature fingerprint vector, calculate optimal hashing function parameters in combination with the distribution density of the icon feature fingerprint vector, and adjust the hash storage unit based on the distribution density. Encode the icon feature fingerprint vector in combination with position-sensitive encoding to obtain an encoded fingerprint vector, construct a multi-level index structure, and add the encoded fingerprint vector to a coarse-grained index layer, and add the icon feature fingerprint vector to a fine-grained index layer to obtain an icon feature multi-dimensional index database;

[0180] The third unit is used to receive a retrieval request and obtain a target icon image, perform feature extraction and encoding on the target icon image, generate a target feature fingerprint vector, and perform a quick search in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. Calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determine the feature similarity based on the feature distance, and select the icon feature fingerprint vectors with a feature similarity greater than a pre-set similarity threshold to generate a similarity candidate set, add it to a deep metric learning model, calculate the relative distance relationship through a triplet loss function, and constrain the feature space distribution in combination with a contrast loss function to obtain a metric feature representation, and calculate the visual similarity, text similarity, and semantic similarity. Determine the weight parameters by determining the variance and discriminant coefficient of each similarity feature, and determine the comprehensive similarity score through weighted summation, sort in descending order, and select the top 3 retrieval results as similar retrieval results for output.

[0181] In the third aspect of the embodiments of the present invention,

[0182] Provided is an electronic device, comprising:

[0183] a processor;

[0184] a memory for storing instructions executable by the processor;

[0185] wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0186] In a fourth aspect of the embodiments of the present invention,

[0187] a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0188] The present invention may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for performing various aspects of the present invention.

[0189] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-dimensional data intelligent retrieval and matching method for graphic and text features, characterized in that Including: Obtain the icon image to be retrieved and add it to the convolutional neural network for feature extraction. Combine multiple convolutional layers and pooling layers to extract the first feature map. Based on the first feature map, input the icon image to be retrieved into the image segmentation attention mechanism network. Determine the spatial dependence relationship of the image region through the spatial attention mechanism. Combine the channel attention module to generate the attention weight map and output the graphic region and the text region. Add the text region to the transducer optical character recognition model, convert it into a feature sequence through the encoder, and perform character-by-character decoding through the multi-head attention mechanism in the decoder to obtain the text content. Extract the feature of the text content through the bidirectional semantic understanding model to obtain the text semantic feature vector. Add the graphic region to the vision transformer network, perform cross-layer feature transfer based on the dense connection structure, and combine the residual connection and the feature pyramid structure to perform feature fusion to obtain the image feature vector; Construct the heterogeneous feature tensor from the image feature vector, the text content, and the text semantic feature vector. Perform singular value decomposition on the heterogeneous feature tensor to obtain the core tensor and the factor matrix. Perform correlation analysis decomposition on the core tensor to obtain the icon feature fingerprint vector. Construct the locality-sensitive hashing index based on the icon feature fingerprint vector. Calculate the optimal hash function parameter in combination with the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density. Encode the icon feature fingerprint vector in combination with the position-sensitive coding to obtain the encoded fingerprint vector. Construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer. Add the icon feature fingerprint vector to the fine-grained index layer to obtain the icon feature multi-dimensional index database; Receive the retrieval request and obtain the target icon image. Perform feature extraction and encoding on the target icon image to generate the target feature fingerprint vector and perform a quick search in the coarse-grained index layer to obtain the initial candidate set containing multiple encoded fingerprint vectors. Calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer. Determine the feature similarity based on the feature distance and select the icon feature fingerprint vectors with the feature similarity greater than the pre-set similarity threshold to generate the similarity candidate set, which is added to the deep metric learning model. Calculate the relative distance relationship through the triplet loss function, and combine the contrast loss function to constrain the feature space distribution to obtain the metric feature representation and calculate the visual similarity, text similarity, and semantic similarity. Determine the weight parameter by determining the variance and discriminant coefficient of each similarity feature, and determine the comprehensive similarity score through weighted summation. Sort in descending order and select the top 3 retrieval results as the similar retrieval results for output.

2. The method according to claim 1, characterized in that Obtain the icon image to be retrieved and add it to a convolutional neural network for feature extraction. Combine multiple convolutional layers and pooling layers to extract the first feature map. Based on the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network. Determine the spatial dependence relationship of the image region through the spatial attention mechanism. Combine the channel attention module to generate an attention weight map and output the graphic region and the text region. Add the text region to the Transformer optical character recognition model, convert it into a feature sequence through the encoder, and perform character-by-character decoding through the multi-head attention mechanism in the decoder to obtain the text content. Extract features from the text content through a bidirectional semantic understanding model to obtain a text semantic feature vector. Add the graphic region to the Vision Transformer network, perform cross-layer feature transfer based on the dense connection structure, and combine the residual connection and the feature pyramid structure for feature fusion to obtain an image feature vector, including: Obtain the icon image to be retrieved and add it to a convolutional neural network, which is composed of multiple convolutional layers and pooling layers connected in sequence. The convolutional kernel size in each convolutional layer decreases gradually from front to back, and the number of convolutional kernels increases gradually from front to back. Extract the first feature map from the icon image to be retrieved through the convolutional layer and the pooling layer; Based on the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network. The image segmentation attention mechanism network performs average pooling and max pooling on the first feature map in the channel dimension respectively through the spatial attention mechanism to obtain the first feature map and the second feature map. Concatenate the first feature map and the second feature map in the channel dimension and then perform convolutional processing to determine the spatial dependence relationship of the image region. Perform global average pooling and max pooling on the first feature map in the spatial dimension respectively through the channel attention module to obtain the first feature vector and the second feature vector. Input the first feature vector and the second feature vector into a fully connected network to generate an attention weight map. Perform weighted processing on the first feature map based on the attention weight map and output the graphic region and the text region; Add the text region to the Transformer optical character recognition model. The encoder of the Transformer optical character recognition model divides the text region into image patches and then performs linear projection to convert it into a feature sequence. The feature sequence undergoes multi-head attention processing to obtain encoded features. The encoded features are input into the decoder, and the decoder performs character-by-character decoding in an autoregressive manner through the multi-head attention mechanism to obtain the text content; Input the text content into a bidirectional semantic understanding model. The bidirectional semantic understanding model uses a multi-layer stacked Transformer encoder to perform bidirectional feature extraction on the text content through the multi-head attention mechanism and combines the residual connection to transfer information between layers to obtain a text semantic feature vector; Add the graphic region to a vision transformer network. After the vision transformer network performs image patch division and linear transformation on the graphic region, it conducts cross-layer feature transfer based on a dense connection structure, constructs a feature pyramid at different scales to extract multi-scale features, and combines residual connections to perform feature fusion on the multi-scale features to obtain an image feature vector.

3. The method according to claim 2, characterized in that, Adding the graphic region to a vision transformer network, after the vision transformer network performs image patch division and linear transformation on the graphic region, conducting cross-layer feature transfer based on a dense connection structure, constructing a feature pyramid at different scales to extract multi-scale features, and combining residual connections to perform feature fusion on the multi-scale features to obtain an image feature vector includes: Add the graphic region to a pre-set transformer network, and perform size normalization processing through bilinear interpolation to obtain a normalized graphic region; Divide the normalized graphic region into a sequence of image patches in a regular grid pattern, convert the pixel matrix of each image patch into an image patch feature vector through a linear projection layer, assign a learnable position encoding vector to each image patch position, add the position encoding vector to the image patch feature vector to obtain a feature sequence integrating position information, and establish the spatial relationship between different image patches; Construct a multi-layer transformer encoder network. The multi-layer transformer encoder network adopts a dense connection structure, splices the feature maps output by each layer in the channel dimension as the input of the next layer, adjusts the number of feature channels through a convolutional layer, and conducts feature transfer between different levels to obtain a transferred feature map; Based on the transferred feature map, construct a feature pyramid structure, obtain large-scale feature maps and small-scale feature maps through pooling operations at different scales for the input features. The large-scale feature maps retain the image detail information, and the small-scale feature maps obtain the information of a large receptive field. Upsample the small-scale feature maps through deconvolution operations to unify the spatial resolutions of the large-scale feature maps and the upsampled small-scale feature maps; Splice the feature maps of different scales with unified resolutions in the channel dimension to obtain multi-scale features, adaptively fuse the multi-scale features through a convolutional layer, and adopt a residual connection mechanism to directly superimpose the input features of the current layer onto the output features to obtain a fused output feature; Perform a global pooling operation on the fused output feature to compress the spatial dimension into a low-dimensional feature vector, process the low-dimensional feature vector through a multi-layer non-linear transformation network, introduce non-linear expression ability through an activation function after each transformation, and output an image feature vector containing local details and global semantics.

4. The method according to claim 1, characterized in that, Construct the image feature vector, the text content, and the text semantic feature vector into a heterogeneous feature tensor, perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and factor matrices, perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector, construct a locality-sensitive hashing index based on the icon feature fingerprint vector, calculate optimal hash function parameters based on the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density, encode the icon feature fingerprint vector by combining position-sensitive coding to obtain an encoded fingerprint vector, construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer, add the icon feature fingerprint vector to the fine-grained index layer, and obtain an icon feature multi-dimensional index database including: Preprocess the image feature vector, the text content, and the text semantic feature vector. Perform normalization processing on the image feature vector to eliminate the scale difference between dimensions. Perform word segmentation and encoding conversion on the text content to convert it into a fixed-length feature representation. Perform normalization processing on the text semantic feature vector to make its distribution range consistent with other features. Organize the preprocessed features into a third-order heterogeneous feature tensor; Perform matrix unfolding on the heterogeneous feature tensor along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Perform singular value decomposition on each unfolded matrix to extract the main components, and recombine the main components according to the pre-set rank parameter to obtain a core tensor and factor matrices; Calculate the correlation coefficients between different dimensions in the core tensor to construct a correlation coefficient matrix, perform eigenvalue decomposition on the correlation coefficient matrix to extract the principal components, calculate the contribution rate of each principal component, and select the combination of principal components with a contribution rate exceeding the preset threshold to form an icon feature fingerprint vector; Divide the feature space into multiple regions, count the number of samples in each region to obtain the regional sample density distribution, determine the number of locality-sensitive hashing functions according to the regional sample density distribution, and perform fine-grained division on the regions with a sample density greater than the first density threshold by increasing the number of hashing functions, and perform coarse-grained division on the regions with a sample density less than the second density threshold by reducing the number of hashing functions; Calculate the variance of each dimension of the icon feature fingerprint vector, sort the dimension importance according to the variance value, assign 16-bit binary encoding to the top 5 dimensions in terms of importance, assign 8-bit binary encoding to the dimensions after the fifth place, determine the binary encoding mapping relationship through eigenvalue interval division, and convert the icon feature fingerprint vector into an encoded fingerprint vector; Construct a multi-level index structure. In the coarse-grained index layer, segment the encoded fingerprint vector according to a 32-bit length, calculate a 32-bit hash value for each segment as an index key to establish a hash table, and the hash table points to a storage bucket. When the hash value of the encoded fingerprint vector is within the Hamming distance threshold of 8, store its index in the same storage bucket. In the fine-grained index layer, construct the icon feature fingerprint vector into a tree structure, and each node of the tree represents the numerical interval of the feature vector in the current dimension, to obtain an icon feature multi-dimensional index database.

5. The method according to claim 4, wherein The heterogeneous feature tensor is matrix-expanded along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Singular value decomposition is performed on each expanded matrix to extract the main components, and the main components are recombined according to a preset rank parameter to obtain the core tensor and factor matrices, including: The image features, text content, and text semantic features of the icon data are constructed into a third-order heterogeneous feature tensor, and the third-order heterogeneous feature tensor includes a sample dimension, a feature type dimension, and an eigenvalue dimension; The third-order heterogeneous feature tensor is expanded along the sample dimension to obtain a first expanded matrix, the number of rows of the first expanded matrix is the number of samples, the number of columns is the product of the feature type dimension and the eigenvalue dimension, expanded along the feature type dimension to obtain a second expanded matrix, the number of rows of the second expanded matrix is the number of feature types, the number of columns is the product of the sample dimension and the eigenvalue dimension, and expanded along the eigenvalue dimension to obtain a third expanded matrix, the number of rows of the third expanded matrix is the sum of the eigenvalue dimensions, and the number of columns is the product of the sample dimension and the feature type dimension; The first expanded matrix, the second expanded matrix, and the third expanded matrix are respectively divided into sub-block matrices of equal size, and the size of each sub-block matrix is one-tenth of the corresponding expanded matrix; Perform iterative singular value decomposition on each sub-block matrix, randomly initialize the unit vector, multiply the unit vector by the sub-block matrix and normalize it to obtain the updated vector, repeat the multiplication and normalization operations until the updated vector converges to obtain the first pair of singular vectors, subtract the rank-one matrix constructed from the first pair of singular vectors from the sub-block matrix to obtain the remaining matrix, and repeat the multiplication and normalization operations on the remaining matrix to obtain the next pair of singular vectors; Add L2 norm constraints to the singular vectors during the singular value decomposition process, and use a dynamic learning rate for iterative optimization. The initial value of the dynamic learning rate is 0.1, and it is adaptively adjusted according to the change of the reconstruction error; Select the largest singular value and its corresponding left and right singular vectors from the singular value decomposition results of each sub-block matrix according to the preset rank parameter, fuse the singular values and singular vectors selected from each sub-block matrix through a collaborative filtering algorithm, form the sample dimension factor matrix with the left singular vectors, and form the feature type dimension factor matrix and the eigenvalue dimension factor matrix with the right singular vectors; Adopt a progressive strategy to construct an initial small-scale core tensor, calculate the reconstruction error, and gradually increase the scale of the core tensor when the reconstruction error is greater than the preset threshold and has not reached the maximum scale limit, repeat the increase until the maximum scale limit is reached to obtain the core tensor.

6. The method according to claim 1, wherein Receiving a retrieval request and obtaining a target icon image, performing feature extraction and encoding on the target icon image, generating a target feature fingerprint vector, and performing a quick retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors, calculating the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determining the feature similarity based on the feature distance, and selecting the icon feature fingerprint vectors with the feature similarity greater than a pre-set similarity threshold to generate a similarity candidate set, adding them to a deep metric learning model, calculating the relative distance relationship through a triplet loss function, and combining a contrastive loss function to constrain the feature space distribution, obtaining a metric feature representation, calculating visual similarity, text similarity, and semantic similarity, determining weight parameters by determining the variance and discriminant coefficient of each similarity feature, determining the comprehensive similarity score through weighted summation, sorting in descending order, and selecting the top 3 retrieval results as similar retrieval results for output, including: Receiving a retrieval request and obtaining a target icon image, performing feature extraction and normalization preprocessing on the target icon image, extracting an image feature vector through a multi-layer convolutional network, extracting a text vector and a semantic feature vector through a pre-trained language model, and concatenating the image feature vector, the text vector, and the semantic feature vector to form a target feature fingerprint vector; Segmenting the target feature fingerprint vector into multiple feature segments according to a preset length, and performing binarization processing on the feature values in each feature segment to obtain binary codes, where the binarization processing includes recording the feature values greater than the mean value of the feature segment as the first identifier and the feature values less than the mean value of the feature segment as the second identifier; Using locality-sensitive hashing in the coarse-grained index layer to map the binary codes to hash values, searching for encoded fingerprint vectors with a distance less than a preset threshold from the hash values in the hash table, adding the encoded fingerprint vectors to the initial candidate set, and if the number of samples in the storage area of the hash table exceeds a pre-set quantity threshold, increasing the number of hash functions in the current storage area for regional subdivision; Traversing the encoded fingerprint vectors in the initial candidate set in the feature tree of the fine-grained index layer, where the internal nodes of the feature tree represent regions in the feature space, the leaf nodes store icon feature fingerprint vectors, calculating the feature distance between the target feature fingerprint vector and the center point of the node, selecting the child node with the closest feature distance to continue traversing until the leaf node, and generating a similarity candidate set from the icon feature fingerprint vectors corresponding to the leaf nodes with a feature distance less than a pre-set similarity threshold; Adding the similarity candidate set to a deep metric learning model, where the deep metric learning model includes a multi-layer fully-connected encoder and an attention mechanism-based metric network, selecting the sample with the closest distance to the target feature fingerprint vector from the similarity candidate set as the positive sample, the sample with the farthest distance as the negative sample, using the target feature fingerprint vector as the anchor sample, and combining the anchor sample, the positive sample, and the negative sample to construct a triplet; Calculate the relative distance relationship for the triple through the triple loss function, and combine the contrast loss function to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance from the negative sample. Calculate the feature representation similarity between icon pairs respectively, compare the similarity with a preset target similarity threshold, and filter the icon pairs to form a feature representation set; Based on the deep metric learning model, obtain the metric feature representation. Calculate the visual similarity, text similarity, and semantic similarity between the icon feature fingerprint vectors in the feature representation set and the target feature fingerprint vector. Determine the weight parameters based on the variance and discriminant coefficient of each similarity feature in the training set, and determine the comprehensive similarity score through weighted summation. Sort the comprehensive similarity scores in descending order, and select the top 3 retrieval results as the similar retrieval results for output.

7. The method according to claim 6, wherein Calculate the relative distance relationship for the triple through the triple loss function, and combine the contrast loss function to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance from the negative sample. Calculate the feature representation similarity between icon pairs respectively, compare the similarity with a preset target similarity threshold, and filter the icon pairs to form a feature representation set, including: Receive a triple composed of a target feature fingerprint vector, a positive sample feature fingerprint vector, and a negative sample feature fingerprint vector, and input the triple into a deep metric learning model. The deep metric learning model includes a multi-layer fully connected encoder network and an attention mechanism-based metric network. The encoder network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence. The output dimensions of the first fully connected layer, the second fully connected layer, and the third fully connected layer decrease in sequence. After each fully connected layer, a non-linear activation layer and a batch normalization layer are connected; Calculate the Euclidean distance between the target feature fingerprint vector and the positive sample feature fingerprint vector through the metric network to obtain the positive sample distance, calculate the Euclidean distance between the target feature fingerprint vector and the negative sample feature fingerprint vector to obtain the negative sample distance, and compare the difference between the positive sample distance and the negative sample distance with a preset interval parameter; Calculate the cosine similarity of the feature representation for the icon pairs in the triple. Set a first target similarity for icon pairs of the same category and a second target similarity for icon pairs of different categories. The first target similarity is greater than the second target similarity, and perform feature constraint based on the difference between the cosine similarity of the feature representation and the corresponding target similarity; Calculate the loss value of the triple, select a preset number of triples with the largest loss values as hard samples, and use the hard samples to update the parameters of the encoder network and the metric network; Perform feature mapping on the feature fingerprint vectors of the icon pairs in the similarity candidate set through the encoder network to obtain the mapped features, calculate the cosine similarity between the mapped features through the metric network, and save the icon pairs with the cosine similarity greater than the preset similarity threshold to the feature representation set.

8. A multi-dimensional data intelligent retrieval and matching system for graphic and text features, which is used to implement the method described in any one of the foregoing claims 1-7, characterized in that Including: The first unit is used to obtain the icon image to be retrieved, add it to a convolutional neural network for feature extraction, combine multiple convolutional layers and pooling layers to extract the first feature map, input the icon image to be retrieved into an image segmentation attention mechanism network based on the first feature map, determine the spatial dependence relationship of the image region through the spatial attention mechanism, generate an attention weight map in combination with a channel attention module, and output a graphic region and a text region. Add the text region to a transducer optical character recognition model, convert it into a feature sequence through an encoder, and perform character-by-character decoding through a multi-head attention mechanism in a decoder to obtain text content. Perform feature extraction on the text content through a bidirectional semantic understanding model to obtain a text semantic feature vector. Add the graphic region to a vision transformer network, perform cross-layer feature transfer based on a dense connection structure, and perform feature fusion in combination with a residual connection and a feature pyramid structure to obtain an image feature vector; The second unit is used to construct the image feature vector, the text content, and the text semantic feature vector into a heterogeneous feature tensor, perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and factor matrices, perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector, construct a locality-sensitive hashing index based on the icon feature fingerprint vector, calculate optimal hash function parameters in combination with the distribution density of the icon feature fingerprint vector, and adjust the hash storage unit based on the distribution density. Encode the icon feature fingerprint vector in combination with position-sensitive coding to obtain an encoded fingerprint vector, construct a multi-level index structure, add the encoded fingerprint vector to a coarse-grained index layer, and add the icon feature fingerprint vector to a fine-grained index layer to obtain an icon feature multi-dimensional index database; The third unit is used to receive a retrieval request, obtain a target icon image, perform feature extraction and encoding on the target icon image, generate a target feature fingerprint vector, perform a quick search in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors, calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determine the feature similarity based on the feature distance, and select the icon feature fingerprint vectors with a feature similarity greater than a pre-set similarity threshold to generate a similarity candidate set, add it to a deep metric learning model, calculate the relative distance relationship through a triplet loss function, and constrain the feature space distribution in combination with a contrastive loss function to obtain a metric feature representation, and calculate visual similarity, text similarity, and semantic similarity. Determine the weight parameters by determining the variance and discriminant coefficient of each similarity feature, determine the comprehensive similarity score through weighted summation, sort in descending order, and select the top 3 retrieval results as similar retrieval results for output.

9. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the computer program instructions implement the method according to any one of claims 1 to 7.

Images