Intelligent retrieval and matching method and system for multi-dimensional data of picture-text features

Abstract

The application provides a kind of multi-dimensional data intelligent retrieval matching method and system of graphic features, it is related to icon image retrieval technical field, comprising: using convolutional neural network, image segmentation attention mechanism network, transformer optical character recognition model and bidirectional semantic understanding model extract the image features, text content and semantic features of icon image, the extracted features are constructed as heterogeneous feature tensor and are singular value decomposed, obtain icon feature fingerprint vector, construct multi-level index based on local sensitive hash, realize fast retrieval, combine depth metric learning model to calculate visual, text and semantic similarity, and according to the variance and discriminant coefficient of each similarity feature, weighted, obtain comprehensive similarity score, output the highest similarity search result.

Description

Technical Field

[0001] This invention relates to the field of icon image retrieval technology, and in particular to a multi-dimensional intelligent retrieval and matching method and system for image and text features. Background Technology

[0002] Icon retrieval is a crucial step in icon registration and management. Its purpose is to find registered icons similar to the icon being searched in the icon database to avoid icon infringement and duplicate registration. With the continuous increase in the number of icon applications, traditional retrieval methods based on text or image features are facing increasing challenges.

[0003] Existing icon retrieval technologies mainly rely on single-modal feature matching, such as searching based on features like the shape, color, and texture of an image or on text descriptions. When dealing with complex icons, such as those that combine graphics and text, the retrieval accuracy and efficiency are low. In addition, traditional retrieval methods are difficult to effectively handle icon deformation, noise, and occlusion, leading to a decrease in the accuracy of retrieval results. They also suffer from insufficient semantic understanding, failing to accurately capture the meaning and connotation of icons and making it difficult to distinguish between similar icons.

[0004] Therefore, a solution is urgently needed to address the problems existing in the current technology. Summary of the Invention

[0005] This invention provides a multi-dimensional intelligent data retrieval and matching method and system based on image and text features, which can at least solve some of the problems existing in the prior art.

[0006] A first aspect of this invention provides a multi-dimensional data intelligent retrieval and matching method based on image and text features, comprising:

[0007] The image of the icon to be retrieved is acquired and added to a convolutional neural network for feature extraction. A first feature map is obtained by combining multiple convolutional and pooling layers. Based on the first feature map, the image of the icon to be retrieved is input into an image segmentation attention mechanism network. The spatial dependence of the image region is determined by the spatial attention mechanism. An attention weight map is generated by combining the channel attention module and the graphic region and text region are output. The text region is added to the transducer optical character recognition model. It is converted into a feature sequence by the encoder and decoded character by character by character by the multi-head attention mechanism in the decoder to obtain the text content. The text content is feature extracted by the bidirectional semantic understanding model to obtain the text semantic feature vector. The graphic region is added to the visual transducer network. Feature cross-layer transfer is performed based on the dense connection structure. Feature fusion is performed by combining residual connections and feature pyramid structure to obtain the image feature vector.

[0008] The image feature vector, the text content, and the text semantic feature vector are constructed into a heterogeneous feature tensor. Singular value decomposition is performed on the heterogeneous feature tensor to obtain a core tensor and a factor matrix. Correlation analysis decomposition is performed on the core tensor to obtain an icon feature fingerprint vector. A local sensitive hash index is constructed based on the icon feature fingerprint vector. The optimal hash function parameters are calculated by combining the distribution density of the icon feature fingerprint vector and the hash storage unit is adjusted based on the distribution density. The icon feature fingerprint vector is encoded by combining position-sensitive coding to obtain an encoded fingerprint vector. A multi-level index structure is constructed and the encoded fingerprint vector is added to the coarse-grained index layer. The icon feature fingerprint vector is added to the fine-grained index layer to obtain an icon feature multidimensional index database.

[0009] The system receives a retrieval request and acquires a target icon image. It then performs feature extraction and encoding on the target icon image to generate a target feature fingerprint vector. A fast retrieval is performed in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. The feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer is calculated. Based on the feature distance, feature similarity is determined, and icon feature fingerprint vectors with feature similarity greater than a pre-set similarity threshold are selected to generate a similarity candidate set. This set is added to the deep metric learning model. The relative distance relationship is calculated using a triplet loss function, and the feature space distribution is constrained by a contrastive loss function to obtain a metric feature representation. Visual similarity, textual similarity, and semantic similarity are calculated. Weight parameters are determined by determining the variance and discriminant coefficient of each similarity feature. A comprehensive similarity score is determined by weighted summation. The results are sorted in descending order, and the top three retrieval results are selected as the output similarity retrieval results.

[0010] In one alternative implementation,

[0011] The process involves acquiring the image of the icon to be retrieved and adding it to a convolutional neural network for feature extraction. A first feature map is obtained by combining multiple convolutional and pooling layers. Based on this first feature map, the image of the icon to be retrieved is input into an image segmentation attention mechanism network. Spatial dependencies of image regions are determined through a spatial attention mechanism. An attention weight map is generated using a channel attention module, and graphic and text regions are output. The text region is added to a transducer optical character recognition model. It is converted into a feature sequence by an encoder and decoded character-by-character using a multi-head attention mechanism in the decoder to obtain the text content. Features are extracted from the text content using a bidirectional semantic understanding model to obtain a text semantic feature vector. The graphic region is added to a visual transducer network. Feature cross-layer transfer is performed based on a dense connection structure. Feature fusion is performed using residual connections and a feature pyramid structure to obtain an image feature vector, including:

[0012] The image of the icon to be retrieved is obtained and added to the convolutional neural network. The convolutional neural network consists of multiple convolutional layers and pooling layers connected in sequence. The size of the convolutional kernel in each convolutional layer decreases from front to back, and the number of convolutional kernels increases from front to back. The first feature map is obtained by extracting features from the image of the icon to be retrieved through the convolutional layers and pooling layers.

[0013] Based on the first feature map, the image of the icon to be retrieved is input into the image segmentation attention mechanism network. The image segmentation attention mechanism network performs average pooling and max pooling on the first feature map in the channel dimension through the spatial attention mechanism to obtain the first feature map and the second feature map. The first feature map and the second feature map are concatenated in the channel dimension and then processed by convolution to determine the spatial dependency of the image region. The first feature map is then processed by the channel attention module to perform global average pooling and max pooling on the first feature map in the spatial dimension to obtain the first feature vector and the second feature vector. The first feature vector and the second feature vector are input into the fully connected network to generate an attention weight map. Based on the attention weight map, the first feature map is weighted and processed to output the graphic region and the text region.

[0014] The text region is added to the converter optical character recognition model. The encoder of the converter optical character recognition model divides the text region into image blocks and then performs linear projection to convert it into a feature sequence. The feature sequence is processed by multi-head attention to obtain encoded features. The encoded features are input into the decoder. The decoder uses an autoregressive method through a multi-head attention mechanism to decode character by character to obtain the text content.

[0015] The text content is input into the bidirectional semantic understanding model, which employs a multi-layer stacked transformer encoder. The model extracts bidirectional features from the text content through a multi-head attention mechanism and combines residual connections to pass inter-layer information, thereby obtaining a text semantic feature vector.

[0016] The graphic region is added to the visual transformer network. After the visual transformer network divides the graphic region into image blocks and performs linear transformation, it performs cross-layer feature transfer based on a dense connection structure. It constructs feature pyramids at different scales to extract multi-scale features and combines residual connections to perform feature fusion on the multi-scale features to obtain the image feature vector.

[0017] In one alternative implementation,

[0018] The graphic region is added to a visual transformer network. After the visual transformer network performs image patching and linear transformation on the graphic region, it performs cross-layer feature transfer based on a dense connection structure. It constructs feature pyramids at different scales to extract multi-scale features. Combined with residual connections, it performs feature fusion on the multi-scale features to obtain an image feature vector, including:

[0019] The graphic region is added to a pre-set transformer network, and the size is normalized by bilinear interpolation to obtain a normalized graphic region.

[0020] The normalized graphic region is divided into a regular grid-like sequence of image blocks. The pixel matrix of each image block is converted into an image block feature vector through a linear projection layer. A learnable position encoding vector is assigned to each image block position. The position encoding vector is added to the image block feature vector to obtain a feature sequence with fused position information, and the spatial relationship between different image blocks is established.

[0021] A multi-layer transformer encoder network is constructed. The multi-layer transformer encoder network adopts a dense connection structure. The feature map output by each layer is concatenated in the channel dimension and used as the input of the next layer. The number of feature channels is adjusted through convolutional layers, and feature transfer is performed between different layers to obtain the transferred feature map.

[0022] Based on the transmitted feature map, a feature pyramid structure is constructed. The input features are pooled at different scales to obtain large-scale feature maps and small-scale feature maps. The large-scale feature map retains image detail information, while the small-scale feature map acquires a large receptive field information. The small-scale feature map is upsampled through deconvolution to unify the spatial resolution of the large-scale feature map and the upsampled small-scale feature map.

[0023] Multi-scale features are obtained by concatenating feature maps of different scales after unifying resolution along the channel dimension. The multi-scale features are then adaptively fused through a convolutional layer. The input features of the current layer are directly superimposed on the output features using a residual connection mechanism to obtain the fused output features.

[0024] The fused output features are subjected to global pooling to compress the spatial dimension into a low-dimensional feature vector. The low-dimensional feature vector is then processed through a multi-layer nonlinear transformation network. After each transformation, an activation function is used to introduce nonlinear expressive power, and the output image feature vector contains local details and global semantics.

[0025] In one alternative implementation,

[0026] The image feature vector, the text content, and the text semantic feature vector are constructed into a heterogeneous feature tensor. Singular value decomposition is performed on the heterogeneous feature tensor to obtain a core tensor and a factor matrix. Correlation analysis is performed on the core tensor to obtain an icon feature fingerprint vector. A local sensitive hash index is constructed based on the icon feature fingerprint vector. Optimal hash function parameters are calculated based on the distribution density of the icon feature fingerprint vector, and the hash storage unit is adjusted based on the distribution density. The icon feature fingerprint vector is encoded using position-sensitive coding to obtain an encoded fingerprint vector. A multi-level index structure is constructed, and the encoded fingerprint vector is added to the coarse-grained index layer. The icon feature fingerprint vector is added to the fine-grained index layer, resulting in an icon feature multidimensional index database comprising:

[0027] The image feature vector, the text content, and the text semantic feature vector are preprocessed. The image feature vector is standardized to eliminate scale differences between dimensions. The text content is segmented and encoded to convert it into a fixed-length feature representation. The text semantic feature vector is normalized to make its distribution range consistent with other features. The preprocessed features are organized into a third-order heterogeneous feature tensor.

[0028] The heterogeneous feature tensor is expanded along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Singular value decomposition is performed on each expanded matrix to extract the main components. The main components are then recombined according to a pre-set rank parameter to obtain the core tensor and factor matrix.

[0029] Calculate the correlation coefficients between different dimensions in the core tensor to construct a correlation coefficient matrix, perform eigenvalue decomposition on the correlation coefficient matrix to extract principal components, calculate the contribution rate of each principal component, and select principal components with contribution rates exceeding a preset threshold to form an icon feature fingerprint vector.

[0030] The feature space is divided into multiple regions, and the number of samples in each region is counted to obtain the region sample density distribution. The number of local sensitive hash functions is determined based on the region sample density distribution. Regions with sample density greater than the first density threshold are divided into fine-grained segments by increasing the number of hash functions, while regions with sample density less than the second density threshold are divided into coarse-grained segments by decreasing the number of hash functions.

[0031] Calculate the variance of each dimension of the icon feature fingerprint vector, sort the dimensions according to the variance values, assign 16-bit binary codes to the top 5 most important dimensions, and assign 8-bit binary codes to the dimensions after the fifth most important dimension. Determine the binary code mapping relationship by dividing the feature value interval, and convert the icon feature fingerprint vector into an encoded fingerprint vector.

[0032] A multi-level index structure is constructed. In the coarse-grained index layer, the encoded fingerprint vector is segmented into 32-bit segments, and a 32-bit hash value is calculated for each segment as the index key to build a hash table. The hash table points to a storage bucket. When the hash value of the encoded fingerprint vector is within the Hamming distance threshold of 8, its index is stored in the same storage bucket. In the fine-grained index layer, the icon feature fingerprint vector is constructed into a tree structure. Each node of the tree represents the numerical range of the feature vector in the current dimension, thus obtaining a multi-dimensional index database of icon features.

[0033] In one alternative implementation,

[0034] The heterogeneous feature tensor is expanded along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Singular value decomposition is performed on each expanded matrix to extract principal components. These principal components are then recombined according to a pre-set rank parameter to obtain the core tensor and factor matrix, including:

[0035] The image features, text content, and text semantic features of the icon data are constructed into a third-order heterogeneous feature tensor, which includes sample dimension, feature type dimension, and feature value dimension.

[0036] The third-order heterogeneous feature tensor is expanded along the sample dimension to obtain a first expanded matrix, where the number of rows in the first expanded matrix is ​​the number of samples and the number of columns is the product of the feature type dimension and the feature value dimension. Expanding along the feature type dimension yields a second expanded matrix, where the number of rows in the second expanded matrix is ​​the number of feature types and the number of columns is the product of the sample dimension and the feature value dimension. Expanding along the feature value dimension yields a third expanded matrix, where the number of rows in the third expanded matrix is ​​the sum of the feature value dimensions and the number of columns is the product of the sample dimension and the feature type dimension.

[0037] The first expansion matrix, the second expansion matrix, and the third expansion matrix are each divided into sub-block matrices of equal size, with each sub-block matrix being one-tenth the size of the corresponding expansion matrix;

[0038] For each sub-block matrix, perform iterative singular value decomposition, randomly initialize unit vectors, multiply the unit vectors with the sub-block matrix and normalize them to obtain an update vector, repeat the multiplication and normalization operation until the update vector converges to obtain the first pair of singular vectors, subtract the rank-one matrix constructed from the first pair of singular vectors from the sub-block matrix to obtain the remaining matrix, and repeat the multiplication and normalization operation on the remaining matrix to obtain the next pair of singular vectors;

[0039] In the singular value decomposition process, L2 norm constraints are added to the singular vectors, and iterative optimization is performed using a dynamic learning rate. The initial value of the dynamic learning rate is 0.1, which is adaptively adjusted according to the changes in reconstruction error.

[0040] Based on the pre-set rank parameter, the largest singular value and its corresponding left and right singular vectors are selected from the singular value decomposition results of each sub-block matrix. The singular values ​​and singular vectors after screening of each sub-block matrix are fused by a collaborative filtering algorithm. The left singular vectors are used to form a sample dimension factor matrix, and the right singular vectors are used to form a feature type dimension factor matrix and an eigenvalue dimension factor matrix.

[0041] An initial small-scale core tensor is constructed using a progressive strategy. The reconstruction error is calculated. When the reconstruction error is greater than a preset threshold but has not reached the maximum size limit, the size of the core tensor is gradually increased. This process is repeated until the maximum size limit is reached, thus obtaining the core tensor.

[0042] In one alternative implementation,

[0043] The system receives a retrieval request and acquires a target icon image. It then performs feature extraction and encoding on the target icon image to generate a target feature fingerprint vector. A fast retrieval is performed in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. The feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer is calculated. Based on the feature distance, feature similarity is determined, and icon feature fingerprint vectors with feature similarity greater than a pre-set similarity threshold are selected to generate a similarity candidate set. This set is added to the deep metric learning model. The relative distance relationship is calculated using a triplet loss function, and the feature space distribution is constrained by a contrastive loss function to obtain a metric feature representation. Visual similarity, text similarity, and semantic similarity are calculated. Weight parameters are determined by determining the variance and discriminant coefficient of each similarity feature. A comprehensive similarity score is determined by weighted summation. The results are sorted in descending order, and the top three retrieval results are selected as the output similarity retrieval results, including:

[0044] The system receives a retrieval request and obtains a target icon image. It performs feature extraction and normalization preprocessing on the target icon image, extracts image feature vectors through a multi-layer convolutional network, extracts text vectors and semantic feature vectors through a pre-trained language model, and concatenates the image feature vectors, the text vectors, and the semantic feature vectors to form a target feature fingerprint vector.

[0045] The target feature fingerprint vector is divided into multiple feature segments according to a preset length. The feature values ​​in each feature segment are binarized to obtain binary codes. The binarization process includes recording feature values ​​greater than the average value of the feature segment as the first identifier and feature values ​​less than the average value of the feature segment as the second identifier.

[0046] Locality-sensitive hashing is used in the coarse-grained index layer to map the binary code to a hash value. The encoded fingerprint vector that is less than a preset threshold away from the hash value is searched in the hash table and added to the initial candidate set. If the number of samples in the storage area in the hash table exceeds a preset threshold, the number of hash functions in the current storage area is increased to further subdivide the area.

[0047] In the feature tree of the fine-grained index layer, the encoded fingerprint vectors in the initial candidate set are traversed. The internal nodes of the feature tree represent regions in the feature space, and the leaf nodes store icon feature fingerprint vectors. The feature distance between the target feature fingerprint vector and the node center point is calculated. The child node with the closest feature distance is selected and traversal continues until the leaf node. The icon feature fingerprint vectors corresponding to the leaf nodes with feature distances less than a preset similarity threshold are used to generate a similarity candidate set.

[0048] The similarity candidate set is added to the deep metric learning model, wherein the deep metric learning model includes a multi-layer fully connected encoder and a metric network based on an attention mechanism. The sample closest to the target feature fingerprint vector is selected from the similarity candidate set as a positive sample, and the sample farthest away is selected as a negative sample. The target feature fingerprint vector is used as an anchor sample, and the anchor sample, the positive sample, and the negative sample are combined to construct a triple.

[0049] The relative distance relationship of the triplet is calculated using the triplet loss function. Combined with the contrastive loss function to constrain the feature space distribution, the distance between the target feature fingerprint vector and the positive sample is less than the distance with the negative sample. The feature representation similarity between the icon pairs is calculated separately. The similarity is compared with the preset target similarity threshold, and the icon pairs are selected to form a feature representation set.

[0050] Based on the deep metric learning model, a metric feature representation is obtained. Visual similarity, text similarity, and semantic similarity are calculated between the icon feature fingerprint vector in the feature representation set and the target feature fingerprint vector. Weight parameters are determined based on the variance and discriminant coefficient of each similarity feature in the training set. A comprehensive similarity score is determined by weighted summation. The comprehensive similarity scores are sorted in descending order, and the top 3 search results are selected as similar search results for output.

[0051] In one alternative implementation,

[0052] The relative distance relationship of the triplet is calculated using a triplet loss function, and the feature space distribution is constrained by a contrastive loss function to ensure that the distance between the target feature fingerprint vector and positive samples is less than the distance with negative samples. The feature representation similarity between icon pairs is calculated separately, and the similarity is compared with a preset target similarity threshold. The selection of icon pairs to form a feature representation set includes:

[0053] The system receives a triplet consisting of a target feature fingerprint vector, a positive sample feature fingerprint vector, and a negative sample feature fingerprint vector. The triplet is then input into a deep metric learning model. The deep metric learning model includes a multi-layer fully connected encoder network and an attention-based metric network. The encoder network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence. The output dimensions of the first fully connected layer, the second fully connected layer, and the third fully connected layer decrease sequentially. Each fully connected layer is followed by a non-linear activation layer and a batch normalization layer.

[0054] The positive sample distance is obtained by calculating the Euclidean distance between the target feature fingerprint vector and the positive sample feature fingerprint vector through the metric network, and the negative sample distance is obtained by calculating the Euclidean distance between the target feature fingerprint vector and the negative sample feature fingerprint vector. The difference between the positive sample distance and the negative sample distance is compared with a preset interval parameter.

[0055] The cosine similarity of the feature representations of the icon pairs in the triplet is calculated. A first target similarity is set for icon pairs of the same category, and a second target similarity is set for icon pairs of different categories. The first target similarity is greater than the second target similarity. Feature constraints are performed based on the difference between the cosine similarity of the feature representations and the corresponding target similarity.

[0056] Calculate the loss value of the triplet, select a preset number of triplets with the largest loss value as hard samples, and use the hard samples for parameter updates of the encoder network and the metric network;

[0057] The feature fingerprint vectors of icon pairs in the similarity candidate set are mapped by the encoder network to obtain mapped features. The cosine similarity between the mapped features is calculated by the metric network. Icon pairs with a cosine similarity greater than a preset similarity threshold are saved to the feature representation set.

[0058] A second aspect of the present invention provides a multi-dimensional data intelligent retrieval and matching system based on image and text features, comprising:

[0059] The first unit is used to acquire the image of the icon to be retrieved and add it to the convolutional neural network for feature extraction. It extracts a first feature map by combining multiple convolutional layers and pooling layers. Based on the first feature map, the image of the icon to be retrieved is input into the image segmentation attention mechanism network. The spatial dependence of the image region is determined by the spatial attention mechanism. The attention weight map is generated by the channel attention module and the graphic region and text region are output. The text region is added to the transducer optical character recognition model. It is converted into a feature sequence by the encoder and decoded character by character by character by the multi-head attention mechanism in the decoder to obtain the text content. The text content is feature extracted by the bidirectional semantic understanding model to obtain the text semantic feature vector. The graphic region is added to the visual transducer network. The feature is transferred across layers based on the dense connection structure. The feature is fused by the residual connection and the feature pyramid structure to obtain the image feature vector.

[0060] The second unit is used to construct a heterogeneous feature tensor from the image feature vector, the text content, and the text semantic feature vector; perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and a factor matrix; perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector; construct a local sensitive hash index based on the icon feature fingerprint vector; calculate the optimal hash function parameters based on the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density; encode the icon feature fingerprint vector using position-sensitive coding to obtain an encoded fingerprint vector; construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer; add the icon feature fingerprint vector to the fine-grained index layer to obtain an icon feature multidimensional index database.

[0061] The third unit is used to receive retrieval requests and obtain target icon images, perform feature extraction and encoding on the target icon images, generate target feature fingerprint vectors, and perform fast retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. It calculates the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determines the feature similarity based on the feature distance, and selects icon feature fingerprint vectors with feature similarity greater than a preset similarity threshold to generate a similarity candidate set, which is added to the deep metric learning model. The relative distance relationship is calculated through the triplet loss function, and the feature space distribution is constrained by the contrastive loss function to obtain the metric feature representation and calculate visual similarity, text similarity, and semantic similarity. The weight parameters are determined by determining the variance and discriminant coefficient of each similarity feature, and the comprehensive similarity score is determined by weighted summation. The results are sorted in descending order, and the top 3 retrieval results are selected as the similarity retrieval results for output.

[0062] A third aspect of the present invention,

[0063] An electronic device is provided, comprising:

[0064] processor;

[0065] Memory used to store processor-executable instructions;

[0066] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0067] Fourth aspect of the embodiments of the present invention,

[0068] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0069] This invention combines image segmentation attention mechanisms, transducer optical character recognition, and bidirectional semantic understanding models to more accurately extract graphic features, text content, and semantic information from icon images. It constructs a multi-dimensional heterogeneous feature tensor, uses locality-sensitive hashing to build a multi-level index structure, and combines position-sensitive coding and distribution density adaptive adjustment of hash storage units to achieve rapid retrieval of massive icon data. A deep metric learning model is used to reorder the candidate set, and more discriminative feature representations are learned through triplet loss and contrastive loss functions. Visual similarity, text similarity, and semantic similarity are combined for comprehensive evaluation, ultimately outputting the retrieval result with the highest similarity, ensuring the reliability of the retrieval results. Attached Figure Description

[0070] Figure 1 This is a flowchart illustrating the intelligent retrieval and matching method for multidimensional data based on image and text features according to an embodiment of the present invention.

[0071] Figure 2 This is a schematic diagram of the structure of the multidimensional data intelligent retrieval and matching system based on image and text features according to an embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0074] Figure 1 This is a flowchart illustrating the intelligent retrieval and matching method for multidimensional data based on image and text features according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0075] S1. Obtain the image of the icon to be retrieved and add it to the convolutional neural network for feature extraction. Combine multiple convolutional layers and pooling layers to extract the first feature map. Based on the first feature map, input the image of the icon to be retrieved into the image segmentation attention mechanism network. Determine the spatial dependency of the image region through the spatial attention mechanism. Combine the channel attention module to generate an attention weight map and output the graphic region and text region. Add the text region to the transducer optical character recognition model. Convert it into a feature sequence through the encoder and decode it character by character through the multi-head attention mechanism in the decoder to obtain the text content. Extract features from the text content through the bidirectional semantic understanding model to obtain the text semantic feature vector. Add the graphic region to the visual transducer network. Perform feature cross-layer transfer based on the dense connection structure. Combine residual connection and feature pyramid structure to perform feature fusion to obtain the image feature vector.

[0076] The image segmentation attention mechanism network is a neural network that introduces an attention mechanism into image segmentation tasks. By emphasizing important regions in the image, it dynamically adjusts the network's focus to improve segmentation accuracy, especially in complex backgrounds or detail processing. The transformer optical character recognition model is a model that uses a transformer architecture for optical character recognition and employs a self-attention mechanism to handle the dependencies between character sequences. The bidirectional semantic understanding model is a model that can simultaneously consider contextual information. It is usually based on a bidirectional neural network structure and can capture the dependencies between words or sentences. It is widely used in natural language processing tasks. The visual transformer network is a visual processing network based on a transformer model. It uses a self-attention mechanism to model global information for each part of the image and can handle long-distance dependencies. It is particularly suitable for processing large-scale images and complex visual tasks. The dense connection structure means that each layer in the network is connected to all previous layers, which improves feature reusability and gradient flow efficiency and reduces the gradient vanishing problem in network training.

[0077] In one alternative implementation,

[0078] The process involves acquiring the image of the icon to be retrieved and adding it to a convolutional neural network for feature extraction. A first feature map is obtained by combining multiple convolutional and pooling layers. Based on this first feature map, the image of the icon to be retrieved is input into an image segmentation attention mechanism network. Spatial dependencies of image regions are determined through a spatial attention mechanism. An attention weight map is generated using a channel attention module, and graphic and text regions are output. The text region is added to a transducer optical character recognition model. It is converted into a feature sequence by an encoder and decoded character-by-character using a multi-head attention mechanism in the decoder to obtain the text content. Features are extracted from the text content using a bidirectional semantic understanding model to obtain a text semantic feature vector. The graphic region is added to a visual transducer network. Feature cross-layer transfer is performed based on a dense connection structure. Feature fusion is performed using residual connections and a feature pyramid structure to obtain an image feature vector, including:

[0079] The image of the icon to be retrieved is obtained and added to the convolutional neural network. The convolutional neural network consists of multiple convolutional layers and pooling layers connected in sequence. The size of the convolutional kernel in each convolutional layer decreases from front to back, and the number of convolutional kernels increases from front to back. The first feature map is obtained by extracting features from the image of the icon to be retrieved through the convolutional layers and pooling layers.

[0080] Based on the first feature map, the image of the icon to be retrieved is input into the image segmentation attention mechanism network. The image segmentation attention mechanism network performs average pooling and max pooling on the first feature map in the channel dimension through the spatial attention mechanism to obtain the first feature map and the second feature map. The first feature map and the second feature map are concatenated in the channel dimension and then processed by convolution to determine the spatial dependency of the image region. The first feature map is then processed by the channel attention module to perform global average pooling and max pooling on the first feature map in the spatial dimension to obtain the first feature vector and the second feature vector. The first feature vector and the second feature vector are input into the fully connected network to generate an attention weight map. Based on the attention weight map, the first feature map is weighted and processed to output the graphic region and the text region.

[0081] The text region is added to the converter optical character recognition model. The encoder of the converter optical character recognition model divides the text region into image blocks and then performs linear projection to convert it into a feature sequence. The feature sequence is processed by multi-head attention to obtain encoded features. The encoded features are input into the decoder. The decoder uses an autoregressive method through a multi-head attention mechanism to decode character by character to obtain the text content.

[0082] The text content is input into the bidirectional semantic understanding model, which employs a multi-layer stacked transformer encoder. The model extracts bidirectional features from the text content through a multi-head attention mechanism and combines residual connections to pass inter-layer information, thereby obtaining a text semantic feature vector.

[0083] The graphic region is added to the visual transformer network. After the visual transformer network divides the graphic region into image blocks and performs linear transformation, it performs cross-layer feature transfer based on a dense connection structure. It constructs feature pyramids at different scales to extract multi-scale features and combines residual connections to perform feature fusion on the multi-scale features to obtain the image feature vector.

[0084] The bidirectional feature extraction refers to extracting features from the preceding and following information of the input sequence simultaneously through a bidirectional network structure (such as a bidirectional LSTM or a bidirectional transformer). It has wide applications in natural language processing and speech recognition, and can improve the ability to capture contextual information. The character-by-character decoding is a method of decoding at the character level, which is often used in optical character recognition tasks. By decoding each character one by one, rather than processing the entire sentence or paragraph of text at the same time, the accuracy and controllability of recognition are improved.

[0085] Retrieve the icon image to be searched. For example, a user uploads an icon image to be searched, in PNG format, with a size of 256x256 pixels.

[0086] The icon image to be retrieved is input into a pre-trained convolutional neural network (CNN). The CNN contains multiple convolutional and pooling layers. For example, a CNN might contain four convolutional layers and three pooling layers. The first convolutional layer uses 64 kernels of size 7x7, the second uses 128 kernels of size 5x5, the third uses 256 kernels of size 3x3, and the fourth uses 512 kernels of size 3x3. All pooling layers use 2x2 max pooling. Convolutional and pooling layers are alternately connected, with each convolutional layer followed by a pooling layer. After processing by the convolutional and pooling layers, the first feature map, with a size of 16x16x512, is extracted.

[0087] The first feature map and the original 256x256 icon image are input into the image segmentation attention mechanism network. This network includes a spatial attention mechanism and a channel attention module. The spatial attention mechanism performs average pooling and max pooling on the first feature map along the channel dimension, resulting in two 16x16x1 feature maps. These two feature maps are concatenated along the channel dimension and then processed through a 3x3 convolutional layer to obtain a 16x16x1 spatial attention map, which reflects the spatial dependencies of image regions. The channel attention module performs global average pooling and max pooling on the first feature map along the spatial dimension, resulting in two 1x1x512 feature vectors. These two feature vectors are input into two fully connected layers with ReLU activation functions, and the outputs are summed and then passed through a Sigmoid activation function to generate a 1x1x512 channel attention weight map. The spatial attention map and the channel attention weight map are multiplied together, and then multiplied by the first feature map to obtain a weighted feature map. Finally, a segmentation network, such as U-Net, is used to process the weighted feature maps and output the segmentation results of graphic and text regions, such as two 256x256 binary images, where pixels with a value of 1 represent the corresponding regions.

[0088] The text region image is input into the optical character recognition model. The encoder of this model divides the text region image into 16x16 image blocks and linearly projects each block into a feature sequence. This feature sequence is then processed by a multi-head attention mechanism to obtain encoded features. These encoded features are input into the decoder, which uses an autoregressive approach with a multi-head attention mechanism to decode each character and obtain the text content. For example, the recognition result might be "example icon".

[0089] Text content is input into a bidirectional semantic understanding model, such as BERT. The bidirectional semantic understanding model uses a multi-layer stacked transformer encoder, performs bidirectional feature extraction on the text content through a multi-head attention mechanism, and combines residual connections to pass inter-layer information to obtain a text semantic feature vector, such as a 768-dimensional vector.

[0090] The image of the graphic region is input into the Visual Transformer Network (ViT). This network divides the image of the graphic region into 16x16 image blocks and performs a linear transformation. It performs feature transfer across layers based on a dense connection structure, and constructs feature pyramids at different scales to extract multi-scale features. It then combines residual connections to fuse the multi-scale features to obtain an image feature vector, such as a 1024-dimensional vector.

[0091] In this embodiment, by combining image features and text semantic features for retrieval, similar icons can be matched more accurately, reducing the false positive rate. It can identify the text content and graphic details in the icons, and supports fine-grained retrieval based on text, graphics, or a combination of both, meeting the diverse retrieval needs of users. Even if the icon image has deformation, noise, or occlusion, the method can still effectively extract features and perform matching, improving the robustness of the retrieval.

[0092] In one alternative implementation,

[0093] The graphic region is added to a visual transformer network. After the visual transformer network performs image patching and linear transformation on the graphic region, it performs cross-layer feature transfer based on a dense connection structure. It constructs feature pyramids at different scales to extract multi-scale features. Combined with residual connections, it performs feature fusion on the multi-scale features to obtain an image feature vector, including:

[0094] The graphic region is added to a pre-set transformer network, and the size is normalized by bilinear interpolation to obtain a normalized graphic region.

[0095] The normalized graphic region is divided into a regular grid-like sequence of image blocks. The pixel matrix of each image block is converted into an image block feature vector through a linear projection layer. A learnable position encoding vector is assigned to each image block position. The position encoding vector is added to the image block feature vector to obtain a feature sequence with fused position information, and the spatial relationship between different image blocks is established.

[0096] A multi-layer transformer encoder network is constructed. The multi-layer transformer encoder network adopts a dense connection structure. The feature map output by each layer is concatenated in the channel dimension and used as the input of the next layer. The number of feature channels is adjusted through convolutional layers, and feature transfer is performed between different layers to obtain the transferred feature map.

[0097] Based on the transmitted feature map, a feature pyramid structure is constructed. The input features are pooled at different scales to obtain large-scale feature maps and small-scale feature maps. The large-scale feature map retains image detail information, while the small-scale feature map acquires a large receptive field information. The small-scale feature map is upsampled through deconvolution to unify the spatial resolution of the large-scale feature map and the upsampled small-scale feature map.

[0098] Multi-scale features are obtained by concatenating feature maps of different scales after unifying resolution along the channel dimension. The multi-scale features are then adaptively fused through a convolutional layer. The input features of the current layer are directly superimposed on the output features using a residual connection mechanism to obtain the fused output features.

[0099] The fused output features are subjected to global pooling to compress the spatial dimension into a low-dimensional feature vector. The low-dimensional feature vector is then processed through a multi-layer nonlinear transformation network. After each transformation, an activation function is used to introduce nonlinear expressive power, and the output image feature vector contains local details and global semantics.

[0100] The pixel matrix refers to the matrix composed of the values ​​of each pixel in the image. It is usually represented by a two-dimensional array and is widely used in image processing, computer vision and other fields. The deconvolution operation is an operation in convolutional neural networks. It is used to upsample the image by reversing the convolution operation to restore the spatial resolution of the image. It is often used in generative adversarial networks and image generation tasks.

[0101] The image region to be processed is obtained and added to a pre-trained visual transformer network. This visual transformer network has been trained on a large-scale dataset and has powerful image feature extraction capabilities. To adapt to the network's input size, the image region is normalized using bilinear interpolation, for example, scaling a 1024x768 image region to 224x224.

[0102] The normalized graphic region is divided into a sequence of regular grid-like image patches. For example, a 224x224 graphic region is divided into 14x14 16x16 image patches. The pixel matrix of each image patch is converted into an image patch feature vector through a linear projection layer. Simultaneously, a learnable positional encoding vector is assigned to each image patch location to record its positional information within the original graphic region. The positional encoding vector is added to the corresponding image patch feature vector to obtain a feature sequence with fused positional information, thereby establishing the spatial relationship between different image patches. Assuming the linear projection layer converts a 16x16 image patch into a 768-dimensional feature vector, and the positional encoding vector is also 768-dimensional, the sum will still be a 768-dimensional feature vector.

[0103] A multi-layer transformer encoder network is constructed. This network employs a densely connected structure, concatenating the feature maps output from each layer along the channel dimension before using them as input to the next layer. For example, if the first transformer encoder layer outputs a feature map with 768 channels and the second layer outputs a feature map with 1024 channels, these two feature maps are concatenated along the channel dimension to obtain a feature map with 1792 channels, which is then used as input to the third layer. To control the feature dimension, a convolutional layer is added after each layer to adjust the number of feature channels, for example, reducing the number of channels from 1792 to 768. This densely connected structure allows for feature transfer between different layers, enabling the network to learn richer feature representations.

[0104] Following the multi-layer transformer encoder network, a feature pyramid structure is constructed. Pooling operations at different scales, such as max pooling, are performed on the feature maps output by the network to obtain large-scale and small-scale feature maps. Large-scale feature maps preserve image detail information, for example, using a 2x2 pooling kernel; small-scale feature maps capture information from a large receptive field, for example, using a 4x4 pooling kernel. Deconvolution operations are then used to upsample the small-scale feature maps to match the spatial resolution of the large-scale feature maps. For example, the feature map obtained after 4x4 pooling is upsampled by a factor of 2.

[0105] Multi-scale features are obtained by concatenating feature maps of different scales at a unified resolution along the channel dimension. For example, feature maps obtained by 2x2 pooling and upsampled 4x4 pooling are concatenated together, and multi-scale features are adaptively fused through convolutional layers. To preserve the original feature information, a residual connection mechanism is used to directly superimpose the input features of the current layer onto the output features, resulting in fused output features.

[0106] Global pooling, such as global average pooling, is performed on the fused output features to compress the spatial dimension into a low-dimensional feature vector. This low-dimensional feature vector is then processed through a multi-layer nonlinear transformation network. After each transformation, an activation function, such as the ReLU function, is applied to introduce nonlinear expressive power, ultimately outputting an image feature vector that contains both local details and global semantics.

[0107] In this embodiment, by combining the visual transformer network and the dense connection structure, richer image features can be extracted, improving the model's expressive power. The feature pyramid structure can effectively fuse feature information at different scales, enabling the model to take into account both local details and global semantics. The introduction of the residual connection mechanism can enhance the model's robustness and improve its generalization ability.

[0108] S2. Construct a heterogeneous feature tensor from the image feature vector, the text content, and the text semantic feature vector. Perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and a factor matrix. Perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector. Construct a local sensitive hash index based on the icon feature fingerprint vector. Calculate the optimal hash function parameters based on the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density. Encode the icon feature fingerprint vector using position-sensitive coding to obtain an encoded fingerprint vector. Construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer. Add the icon feature fingerprint vector to the fine-grained index layer to obtain an icon feature multidimensional index database.

[0109] The heterogeneous feature tensor is a tensor used to represent complex multidimensional data, typically containing different types of feature data (such as numerical, categorical, etc.). It is used to handle problems with multiple features and heterogeneous data sources, and is commonly used in multimodal learning and recommendation systems. The core tensor is a low-dimensional tensor representing the main features of the data, while the factor matrix corresponds to the representation of each dimension in the tensor. The core tensor is reconstructed by combining it with the factor matrix. The optimal hash function is a function that maps data to a low-dimensional space through hashing operations. Its goal is to make similar data points as close as possible in the hashed space, thereby improving the efficiency of data storage and retrieval. It is commonly used in applications such as approximate nearest neighbor search. The hash storage unit is a data storage structure used to store data... Data points mapped by hash functions are often used in efficient data storage and querying by reducing storage space and accelerating data retrieval. Position-sensitive coding is a coding method used to represent positional features in data. By considering the relative position of data points in space, the model can capture information dependencies in spatial or temporal dimensions. It is widely used in fields such as natural language processing and image processing. The coarse-grained index layer is a method for indexing large-scale data. By dividing and indexing data in a coarse-grained manner, it improves retrieval efficiency and is often used in databases and information retrieval systems. The fine-grained index layer is a further refinement based on the coarse-grained index, providing a more accurate index to improve query speed. It is often used in multi-level index structures, especially when high-precision queries are required.

[0110] In one alternative implementation,

[0111] The image feature vector, the text content, and the text semantic feature vector are constructed into a heterogeneous feature tensor. Singular value decomposition is performed on the heterogeneous feature tensor to obtain a core tensor and a factor matrix. Correlation analysis is performed on the core tensor to obtain an icon feature fingerprint vector. A local sensitive hash index is constructed based on the icon feature fingerprint vector. Optimal hash function parameters are calculated based on the distribution density of the icon feature fingerprint vector, and the hash storage unit is adjusted based on the distribution density. The icon feature fingerprint vector is encoded using position-sensitive coding to obtain an encoded fingerprint vector. A multi-level index structure is constructed, and the encoded fingerprint vector is added to the coarse-grained index layer. The icon feature fingerprint vector is added to the fine-grained index layer, resulting in an icon feature multidimensional index database comprising:

[0112] The image feature vector, the text content, and the text semantic feature vector are preprocessed. The image feature vector is standardized to eliminate scale differences between dimensions. The text content is segmented and encoded to convert it into a fixed-length feature representation. The text semantic feature vector is normalized to make its distribution range consistent with other features. The preprocessed features are organized into a third-order heterogeneous feature tensor.

[0113] The heterogeneous feature tensor is expanded along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Singular value decomposition is performed on each expanded matrix to extract the main components. The main components are then recombined according to a pre-set rank parameter to obtain the core tensor and factor matrix.

[0114] Calculate the correlation coefficients between different dimensions in the core tensor to construct a correlation coefficient matrix, perform eigenvalue decomposition on the correlation coefficient matrix to extract principal components, calculate the contribution rate of each principal component, and select principal components with contribution rates exceeding a preset threshold to form an icon feature fingerprint vector.

[0115] The feature space is divided into multiple regions, and the number of samples in each region is counted to obtain the region sample density distribution. The number of local sensitive hash functions is determined based on the region sample density distribution. Regions with sample density greater than the first density threshold are divided into fine-grained segments by increasing the number of hash functions, while regions with sample density less than the second density threshold are divided into coarse-grained segments by decreasing the number of hash functions.

[0116] Calculate the variance of each dimension of the icon feature fingerprint vector, sort the dimensions according to the variance values, assign 16-bit binary codes to the top 5 most important dimensions, and assign 8-bit binary codes to the dimensions after the fifth most important dimension. Determine the binary code mapping relationship by dividing the feature value interval, and convert the icon feature fingerprint vector into an encoded fingerprint vector.

[0117] A multi-level index structure is constructed. In the coarse-grained index layer, the encoded fingerprint vector is segmented into 32-bit segments, and a 32-bit hash value is calculated for each segment as the index key to build a hash table. The hash table points to a storage bucket. When the hash value of the encoded fingerprint vector is within the Hamming distance threshold of 8, its index is stored in the same storage bucket. In the fine-grained index layer, the icon feature fingerprint vector is constructed into a tree structure. Each node of the tree represents the numerical range of the feature vector in the current dimension, thus obtaining a multi-dimensional index database of icon features.

[0118] The correlation coefficient matrix is ​​a matrix that represents the linear correlation between multiple variables. The elements in the matrix represent the correlation coefficient between each pair of variables. It is commonly used in statistics and machine learning to analyze the dependency between variables. The locality-sensitive hash function is a method that maps similar data points to the same bucket by hashing. Its goal is to map similar data points to similar hash values ​​to accelerate the search for approximate nearest neighbors. The Hamming distance threshold is a metric used to measure the difference between two binary strings of the same length. It is commonly used in string comparison and data retrieval. The smaller the Hamming distance, the more similar the two strings are.

[0119] Feature extraction and preprocessing are performed on the icon image data. Visual feature vectors are extracted from the images, for example, using convolutional neural networks to extract depth features. Simultaneously, the text content of the icons is extracted, and semantic feature vectors are extracted using natural language processing techniques, such as using the BERT model to obtain word vectors. The image feature vectors are standardized, for example, using Z-score standardization to eliminate scale differences between different dimensions. The text content is segmented and encoded, for example, using Word2Vec to convert the text into a fixed-length feature representation. The text semantic feature vectors are normalized, for example, using MinMaxScaler to make their distribution consistent with other features. The preprocessed image feature vectors, text content features, and text semantic feature vectors are organized into a third-order heterogeneous feature tensor. For example, assuming there are 1000 icon samples, the image feature vector dimension is 512, the text content feature dimension is 200, and the text semantic feature vector dimension is 768, then a 1000x3x(512+200+768) third-order tensor can be constructed.

[0120] Perform Tucker decomposition on the heterogeneous feature tensor. Expand the third-order tensor into matrices along the sample dimension, feature type dimension, and eigenvalue dimension. Perform singular value decomposition on each expanded matrix to extract the principal components. For example, setting the rank parameter to (100, 3, 500) will yield a 100x3x500 core tensor and three factor matrices.

[0121] Correlation analysis decomposition is performed on the core tensor to obtain the icon feature fingerprint vector. Correlation coefficients between different dimensions of the core tensor are calculated, and a correlation coefficient matrix is ​​constructed. Eigenvalue decomposition is performed on the correlation coefficient matrix to extract principal components. The contribution rate of each principal component is calculated; for example, a threshold of 85% is set, and principal components with contribution rates exceeding the threshold are selected to form the icon feature fingerprint vector. Assuming the first 100 principal components are selected, the icon feature fingerprint vector will have a dimension of 100.

[0122] A Locality Sensitive Hash Index (LSI) is constructed based on icon-based feature fingerprint vectors. The feature space is divided into multiple regions; for example, dividing each dimension into 10 intervals results in 10 to the power of 100 regions. The number of samples within each region is counted to obtain the region's sample density distribution. The number of LSIs is determined based on the region's sample density distribution. For example, a first density threshold of 100 and a second density threshold of 10 are set. Regions with a sample density greater than 100 are finely divided by increasing the number of hash functions, for example, using 5 hash functions; regions with a sample density less than 10 are coarsely divided by decreasing the number of hash functions, for example, using 2 hash functions; and other regions use 3 hash functions.

[0123] Position-sensitive coding is used to encode the icon feature fingerprint vector. The variance of each dimension of the icon feature fingerprint vector is calculated, and the dimensions are ranked according to their importance. The top 5 most important dimensions are assigned 16-bit binary codes, and dimensions after the fifth most important dimension are assigned 8-bit binary codes. The binary coding mapping relationship is determined by dividing the feature value range, converting the icon feature fingerprint vector into an encoded fingerprint vector. For example, assuming a dimension has the largest variance, with a value range of 0 to 1, then 0 to 0.0625 can be mapped to 0000000000000000, 0.0625 to 0.125 can be mapped to 000000000000001, and so on.

[0124] A multi-level index structure is constructed. In the coarse-grained index layer, the encoded fingerprint vector is segmented into 32-bit segments, and a 32-bit hash value is calculated for each segment as the index key to build a hash table. The hash table points to storage buckets. When the hash value of the encoded fingerprint vector is within the Hamming distance threshold of 8, its index is stored in the same storage bucket. In the fine-grained index layer, the icon feature fingerprint vector is constructed into a tree structure. Each node of the tree represents the numerical range of the feature vector in the current dimension. For example, the root node represents the first dimension, its child nodes represent different value ranges in that dimension, and so on, with the leaf nodes storing pointers to specific icon samples. This completes the construction of the multi-dimensional index database for icon features.

[0125] In this embodiment, by combining image features, text content, and semantic features, and utilizing Tucker decomposition and correlation analysis to extract icon feature fingerprint vectors, icon information can be expressed more comprehensively, thereby improving retrieval accuracy. The design of locality-sensitive hashing and a multi-level index structure can effectively reduce the retrieval space and accelerate the retrieval speed. Simultaneously, position-sensitive coding can further improve retrieval efficiency, handle large-scale icon data, support the fusion of multiple feature types, and has good scalability.

[0126] In one alternative implementation,

[0127] The heterogeneous feature tensor is expanded along the sample dimension, feature type dimension, and eigenvalue dimension respectively. Singular value decomposition is performed on each expanded matrix to extract principal components. These principal components are then recombined according to a pre-set rank parameter to obtain the core tensor and factor matrix, including:

[0128] The image features, text content, and text semantic features of the icon data are constructed into a third-order heterogeneous feature tensor, which includes sample dimension, feature type dimension, and feature value dimension.

[0129] The third-order heterogeneous feature tensor is expanded along the sample dimension to obtain a first expanded matrix, where the number of rows in the first expanded matrix is ​​the number of samples and the number of columns is the product of the feature type dimension and the feature value dimension. Expanding along the feature type dimension yields a second expanded matrix, where the number of rows in the second expanded matrix is ​​the number of feature types and the number of columns is the product of the sample dimension and the feature value dimension. Expanding along the feature value dimension yields a third expanded matrix, where the number of rows in the third expanded matrix is ​​the sum of the feature value dimensions and the number of columns is the product of the sample dimension and the feature type dimension.

[0130] The first expansion matrix, the second expansion matrix, and the third expansion matrix are each divided into sub-block matrices of equal size, with each sub-block matrix being one-tenth the size of the corresponding expansion matrix;

[0131] For each sub-block matrix, perform iterative singular value decomposition, randomly initialize unit vectors, multiply the unit vectors with the sub-block matrix and normalize them to obtain an update vector, repeat the multiplication and normalization operation until the update vector converges to obtain the first pair of singular vectors, subtract the rank-one matrix constructed from the first pair of singular vectors from the sub-block matrix to obtain the remaining matrix, and repeat the multiplication and normalization operation on the remaining matrix to obtain the next pair of singular vectors;

[0132] In the singular value decomposition process, L2 norm constraints are added to the singular vectors, and iterative optimization is performed using a dynamic learning rate. The initial value of the dynamic learning rate is 0.1, which is adaptively adjusted according to the changes in reconstruction error.

[0133] Based on the pre-set rank parameter, the largest singular value and its corresponding left and right singular vectors are selected from the singular value decomposition results of each sub-block matrix. The singular values ​​and singular vectors after screening of each sub-block matrix are fused by a collaborative filtering algorithm. The left singular vectors are used to form a sample dimension factor matrix, and the right singular vectors are used to form a feature type dimension factor matrix and an eigenvalue dimension factor matrix.

[0134] An initial small-scale core tensor is constructed using a progressive strategy. The reconstruction error is calculated. When the reconstruction error is greater than a preset threshold but has not reached the maximum size limit, the size of the core tensor is gradually increased. This process is repeated until the maximum size limit is reached, thus obtaining the core tensor.

[0135] The sub-block matrix refers to a partial sub-matrix within a matrix, typically obtained by dividing the matrix into blocks. Each sub-block matrix contains a portion of the data in the matrix and is commonly used in matrix decomposition and optimization problems. The rank-one matrix is ​​a matrix with a rank of 1, meaning it can be represented as the outer product of two vectors. Rank-one matrices are frequently used in tasks such as data dimensionality reduction and feature representation. The collaborative filtering algorithm is a commonly used algorithm in recommendation systems. By analyzing the similarity between users or items, it predicts content that users may be interested in based on historical behavior or preferences, and is widely applied in e-commerce, movie recommendations, and other fields.

[0136] Collect and preprocess icon data. Retrieve icon data from an icon database, including icon images, text content, and semantic features of the text. Standardize icon images, such as by uniformizing size and converting color spaces. Preprocess the text content by word segmentation and stop word removal. Extract semantic features of the text content using a pre-trained word vector model.

[0137] Construct a third-order heterogeneous feature tensor. Organize the preprocessed icon data into a third-order tensor. The three dimensions of the tensor represent the sample, feature type, and feature value, respectively. For example, the image feature vector, text content feature vector, and semantic feature vector of a certain icon are filled into the corresponding positions in the tensor. Assuming there are 1000 icon samples, with an image feature dimension of 512, a text content feature dimension of 256, and a semantic feature dimension of 768, a tensor of 1000x3x(512+256+768) can be constructed.

[0138] The tensor is expanded and subjected to singular value decomposition (SVD). The constructed third-order tensor is expanded into three matrices along the sample dimension, feature type dimension, and eigenvalue dimension, respectively. For example, expanding along the sample dimension yields a matrix with 1000 rows and 3*(512+256+768) columns. Each expanded matrix is ​​divided into equal-sized sub-matrices, each sub-matrix being one-tenth the size of the corresponding expanded matrix. Iterative SVD is performed on each sub-matrix to extract principal components. During the iterative SVD process, a dynamic learning rate is used for optimization, with an initial learning rate of 0.1, adaptively adjusted based on the reconstruction error. Simultaneously, L2 norm constraints are added to the singular vectors to prevent overfitting.

[0139] In the singular value decomposition (SVD) process, important singular values ​​and singular vectors are selected. Based on a pre-defined rank parameter, the largest number of singular values ​​and their corresponding left and right singular vectors are selected from the SVD results of each sub-block matrix. For example, if the rank parameter is set to 10, the top 10 largest singular values ​​and their corresponding singular vectors are selected. A collaborative filtering algorithm is then used to fuse the selected singular values ​​and vectors from each sub-block matrix. The left singular vectors are used to form a sample dimension factor matrix, and the right singular vectors are used to form a feature type dimension factor matrix and an eigenvalue dimension factor matrix.

[0140] A progressive strategy is used to construct the core tensor. An initial small-scale core tensor is built, and the reconstruction error is calculated. If the reconstruction error is greater than a preset threshold, and the core tensor has not reached its maximum size limit, the size of the core tensor is gradually increased until the preset maximum size limit is reached or the reconstruction error is less than the threshold.

[0141] For example, suppose we perform singular value decomposition on a 100x3x1536 sub-block matrix, with the rank parameter set to 10. After singular value decomposition and filtering, we obtain 10 singular values ​​and corresponding 10 left singular vectors (100-dimensional) and 10 right singular vectors (4608-dimensional). We then use these 10 left singular vectors to form a 100x10 sample factor matrix. We perform similar operations and fusion on the singular values ​​and singular vectors of all sub-block matrices to finally obtain the complete core tensor and factor matrix.

[0142] In this embodiment, by fusing multimodal features, icon information can be described more comprehensively, thereby improving the accuracy and recall of icon retrieval. Multimodal feature fusion can better capture the fine-grained features of icons, thereby improving the accuracy of icon classification. Tensor decomposition technology can effectively reduce data dimensionality and computational complexity, thereby improving the efficiency of icon analysis.

[0143] S3. Receive a retrieval request and obtain a target icon image. Perform feature extraction and encoding on the target icon image to generate a target feature fingerprint vector. Perform a fast retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. Calculate the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer. Determine the feature similarity based on the feature distance and select icon feature fingerprint vectors with feature similarity greater than a preset similarity threshold to generate a similarity candidate set. Add the set to the deep metric learning model. Calculate the relative distance relationship using a triplet loss function. Combine this with a contrastive loss function to constrain the feature space distribution, obtain the metric feature representation, and calculate visual similarity, text similarity, and semantic similarity. Determine the weight parameters by determining the variance and discriminant coefficient of each similarity feature. Determine the comprehensive similarity score by weighted summation. Sort the results in descending order and select the top 3 retrieval results as the output similarity retrieval results.

[0144] The fast retrieval is a technique for quickly finding and matching target items in a database. The deep metric learning model is a model that uses deep neural networks for metric learning. Its purpose is to learn a mapping function that maps input data to an embedding space, making the distance between similar data points smaller and the distance between dissimilar data points larger. The triplet loss function is a loss function commonly used in deep metric learning to optimize the model's embedding space.

[0145] In one alternative implementation,

[0146] The system receives a retrieval request and acquires a target icon image. It then performs feature extraction and encoding on the target icon image to generate a target feature fingerprint vector. A fast retrieval is performed in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. The feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer is calculated. Based on the feature distance, feature similarity is determined, and icon feature fingerprint vectors with feature similarity greater than a pre-set similarity threshold are selected to generate a similarity candidate set. This set is added to the deep metric learning model. The relative distance relationship is calculated using a triplet loss function, and the feature space distribution is constrained by a contrastive loss function to obtain a metric feature representation. Visual similarity, text similarity, and semantic similarity are calculated. Weight parameters are determined by determining the variance and discriminant coefficient of each similarity feature. A comprehensive similarity score is determined by weighted summation. The results are sorted in descending order, and the top three retrieval results are selected as the output similarity retrieval results, including:

[0147] The system receives a retrieval request and obtains a target icon image. It performs feature extraction and normalization preprocessing on the target icon image, extracts image feature vectors through a multi-layer convolutional network, extracts text vectors and semantic feature vectors through a pre-trained language model, and concatenates the image feature vectors, the text vectors, and the semantic feature vectors to form a target feature fingerprint vector.

[0148] The target feature fingerprint vector is divided into multiple feature segments according to a preset length. The feature values ​​in each feature segment are binarized to obtain binary codes. The binarization process includes recording feature values ​​greater than the average value of the feature segment as the first identifier and feature values ​​less than the average value of the feature segment as the second identifier.

[0149] Locality-sensitive hashing is used in the coarse-grained index layer to map the binary code to a hash value. The encoded fingerprint vector that is less than a preset threshold away from the hash value is searched in the hash table and added to the initial candidate set. If the number of samples in the storage area in the hash table exceeds a preset threshold, the number of hash functions in the current storage area is increased to further subdivide the area.

[0150] In the feature tree of the fine-grained index layer, the encoded fingerprint vectors in the initial candidate set are traversed. The internal nodes of the feature tree represent regions in the feature space, and the leaf nodes store icon feature fingerprint vectors. The feature distance between the target feature fingerprint vector and the node center point is calculated. The child node with the closest feature distance is selected and traversal continues until the leaf node. The icon feature fingerprint vectors corresponding to the leaf nodes with feature distances less than a preset similarity threshold are used to generate a similarity candidate set.

[0151] The similarity candidate set is added to the deep metric learning model, wherein the deep metric learning model includes a multi-layer fully connected encoder and a metric network based on an attention mechanism. The sample closest to the target feature fingerprint vector is selected from the similarity candidate set as a positive sample, and the sample farthest away is selected as a negative sample. The target feature fingerprint vector is used as an anchor sample, and the anchor sample, the positive sample, and the negative sample are combined to construct a triple.

[0152] The relative distance relationship of the triplet is calculated using the triplet loss function. Combined with the contrastive loss function to constrain the feature space distribution, the distance between the target feature fingerprint vector and the positive sample is less than the distance with the negative sample. The feature representation similarity between the icon pairs is calculated separately. The similarity is compared with the preset target similarity threshold, and the icon pairs are selected to form a feature representation set.

[0153] Based on the deep metric learning model, a metric feature representation is obtained. Visual similarity, text similarity, and semantic similarity are calculated between the icon feature fingerprint vector in the feature representation set and the target feature fingerprint vector. Weight parameters are determined based on the variance and discriminant coefficient of each similarity feature in the training set. A comprehensive similarity score is determined by weighted summation. The comprehensive similarity scores are sorted in descending order, and the top 3 search results are selected as similar search results for output.

[0154] The binarization process is the conversion of input data (such as images, text, etc.) into binary format. It is typically used for feature encoding, image compression, or reducing data storage requirements. Locality-sensitive hashing is an algorithm for fast retrieval of high-dimensional data. Its core idea is to construct a hash function so that similar data points are mapped to the same hash bucket. The leaf node is the terminal node in a tree structure (such as a decision tree, random forest, etc.), representing the final classification or regression result of the tree. The anchor sample in metric learning refers to one of the samples used to define the similarity metric, usually a reference point for comparison with other samples.

[0155] Receive search requests and retrieve target icon images. For example, a user uploads an icon image containing both graphics and text as the search target.

[0156] Feature extraction and normalization preprocessing are performed on the target icon image to resize it to a uniform size, such as 224x224 pixels. A multi-layer convolutional network (e.g., ResNet50) is used to extract image feature vectors. Assume the extracted image feature vector has a dimension of 2048. Simultaneously, a pre-trained language model (e.g., BERT) is used to extract text vectors and semantic feature vectors corresponding to the icon text. Assume the text vector has a dimension of 768 and the semantic feature vector has a dimension of 512. Finally, the image feature vector, text vector, and semantic feature vector are concatenated to form the target feature fingerprint vector, with a dimension of 2048 + 768 + 512 = 3328.

[0157] The target feature fingerprint vector is segmented into multiple feature segments according to a preset length. For example, a 3328-dimensional feature vector is segmented into 6 feature segments of length 512, and the remaining 256-dimensional feature segments are processed separately. The feature values ​​in each feature segment are binarized to obtain binary codes. Specifically, the mean of each feature segment is calculated, and feature values ​​greater than the mean are recorded as "1" and feature values ​​less than the mean are recorded as "0". For example, if the mean of a feature segment is 0.5, and the feature values ​​of that feature segment are [0.6, 0.3, 0.7, 0.2], then the corresponding binary code is [1, 0, 1, 0].

[0158] Locality-Sensitive Hashing (LSH) is used in the coarse-grained indexing layer to map binary code to hash values. For example, the SimHash algorithm is used to map the binary code of each feature segment to a 64-bit hash value. Encoded fingerprint vectors with a distance less than a preset threshold from the target hash value are searched in the hash table. For example, with a threshold of 3, encoded fingerprint vectors with a Hamming distance less than or equal to 3 are searched and added to the initial candidate set. If the number of samples in a stored region in the hash table exceeds a preset threshold (e.g., 1000), the number of hash functions in the current stored region is increased (e.g., from 3 to 4) to further subdivide the region and improve retrieval efficiency.

[0159] The encoded fingerprint vectors in the initial candidate set are traversed in the feature tree of the fine-grained index layer. Internal nodes of the feature tree represent regions in the feature space, and leaf nodes store icon feature fingerprint vectors. The feature distance (e.g., Euclidean distance) between the target feature fingerprint vector and the node center point is calculated. The child node with the closest feature distance is selected, and traversal continues until a leaf node is reached. Icon feature fingerprint vectors corresponding to leaf nodes with feature distances less than a pre-set similarity threshold (e.g., 0.8) are used to generate a similarity candidate set.

[0160] A similarity candidate set is added to the deep metric learning model. The deep metric learning model comprises a multi-layer fully connected encoder and an attention-based metric network. From the similarity candidate set, the sample closest to the target feature fingerprint vector is selected as the positive sample, and the sample furthest away is selected as the negative sample. The target feature fingerprint vector is used as an anchor sample, and triples are constructed by combining the anchor sample, positive sample, and negative sample. For example, if the target icon image is closest to icon A in the candidate set and furthest away from icon B, then a triple (target icon, icon A, icon B) is constructed.

[0161] The relative distance relationship of triplet pairs is calculated using a triplet loss function. Combined with a contrastive loss function, the feature space distribution is constrained to ensure that the distance between the target feature fingerprint vector and positive samples is less than the distance with negative samples. The feature representation similarity between icon pairs is then calculated. This similarity is compared to a preset target similarity threshold, and icon pairs are selected to form a feature representation set.

[0162] Based on a deep metric learning model, metric feature representations are obtained. Visual similarity, textual similarity, and semantic similarity are calculated between the icon feature fingerprint vector and the target feature fingerprint vector in the feature representation set. For example, cosine similarity is used to calculate visual similarity, textual similarity, and semantic similarity. Weight parameters are determined based on the variance and discriminant coefficient of each similarity feature in the training set. For example, visual similarity has a larger variance and a higher discriminant coefficient, so it is assigned a larger weight. A weighted summation is used to determine the comprehensive similarity score. The comprehensive similarity scores are sorted in descending order, and the top 3 search results are selected as the output similarity search results.

[0163] In this embodiment, a two-level retrieval architecture of coarse-grained index and fine-grained index is used to quickly filter the candidate set, narrow the retrieval range, and improve retrieval efficiency. A deep metric learning model is used to learn more discriminative feature representations. Combined with visual, text, and semantic multi-dimensional similarity measurements, the retrieval accuracy is improved. The target icon image is preprocessed by normalization and combined with binarization and locality-sensitive hashing algorithm to enhance the robustness of the retrieval system and make it more resistant to interference factors such as image noise and deformation.

[0164] In one alternative implementation,

[0165] The relative distance relationship of the triplet is calculated using a triplet loss function, and the feature space distribution is constrained by a contrastive loss function to ensure that the distance between the target feature fingerprint vector and positive samples is less than the distance with negative samples. The feature representation similarity between icon pairs is calculated separately, and the similarity is compared with a preset target similarity threshold. The selection of icon pairs to form a feature representation set includes:

[0166] The system receives a triplet consisting of a target feature fingerprint vector, a positive sample feature fingerprint vector, and a negative sample feature fingerprint vector. The triplet is then input into a deep metric learning model. The deep metric learning model includes a multi-layer fully connected encoder network and an attention-based metric network. The encoder network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence. The output dimensions of the first fully connected layer, the second fully connected layer, and the third fully connected layer decrease sequentially. Each fully connected layer is followed by a non-linear activation layer and a batch normalization layer.

[0167] The positive sample distance is obtained by calculating the Euclidean distance between the target feature fingerprint vector and the positive sample feature fingerprint vector through the metric network, and the negative sample distance is obtained by calculating the Euclidean distance between the target feature fingerprint vector and the negative sample feature fingerprint vector. The difference between the positive sample distance and the negative sample distance is compared with a preset interval parameter.

[0168] The cosine similarity of the feature representations of the icon pairs in the triplet is calculated. A first target similarity is set for icon pairs of the same category, and a second target similarity is set for icon pairs of different categories. The first target similarity is greater than the second target similarity. Feature constraints are performed based on the difference between the cosine similarity of the feature representations and the corresponding target similarity.

[0169] Calculate the loss value of the triplet, select a preset number of triplets with the largest loss value as hard samples, and use the hard samples for parameter updates of the encoder network and the metric network;

[0170] The feature fingerprint vectors of icon pairs in the similarity candidate set are mapped by the encoder network to obtain mapped features. The cosine similarity between the mapped features is calculated by the metric network. Icon pairs with a cosine similarity greater than a preset similarity threshold are saved to the feature representation set.

[0171] Positive samples refer to samples similar to anchor samples. In metric learning, positive samples should have feature representations similar to anchor samples. Hard samples refer to samples that are difficult to classify or separate for model training. They are usually used to enhance the model's learning ability. In deep learning, hard samples are used for reinforcement learning through specific sampling strategies or weighting methods to improve the model's robustness and generalization ability.

[0172] Prepare an icon image dataset for model training. This dataset contains a large number of icon images, each labeled with its category. For example, we collect 10,000 icon images, covering 1,000 different icon categories, with 10 images in each category. Randomly select a target icon image, a positive sample icon image of the same category, and a negative sample icon image of a different category from the dataset to form a triple for model training. For example, randomly select an "apple" icon image as the target image, another "apple" icon image as a positive sample, and a "banana" icon image as a negative sample to form a triple.

[0173] A deep metric learning model is constructed, consisting of an encoder network and a metric network. The encoder network converts the input icon image into feature vectors. Using ResNet50 as an example, it contains multiple convolutional and pooling layers to extract image features. The metric network calculates the distance or similarity between two feature vectors. An attention-based metric network is employed here to better capture subtle differences between feature vectors. Assuming the encoder network outputs a feature vector with a dimension of 512, the attention mechanism calculates the weights of different dimensions of features between the two feature vectors to obtain a weighted feature representation, and then calculates their Euclidean distance.

[0174] The model is trained using triplet loss and contrastive loss functions. The prepared triplets are input into the deep metric learning model, and the encoder network extracts feature vectors from the target image, positive samples, and negative samples, respectively. The metric network calculates the distance between the target feature vector and the positive sample feature vector (positive sample distance), and the distance between the target feature vector and the negative sample feature vector (negative sample distance). The goal of the triplet loss is to make the positive sample distance less than the negative sample distance, with at least a preset margin between them, such as 0.2. Simultaneously, the cosine similarity of the icon pairs in the triplet is calculated and compared with a preset target similarity. For example, for icon pairs of the same category, the target similarity is set to 0.9; for icon pairs of different categories, the target similarity is set to 0.1. Feature constraints by minimizing similarity differences further improve model performance. During training, the triplets with the largest loss values ​​(e.g., 128) are selected as hard samples for updating model parameters, accelerating model convergence.

[0175] Icon image retrieval is performed. The icon image to be retrieved is input into a trained deep metric learning model to obtain its feature vector. This feature vector is compared with the feature vectors of all icon images in the database, and the cosine similarity between them is calculated. Icon images with a similarity greater than a preset threshold, such as 0.8, are filtered out to form the final retrieval results.

[0176] In this embodiment, through deep metric learning, the model can learn more discriminative icon image feature representations, thereby improving retrieval accuracy and more effectively identifying similar icons. By combining triplet loss and contrast loss for training, the model becomes more robust to subtle changes in images, such as lighting, angle, and deformation. Even if icon images have certain differences, it can accurately determine their similarity. By pre-calculating the feature vectors of all icon images in the database, the retrieval process only needs to calculate the similarity between the image to be retrieved and the images in the database, which greatly improves retrieval efficiency and can quickly find similar icons from massive amounts of data.

[0177] Figure 2 This is a schematic diagram of the structure of the multi-dimensional data intelligent retrieval and matching system based on image and text features according to an embodiment of the present invention, as shown below. Figure 2 As shown, the system includes:

[0178] The first unit is used to acquire the image of the icon to be retrieved and add it to the convolutional neural network for feature extraction. It extracts a first feature map by combining multiple convolutional layers and pooling layers. Based on the first feature map, the image of the icon to be retrieved is input into the image segmentation attention mechanism network. The spatial dependence of the image region is determined by the spatial attention mechanism. The attention weight map is generated by the channel attention module and the graphic region and text region are output. The text region is added to the transducer optical character recognition model. It is converted into a feature sequence by the encoder and decoded character by character by character by the multi-head attention mechanism in the decoder to obtain the text content. The text content is feature extracted by the bidirectional semantic understanding model to obtain the text semantic feature vector. The graphic region is added to the visual transducer network. The feature is transferred across layers based on the dense connection structure. The feature is fused by the residual connection and the feature pyramid structure to obtain the image feature vector.

[0179] The second unit is used to construct a heterogeneous feature tensor from the image feature vector, the text content, and the text semantic feature vector; perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and a factor matrix; perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector; construct a local sensitive hash index based on the icon feature fingerprint vector; calculate the optimal hash function parameters based on the distribution density of the icon feature fingerprint vector and adjust the hash storage unit based on the distribution density; encode the icon feature fingerprint vector using position-sensitive coding to obtain an encoded fingerprint vector; construct a multi-level index structure and add the encoded fingerprint vector to the coarse-grained index layer; add the icon feature fingerprint vector to the fine-grained index layer to obtain an icon feature multidimensional index database.

[0180] The third unit is used to receive retrieval requests and obtain target icon images, perform feature extraction and encoding on the target icon images, generate target feature fingerprint vectors, and perform fast retrieval in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors. It calculates the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determines the feature similarity based on the feature distance, and selects icon feature fingerprint vectors with feature similarity greater than a preset similarity threshold to generate a similarity candidate set, which is added to the deep metric learning model. The relative distance relationship is calculated through the triplet loss function, and the feature space distribution is constrained by the contrastive loss function to obtain the metric feature representation and calculate visual similarity, text similarity, and semantic similarity. The weight parameters are determined by determining the variance and discriminant coefficient of each similarity feature, and the comprehensive similarity score is determined by weighted summation. The results are sorted in descending order, and the top 3 retrieval results are selected as the similarity retrieval results for output.

[0181] A third aspect of the present invention,

[0182] An electronic device is provided, comprising:

[0183] processor;

[0184] Memory used to store processor-executable instructions;

[0185] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0186] Fourth aspect of the embodiments of the present invention,

[0187] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0188] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent retrieval and matching of multi-dimensional data of text and graphics features, characterized in that, The method comprises: obtaining an icon image to be searched and adding it to a convolutional neural network for feature extraction, combining multiple convolutional layers and pooling layers to extract a first feature map, inputting the icon image to be searched into an image segmentation attention mechanism network based on the first feature map, determining the spatial dependence of the image region through a spatial attention mechanism, combining a channel attention module to generate an attention weight map and output a graphic region and a text region, adding the text region to a transformer optical character recognition model, converting it into a feature sequence through an encoder and decoding it character by character through a multi-head attention mechanism in the decoder to obtain text content, extracting features of the text content through a bidirectional semantic understanding model to obtain a text semantic feature vector, adding the graphic region to a visual transformer network, performing feature cross-layer transmission based on a dense connection structure, combining a residual connection and a feature pyramid structure to perform feature fusion to obtain an image feature vector; constructing the image feature vector, the text content and the text semantic feature vector into a heterogeneous feature tensor, performing singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and a factor matrix, decomposing the core tensor through correlation analysis to obtain an icon feature fingerprint vector, constructing a local sensitive hash index based on the icon feature fingerprint vector, calculating the optimal hash function parameter based on the distribution density of the icon feature fingerprint vector and adjusting the hash storage unit based on the distribution density, encoding the icon feature fingerprint vector through location sensitive coding to obtain an encoded fingerprint vector, constructing a multi-level index structure and adding the encoded fingerprint vector to a coarse-grained index layer, adding the icon feature fingerprint vector to a fine-grained index layer to obtain an icon feature multi-dimensional index database; receiving a search request and obtaining a target icon image, performing feature extraction and encoding on the target icon image, generating a target feature fingerprint vector and performing fast search in the coarse-grained index layer to obtain an initial candidate set containing multiple encoded fingerprint vectors, calculating the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determining the feature similarity based on the feature distance and selecting the icon feature fingerprint vector with a feature similarity greater than a pre-set similarity threshold to generate a similarity candidate set, adding it to a deep metric learning model, calculating the relative distance relationship through a triplet loss function, combining a contrast loss function to constrain the feature space distribution, obtaining a metric feature representation and calculating the visual similarity, the text similarity and the semantic similarity, determining the weight parameter by determining the variance and discrimination coefficient of each similarity feature, determining the comprehensive similarity score through weighted summation, descending order sorting and selecting the top 3 search results as the similar search results for output.

2. The method of claim 1, wherein, The icon image to be retrieved is obtained and added to a convolutional neural network for feature extraction, a plurality of convolutional layers and pooling layers are combined to extract a first feature map, the icon image to be retrieved is input into an image segmentation attention mechanism network based on the first feature map, the spatial dependence relationship of the image region is determined through the spatial attention mechanism, the attention weight map is generated by combining the channel attention module, and the graphic region and the text region are output, the text region is added to the transformer optical character recognition model, the feature sequence is converted through the encoder, and the character-by-character decoding is performed through the multi-head attention mechanism in the decoder to obtain the text content, the text content is feature extracted through the bidirectional semantic understanding model, and a text semantic feature vector is obtained, the graphic region is added to the visual transformer network, feature cross-layer transmission is performed based on the dense connection structure, and the image feature vector is obtained through feature fusion combining the residual connection and the feature pyramid structure, including: The icon image to be retrieved is obtained and added to a convolutional neural network, the convolutional neural network is composed of a plurality of convolutional layers and pooling layers connected in sequence, the size of the convolution kernel in each convolutional layer decreases step by step from front to back, and the number of convolution kernels increases step by step from front to back, the first feature map is obtained through feature extraction of the icon image to be retrieved by the convolutional layer and the pooling layer; The icon image to be retrieved is input into an image segmentation attention mechanism network based on the first feature map, the first feature map and the second feature map are obtained by respectively performing average pooling and maximum pooling on the first feature map in the channel dimension through the spatial attention mechanism of the image segmentation attention mechanism network, the spatial dependence relationship of the image region is determined after the first feature map and the second feature map are spliced in the channel dimension and then subjected to convolution processing, the first feature vector and the second feature vector are obtained by respectively performing global average pooling and maximum pooling on the first feature map in the spatial dimension through the channel attention module, the attention weight map is generated by inputting the first feature vector and the second feature vector into a fully connected network, and the first feature map is weighted and processed based on the attention weight map to output the graphic region and the text region; The text region is added to the transformer optical character recognition model, the encoder of the transformer optical character recognition model divides the text region into image blocks and then performs linear projection to convert the image blocks into feature sequences, the encoding features are obtained through multi-head attention processing of the feature sequences, the encoding features are input into the decoder, and the character-by-character decoding is performed through the multi-head attention mechanism of the decoder to obtain the text content; The text content is input into the bidirectional semantic understanding model, the bidirectional semantic understanding model adopts a multi-layer stacked transformer encoder, performs bidirectional feature extraction on the text content through the multi-head attention mechanism, combines the residual connection to transmit inter-layer information, and obtains a text semantic feature vector. The graphic region is added to a visual transformer network, after image block division and linear transformation of the graphic region, feature cross-layer transmission is performed based on a dense connection structure, a feature pyramid is constructed at different scales to extract multi-scale features, and image feature vectors are obtained by combining residual connection for feature fusion of the multi-scale features.

3. The method of claim 2, wherein, The graphic region is added to a visual transformer network, after image block division and linear transformation of the graphic region, feature cross-layer transmission is performed based on a dense connection structure, a feature pyramid is constructed at different scales to extract multi-scale features, and image feature vectors are obtained by combining residual connection for feature fusion of the multi-scale features. The graphic region is added to a pre-set transformer network, and size normalization processing is performed through bilinear interpolation to obtain a normalized graphic region. The normalized graphic region is divided into a regular grid-shaped image block sequence, a pixel matrix of each image block is converted into an image block feature vector through a linear projection layer, a learnable position encoding vector is assigned to each image block position, the position encoding vector and the image block feature vector are added to obtain a feature sequence fused with position information, and a spatial relationship between different image blocks is established. A multi-layer transformer encoder network is constructed, the multi-layer transformer encoder network adopts a dense connection structure, feature maps output by each layer are spliced in a channel dimension to be used as input of a next layer, a convolutional layer is used to adjust a feature channel number, and feature transmission is performed between different levels to obtain a transmission feature map. A feature pyramid structure is constructed based on the transmission feature map, large-scale feature maps and small-scale feature maps are obtained through different scale pooling operations on input features, the large-scale feature maps retain image detail information, and the small-scale feature maps obtain large-range receptive field information, the small-scale feature maps are up-sampled through a deconvolutional operation, and the spatial resolutions of the large-scale feature maps and the up-sampled small-scale feature maps are unified. Different scale feature maps with unified resolutions are spliced in a channel dimension to obtain multi-scale features, the multi-scale features are adaptively fused through a convolutional layer, a residual connection mechanism is used to directly stack input features of a current layer to output features to obtain fused output features. A global pooling operation is performed on the fused output features to compress a spatial dimension into a low-dimensional feature vector, the low-dimensional feature vector is processed through a multi-layer nonlinear transformation network, a nonlinear expression capability is introduced through an activation function after each layer transformation, and image feature vectors containing local details and global semantics are output.

4. The method of claim 1, wherein, The image feature vector, the text content and the text semantic feature vector are constructed into a heterogeneous feature tensor, singular value decomposition is performed on the heterogeneous feature tensor to obtain a core tensor and a factor matrix, correlation analysis decomposition is performed on the core tensor to obtain an icon feature fingerprint vector, a local sensitive hash index is constructed based on the icon feature fingerprint vector, optimal hash function parameters are calculated combined with the distribution density of the icon feature fingerprint vector and the hash storage unit is adjusted based on the distribution density, the icon feature fingerprint vector is encoded combined with a location sensitive code to obtain an encoded fingerprint vector, a multi-level index structure is constructed and the encoded fingerprint vector is added to a coarse-grained index layer and the icon feature fingerprint vector is added to a fine-grained index layer to obtain an icon feature multi-dimensional index database, including: The image feature vector, the text content and the text semantic feature vector are preprocessed, standardization processing is performed on the image feature vector to eliminate dimensional scale differences, word segmentation and encoding conversion are performed on the text content to represent fixed-length features, normalization processing is performed on the text semantic feature vector to make the distribution range consistent with other features, and the preprocessed features are organized into a three-order heterogeneous feature tensor; The heterogeneous feature tensor is respectively matrix unfolded along a sample dimension, a feature type dimension and a feature value dimension, singular value decomposition is performed on each unfolded matrix to extract main components, and the main components are reorganized according to a pre-set rank parameter to obtain a core tensor and a factor matrix; A correlation coefficient matrix is constructed by calculating correlation coefficients between different dimensions in the core tensor, eigenvalue decomposition is performed on the correlation coefficient matrix to extract main components, the contribution rate of each main component is calculated, and a main component whose contribution rate exceeds a pre-set threshold is selected to form an icon feature fingerprint vector; A feature space is divided into multiple regions, the number of samples in each region is counted to obtain a regional sample density distribution, the number of local sensitive hash functions is determined according to the regional sample density distribution, a region with a sample density greater than a first density threshold is divided in a fine-grained manner by increasing the number of hash functions, and a region with a sample density less than a second density threshold is divided in a coarse-grained manner by reducing the number of hash functions; The variances of the dimensions of the icon feature fingerprint vector are calculated, the importance of the dimensions is sorted according to the variance values, the top 5 dimensions in importance are assigned 16-bit binary codes, the dimensions after the fifth are assigned 8-bit binary codes, a binary code mapping relationship is determined through feature value interval division, and the icon feature fingerprint vector is converted into an encoded fingerprint vector; A multi-level index structure is constructed, the encoded fingerprint vector is segmented according to a 32-bit length in the coarse-grained index layer, a 32-bit hash value is calculated for each segment as an index key to establish a hash table, the hash table points to a storage bucket, when the hash values of the encoded fingerprint vectors are within a Hamming distance threshold of 8, they are indexed and stored into the same storage bucket, and the icon feature fingerprint vector is constructed into a tree structure in the fine-grained index layer, each node of the tree represents a numerical interval of the feature vector in the current dimension, and an icon feature multi-dimensional index database is obtained.

5. The method of claim 4, wherein, The heterogeneous feature tensor is respectively matrix unfolded along the sample dimension, the feature type dimension and the feature value dimension, and a principal component is extracted by performing singular value decomposition on each unfolded matrix, and the principal component is reorganized according to a pre-set rank parameter to obtain a core tensor and a factor matrix, including: An image feature, a text content and a text semantic feature of icon data are constructed into a three-order heterogeneous feature tensor, and the three-order heterogeneous feature tensor includes a sample dimension, a feature type dimension and a feature value dimension; The three-order heterogeneous feature tensor is unfolded along the sample dimension to obtain a first unfolded matrix, the number of rows of the first unfolded matrix is the number of samples, and the number of columns is the product of the feature type dimension and the feature value dimension, the three-order heterogeneous feature tensor is unfolded along the feature type dimension to obtain a second unfolded matrix, the number of rows of the second unfolded matrix is the number of feature types, and the number of columns is the product of the sample dimension and the feature value dimension, and the three-order heterogeneous feature tensor is unfolded along the feature value dimension to obtain a third unfolded matrix, the number of rows of the third unfolded matrix is the sum of the feature value dimensions, and the number of columns is the product of the sample dimension and the feature type dimension; The first unfolded matrix, the second unfolded matrix and the third unfolded matrix are respectively divided into sub-block matrices of equal size, and the size of each sub-block matrix is one tenth of the corresponding unfolded matrix; An iterative singular value decomposition is performed on each sub-block matrix, a unit vector is randomly initialized, the unit vector is multiplied by the sub-block matrix and normalized to obtain an updated vector, the multiplication and normalization operation is repeatedly performed until the updated vector converges to obtain a first pair of singular vectors, a rank-one matrix constructed by the first pair of singular vectors is subtracted from the sub-block matrix to obtain a residual matrix, and the multiplication and normalization operation is repeatedly performed on the residual matrix to obtain a next pair of singular vectors; An L2 norm constraint is added to the singular vector during the singular value decomposition process, and an iterative optimization is performed by using a dynamic learning rate, an initial value of the dynamic learning rate is 0.1, and the dynamic learning rate is adaptively adjusted according to a reconstruction error change; The largest singular value and the corresponding left singular vector and right singular vector are selected from the singular value decomposition results of each sub-block matrix according to a pre-set rank parameter, the singular values and the singular vectors filtered by the sub-block matrices are fused by using a collaborative filtering algorithm, the left singular vector is composed into a sample dimension factor matrix, and the right singular vector is composed into a feature type dimension factor matrix and a feature value dimension factor matrix; An initial small-scale core tensor is constructed by using a progressive strategy, a reconstruction error is calculated, the core tensor scale is gradually increased when the reconstruction error is greater than a pre-set threshold value and the maximum scale limit is not reached, the core tensor scale is repeatedly increased until the maximum scale limit is reached, and the core tensor is obtained.

6. The method of claim 1, wherein, receiving a search request and obtaining a target icon image, performing feature extraction and encoding on the target icon image, generating a target feature fingerprint vector and performing fast search in the coarse-grained index layer to obtain an initial candidate set containing a plurality of encoded fingerprint vectors, calculating the feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determining the feature similarity based on the feature distance and selecting the icon feature fingerprint vector with a feature similarity greater than a pre-set similarity threshold to generate a similarity candidate set, adding to a deep metric learning model, calculating the relative distance relationship through a triplet loss function, combining a contrast loss function to constrain the feature space distribution, obtaining a metric feature representation and calculating a visual similarity, a textual similarity and a semantic similarity, determining a weight parameter by determining the variance and discrimination coefficient of each similarity feature, determining a comprehensive similarity score by weighted summation, and sorting in descending order and selecting the top 3 search results as the similar search results output including: receiving a search request and obtaining a target icon image, performing feature extraction and normalization preprocessing on the target icon image, extracting an image feature vector through a multi-layer convolutional network, extracting a text vector and a semantic feature vector through a pre-trained language model, and splicing the image feature vector, the text vector and the semantic feature vector to form a target feature fingerprint vector; segmenting the target feature fingerprint vector into a plurality of feature segments according to a pre-set length, and performing binary processing on the feature values in each feature segment to obtain a binary code, wherein the binary processing includes recording a feature value greater than the average value of the feature segment as a first identifier and recording a feature value less than the average value of the feature segment as a second identifier; mapping the binary code to a hash value using local sensitive hashing in the coarse-grained index layer, searching for an encoded fingerprint vector with a distance less than a pre-set threshold from the hash value in a hash table, adding the encoded fingerprint vector to an initial candidate set, and if the number of samples in the storage area in the hash table exceeds a pre-set number threshold, increasing the number of hash functions in the current storage area for area subdivision; traversing the encoded fingerprint vectors in the initial candidate set in the feature tree of the fine-grained index layer, wherein the internal nodes of the feature tree represent areas in the feature space and the leaf nodes store icon feature fingerprint vectors, calculating the feature distance between the target feature fingerprint vector and the node center point, selecting the child node with the closest feature distance to continue traversing until the leaf node, and generating a similarity candidate set from the icon feature fingerprint vectors corresponding to the leaf nodes with a feature distance less than a pre-set similarity threshold; adding the similarity candidate set to a deep metric learning model, wherein the deep metric learning model includes a multi-layer fully connected encoder and a metric network based on an attention mechanism, selecting a sample closest to the target feature fingerprint vector as a positive sample and a sample farthest from the target feature fingerprint vector as a negative sample from the similarity candidate set, combining the anchor sample, the positive sample and the negative sample to construct a triplet, and The triplet is calculated by a triplet loss function to calculate a relative distance relationship, and a contrastive loss function is combined to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance between the target feature fingerprint vector and the negative sample, and the feature representation similarity between the icon pairs is calculated respectively, the similarity is compared with a preset target similarity threshold, and the icon pairs are screened to form a feature representation set; Based on the deep metric learning model, the visual similarity, the textual similarity and the semantic similarity are calculated between the icon feature fingerprint vectors in the feature representation set and the target feature fingerprint vector, the weight parameters are determined based on the variance and the discriminant coefficient of each similarity feature in the training set, the comprehensive similarity score is determined by weighted summation, the comprehensive similarity score is sorted in descending order, and the top 3 search results are selected as the similar search results.

7. The method of claim 6, wherein, The triplet is calculated by a triplet loss function to calculate a relative distance relationship, and a contrastive loss function is combined to constrain the feature space distribution, so that the distance between the target feature fingerprint vector and the positive sample is less than the distance between the target feature fingerprint vector and the negative sample, and the feature representation similarity between the icon pairs is calculated respectively, the similarity is compared with a preset target similarity threshold, and the icon pairs are screened to form a feature representation set includes: A triplet composed of a target feature fingerprint vector, a positive sample feature fingerprint vector and a negative sample feature fingerprint vector is received, and the triplet is input into a deep metric learning model, the deep metric learning model includes a multi-layer fully connected encoder network and a metric network based on an attention mechanism, the encoder network includes a first fully connected layer, a second fully connected layer and a third fully connected layer connected in turn, and the output dimensions of the first fully connected layer, the second fully connected layer and the third fully connected layer decrease in turn, and a nonlinear activation layer and a batch normalization layer are connected after each fully connected layer; The Euclidean distance between the target feature fingerprint vector and the positive sample feature fingerprint vector is calculated by the metric network to obtain a positive sample distance, the Euclidean distance between the target feature fingerprint vector and the negative sample feature fingerprint vector is calculated to obtain a negative sample distance, and the difference between the positive sample distance and the negative sample distance is compared with a preset interval parameter; The cosine similarity of the feature representation of the icon pairs in the triplet is calculated, the first target similarity is set for the icon pairs of the same category, the second target similarity is set for the icon pairs of different categories, the first target similarity is greater than the second target similarity, and the feature constraint is performed based on the difference between the cosine similarity of the feature representation and the corresponding target similarity; The loss value of the triplet is calculated, a preset number of triplets with the maximum loss value are selected as difficult samples, and the difficult samples are used for parameter updating of the encoder network and the metric network; The feature fingerprints of the icon pairs in the similarity candidate set are mapped to mapping features by the encoder network, the cosine similarity between the mapping features is calculated by the metric network, and the icon pairs with a cosine similarity greater than a preset similarity threshold are saved to a feature representation set.

8. A multi-dimensional data intelligent search matching system for text and image features, for implementing the method of any one of the preceding claims 1-7, characterized in that, includes: The first unit is configured to obtain an icon image to be searched and add the icon image to a convolutional neural network for feature extraction, obtain a first feature map by combining a plurality of convolutional layers and pooling layers, input the icon image to be searched into an image segmentation attention mechanism network based on the first feature map, determine a spatial dependency of an image region by a spatial attention mechanism, generate an attention weight map by combining a channel attention module, output a graphic region and a text region, add the text region to a transformer optical character recognition model, convert the text region into a feature sequence by an encoder, decode the feature sequence by a multi-head attention mechanism in a decoder to obtain text content, extract a feature of the text content by a bidirectional semantic understanding model to obtain a text semantic feature vector, add the graphic region to a visual transformer network, perform feature cross-layer transmission based on a dense connection structure, combine a residual connection and a feature pyramid structure to perform feature fusion to obtain an image feature vector; The second unit is configured to construct the image feature vector, the text content, and the text semantic feature vector into a heterogeneous feature tensor, perform singular value decomposition on the heterogeneous feature tensor to obtain a core tensor and a factor matrix, perform correlation analysis decomposition on the core tensor to obtain an icon feature fingerprint vector, construct a local sensitive hash index based on the icon feature fingerprint vector, calculate optimal hash function parameters based on a distribution density of the icon feature fingerprint vector, adjust a hash storage unit based on the distribution density, encode the icon feature fingerprint vector by combining a location sensitive coding to obtain an encoded fingerprint vector, construct a multi-level index structure, and add the encoded fingerprint vector to a coarse-grained index layer and the icon feature fingerprint vector to a fine-grained index layer to obtain an icon feature multi-dimensional index database. The third unit is configured to receive a search request, obtain a target icon image, perform feature extraction and encoding on the target icon image, generate a target feature fingerprint vector, perform fast search on the coarse-grained index layer to obtain an initial candidate set containing a plurality of encoded fingerprint vectors, calculate a feature distance between the icon feature fingerprint vector corresponding to the encoded fingerprint vector and the target feature fingerprint vector in the fine-grained index layer, determine a feature similarity based on the feature distance, select an icon feature fingerprint vector with a feature similarity greater than a pre-set similarity threshold to generate a similarity candidate set, add the icon feature fingerprint vector to a deep metric learning model, calculate a relative distance relationship by a triplet loss function, constrain a feature space distribution by a contrastive loss function, obtain a metric feature representation, calculate a visual similarity, a text similarity, and a semantic similarity, determine a weight parameter by determining a variance and a discrimination coefficient of each similarity feature, determine a comprehensive similarity score by weighted summation, sort in descending order, and select the top 3 search results as similar search results.

9. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1 to 7.

Images