Multi-modal feature matching method and system based on artificial intelligence large model

Abstract

The invention relates to the technical field of multi-modal feature matching, and discloses a multi-modal feature matching method and system based on an artificial intelligence large model, and the method comprises the steps: obtaining an original image, carrying out the region division, extracting a visual description vector, and obtaining image region representation data; obtaining descriptive texts, performing entity recognition, and extracting semantic feature vectors; calculating an initial similarity matrix of the image region representation data and the semantic feature vector, and obtaining a preliminary weight set through dynamic association capture; carrying out weight optimization on possible local feature interference to obtain an optimized weight set; according to the optimized weight set, performing feature fusion on the image region representation data and the text entity description set to obtain a multi-modal feature set; and performing screening according to a preset threshold value, and determining a final matching result between the image region and the text entity. According to the method, high-precision semantic alignment and low-loss feature fusion between images and texts can be realized, and a fine-grained matching result is obtained.

Description

A Multimodal Feature Matching Method and System Based on Large-Scale Artificial Intelligence Models Technical Field

[0001] This invention relates to the field of multimodal feature matching technology, and in particular to a multimodal feature matching method and system based on large artificial intelligence models. Background Technology

[0002] Currently, artificial intelligence and cross-modal information processing are developing rapidly. Content understanding and semantic alignment of multimodal data such as images, text, audio, and video have become core technologies driving progress in fields such as intelligent retrieval, content recommendation, and human-computer interaction. Multimodal feature matching aims to establish semantic relationships between different modalities of data through reinforcement learning, achieving a unified understanding and efficient utilization of complex information. Especially in image and text matching tasks, its accuracy directly determines the practicality and user experience of applications such as cross-media retrieval, intelligent annotation, and visual question answering.

[0003] In existing technologies, multimodal matching methods typically rely on pre-trained visual and language models to extract features separately, and then perform alignment through simple similarity calculations (such as cosine similarity). While these methods achieve cross-modal semantic association to some extent, they often struggle to handle fine-grained correspondences between multiple objects in an image and multiple entities in text. They are easily influenced by irrelevant local features or background information, leading to deviations in matching results in complex scenarios. For example, in a news image containing people, objects, and scene elements, if the text description involves multiple entities and their attributes, traditional methods often fail to accurately distinguish which region in the image corresponds to which entity in the text, thus affecting the overall alignment accuracy. Furthermore, existing methods have limited performance in handling dynamic associations and contextual dependencies, often relying on static weights or fixed attention mechanisms, making it difficult to adapt to changes in content composition.

[0004] In summary, existing technologies still suffer from problems such as strong local feature interference, coarse semantic alignment granularity, and weak dynamic association capture in multimodal feature matching. These limitations make it difficult to achieve high-precision, interpretable, and adaptive image-text matching and low-loss feature fusion in real-world scenarios, thus restricting their further application in intelligent content understanding and interaction systems. Summary of the Invention

[0005] This invention provides a multimodal feature matching method and system based on a large artificial intelligence model, which can effectively calculate the alignment weights between image regions and text entities and achieve low-loss feature fusion in image-text matching, thereby obtaining fine-grained matching results.

[0006] Firstly, to address the aforementioned technical problems, this invention provides a multimodal feature matching method based on a large-scale artificial intelligence model, comprising: acquiring an original image; dividing the original image into regions; extracting a visual description vector for each region; normalizing the visual description vector to obtain image region representation data; acquiring descriptive text; performing entity recognition on the descriptive text; extracting a text entity description set; analyzing the modification relationships of the text entity description set; and determining a semantic feature vector by combining the image region representation data; calculating an initial similarity matrix based on the image region representation data and the semantic feature vector, and performing dynamic association capture, according to... The dynamic association process weights and normalizes the initial similarity matrix to obtain a preliminary weight set; it identifies and locates local feature interference regions in the preliminary weight set, optimizes the weight distribution based on these regions, and obtains an optimized weight set; based on the optimized weight set, it performs feature fusion on the image region representation data and the text entity description set, extracts modal complementarity information, performs depth mapping on the modal complementarity information, and obtains a multimodal feature set; it then filters the multimodal feature set according to a preset threshold to obtain feature pairs that meet fine-grained matching requirements, and determines the final matching result between the image region and the text entity.

[0007] Secondly, this invention provides a multimodal feature matching system based on a large artificial intelligence model, comprising: a visual feature representation module, used to acquire an original image, divide the original image into regions, extract visual description vectors for each region, and normalize the visual description vectors to obtain image region representation data; a semantic feature extraction module, used to acquire descriptive text, perform entity recognition on the descriptive text, extract a set of text entity descriptions, analyze the modification relationships of the text entity description sets, and determine semantic feature vectors by combining the image region representation data; and a weight distribution acquisition module, used to calculate an initial similarity matrix based on the image region representation data and the semantic feature vectors, and perform dynamic association capture based on the dynamic relationship... The initial similarity matrix is ​​weighted and normalized to obtain a preliminary weight set. A weight distribution optimization module identifies and locates local feature interference regions in the preliminary weight set, and optimizes the weight distribution based on these regions to obtain an optimized weight set. A modal feature acquisition module performs feature fusion on the image region representation data and the text entity description set based on the optimized weight set, extracts modal complementarity information, and performs depth mapping on the modal complementarity information to obtain a multimodal feature set. A filtering and matching output module filters the multimodal feature set according to a preset threshold to obtain feature pairs that meet fine-grained matching requirements, and determines the final matching result between the image region and the text entity.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention divides the original image into regions, extracts independent image region representation data sets, and combines text parsing tools to perform entity recognition and semantic vector extraction on descriptive text, thereby constructing high-quality visual representations and text representations respectively. This multimodal feature separation and structured extraction process overcomes the drawbacks of the traditional method of mixing whole-image features and full-text embedding, and coarse granularity, and realizes refined modeling of local semantics of images and text entity attributes, providing a clear and distinguishable feature basis for subsequent cross-modal alignment, and effectively reducing matching noise caused by feature interference between different objects.

[0009] (2) This invention calculates an initial similarity matrix between image region representations and text semantic features, and introduces a dynamic association capture mechanism to generate a preliminary weight set. Furthermore, it combines a weighted adjustment strategy to suppress and optimize local feature interference. This method realizes a shift from static similarity calculation to dynamic, interpretable alignment weight learning, adaptively focusing on semantically relevant regions and entities, weakening the influence of irrelevant or interfering features, thereby significantly improving the robustness and accuracy of the alignment process. It solves the problem of traditional matching methods being susceptible to local interference and having weak semantic associations in complex multi-object scenarios.

[0010] (3) Based on the optimized alignment weight set, this invention achieves feature fusion of image region representation data and text entity description through a deep mapping network, generating a unified multimodal feature set, and completes fine-grained matching with the help of a threshold screening mechanism. Finally, the alignment weight is continuously optimized through result verification and feedback loop. This process constructs a complete closed-loop system of "region segmentation - entity recognition - dynamic alignment - weight optimization - feature fusion - matching verification", which makes the matching process adaptive, iterative and interpretable, realizing the evolution from shallow feature interaction to deep semantic fusion, and significantly improving the understanding accuracy and application reliability of multimodal content in tasks such as retrieval, recommendation and generation. Attached Figure Description

[0011] Figure 1 is a flowchart illustrating an embodiment of the multimodal feature matching method based on a large artificial intelligence model provided by the present invention; Figure 2 is a flowchart illustrating an embodiment of obtaining a multimodal feature set provided by the present invention; Figure 3 is a structural diagram illustrating an embodiment of the multimodal feature matching system based on a large artificial intelligence model provided by the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Referring to Figure 1, the first embodiment of the present invention provides a multimodal feature matching method based on a large artificial intelligence model, including the following steps: S11, acquiring an original image, dividing the original image into regions, extracting visual description vectors for each region, and normalizing the visual description vectors to obtain image region representation data; S12, acquiring descriptive text, performing entity recognition on the descriptive text, extracting text entity description sets, analyzing the modification relationships of the text entity description sets, and determining semantic feature vectors by combining the image region representation data; S13, calculating an initial similarity matrix based on the image region representation data and the semantic feature vectors, and performing dynamic association capture, based on the... The dynamic association performs weighted and normalized processing on the initial similarity matrix to obtain a preliminary weight set; S14, identifies and locates local feature interference regions in the preliminary weight set, and optimizes the weight distribution based on the local feature interference regions to obtain an optimized weight set; S15, performs feature fusion on the image region representation data and the text entity description set based on the optimized weight set, extracts modal complementarity information, and performs depth mapping processing on the modal complementarity information to obtain a multimodal feature set; S16, filters the multimodal feature set according to a preset threshold to obtain feature pairs that meet the fine-grained matching requirements, and determines the final matching result between the image region and the text entity.

[0014] In step S11, acquiring the original image, dividing the original image into regions, extracting the visual description vector of each region, and normalizing the visual description vector to obtain image region representation data includes: acquiring the original image; generating a region mask matrix based on the original image using a semantic segmentation network; extracting independent region images from the original image based on the region mask matrix; calculating the local feature set of the independent region images; performing feature encoding mapping on the local feature set to obtain a visual description vector; and normalizing the visual description vector to obtain image region representation data.

[0015] First, the original image is acquired, and a region mask matrix is ​​generated using a semantic segmentation network model. It should be noted that the original image is typically obtained from an image acquisition device, such as a camera or scanner, and its format can be RGB, grayscale, or other multi-channel images. The semantic segmentation network model employs a deep convolutional neural network based on an encoder-decoder structure, using the DeepLabV3+ architecture. Its encoder typically uses backbone networks such as ResNet or EfficientNet pre-trained on large datasets like ImageNet to extract multi-scale image features. The decoder, through modules such as upsampling, skip connections, and Spatial Convolutional Pooling Pyramid (ASPP), fuses the high-level semantic features output by the encoder with low-level detail features, ultimately predicting the semantic category (e.g., "person," "vehicle," "sky," "building," etc.) for each pixel in the image and outputting a region mask matrix with the same spatial size as the input image. Each position in this matrix contains an integer value representing the semantic region number to which that pixel belongs. The pre-training process of the semantic segmentation network model is as follows: A large-scale image dataset containing multiple scenes (such as indoor homes, city street scenes, and natural scenery) is constructed. This dataset includes the publicly available datasets COCO-Stuff (approximately 160,000 images) and ADE20K (approximately 20,000 images), divided in an 8:1:1 ratio. Each pixel in the dataset is precisely labeled with its category, forming a pixel-level label mask. The dataset is then divided into training, validation, and test sets in an 8:1:1 ratio. The model's loss function is a weighted combination of cross-entropy loss and Dice loss (with a weight ratio of 1:1). The Adam optimizer (β1=0.9, β2=0.999) is used for parameter updates, with an initial learning rate set to 0.001, dynamically adjusted using a cosine annealing strategy (decaying every 10 epochs, with a minimum learning rate of 1e-6). The convergence criterion is that the variation of the mean Intersection over Union (mIoU) on the validation set is less than 0.01% for five consecutive epochs. During training, data augmentation techniques (such as random flipping, scaling, and cropping) are introduced to improve the model's generalization ability. The trained model outputs a region mask matrix with the same size as the original image, where each element represents the object category index of the corresponding pixel (e.g., 0 for background, 1 for people, 2 for vehicles, etc.).

[0016] Subsequently, based on the region mask matrix, independent region images are extracted from the original image, and the local feature set of the independent region images is calculated. It should be noted that the non-zero class indices in the region mask matrix are traversed, and for each independent index value, the corresponding pixel coordinate set is extracted using the matrix index. Based on the pixel coordinate set, the minimum bounding rectangle covering the region is determined, and the image matting operation is performed on the original image using this rectangle coordinates. For the matted region images, the system fills the redundant background outside the mask with zero pixels, thereby obtaining independent region images with accurate edge contours. For each independent region image, its local feature set is calculated. Specifically, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract keypoints and their descriptors. First, extreme points are detected in the difference Gaussian scale space, then the principal direction of the keypoints is calculated, and finally a 128-dimensional local feature descriptor is generated. For a complex independent region image, hundreds of such local feature descriptors are extracted, forming a local feature set.

[0017] Then, feature encoding mapping is performed on the local feature set to obtain the visual description vector. It should be noted that the Local Aggregated Descriptor Vector (VLAD) encoding method is used, and a codebook containing K visual words is trained on the local features of a large number of training images using the K-means clustering algorithm, where K is typically 64 or 128. Next, for the local feature set of the current independent region image, the residual vector between the local feature vector and the nearest visual word is calculated. Specifically, the Euclidean distance between the local features and the K visual words in the codebook is calculated, and the visual word corresponding to the minimum Euclidean distance is taken as the nearest visual word. The local feature vector is subtracted from its corresponding best visual word to obtain the residual vector. The residual vectors of all visual words belonging to the same visual word are accumulated, and finally, the accumulated residual vectors of all visual words are concatenated to obtain the visual description vector.

[0018] Finally, the visual description vectors are normalized to obtain image region representation data. L2 normalization is performed on each visual description vector, i.e., L2 normalization is calculated. ,in, This is the normalized visual description vector. The visual description vector, Representing vectors The Euclidean norm is used to obtain image region representation data after normalization.

[0019] For example, consider an interior design scene image containing multiple pieces of furniture. First, the image is input into a semantic segmentation model. The model outputs a mask matrix, precisely marking the pixel regions of "sofa," "coffee table," and "floor lamp." The system then crops out the "coffee table" region separately based on the mask, extracting its wood grain texture and edge corners as a local feature set (containing approximately 300 SIFT feature points). Next, using a pre-trained codebook containing 64 visual words, these 300 feature points are mapped and aggregated into an 8192-dimensional (...) The visual description vector of the "coffee table" area is obtained. Finally, after L2 normalization, the image representation data of the "coffee table" area is obtained.

[0020] In step S12, the process of acquiring descriptive text, performing entity recognition on the descriptive text, extracting a set of text entity descriptions, analyzing the modification relationships of the text entity descriptions, and determining semantic feature vectors by combining the image region representation data includes: acquiring descriptive text; performing sequence labeling on the descriptive text using a conditional random field to obtain a set of text entity descriptions; analyzing the modification relationships between entities in the text entity descriptions using dependency parsing to generate an initial entity vector matrix; aligning the initial entity vector matrix with the image region representation data using multimodal attention to obtain an alignment vector distribution; and performing fully connected layer mapping processing on the alignment vector distribution to obtain semantic feature vectors.

[0021] First, descriptive text is obtained and sequence-annotated using a Conditional Random Field (CRF) model to obtain a set of text entity descriptions. It should be noted that semantic analysis is performed on the original image to obtain natural language descriptive text reflecting the main content and entity information of the image, and the descriptive text is then segmented and stop-word removed. The processed text sequence is then input into a pre-trained CRF model for sequence annotation. It should be noted that the CRF model is built on a Bidirectional Long Short-Term Memory (BiLSTM) network, forming a BiLSTM-CRF architecture. The BiLSTM layer is used to extract contextual features of the text, and the CRF layer is used to constrain the legality of the prediction results using label transition probabilities. The pre-training process of this model is as follows: a large image-text corpus containing rich entity descriptions is collected, and professional annotators use the BIO annotation system to annotate entities in the text (such as "person," "table," and "red car"); a training set is constructed, using the text portions of the public datasets MS-COCO and Flickr 30k, supplemented by Wikipedia entity-annotated corpus, with a total scale of approximately 150,000 text entries. The data was divided into training, validation, and test sets in an 8:1:1 ratio, and the loss function was defined as negative log-likelihood loss. The Adam optimizer (β1=0.9, β2=0.999) was used for parameter updates, with an initial learning rate of 0.001 and an exponential decay strategy (decay rate of 0.95 per epoch). The convergence criterion was that the F1 score for entity recognition on the validation set changed by less than 0.01% for three consecutive epochs. After training, the model could output the label sequence corresponding to each word in the text, thus obtaining a set of text entity descriptions. For example, inputting descriptive text, such as "A girl wearing a red dress is playing soccer in the park," into the model, the model outputs entities such as "girl" (PERSON), "red dress" (ATTR), "park" (LOC), "kick" (ACTION), and "soccer ball" (OBJECT), along with their types, constituting the set of text entity descriptions. ,in The number of entities identified, each entity It includes its text fragments and entity types.

[0022] Subsequently, a dependency parser is used to parse the modification relationships between entities, generating an initial entity vector matrix. It should be noted that the dependency parser, such as one based on the Arc-Eager transfer system, parses the dependency relationships between each entity and its modifiers in the text entity description set, such as subject-verb and attributive-head relationships. Based on the parsing results, the word vectors of the entity words and their modifiers are aggregated. For example, for "red sports car," the Word2Vec vectors or BERT embedding vectors of "red" and "sports car" are weighted and averaged according to dependency weights to generate the initial entity vector matrix. The dependency weights are obtained by analyzing the statistical distribution of the point mutual information between modifiers and headwords in a large-scale corpus. This is achieved by pre-calculating the point mutual information index of all commonly used adjective and noun pairs in the large-scale corpus and mapping it to a weight interval of [0.1, 0.9] as the dependency weights. For example, the weight for an adjective modifying a noun is set to 0.6, and the weight for the noun itself is set to 0.4.

[0023] Next, the initial entity vector matrix is ​​aligned with the image region representation data obtained in step S11 using multimodal attention to obtain the alignment vector distribution. It should be noted that the matrix formed by the image region representation data is... The initial entity vector matrix is Using text features as queries and image features as keys and values, an attention weight matrix is ​​calculated. The formula is ,in, and These are the learnable linear transformation matrices for the query and key, respectively. and As a learnable linear transformation matrix that maps semantic feature vectors and image region representation data to the common attention subspace, its value is not fixed in advance. Instead, during multimodal joint training based on a large-scale image-text pairing dataset, the parameters are initialized using the He method, and the gradient is calculated through backpropagation algorithm and continuously updated using the Adam optimizer as the model minimizes the contrastive loss function until convergence is determined. It is a fixed hyperparameter determined according to the feature channel dimension (such as 64 or 128) preset by the model architecture. In the formula, its square root is usually taken as the scaling factor to perform numerical smoothing on the dot product result. This prevents the dot product value from being too large due to the vector dimension being too high, which would cause the Softmax function to enter the saturation region with minimal gradient, thus ensuring the stability and convergence efficiency of model training. The function operates on each row of the matrix such that the sum of the attention weights for each text entity across all image regions is 1. Based on the attention weight matrix... Calculate the alignment vector distribution ,in The transformation matrix for the value is determined in the same way. and .

[0024] Finally, the alignment vector distribution is mapped using a fully connected layer to obtain semantic feature vectors. It should be noted that the alignment vector distribution is input into a neural network containing two fully connected layers and a ReLU activation function. This neural network includes the non-linear activation function ReLU and possible layer normalization. This mapping process can be represented as follows: The output of this network is the final semantic feature vector, whose dimensions are aligned with the image region representation data. For example, the input descriptive text is "A retro-style wooden coffee table is placed in the center of the living room." The CRF model identifies the core entity "coffee table," and the dependency parser extracts the modifiers "retro style" and "wooden." The system analysis yields the following weights: "coffee table" (center word) weight 0.5, "retro style" (adjective) weight 0.25, and "wooden" (noun modifier) ​​weight 0.25. The initial entity vector is calculated as follows: Then, through an attention mechanism, the attention score between the "coffee table" entity vector and the "sofa" region in the image is calculated to be 0.15, the score with the "floor" region is 0.1, and the score with the "coffee table" image region is as high as 0.75. The final generated semantic feature vector aggregates 75% of the coffee table region features in the image, forming a deep semantic representation with image-text alignment.

[0025] In step S13, the step of calculating an initial similarity matrix based on the image region representation data and the semantic feature vector, and performing dynamic association capture, and then weighting and normalizing the initial similarity matrix based on the dynamic association to obtain a preliminary weight set includes: calculating the projection overlap between the semantic feature vector and the image region representation data to obtain an initial similarity matrix; extracting spatial association features through convolution operations based on the neighborhood distribution of the initial similarity matrix; performing serial dependency analysis on the spatial association features to determine a dynamic association factor set; and performing weighted modulation and normalized exponential function operations on the initial similarity matrix based on the dynamic association factor set to obtain a preliminary weight set.

[0026] First, the projection overlap between the semantic feature vector and the image region representation data is calculated to obtain an initial similarity matrix. It should be noted that the projection overlap is calculated using the cosine similarity formula. ,in, To calculate the projection overlap between the representation data of the i-th image region and the semantic feature vector of the j-th region, The image region is characterized by data. For example, given the semantic feature vectors, there are M image region representation data and N semantic feature vectors, and the projection overlap degree will be calculated. As the element in the i-th row and j-th column of the matrix, an initial similarity matrix of dimension M×N is finally constructed.

[0027] Subsequently, based on the neighborhood distribution of the initial similarity matrix, spatial association features are extracted through convolution operations. Since adjacent elements in the similarity matrix may correspond to spatially adjacent regions in an image or semantically similar entities in text, a convolutional neural network (CNN) is used to capture these local patterns. Specifically, a convolutional neural network containing multiple... The convolutional layer with convolutional kernels performs convolution operations on the initial similarity matrix. The convolution stride is set to 1, and zero-padding is used at the edges to maintain consistent feature map size. The parameters of the convolutional kernels are initialized using the He initialization method, and the matching error loss function is used as the optimization objective during the overall model training process. The parameters of the convolutional kernels are iteratively updated through the backpropagation algorithm. The output of the convolution operation is normalized by the BatchNorm layer and activated by ReLU to obtain spatial correlation features reflecting local matching patterns.

[0028] Subsequently, a sequence dependency analysis was performed on the spatial association features to determine the set of dynamic association factors. Considering the sequential dependencies between text entities within a sentence, a Bidirectional Long Short-Term Memory (BiLSTM) network was used to scan the spatial association features along the text dimension. The BiLSTM network consists of forward and backward LSTM layers, with each LSTM unit containing a forget gate, input gate, and output gate to regulate feature transfer across time steps. Hyperparameter settings included 256 hidden layer nodes, a dropout rate of 0.3 to prevent overfitting, and a time step corresponding to the maximum sequence length of the text entities. During scanning, the forward layer captured the forward semantic logic between entities, while the backward layer captured the reverse context. The pre-training process of the BiLSTM network involved self-supervised pre-training on Wikipedia and a general image-text corpus using a masked entity prediction task to acquire basic semantic association capabilities. Subsequently, supervised fine-tuning was performed in a matching task using a multimodal contrastive loss function to ensure the model could accurately quantify the matching relevance between image features and text sequences. The spatial association features were then... All associated features of a column represent the first The matching feature vectors of each text entity and all image regions are input into a BiLSTM network to obtain a sequence of hidden states containing contextual information. Then, a fully connected layer activated by a sigmoid is used to map the hidden states into dynamic correlation factors ranging from (0, 1). This factor characterizes the first [factor] in the current context. The image region and the first Association confidence of each text entity. This constitutes a set of dynamically related factors.

[0029] Finally, based on the set of dynamic correlation factors, the initial similarity matrix is ​​weighted and modulated using a normalized exponential function to obtain a preliminary weight set. (The formula is used to...) The initial similarity matrix is ​​modulated, where The adjustment coefficient is determined by observing the change curve of the matching recall on the validation set, and the peak inflection point of the change curve is taken as the adjustment coefficient. The value is the adjustment coefficient, typically between 0.5 and 1.0. For example, [the value is...]. Iterative testing was performed within the range [0,2] with a step size of 0.1, and it was found that... When the value is 0.5, the curve reaches its peak inflection point, and the improvement in match recall is most significant; therefore, it is set to 0.5. This is the modulated similarity matrix. The modulated similarity matrix is ​​normalized along the image region dimension using a normalization exponential function to obtain a preliminary weight set. The normalization exponential function is: ,in, This is a temperature coefficient used to control the smoothness of the distribution. The value of is determined through a grid search, specifically, a search is performed in the parameter space [0.01, 0.05, 0.07, 0.1, 0.2], with the matching accuracy on the validation set used as the evaluation metric. For example, when When = 0.07, the Softmax function can effectively widen the difference between positive and negative samples without causing gradient saturation, therefore let . It is 0.07. Indicates the first The text entity focuses on the first Alignment weights for each image region The amount of data is used to characterize the image region.

[0030] For example, suppose that in the above scenario, the text mentions "sofa" in addition to "coffee table". The initial similarity matrix shows that the text vector for "coffee table" has a similarity of 0.75 with the "coffee table" region in the image, but also a similarity of 0.6 with the "wooden floor" region in the image, which has a similar color. Through convolution operations and BiLSTM analysis, the BiLSTM outputs the dynamic association factor between "coffee table" and "sofa". The dynamic correlation factor between the "coffee table" and the "floor" is 0.2, and an adjustment coefficient is set. The corrected similarity of the "coffee table" area is then... The "floor" area is .go through After normalization, in the initial weight distribution, the weight of the "coffee table" area increased to 0.87, while the weight of the "floor" area was suppressed to 0.12.

[0031] In step S14, identifying and locating local feature interference regions in the preliminary weight set, and optimizing the weight distribution based on the local feature interference regions to obtain an optimized weight set, includes: calculating the spatial distribution entropy of the preliminary weight set; locating local feature interference regions based on the spatial distribution entropy; generating an interference suppression factor matrix based on the local feature interference regions; and adjusting the weights of the preliminary weight set using the interference suppression factor matrix to obtain the optimized weight set.

[0032] First, based on the spatial distribution entropy of the preliminary weight set, local feature interference regions are located. It should be noted that the spatial distribution entropy of the weight vector corresponding to each text entity in the preliminary weight set is calculated; specifically, for the... There are text entities, and their weight vectors are: Spatial distribution entropy The calculation formula is ,in, Minimum value (e.g.) To avoid errors in logarithmic calculations, entropy value This reflects the degree of concentration in the matching between the entity and the image region. The entropy value represents the amount of data in the image region. If the entropy value is too high, it indicates that the entity has a certain similarity to multiple regions in the image, suggesting matching ambiguity or background noise interference. If the entropy value is too low, it indicates that the matching is too concentrated, potentially ignoring the multi-scale features of the entity. An interference judgment threshold is set. This threshold is obtained by statistically analyzing the entropy distribution of correctly matched samples and taking its 95th quantile. If If the weight distribution corresponding to the entity is found to have local feature interference, it needs to be located. The location method is as follows: in the weight vector, find the index of the region whose value is significantly lower than the main peak value but greater than the mean background noise value, and mark it as the interference region. Specifically, "significantly lower than the main peak value" is defined as the weight value of the region being less than 60% of the maximum weight value, and "greater than the mean background noise value" is defined as... It should be noted that the background noise interference is obtained as follows: during the semantic segmentation process in step S11, the model labels non-subject object regions (such as walls, ceilings, blurred backgrounds, etc.) as the "background" category (usually with an index of 0). The system collects preliminary weight values ​​calculated from the "background" regions and arbitrary text entities of a large number of images in the training set, statistically analyzes the background noise distribution, and thus calculates the mean of the background noise weights. and standard deviation .

[0033] Subsequently, an interference suppression factor matrix is ​​generated based on the local feature interference regions. The initial weight set is then weighted and adjusted using this interference suppression factor matrix to obtain an optimized weight set. For regions determined to be interference... Set its inhibitory factor The calculation uses a bell-shaped function. in, The mean of the background noise weights. The noise standard deviation is obtained by statistically estimating the weights of the non-matching regions. The maximum suppression strength is empirically set to 0.8. This value is determined based on the signal-to-noise ratio balance principle. Specifically, sensitivity analysis of the suppression strength in simulation tests revealed that if the suppression strength is too low (e.g., <0.5), it cannot effectively eliminate the influence of semantically similar objects (e.g., the interference of the TV cabinet on the coffee table); if the suppression strength is too high (e.g., >0.9), it is easy for the gradient to become unstable during backpropagation due to drastic fluctuations in the weight distribution, and even mistakenly killing edge features with certain correlations. Setting it to 0.8 can suppress significant noise while retaining weak contextual correlation information in the image background, so that the model achieves the optimal balance between robustness and feature integrity. For non-interference areas, Set to 1, thus constructing the interference suppression factor matrix. The initial weight set is then weighted and adjusted using this interference suppression factor matrix to obtain the optimized weight set. The calculation formula is as follows: , This is the adjusted alignment weight. After adjustment, it needs to be re-aligned. Each column is normalized to ensure the probability sum is 1, resulting in an optimized weight set. Specifically, L1 normalization is used to calculate the sum of the adjusted weight vectors. The normalized weights are obtained. .

[0034] As shown in Figure 2, in step S15, the step of performing feature fusion on the image region representation data and the text entity description set according to the optimized weight set, extracting modal complementarity information, and performing deep mapping processing on the modal complementarity information to obtain a multimodal feature set includes: S151, performing weighted processing on the image region representation data and the text entity description according to the optimized weight set to obtain a weighted visual semantic vector and a weighted text embedding vector; S152, constructing a cross-modal mapping relationship matrix based on the weighted visual semantic vector and the weighted text embedding vector, and extracting modal complementarity information; S153, projecting the modal complementarity information onto the fusion feature space through a preset deep mapping network, and obtaining a joint representation vector through joint distribution alignment calculation; S154, performing feature reconstruction and principal component analysis dimensionality reduction processing on the joint representation vector to obtain a multimodal feature set.

[0035] First, based on the optimized weight set, the image region representation data and text entity descriptions are weighted to obtain a weighted visual semantic vector and a weighted text embedding vector. Specifically, the optimized weight set is then used... Image region representation data By performing weighted summation, we obtain the weighted visual semantic vector for each text entity. At the same time, the original text entity description vectors are preserved. As a weighted text embedding vector (the weight can be considered as 1).

[0036] Subsequently, a cross-modal mapping matrix is ​​constructed based on the weighted visual semantic vector and the weighted text embedding vector, and modal complementarity information is extracted. It should be noted that a gating mechanism-based fusion method is used to construct the cross-modal mapping matrix and extract modal complementarity information; specifically, the gating vector is calculated. ,in This represents vector concatenation, whereby the weighted visual semantic vector and the weighted text embedding vector are joined end-to-end to form a joint vector, where... This is a gating weight matrix used to learn the correlation strength between visual features and text features across different dimensions. This is the bias vector, used to balance the model's prior preferences for different modalities. and The values ​​are not fixed in advance, but are initialized using the He method during multimodal joint training based on a large-scale image-text matching dataset. The loss is calculated using the backpropagation algorithm by calculating the contrastive loss function of the final matching task. and The gradient is calculated, and the Adam optimizer is used to update the parameters until convergence is determined. The function is the Sigmoid function. Then, mode complementarity information is extracted using the gated vector. ,in The Hadamard product is the product of the elements of two vectors.

[0037] Subsequently, the modal complementarity information is input into the preset deep mapping network, projected onto the fusion feature space, and a joint representation vector is obtained through joint distribution alignment calculation. It should be noted that the deep mapping network is designed as a multilayer perceptron (MLP) structure, including an input layer, three hidden layers (with 1024, 512, and 256 nodes respectively), and an output layer. Residual connections and LayerNorm techniques are used between the hidden layers to accelerate convergence and prevent gradient vanishing. The pre-training process of this network adopts a contrastive learning strategy, constructing positive sample pairs (matching image-text pairs) and negative sample pairs (mismatching image-text pairs), and training is performed using the InfoNCE loss function with a temperature parameter τ=0.07. The optimizer uses Adam (β1=0.9, β2=0.999), with an initial learning rate of 0.001, and the learning rate decay uses a cosine annealing strategy (period of 50 epochs). The convergence criterion is that the change in validation set matching accuracy is less than 0.01% for 5 consecutive epochs. The training data comes from the publicly available multimodal datasets MS-COCO (approximately 120,000 image-text pairs) and Flickr 30k (approximately 30,000 pairs). The datasets are divided into training, validation, and test sets in an 8:1:1 ratio to ensure the model's generalization ability. The trained deep mapping network is input and projected onto the fusion feature space to obtain preliminary feature vectors. To further eliminate distribution differences between modalities, joint distribution alignment calculation is performed in the fusion feature space. A maximum mean difference minimization strategy is adopted to pull the feature distribution of the current batch towards a Gaussian distribution, resulting in a joint representation vector. Specifically, in the feature space, the data is mapped to the reproducing kernel Hilbert space (RKHS) using a Gaussian kernel function, and the distance between the source domain (the current fusion feature distribution) and the target domain (the Gaussian distribution mean embedding) is calculated. During the training process of the deep mapping network, It is added as a regularization term to the total loss function to make the distribution of the fused features approximate the standard Gaussian distribution.

[0038] Finally, the joint representation vector is subjected to feature reconstruction and densification to obtain a multimodal feature set. Feature reconstruction is performed through a decoder network whose structure is symmetrically designed with the deep mapping network (as the encoder). For example, if the encoder layer is 1024->512->256, then the decoder layer is 256->512->1024. The decoder receives the joint representation vector as input and outputs the reconstructed visual features. and text features The system calculates the mean squared error (MSE) between the original features and the reconstructed features as the reconstruction loss. ,in and The weighted visual semantic vector and the weighted text embedding vector obtained in step S151 are used. By minimizing this loss during training, the intermediate joint representation vector is forced to retain key features sufficient to reconstruct the original information. The compaction process involves principal component analysis, calculating the covariance matrix of the joint representation vector on the training set, performing eigenvalue decomposition, and selecting the first... The eigenvectors corresponding to the largest eigenvalues ​​(e.g., 128) form a projection matrix, which maps the high-dimensional joint vector to a low-dimensional compact space, resulting in a multimodal feature set.

[0039] For example, for the alignment feature of "wooden coffee table," the visual vector might emphasize pixel features such as "brown" and "texture," while the text vector might emphasize conceptual features such as "furniture" and "support." In the gated vector (512 dimensions) calculated by the gating mechanism, the value corresponding to the "color" and "texture" dimensions is 0.85 (biased towards vision), and the value corresponding to the "object category" dimension is 0.2 (biased towards text). Depending on the context, if the task is to distinguish materials, the gated vector tends towards 1, retaining more visual texture information; if the task is to classify objects, the gate tends towards 0, retaining textual concepts. Under the MMD constraint, the fused vector is mapped to a standard Gaussian distribution space, making the features of "coffee table" closer to "table" and further away from "car." Finally, after densification, a simplified 128-dimensional vector is output, resulting in a multimodal feature set.

[0040] In step S16, the step of filtering the multimodal feature set according to a preset threshold to obtain feature pairs that meet the fine-grained matching requirements and determining the final matching result between the image region and the text entity includes: calculating the similarity between the multimodal feature sets and constructing a feature similarity matrix; extracting index pairs greater than a preset global association threshold from the feature similarity matrix to obtain a candidate feature pair set; calculating the local alignment degree of the candidate feature pair set to obtain a fine-grained matching confidence score; filtering feature pairs in the candidate feature pair set whose fine-grained matching confidence score is higher than a preset fine-grained filtering threshold to obtain a target feature pair sequence; and constructing a bidirectional index table of the image region and the text entity based on the target feature pair sequence to obtain the final matching result.

[0041] First, for the fused multimodal feature set, calculate the cosine similarity between each pair of features to construct a feature similarity matrix. Specifically, assume that the multimodal feature set contains... Image region fusion features and Text entity fusion features For any pair of features The formula for calculating cosine similarity is: Iterate through all combinations to obtain A feature similarity matrix of dimension.

[0042] Subsequently, extract features with similarity greater than a preset global association threshold from the feature similarity matrix. Index pairs. The method for obtaining the similarity is as follows: calculate the mean similarity of all positive sample pairs on the validation set. and standard deviation and the mean similarity of negative sample pairs and standard deviation ,set up This involves taking the similarity values ​​corresponding to the equal error rate (EER) points of the two distributions. Index pairs that meet the criteria are then extracted to form a candidate feature pair set.

[0043] Next, the local alignment degree of the candidate feature pairs is calculated to obtain a fine-grained matching confidence score. The local alignment degree considers not only vector similarity but also spatial geometric consistency. For example, if the text describes "the chair on the left" and "the table on the right," then it is necessary to check whether the relative positions of the center coordinates of the corresponding image regions conform to a "left-right" relationship, defining a spatial consistency score. Specifically, the centroid coordinates of the image region mask are extracted and normalized into an image geometric orientation vector. Simultaneously, dependency parsing is used to identify locative words describing the entity in the text, and the text semantic orientation vector is obtained based on the correspondence between the locative words and a preset spatial mapping table. The cosine similarity between the image geometric orientation vector and the text semantic orientation vector is calculated as a spatial consistency score. Combining vector similarity Calculate fine-grained matching confidence. ,in These are weighting coefficients, typically set to 0.7 and 0.3.

[0044] Subsequently, candidates with a fine-grained matching confidence level higher than a preset fine-grained filtering threshold are selected. The characteristic pair. The ratio is usually set. More stringent requirements can be set, for example, to 0.85, to ensure high precision in the final output. Feature pairs that meet the conditions constitute the target feature pair sequence.

[0045] Finally, a bidirectional index table of image regions and text entities is constructed based on the target features. This index table contains two hash maps: Map_Image_to_Text (Key is the image region ID, Value is a list of matching text entity IDs) and Map_Text_to_Image (Key is the text entity ID, Value is a list of matching image region IDs). Through this bidirectional index table, the system can quickly respond to text queries based on image regions or image location requests based on text entities, and output the final matching result. For example, after the above calculation, the system finds that the dot product of the fused feature vector (128 dimensions) of "coffee table text" and "coffee table image region" is 0.92, the product of moduli is 1, and the cosine similarity is 0.92. Setting the global threshold to 0.75, the pair passes the screening. Further checking the spatial location, the text does not mention orientation, therefore... Set the default value to 1.0. Calculate fine-grained confidence scores. This value is higher than the fine-grained screening threshold. Therefore, the system records in the bidirectional index table that the image region (coffee table) with ID 102 corresponds to the text entity (coffee table) with ID 5. The final output clearly indicates that the brown rectangular area in the center of the image is the "wooden coffee table" described in the text.

[0046] After step S16, the method further includes: comparing the final matching result with the preset real label to extract a set of mismatched samples; calculating the matching accuracy feedback value; if the matching accuracy feedback value is lower than the preset matching threshold, constructing a loss function based on the set of mismatched samples; using the loss function to perform inverse differentiation calculation on the optimized weight set to obtain gradient information; updating the optimized weight set based on the gradient information to obtain an updated weight set.

[0047] First, the final matching result is compared with preset real labels to extract a set of mismatched samples. It should be noted that the preset real labels refer to the pre-annotated correspondences between image regions and text entities during model training, typically existing as a (region index, entity index) pairing list. The final matching result is then compared item by item with the preset real labels. The comparison process includes checking whether each predicted match pair exists in the real label list. Furthermore, match pairs that exist in the real labels but not in the predicted result are also considered missed matches, resulting in a set of mismatched samples. For example, in "office scene" recognition, if the system mismatches "monitor" as "laptop," this pair of data is stored in the mismatched sample set.

[0048] Subsequently, the matching accuracy feedback value is calculated. If the matching accuracy feedback value is lower than a preset matching threshold, a loss function is constructed based on the set of mismatched samples. It should be noted that the matching accuracy feedback value is obtained by calculating the ratio of the number of correctly matched pairs to the total number of matched pairs in the final matching result. The matching threshold is the preset matching threshold, calculated by taking the 90th percentile of the matching accuracy feedback values ​​of historical matched pairs. For example, the preset matching threshold is 0.85. A contrastive loss function is constructed based on the set of mismatched samples. Specifically, for mismatched samples, the optimized weight distribution generated in step S14 is extracted, and a feedback loss function is constructed. ,in, It is the cosine similarity between the incorrect image regions and text entities in the set of mismatched samples. When a mismatch occurs, the text entity is located to the correct image region index based on the manually labeled truth value, and then the value corresponding to the index is extracted from the initial similarity matrix or preliminary weight set generated in step S13.

[0049] Next, the loss function is used to perform back-derivative calculation on the optimized weight set to obtain gradient information. Specifically, along the computation path of the neural network, the gradient is calculated. For optimizing the weight matrix The partial derivatives of are used as the gradient information.

[0050] Finally, the optimized weight set is updated based on the gradient information to obtain the updated weight set. Stochastic gradient descent is used to adjust the weight distribution according to a preset step size. Specifically, the update formula is as follows: ,in, For the updated weights, For the optimized weights, To preset the step size, an "exponential decay strategy" is adopted, with an initial step size of 0.001 to ensure that the update process does not skip local optima. As the number of feedback iterations increases, the step size is adjusted according to... Decrease successively The gradient information, Let be the loss function.

[0051] For example, in a certain matching, a mismatch resulted in an accuracy feedback value of 0.78 (below the threshold of 0.85). The system discovered through back-differentiation that the error was caused by the excessively high weight of the "television" background region. Through gradient backpropagation, the system dynamically fine-tuned the sensitivity of the inhibition factor for this region in subsequent similar tasks from 0.8 to 0.82. In the updated weight set, the interference weight of "television" decreased from 0.3 to 0.12, and the matching accuracy subsequently improved to above 0.88.

[0052] In summary, this invention discloses a multimodal feature matching method based on a large-scale artificial intelligence model, including: acquiring an original image; dividing the original image into regions; extracting visual description vectors for each region; normalizing the visual description vectors to obtain image region representation data; acquiring descriptive text; performing entity recognition on the descriptive text; extracting text entity description sets; analyzing the modification relationships of the text entity description sets; and determining semantic feature vectors by combining the image region representation data with the semantic feature vectors; calculating an initial similarity matrix based on the image region representation data and the semantic feature vectors; and performing dynamic association capture based on the dynamic association... The initial similarity matrix is ​​weighted and normalized to obtain a preliminary weight set. Local feature interference regions in the preliminary weight set are identified and located. The weight distribution is optimized based on these local feature interference regions to obtain an optimized weight set. Based on the optimized weight set, feature fusion is performed on the image region representation data and the text entity description set to extract modal complementarity information. This modal complementarity information is then subjected to depth mapping to obtain a multimodal feature set. The multimodal feature set is then filtered according to a preset threshold to obtain feature pairs that meet fine-grained matching requirements, thus determining the final matching result between the image region and the text entity. This invention achieves efficient calculation of alignment weights and low-loss feature fusion between image regions and text entities in image-text matching through multimodal feature matching, thereby obtaining fine-grained matching results.

[0053] Referring to Figure 3, the second embodiment of the present invention provides a multimodal feature matching system based on a large artificial intelligence model, comprising: a visual feature representation module, used to acquire an original image, divide the original image into regions, extract visual description vectors for each region, and normalize the visual description vectors to obtain image region representation data; a semantic feature extraction module, used to acquire descriptive text, perform entity recognition on the descriptive text, extract a set of text entity descriptions, analyze the modification relationships of the text entity description sets, and determine semantic feature vectors by combining the image region representation data; and a weight distribution acquisition module, used to calculate an initial similarity matrix based on the image region representation data and the semantic feature vectors, and perform dynamic association capture, based on the dynamic... The initial similarity matrix is ​​weighted and normalized to obtain a preliminary weight set. A weight distribution optimization module identifies and locates local feature interference regions in the preliminary weight set, and optimizes the weight distribution based on these regions to obtain an optimized weight set. A modal feature acquisition module performs feature fusion on the image region representation data and the text entity description set based on the optimized weight set, extracts modal complementarity information, and performs depth mapping on the modal complementarity information to obtain a multimodal feature set. A filtering and matching output module filters the multimodal feature set according to a preset threshold to obtain feature pairs that meet fine-grained matching requirements, and determines the final matching result between the image region and the text entity.

[0054] It should be noted that the multimodal feature matching system based on a large AI model provided in this embodiment of the invention is used to execute all the process steps of the multimodal feature matching method based on a large artificial intelligence model in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0055] This invention also provides an electronic device. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as an image processing program. When the processor executes the computer program, it implements the steps in the above-described embodiments of the multimodal feature matching system method based on a large AI model, such as step S11 shown in FIG1. ​​Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described system embodiments, such as an image processing module.

[0056] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0057] The electronic device may be a desktop computer, laptop, handheld computer, or smart tablet, etc. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0058] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting all parts of the electronic device via various interfaces and lines.

[0059] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0060] If the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0061] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0062] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A multimodal feature matching method based on a large artificial intelligence model, characterized in that, include: The process involves: acquiring an original image, dividing the original image into regions, extracting visual description vectors for each region, and normalizing the visual description vectors to obtain image region representation data; acquiring descriptive text, performing entity recognition on the descriptive text, extracting text entity description sets, analyzing the modification relationships of the text entity description sets, and combining the image region representation data to determine semantic feature vectors; calculating an initial similarity matrix based on the image region representation data and the semantic feature vectors, and performing dynamic association capture; weighting and normalizing the initial similarity matrix based on the dynamic association to obtain a preliminary weight set; identifying and locating local feature interference regions in the preliminary weight set, optimizing the weight distribution based on the local feature interference regions to obtain an optimized weight set; and performing feature fusion on the image region representation data and the text entity description sets based on the optimized weight set to extract modal complementarity information, and performing depth mapping processing on the modal complementarity information to obtain a multimodal feature set. The multimodal feature set is filtered according to a preset threshold to obtain feature pairs that meet the fine-grained matching requirements, and the final matching result between the image region and the text entity is determined.

2. The multimodal feature matching method based on a large artificial intelligence model according to claim 1, characterized in that, The process of acquiring the original image, dividing the original image into regions, extracting visual description vectors for each region, and normalizing the visual description vectors to obtain image region representation data includes: acquiring the original image; generating a region mask matrix based on the original image using a semantic segmentation network; extracting independent region images from the original image based on the region mask matrix; calculating local feature sets for the independent region images; performing feature encoding mapping on the local feature sets to obtain visual description vectors; and normalizing the visual description vectors to obtain image region representation data.

3. The multimodal feature matching method based on a large artificial intelligence model according to claim 1, characterized in that, The process of acquiring descriptive text, performing entity recognition on the descriptive text, extracting a set of text entity descriptions, analyzing the modification relationships of the text entity descriptions, and determining semantic feature vectors by combining the image region representation data includes: acquiring descriptive text; performing sequence labeling on the descriptive text using a conditional random field to obtain a set of text entity descriptions; analyzing the modification relationships between entities in the text entity descriptions using dependency parsing to generate an initial entity vector matrix; aligning the initial entity vector matrix with the image region representation data using multimodal attention to obtain an alignment vector distribution; and performing fully connected layer mapping processing on the alignment vector distribution to obtain semantic feature vectors.

4. The multimodal feature matching method based on a large artificial intelligence model according to claim 1, characterized in that, The step of calculating an initial similarity matrix based on the image region representation data and the semantic feature vector, and performing dynamic association capture, and then weighting and normalizing the initial similarity matrix based on the dynamic association to obtain a preliminary weight set includes: calculating the projection overlap between the semantic feature vector and the image region representation data to obtain an initial similarity matrix; extracting spatial association features through convolution operations based on the neighborhood distribution of the initial similarity matrix; performing serial dependency analysis on the spatial association features to determine a dynamic association factor set; and performing weighted modulation and normalized exponential function operations on the initial similarity matrix based on the dynamic association factor set to obtain a preliminary weight set.

5. The multimodal feature matching method based on a large artificial intelligence model according to claim 1, characterized in that, The step of identifying and locating local feature interference regions in the preliminary weight set, and optimizing the weight distribution based on the local feature interference regions to obtain an optimized weight set includes: calculating the spatial distribution entropy of the preliminary weight set; locating local feature interference regions based on the spatial distribution entropy; generating an interference suppression factor matrix based on the local feature interference regions; and adjusting the weights of the preliminary weight set using the interference suppression factor matrix to obtain the optimized weight set.

6. The multimodal feature matching method based on a large artificial intelligence model according to claim 1, characterized in that, The step of fusing features of the image region representation data and the text entity description set according to the optimized weight set, extracting modal complementarity information, and performing deep mapping processing on the modal complementarity information to obtain a multimodal feature set includes: weighting the image region representation data and the text entity description according to the optimized weight set to obtain a weighted visual semantic vector and a weighted text embedding vector; constructing a cross-modal mapping relationship matrix based on the weighted visual semantic vector and the weighted text embedding vector to extract modal complementarity information; projecting the modal complementarity information onto the fusion feature space through a preset deep mapping network, and obtaining a joint representation vector through joint distribution alignment calculation; and performing feature reconstruction and principal component analysis dimensionality reduction processing on the joint representation vector to obtain a multimodal feature set.

7. The multimodal feature matching method based on a large artificial intelligence model according to claim 1, characterized in that, The step of filtering the multimodal feature set according to a preset threshold to obtain feature pairs that meet the fine-grained matching requirements and determining the final matching result between the image region and the text entity includes: calculating the similarity between the multimodal feature sets and constructing a feature similarity matrix; extracting index pairs greater than a preset global association threshold from the feature similarity matrix to obtain a candidate feature pair set; calculating the local alignment degree of the candidate feature pair set to obtain a fine-grained matching confidence score; filtering feature pairs in the candidate feature pair set whose fine-grained matching confidence score is higher than a preset fine-grained filtering threshold to obtain a target feature pair sequence; and constructing a bidirectional index table between the image region and the text entity based on the target feature pair sequence to obtain the final matching result.

8. The multimodal feature matching method based on a large artificial intelligence model according to claim 1, characterized in that, After filtering the multimodal feature set according to a preset threshold to obtain feature pairs that meet the fine-grained matching requirements and determining the final matching result between the image region and the text entity, the method further includes: comparing the final matching result with a preset real label to extract a set of mismatched samples; calculating a matching accuracy feedback value; if the matching accuracy feedback value is lower than a preset matching threshold, constructing a loss function based on the set of mismatched samples; using the loss function to perform inverse differentiation calculation on the optimized weight set to obtain gradient information; and updating the optimized weight set according to the gradient information to obtain an updated weight set.

9. A multimodal feature matching system based on a large artificial intelligence model, characterized in that, include: The visual feature representation module is used to acquire the original image, divide the original image into regions, extract the visual description vector of each region, and normalize the visual description vector to obtain image region representation data. The semantic feature extraction module is used to acquire descriptive text, perform entity recognition on the descriptive text, extract a set of text entity descriptions, analyze the modification relationships of the set of text entity descriptions, and determine the semantic feature vector by combining the image region representation data. The weight distribution acquisition module is used to calculate an initial similarity matrix based on the image region representation data and the semantic feature vector, and perform dynamic association capture. Based on the dynamic association, the initial similarity matrix is ​​weighted and normalized to obtain a preliminary weight set. The weight distribution optimization module is used to identify and locate local feature interference regions in the preliminary weight set, and optimize the weight distribution based on the local feature interference regions to obtain an optimized weight set. The modal feature acquisition module is used to perform feature fusion on the image region representation data and the text entity description set based on the optimized weight set, extract modal complementary information, and perform depth mapping processing on the modal complementary information to obtain a multimodal feature set. The filtering and matching output module is used to filter the multimodal feature set according to a preset threshold, obtain feature pairs that meet the fine-grained matching requirements, and determine the final matching result between the image region and the text entity.

Images