A method, system, terminal and storage medium for image-text matching based on uncertainty-aware multi-view fine-grained alignment

Abstract

The application relates to the technical field of information retrieval, and discloses a kind of based on uncertainty perception multi-view fine-grained alignment's picture-text matching method, system, terminal and storage medium, the method comprises: by freezing visual language big model to original image text pair is structured extraction, obtains structured scene graph;The structured scene graph is constrained, obtains multiple view variant images, and the view variant image is mapped, obtains regional level diagonal Gaussian distribution, the global Gaussian distribution representation is obtained by weighting aggregation to regional level diagonal Gaussian distribution;Global Gaussian distribution representation is selected by second-order Wasserstein distance ordering, obtains three-dimensional feature, and global Gaussian distribution representation is constrained, obtains structure matching score, and the retrieval ordering result is obtained by weighting fusion to three-dimensional feature and structure matching score.The application is based on scene graph extraction and weighted aggregation, realizes the accurate retrieval matching of the semantics between image and text.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and in particular to a method, system, terminal, and computer-readable storage medium for image and text matching based on uncertainty-aware multi-view fine-grained alignment. Background Technology

[0002] Early dual-encoder methods (such as VSE++ (Visual-Semantic Embedding++) and SCAN (Stacked Cross Attention Networks)) used global pooling features for similarity calculation, which was computationally efficient, but could not model fine-grained correspondences of local semantics. Subsequent works introduced the interaction between regions and lexical units to improve performance, such as Similarity Reasoning and Filtration (SGRAF) and Negative-Aware Attention Framework (NAAF). However, these methods still rely on local similarity scores and lack explicit constraints on the overall topological structure of objects, attributes, and relationships, resulting in insufficient fine-grained alignment and combinatorial semantic reasoning.

[0003] Mainstream methods represent both image regions and text phrases as deterministic point embeddings, ignoring the semantic ambiguity of multiple texts per image and multiple images per text in real-world data. Existing probabilistic embedding methods, such as Probabilistic Cross-Modal Embedding (PCME), Probabilistic Cross-Modal Embedding++ (PCME++), and Probabilistic Adapter for Frozen Vision-Language Models (ProbVLM), although incorporating Gaussian distributions, all model the uncertainty of the entire image at the global image level, failing to capture the differences in local semantic uncertainty between different regions. Their ability to characterize fine-grained ambiguity in complex scenes is limited. Secondly, false negatives lead to biased comparison optimization: although there are hard negative sample mining mechanisms, there is a lack of explicit identification and probability weighting of false negatives, resulting in a biased optimization direction and affecting the final semantic matching performance.

[0004] Existing image-text retrieval methods, such as dual encoders based on global pooling, only perform local optimizations during feature extraction or similarity calculation. They cannot explicitly model and distinguish semantic uncertainties at the region level during feature encoding. This makes it difficult to achieve accurate matching of structured semantics between images and text when facing fine-grained semantic scenarios that require precise differentiation of object-attribute-relationship topologies (such as the relationship reversal between dog chasing cat and cat chasing dog). This has become an urgent problem to be solved.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for image-text matching based on uncertainty-aware multi-view fine-grained alignment. This aims to solve the problem in the prior art that it is difficult to achieve accurate matching of structured semantics between images and text when facing fine-grained combined semantic scenarios that require precise differentiation of object-attribute-relationship topology.

[0007] To achieve the above objectives, the present invention provides a text-image matching method based on uncertainty-aware multi-view fine-grained alignment, the text-image matching method based on uncertainty-aware multi-view fine-grained alignment includes the following steps: Obtain the original image-text pairs, and perform structured extraction on the original image-text pairs using the frozen visual language large model to obtain a structured scene map; A candidate bounding box set is obtained, and the structured scene graph is constrained according to the candidate bounding box set to obtain multiple view variant images. All the view variant images are mapped through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation. The global Gaussian distribution representation is selected by second-order Wasserstein distance sorting to obtain three-dimensional features. The global mean vector of the global Gaussian distribution representation is constrained to obtain the structure matching score. The distribution similarity of the three-dimensional features and the structure matching score are weighted and fused to obtain the matching ranking result.

[0008] Optionally, the image-text matching method based on uncertainty-aware multi-view fine-grained alignment, wherein obtaining the original image-text pair and performing structured extraction on the original image-text pair by freezing the large visual language model to obtain a structured scene graph specifically includes: Obtain the original image text pair and the abnormal sample, and filter the original image text according to the abnormal sample to obtain the target image; The target image is structurally extracted by freezing the large visual language model to obtain entities, relationships, and attributes. The entities, relationships, and attributes are then fused to obtain a structured scene graph.

[0009] Optionally, the image-text matching method based on uncertainty-aware multi-view fine-grained alignment includes view variant images comprising visual variants, context variants, and style variants. The global Gaussian distribution representation includes a global mean vector and a global variance vector; The process of obtaining a candidate bounding box set, constraining the structured scene graph based on the candidate bounding box set to obtain multiple view variant images, mapping all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution, and weighting and aggregating the region-level diagonal Gaussian distribution based on the law of total variance through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation, specifically including: Obtain a confidence threshold, and perform detection on the target image based on the confidence threshold to obtain a set of candidate bounding boxes; The structured scene graph is constrained based on the candidate bounding box set to obtain visual variants, context variants, and style variants. Re-detection and fine-grained matching are performed on the visual variants, context variants, and style variants to obtain an enhancement candidate pool. The enhancement candidate pool is filtered based on random sampling to obtain positive samples of view variants, positive samples of context variants, and positive samples of style variants. By freezing the large visual language model, mean pooling is performed on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain visual region features; By freezing the mean projection layer and variance projection layer of the large visual language model, the visual region features are mapped to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated using a scaling dot product attention mechanism to obtain the global mean vector and the global variance vector.

[0010] Optionally, in the image-text matching method based on uncertainty-aware multi-view fine-grained alignment, the visual region features include appearance features, spatial location features, and high-level semantic embedding features. The step of freezing the large visual language model and performing mean pooling on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain visual region features specifically includes: By freezing the large visual language model, feature extraction is performed on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain a high-level semantic feature vector. By freezing the visual language big model, feature encoding is performed on the high-level semantic feature vector to obtain appearance features, spatial location features, and high-level semantic embedding features.

[0011] Optionally, the image-text matching method based on uncertainty-aware multi-view fine-grained alignment, wherein selecting the global Gaussian distribution representation by second-order Wasserstein distance sorting to obtain three-dimensional features, constraining the global mean vector of the global Gaussian distribution representation to obtain a structure matching score, and weightedly fusing the distribution similarity of the three-dimensional features and the structure matching score to obtain a matching ranking result, specifically includes: The global mean vector and global variance vector are selected by sorting by second-order Wasserstein distance to obtain a difficult negative sample pool. The three-dimensional features are then extracted from the difficult negative sample pool. By constraining the global mean vector, multiple graph node features are obtained; Based on all the graph node features, a visual graph and a text graph are determined, and the visual graph and the text graph are structurally matched to obtain a structural matching score. The distribution similarity of the three-dimensional features and the structural matching score are weighted and fused according to a preset fusion coefficient to obtain the matching ranking result.

[0012] Optionally, the image-text matching method based on uncertainty-aware multi-view fine-grained alignment, wherein determining the visual image and text image according to the features of all the graph nodes, and performing structural matching on the visual image and the text image to obtain a structural matching score, specifically includes: Based on the features of all the graph nodes, a visual graph and a text graph are determined. The visual graph and the text graph are extracted to obtain a target feature vector. The target feature vector is then mapped to obtain a diagonal Gaussian probability distribution. The diagonal Gaussian probability distribution is aggregated using an attention pooling mechanism to obtain a structure matching score.

[0013] Optionally, the image-text matching method based on uncertainty-aware multi-view fine-grained alignment, wherein the step of selecting the global mean vector and global variance vector representation by second-order Wasserstein distance sorting to obtain a difficult negative sample pool, extracting the three-dimensional features from the difficult negative sample pool, further includes: The three-dimensional features are predicted by freezing the MLP layer of the visual language large model to obtain the true negative probability. The true negative probability is then divided according to a preset score threshold to obtain sparse pseudo-labels. Based on the sparse pseudo-labels, the confidence scores of the true negative probabilities are weighted to obtain the contrast loss, annealing KL regularization loss, confidence supervision loss, and graph structure ranking loss. The optimal visual language large model is obtained by backpropagating the GSMN parameters and projection layer parameters of the frozen visual language large model based on the contrast loss, the annealing KL regularization loss, the confidence supervision loss, and the graph structure ranking loss.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a text-image matching system based on uncertainty-aware multi-view fine-grained alignment, wherein the text-image matching system based on uncertainty-aware multi-view fine-grained alignment is as follows: The feature structure extraction module is used to obtain the original image text pairs, and to extract the structure of the original image text pairs by freezing the visual language big model to obtain a structured scene map; The image semantic extraction module is used to obtain a set of candidate bounding boxes, constrain the structured scene graph according to the set of candidate bounding boxes, obtain multiple view variant images, map all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution, and based on the law of total variance, perform weighted aggregation on the region-level diagonal Gaussian distribution through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation; The similarity retrieval module is used to select the global Gaussian distribution representation by sorting it by second-order Wasserstein distance to obtain three-dimensional features, constrain the global mean vector of the global Gaussian distribution representation to obtain a structure matching score, and perform weighted fusion of the distribution similarity of the three-dimensional features and the structure matching score to obtain the matching ranking result.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a text-image matching program based on uncertainty-aware multi-view fine-grained alignment, and when the text-image matching program based on uncertainty-aware multi-view fine-grained alignment is executed by a processor, it implements the steps of the text-image matching method based on uncertainty-aware multi-view fine-grained alignment as described above.

[0016] This invention acquires original image-text pairs, performs structured extraction on these pairs using a frozen visual language model, and obtains a structured scene graph. A candidate bounding box set is then acquired, and the structured scene graph is constrained based on this set to obtain multiple view variant images. All view variant images are mapped using the mean projection layer and variance projection layer of the frozen visual language model to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated using a scaling dot product attention mechanism to obtain a global Gaussian distribution representation. This global Gaussian distribution representation is then selected using second-order Wasserstein distance sorting to obtain three-dimensional features. The global mean vector of the global Gaussian distribution representation is constrained to obtain a structure matching score. The distribution similarity of the three-dimensional features and the structure matching score are weighted and fused to obtain a matching ranking result. This invention, based on scene graph extraction and weighted aggregation, achieves accurate retrieval and matching of structured semantics between images and text. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of the image-text matching method based on uncertainty-aware multi-view fine-grained alignment of the present invention; Figure 2 This is a schematic diagram of the overall framework of UMFA, a preferred embodiment of the image-text matching method based on uncertainty-aware multi-view fine-grained alignment of the present invention. Figure 3 This is a comparative schematic diagram of a preferred embodiment of the image-text matching method based on uncertainty-aware multi-view fine-grained alignment of the present invention; Figure 4 This is a schematic diagram of rejection curve analysis on MSCOCO 5K, a preferred embodiment of the image-text matching method based on uncertainty-aware multi-view fine-grained alignment of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the image and text matching system based on uncertainty-aware multi-view fine-grained alignment of the present invention; Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the device of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] Existing image-text retrieval methods, such as dual encoders based on global pooling, only perform local optimizations during feature extraction or similarity calculation. They cannot explicitly model and distinguish semantic uncertainties at the region level during feature encoding. This makes it difficult to achieve accurate matching of structured semantics between images and text when faced with fine-grained semantic scenarios that require precise differentiation of object-attribute-relationship topologies (such as determining the relationship reversal between a dog chasing a cat and a cat chasing a dog). Therefore, there is a need for an image-text matching method based on uncertainty-aware multi-view fine-grained alignment, which can achieve accurate retrieval and matching of structured semantics between images and text based on scene graph extraction and weighted aggregation.

[0020] The preferred embodiment of the image-text matching method based on uncertainty-aware multi-view fine-grained alignment described in this invention, such as... Figure 1 and Figure 2 As shown, the image-text matching method based on uncertainty-aware multi-view fine-grained alignment includes the following steps: Step S10: Obtain the original image-text pair, and perform structured extraction on the original image-text pair by freezing the visual language big model to obtain a structured scene map.

[0021] Step S10 includes: Step S11: Obtain the original image text pair and abnormal samples, and filter the original image text according to the abnormal samples to obtain the target image; Step S12: Extract the target image into a structured form by freezing the visual language big model to obtain entities, relationships and attributes. Then, fuse the entities, relationships and attributes to obtain a structured scene graph.

[0022] Specifically, the original image text pairs and abnormal samples are obtained. Based on the abnormal samples, the original image text is filtered (filtering low-quality detection results, including missed detections (e.g., not detecting the kitten in the image), false detections (e.g., misidentifying the background area as a dog), and inaccurate localization (e.g., the bounding box only partially covers the target instead of completely covering it) (average filtering rate 12.3%)), resulting in the target image ((Image A, 0.92), (Image B, 0.85), (Image C, 0.71), ...). This is then processed by freezing the large visual language model (calling...). The Qwen-VL large model is frozen to perform structured extraction on the target image, obtaining entities (such as object nouns like "cat", "dog", "grass", etc.), relationships (such as verbs describing the interaction between objects like "chasing" and "located"), and attributes (such as adjectives modifying objects like "black" and "color"). The entities, relationships, and attributes are then fused to obtain a structured scene graph (represented in the form of a directed graph, with nodes containing entity nodes such as "cat" and "dog", attribute nodes such as "black" and "color", and relationship edge nodes such as "chasing" and "located").

[0023] In this embodiment, abnormal samples are filtered out (mainly including four categories: (1) incorrectly labeled samples, such as a dog in the figure but the label is mistakenly written as a cat; (2) severely occluded samples, such as a cat that is completely occluded and only its tail is visible; (3) blurry / low-resolution samples, which are low-quality data that are almost impossible to identify; (4) irrelevant noise samples, such as mixed-in face images).

[0024] For example, such as Figure 2 As shown, the graph consists of three core components: scene graph-guided synthesis enhancement, uncertainty-aware contrast alignment, and graph structure relation constraints. Given an image-text pair (I, T), the model first extracts a structured semantic representation, then generates multi-granularity image variants to expand the coverage of hard negative samples. The uncertainty learning module maps the representation to a diagonal Gaussian distribution space, performs fine-grained alignment under the Wasserstein distance metric, and reduces false penalties through a false negative sample probability estimation branch. The graph structure module suppresses the phenomenon of local similarity but mismatched relationships by explicitly constraining the consistency of object-attribute-relationship. In the inference phase, a fixed fusion coefficient is used for bidirectional retrieval ranking. Figure 2 The meanings of the symbols in the text are as follows: To compare losses, For KL divergence annealing regularization loss, For graph structure sorting loss, Supervision loss for false label confidence; This represents the distribution similarity score calculated based on Wassstein distance. This represents the text-image cosine similarity score from CLIP (Contrastive Language-Image Pre-training). Indicates the entity overlap rate between the query text and the candidate text. This represents the true negative sample probability predicted by the MLP (Multilayer Perceptron). Adaptive weighting coefficients for comparison loss; Represents the parameterized Gaussian distribution of each region; Figure 2 In this context, the negative sample distance blocking refers to the Wasserstein distance (a differentiable distance function used to measure the difference between two Gaussian distributions).

[0025] Step S20: Obtain a candidate bounding box set, constrain the structured scene graph according to the candidate bounding box set, and obtain multiple view variant images. Map all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation.

[0026] Step S20 includes: Step S21: Obtain the confidence threshold, and detect the target image according to the confidence threshold to obtain a set of candidate bounding boxes; Step S22: Constrain the structured scene graph according to the candidate bounding box set to obtain visual variants, context variants and style variants. Perform re-detection and fine-grained matching on the visual variants, context variants and style variants to obtain an enhancement candidate pool. Filter the enhancement candidate pool according to random sampling to obtain positive samples of view variants, positive samples of context variants and positive samples of style variants. Step S23: By freezing the large visual language model, mean pooling is performed on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain visual region features; Step S24: Map the visual region features by freezing the mean projection layer and variance projection layer of the large visual language model to obtain a region-level diagonal Gaussian distribution; Step S25: Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated through a scaling dot product attention mechanism to obtain the global mean vector and the global variance vector.

[0027] Specifically, a confidence threshold is obtained, and the target image is detected based on the confidence threshold to obtain a candidate bounding box set (Batch calling GroundingDINO, performing open set object detection with entity names as prompts, and outputting a candidate bounding box set with confidence (such as the bounding rectangle coordinates output for each target object detected in the image, labeled with the spatial range of "cat" [x1,y1,x2,y2]), filtering out boxes with confidence scores below 0.3), and the structured scene graph is constrained based on the candidate bounding box set to obtain visual variants, context variants, and style variants (Batch calling GLIGEN (a localization-based language-image generation model, GroundingDINO) based on the three types of prompts). Language-ImageGeneration generates variant images. Re-detection and fine-grained matching are performed on the visual variants, context variants, and style variants to obtain an enhancement candidate pool (samples with CLIP similarity below 0.7 are filtered, and re-detection and fine-grained matching are performed on the visual variants, context variants, and style variants to construct the enhancement candidate pool). The enhancement candidate pool is then filtered based on random sampling to obtain positive samples of viewpoint variants, context variants, and style variants (during training, one variant is randomly sampled from each anchor point as an additional positive sample, including positive samples of viewpoint variants, context variants, and style variants). The positive samples of viewpoint variants, context variants, and style variants are then averaged and pooled using the frozen visual language model to obtain visual region features (the last patch is extracted using the frozen CLIP ViT-B / 32). The token is used to perform mean pooling on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant, based on the token subset within the bounding box, to obtain visual region features. These visual region features are then mapped through the mean projection layer and variance projection layer of the frozen visual language model to obtain a region-level diagonal Gaussian distribution (subsequently, through learnable mean projection layers and variance projection layers, the region features and phrase features are mapped to region-level diagonal Gaussian distributions respectively). Based on the law of total variance (using the law of total variance to calculate the global mean and global variance), scaling the dot product attention... The force mechanism performs weighted aggregation on the region-level diagonal Gaussian distribution to obtain a global mean vector and a global variance vector (the global mean vector and global variance vector are calculated by weighted aggregation of the distribution parameters of each region through a scaling dot product attention mechanism and using the law of total variance). The view variant image (i.e., a semantically preserved but visually different derivative image generated for the same original image, i.e., a visual variant image represents a derivative image with the same semantics but different visuals) includes visual variants, context variants, and style variants. The global Gaussian distribution representation includes a global mean vector and a global variance vector.

[0028] In this embodiment, positive samples are generated from different shooting angles, such as "orange cat lying on the windowsill from the front" and "orange cat lying on the windowsill from the side" and "orange cat lying on the windowsill from above". Positive samples are generated from different shooting angles, such as "orange cat lying on the windowsill" and "orange cat lying on the grass" when the original image is "orange cat lying on the windowsill". Positive samples are generated from different backgrounds, such as "orange cat lying on the desk" and "orange cat lying on the grass" when the original image is "orange cat realistic photo". Positive samples are generated from different styles, such as "orange cat cartoon illustration" and "orange cat watercolor painting". During training, each original image is randomly sampled from one of the above three categories as its additional positive sample, so that the model learns a visually invariant representation of the same query text.

[0029] As an example, the aggregation of data into a global Gaussian distribution using attention pooling refers to the aggregation of the Gaussian distributions of various regions in an image into a unified two-parameter probability model through the attention pooling mechanism. This model specifically includes two core components: one is the global mean vector. The overall semantic center of the image in the embedding space (e.g., the image of a "black dog chasing a cat") represents the overall semantic center of the image. The vector points to a comprehensive semantic region such as "animal interaction" and "outdoor scene"; the second is the global variance vector. (Or a diagonal covariance matrix), encoding the magnitude of uncertainty in each dimension (e.g., higher color variance in the "cat" region because cat fur color varies, lower texture variance in the grass region because grass texture is relatively stable), used to adaptively reduce gradient weights for high-uncertainty samples (i.e., semantically ambiguous, potentially ambiguous hard examples) in contrastive learning. The entire image is ultimately modeled as... It uses probability distributions in the form of a graph rather than deterministic points, thus naturally expressing the inherent ambiguity of a single graph having multiple meanings.

[0030] Step S23 includes: Step S231: Extract features from the viewpoint variant positive samples, the context variant positive samples, and the style variant positive samples by freezing the visual language big model to obtain a high-level semantic feature vector; Step S232: Encode the high-level semantic feature vector by freezing the visual language big model to obtain appearance features, spatial location features and high-level semantic embedding features.

[0031] Specifically, by freezing the visual language big model, features are extracted from the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain high-level semantic feature vectors (referring to high-level semantic feature vectors extracted from a specific region of interest (ROI) in the image by object detection). The high-level semantic feature vectors are then encoded by freezing the visual language big model to obtain appearance features, spatial location features, and high-level semantic embedding features.

[0032] In this embodiment, the high-level semantic feature vector comprehensively encodes three types of information: first, appearance features, such as the color texture and shape outline of objects within the region (taking the region where the "cat" is located as an example, including visual appearances such as orange fur, white belly, round face and pointed ears); second, spatial location features, namely the coordinate position of the region in the image and the size of the bounding box (representing the orientation information of the object in the upper left / lower right of the image); and third, high-level semantic embedding features, such as the vector representation mapped to the shared space after CLIP encoding (capturing the abstract semantic concepts such as "cat", "animal", and "pet" corresponding to the region). Taking the image of "black dog chasing cat" as an example, the visual region features extracted from the bounding box where the cat is located are a comprehensive vector that simultaneously covers the above-mentioned appearance, spatial and semantic information; the text side encodes and represents each phrase independently.

[0033] Step S30: Select the global Gaussian distribution representation by sorting the second-order Wasserstein distance to obtain three-dimensional features. Constrain the global mean vector of the global Gaussian distribution representation to obtain the structure matching score. Weightedly fuse the distribution similarity of the three-dimensional features and the structure matching score to obtain the matching ranking result.

[0034] Step S30 includes: Step S31: Select the global mean vector and global variance vector representation by sorting by second-order Wasserstein distance to obtain a difficult negative sample pool; extract the three-dimensional features from the difficult negative sample pool. Step S32: Constrain the global mean vector to obtain multiple graph node features; Step S33: Determine the visual graph and text graph based on all the graph node features, perform structural matching on the visual graph and the text graph, and obtain the structural matching score; Step S34: Perform weighted fusion of the distribution similarity of the three-dimensional features and the structure matching score according to the preset fusion coefficient to obtain the matching ranking result.

[0035] Specifically, after step S31, the method further includes: predicting the three-dimensional features by freezing the MLP layer of the visual language large model to obtain the true negative probability; dividing the true negative probability according to a preset score threshold to obtain sparse pseudo-labels; calculating the confidence score of the true negative probability based on the sparse pseudo-labels to obtain the contrast loss, annealing KL regularization loss, confidence supervision loss, and graph structure ranking loss; and adjusting the GSMN (Gated Semantic Memory) of the frozen visual language large model according to the contrast loss, the annealing KL regularization loss, the confidence supervision loss, and the graph structure ranking loss. The parameters of the gated semantic memory network (Gated Semantic Memory Network) and the projection layer parameters are backpropagated to obtain the optimal visual language large model. The global mean vector and global variance vector are selected by second-order Wasserstein distance ranking to obtain a difficult negative sample pool. Three-dimensional features are extracted from this pool (for each training sample, within a batch, the 32 closest text candidates are selected by Wasserstein distance ranking as the difficult negative sample pool; for each candidate in the pool, three-dimensional features are extracted; these three-dimensional features represent the multi-dimensional correlation at the semantic level, but do not represent spatial meaning). The global mean vector is constrained to obtain multiple graph node features (referring to the feature information carried by each node in the scene graph structure; for entity nodes such as "..."). The feature of "cat" includes the visual appearance features and semantic embedding features of the object. For relation edge nodes such as "chasing", the feature encodes the type and direction information of the relation. Based on the features of all the graph nodes, a visual graph and a text graph are determined. The visual graph and the text graph are structurally matched to obtain a structural matching score (the sum of the structural similarity scores in the two directions from image to text and from text to image is calculated as the structural matching score). The distribution similarity of the three-dimensional features and the structural matching score are weighted and fused according to a preset fusion coefficient to obtain the matching ranking result. Furthermore, the GSMN parameters are optimized end-to-end through bidirectional triplet ranking loss (with a fixed interval value) so that the graph structure score of the matched image-text pair is higher than that of the unmatched pair, and cross-modal topological consistency constraints are applied.

[0036] For example, such as Figure 3 As shown, Figure 3 (a) indicates that, taking the image of "dog chasing cat" and the text "cat chasing dog" as an example, deterministic embedding makes the similarity between the two as high as 0.95. The relation-reversed samples cannot be effectively distinguished in the feature space. The difficult negative sample pushing mechanism actually pushes semantically similar synonyms away incorrectly, destroying the embedding space structure. Figure 3(b) in the example explicitly models the object-attribute-relationship topology using scene graphs, effectively distinguishing relation-reversal scenes. Gaussian distribution embeddings represent semantic uncertainty, and synonymous concepts are naturally aligned through distribution overlap. Simultaneously, scene graph-guided multi-view synthesis generates multiple semantically consistent variant samples from a single image, expanding the coverage of combined relationships in the training data. Furthermore, as... Figure 4 As shown, Figure 4 In the figure, (a) represents the image-to-text recall rate. Figure 4 In the figure, (b) represents the text-to-image recall rate. Figure 4 In this context, (c) represents the sum of all recall rates. Figure 4 After gradually removing high-uncertainty samples, it exhibits a steep performance gain curve, significantly outperforming the pseudo-uncertainty baseline and the random rejection scheme, verifying that the learned variance effectively characterizes the sample difficulty.

[0037] In this embodiment, the MLP classifier first outputs a confidence score for each candidate as a "true negative sample" based on three-dimensional features: distribution similarity, text similarity, and entity overlap rate. (For example, for the candidate image "white cat sleeping" in the query "black dog chasing cat", the model outputs a low score of 0.12 after combining the three features, judging it as a true negative sample; while for the candidate "cat lying on the grass", it outputs a high score of 0.89, judging it as a suspected positive sample). Then, an adaptive threshold is set based on the validation set quantile (e.g., the top 30% quantile of the validation set prediction score is taken as the cutoff). Only candidates exceeding the threshold are given pseudo-labels (e.g., "cat lying on the grass" is marked as a +1 pseudo-positive sample and participates in the contrastive loss calculation, while "white cat sleeping" is not marked and does not participate in this round of gradient update because it is below the threshold). Thus, "sparse but accurate" false negative sample screening is achieved without manual review of each item. The normalized weighting coefficient is calculated by multiplying the confidence index and the difficulty term, and the contrastive loss is weighted accordingly.

[0038] As an example, the joint loss calculation and backpropagation: calculate the weighted contrastive loss, annealing KL regularization loss, confidence supervision loss, and graph structure ranking loss, and jointly backpropagate to update the GSMN parameters and projection layer parameters (CLIP backbone is kept frozen), using the AdamW optimizer (learning rate 1e-5, cosine decay, batch size 128, 4 A100 GPUs, 30 epochs in total).

[0039] Step S33 includes: Step S331: Determine the visual graph and text graph based on all the graph node features, extract the visual graph and text graph to obtain the target feature vector, and map the target feature vector to obtain a diagonal Gaussian probability distribution; Step S332: Aggregate the diagonal Gaussian probability distribution through attention pooling mechanism to obtain the structure matching score.

[0040] Specifically, a visual graph and a text graph are determined based on the features of all the graph nodes. The visual graph and the text graph are extracted to obtain target feature vectors. The target feature vectors are mapped to obtain a diagonal Gaussian probability distribution (the feature vectors of each visual region and the vectors of each text phrase are mapped to diagonal Gaussian probability distributions respectively). The diagonal Gaussian probability distributions are aggregated through an attention pooling mechanism to obtain a structure matching score (the distribution similarity and graph matching score are calculated, i.e., the structure matching score).

[0041] In this embodiment, Gaussian distribution representations are extracted from the query image and text. Through learnable mean projection layers and variance projection layers, the feature vectors of each visual region and the vectors of each text phrase, i.e., the target feature vectors, are mapped to a diagonal Gaussian probability distribution. This distribution is uniquely determined by two sets of parameters: the mean vector. (such as the "cat" area) (pointing to the semantic direction of "feline / furry / small pet" in the embedding space) and the diagonal variance vector (Record the magnitude of uncertainty in each dimension; taking the image of a "black dog chasing a tabby cat" as an example: the "tabby cat" area varies in appearance...) Higher, "grassland" areas are characterized by their uniform texture. Lower, the "pursuit" relationship is abstract The highest Gaussian distribution is obtained by weighting and aggregating the Gaussian distributions of all regions into a global Gaussian distribution for the entire image using an attention pooling mechanism. The distribution similarity and graph matching score are calculated. The bidirectional search Top-K ranking results are output according to the weighted fusion scores (coefficients 0.7 and 0.3).

[0042] As an example, the method (UMFA) was validated on two tasks: image-to-text retrieval and fine-grained understanding. The main retrieval experiment reported the I2T / T2I Recall@1 and Rsum on the MSCOCO 5K subset. Table 1 provides a comparison under two typical backbones: Under the ResNet-101 region feature setting, compared with DivEmbed, CUSA, etc., UMFA achieved better results in I2T R@1, T2IR@1, and Rsum (e.g., I2T (image-to-text retrieval direction) R@1 (Top-1 recall) reached 63.1, and Rsum reached 448.1), indicating that it can still improve the overall alignment and cumulative retrieval metrics under the region encoding framework. With CLIP, compared to baselines such as CLIP fine-tuning and PCME++, I2TR@1 is further improved to 68.5, T2IR@1 to 52.4, and Rsum to 471.1. This indicates that under the unified vision-language encoding and the introduced multi-module constraints, the main retrieval performance has the advantage of repeatable observation compared to similar methods.

[0043] Specifically, the performance comparison of image and text retrieval is shown in Table 1: Table 1: Comparison of Image and Text Search Performance

[0044] In this embodiment, Table 2 compares the model's ability to grasp fine-grained semantics from multiple dimensions on fine-grained diagnostic sets such as ARO. It is evident that large-parameter visual language or generative models may still perform poorly on relational metrics even with significantly more parameters; however, this invention achieves high scores on relations, attributes, and average terms with a trainable scale of approximately 151 million models. This demonstrates that the UMFA method does not rely solely on increasing model size, but rather achieves verifiable gains in fine-grained semantic tasks through mechanisms such as structured alignment and difficult examples / graph structures, thus supporting the invention's objective of text-image meaning matching.

[0045] Furthermore, the results of the fine-grained diagnostic tests are shown in Table 2: Table 2: Results of Fine-Grained Diagnostic Tests

[0046] In this embodiment, Table 3 shows the module-level ablation of the present invention on MSCOCO (w / o indicates the variant after removing the corresponding sub-module in the complete process), and uniformly reports bidirectional R@1 / 5 / 10 and Rsum. Compared with the complete model (I2T R@1 68.5, Rsum 471.1), when the generation branch (w / oSyn), Gaussian / true negative class correlation constraints (w / o Gauss+TNC), and graph structure matching (w / o Graph) are removed in sequence, the Rsum decreases to 468.6, 467.2, and 464.5 respectively, and the Recall of each item shows a consistent decline. This result indicates that each sub-module makes a positive contribution to the overall framework; among them, the removal of graph-related components brings a relatively larger performance degradation, which is consistent with the technical route of incorporating object relationship / structural information into cross-modal alignment in this paper. The experiment confirms the correspondence between the corresponding technical features and the overall technical effect.

[0047] Specifically, the module-level ablation is shown in Table 3: Table 3: Module-level Ablation

[0048] Furthermore, such as Figure 5 As shown, based on the above-mentioned image-text matching method based on uncertainty-aware multi-view fine-grained alignment, the present invention also provides an image-text matching system based on uncertainty-aware multi-view fine-grained alignment, wherein the image-text matching system based on uncertainty-aware multi-view fine-grained alignment includes: The feature structure extraction module 51 is used to obtain the original image text pair, and to extract the structure of the original image text pair by freezing the visual language big model to obtain the structured scene map; Image semantic extraction module 52 is used to obtain a set of candidate bounding boxes, constrain the structured scene graph according to the set of candidate bounding boxes, obtain multiple view variant images, map all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution, and based on the law of total variance, perform weighted aggregation on the region-level diagonal Gaussian distribution through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation; The similarity retrieval module 53 is used to select the global Gaussian distribution representation by sorting by second-order Wasserstein distance to obtain three-dimensional features, constrain the global mean vector of the global Gaussian distribution representation to obtain a structure matching score, and perform weighted fusion of the distribution similarity of the three-dimensional features and the structure matching score to obtain a matching ranking result.

[0049] Furthermore, such as Figure 6 As shown, based on the above-mentioned image and text matching method and system based on uncertainty-aware multi-view fine-grained alignment, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0050] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a text-image matching program 40 based on uncertainty-aware multi-view fine-grained alignment, which can be executed by the processor 10 to implement the text-image matching method based on uncertainty-aware multi-view fine-grained alignment in this application.

[0051] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the image-text matching method based on uncertainty-aware multi-view fine-grained alignment.

[0052] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminals communicate with each other via a system bus.

[0053] In one embodiment, when processor 10 executes the text-image matching program 40 based on uncertainty-aware multi-view fine-grained alignment in memory 20, the following steps are performed: Obtain the original image-text pairs, and perform structured extraction on the original image-text pairs using the frozen visual language large model to obtain a structured scene map; A candidate bounding box set is obtained, and the structured scene graph is constrained according to the candidate bounding box set to obtain multiple view variant images. All the view variant images are mapped through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation. The global Gaussian distribution representation is selected by second-order Wasserstein distance sorting to obtain three-dimensional features. The global mean vector of the global Gaussian distribution representation is constrained to obtain the structure matching score. The distribution similarity of the three-dimensional features and the structure matching score are weighted and fused to obtain the matching ranking result. Specifically, obtaining the original image-text pairs, and performing structured extraction on the original image-text pairs using a frozen visual language model to obtain a structured scene graph, includes: Obtain the original image text pair and the abnormal sample, and filter the original image text according to the abnormal sample to obtain the target image; The target image is structurally extracted by freezing the large visual language model to obtain entities, relationships, and attributes. The entities, relationships, and attributes are then fused to obtain a structured scene graph.

[0054] The view variant images include visual variants, context variants, and style variants; The global Gaussian distribution representation includes a global mean vector and a global variance vector; The process of obtaining a candidate bounding box set, constraining the structured scene graph based on the candidate bounding box set to obtain multiple view variant images, mapping all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution, and weighting and aggregating the region-level diagonal Gaussian distribution based on the law of total variance through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation, specifically including: Obtain a confidence threshold, and perform detection on the target image based on the confidence threshold to obtain a set of candidate bounding boxes; The structured scene graph is constrained based on the candidate bounding box set to obtain visual variants, context variants, and style variants. Re-detection and fine-grained matching are performed on the visual variants, context variants, and style variants to obtain an enhancement candidate pool. The enhancement candidate pool is filtered based on random sampling to obtain positive samples of view variants, positive samples of context variants, and positive samples of style variants. By freezing the large visual language model, mean pooling is performed on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain visual region features; By freezing the mean projection layer and variance projection layer of the large visual language model, the visual region features are mapped to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated using a scaling dot product attention mechanism to obtain the global mean vector and the global variance vector.

[0055] The visual region features include appearance features, spatial location features, and high-level semantic embedding features; The step of freezing the large visual language model and performing mean pooling on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain visual region features specifically includes: By freezing the large visual language model, feature extraction is performed on the positive samples of the viewpoint variant, the positive samples of the context variant, and the positive samples of the style variant to obtain a high-level semantic feature vector. By freezing the visual language big model, feature encoding is performed on the high-level semantic feature vector to obtain appearance features, spatial location features, and high-level semantic embedding features.

[0056] Specifically, the process involves selecting the global Gaussian distribution representation using second-order Wasserstein distance ranking to obtain three-dimensional features, constraining the global mean vector of the global Gaussian distribution representation to obtain a structure matching score, and weightedly fusing the distribution similarity of the three-dimensional features and the structure matching score to obtain a matching ranking result. This process includes: The global mean vector and global variance vector are selected by sorting by second-order Wasserstein distance to obtain a difficult negative sample pool. The three-dimensional features are then extracted from the difficult negative sample pool. By constraining the global mean vector, multiple graph node features are obtained; Based on all the graph node features, a visual graph and a text graph are determined, and the visual graph and the text graph are structurally matched to obtain a structural matching score. The distribution similarity of the three-dimensional features and the structural matching score are weighted and fused according to a preset fusion coefficient to obtain the matching ranking result.

[0057] Specifically, the step of determining the visual graph and text graph based on the features of all the graph nodes, and performing structure matching on the visual graph and the text graph to obtain a structure matching score, includes: Based on the features of all the graph nodes, a visual graph and a text graph are determined. The visual graph and the text graph are extracted to obtain a target feature vector. The target feature vector is then mapped to obtain a diagonal Gaussian probability distribution. The diagonal Gaussian probability distribution is aggregated using an attention pooling mechanism to obtain a structure matching score.

[0058] The process includes selecting the global mean vector and global variance vector representations through second-order Wasserstein distance sorting to obtain a difficult negative sample pool, extracting three-dimensional features from the difficult negative sample pool, and then further including: The three-dimensional features are predicted by freezing the MLP layer of the visual language large model to obtain the true negative probability. The true negative probability is then divided according to a preset score threshold to obtain sparse pseudo-labels. Based on the sparse pseudo-labels, the confidence scores of the true negative probabilities are weighted to obtain the contrast loss, annealing KL regularization loss, confidence supervision loss, and graph structure ranking loss. The optimal visual language large model is obtained by backpropagating the GSMN parameters and projection layer parameters of the frozen visual language large model based on the contrast loss, the annealing KL regularization loss, the confidence supervision loss, and the graph structure ranking loss.

[0059] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a text-image matching program based on uncertainty-aware multi-view fine-grained alignment, and the text-image matching program based on uncertainty-aware multi-view fine-grained alignment, when executed by a processor, implements the steps of the text-image matching method based on uncertainty-aware multi-view fine-grained alignment as described above.

[0060] In summary, this invention provides a method, system, terminal, and storage medium for image-text matching based on uncertainty-aware multi-view fine-grained alignment. The method includes: acquiring original image-text pairs; performing structured extraction on the original image-text pairs using a frozen visual language large model to obtain a structured scene graph; acquiring a candidate bounding box set; constraining the structured scene graph based on the candidate bounding box set to obtain multiple view variant images; mapping all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution; weighting and aggregating the region-level diagonal Gaussian distribution based on the law of total variance using a scaling dot product attention mechanism to obtain a global Gaussian distribution representation; selecting from the global Gaussian distribution representation using second-order Wasserstein distance sorting to obtain three-dimensional features; constraining the global mean vector of the global Gaussian distribution representation to obtain a structure matching score; and weightedly fusing the distribution similarity of the three-dimensional features and the structure matching score to obtain a matching ranking result. This invention, based on scene graph extraction and weighted aggregation, achieves accurate retrieval and matching of structured semantics between images and text.

[0061] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal system that includes that element.

[0062] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0063] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A text-image matching method based on uncertainty-aware multi-view fine-grained alignment, characterized in that, The image-text matching method based on uncertainty-aware multi-view fine-grained alignment includes: Obtain the original image-text pairs, and perform structured extraction on the original image-text pairs using the frozen visual language large model to obtain a structured scene map; A candidate bounding box set is obtained, and the structured scene graph is constrained according to the candidate bounding box set to obtain multiple view variant images. All the view variant images are mapped through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the region-level diagonal Gaussian distribution is weighted and aggregated through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation. The global Gaussian distribution representation is selected by second-order Wasserstein distance sorting to obtain three-dimensional features. The global mean vector of the global Gaussian distribution representation is constrained to obtain the structure matching score. The distribution similarity of the three-dimensional features and the structure matching score are weighted and fused to obtain the matching ranking result. The process involves selecting the global Gaussian distribution representation using second-order Wasserstein distance ranking to obtain three-dimensional features, constraining the global mean vector of the global Gaussian distribution representation to obtain a structure matching score, and then weighting and fusing the distribution similarity of the three-dimensional features and the structure matching score to obtain a matching ranking result. Specifically, this includes: The global mean vector and global variance vector are selected by sorting by second-order Wasserstein distance to obtain a difficult negative sample pool. The three-dimensional features are then extracted from the difficult negative sample pool. By constraining the global mean vector, multiple graph node features are obtained; Based on all the graph node features, a visual graph and a text graph are determined, and the visual graph and the text graph are structurally matched to obtain a structural matching score. The distribution similarity of the three-dimensional features and the structural matching score are weighted and fused according to a preset fusion coefficient to obtain the matching ranking result; The view variant image refers to a derived image that has the same semantics but different visuals; The three-dimensional features represent the degree of multi-dimensional association at the semantic level.

2. The image-text matching method based on uncertainty-aware multi-view fine-grained alignment according to claim 1, characterized in that, The process of obtaining the original image-text pairs and then performing structured extraction on the original image-text pairs using a frozen visual language model to obtain a structured scene graph specifically includes: Obtain the original image-text pair and the abnormal sample; filter the original image-text pair according to the abnormal sample to obtain the target image. The target image is structurally extracted by freezing the large visual language model to obtain entities, relationships, and attributes. The entities, relationships, and attributes are then fused to obtain a structured scene graph.

3. The image-text matching method based on uncertainty-aware multi-view fine-grained alignment according to claim 2, characterized in that, The view variant images include visual variants, context variants, and style variants; The global Gaussian distribution representation includes a global mean vector and a global variance vector; The process of obtaining a candidate bounding box set, constraining the structured scene graph based on the candidate bounding box set to obtain multiple view variant images, mapping all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution, and weighting and aggregating the region-level diagonal Gaussian distribution based on the law of total variance through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation, specifically including: Obtain a confidence threshold, and perform detection on the target image based on the confidence threshold to obtain a set of candidate bounding boxes; The structured scene graph is constrained based on the candidate bounding box set to obtain visual variants, context variants, and style variants. Re-detection and fine-grained matching are performed on the visual variants, context variants, and style variants to obtain an enhancement candidate pool. The enhancement candidate pool is filtered based on random sampling to obtain positive samples of visual variants, context variants, and style variants. By freezing the large visual language model, mean pooling is performed on the positive samples of the visual variant, the positive samples of the context variant, and the positive samples of the style variant to obtain visual region features; By freezing the mean projection layer and variance projection layer of the large visual language model, the features of the visual region are mapped to obtain a region-level diagonal Gaussian distribution. Based on the law of total variance, the regional-level diagonal Gaussian distribution is weighted and aggregated through a scaling dot product attention mechanism to obtain the global mean vector and the global variance vector.

4. The image-text matching method based on uncertainty-aware multi-view fine-grained alignment according to claim 3, characterized in that, The visual region features include appearance features, spatial location features, and high-level semantic embedding features; The step of performing mean pooling on the positive samples of visual variants, the positive samples of context variants, and the positive samples of style variants by freezing the large visual language model to obtain visual region features specifically includes: By freezing the large visual language model, feature extraction is performed on the positive samples of the visual variant, the positive samples of the context variant, and the positive samples of the style variant to obtain a high-level semantic feature vector. By freezing the visual language big model, feature encoding is performed on the high-level semantic feature vector to obtain appearance features, spatial location features, and high-level semantic embedding features.

5. The image-text matching method based on uncertainty-aware multi-view fine-grained alignment according to claim 1, characterized in that, The step of determining the visual graph and text graph based on the features of all the graph nodes, and performing structural matching on the visual graph and the text graph to obtain a structural matching score, specifically includes: Based on the features of all the graph nodes, a visual graph and a text graph are determined. The visual graph and the text graph are extracted to obtain a target feature vector. The target feature vector is then mapped to obtain a diagonal Gaussian probability distribution. The diagonal Gaussian probability distribution is aggregated using an attention pooling mechanism to obtain a structure matching score.

6. The image-text matching method based on uncertainty-aware multi-view fine-grained alignment according to claim 5, characterized in that, The process involves selecting the global mean vector and global variance vector representations using second-order Wasserstein distance sorting to obtain a difficult negative sample pool, extracting three-dimensional features from the difficult negative sample pool, and then further including: The three-dimensional features are predicted by freezing the MLP layer of the visual language large model to obtain the true negative probability. The true negative probability is then divided according to a preset score threshold to obtain sparse pseudo-labels. Based on the sparse pseudo-labels, the confidence scores of the true negative probabilities are weighted to obtain the contrast loss, annealing KL regularization loss, confidence supervision loss, and graph structure ranking loss. The optimal visual language large model is obtained by backpropagating the GSMN parameters and projection layer parameters of the frozen visual language large model based on the contrast loss, the annealing KL regularization loss, the confidence supervision loss, and the graph structure ranking loss.

7. A text-image matching system based on uncertainty-aware multi-view fine-grained alignment, characterized in that, The image-text matching system based on uncertainty-aware multi-view fine-grained alignment is applied to the image-text matching method based on uncertainty-aware multi-view fine-grained alignment as described in any one of claims 1-6, wherein the image-text matching system based on uncertainty-aware multi-view fine-grained alignment comprises: The feature structure extraction module is used to obtain the original image text pairs, and to extract the structure of the original image text pairs by freezing the visual language big model to obtain a structured scene map; The image semantic extraction module is used to obtain a set of candidate bounding boxes, constrain the structured scene graph according to the set of candidate bounding boxes, obtain multiple view variant images, map all the view variant images through the mean projection layer and variance projection layer of the frozen visual language large model to obtain a region-level diagonal Gaussian distribution, and based on the law of total variance, perform weighted aggregation on the region-level diagonal Gaussian distribution through a scaling dot product attention mechanism to obtain a global Gaussian distribution representation; The similarity retrieval module is used to select the global Gaussian distribution representation by sorting it by second-order Wasserstein distance to obtain three-dimensional features, constrain the global mean vector of the global Gaussian distribution representation to obtain a structure matching score, and perform weighted fusion of the distribution similarity of the three-dimensional features and the structure matching score to obtain the matching ranking result.

8. A terminal, characterized in that, The terminal includes: a memory, a processor, and a text-image matching program based on uncertainty-aware multi-view fine-grained alignment stored in the memory and executable on the processor. When the text-image matching program based on uncertainty-aware multi-view fine-grained alignment is executed by the processor, it implements the steps of the text-image matching method based on uncertainty-aware multi-view fine-grained alignment as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a text-image matching program based on uncertainty-aware multi-view fine-grained alignment, which, when executed by a processor, implements the steps of the text-image matching method based on uncertainty-aware multi-view fine-grained alignment as described in any one of claims 1-6.

Images