Multi-view bridging fine-grained image-text cross-modal alignment method, device, equipment and medium

Abstract

The application relates to a multi-view bridging fine-grained image-text cross-modal alignment method, device, equipment and medium. The method comprises the following steps: fusing front and side view images to extract more abundant image information, mediating an intermediate state via a bridge to generate a modality gap that compensates for global feature alignment of images and texts, and finally comprehensively estimating image-text similarity through a global-local cross alignment similarity score. Compared with an existing single-view (front view) image-text cross-modal contrast alignment method, the method can compensate for a cross-modal gap by fusing more source image information, and can more fully and finely interact with inter-modal and intra-modal information, so that subsequent zero-shot image classification based on a category text prompt is more accurate.

Description

Technical Field

[0001] This application relates to the field of cross-modal processing technology of images and text, and in particular to a method, apparatus, device and medium for fine-grained cross-modal alignment of images and text using multi-view bridging. Background Technology

[0002] Image target recognition, with its advantages of non-invasiveness, high efficiency, and low cost, is widely used in fields such as security monitoring, autonomous driving, and industrial inspection. The visual information acquired is the core foundation for deep analysis such as target classification and semantic understanding. With the development of deep learning, cross-modal analysis of images and text has become a research hotspot, aiming to establish the association between visual features (such as shape and component structure) and text semantics (such as category and attribute description) to achieve tasks such as "image meaning recognition" and "text retrieval of images".

[0003] However, these technologies heavily rely on large-scale manually labeled data, requiring image category and bounding box annotations, as well as text semantic alignment annotations, which is costly and time-consuming. Moreover, in scenarios where annotations are scarce, such as rare targets (e.g., special industrial defects) and emerging tasks (e.g., novel scene detection), the model's generalization ability is insufficient, making it difficult to effectively associate cross-modal information and limiting practical applications.

[0004] To reduce reliance on annotation, zero-shot learning has become a core approach, with Visual-Language Pre-training (VLP) technology standing out. This technique leverages large-scale, general image-text pairs (such as publicly available internet data) to learn cross-modal associations without requiring additional annotations for specific tasks, providing a feasible solution for zero-shot / few-shot scenarios. Existing mainstream technologies integrate image-text features through feature fusion modules and optimize alignment performance using loss functions: for example, CLIP optimizes global image-text matching through contrastive learning and cosine similarity, driving the field's development; subsequent advancements such as ALIGN and BLIP have improved contrastive learning and enhanced local interactions to optimize alignment performance in general scenarios, validating the potential of VLP in cross-modal tasks.

[0005] However, existing technologies still suffer from two major shortcomings in complex scenarios: First, insufficient utilization of multi-view information. General recognition often faces multi-view data of targets (such as different shooting angles and poses). Existing technologies mostly design solutions for single views, failing to effectively model complementary relationships between views (such as frontal contours and side structures), and lacking information compensation when views are missing, resulting in insufficient stability of cross-modal associations. Second, poor cross-modal alignment. There are complex many-to-many relationships between local image features (such as car wheels and mobile phone screens) and multi-semantic descriptions in text (such as "red car wheel hubs"). Existing technologies mostly use global feature cosine similarity measurement, which makes it difficult to capture local and subdivided semantic correspondences and to bridge modal gaps, resulting in insufficient semantic matching and affecting the accuracy of associations. Summary of the Invention

[0006] Therefore, it is necessary to provide a method, apparatus, device, and medium for efficient cross-modal alignment of multi-view bridging and fine-grained text and image cross-modal alignment, which can address the above-mentioned technical problems.

[0007] A fine-grained cross-modal alignment method for multi-view bridging graphics and text, the method comprising: Obtain a training sample set, which includes multiple training pairs. Each training pair includes sample images of the same target obtained from two different perspectives and corresponding text information. The cross-modal feature extraction model is trained using each training pair in the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information in the training pair respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain fused image features. The text features and fused image features are projected into a shared space using a cross-modal bridge to obtain image intermediate state and text intermediate state. An intermediate state loss is constructed based on the image intermediate state and text intermediate state. Text reconstruction loss and image reconstruction loss are constructed based on the text features and text intermediate state, the fused image features and image intermediate state, respectively. The global-local cross-alignment unit is used to obtain the similarity score matrix between image and text and between text and image based on the text features and the fused image features, thereby obtaining the global-local cross-alignment loss; The cross-modal feature extraction model is trained based on the intermediate state loss, text reconstruction loss, image reconstruction loss, and global-local cross-alignment loss until convergence, thus obtaining the trained cross-modal feature extraction model. The target images and corresponding prompt texts are obtained from two different perspectives of the same target. The target images and prompt texts are then input into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

[0008] In one embodiment, the image encoder is a visual encoder that extracts global visual features and local visual features from the first viewpoint sample image and the second viewpoint sample image, respectively. The text encoder employs a target-related language model to extract global and local text features of the text information.

[0009] In one embodiment, before inputting the text information into the text encoder, the text information is further semantically cleaned and standardized using a large language model to obtain purified text information, which is then input into the text encoder.

[0010] In one embodiment, in the multi-view fusion unit: For visual features from two different perspectives, a view type encoding sequence and a missing state identifier sequence are constructed respectively. The view type encoding sequence is used to identify whether the visual feature belongs to a first perspective or a second perspective, and the missing state identifier sequence is used to identify whether there is a missing feature. The global and local visual features of the first and second view sample images are fused together to obtain a multi-view global feature sequence and a multi-view local feature sequence. The multi-view global feature sequence and the multi-view local feature sequence are concatenated and superimposed with the view type encoding sequence and the missing state identifier sequence of the corresponding view along the feature channel dimension to obtain the enhanced multi-view global feature sequence and the enhanced multi-view local feature sequence. Based on the enhanced multi-view global feature sequence and the enhanced multi-view local feature sequence, a multi-head self-attention mechanism is used to calculate the correlation weights between features from different perspectives and complete feature interaction. The features output by the attention mechanism are subjected to average pooling along the sequence length dimension to aggregate global information. The pooling result is input into a linear projection layer for dimension fine-tuning and feature mapping to obtain the fused image features, which include fused image global features and fused image local features.

[0011] In one embodiment, the cross-modal bridge is constructed from learnable vectors, and the image intermediate and text intermediate are represented as follows: ; ; In the above formula, and These are the global features and intermediate states of the fused image, respectively. and These refer to the global text features and the text intermediate state, respectively. express transpose, This indicates a cross-modal bridge.

[0012] In one embodiment, in the global-local cross-alignment unit: The global features and local features of the fused image are concatenated along the feature channel dimension to obtain globally enhanced local fused image features; The global text features and local text features are concatenated along the feature channel dimension to obtain globally enhanced local text features; The semantic association weights between the globally enhanced local fused image features and the global text features are calculated using a cross-modal attention mechanism to obtain the first interaction feature; The visual-semantic association weights between the globally enhanced local text features and the fused global image features are calculated using a cross-modal attention mechanism to obtain the second interaction feature; The first interaction feature and the second interaction feature are respectively passed through a multilayer perceptron, and after nonlinear mapping and dimension adjustment, the image-to-text similarity score matrix and the text-to-image similarity score matrix are obtained. The global-local cross-alignment loss is calculated based on the image-to-text similarity score matrix and the text-to-image similarity score matrix.

[0013] In one embodiment, after obtaining cross-modal aligned image features and text features, downstream tasks are performed based on the image features and text features; The prompt text is related to the downstream task.

[0014] This application also provides a multi-view bridging fine-grained cross-modal alignment device for graphics and text, the device comprising: The training sample set acquisition module is used to acquire a training sample set, which includes multiple training pairs. Each training pair includes sample images of the same target acquired from two different perspectives and corresponding text information. The feature extraction module is used to train the cross-modal feature extraction model using each training pair in the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information in the training pair respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain the fused image features. A bridge-mediated intermediate state generation module is used to project the text features and fused image features into a shared space using a cross-modal bridge to obtain image intermediate states and text intermediate states. An intermediate state loss is constructed based on the image intermediate states and text intermediate states. Text reconstruction loss and image reconstruction loss are constructed based on the text features and text intermediate states, the fused image features and image intermediate states, respectively. The global-local cross-alignment module is used to obtain the image-to-text and text-to-image similarity score matrices based on the text features and the fused image features using the global-local cross-alignment unit, thereby obtaining the global-local cross-alignment loss; The cross-modal feature extraction model training module is used to train the cross-modal feature extraction model based on the intermediate state loss, text reconstruction loss, image reconstruction loss and global-local cross-alignment loss until convergence, so as to obtain the trained cross-modal feature extraction model. The image-text cross-modal alignment module is used to acquire target images of the same target from two different perspectives, as well as corresponding prompt text. The target images and prompt text are then input into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described multi-view bridging fine-grained text-image cross-modal alignment method.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described multi-view bridging fine-grained cross-modal alignment method for graphics and text.

[0017] The aforementioned multi-view bridging fine-grained image-text cross-modal alignment method, apparatus, device, and medium train a cross-modal feature extraction model, including an image encoder, a text encoder, and a multi-view fusion unit, using training pairs from a training sample set. The image and text encoders process two sample images and text information from the training pairs, respectively, to obtain image features and text features from two different perspectives. The multi-view fusion unit then fuses these image features to obtain fused image features. A cross-modal bridge projects the text features and fused image features into a shared space, resulting in an image intermediate state and a text intermediate state. An intermediate state loss is constructed based on the image intermediate state and the text intermediate state. Finally, based on the text features and the text intermediate state, the fused image... This method constructs text reconstruction loss and image reconstruction loss based on image features and intermediate states. Simultaneously, a global-local cross-alignment unit is used to obtain image-to-text and text-to-image similarity score matrices based on text features and fused image features, thus yielding a global-local cross-alignment loss. The cross-modal feature extraction model is trained using the intermediate state loss, text reconstruction loss, image reconstruction loss, and global-local cross-alignment loss until convergence, resulting in a trained cross-modal feature extraction model. Target images and prompt text obtained from two different perspectives of the same object are input into the trained cross-modal feature extraction model to obtain cross-modal aligned image and text features, providing core feature support for subsequent downstream tasks without requiring additional modality adaptation processing. This method fully leverages multi-view fusion to extract multi-angle visual information of the target and uses multi-loss joint optimization to ensure modal feature integrity and alignment accuracy. It effectively solves the problems of feature representation bias caused by insufficient information from a single view and alignment deviation caused by the cross-modal gap, significantly improving the model's generalization performance in scenarios with scarce annotations (such as rare target recognition and novel perspective image analysis) and reducing reliance on large-scale manually annotated data. Meanwhile, the image features and text features after cross-modal alignment have high semantic consistency, which can be directly adapted to downstream tasks in multiple fields. This not only reduces the feature preprocessing cost and development cycle of downstream tasks, but also improves the inference efficiency and result accuracy of downstream tasks (such as medical image report generation and industrial defect annotation) through accurate modal association. It is suitable for complex real-world scenarios such as security monitoring, medical diagnosis, autonomous driving, and industrial inspection. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a multi-view bridging fine-grained cross-modal alignment method for text and images in one embodiment. Figure 2 This is a schematic diagram of a multi-view cross-modal training framework in one embodiment; Figure 3This is a schematic diagram of a large language model template instruction for purifying text based on report text and disease keywords in one embodiment. Figure 4 This is a structural block diagram of a multi-view bridging fine-grained image-text cross-modal alignment device in one embodiment; Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In this application, as Figure 1 As shown, a fine-grained cross-modal alignment method for multi-view bridging is provided, including the following steps: Step S100: Obtain a training sample set. The training sample set includes multiple training pairs. Each training pair includes sample images of the same target obtained from two different perspectives and corresponding text information.

[0021] Step S110: The cross-modal feature extraction model is trained using training pairs from the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information from the training pairs respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain the fused image features.

[0022] Step S120: The text features and the fused image features are projected into a shared space using a cross-modal bridge to obtain the image intermediate state and the text intermediate state. The intermediate state loss is constructed based on the image intermediate state and the text intermediate state. The text reconstruction loss and the image reconstruction loss are constructed based on the text features and the text intermediate state, the fused image features and the image intermediate state, respectively.

[0023] Step S130: Using the global-local cross-alignment unit, the similarity score matrix between image and text and between text and image is obtained based on the text features and the fused image features, thereby obtaining the global-local cross-alignment loss.

[0024] Step S140: Train the cross-modal feature extraction model based on intermediate state loss, text reconstruction loss, image reconstruction loss, and global-local cross-alignment loss until convergence, and obtain the trained cross-modal feature extraction model.

[0025] Step S150: Obtain target images of the same target from two different perspectives, along with corresponding prompt text. Input the target images and prompt text into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

[0026] To improve the accuracy and generalization ability of zero-shot image analysis tasks (such as target recognition, attribute retrieval, and scene understanding), this application proposes a pre-training framework that combines multi-view fusion and intermediate-state bridging for cross-modal alignment. The main reasons for the performance limitations of existing zero-shot image analysis methods are twofold: first, they lack sufficient utilization of multi-view information, and in real-world scenarios, there is a lack of effective information compensation mechanisms for common viewpoint gaps (such as view gaps due to specific shooting angles, poses, and lighting conditions); second, there is a cross-modal gap and semantic mismatch between general images (such as images of industrial parts, security monitoring images, and autonomous driving scene images) and text descriptions (such as target attribute descriptions and scene semantic descriptions), and single global feature alignment is insufficient to capture complex fine-grained correspondences (such as the relationship between local defects in parts and defect types in text descriptions, and the relationship between monitored target actions and behavioral details in text descriptions).

[0027] Based on this, this method achieves full mining of multi-view information and accurate alignment of cross-modal semantics through the collaborative work of three key technical modules. Specifically, this method includes: First, a multi-view fusion unit is constructed. This unit employs a state-aware mechanism: it utilizes a zero-padding placeholder and combines explicit missing states with view type embeddings. This design allows the model to dynamically switch its encoding strategy—co-aggregating complementary information when two views coexist, or using learnable state embeddings to compensate for missing modal biases when only a frontal view is available. Through this view generation and compensation mechanism, robust and comprehensive image representations can still be constructed even when side views are missing, effectively improving the model's ability to utilize multi-source information.

[0028] Secondly, a cross-modal knowledge bridge is designed to generate intermediate states of visual / textual information for global feature cross-modal alignment within a unified semantic space. This bridge, acting as a shared semantic dictionary, contains a set of learnable basic concept vectors, providing a unified interaction space for image and text features. Through joint optimization of feature reconstruction loss and contrastive learning loss, features from both modalities are aligned in the shared space, significantly narrowing the modal gap between visual and textual features.

[0029] Finally, a global-local cross-alignment mechanism is established. Based on a unified global semantic space optimized by a bridging mechanism, this mechanism calculates fine-grained interactions between the enhanced local features of the image and the global features of the text description, as well as between the enhanced local features of the text description and the global features of the image, through cross-attention calculation, generating a semantically rich fused representation. A lightweight projector then converts the fused features into a final similarity score, replacing the traditional single global similarity calculation, thus achieving accurate modeling of the complex correspondence between the target image and the report.

[0030] This method, through a progressive design of the three technical layers mentioned above, systematically solves the problems of insufficient utilization of multiple views and inadequate cross-modal alignment while maintaining the advantages of zero-shot learning. It only requires pre-training with large-scale image-text pairs, enabling rapid target classification or other downstream tasks by calculating the similarity between images and possible text categories based on simple textual descriptions without specific annotations. This significantly improves the model's generalization performance while maintaining convenient data acquisition.

[0031] like Figure 2 As shown, this paper proposes a multi-view cross-modal training framework under which the cross-modal feature extraction model is trained in an unsupervised manner.

[0032] In step S100, in order to adapt to different downstream task scenarios, the training pairs of the training sample set can be designed in various forms according to the needs of the application domain.

[0033] In one embodiment, the training sample set can be remote sensing image scene training pairs. For downstream tasks such as remote sensing target recognition and land cover classification, the sample images of the training pairs use dual-view remote sensing images of the same geographic target, and the text information is a professional semantic description of the geographic target. For example, the dual-view remote sensing images are "orthogonal view remote sensing image" and "30° oblique view remote sensing image" of a certain farmland area; if the target is an urban building area, it can be replaced with "zenith view remote sensing image" (showing the building roof layout and floor area ratio) and "45° side view remote sensing image" (showing the building facade structure, building height and surrounding green distribution). Corresponding text information: It needs to be combined with the professional attributes of the remote sensing target, including key information such as "target type, spatial characteristics, and state parameters". For example, the text of the farmland scene is "a winter wheat planting field in a certain region of a certain province. The plot area is about 2.3 acres from the orthogonal view. It is rectangular in shape and adjacent to a rural road on the east side. From the oblique view, the crop growth height is uniform and there are no obvious traces of pests and diseases". The text of the building area is "an office building complex in the commercial core area of ​​a certain city. From the zenith view, there are a total of 8 buildings. The roofs are all flat and equipped with photovoltaic panels. From the 45° side view, the building height is about 80-120 meters. The facade is a glass curtain wall and the ground floor is a 3-story commercial podium".

[0034] In another embodiment, for downstream tasks such as medical image diagnosis and report generation, the sample images for the training pair are dual-view medical images of the same lesion / organ, and the text information is the corresponding clinical radiology report segment. In this paper, lung medical images are used as an example for the training pair.

[0035] Specifically, the training pairs consist of two-dimensional medical chest X-ray images and corresponding report texts, where the image portion includes both frontal and side (if any) views from the same examination. The training sample set can then be represented as paired two-dimensional chest X-ray images and report texts. ,in It is a multi-view image tuple , representing front and side view images. This is the corresponding medical report. The training sample set can use the open-source dataset MIMIC-CXRJPG.

[0036] In step S110, the image encoder uses a visual encoder to extract global and local visual features from the first-view sample image (front) and the second-view sample image (side), respectively.

[0037] In this embodiment, the visual encoder may specifically be a Vision Transformer.

[0038] Specifically, the image encoder is used to extract global and local visual features from multi-view image tuples. Input image tuples After being divided into N patches, the images are fed into the encoder to generate multi-view image features, i.e., first-view sample image features. Second-view sample image features :in, Including global features of the front view and front view local features ; Including global features of the side view and side view local features .

[0039] In this embodiment, the text encoder employs the target-related language model BioClinical BERT to extract global and local text features of the text information. For example, when the sample image is a medical image, a biomedical language model can be used as the text encoder to extract features from the text report. This text encoder then processes the purified input report. Encoding as global text features and local text features M represents the maximum text length.

[0040] In other embodiments, when the sample image is a remote sensing image, the text encoder may employ a variant of BERT pre-trained on remote sensing domain text data.

[0041] In this embodiment, before inputting the text information into the text encoder, the text information is semantically cleaned and standardized using a large language model to obtain purified text information, which is then input into the text encoder.

[0042] Specifically, the report text in the open-source dataset MIMIC-CXR JPG is purified using a large language model to remove redundancy and obtain the purified report text. A CSV file corresponding to the pre-trained multimodal dataset is constructed, which must include information such as the image path, view type, original text content, and purified text content for each inspection. Specific large language model instruction prompts are as follows: Figure 3 As shown. Each tip includes a manually curated single-sample example to enhance specificity and relevance.

[0043] Similarly, when applied to other image task scenarios, the text in the applied image dataset can be purified and redundant removed in a similar way. Simply replace "radiologist" with the name of an expert in other professional fields, and then input the purified text information into the text encoder for processing.

[0044] In this embodiment, in the multi-view fusion unit: a view type encoding sequence and a missing state identifier sequence are constructed for the visual features of two different viewpoints, respectively. The view type encoding sequence is used to identify whether the visual feature belongs to the first viewpoint or the second viewpoint, and the missing state identifier sequence is used to identify whether there is a missing feature. The global and local visual features of the first viewpoint sample image and the second viewpoint sample image are fused accordingly to obtain a multi-view global feature sequence and a multi-view local feature sequence. The multi-view global feature sequence and the multi-view local feature sequence are concatenated and superimposed with the view type encoding sequence and the missing state identifier sequence of the corresponding viewpoint along the feature channel dimension to obtain an enhanced multi-view global feature sequence and an enhanced multi-view local feature sequence. Based on the enhanced multi-view global feature sequence and the enhanced multi-view local feature sequence, a multi-head self-attention mechanism is used to calculate the correlation weight between features of different viewpoints and complete feature interaction. An average pooling operation is performed on the features output by the attention mechanism along the sequence length dimension to aggregate global information. The pooling result is input into a linear projection layer for dimensional fine-tuning and feature mapping to obtain fused image features, which include fused image global features and fused image local features.

[0045] Specifically, in the view type coding sequence, "0" indicates the first view (such as the front view) and "1" indicates the second view (such as the side view).

[0046] Specifically, in the missing state identifier sequence, "0" indicates the presence of a viewpoint, and "1" indicates the absence of a viewpoint (e.g., "1" indicates no image from the second viewpoint). That is, even when an image from a certain viewpoint is missing, the multi-view fusion unit can still be used to fuse image features from multiple viewpoints.

[0047] In step S120, a learnable vector is constructed as a cross-modal bridge. Global features of the fused image Global text features and cross-modal bridge Projecting onto a shared potential space. Cross-modal bridge. Alignment among the three is ensured by using reconstruction loss and contrast loss.

[0048] Specifically, the image intermediate state and the text intermediate state are represented as follows: ; ; In the above formula, and These are the global features and intermediate states of the fused image, respectively. and These refer to the global text features and the text intermediate state, respectively. express transpose, This indicates a cross-modal bridge.

[0049] In this embodiment, intermediate state loss, text reconstruction loss, and image reconstruction loss are used to construct the knowledge bridging loss; ; Among them, intermediate state loss For InfoNCE loss, and for text reconstruction loss and image reconstruction loss The MSE loss is used for reconstruction loss. Reconstruction loss ensures that the knowledge bridge can effectively capture image and text information, while InfoNCE loss is used to align the representations of image intermediates and text intermediates in the latent space.

[0050] In step 130, within the global-local cross-alignment unit: the global features and local features of the fused image are concatenated along the feature channel dimension to obtain globally enhanced local fused image features. Global text features and local text features are concatenated along the feature channel dimension to obtain globally enhanced local text features. Globally enhanced local fusion image features are computed through a cross-modal attention mechanism. With global text features The semantic association weights are calculated, and the first interaction features are obtained. Then, the globally enhanced local text features are calculated through a cross-modal attention mechanism. Global features of the fused image The visual-semantic association weights are calculated, and the second interaction feature is obtained. The first and second interaction features are then passed through a multilayer perceptron, and after nonlinear mapping and dimensionality adjustment, the image-to-text similarity score matrix and the text-to-image similarity score matrix are obtained. Based on image-to-text similarity score matrices and text-to-image similarity score matrices. Calculate the global-local cross-alignment loss. ,in: ; ; In this embodiment, InfoNCE loss is used to optimize the obtained image-to-text and text-to-image similarity score matrices. and To maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs.

[0051] In step S140, the total loss used when training the cross-modal feature extraction model is: ; By employing this alignment strategy of globally enhancing local cross-connection and knowledge bridging to generate intermediate states, the proposed multi-view cross-modal training framework can effectively handle multi-view data, narrow the modal gap, and achieve more accurate and robust semantic alignment between images and reports, thereby significantly improving the performance of zero-shot classification tasks.

[0052] In this embodiment, after training the cross-modal feature extraction model using a large number of training pairs within a multi-view cross-modal training framework, it is also tested. The input consists of a two-dimensional chest X-ray image to be classified and possible disease category text prompts. The similarity scores between the two-dimensional chest X-ray image and each disease category text prompt are obtained. Based on each similarity score and the actual category label, the AUROC (Area Under the Receiver Operating Characteristic curve, i.e., the area under the ROC curve) is calculated to judge the performance of the cross-modal feature extraction model. When the performance meets the preset conditions, the trained cross-modal feature extraction model is obtained.

[0053] In step S150, after obtaining cross-modal aligned image features and text features, downstream tasks are performed based on the image features and text features. Downstream tasks include target recognition, image detection, etc.

[0054] In this embodiment, the prompt text is related to the downstream task. For example, in the downstream task, the prompt text may contain multiple categories related to the target for classifying the target.

[0055] Among the aforementioned multi-view bridging fine-grained image-text cross-modal alignment methods, a new method, MV-Bridge, is proposed. This method, based on multi-angle views and corresponding report text, learns the semantic dependencies between images and text through cross-modal alignment pre-training and is used for subsequent zero-shot image classification based on category-based text prompts. This method proposes a View-Aware Fusion (VAAF) unit to fuse image features from both frontal and side views, obtaining global and local features of the fused image. This unit employs a state-aware mechanism: it utilizes a zero-filler placeholder and combines explicit missing states with view type embeddings. This design allows the model to dynamically switch its encoding strategy—collaboratively aggregating complementary information when two views coexist, or using learnable state embeddings to compensate for missing modal bias when only the frontal view is available. Through this view generation and compensation mechanism, robust and comprehensive image representations can still be constructed even when side views are missing, effectively improving the model's ability to utilize multi-source information.

[0056] This method proposes a cross-modal bridge, which uses a knowledge bridge constructed based on learnable vectors as a medium to generate global features of intermediate image states and global features of intermediate text states, and performs cross-modal alignment of global features under a unified semantic space, which to some extent makes up for the modal gap when directly performing image-text global feature comparison learning.

[0057] This method proposes a Global-Local Cross-Alignment (GLCA) unit, which utilizes global features to supplement local feature information within a modality at the level of cross-modal alignment of local features, promoting deep and fine-grained interaction between global and local features between image and text modalities. Furthermore, the final similarity score obtained by processing the interacted features through a multilayer perceptron serves as a novel measure of cross-modal feature similarity, improving subsequent zero-shot classification performance.

[0058] This method extracts richer image information by fusing frontal and side view images. A bridging mechanism mediates intermediate state generation to bridge the modal gap in global feature alignment between images and text. Finally, the similarity score obtained through global-local cross-alignment provides a more comprehensive estimate of image-text similarity. Compared to existing cross-modal comparison alignment methods based on single-view (frontal view) images and text, this method, by fusing more source image information and bridging the cross-modal gap, achieves more thorough and finer-grained interaction of information both between and within modalities. This results in higher accuracy and AUROC scores when performing subsequent zero-shot image classification based on category-based text prompts.

[0059] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0060] In one embodiment, such as Figure 4 As shown, a multi-view bridging fine-grained image-text cross-modal alignment device is provided, including: a training sample set acquisition module 200, a feature extraction module 210, a bridge-mediated intermediate state generation module 220, a global-local cross-alignment module 230, a cross-modal feature extraction model training module 240, and an image-text cross-modal alignment module 250, wherein: The training sample set acquisition module 200 is used to acquire a training sample set, which includes multiple training pairs. Each training pair includes sample images of the same target acquired from two different perspectives and corresponding text information. The feature extraction module 210 is used to train the cross-modal feature extraction model using each training pair in the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information in the training pair respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain fused image features. The bridge-mediated intermediate state generation module 220 is used to project the text features and fused image features into a shared space using a cross-modal bridge to obtain image intermediate state and text intermediate state, construct intermediate state loss based on image intermediate state and text intermediate state, and construct text reconstruction loss and image reconstruction loss based on text features and text intermediate state, fused image features and image intermediate state, respectively. The global-local cross-alignment module 230 is used to obtain the similarity score matrix between image and text and between text and image based on the text features and the fused image features using the global-local cross-alignment unit, thereby obtaining the global-local cross-alignment loss; The cross-modal feature extraction model training module 240 is used to train the cross-modal feature extraction model based on the intermediate state loss, text reconstruction loss, image reconstruction loss and the global-local cross-alignment loss until convergence, so as to obtain the trained cross-modal feature extraction model. The image-text cross-modal alignment module 250 is used to acquire target images of the same target from two different perspectives, as well as corresponding prompt text. The target images and prompt text are input into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

[0061] Specific limitations regarding the multi-view bridging fine-grained cross-modal alignment device can be found in the limitations of the multi-view bridging fine-grained cross-modal alignment method described above, and will not be repeated here. Each module in the aforementioned multi-view bridging fine-grained cross-modal alignment device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0062] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a multi-view bridging fine-grained cross-modal alignment method for graphics and text. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0063] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0064] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Obtain a training sample set, which includes multiple training pairs. Each training pair includes sample images of the same target obtained from two different perspectives and corresponding text information. The cross-modal feature extraction model is trained using each training pair in the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information in the training pair respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain fused image features. The text features and fused image features are projected into a shared space using a cross-modal bridge to obtain image intermediate state and text intermediate state. An intermediate state loss is constructed based on the image intermediate state and text intermediate state. Text reconstruction loss and image reconstruction loss are constructed based on the text features and text intermediate state, the fused image features and image intermediate state, respectively. The global-local cross-alignment unit is used to obtain the similarity score matrix between image and text and between text and image based on the text features and the fused image features, thereby obtaining the global-local cross-alignment loss; The cross-modal feature extraction model is trained based on the intermediate state loss, text reconstruction loss, image reconstruction loss, and global-local cross-alignment loss until convergence, thus obtaining the trained cross-modal feature extraction model. The target images and corresponding prompt texts are obtained from two different perspectives of the same target. The target images and prompt texts are then input into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

[0065] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain a training sample set, which includes multiple training pairs. Each training pair includes sample images of the same target obtained from two different perspectives and corresponding text information. The cross-modal feature extraction model is trained using each training pair in the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information in the training pair respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain fused image features. The text features and fused image features are projected into a shared space using a cross-modal bridge to obtain image intermediate state and text intermediate state. An intermediate state loss is constructed based on the image intermediate state and text intermediate state. Text reconstruction loss and image reconstruction loss are constructed based on the text features and text intermediate state, the fused image features and image intermediate state, respectively. The global-local cross-alignment unit is used to obtain the similarity score matrix between image and text and between text and image based on the text features and the fused image features, thereby obtaining the global-local cross-alignment loss; The cross-modal feature extraction model is trained based on the intermediate state loss, text reconstruction loss, image reconstruction loss, and global-local cross-alignment loss until convergence, thus obtaining the trained cross-modal feature extraction model. The target images and corresponding prompt texts are obtained from two different perspectives of the same target. The target images and prompt texts are then input into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

[0066] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A fine-grained cross-modal alignment method for multi-view bridging graphics and text, characterized in that, The method includes: Obtain a training sample set, which includes multiple training pairs. Each training pair includes sample images of the same target obtained from two different perspectives and corresponding text information. The cross-modal feature extraction model is trained using each training pair in the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information in the training pair respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain fused image features. The text features and fused image features are projected into a shared space using a cross-modal bridge to obtain image intermediate state and text intermediate state. An intermediate state loss is constructed based on the image intermediate state and text intermediate state. Text reconstruction loss and image reconstruction loss are constructed based on the text features and text intermediate state, the fused image features and image intermediate state, respectively. The global-local cross-alignment unit is used to obtain the similarity score matrix between image and text and between text and image based on the text features and the fused image features, thereby obtaining the global-local cross-alignment loss; The cross-modal feature extraction model is trained based on the intermediate state loss, text reconstruction loss, image reconstruction loss, and global-local cross-alignment loss until convergence, thus obtaining the trained cross-modal feature extraction model. The target images and corresponding prompt texts are obtained from two different perspectives of the same target. The target images and prompt texts are then input into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

2. The multi-view bridging fine-grained cross-modal alignment method for graphics and text according to claim 1, characterized in that, The image encoder is a visual encoder that extracts global and local visual features from the first-view sample image and the second-view sample image, respectively. The text encoder employs a target-related language model to extract global and local text features of the text information.

3. The multi-view bridging fine-grained cross-modal alignment method for graphics and text according to claim 2, characterized in that, Before inputting the text information into the text encoder, the text information is also semantically cleaned and standardized using a large language model to obtain purified text information, which is then input into the text encoder.

4. The multi-view bridging fine-grained cross-modal alignment method for text and images according to claim 2, characterized in that, In the multi-view fusion unit: For visual features from two different perspectives, a view type encoding sequence and a missing state identifier sequence are constructed respectively. The view type encoding sequence is used to identify whether the visual feature belongs to a first perspective or a second perspective, and the missing state identifier sequence is used to identify whether there is a missing feature. The global and local visual features of the first and second view sample images are fused together to obtain a multi-view global feature sequence and a multi-view local feature sequence. The multi-view global feature sequence and the multi-view local feature sequence are concatenated and superimposed with the view type encoding sequence and the missing state identifier sequence of the corresponding view along the feature channel dimension to obtain the enhanced multi-view global feature sequence and the enhanced multi-view local feature sequence. Based on the enhanced multi-view global feature sequence and the enhanced multi-view local feature sequence, a multi-head self-attention mechanism is used to calculate the correlation weights between features from different perspectives and complete feature interaction. The features output by the attention mechanism are subjected to average pooling along the sequence length dimension to aggregate global information. The pooling result is input into a linear projection layer for dimensional fine-tuning and feature mapping to obtain the fused image features, which include fused image global features and fused image local features.

5. The multi-view bridging fine-grained cross-modal alignment method for text and images according to claim 4, characterized in that, The cross-modal bridge is constructed from learnable vectors, and the image intermediate and text intermediate are represented as follows: In the above formula, and These are the global features and intermediate states of the fused image, respectively. and These refer to the global text features and the text intermediate state, respectively. express transpose, This indicates a cross-modal bridge.

6. The multi-view bridging fine-grained cross-modal alignment method for text and images according to claim 4, characterized in that, In the global-local cross-alignment unit: The global features and local features of the fused image are concatenated along the feature channel dimension to obtain globally enhanced local fused image features; The global text features and local text features are concatenated along the feature channel dimension to obtain globally enhanced local text features; The semantic association weights between the globally enhanced local fused image features and the global text features are calculated using a cross-modal attention mechanism to obtain the first interaction feature; The visual-semantic association weights between the globally enhanced local text features and the fused global image features are calculated using a cross-modal attention mechanism to obtain the second interaction feature; The first interaction feature and the second interaction feature are respectively passed through a multilayer perceptron, and after nonlinear mapping and dimension adjustment, the image-to-text similarity score matrix and the text-to-image similarity score matrix are obtained. The global-local cross-alignment loss is calculated based on the image-to-text similarity score matrix and the text-to-image similarity score matrix.

7. The multi-view bridging fine-grained cross-modal alignment method for graphics and text according to any one of claims 1-6, characterized in that, After obtaining cross-modal aligned image and text features, downstream tasks are performed based on the image and text features. The prompt text is related to the downstream task.

8. A multi-view bridging fine-grained cross-modal alignment device for graphics and text, characterized in that, The device includes: The training sample set acquisition module is used to acquire a training sample set, which includes multiple training pairs. Each training pair includes sample images of the same target acquired from two different perspectives and corresponding text information. The feature extraction module is used to train the cross-modal feature extraction model using each training pair in the training sample set. The cross-modal feature extraction model includes an image encoder, a text encoder, and a multi-view fusion unit. The image encoder and the text encoder process the two sample images and text information in the training pair respectively to obtain image features and text features from two perspectives. Then, the multi-view fusion unit performs feature fusion based on the image features from the two perspectives to obtain the fused image features. A bridge-mediated intermediate state generation module is used to project the text features and fused image features into a shared space using a cross-modal bridge to obtain image intermediate states and text intermediate states. An intermediate state loss is constructed based on the image intermediate states and text intermediate states. Text reconstruction loss and image reconstruction loss are constructed based on the text features and text intermediate states, the fused image features and image intermediate states, respectively. The global-local cross-alignment module is used to obtain the image-to-text and text-to-image similarity score matrices based on the text features and the fused image features using the global-local cross-alignment unit, thereby obtaining the global-local cross-alignment loss; The cross-modal feature extraction model training module is used to train the cross-modal feature extraction model based on the intermediate state loss, text reconstruction loss, image reconstruction loss and global-local cross-alignment loss until convergence, so as to obtain the trained cross-modal feature extraction model. The image-text cross-modal alignment module is used to acquire target images of the same target from two different perspectives, as well as corresponding prompt text. The target images and prompt text are then input into the trained cross-modal feature extraction model to obtain cross-modal aligned image features and text features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images