Image-text alignment method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an image-text alignment method and apparatus, a computer device, a readable storage medium and a program product. The method comprises the steps of obtaining a medical image and a to-be-analyzed text for the medical image; obtaining a first image distribution feature of the medical image and a first text distribution feature of the to-be-analyzed text; respectively mapping the first image distribution feature and the first text distribution feature to a hyperbolic space to obtain a second image distribution feature and a second text distribution feature; and determining an alignment result between the medical image and the to-be-analyzed text according to the second image distribution feature and the second text distribution feature. By adopting the method, the semantic uncertainty can be better captured, so that the accuracy of an alignment result is improved.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium and computer program product for aligning images and texts. Background Art

[0002] In the medical field, image-text alignment representation learning is a common processing method. Given the natural implicit relationship between images and text, the image-text alignment representation learning method currently relies on the hyperbolic point embedding method. This method takes advantage of the hierarchical modeling of hyperbolic space and can better model image-text data with visual semantic hierarchical partial order.

[0003] Currently, the method of point embedding is mostly used to model image and text data for image and text alignment in hyperbolic space. However, point embedding cannot capture semantic uncertainty, that is, images have multiple interpretations and texts point to different images. This phenomenon is more obvious in the medical field. Summary of the invention

[0004] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium and computer program product for image-text alignment to address the technical problem that the above-mentioned point embedding method cannot capture semantic uncertainty.

[0005] In a first aspect, the present application provides a method for aligning text and images. The method comprises:

[0006] Acquire a medical image and a text to be analyzed for the medical image;

[0007] Acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed;

[0008] Mapping the first image distribution feature and the first text distribution feature to a hyperbolic space respectively to obtain a second image distribution feature and a second text distribution feature;

[0009] An alignment result between the medical image and the text to be analyzed is determined according to the second image distribution feature and the second text distribution feature.

[0010] In one embodiment, the obtaining of the first image distribution feature of the medical image and the first text distribution feature of the text to be analyzed includes:

[0011] Respectively extracting image features of the medical image and text features of the text to be analyzed;

[0012] The image feature is transformed to obtain the first image distribution feature, and the text feature is transformed to obtain the first text distribution feature.

[0013] In one embodiment, the first image distribution feature includes the image distribution expectation and image distribution variance of the medical image; the first text distribution feature includes the text distribution expectation and text distribution variance of the text to be analyzed;

[0014] The step of mapping the first image distribution feature and the first text distribution feature to a hyperbolic space to obtain a second image distribution feature and a second text distribution feature comprises:

[0015] Performing mapping processing on the image distribution expectation in the first image distribution feature to obtain a new image distribution expectation, and constructing the second image distribution feature based on the new image distribution expectation and the image distribution variance;

[0016] Mapping processing is performed on the text distribution expectation in the first text distribution feature to obtain a new text distribution expectation, and the second text distribution feature is constructed based on the new text distribution expectation and the text distribution variance.

[0017] In one embodiment, determining the alignment result between the medical image and the text to be analyzed according to the second image distribution feature and the second text distribution feature includes:

[0018] Acquire the similarity between the second image distribution feature and the second text distribution feature;

[0019] When the similarity is greater than a threshold, it is determined that the alignment result between the medical image and the text to be analyzed is an alignment of the two.

[0020] In one embodiment, the image-text alignment method is implemented by an image-text alignment model, and the image-text alignment model is trained in the following manner:

[0021] Acquire a sample image-text pair; the sample image-text pair comprises a sample medical image and a diagnostic text of the sample medical image;

[0022] Inputting the sample image-text pair into an initial image-text alignment model to obtain sample image distribution features of the sample medical image and sample text distribution features of the diagnostic text;

[0023] A loss value is obtained according to the sample image distribution characteristics and the sample text distribution characteristics, and the initial image-text alignment model is trained with the goal of reducing the loss value to obtain the image-text alignment model.

[0024] In one embodiment, obtaining the loss value according to the sample image distribution feature and the sample text distribution feature includes:

[0025] Inputting the sample image distribution feature and the sample text distribution feature into an inclusive loss function to obtain a first loss value;

[0026] Inputting the distribution expectation in the sample image distribution feature and the distribution expectation in the sample text distribution feature into a contrast loss function to obtain a second loss value;

[0027] The loss value is obtained based on the first loss value and the second loss value.

[0028] In a second aspect, the present application also provides a device for aligning text and images. The device comprises:

[0029] A data acquisition unit, used for acquiring a medical image and a text to be analyzed for the medical image;

[0030] A feature acquisition unit, used to acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed;

[0031] A feature mapping unit, used to map the first image distribution feature and the first text distribution feature to a hyperbolic space respectively, to obtain a second image distribution feature and a second text distribution feature;

[0032] The image-text alignment unit is used to determine the alignment result between the medical image and the text to be analyzed according to the second image distribution feature and the second text distribution feature.

[0033] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0034] Acquire a medical image and a text to be analyzed for the medical image;

[0035] Acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed;

[0036] Mapping the first image distribution feature and the first text distribution feature to a hyperbolic space respectively to obtain a second image distribution feature and a second text distribution feature;

[0037] An alignment result between the medical image and the text to be analyzed is determined according to the second image distribution feature and the second text distribution feature.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Acquire a medical image and a text to be analyzed for the medical image;

[0040] Acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed;

[0041] Mapping the first image distribution feature and the first text distribution feature to a hyperbolic space respectively to obtain a second image distribution feature and a second text distribution feature;

[0042] An alignment result between the medical image and the text to be analyzed is determined according to the second image distribution feature and the second text distribution feature.

[0043] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Acquire a medical image and a text to be analyzed for the medical image;

[0045] Acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed;

[0046] Mapping the first image distribution feature and the first text distribution feature to a hyperbolic space respectively to obtain a second image distribution feature and a second text distribution feature;

[0047] An alignment result between the medical image and the text to be analyzed is determined according to the second image distribution feature and the second text distribution feature.

[0048] The above-mentioned image-text alignment method, device, computer equipment, storage medium and computer program product, when performing image-text alignment recognition on a medical image and the text to be analyzed of the medical image, first obtain the first image distribution feature of the medical image and the first text distribution feature of the text to be analyzed respectively, and then map the first image distribution feature and the first text distribution feature to the hyperbolic space respectively to obtain the second image distribution feature and the second text distribution feature. Finally, according to the second image distribution feature and the second text distribution feature, the alignment result between the medical image and the text to be analyzed is determined. On the one hand, this method uses non-Euclidean hyperbolic space to model image-text data, and uses the advantages of hyperbolic space for hierarchical modeling and the generalization ability of cross-modal contrast learning in zero-shot learning. Compared with Euclidean space, better data representation can be obtained, thereby improving the accuracy of image-text alignment results. On the other hand, by replacing the traditional point embedding method with the density-based distribution embedding method of image distribution features and text distribution features, semantic uncertainty can be better captured, thereby improving the accuracy of alignment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG1 is a schematic diagram of a flow chart of a method for aligning text and images in one embodiment;

[0050] FIG2 is a schematic diagram of the structure of a first conversion module and a second conversion module in a picture-text alignment model in one embodiment;

[0051] FIG3 is a schematic diagram of a flow chart of a method for training an image-text alignment model in one embodiment;

[0052] FIG4 is a schematic diagram of the structure of an image-text alignment model in one embodiment;

[0053] FIG5 is a schematic diagram of a flow chart of a method for aligning text and images in another embodiment;

[0054] FIG6 is a structural block diagram of a device for aligning images and texts in one embodiment;

[0055] FIG. 7 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0058] In one embodiment, as shown in FIG1 , a method for aligning images and texts is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0059] Step S110, obtaining a medical image and a text to be analyzed for the medical image.

[0060] The medical image is an image including a medical object, which may be a two-dimensional image. For example, the medical image may be an X-ray of the heart. The medical object may be, for example, a human body or a part or tissue of the human body.

[0061] The text to be analyzed represents a text description for analyzing a medical image. For example, if the medical image is an X-ray of the lungs, the corresponding text to be analyzed may be "whether the patient has pneumonia", "type of pneumonia", etc.

[0062] Step S120, obtaining a first image distribution feature of a medical image and a first text distribution feature of a text to be analyzed.

[0063] The first image distribution feature is used to characterize the density distribution of the medical image, and the first text distribution feature is used to characterize the density distribution of the text to be analyzed. The first image distribution feature and the first text distribution feature are both features in the Euclidean space. The Euclidean space refers to a real vector space with a metric function, which is based on the Euclidean distance.

[0064] In a specific implementation, the image features of the medical image and the text features of the text to be analyzed can be first extracted, and then the image features can be converted to obtain the first image distribution features, and the text features can be converted to obtain the first text distribution features. More specifically, the image features of the medical image can be extracted by an image encoder, and the text features of the text to be analyzed can be extracted by a text encoder. The image features are converted into image distribution features in Euclidean space by a conversion module built based on a fully connected layer (FC), which is recorded as the first image distribution feature. Similarly, the text features are converted into text distribution features in Euclidean space by a conversion module built based on a fully connected layer (FC), which is recorded as the first text distribution feature.

[0065] Step S130 , mapping the first image distribution feature and the first text distribution feature to a hyperbolic space respectively to obtain a second image distribution feature and a second text distribution feature.

[0066] Among them, hyperbolic space is a non-Euclidean space, which is suitable for representing data with complex structures and hierarchical relationships, and can better represent some important multimodal relationships.

[0067] It can be understood that there is a natural implication relationship between images and texts, specifically, text implies images. For example, the text is "pneumonia", and there may be multiple corresponding images, and hyperbolic space has the advantage of hierarchical modeling. Therefore, this application uses hyperbolic density embedding in hyperbolic space to perform visual semantic representation learning, so that when modeling graphic and text data with potential hierarchical features in non-Euclidean hyperbolic space, better data representation can be obtained.

[0068] In a specific implementation, based on a hyperbolic mapping relationship, the first image distribution feature can be mapped from Euclidean space to hyperbolic space to obtain a second image distribution feature, and the first text distribution feature can be mapped from Euclidean space to hyperbolic space to obtain a second text distribution feature.

[0069] Step S140: determining an alignment result between the medical image and the text to be analyzed according to the second image distribution feature and the second text distribution feature.

[0070] In a specific implementation, the distance between the second image distribution feature and the second text distribution feature can be measured by a metric formula in a hyperbolic space to determine the correlation between the two, and then determine the alignment result between the medical image and the text to be analyzed.

[0071] In the above-mentioned image-text alignment method, when performing image-text alignment recognition on a medical image and the text to be analyzed of the medical image, the first image distribution feature of the medical image and the first text distribution feature of the text to be analyzed are first obtained respectively, and then the first image distribution feature and the first text distribution feature are mapped to the hyperbolic space respectively to obtain the second image distribution feature and the second text distribution feature. Finally, the alignment result between the medical image and the text to be analyzed is determined according to the second image distribution feature and the second text distribution feature. On the one hand, this method uses non-Euclidean hyperbolic space to model image-text data, and uses the advantages of hyperbolic space for hierarchical modeling and the generalization ability of cross-modal contrast learning in zero-shot learning. Compared with Euclidean space, better data representation can be obtained, thereby improving the accuracy of image-text alignment results. On the other hand, by replacing the traditional point embedding method with the density-based distribution embedding method of image distribution features and text distribution features, semantic uncertainty can be better captured, thereby improving the accuracy of alignment results.

[0072] In an exemplary embodiment, step S120, obtaining a first image distribution feature of a medical image and a first text distribution feature of a text to be analyzed, includes:

[0073] Step S121, extracting image features of the medical image and text features of the text to be analyzed respectively.

[0074] In the specific implementation, the image features of medical images can be extracted through the image encoder, and the text features of the text to be analyzed can be extracted through the text encoder. Among them, the image encoder can use VIT (Vision Transformer) as the image encoder. Vision Transformer regards the entire image as a series of "patches", and these patches are regarded as "words" input into the Transformer model. Among them, the text encoder can use BioClinicalBERT (a biomedical pre-trained model based on BERT). BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language representation model.

[0075] In practical applications, medical images can be input into the visual transformer as an image encoder, and the output of the visual transformer can be pooled to obtain image features. Similarly, the text to be analyzed can be input into the BioClinicalBERT as a text encoder, and word-level embedding can be generated by BioClinicalBERT, with the output of the special [CLS] tag as the text feature.

[0076] Step S122: transform the image features to obtain first image distribution features, and transform the text features to obtain first text distribution features.

[0077] In a specific implementation, the first conversion module can perform feature conversion processing on the image features to obtain the first image distribution features, and the second conversion module can perform feature conversion processing on the text features to obtain the first text distribution features. The first conversion module and the second conversion module have the same model structure and different model parameters.

[0078] More specifically, the first conversion module and the second conversion module are both constructed based on a fully connected layer (FC layer). Referring to FIG2 , a schematic diagram of the structure of the first conversion module and the second conversion module is shown in an embodiment. As shown in FIG2 , when the image features are converted by the first conversion module, the image features are converted to Input the first conversion module, in which the image features are first transformed through a fully connected layer (FC layer) The features are reorganized and then output through the ReLU layer. The output features are further reorganized through two fully connected layers (FC layers) to output the expected image distribution of the medical image. and image distribution variance , as the first image distribution feature, the transformation process can be expressed as: .

[0079] Similarly, when the text features are transformed by the second transformation module, the text features Input the second conversion module, in which the text features are first transformed through a fully connected layer (FC layer) The features are reorganized and then output through the ReLU layer. The output features are further reorganized through two fully connected layers (FC layers) to output the expected text distribution of the text to be analyzed. and text distribution variance , as the first text distribution feature, the transformation process can be expressed as: .

[0080] Among them, ReLU (Rectified Linear Unit) is a linear rectification function, also known as a rectified linear unit, which is a commonly used activation function in artificial neural networks. The fully connected layer (FC layer) is used to reorganize all local information.

[0081] It can be understood that due to the difference in processing image features and text features, although the first conversion module and the second conversion module have the same structure, the model parameters are different, specifically, the parameters of each fully connected layer (FClayer) in the two conversion modules are different.

[0082] In this embodiment, the image features of the medical image and the text features of the text to be analyzed are first extracted, and then the image features and the text features are converted into image distribution features and text distribution features through the first conversion module and the second conversion module respectively. The image distribution features and the text distribution features are used to align the image and the text. This density-based distribution embedding method replaces the traditional point embedding method, which can better capture the semantic uncertainty of the image and the text.

[0083] In an exemplary embodiment, the first image distribution feature includes an image distribution expectation and an image distribution variance of a medical image; and the first text distribution feature includes a text distribution expectation and a text distribution variance of a text to be analyzed.

[0084] The above-mentioned step S130 maps the first image distribution feature and the first text distribution feature to the hyperbolic space respectively to obtain the second image distribution feature and the second text distribution feature, including: mapping the image distribution expectation in the first image distribution feature to obtain a new image distribution expectation, and constructing the second image distribution feature based on the new image distribution expectation and the image distribution variance; mapping the text distribution expectation in the first text distribution feature to obtain a new text distribution expectation, and constructing the second text distribution feature based on the new text distribution expectation and the text distribution variance.

[0085] Among them, the image distribution expectation of the medical image and the text distribution expectation of the text to be analyzed are both vectors, and the image distribution variance of the medical image and the text distribution variance of the text to be analyzed are both scalars.

[0086] Specifically, the image distribution expectation in the first image distribution feature can be mapped based on the hyperbolic mapping relationship, and the text distribution expectation in the first text distribution feature can be mapped. The hyperbolic mapping relationship of the two mappings can be expressed as follows:

[0087] The hyperbolic mapping relationship of image distribution expectation is:

[0088] (1)

[0089] Hyperbolic mapping relationship of text distribution expectation:

[0090] (2)

[0091] Among them, in formula (1) represents the new image distribution expectation after mapping to the hyperbolic space, represents the expected distribution of images in Euclidean space. represents the new text distribution expectation after mapping to the hyperbolic space, Represents the expected distribution of text in Euclidean space.

[0092] The image distribution expectation in the first image distribution feature Input formula (1) for mapping and obtain the new image distribution expectation , based on the new image distribution expectation and the image distribution variance in the first image distribution feature , forming the image distribution feature of the hyperbolic space as the second image distribution feature. Among them, the image distribution variance is a scalar, so it can be further processed by the unit matrix, specifically by multiplying the image distribution variance with the unit matrix. The second image distribution feature obtained can be expressed as: ,in, Represents the identity matrix.

[0093] Similarly, the text distribution expectation in the first text distribution feature is Input formula (2) for mapping and obtain the new text distribution expectation , based on the new text distribution expectation and the text distribution variance in the first text distribution feature , forming the text distribution feature of the hyperbolic space as the second text distribution feature. Among them, the text distribution variance is a scalar, so it can be further processed by the unit matrix, specifically by multiplying the text distribution variance with the unit matrix. The second text distribution feature obtained can be expressed as: ,in, Represents the identity matrix.

[0094] In this embodiment, the mapping of the image distribution expectation in the first image distribution feature and the text distribution expectation in the first text distribution feature from the Euclidean space to the hyperbolic space is respectively realized through the hyperbolic mapping relationship, and the image distribution feature in the hyperbolic space is obtained by combining the new image distribution expectation obtained by the mapping and the original image distribution variance, and the text distribution feature in the hyperbolic space is obtained by combining the new text distribution expectation obtained by the mapping and the original text distribution variance, thereby realizing the modeling of image and text data in non-Euclidean hyperbolic space. Compared with the Euclidean space, better data representation can be obtained, thereby improving the accuracy of the image-text alignment results.

[0095] In an exemplary embodiment, in the above step S140, the alignment result between the medical image and the text to be analyzed is determined based on the second image distribution feature and the second text distribution feature, including: obtaining the similarity between the second image distribution feature and the second text distribution feature; when the similarity is greater than a threshold, determining that the alignment result between the medical image and the text to be analyzed is alignment of the two.

[0096] In a specific implementation, the similarity indicates the degree of match between the second image distribution feature and the second text distribution feature, and corresponds to the degree of match or distance between the medical image and the text to be analyzed. If the similarity is greater than a threshold, indicating that the distance between the two is small, it is determined that the medical image and the text to be analyzed are relatively related, and the alignment result between the two is that the two are aligned. On the contrary, if the similarity is less than or equal to the threshold, indicating that the distance between the two is large, it is determined that the medical image and the text to be analyzed are greatly different, and the alignment result between the two is that the two are not aligned.

[0097] The similarity between the second image distribution feature and the second text distribution feature is measured by a measurement formula in a hyperbolic space, specifically, by using the contrast loss function shown in the following formula (4) for measurement.

[0098] In this embodiment, the similarity calculation based on distribution features can better capture the semantic information of medical images and the corresponding to be analyzed, rather than relying solely on simple point embedding matching. This method improves the accuracy of alignment because it takes into account the overall distribution of features and can better reflect the relevance of image content and text. In addition, through density-based distribution features, the model can capture semantic uncertainty in images and text, making it more flexible in determining alignment.

[0099] In an exemplary embodiment, the above-mentioned image-text alignment method can be implemented by an image-text alignment model. As shown in FIG3 , the image-text alignment model used to implement the above-mentioned image-text alignment method is trained by the following steps:

[0100] Step S310, obtaining a sample image-text pair; the sample image-text pair includes a sample medical image and a diagnostic text of the sample medical image;

[0101] Step S320, inputting the sample image-text pair into the initial image-text alignment model to obtain sample image distribution features of the sample medical image and sample text distribution features of the diagnosis text;

[0102] Step S330, obtaining a loss value according to the sample image distribution characteristics and the sample text distribution characteristics, and training the initial image-text alignment model with the goal of reducing the loss value to obtain an image-text alignment model.

[0103] The diagnosis text indicates the description text of the diagnosis result of the sample medical image, and may be the text in the imaging report corresponding to the sample medical image. For example, the sample medical image is an X-ray of the heart, and the diagnosis text of the X-ray is "enlargement of the heart outline size".

[0104] Specifically, the image-text alignment model is designed to learn the semantic relationship between the image and the text in the image-text pair. The text that describes the image itself can clearly indicate the characteristics of the image. This direct correspondence enables the model to effectively learn the connection between image features and text expressions, thereby establishing a more accurate embedding space. Therefore, when training the image-text alignment model, the text in the sample image-text pair obtained should be a diagnostic text that can directly indicate the characteristics of the sample medical image.

[0105] When training the model, the sample image-text pair is used as the input variable and the initial image-text alignment model is input. In the image-text alignment model, the image features of the sample medical image and the text features of the diagnostic text are first extracted, and then converted into image distribution features and text distribution features in the Euclidean space. Further, the two distribution features are mapped from the Euclidean space to the hyperbolic space to obtain the sample image distribution features of the sample medical image and the sample text distribution features of the diagnostic text. The sample image distribution features and the sample text distribution features are substituted into the loss function to calculate the loss value, and then the initial image-text alignment model is trained with the goal of reducing the loss value until the training end condition is reached, and the training is terminated to obtain the image-text alignment model. Among them, the training end condition can be the convergence of the loss value, or reaching a preset number of training times.

[0106] In this embodiment, when training the image-text alignment model, the text of the sample image-text pair used is the diagnostic text that directly indicates the characteristics of the sample medical image, thereby establishing a direct correspondence between the sample medical image and the diagnostic text, so that the image-text alignment model can effectively learn the relationship between image features and text expressions, thereby establishing a more accurate embedding space, improving the training effect of the model, and enabling the model to learn how to understand the image content, so that in the application stage it can analyze and identify the relationship between the image and the text to be analyzed, and can answer various types of inquiries.

[0107] In an exemplary embodiment, in the above step S330, obtaining the loss value according to the sample image distribution characteristics and the sample text distribution characteristics includes:

[0108] Step S331, inputting the sample image distribution feature and the sample text distribution feature into the inclusive loss function to obtain a first loss value;

[0109] Step S332, inputting the distribution expectation in the sample image distribution feature and the distribution expectation in the sample text distribution feature into the contrast loss function to obtain a second loss value;

[0110] Step S333: obtaining a loss value based on the first loss value and the second loss value.

[0111] Specifically, there is a visual semantic hierarchical partial order relationship (i.e., implication relationship) between images and texts, specifically, text implies images, such as the text "dog" contains all dog images. Therefore, it is necessary to use the implication loss function to constrain the partial order relationship. However, what is modeled in this application is the partial order relationship between the distribution features of images and texts. Therefore, the traditional implication loss function is not applicable. Therefore, this application proposes to use the inclusion loss function to constrain the image and text distribution features.

[0112] The inclusion loss function can be expressed as:

[0113] (3)

[0114] in, represents the first loss value, and Respectively represent the sample image distribution characteristics and sample text distribution characteristics, is the KL divergence function, which is used to measure and The difference between the two distributions, is a predefined hyperparameter used to give a boundary (i.e., the tolerable difference boundary value between the two distributions).

[0115] In addition to measuring the partial order relationship between image and text, it is also necessary to measure the similarity between image and text features. Therefore, it is also necessary to calculate the similarity between the sample image distribution features and the sample text distribution features through the contrast loss function, specifically the similarity between the distribution expectation in the sample image distribution features and the distribution expectation in the sample text distribution features.

[0116] The contrast loss function can be expressed as:

[0117] (4)

[0118] in, .

[0119] in, represents the second loss value, and They represent the expected distribution of sample image distribution features and the expected distribution of sample text distribution features, respectively. is a predefined curvature. and For n+1-dimensional data, and can be regarded as the first dimension data, and It can be regarded as n-dimensional data after the first dimension.

[0120] After obtaining the first loss value and the second loss value through the inclusion loss function and the contrast loss function respectively, the first loss value and the second loss value can be summed to obtain the total loss value, that is, the loss value .

[0121] Furthermore, in one embodiment, the sample image-text pairs include positive sample pairs and negative sample pairs, wherein the positive sample pairs represent image-text pairs belonging to the same sample object, and the negative sample pairs represent image-text pairs of different sample objects. When calculating the loss value for model training, different training methods can be used for different sample pair types. Specifically, for positive sample pairs, the training goal is to make the sample image distribution features and the sample text distribution features as similar as possible; for negative sample pairs, the training goal is to make the sample image distribution features and the sample text distribution features as different as possible.

[0122] In this embodiment, by combining contrast loss and inclusion loss, the measurement of the partial order relationship between image distribution features and text distribution features and the measurement of the features themselves are comprehensively considered, so that the loss value can be made more accurate from multiple levels, thereby improving the alignment and recognition effect of the trained image-text alignment model.

[0123] It can be understood that in the related art, combining the advantages of hyperbolic space hierarchical modeling and the generalization ability of cross-modal contrastive learning in zero-shot learning, the contrastive model uses the Lorentz manifold to map image features and text features to points in the hyperbolic space respectively, and uses the implied angle constraint to learn the prior of the partial order structure between images and text in the hyperbolic space. However, the way of mapping objects to points in space has obvious limitations, that is, it is impossible to express semantic uncertainty, that is, an image can write different summaries from different angles, and a summary can also describe different but similar images. In medical images and imaging reports, this phenomenon is even more obvious. Therefore, the present application proposes a new method that can introduce a distribution embedding strategy to model semantic uncertainty and partial order priors between images and texts while taking advantage of the hierarchical feature capture of hyperbolic space.

[0124] In one embodiment, in order to facilitate those skilled in the art to understand the embodiments of the present application, the following will be explained in conjunction with specific examples of the accompanying drawings. Referring to Figure 4, a structural schematic diagram of a graph-text alignment model is shown, including an image encoder, a text encoder, a first conversion module, a second conversion module, a first hyperbolic mapping module, and a second hyperbolic mapping module, wherein the image encoder is used to encode medical images to obtain image features; the text encoder is used to encode text to be analyzed to obtain text features; the first conversion module is used to perform feature conversion on image features to obtain first image distribution features; the second conversion module is used to perform feature conversion on text features to obtain first text distribution features; the first hyperbolic mapping module is used to map the first image distribution features to a hyperbolic space to obtain a second image distribution feature; the second hyperbolic mapping module is used to map the first text distribution features to a hyperbolic space to obtain a second text distribution feature; the process of applying the graph-text alignment model to perform graph-text alignment is shown in Figure 5, and includes the following steps:

[0125] Step S510, obtaining a medical image and a text to be analyzed for the medical image;

[0126] Step S520, inputting the medical image into an image encoder to obtain image features, and inputting the text to be analyzed into a text encoder to obtain text features;

[0127] Step S530, inputting the image features into the first conversion module for feature conversion processing to obtain the image distribution expectation and image distribution variance of the medical image as the first image distribution feature, and inputting the text features into the second conversion module for feature conversion processing to obtain the text distribution expectation and text distribution variance of the text to be analyzed as the first text distribution feature;

[0128] Step S540, inputting the image distribution expectation in the first image distribution feature into the first hyperbolic mapping module for mapping processing to obtain a new image distribution expectation, and constructing a second image distribution feature based on the new image distribution expectation and the image distribution variance;

[0129] Step S550, inputting the text distribution expectation in the first text distribution feature into a second hyperbolic mapping module for mapping processing to obtain a new text distribution expectation, and constructing a second text distribution feature based on the new text distribution expectation and the text distribution variance;

[0130] Step S560, obtaining the similarity between the second image distribution feature and the second text distribution feature;

[0131] Step S570: When the similarity is greater than the threshold, determining that the alignment result between the medical image and the text to be analyzed is an alignment of the two.

[0132] The image-text alignment method provided in this embodiment, on the one hand, utilizes non-Euclidean hyperbolic space to model image-text data, and utilizes the advantages of hyperbolic space in hierarchical modeling and the generalization ability of cross-modal contrastive learning in zero-shot learning. Compared with Euclidean space, it can obtain better data representation, thereby improving the accuracy of image-text alignment results. On the other hand, by converting image features and text features into image distribution features and text distribution features, this density-based distribution embedding method replaces the traditional point embedding method, which can better capture semantic uncertainty, thereby improving the accuracy of alignment results.

[0133] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0134] Based on the same inventive concept, the embodiment of the present application also provides a device for aligning text and images for implementing the method for aligning text and images involved above. The implementation scheme provided by the device to solve the problem to be analyzed is similar to the implementation scheme recorded in the method above, so the specific limitations in one or more embodiments of the device for aligning text and images provided below can refer to the limitations of the method for aligning text and images above, and will not be repeated here.

[0135] In one embodiment, as shown in FIG6 , a device for aligning images and texts is provided, comprising:

[0136] A data acquisition unit 610 is used to acquire a medical image and a text to be analyzed for the medical image;

[0137] A feature acquisition unit 620, configured to acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed;

[0138] A feature mapping unit 630, used to map the first image distribution feature and the first text distribution feature to a hyperbolic space to obtain a second image distribution feature and a second text distribution feature;

[0139] The image-text alignment unit 640 is used to determine the alignment result between the medical image and the text to be analyzed according to the second image distribution feature and the second text distribution feature.

[0140] In one embodiment, the feature acquisition unit 620 is used to extract image features of the medical image and text features of the text to be analyzed respectively; transform the image features to obtain first image distribution features, and transform the text features to obtain first text distribution features.

[0141] In one embodiment, the first image distribution feature includes the image distribution expectation and the image distribution variance of the medical image; the first text distribution feature includes the text distribution expectation and the text distribution variance of the text to be analyzed; the feature mapping unit 630 is also used to map the image distribution expectation in the first image distribution feature to obtain a new image distribution expectation, and construct a second image distribution feature based on the new image distribution expectation and the image distribution variance; and map the text distribution expectation in the first text distribution feature to obtain a new text distribution expectation, and construct a second text distribution feature based on the new text distribution expectation and the text distribution variance.

[0142] In one of the embodiments, the image-text alignment unit 640 is further used to obtain the similarity between the second image distribution feature and the second text distribution feature; when the similarity is greater than a threshold, the alignment result between the medical image and the text to be analyzed is determined to be an alignment of the two.

[0143] In one of the embodiments, the image-text alignment method is implemented by an image-text alignment model, and the above-mentioned device also includes a model training module, which is used to obtain sample image-text pairs; the sample image-text pairs include sample medical images and diagnostic texts of the sample medical images; the sample image-text pairs are input into the initial image-text alignment model to obtain sample image distribution features of the sample medical images and sample text distribution features of the diagnostic text; the loss value is obtained according to the sample image distribution features and the sample text distribution features, and the initial image-text alignment model is trained with the goal of reducing the loss value to obtain the image-text alignment model.

[0144] In one of the embodiments, the model training module is also used to input the sample image distribution features and the sample text distribution features into the inclusive loss function to obtain a first loss value; input the distribution expectation in the sample image distribution features and the distribution expectation in the sample text distribution features into the contrast loss function to obtain a second loss value; and obtain a loss value based on the first loss value and the second loss value.

[0145] Each module in the above-mentioned image-text alignment device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0146] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for aligning text and images is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, a trackball or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0147] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0150] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for aligning images and texts, characterized in that: The method comprises: Acquire a medical image and a text to be analyzed for the medical image; Acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed; Mapping the first image distribution feature and the first text distribution feature to a hyperbolic space respectively to obtain a second image distribution feature and a second text distribution feature; An alignment result between the medical image and the text to be analyzed is determined according to the second image distribution feature and the second text distribution feature.

2. The method according to claim 1, characterized in that The obtaining of the first image distribution feature of the medical image and the first text distribution feature of the text to be analyzed includes: Respectively extracting image features of the medical image and text features of the text to be analyzed; The image feature is transformed to obtain the first image distribution feature, and the text feature is transformed to obtain the first text distribution feature.

3. The method according to claim 1, characterized in that: The first image distribution feature includes the image distribution expectation and the image distribution variance of the medical image; the first text distribution feature includes the text distribution expectation and the text distribution variance of the text to be analyzed; The step of mapping the first image distribution feature and the first text distribution feature to a hyperbolic space to obtain a second image distribution feature and a second text distribution feature comprises: Performing mapping processing on the image distribution expectation in the first image distribution feature to obtain a new image distribution expectation, and constructing the second image distribution feature based on the new image distribution expectation and the image distribution variance; Mapping processing is performed on the text distribution expectation in the first text distribution feature to obtain a new text distribution expectation, and the second text distribution feature is constructed based on the new text distribution expectation and the text distribution variance.

4. The method according to claim 1, characterized in that: The step of determining the alignment result between the medical image and the text to be analyzed according to the second image distribution feature and the second text distribution feature includes: Acquire the similarity between the second image distribution feature and the second text distribution feature; When the similarity is greater than a threshold, it is determined that the alignment result between the medical image and the text to be analyzed is an alignment of the two.

5. The method according to claim 1, characterized in that The image-text alignment method is implemented by an image-text alignment model, and the image-text alignment model is trained in the following way: Acquire a sample image-text pair; the sample image-text pair comprises a sample medical image and a diagnostic text of the sample medical image; Inputting the sample image-text pair into an initial image-text alignment model to obtain sample image distribution features of the sample medical image and sample text distribution features of the diagnosis text; A loss value is obtained according to the sample image distribution characteristics and the sample text distribution characteristics, and the initial image-text alignment model is trained with the goal of reducing the loss value to obtain the image-text alignment model.

6. The method according to claim 5, characterized in that The obtaining of the loss value according to the sample image distribution feature and the sample text distribution feature includes: Inputting the sample image distribution feature and the sample text distribution feature into an inclusive loss function to obtain a first loss value; Inputting the distribution expectation in the sample image distribution feature and the distribution expectation in the sample text distribution feature into a contrast loss function to obtain a second loss value; The loss value is obtained based on the first loss value and the second loss value.

7. A device for aligning images and texts, characterized in that: The device comprises: A data acquisition unit, used for acquiring a medical image and a text to be analyzed for the medical image; A feature acquisition unit, used to acquire a first image distribution feature of the medical image and a first text distribution feature of the text to be analyzed; A feature mapping unit, used to map the first image distribution feature and the first text distribution feature to a hyperbolic space respectively, to obtain a second image distribution feature and a second text distribution feature; The image-text alignment unit is used to determine the alignment result between the medical image and the text to be analyzed according to the second image distribution feature and the second text distribution feature.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the image-text alignment method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image-text alignment method described in any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image-text alignment method described in any one of claims 1 to 6 are implemented.

Images