A method for joint training of an image encoder and a text encoder, an image query method, a health status assessment method, and a device based on contrastive learning.

Abstract

This application discloses a joint training method for an image encoder and a text encoder based on contrastive learning, an image query method, a health status assessment method, and an apparatus, relating to the field of medical image understanding. The method includes: inputting various 3D multimodal brain MRI images and various Chinese medical reports into an image encoder model and a text encoder model respectively, obtaining one-dimensional image vectors and one-dimensional text vectors; calculating the loss for the current iteration using a fusion loss function based on the image-text similarity matrix, the text-image similarity matrix, and the semantic similarity matrix, and optimizing the parameters of the image encoder model and the text encoder model; and obtaining the trained image encoder model and text encoder model when the iteration termination condition is met. This application achieves downstream tasks such as zero-shot classification of images and text and cross-modal mutual retrieval.

Description

Technical Field

[0001] This application relates to the field of medical image understanding, and in particular to a method for joint training of image encoder and text encoder based on contrastive learning, a method for querying 3D multimodal brain MRI images, and a method and device for assessing target health status based on 3D multimodal brain MRI images. Background Technology

[0002] In recent years, the rapid development of artificial intelligence technology has brought revolutionary changes to all walks of life. Especially in the fields of computer vision and natural language processing, the rise of deep learning technology has led the trend of technological innovation. In the medical field, this technology is widely used in medical image analysis, assisted diagnosis, and automatic generation of image reports, significantly improving the efficiency and accuracy of medical services. However, most current deep learning-based medical image processing requires a large amount of manual annotation as labels to train the neural network. For medical images, especially 3D and multimodal images, the amount of data is relatively scarce, and annotation needs to be done manually by professional doctors.

[0003] In the specific field of medical image processing, brain MRI (Magnetic Resonance Imaging) images serve as crucial diagnostic evidence, and the complexity of their processing and analysis is particularly prominent. Because brain MRI images typically contain multiple modalities (such as T1 (T1-weighted imaging), T2 (T2-weighted imaging), ADC (Apparent Diffusion Coefficient), and DWI (Diffusion-weighted Imaging), each modality provides unique anatomical and pathological information. Therefore, effectively fusing this multimodal information while preserving the spatial integrity of the 3D image has become one of the current technical challenges. Existing contrastive learning techniques mostly focus on 2D image processing. When 3D images are simply segmented into 2D slice sequences for processing, important spatial structural information is inevitably lost, thus affecting the accuracy of diagnosis.

[0004] Furthermore, natural language processing in medical reports faces unique challenges. The high degree of similarity, homogenized semantic content, and complexity of technical terminology in Chinese medical reports make it difficult to directly apply traditional natural language processing techniques and achieve ideal results. Summary of the Invention

[0005] The purpose of this application is to provide a joint training method for image encoders and text encoders based on contrastive learning, an image query method, a health status assessment method and device, which can realize downstream tasks such as zero-shot classification of 3D multimodal images and text and cross-modal mutual retrieval.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a joint training method for an image encoder and a text encoder based on contrastive learning, including:

[0008] 3D multimodal brain MRI images and corresponding Chinese medical reports were obtained to form a training sample set;

[0009] Each 3D multimodal brain MRI image in the current batch of training sample subset is input into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subset corresponds one-to-one with each Chinese medical report;

[0010] Input each Chinese medical report in the current batch of training sample subset into the text encoder model to obtain the one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model;

[0011] Calculate the similarity between each one-dimensional image vector and each one-dimensional text vector, and construct the image-text similarity matrix and the text-image similarity matrix; wherein the text-image similarity matrix is ​​the transpose of the image-text similarity matrix;

[0012] Calculate the semantic similarity between any two Chinese medical reports in the training sample subset of the current batch, and construct a semantic similarity matrix;

[0013] Based on the image-text similarity matrix, text-image similarity matrix, and semantic similarity matrix, the loss for the current iteration is calculated using a fusion loss function; the fusion loss function includes contrastive learning loss and distribution loss.

[0014] The parameters of the image encoder model and the text encoder model are optimized based on the loss of the current iteration;

[0015] Determine whether the iteration termination condition has been met and obtain the determination result;

[0016] If the judgment result is yes, then the image encoder model and text encoder model with optimized output parameters are used as the trained image encoder model and text encoder model.

[0017] If the judgment result is negative, then the current batch of training sample subsets is re-acquired in the training sample set, and the step of "inputting each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model" is returned, and the next iteration optimization is performed.

[0018] Optionally, 3D multimodal brain MRI images and corresponding Chinese medical reports are acquired to form a training sample set, which then includes:

[0019] 3D multimodal brain MRI images are acquired, and image preprocessing is performed on the 3D multimodal brain MRI images. The image preprocessing includes: resampling, Hu value truncation, maximum and minimum normalization, and data augmentation. Specifically, the data augmentation includes: image rotation, image scaling, elastic deformation, intensity modification, and contrast modification.

[0020] Break down multiple modalities in Chinese medical reports into descriptions of individual modalities;

[0021] Extract the modality name, modality location, modality part, and modality performance keywords from the description of each modality, and fill them into a template style to obtain a preprocessed Chinese medical report. The template style is as follows: in the modality {modal}, the {side} and {part} are represented by {performance}. Here, {modal} represents the extracted modality name, {side} represents the extracted modality location, {part} represents the extracted modality part, and {performance} represents the extracted modality performance keywords.

[0022] Optionally, the formula for calculating the semantic similarity is:

[0023] t sim =TextDice*(w loc +w per );

[0024]

[0025] Among them, t sim The semantic similarity between processed Chinese medical reports t1 and t2 is represented by TextDice, and the formal similarity between processed Chinese medical reports t1 and t2 is represented by Len. overlap Len indicates the number of overlapping characters between the processed Chinese medical reports t1 and t2. t1 This indicates the number of characters in the processed Chinese medical report t1, Len t2 w represents the number of characters in the processed Chinese medical report t2.loc This represents the keyword score of the lesion site obtained based on modal orientation and modal location, w per This represents the keyword score of the image performance obtained based on the modal performance keywords.

[0026] Optionally, the formula for calculating the similarity between each one-dimensional image vector and each one-dimensional text vector is:

[0027]

[0028] in, This represents the image-text similarity between the i-th 3D multimodal brain MRI image and the j-th Chinese medical report in the current batch. This represents the i-th 3D multimodal brain MRI image in the current batch. This represents the j-th Chinese medical report in the current batch, v i Let t represent the i-th one-dimensional image vector in the current batch. j Let S(v) represent the j-th one-dimensional text vector in the current batch. i ,t j ) represents the cosine similarity between the i-th one-dimensional image vector and the j-th one-dimensional text vector in the current batch, exp(·) represents taking the exponent of the expression within the parentheses, τ represents the learnable temperature coefficient, and N is the preset batch size.

[0029] Optionally, the formula for calculating the fusion loss function is:

[0030] L=αL clip +βL ts ;

[0031]

[0032] L v2t =CE(E N ,P v2t );

[0033] L t2v =CE(E N ,P t2v );

[0034]

[0035] L ts-v2t =KL1(T sim ,P v2t );

[0036] L ts-t2v =KL2(T sim ,P t2v );

[0037] Where L represents the fusion loss function, L clip This represents the contrastive learning loss, where α is the first weight and L is the second weight. ts This represents the distributed loss, where β is the second weight, and L... v2t L represents the image-text similarity loss. t2v E represents the text-image similarity loss, CE represents the cross-entropy loss function, and E N P represents the identity matrix. v2t Let P represent the image-text similarity matrix. t2v L represents the text-image similarity matrix. ts-v2t L represents the first intermediate function. ts-t2v T represents the second intermediate function. sim Let KL1(·,·) denote the KL divergence between the semantic similarity matrix within the brackets and the image-text similarity matrix, and KL2(·,·) denote the KL divergence between the semantic similarity matrix within the brackets and the text-image similarity matrix.

[0038] Optionally, the step of inputting each 3D multimodal brain MRI image from the current batch of training sample subsets into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model further includes: performing a learning rate warm-up method for a total of 5000 iterations, adjusting the learning rate once for each iteration, and the adjustment formula for the learning rate is as follows:

[0039]

[0040] in, Let lr be the learning rate for the current iteration. init This represents the initial learning rate of the image encoder model or text encoder model, where t is the current iteration number. max This represents the total number of iterations.

[0041] Optionally, optimizing the parameters of the image encoder model and the text encoder model also includes updating the learning rate during training using a multinomial learning rate descent strategy, with the learning rate update formula as follows:

[0042]

[0043] Among them, lr (e) The learning rate is represented by 'e', ​​and the current period is 'e'. A period is defined as 250 iterations. max This represents the total number of cycles.

[0044] Secondly, this application provides a method for querying 3D multimodal brain MRI images, including:

[0045] The trained image encoder model is used to process each 3D multimodal brain MRI image stored in the database to obtain a one-dimensional image vector corresponding to each image.

[0046] The system receives query conditions input by the user, performs text preprocessing on the query conditions, and inputs the preprocessed query conditions into the trained text encoder model to obtain the target one-dimensional text vector.

[0047] The similarity between the target one-dimensional text vector and each one-dimensional image vector is calculated and compared. The 3D multimodal brain MRI image corresponding to the one-dimensional image vector with the highest similarity value is selected as the query result.

[0048] Thirdly, this application provides a method for assessing target health status based on 3D multimodal brain MRI images, including:

[0049] Construct a target health status assessment model, which includes a trained image encoder model and a binary classification model;

[0050] 3D multimodal brain MRI images containing target and non-target health states are input into a trained image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image.

[0051] The one-dimensional image vectors corresponding to each 3D multimodal brain MRI image are used as training samples to train the binary classification model, thereby obtaining a trained binary classification model to assess the health status of the target.

[0052] Fourthly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the image encoder-text encoder joint training method based on contrastive learning as described above, or the steps of the 3D multimodal brain MRI image query method, or the steps of the target health status assessment method based on 3D multimodal brain MRI images.

[0053] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0054] This application provides a joint training method for image encoders and text encoders based on contrastive learning, an image query method, a health status assessment method, and an apparatus. Through image preprocessing and enhancement steps, it addresses the problems of inconsistent raw data quality, noise interference, and the difficulty of direct processing of 3D multimodal brain MRI images. Image preprocessing improves image quality, and data augmentation increases the diversity of training samples, enhancing the model's generalization ability. The image encoder model efficiently converts complex 3D images into low-dimensional one-dimensional image vectors, while the text encoder model converts Chinese medical reports into low-dimensional one-dimensional text vectors. Simultaneously, the modal descriptions in the Chinese medical reports are templated, simplifying the structure of the text data. By optimizing the parameters of the image encoder and text encoder models using a fusion loss function based on the semantic similarity matrix, image-text cosine similarity matrix, and text-image cosine similarity matrix, a well-trained image encoder and text encoder model are obtained. This achieves efficient and accurate matching and understanding between images and text, solving the problem of a large semantic gap and difficulty in establishing effective associations between multimodal image and text data. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is an application environment diagram of a joint training method for image encoder-text encoder based on contrastive learning in one embodiment of this application;

[0057] Figure 2 A flowchart illustrating a joint training method for an image encoder and text encoder based on contrastive learning, provided in an embodiment of this application;

[0058] Figure 3 This is a data processing method according to an embodiment of the present application. Figure 3 (a) Preprocessing methods for 3D multimodal brain MRI images. Figure 3 (b) A method for splitting Chinese medical reports into templated text data;

[0059] Figure 4 This is a comparative learning training framework according to an embodiment of this application;

[0060] Figure 5 This is a semantic similarity calculation method according to an embodiment of this application;

[0061] Figure 6This application presents a method for calculating the semantic similarity of multi-sentence long text samples according to an embodiment of the present application.

[0062] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] The image encoder-text encoder joint training method based on contrastive learning provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send 3D multimodal brain MRI images and corresponding Chinese medical reports to server 104. After receiving the 3D multimodal brain MRI images and corresponding Chinese medical reports, server 104 constructs a training sample set. Server 104 inputs each 3D multimodal brain MRI image from the current batch of training sample subset into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subset corresponds one-to-one with each Chinese medical report; each Chinese medical report in the current batch of training sample subset is input into the text encoder model to obtain the one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model; the similarity between each one-dimensional image vector and each one-dimensional text vector is calculated respectively, and an image-text similarity matrix and a text-image similarity matrix are constructed; wherein, the text-image similarity matrix is ​​the transpose of the image-text similarity matrix; the semantic similarity between any two Chinese medical reports in the current batch of training sample subset is calculated, and a semantic similarity matrix is ​​constructed. The semantic similarity matrix is ​​used; based on the image-text similarity matrix, text-image similarity matrix, and semantic similarity matrix, the loss for the current iteration is calculated using a fusion loss function, which includes contrastive learning loss and distribution loss; the parameters of the image encoder model and text encoder model are optimized based on the loss for the current iteration; it is determined whether the iteration termination condition has been met, and the determination result is obtained; if the determination result is yes, the optimized image encoder model and text encoder model are output as the trained image encoder model and text encoder model; if the determination result is no, the current batch of training samples is re-acquired in the training sample set, and the step of "inputting each 3D multimodal brain MRI image in the current batch of training sample subset into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model" is returned, and the next iteration optimization is performed.

[0066] Server 104 can feed back the trained image encoder model and text encoder model to terminal 102. Furthermore, in some embodiments, the image encoder-text encoder joint training method based on contrastive learning can also be implemented separately by server 104 or terminal 102. For example, terminal 102 can directly process the 3D multimodal brain MRI images and the corresponding Chinese medical reports, or server 104 can retrieve the 3D multimodal brain MRI images and the corresponding Chinese medical reports from the data storage system and process them.

[0067] In one exemplary embodiment, such as Figure 2 As shown, a joint training method for an image encoder and a text encoder based on contrastive learning is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 210. Wherein:

[0068] Step 201: Obtain 3D multimodal brain MRI images and corresponding Chinese medical reports to form a training sample set.

[0069] Step 202: Input each 3D multimodal brain MRI image in the current batch of training sample subset into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subset corresponds one-to-one with each Chinese medical report.

[0070] Step 203: Input each Chinese medical report in the current batch of training sample subset into the text encoder model to obtain the one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model.

[0071] Step 204: Calculate the similarity between each one-dimensional image vector and each one-dimensional text vector, and construct the image-text similarity matrix and the text-image similarity matrix; wherein the text-image similarity matrix is ​​the transpose of the image-text similarity matrix.

[0072] Step 205: Calculate the semantic similarity between any two Chinese medical reports in the training sample subset of the current batch, and construct a semantic similarity matrix.

[0073] Step 206: Based on the image-text similarity matrix, the text-image similarity matrix, and the semantic similarity matrix, calculate the loss for the current iteration using a fusion loss function; the fusion loss function includes contrastive learning loss and distribution loss.

[0074] Step 207: Optimize the parameters of the image encoder model and the text encoder model based on the loss of the current iteration.

[0075] Step 208: Determine whether the iteration termination condition has been met and obtain the determination result.

[0076] Step 209: If the judgment result is yes, then the image encoder model and text encoder model with optimized output parameters are used as the trained image encoder model and text encoder model.

[0077] Step 210: If the judgment result is negative, then re-acquire the current batch of training sample subsets in the training sample set, return to step 202, and perform the next iteration optimization.

[0078] By implementing steps 201 to 210 above, downstream tasks such as zero-shot classification and cross-modal mutual retrieval of 3D multimodal images and text can be achieved. The specific training process is as follows: Figure 4 As shown.

[0079] Among them, implementing this method, such as Figure 3 (a) and Figure 3 As shown in (b), 3D multimodal brain MRI images and corresponding Chinese medical reports are acquired to form a training sample set, which then includes:

[0080] 3D multimodal brain MRI images are acquired, and image preprocessing is performed on the 3D multimodal brain MRI images. The image preprocessing includes: resampling, Hu value truncation, maximum and minimum normalization, and data augmentation. Specifically, the data augmentation includes: image rotation, image scaling, elastic deformation, intensity modification, and contrast modification.

[0081] Break down multiple modalities in Chinese medical reports into descriptions of individual modalities.

[0082] Extract the modality name, modality location, modality part, and modality performance keywords from the description of each modality, and fill them into a template style to obtain a preprocessed Chinese medical report. The template style is as follows: in the modality {modal}, the {side} and {part} are represented by {performance}. Here, {modal} represents the extracted modality name, {side} represents the extracted modality location, {part} represents the extracted modality part, and {performance} represents the extracted modality performance keywords.

[0083] In another exemplary embodiment of this application, the formula for calculating semantic similarity is:

[0084] t sim =TextDice*(w loc +w per );

[0085]

[0086] Among them, t sim The semantic similarity between processed Chinese medical reports t1 and t2 is represented by TextDice, and the formal similarity between processed Chinese medical reports t1 and t2 is represented by Len. overlap Len indicates the number of overlapping characters between the processed Chinese medical reports t1 and t2. t1 This indicates the number of characters in the processed Chinese medical report t1, Len t2 w represents the number of characters in the processed Chinese medical report t2. loc This represents the keyword score of the lesion site obtained based on modal orientation and modal location, w per This represents the keyword score of the image performance obtained based on the modal performance keywords.

[0087] The formula for calculating the similarity between each one-dimensional image vector and each one-dimensional text vector is:

[0088]

[0089] in, This represents the image-text similarity between the i-th 3D multimodal brain MRI image and the j-th Chinese medical report in the current batch. This represents the i-th 3D multimodal brain MRI image in the current batch. This represents the j-th Chinese medical report in the current batch, v i Let t represent the i-th one-dimensional image vector in the current batch. j Let S(v) represent the j-th one-dimensional text vector in the current batch. i ,t j ) represents the cosine similarity between the i-th one-dimensional image vector and the j-th one-dimensional text vector in the current batch, exp(·) represents taking the exponent of the expression within the parentheses, τ represents the learnable temperature coefficient, and N is the preset batch size.

[0090] The formula for calculating the fusion loss function is:

[0091] L=αL clip +βL ts ;

[0092]

[0093] L v2t =CE(E N ,P v2t );

[0094] L t2v =CE(E N ,P t2v );

[0095]

[0096] L ts-v2t =KL1(T sim ,P v2t );

[0097] L ts-t2v =KL2(T sim ,P t2v );

[0098] Where L represents the fusion loss function, L clip This represents the contrastive learning loss, where α is the first weight and l ts This represents the distributed loss, where β is the second weight, and L... v2t L represents the image-text similarity loss. t2v E represents the text-image similarity loss, CE represents the cross-entropy loss function, and E N P represents the identity matrix. v2t Let P represent the image-text similarity matrix. t2v L represents the text-image similarity matrix. ts-v2t L represents the first intermediate function. ts-t2v T represents the second intermediate function. sim Let KL1(·,·) denote the KL divergence between the semantic similarity matrix within the brackets and the image-text similarity matrix, and KL2(·,·) denote the KL divergence between the semantic similarity matrix within the brackets and the text-image similarity matrix.

[0099] We consider {side} and {part} as keywords for lesion location, and {performance} as keywords for imaging manifestation. For lesion location keywords, we check whether they contain the same standard lesion location terminology; if the result is True, the keyword score is w. loc If the value is 0.5, and the result is False, then the keyword score is w. loc The score is 0; for image representation keywords, check whether they contain the same standard image representation terms. If the result is True, the keyword score is w. per If the value is 0.5, and the result is False, then the keyword score is w.per The value is 0. The calculation process for mode T2 is as follows: Figure 5 As shown.

[0100] In this implementation, the description of the single modality is given in multiple ways, and templated text data for each description is obtained and then concatenated.

[0101] For templated text data with multiple clauses and long durations, the semantic similarity between each pair of templated texts containing only one type of representation is calculated, and the average of all semantic similarities is taken as the semantic similarity of the templated text data with multiple clauses and long durations. For example, the modal ADC description is: ① In modal ADC, it shows a high signal in the right thalamus; ② In modal ADC, it shows a low signal in both occipital lobes. The modal T1 description is: ① In modal T1, it shows a long signal shadow in the right basal ganglia; ② In modal T1, it shows a long signal shadow in the right thalamus; ③ In modal T1, it shows a long signal shadow in the right ventricle. Figure 6 As shown, the semantic similarity of the multi-sentence long text templated text data of modal ADC and modal T1 is calculated.

[0102] In another exemplary embodiment of this application, a learning rate warm-up method is used to perform a total of 5000 iterations before the formal training of the image encoder model and the text encoder model. The learning rate is adjusted once for each iteration, and the adjustment formula for the learning rate is as follows:

[0103]

[0104] in, Let lr be the learning rate for the current iteration. init This represents the initial learning rate of the image encoder model or text encoder model, where t is the current iteration number. max This represents the total number of iterations.

[0105] During the training of both the image encoder and text encoder models, a polynomial learning rate descent strategy is used to update the learning rate. The initial learning rate (lr) for the image encoder model is... image The initial learning rate lr of the text encoder model is 0.0002. text It is 0.00005. The learning rate decreases continuously as the number of training rounds increases. The formula for updating the learning rate is as follows:

[0106]

[0107] Among them, lr (e)The learning rate is represented by 'e', ​​and the current period is 'e'. A period is defined as 250 iterations. max This represents the total number of cycles.

[0108] This application also provides an application scenario in which the above-mentioned image encoder-text encoder joint training method based on contrastive learning is applied. Specifically, the image encoder-text encoder joint training method based on contrastive learning provided in this embodiment can be applied in a medical image-assisted diagnosis scenario. This scenario includes an image acquisition stage, an image preprocessing stage, an image analysis stage, and a diagnostic report generation stage; image data enters the image preprocessing stage from the image acquisition stage, and then enters the image analysis stage after the preprocessing steps. The image encoder-text encoder joint training method based on contrastive learning provided in this embodiment belongs to the image analysis stage in medical image-assisted diagnosis. Specifically, in the analysis process, based on the preprocessed 3D multimodal brain MRI images and the corresponding Chinese medical reports, the trained image encoder model and text encoder model can be used to perform joint analysis of images and text. By calculating the similarity between one-dimensional image vectors and one-dimensional text vectors, the model can capture the potential correlation between images and text, thereby assisting doctors in making more accurate diagnoses and generating corresponding diagnostic reports.

[0109] This application addresses the challenges of high cost of medical image annotation, difficulty in processing 3D multimodal images, and high similarity and limited semantic content in Chinese medical reports. It proposes a joint training method for an image encoder and a text encoder based on contrastive learning, enabling the training of neural networks with unlabeled data. This application preprocesses multimodal images to a unified standard; decomposes Chinese medical reports into different modal representations and converts them to template styles; constructs a semantic similarity matrix to quantify the semantic similarity between different texts; uses ResNet (Residual Network) to encode the multimodal images and BERT (Bidirectional Encoder Representations from Transformers) to encode the corresponding text data; and calculates the loss for the current iteration using a fusion loss function based on the image-text similarity matrix, the text-image similarity matrix, and the semantic similarity matrix. The fusion loss function includes contrastive learning loss (Information Noise Contrastive Estimation, InfoNCE) and distribution loss to optimize the parameters of the image encoder model and the text encoder model. Through continuous iterative optimization, the joint training method proposed in this application has gradually endowed the image encoder model and text encoder model with strong zero-shot learning capabilities and transfer learning application capabilities. When faced with new and unseen medical images and reports, they can utilize learned knowledge and patterns for reasoning and judgment, assisting doctors in making rapid and accurate diagnoses. In summary, this application, to a certain extent, solves the problems of high cost of medical image annotation, difficulty in 3D multimodal image processing, and high similarity and limited semantic content in Chinese medical reports, providing an efficient and accurate solution for the field of medical image-assisted diagnosis.

[0110] In one exemplary embodiment, a method for querying 3D multimodal brain MRI images is provided, specifically including:

[0111] The trained image encoder model is used to process each 3D multimodal brain MRI image stored in the database to obtain a one-dimensional image vector corresponding to each image.

[0112] The system receives query conditions input by the user, performs text preprocessing on the query conditions, and inputs the preprocessed query conditions into the trained text encoder model to obtain the target one-dimensional text vector.

[0113] The similarity between the target one-dimensional text vector and each one-dimensional image vector is calculated and compared. The 3D multimodal brain MRI image corresponding to the one-dimensional image vector with the highest similarity value is selected as the query result.

[0114] In another exemplary embodiment, a method for assessing the health status of a target based on 3D multimodal brain MRI images is provided, specifically including:

[0115] Construct a target health state model, which includes a trained image encoder model and a binary classification model.

[0116] 3D multimodal brain MRI images containing both target and non-target health states are input into a trained image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image.

[0117] The one-dimensional image vectors corresponding to each 3D multimodal brain MRI image are used as training samples to train the binary classification model, thereby obtaining a trained binary classification model to assess the target's health status.

[0118] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores video tag processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a contrastive learning-based image encoder-text encoder joint training method, an image query method, or a health status assessment method.

[0119] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0120] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0123] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A joint training method for an image encoder and a text encoder based on contrastive learning, characterized in that, The image encoder-text encoder joint training method based on contrastive learning includes: 3D multimodal brain MRI images and corresponding Chinese medical reports were obtained to form a training sample set; Each 3D multimodal brain MRI image in the current batch of training sample subset is input into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subset corresponds one-to-one with each Chinese medical report; Input each Chinese medical report in the current batch of training sample subset into the text encoder model to obtain the one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model; Calculate the similarity between each one-dimensional image vector and each one-dimensional text vector, and construct the image-text similarity matrix and the text-image similarity matrix; wherein the text-image similarity matrix is ​​the transpose of the image-text similarity matrix; Calculate the semantic similarity between any two Chinese medical reports in the training sample subset of the current batch, and construct a semantic similarity matrix; Based on the image-text similarity matrix, text-image similarity matrix, and semantic similarity matrix, the loss for the current iteration is calculated using a fusion loss function; the fusion loss function includes contrastive learning loss and distribution loss. The parameters of the image encoder model and the text encoder model are optimized based on the loss of the current iteration; Determine whether the iteration termination condition has been met and obtain the determination result; If the judgment result is yes, then the image encoder model and text encoder model with optimized output parameters are used as the trained image encoder model and text encoder model. If the judgment result is negative, then the current batch of training sample subsets is re-acquired in the training sample set, and the step of "inputting each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model" is returned, and the next iteration optimization is performed.

2. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that, The training sample set is constructed by acquiring 3D multimodal brain MRI images and corresponding Chinese medical reports. This process also includes: 3D multimodal brain MRI images are acquired, and image preprocessing is performed on the 3D multimodal brain MRI images. The image preprocessing includes: resampling, Hu value truncation, maximum and minimum normalization, and data augmentation. Specifically, the data augmentation includes: image rotation, image scaling, elastic deformation, intensity modification, and contrast modification. Break down multiple modalities in Chinese medical reports into descriptions of individual modalities; Extract the modality name, modality location, modality part, and modality performance keywords from the description of each modality, and fill them into a template style to obtain a preprocessed Chinese medical report. The template style is as follows: in the modality {modal}, the {side} and {part} are represented by {performance}. Here, {modal} represents the extracted modality name, {side} represents the extracted modality location, {part} represents the extracted modality part, and {performance} represents the extracted modality performance keywords.

3. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that, The formula for calculating semantic similarity is: ； ； in, This indicates the processed Chinese medical report. t 1 and t Semantic similarity between 2 This indicates the processed Chinese medical report. t 1 and t Formal similarity between 2 This indicates the processed Chinese medical report. t 1 and t The number of overlapping characters between 2, This indicates the processed Chinese medical report. t 1 character count, This indicates the processed Chinese medical report. t 2 characters This represents the keyword score for the lesion location obtained based on modal orientation and modal location. This represents the keyword score of the image performance obtained based on modal performance keywords.

4. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that, The formula for calculating the similarity between each one-dimensional image vector and each one-dimensional text vector is: ; in, This represents the image-text similarity between the i-th 3D multimodal brain MRI image and the j-th Chinese medical report in the current batch. This represents the i-th 3D multimodal brain MRI image in the current batch. This represents the j-th Chinese medical report in the current batch. This represents the i-th one-dimensional image vector in the current batch. This represents the j-th one-dimensional text vector in the current batch. This represents the cosine similarity between the i-th one-dimensional image vector and the j-th one-dimensional text vector in the current batch. This indicates that the expression within the parentheses is being exponentially evaluated. This represents the learnable temperature coefficient, and N is the preset batch size.

5. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that, The formula for calculating the fusion loss function is as follows: ； ； ； ； ； ； ； Where L represents the fusion loss function, Indicates the contrast learning loss. As the first weight, Indicates distributed loss. As the second weight, This represents the image-text similarity loss. The text-image similarity loss is represented by CE, which stands for cross-entropy loss function. Represents the identity matrix. Represents the image-text similarity matrix. Represents the text-image similarity matrix. This represents the first intermediate function. This represents the second intermediate function. Represents the semantic similarity matrix. This indicates the calculation of the KL divergence between the semantic similarity matrix within the parentheses and the image-text similarity matrix. This indicates the calculation of the KL divergence between the semantic similarity matrix within the brackets and the text-image similarity matrix.

6. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that, The process of inputting each 3D multimodal brain MRI image from the current batch of training samples into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model also includes: performing a learning rate warm-up method for a total of 5000 iterations, adjusting the learning rate once for each iteration, and the adjustment formula for the learning rate is as follows: ； in, The learning rate for the current iteration. This represents the initial learning rate of the image encoder model or the text encoder model. This represents the current iteration number. This represents the total number of iterations.

7. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that, Optimizing the parameters of the image encoder model and the text encoder model also includes updating the learning rate during training using a multinomial learning rate descent strategy. The update formula for the learning rate is as follows: ； in, This represents the updated learning rate. This indicates the current period, and defines 250 iterations as one period. This represents the total number of cycles.

8. A method for querying 3D multimodal brain MRI images, characterized in that, The image encoder model and the text encoder model are trained based on the image encoder-text encoder joint training method based on contrastive learning as described in claim 1. The 3D multimodal brain MRI image query method includes: The trained image encoder model is used to process each 3D multimodal brain MRI image stored in the database to obtain a one-dimensional image vector corresponding to each image. The system receives query conditions input by the user, performs text preprocessing on the query conditions, and inputs the preprocessed query conditions into the trained text encoder model to obtain the target one-dimensional text vector. The similarity between the target one-dimensional text vector and each one-dimensional image vector is calculated and compared. The 3D multimodal brain MRI image corresponding to the one-dimensional image vector with the highest similarity value is selected as the query result.

9. A method for assessing target health status based on 3D multimodal brain MRI images, characterized in that, The image encoder model is trained based on the image encoder-text encoder joint training method based on contrastive learning as described in claim 1, and the target health status assessment method based on 3D multimodal brain MRI images includes: Construct a target health status assessment model, which includes a trained image encoder model and a binary classification model; 3D multimodal brain MRI images containing target and non-target health states are input into a trained image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image. The one-dimensional image vectors corresponding to each 3D multimodal brain MRI image are used as training samples to train the binary classification model, thereby obtaining a trained binary classification model to assess the target's health status.

10. A computer device, comprising: The memory, the processor, and the computer program stored in the memory and executable on the processor are characterized in that the processor executes the computer program to implement the image encoder-text encoder joint training method based on contrastive learning according to any one of claims 1-7, or the 3D multimodal brain MRI image query method according to claim 8, or the target health status assessment method based on 3D multimodal brain MRI images according to claim 9.

Images