Comparative learning-based image encoder-text encoder joint training method, image query method, health status evaluation method and equipment

Abstract

The invention discloses an image encoder-text encoder joint training method, an image query method, a health state evaluation method and equipment based on comparative learning, and relates to the field of medical image understanding. The method comprises the following steps: respectively inputting each 3D multi-modal brain MRI image and each Chinese medical report into an image encoder model and a text encoder model to obtain a one-dimensional image vector and a one-dimensional text vector; according to the image-text similarity matrix, the text-image similarity matrix and the semantic similarity matrix, using a fusion loss function to calculate the loss of the current iteration, and optimizing the parameters of the image encoder model and the text encoder model; and obtaining a trained image encoder model and a trained text encoder model when an iteration ending condition is met. Downstream tasks such as zero sample classification and cross-modal mutual retrieval of images and texts are realized.

Description

Technical Field

[0001] The present application relates to the field of medical image understanding, and in particular to an image encoder-text encoder joint training method based on contrastive learning, a 3D multimodal brain MRI image query method, and a target health status assessment method and device based on 3D multimodal brain MRI images. Background Art

[0002] In recent years, the rapid development of artificial intelligence technology has brought revolutionary changes to all walks of life, especially in the fields of computer vision and natural language processing. The rise of deep learning technology has led the trend of technological innovation. In the medical field, this technology is widely used in medical image analysis, auxiliary diagnosis, and automatic generation of imaging reports, which significantly improves the efficiency and accuracy of medical services. However, most of the current deep learning-based medical image processing requires a large amount of manual annotation as labels to train neural networks. For medical images, especially 3D images and multimodal images, the amount of data is relatively scarce, and the annotations need to be manually made by professional doctors.

[0003] In the specific field of medical image processing, brain MRI (Magnetic Resonance Imaging) images are the key basis for diagnosis, and the complexity of their processing and analysis is particularly prominent. Since brain MRI images usually contain multiple modalities (such as T1 (T1-weighted imaging, T1-weighted imaging), T2 (T2-weighted imaging, T2-weighted imaging, ADC (Apparent Diffusion Coefficient, DWI (Diffusion-weighted Imaging, diffusion-weighted imaging)), etc.), each modality provides unique anatomical and pathological information. Therefore, how to effectively fuse these multi-modal information while retaining the spatial integrity of 3D images has become one of the current technical challenges. Most of the existing contrastive learning techniques focus on 2D image processing. When 3D images are simply divided into 2D slice sequences for processing, important spatial structural information will inevitably be lost, which will affect the accuracy of diagnosis.

[0004] In addition, natural language processing of medical reports also faces unique challenges. Due to the high similarity, unified semantic content and complexity of professional terminology in Chinese medical reports, it is difficult to directly apply traditional natural language processing technology and achieve ideal results. Summary of the invention

[0005] The purpose of this application is to provide an image encoder-text encoder joint training method, image query method, health status assessment method and device based on contrastive learning, which can realize downstream tasks such as zero-sample classification and cross-modal mutual retrieval of 3D multimodal images and texts.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides an image encoder-text encoder joint training method based on contrastive learning, comprising:

[0008] Acquire 3D multimodal brain MRI images and Chinese medical reports corresponding to the 3D multimodal brain MRI images to form a training sample set;

[0009] Input each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subsets corresponds to each Chinese medical report one by one;

[0010] Input each Chinese medical report in the current batch of training sample subsets into the text encoder model to obtain a one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model;

[0011] Calculate the similarity between each one-dimensional image vector and each one-dimensional text vector respectively, and construct an image-text similarity matrix and a text-image similarity matrix; wherein the text-image similarity matrix is ​​the transpose of the image-text similarity matrix;

[0012] Calculate the semantic similarity between any two Chinese medical reports in the current batch of training sample subsets and construct a semantic similarity matrix;

[0013] According to the image-text similarity matrix, the text-image similarity matrix and the semantic similarity matrix, the loss of the current iteration is calculated using a fusion loss function; the fusion loss function includes contrastive learning loss and distribution loss;

[0014] Optimize the parameters of the image encoder model and the text encoder model according to the loss of the current iteration;

[0015] Determine whether the iteration end condition is reached and obtain the determination result;

[0016] If the judgment result is yes, outputting the image encoder model and the text encoder model after parameter optimization as the trained image encoder model and the text encoder model;

[0017] If the judgment result is no, then re-obtain the training sample subset of the current batch in the training sample set, return to the step of "inputting each 3D multimodal brain MRI image in the training sample subset of the current batch into the image encoder model, and obtaining the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model" to perform the next iterative optimization.

[0018] Optionally, a 3D multimodal brain MRI image and a Chinese medical report corresponding to the 3D multimodal brain MRI image are obtained to form a training sample set, and then the following is further included:

[0019] Acquire a 3D multimodal brain MRI image, and perform image preprocessing on the 3D multimodal brain MRI image, wherein the image preprocessing includes: resampling, Hu value truncation, maximum and minimum normalization, and data enhancement; wherein the data enhancement specifically includes: image rotation, image scaling, elastic deformation, intensity change, and contrast change;

[0020] Split multiple modalities in Chinese medical reports into descriptions of a single modality;

[0021] The modal name, modal orientation, modal part, and modal performance keywords in the description of each modality are extracted, and filled into a template style to obtain a Chinese medical report after text preprocessing; wherein the template style is: in modality {modal}, in {side}{part} is performed as {performance}; wherein {modal} represents the extracted modal name, {side} represents the extracted modal orientation, {part} represents the extracted modal part, and {performance} represents the extracted modal performance keywords.

[0022] Optionally, the calculation formula of the semantic similarity is:

[0023] t sim =TextDice*(w loc +w per );

[0024]

[0025] Among them, t sim represents the semantic similarity between the processed Chinese medical reports t1 and t2, TextDice represents the formal similarity between the processed Chinese medical reports t1 and t2, Len overlap Indicates the number of overlapping characters between the processed Chinese medical reports t1 and t2, Len t1 Indicates the number of characters in the processed Chinese medical report t1, Len t2 represents the number of characters w in the processed Chinese medical report t2loc represents the keyword score of the lesion site according to the modal orientation and modal location, w per Indicates the keyword score of the image expression obtained based on the modality expression keyword.

[0026] Optionally, the similarity between each one-dimensional image vector and each one-dimensional text vector is calculated as follows:

[0027]

[0028] in, represents the image-text similarity between the i-th 3D multimodal brain MRI image and the j-th Chinese medical report in the current batch, represents the i-th 3D multimodal brain MRI image in the current batch, represents the jth Chinese medical report in the current batch, v i represents the i-th one-dimensional image vector in the current batch, t j represents the j-th one-dimensional text vector in the current batch, S(v i ,t j ) represents the cosine similarity between the i-th one-dimensional image vector and the j-th one-dimensional text vector in the current batch, exp(·) represents the exponential of the expression in the brackets, τ represents the learnable temperature coefficient, and N is the preset batch size.

[0029] Optionally, the calculation formula of the fusion loss function is:

[0030] L=αL clip +βL ts ;

[0031]

[0032] L v2t =CE(E N ,P v2t );

[0033] L t2v =CE(E N ,P t2v );

[0034]

[0035] L ts-v2t =KL1(T sim ,P v2t );

[0036] L ts-t2v =KL2(T sim ,P t2v );

[0037] Among them, L represents the fusion loss function, L clip represents contrastive learning loss, α is the first weight, L ts represents the distribution loss, β is the second weight, L v2t represents the image-text similarity loss, L t2v represents the text-image similarity loss, CE represents the cross entropy loss function, E N represents the identity matrix, P v2t represents the image-text similarity matrix, P t2v represents the text-image similarity matrix, L ts-v2t represents the first intermediate function, L ts-t2v represents the second intermediate function, T sim represents the semantic similarity matrix, KL1(·,·) represents the KL divergence between the semantic similarity matrix in the brackets and the image-text similarity matrix, and KL2(·,·) represents the KL divergence between the semantic similarity matrix in the brackets and the text-image similarity matrix.

[0038] Optionally, the method of inputting each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model also includes: using a learning rate warm-up method for a total of 5000 iterations, adjusting the learning rate once per iteration, and the learning rate adjustment formula is as follows:

[0039]

[0040] in, is the learning rate of the current iteration, lr init represents the initial learning rate of the image encoder model or text encoder model, t is the current number of iterations, and t max is the total number of iterations.

[0041] Optionally, optimizing the parameters of the image encoder model and the text encoder model further includes: updating the learning rate using a polynomial learning rate descent strategy during the training process, and the learning rate update formula is as follows:

[0042]

[0043] Among them, lr (e) represents the updated learning rate, e represents the current cycle, and 250 iterations are defined as one cycle. max is the total number of cycles.

[0044] In a second aspect, the present application provides a 3D multimodal brain MRI image query method, comprising:

[0045] Using the trained image encoder model, each 3D multimodal brain MRI image stored in the database is processed to obtain a one-dimensional image vector corresponding to each image;

[0046] Receive the query condition input by the user, perform text preprocessing on the query condition, and input the query condition after text preprocessing into the trained text encoder model to obtain the target one-dimensional text vector;

[0047] The similarity between the target one-dimensional text vector and each one-dimensional image vector is calculated and compared respectively, and the 3D multimodal brain MRI image corresponding to the one-dimensional image vector with the largest similarity value is selected as the query result.

[0048] In a third aspect, the present application provides a target health status assessment method based on 3D multimodal brain MRI images, comprising:

[0049] Constructing a target health status evaluation model, wherein the target health status evaluation model includes a trained image encoder model and a binary classification model;

[0050] Inputting 3D multimodal brain MRI images including target health states and non-target health states into the trained image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image;

[0051] The one-dimensional image vectors corresponding to each 3D multimodal brain MRI image are used as training samples to train the binary classification model to obtain a trained binary classification model, thereby realizing the evaluation of the target health status.

[0052] In a fourth aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the image encoder-text encoder joint training method based on contrastive learning, or the steps of the 3D multimodal brain MRI image query method, or the steps of the target health status assessment method based on 3D multimodal brain MRI images as described above.

[0053] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0054] The present application provides an image encoder-text encoder joint training method, an image query method, a health status assessment method and a device based on contrastive learning. Through image preprocessing and enhancement steps, the problems of uneven quality of raw data of 3D multimodal brain MRI images, noise interference and difficulty in direct processing are solved. Image preprocessing improves image quality, and increases the diversity of training samples through data enhancement, thereby improving the generalization ability of the model. The image encoder model is used to efficiently convert complex 3D images into low-dimensional one-dimensional image vectors, and the text encoder model is used to convert Chinese medical reports into low-dimensional one-dimensional text vectors. At the same time, the modal descriptions in the Chinese medical reports are templated to simplify the structure of text data. By optimizing the parameters of the image encoder model and the text encoder model according to the semantic similarity matrix, the image-text cosine similarity matrix and the text-image cosine similarity matrix, and using the fusion loss function, the trained image encoder model and the text encoder model are obtained, which realizes efficient and accurate matching and understanding between images and texts, and solves the problem that the semantic gap between multimodal images and text data is large and it is difficult to establish effective associations. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 This is an application environment diagram of an image encoder-text encoder joint training method based on contrastive learning in an embodiment of the present application;

[0057] Figure 2 A flowchart of a method for joint training of an image encoder and a text encoder based on contrastive learning provided in one embodiment of the present application;

[0058] Figure 3 A data processing method according to an embodiment of the present application is provided. Figure 3 (a) is the preprocessing method of 3D multimodal brain MRI images. Figure 3 (b) A method for splitting Chinese medical reports into templated text data;

[0059] Figure 4 A contrastive learning training framework according to an embodiment of the present application;

[0060] Figure 5 A method for calculating semantic similarity according to an embodiment of the present application;

[0061] Figure 6A method for calculating semantic similarity of a multi-sentence long text sample according to an embodiment of the present application;

[0062] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0065] The image encoder-text encoder joint training method based on contrastive learning provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the 3D multimodal brain MRI image and the Chinese medical report corresponding to the 3D multimodal brain MRI image to the server 104. After the server 104 receives the 3D multimodal brain MRI image and the Chinese medical report corresponding to the 3D multimodal brain MRI image, a training sample set is formed. The server 104 inputs each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subsets corresponds to each Chinese medical report one by one; each Chinese medical report in the current batch of training sample subsets is input into the text encoder model to obtain a one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model; the similarity between each one-dimensional image vector and each one-dimensional text vector is calculated respectively, and an image-text similarity matrix and a text-image similarity matrix are constructed; wherein the text-image similarity matrix is ​​the transpose of the image-text similarity matrix; the semantic similarity between any two Chinese medical reports in the current batch of training sample subsets is calculated, and a text-image similarity matrix is ​​constructed. semantic similarity matrix; according to the image-text similarity matrix, the text-image similarity matrix and the semantic similarity matrix, the loss of the current iteration is calculated using the fusion loss function; the fusion loss function includes contrastive learning loss and distribution loss; according to the loss of the current iteration, the parameters of the image encoder model and the text encoder model are optimized; whether the iteration end condition is met, and a judgment result is obtained; if the judgment result is yes, the image encoder model and the text encoder model with optimized parameters are output as the trained image encoder model and the text encoder model; if the judgment result is no, the training sample subset of the current batch is re-acquired in the training sample set, and the step of "inputting each 3D multimodal brain MRI image in the training sample subset of the current batch into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model" is returned to perform the next iterative optimization.

[0066] The server 104 can feed back the obtained trained image encoder model and text encoder model to the terminal 102. In addition, in some embodiments, the image encoder-text encoder joint training method based on contrastive learning can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly process the 3D multimodal brain MRI image and the Chinese medical report corresponding to the 3D multimodal brain MRI image, or the server 104 can obtain the 3D multimodal brain MRI image and the Chinese medical report corresponding to the 3D multimodal brain MRI image from the data storage system, and process the 3D multimodal brain MRI image and the Chinese medical report corresponding to the 3D multimodal brain MRI image.

[0067] In an exemplary embodiment, Figure 2 As shown, a method for joint training of an image encoder and a text encoder based on contrastive learning is provided. The method is executed by a computer device, and specifically can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 210. Among them:

[0068] Step 201 : Acquire 3D multimodal brain MRI images and Chinese medical reports corresponding to the 3D multimodal brain MRI images to form a training sample set.

[0069] Step 202, input each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model, and obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subsets corresponds to each Chinese medical report one by one.

[0070] Step 203: input each Chinese medical report in the current batch of training sample subsets into the text encoder model to obtain a one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model.

[0071] Step 204, respectively calculate the similarity between each one-dimensional image vector and each one-dimensional text vector, and construct an image-text similarity matrix and a text-image similarity matrix; wherein the text-image similarity matrix is ​​the transpose of the image-text similarity matrix.

[0072] Step 205 , calculating the semantic similarity between any two Chinese medical reports in the current batch of training sample subsets, and constructing a semantic similarity matrix.

[0073] Step 206, calculating the loss of the current iteration using a fusion loss function according to the image-text similarity matrix, the text-image similarity matrix and the semantic similarity matrix; the fusion loss function includes contrastive learning loss and distribution loss.

[0074] Step 207, optimizing the parameters of the image encoder model and the text encoder model according to the loss of the current iteration.

[0075] Step 208, determine whether the iteration end condition is met and obtain a determination result.

[0076] Step 209: If the judgment result is yes, the image encoder model and the text encoder model after the parameter optimization are output as the trained image encoder model and the text encoder model.

[0077] Step 210: If the judgment result is no, then the training sample subset of the current batch is re-obtained from the training sample set, and the process returns to step 202 to perform the next iterative optimization.

[0078] By implementing the above steps 201 to 210, downstream tasks such as zero-shot classification and cross-modal mutual retrieval of 3D multimodal images and texts can be realized. The specific training process is as follows: Figure 4 shown.

[0079] Among them, implementing this embodiment, such as Figure 3 (a) and Figure 3 As shown in (b), a 3D multimodal brain MRI image and a Chinese medical report corresponding to the 3D multimodal brain MRI image are obtained to form a training sample set, and then the following is also included:

[0080] Acquire 3D multimodal brain MRI images, and perform image preprocessing on the 3D multimodal brain MRI images, wherein the image preprocessing includes: resampling, Hu value truncation, maximum and minimum normalization, and data enhancement; wherein the data enhancement specifically includes: image rotation, image scaling, elastic deformation, intensity change, and contrast change.

[0081] Description of splitting multiple modalities in Chinese medical reports into a single modality.

[0082] The modal name, modal orientation, modal part, and modal performance keywords in the description of each modality are extracted, and filled into a template style to obtain a Chinese medical report after text preprocessing; wherein the template style is: in modality {modal}, in {side}{part} is performed as {performance}; wherein {modal} represents the extracted modal name, {side} represents the extracted modal orientation, {part} represents the extracted modal part, and {performance} represents the extracted modal performance keywords.

[0083] In another exemplary embodiment of the present application, the calculation formula of semantic similarity is:

[0084] t sim =TextDice*(w loc +w per );

[0085]

[0086] Among them, t sim represents the semantic similarity between the processed Chinese medical reports t1 and t2, TextDice represents the formal similarity between the processed Chinese medical reports t1 and t2, Len overlap Indicates the number of overlapping characters between the processed Chinese medical reports t1 and t2, Len t1 Indicates the number of characters in the processed Chinese medical report t1, Len t2 represents the number of characters in the processed Chinese medical report t2, w loc represents the keyword score of the lesion site according to the modal orientation and modal location, w per Indicates the keyword score of the image expression obtained based on the modality expression keyword.

[0087] The calculation formula for the similarity between each one-dimensional image vector and each one-dimensional text vector is:

[0088]

[0089] in, represents the image-text similarity between the i-th 3D multimodal brain MRI image and the j-th Chinese medical report in the current batch, represents the i-th 3D multimodal brain MRI image in the current batch, represents the jth Chinese medical report in the current batch, v i represents the i-th one-dimensional image vector in the current batch, t j represents the j-th one-dimensional text vector in the current batch, S(v i ,t j ) represents the cosine similarity between the i-th one-dimensional image vector and the j-th one-dimensional text vector in the current batch, exp(·) represents the exponential of the expression in the brackets, τ represents the learnable temperature coefficient, and N is the preset batch size.

[0090] The calculation formula of the fusion loss function is:

[0091] L=αL clip +βL ts ;

[0092]

[0093] L v2t =CE(E N ,P v2t );

[0094] L t2v =CE(E N ,P t2v );

[0095]

[0096] L ts-v2t =KL1(T sim ,P v2t );

[0097] L ts-t2v =KL2(T sim ,P t2v );

[0098] Among them, L represents the fusion loss function, L clip represents contrastive learning loss, α is the first weight, l ts represents the distribution loss, β is the second weight, L v2t represents the image-text similarity loss, L t2v represents the text-image similarity loss, CE represents the cross entropy loss function, E N represents the identity matrix, P v2t represents the image-text similarity matrix, P t2v represents the text-image similarity matrix, L ts-v2t represents the first intermediate function, L ts-t2v represents the second intermediate function, T sim represents the semantic similarity matrix, KL1(·,·) represents the KL divergence between the semantic similarity matrix in the brackets and the image-text similarity matrix, and KL2(·,·) represents the KL divergence between the semantic similarity matrix in the brackets and the text-image similarity matrix.

[0099] {side}{part} is considered as the keyword of the lesion site, and {performance} is considered as the keyword of the image performance. For the lesion site keyword, check whether it contains the same standard lesion site noun. If the result is True, the keyword score w loc is 0.5, and if the result is False, the keyword score w loc is 0; for the image expression keywords, check whether they contain the same standard image expression nouns. If the result is True, the keyword score w per is 0.5, and if the result is False, the keyword score wper is 0. The calculation process for mode T2 is as follows Figure 5 shown.

[0100] In this implementation, when there are multiple descriptions of the single modality, templated text data for each description is obtained and concatenated.

[0101] For multi-sentence long text templated text data, calculate the templated texts of all simple text descriptions that only contain one expression description, calculate the semantic similarity between each two, and calculate the average of all semantic similarities as the semantic similarity of the multi-sentence long text templated text data. For example, the description of modality ADC is: ① In modality ADC, it shows a high signal in the right thalamus; ② In modality ADC, it shows a low signal in the bilateral occipital lobes. The description of modality T1 is: ① In modality T1, it shows a long signal shadow in the right basal ganglia; ② In modality T1, it shows a long signal shadow in the right thalamus; ③ In modality T1, it shows a long signal shadow in the right ventricle. Figure 6 As shown, the semantic similarity of multi-sentence long text templated text data of modality ADC and modality T1 is calculated.

[0102] In another exemplary embodiment of the present application, a learning rate warm-up method is used. Before the image encoder model and the text encoder model are formally trained, a total of 5000 iterations are performed, and the learning rate is adjusted once per iteration. The adjustment formula of the learning rate is as follows:

[0103]

[0104] in, is the learning rate of the current iteration, lr init represents the initial learning rate of the image encoder model or text encoder model, t is the current number of iterations, and t max is the total number of iterations.

[0105] During the training of the image encoder model and the text encoder model, the polynomial learning rate descent strategy (Poly Learning Rate Policy) is also used to update the learning rate. The initial learning rate lr of the image encoder model is image is 0.0002, the initial learning rate lr of the text encoder model text is 0.00005. The learning rate decays as the number of training rounds increases. The learning rate update formula is as follows:

[0106]

[0107] Among them, lr (e)represents the updated learning rate, e represents the current cycle, and 250 iterations are defined as one cycle. max is the total number of cycles.

[0108] The present application also provides an application scenario, which applies the above-mentioned image encoder-text encoder joint training method based on contrastive learning. Specifically: the image encoder-text encoder joint training method based on contrastive learning provided in this embodiment can be applied in the medical image-assisted diagnosis scenario. The scenario includes an image acquisition link, an image preprocessing link, an image analysis link, and a diagnosis report generation link; the image data enters the image preprocessing link from the image acquisition link, and enters the image analysis link after the preprocessing step. The image encoder-text encoder joint training method based on contrastive learning provided in this embodiment belongs to the image analysis link in medical image-assisted diagnosis. Specifically, in the analysis process, the image and text can be jointly analyzed using the trained image encoder model and text encoder model based on the preprocessed 3D multimodal brain MRI image and the corresponding Chinese medical report. By calculating the similarity between the one-dimensional image vector and the one-dimensional text vector, the model can capture the potential correlation between the image and the text, thereby assisting doctors to make a more accurate diagnosis and generate a corresponding diagnosis report.

[0109] In response to the challenges of high cost of medical image annotation, difficulty in processing 3D multimodal images, high similarity and single semantic content of Chinese medical reports, this application proposes a joint training method of image encoder and text encoder based on contrastive learning, which realizes the training of neural networks with unlabeled data. This application preprocesses multimodal images to a unified standard; splits Chinese medical reports into different modal expressions and converts them into template styles; constructs a semantic similarity matrix to quantify the semantic similarity between different texts; uses ResNet (Residual Network) to encode multimodal images and uses BERT (Bidirectional Encoder Representations from Transformers) to encode corresponding text data; calculates the loss of the current iteration based on the image-text similarity matrix, text-image similarity matrix and semantic similarity matrix using a fusion loss function; the fusion loss function includes contrastive learning loss (Information Noise Contrastive Estimation, InfoNCE) and distribution loss to optimize the parameters of the image encoder model and the text encoder model. Through continuous iterative optimization, the joint training method proposed in this application has gradually enabled the image encoder model and the text encoder model to have strong zero-sample learning capabilities and transfer learning application capabilities. When faced with new, unseen medical images and reports, the learned knowledge and patterns can be used for reasoning and judgment to assist doctors in making quick and accurate diagnoses. In summary, this application has, to a certain extent, solved the problems of high medical image annotation costs, difficulties in 3D multimodal image processing, and high similarity and single semantic content in Chinese medical reports, providing an efficient and accurate solution for the field of medical image-assisted diagnosis.

[0110] In an exemplary embodiment, a 3D multimodal brain MRI image query method is provided, which specifically includes:

[0111] Using the trained image encoder model, each 3D multimodal brain MRI image stored in the database is processed to obtain a one-dimensional image vector corresponding to each image.

[0112] Receive the query conditions input by the user, perform text preprocessing on the query conditions, and input the query conditions after text preprocessing into the trained text encoder model to obtain the target one-dimensional text vector.

[0113] The similarity between the target one-dimensional text vector and each one-dimensional image vector is calculated and compared respectively, and the 3D multimodal brain MRI image corresponding to the one-dimensional image vector with the largest similarity value is selected as the query result.

[0114] In another exemplary embodiment, a method for evaluating the health status of a target based on 3D multimodal brain MRI images is provided, specifically comprising:

[0115] A target health status model is constructed, wherein the target health status model includes a trained image encoder model and a binary classification model.

[0116] The 3D multimodal brain MRI images containing the target health state and the non-target health state are input into the trained image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image.

[0117] The one-dimensional image vectors corresponding to each 3D multimodal brain MRI image are used as training samples to train the binary classification model to obtain a trained binary classification model, thereby realizing the evaluation of the target health status.

[0118] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video tag processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for joint training of an image encoder-text encoder based on contrastive learning, an image query method or a health status assessment method is implemented.

[0119] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0122] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0123] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0124] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for joint training of image encoder and text encoder based on contrastive learning, characterized in that: The image encoder-text encoder joint training method based on contrastive learning includes: Acquire 3D multimodal brain MRI images and Chinese medical reports corresponding to the 3D multimodal brain MRI images to form a training sample set; Input each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model; the training sample subset is obtained from the training sample set; each 3D multimodal brain MRI image in the current batch of training sample subsets corresponds to each Chinese medical report one by one; Input each Chinese medical report in the current batch of training sample subsets into the text encoder model to obtain a one-dimensional text vector corresponding to each Chinese medical report output by the text encoder model; Calculate the similarity between each one-dimensional image vector and each one-dimensional text vector respectively, and construct an image-text similarity matrix and a text-image similarity matrix; wherein the text-image similarity matrix is ​​the transpose of the image-text similarity matrix; Calculate the semantic similarity between any two Chinese medical reports in the current batch of training sample subsets and construct a semantic similarity matrix; According to the image-text similarity matrix, the text-image similarity matrix and the semantic similarity matrix, the loss of the current iteration is calculated using a fusion loss function; the fusion loss function includes contrastive learning loss and distribution loss; Optimize the parameters of the image encoder model and the text encoder model according to the loss of the current iteration; Determine whether the iteration end condition is met and obtain the determination result; If the judgment result is yes, outputting the image encoder model and the text encoder model after parameter optimization as the trained image encoder model and the text encoder model; If the judgment result is no, then the training sample subset of the current batch is re-obtained from the training sample set, and the step of "inputting each 3D multimodal brain MRI image in the training sample subset of the current batch into the image encoder model to obtain the one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model" is returned to perform the next iterative optimization.

2. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that: Obtain 3D multimodal brain MRI images and Chinese medical reports corresponding to the 3D multimodal brain MRI images to form a training sample set, which also includes: Acquire a 3D multimodal brain MRI image, and perform image preprocessing on the 3D multimodal brain MRI image, wherein the image preprocessing includes: resampling, Hu value truncation, maximum and minimum normalization, and data enhancement; wherein the data enhancement specifically includes: image rotation, image scaling, elastic deformation, intensity change, and contrast change; Split multiple modalities in Chinese medical reports into descriptions of a single modality; The modal name, modal orientation, modal part, and modal performance keywords in the description of each modality are extracted, and filled into a template style to obtain a Chinese medical report after text preprocessing; wherein the template style is: in modality {modal}, in {side}{part} is performed as {performance}; wherein {modal} represents the extracted modal name, {side} represents the extracted modal orientation, {part} represents the extracted modal part, and {performance} represents the extracted modal performance keywords.

3. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that: The calculation formula of the semantic similarity is: t sim =TextDice*(w loc +w per ); Among them, t sim represents the semantic similarity between the processed Chinese medical reports t1 and t2, TextDice represents the formal similarity between the processed Chinese medical reports t1 and t2, Len overlap Indicates the number of overlapping characters between the processed Chinese medical reports t1 and t2, Len t1 Indicates the number of characters in the processed Chinese medical report t1, Len t2 represents the number of characters in the processed Chinese medical report t2, w loc represents the keyword score of the lesion site according to the modal orientation and modal location, w per Indicates the keyword score of the image expression obtained based on the modality expression keyword.

4. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that: The calculation formula for the similarity between each one-dimensional image vector and each one-dimensional text vector is: in, represents the image-text similarity between the i-th 3D multimodal brain MRI image and the j-th Chinese medical report in the current batch, represents the i-th 3D multimodal brain MRI image in the current batch, represents the jth Chinese medical report in the current batch, v i represents the i-th one-dimensional image vector in the current batch, t j represents the j-th one-dimensional text vector in the current batch, S(v i ,t j ) represents the cosine similarity between the i-th one-dimensional image vector and the j-th one-dimensional text vector in the current batch, exp(·) represents the exponential of the expression in the brackets, τ represents the learnable temperature coefficient, and N is the preset batch size.

5. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that: The calculation formula of the fusion loss function is: L=al clip +βL ts ; IT v2t =CE(E N ,P v2t ); IT t2v =CE(E N ,P t2v ); 50 ts-v2t =KL1(T sim ,P v2t ); L ts-t2v =KL2(T sim ,P t2v ); Among them, L represents the fusion loss function, L clip represents contrastive learning loss, α is the first weight, L ts represents the distribution loss, β is the second weight, L v2t represents the image-text similarity loss, L t2v represents the text-image similarity loss, CE represents the cross entropy loss function, E N represents the identity matrix, P v2t represents the image-text similarity matrix, P t2v represents the text-image similarity matrix, L ts-v2t represents the first intermediate function, L ts-t2v represents the second intermediate function, T sim represents the semantic similarity matrix, KL1(·,·) represents the KL divergence between the semantic similarity matrix in the brackets and the image-text similarity matrix, and KL2(·,·) represents the KL divergence between the semantic similarity matrix in the brackets and the text-image similarity matrix.

6. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that: The method includes inputting each 3D multimodal brain MRI image in the current batch of training sample subsets into the image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image output by the image encoder model, and also includes: using a learning rate warm-up method for a total of 5000 iterations, adjusting the learning rate once per iteration, and the adjustment formula of the learning rate is as follows: in, is the learning rate of the current iteration, lr init represents the initial learning rate of the image encoder model or text encoder model, t is the current number of iterations, and t max is the total number of iterations.

7. The image encoder-text encoder joint training method based on contrastive learning according to claim 1, characterized in that: Optimizing the parameters of the image encoder model and the text encoder model also includes: updating the learning rate using a polynomial learning rate descent strategy during training. The learning rate update formula is as follows: Among them, lr (e) represents the updated learning rate, e represents the current cycle, and 250 iterations are defined as one cycle. max is the total number of cycles.

8. A 3D multimodal brain MRI image query method, characterized in that: The 3D multimodal brain MRI image query method comprises: Using the trained image encoder model, each 3D multimodal brain MRI image stored in the database is processed to obtain a one-dimensional image vector corresponding to each image; Receive the query condition input by the user, perform text preprocessing on the query condition, and input the query condition after text preprocessing into the trained text encoder model to obtain the target one-dimensional text vector; The similarity between the target one-dimensional text vector and each one-dimensional image vector is calculated and compared respectively, and the 3D multimodal brain MRI image corresponding to the one-dimensional image vector with the largest similarity value is selected as the query result.

9. A target health status assessment method based on 3D multimodal brain MRI images, characterized in that: The target health status evaluation method based on 3D multimodal brain MRI images includes: Constructing a target health status evaluation model, wherein the target health status evaluation model includes a trained image encoder model and a binary classification model; Inputting 3D multimodal brain MRI images including target health states and non-target health states into the trained image encoder model to obtain a one-dimensional image vector corresponding to each 3D multimodal brain MRI image; The one-dimensional image vectors corresponding to each 3D multimodal brain MRI image are used as training samples to train the binary classification model to obtain a trained binary classification model, thereby realizing the evaluation of the target health status.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image encoder-text encoder joint training method based on contrastive learning as described in any one of claims 1 to 7, or the 3D multimodal brain MRI image query method as described in claim 8, or the target health status assessment method based on 3D multimodal brain MRI images as described in claim 9.

Images