Text-to-image conversion method and device, computer equipment and storage medium

By using the pretrained encoder model and coupling layer to optimize the pathological text and image data, the accuracy problem of pathological text to image is solved, high-quality pathological state image generation is achieved, and the clinical application value is enhanced.

CN119046489AActive Publication Date: 2024-11-29PING AN TECH (SHENZHEN) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411165221.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The prior art is difficult to accurately and effectively convert pathological text into intuitive pathological state images, especially when dealing with specialized terms and complex lesion descriptions unique to the medical field, and it is difficult to maintain sufficient resolution and authenticity, limiting its clinical practice. value of application.

Method used

The pre-trained encoder model and preset coupling layer are used to extract vectors and calculate the loss value of the pathological text data, optimize the model to generate pathological state images, and accurately generate images through Gaussian blur processing and feature decoding.

Benefits of technology

It effectively realizes the generation of corresponding pathological state images based on the input pathological text, improves the accuracy and detail restoration ability of generated images, and enhances the visual support of clinical diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046489A_ABST
    Figure CN119046489A_ABST
Patent Text Reader

Abstract

The embodiment of the invention belongs to the field of digital medical treatment, and relates to a text-to-image method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining first pathological text and image pair data; carrying out vector extraction and loss value calculation on the first pathological graph pair data to obtain a first total loss value; optimizing a pre-trained encoder model and a preset coupling layer according to the first total loss value to obtain an optimized encoder model and an optimized coupling layer; acquiring second pathological text and graph pair data; carrying out vector extraction and loss value calculation on the second pathological graph pair data to obtain a second total loss value; re-optimizing the optimized encoder model and the optimized coupling layer according to the second total loss value to obtain an effective encoder model and an effective coupling layer; and inputting the actual pathological text into the effective encoder model and the effective coupling layer, and carrying out image decoding to obtain a pathological state image. The corresponding pathological state image can be effectively generated according to the input pathological text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, specifically to the field of digital medicine, and in particular to a text-to-image method, device, computer equipment and storage medium. Background Art

[0002] In current medical diagnosis and research, pathological texts, as an important carrier for describing disease characteristics and progression, carry rich clinical information. However, although pathological texts can record the morphological changes, cytological characteristics, and molecular abnormalities of the disease in detail, in order to display disease information more intuitively, they usually need to be viewed in conjunction with pathological state images. In most cases, the acquisition of pathological state images requires high-precision observation. Therefore, how to convert pathological texts into pathological state images has always been a problem to be solved in this field.

[0003] In terms of converting pathological text into intuitive pathological state images, the field has always faced considerable technical challenges. First, there is a complex mapping relationship between pathological text and pathological images, which not only involves the correspondence between vocabulary and image features, but also covers a deep understanding of the semantic level. Although existing computer vision and natural language processing technologies have made significant progress, it is difficult to accurately and effectively recognize professional terms, complex lesion descriptions, and highly specialized pathological languages ​​that are unique to the medical field.

[0004] Secondly, the generation of pathological images requires a high degree of accuracy and detail restoration capabilities. Pathological conditions are often reflected in the microscopic structure of cells, tissue arrangement, and subtle differences in the lesion area, which are crucial for accurate diagnosis of the disease. However, current technology has difficulty maintaining sufficient resolution and authenticity when generating images, which limits its application value in clinical practice and brings difficulties to the development of pathological research. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a text-to-image method, apparatus, computer device and storage medium to solve the problem of being unable to accurately and effectively generate a corresponding pathological state image based on the input pathological text.

[0006] In order to solve the above technical problems, the present application embodiment provides a text-to-image method, which adopts the following technical solution:

[0007] Acquiring first pathological text-image pair data;

[0008] Performing vector extraction on the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculating the loss value according to the extracted first feature vector to obtain a first total loss value;

[0009] Optimizing the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain an optimized encoder model and an optimized coupling layer;

[0010] Acquiring second pathological text-image pair data;

[0011] Performing vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculating a loss value according to the extracted second feature vector to obtain a second total loss value;

[0012] Re-optimize the optimized encoder model and the optimized coupling layer according to the second total loss value to obtain an effective encoder model and an effective coupling layer;

[0013] The actual pathological text is obtained, inputted into the effective encoder model and the effective coupling layer, and image decoding is performed to obtain a pathological state image.

[0014] Furthermore, the step of obtaining the first pathological text-image pair data specifically includes:

[0015] Acquire a first data identifier, and extract pathological text data and pathological image data from a database according to the first data identifier;

[0016] Performing Gaussian blur processing on the pathological image data to obtain pathological Gaussian image data;

[0017] The pathological text data and the pathological Gaussian image data are associated to obtain the first pathological text-image pair data.

[0018] Further, the step of performing vector extraction on the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculating the loss value according to the extracted first feature vector to obtain the first total loss value specifically includes:

[0019] Inputting the first pathological text-image pair data into the pre-trained encoder model for feature encoding to obtain a first feature encoding vector;

[0020] Inputting the first feature coding vector into the preset coupling layer for feature conversion to obtain the first feature vector;

[0021] Perform text loss calculation and image loss calculation on the first feature vector to obtain a first total loss value.

[0022] Furthermore, the step of performing text loss calculation and image loss calculation on the first feature vector to obtain a first total loss value specifically includes:

[0023] Extracting a first text feature vector from the first feature vector;

[0024] Obtaining a dimension corresponding to the first text feature vector, and performing loss estimation calculation according to the dimension corresponding to the first text feature vector and a standard Gaussian distribution to obtain a first text loss;

[0025] Extracting a first image feature vector from the first feature vector;

[0026] Inputting the first image feature vector into a preset image decoder for decoding to obtain a first image decoding vector;

[0027] Extracting the pathological Gaussian image data in the first pathological text-image pair data, and performing loss estimation calculation based on the first image decoding vector and the pathological Gaussian image data to obtain a first image loss;

[0028] The first text loss and the first image loss are summed to obtain the first total loss value.

[0029] Furthermore, the step of obtaining the second pathological text-image pair data specifically includes:

[0030] Acquire a second data identifier, and extract Gaussian-distributed pathological text data and pathological image data from a database according to the second data identifier;

[0031] The Gaussian-distributed pathological text data and the pathological image data are associated to obtain the second pathological text-image pair data.

[0032] Further, the step of performing vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculating the loss value according to the extracted second feature vector to obtain the second total loss value specifically includes:

[0033] Inputting the second pathological text-image pair data into the optimized coupling layer to obtain a coupling vector;

[0034] Inputting the coupling vector into an optimized encoder model for feature encoding to obtain the second feature vector;

[0035] Perform text loss calculation and image loss calculation on the second feature vector to obtain a second total loss value.

[0036] Furthermore, the step of performing text loss calculation and image loss calculation on the second feature vector to obtain a second total loss value specifically includes:

[0037] Extracting a second text feature vector from the second feature vector;

[0038] Obtaining the dimension corresponding to the second text feature vector, and performing loss estimation calculation according to the dimension corresponding to the second text feature vector and a standard Gaussian distribution to obtain a second text loss;

[0039] extracting a second image feature vector from the second feature vector;

[0040] Inputting the second image feature vector into a preset image decoder for decoding to obtain a second image decoding vector;

[0041] Extracting the pathological image data in the second pathological text-image pair data, and performing loss estimation calculation based on the second image decoding vector and the pathological Gaussian image data in the second pathological text-image pair data to obtain a second image loss;

[0042] The second text loss and the second image loss are summed to obtain the second total loss value.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a text-to-image device, which adopts the following technical solution:

[0044] A first data acquisition module, used for acquiring first pathological text-image pair data;

[0045] A first loss calculation module, used for performing vector extraction on the first pathological text-image pair data according to a pre-trained encoder model and a preset coupling layer, and performing loss value calculation according to the extracted first feature vector to obtain a first total loss value;

[0046] A first optimization module, used to optimize the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain an optimized encoder model and an optimized coupling layer;

[0047] A second data acquisition module, used to acquire second pathology text-image pair data;

[0048] A second loss calculation module, used for performing vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and performing loss value calculation according to the extracted second feature vector to obtain a second total loss value;

[0049] A second optimization module, configured to re-optimize the optimized encoder model and the optimized coupling layer according to the second total loss value to obtain an effective encoder model and an effective coupling layer;

[0050] The image conversion module is used to obtain the actual pathological text, input the actual pathological text into the effective encoder model and the effective coupling layer, and perform image decoding to obtain a pathological state image.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0052] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above text-to-image methods when executing the computer-readable instructions.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0054] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text-to-image method described in any one of the above items are implemented.

[0055] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment obtains the first pathological text-image data; extracts the vector of the first pathological text-image data according to the pre-trained encoder model and the preset coupling layer, and calculates the loss value according to the extracted first feature vector to obtain the first total loss value; optimizes the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain the optimized encoder model and the optimized coupling layer; obtains the second pathological text-image data; extracts the vector of the second pathological text-image data according to the optimized encoder model and the optimized coupling layer, and calculates the loss value according to the extracted second feature vector to obtain the second total loss value; optimizes the optimized encoder model and the optimized coupling layer again according to the second total loss value to obtain the effective encoder model and the effective coupling layer; obtains the actual pathological text, inputs the actual pathological text into the effective encoder model and the effective coupling layer, and performs image decoding to obtain the pathological state image. Thus, the corresponding pathological state image is effectively generated according to the input pathological text, and the accuracy of the generated pathological state image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0058] Figure 2A flowchart of an embodiment of a text-to-image method according to the present application;

[0059] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S10;

[0060] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S20;

[0061] Figure 5 yes Figure 4 A flowchart of a specific implementation of step S203;

[0062] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S40;

[0063] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S50;

[0064] Figure 8 yes Figure 7 A flowchart of a specific implementation of step S503;

[0065] Fig. 9 is a structural schematic diagram of an embodiment of a text-to-image device according to the present application;

[0066] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0068] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0070] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0071] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0072] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0073] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0074] It should be noted that the text-to-image method provided in the embodiment of the present application is generally executed by a server, and accordingly, the text-to-image device is generally arranged in the server.

[0075] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0076] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for converting text to image according to the present application. The method for converting text to image comprises the following steps:

[0077] Step S10, obtaining first pathological text-image pair data;

[0078] In this embodiment, the first pathological image-text pair data includes pathological text data and pathological image data, and the content described by the pathological text data corresponds to the pathological image data, wherein the pathological image data is Gaussian blurred to form pathological Gaussian image data, and the image noise and detail level are reduced by calculating the weighted average of the pixels around each pixel in the image, thereby obtaining the pathological Gaussian image data after normal distribution blur processing.

[0079] Step S20, performing vector extraction on the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculating the loss value according to the extracted first feature vector to obtain a first total loss value;

[0080] In this embodiment, the pre-trained encoder model is a masked multimodal encoder MME model trained with sample pathology text data and sample pathology image data. The masked multimodal encoder MME model is a model that can simultaneously process both text and image modal data. It uses deep learning architectures such as Transformer to capture cross-modal interaction information, and randomly masks part of the input data (such as words in text or areas in images) to train the model to predict the masked content, thereby learning the intrinsic representation of the data, thereby achieving effective alignment of text and image modal data, so as to accurately understand and generate cross-modal information (i.e., input text to generate images or input images to generate text).

[0081] In this embodiment, the preset coupling layer is an intermediate layer that promotes interaction and fusion between different modal data, and is used to convert feature vectors of different modalities into cross-modal feature vectors, which integrate feature vectors from different modalities through specific fusion strategies (such as weighted summation, splicing, attention mechanism, etc.) to form a cross-module feature vector representation. The first feature vector refers to a feature vector of a cross-module representation containing a pathological text vector and a pathological image vector. By calculating the loss value of the first feature vector, a first total loss value that can effectively evaluate the model performance and guide the model optimization is obtained.

[0082] Step S30, optimizing the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain an optimized encoder model and an optimized coupling layer;

[0083] In this embodiment, the parameters of the pre-trained encoder model and the preset coupling layer are optimized by the gradient descent method or the back propagation algorithm. The gradient descent method calculates the gradient of the first total loss value with respect to the model parameters and updates the parameters in the opposite direction of the gradient to reduce the loss value. The back propagation algorithm calculates the gradient of the first total loss value with respect to each parameter and updates the parameters of the pre-trained encoder model and the preset coupling layer according to the gradient to obtain an optimized encoder model and an optimized coupling layer.

[0084] Step S40, obtaining second pathology text-image pair data;

[0085] In this embodiment, the second pathological text-image pair data includes pathological text data and pathological image data with Gaussian distribution. The pathological text data with Gaussian distribution refers to the pathological text data statistically calculated according to Gaussian distribution. The steps of obtaining the pathological text data include extracting text features and digitizing features. When calculating the pathological text data, it is first necessary to extract useful features from the text. These features include vocabulary frequency, the number of occurrences of specific medical terms, etc. Then, the extracted text features are digitized for statistical analysis. For example, the vocabulary frequency can be expressed as the number of times each vocabulary appears in the text. After obtaining the digitized text features, the digitized text features are added to the corresponding pathological text data, thereby obtaining the pathological text data with Gaussian distribution.

[0086] Step S50, performing vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculating a loss value according to the extracted second feature vector to obtain a second total loss value;

[0087] In this embodiment, the optimized encoder model refers to the masked multimodal encoder MME model optimized according to the first total loss value, and the optimized coupling layer refers to the network middle layer (coupling layer) optimized according to the first total loss value. By performing vector extraction on the second pathological text map data and calculating the loss value based on the extracted second feature vector, a second total loss value for evaluating the performance of the optimized encoder model and the optimized coupling layer is obtained, thereby facilitating further updating of the optimized encoder model and the optimized coupling layer based on the second total loss value.

[0088] Step S60, re-optimizing the optimized encoder model and the optimized coupling layer according to the second total loss value to obtain an effective encoder model and an effective coupling layer;

[0089] In this embodiment, the parameters of the optimized encoder model and the optimized coupling layer are optimized by the gradient descent method or the back propagation algorithm. The gradient descent method calculates the gradient of the second total loss value with respect to the model parameters and updates the parameters in the opposite direction of the gradient to reduce the loss value. The back propagation algorithm calculates the gradient of the second total loss value with respect to each parameter and updates the parameters of the optimized encoder model and the optimized coupling layer according to the gradient to obtain an effective encoder model and an effective coupling layer.

[0090] Step S70, obtaining actual pathological text, inputting the actual pathological text into the effective encoder model and the effective coupling layer, and performing image decoding to obtain a pathological state image.

[0091] In this embodiment, the actual pathological text refers to a real pathological text, which is derived from the pathological records of hospital patients. The actual pathological text is input into the effective encoder model and the effective coupling layer, and then the output feature vector is image decoded to obtain a pathological state image corresponding to the actual pathological text. The image decoding of this embodiment is performed by a preset image decoder, which is a picture decoder (Decoder). The picture decoder (Decoder) performs decoding calculations by integrating a built-in decoder (such as png-decoder and jpeg-decoder), and the vector output by the effective encoder model and the effective coupling layer is decoded into a pathological state image through image decoding.

[0092] In this embodiment, the pathological text data and pathological image data in the above method can be derived from the pathological records recorded in the database of the medical system, wherein the pathological records may include pathological diagnosis reports, pathological images, etc., and the pathological text data may include diagnostic reports, case descriptions, medical history records, etc. written by pathologists. These data exist in the form of text and can be extracted, cleaned and converted by natural language processing technology for subsequent statistical analysis. Pathological image data may include tissue section images, cytological images, etc. under a microscope, and these image data may be feature extracted and converted by image processing technology for statistical analysis such as Gaussian distribution.

[0093] This embodiment obtains the first pathological text-image data; extracts the vector of the first pathological text-image data according to the pre-trained encoder model and the preset coupling layer, and calculates the loss value according to the extracted first feature vector to obtain the first total loss value; optimizes the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain the optimized encoder model and the optimized coupling layer; obtains the second pathological text-image data; extracts the vector of the second pathological text-image data according to the optimized encoder model and the optimized coupling layer, and calculates the loss value according to the extracted second feature vector to obtain the second total loss value; optimizes the optimized encoder model and the optimized coupling layer again according to the second total loss value to obtain the effective encoder model and the effective coupling layer; obtains the actual pathological text, inputs the actual pathological text into the effective encoder model and the effective coupling layer, and performs image decoding to obtain the pathological state image. Thus, the corresponding pathological state image is effectively generated according to the input pathological text, and the accuracy of the generated pathological state image is improved.

[0094] Continue to refer Figure 3 In some optional implementations of this embodiment, step S10 includes the following steps:

[0095] Step S101, obtaining a first data identifier, and extracting pathological text data and pathological image data from a database according to the first data identifier;

[0096] In this embodiment, the first data identifier is identification information pointing to the pathological text data and the pathological image data. The pathological text data and the pathological image data correspond to each other and are retrieved from the database through the first data identifier.

[0097] Step S102, performing Gaussian blur processing on the pathological image data to obtain pathological Gaussian image data;

[0098] In this embodiment, the Gaussian blur processing of the pathological image data includes the following steps: Selecting blur radius and standard deviation value: When processing pathological image data, it is first necessary to select a suitable blur radius and standard deviation value. The blur radius determines the pixel range involved in the calculation, while the standard deviation value directly affects the degree of blur. Generate a weight matrix: According to the selected standard deviation value, the weight of each pixel is calculated using the Gaussian formula to generate a weight matrix. The center point of this matrix corresponds to the current pixel in the image, and the weights of the surrounding points are determined according to their distance from the center point. Calculate the weighted average: Apply the weight matrix to the pathological image data to calculate the weighted average of each pixel. This process involves multiplying the color value of the current pixel and its surrounding pixels (for color images, usually the color values ​​of the three RGB channels) with the corresponding weights and then summing them. Generate pathological Gaussian image data: Repeat the above calculation process for each pixel in the image, and finally obtain the pathological image data processed by Gaussian blur, that is, pathological Gaussian image data.

[0099] Step S103, associating the pathological text data with the pathological Gaussian image data to obtain the first pathological text-image pair data.

[0100] In this embodiment, the pathological text data and the pathological Gaussian image data are paired and a data set is created. The paired pathological text data and pathological Gaussian image data are saved in the data set, thereby forming a data association to obtain the first pathological text-image pair data.

[0101] This embodiment obtains a first data identifier, extracts pathological text data and pathological image data from a database according to the first data identifier; performs Gaussian blur processing on the pathological image data to obtain pathological Gaussian image data; and associates the pathological text data with the pathological Gaussian image data, thereby effectively obtaining first pathological text-image pair data including pathological text data and pathological Gaussian image data, so as to facilitate subsequent feature vector extraction and loss value calculation based on the first pathological text-image pair data.

[0102] Continue to refer Figure 4 In some optional implementations of this embodiment, step S20 includes the following steps:

[0103] Step S201, inputting the first pathological text-image pair data into the pre-trained encoder model for feature encoding to obtain a first feature encoding vector;

[0104] In this embodiment, feature encoding refers to the process of converting raw data (such as categorical variables, dates, text, etc.) into a numerical form that can be understood and processed by the machine learning model. The first feature encoding vector includes a first pathological text vector and a first pathological image vector. Through feature encoding, the first feature encoding vector is effectively obtained to facilitate subsequent analysis and processing.

[0105] Step S202, inputting the first feature coding vector into the preset coupling layer for feature conversion to obtain the first feature vector;

[0106] In this embodiment, the first feature vector refers to a feature vector including a cross-module representation of a first pathological text vector and a first pathological image vector. The preset coupling layer is used to convert feature vectors of different modalities into cross-modal feature vectors, and integrate feature vectors from different modalities through specific fusion strategies (such as weighted summation, splicing, attention mechanism), thereby completing feature conversion and obtaining the first feature vector.

[0107] Step S203: Perform text loss calculation and image loss calculation on the first feature vector to obtain a first total loss value.

[0108] In this embodiment, text loss calculation refers to calculating the loss value of the part of the first feature vector corresponding to the text vector, and image loss calculation refers to calculating the loss value of the part of the first feature vector corresponding to the image vector. The first total loss value is obtained by calculating the first feature vector based on the text loss calculation and the image loss calculation, reflecting the value of the text part loss and the image part loss in the model output result. The first total loss value can effectively reflect the deviation of the pre-trained encoder model and the preset coupling layer in the result output, thereby facilitating the subsequent optimization operation based on the first total loss value.

[0109] This embodiment obtains a first feature encoding vector by inputting the first pathological text-image pair data into the pre-trained encoder model for feature encoding; inputting the first feature encoding vector into the preset coupling layer for feature conversion to obtain the first feature vector; and performing text loss calculation and image loss calculation on the first feature vector to obtain a first total loss value. Thus, the first total loss value that can reflect the output result deviation of the first pathological text-image pair data after passing through the pre-trained encoder model and the preset coupling layer is effectively obtained, so as to facilitate the subsequent optimization of the pre-trained encoder model and the preset coupling layer according to the first total loss value.

[0110] Continue to refer Figure 5 In some optional implementations of this embodiment, step S203 specifically includes the following steps:

[0111] Step S2031, extracting a first text feature vector from the first feature vector;

[0112] In this embodiment, the first text feature vector is a vector corresponding to the text portion in the first feature vector, and the first text feature vector is obtained by separating the portion related to the text from the first feature vector. The separation operation can be completed by identifying the portion related to the text in the first feature vector through a machine learning algorithm, and then performing feature selection and extraction.

[0113] Step S2032, obtaining the dimension corresponding to the first text feature vector, and performing loss estimation calculation according to the dimension corresponding to the first text feature vector and the standard Gaussian distribution to obtain the first text loss;

[0114] In this embodiment, the dimension of the first text feature vector is obtained, and the dimension includes the vocabulary frequency, sentence length, and the number of occurrences of specific medical terms in the first text feature vector. Using the standard Gaussian distribution (i.e., a normal distribution with a mean of 0 and a variance of 1) as a reference distribution, the first text loss representing the degree of difference between the first text feature vector and the standard Gaussian distribution is obtained by calculating the difference or distance between the first feature vector and the standard Gaussian distribution, and the calculation can be implemented using the KL divergence (Kullback-Leibler Divergence). KL divergence (Kullback-LeiblerDivergence), also known as relative entropy (Relative Entropy), information divergence (Information Divergence), information gain (Information Gain) or KLIC, is an important concept in information theory, which is used to quantify the difference between two probability distributions. The calculation formula of KL divergence is D(P||Q)=ΣP(x)*log(P(x) / Q(x)). In this embodiment, D(P||Q) corresponds to the difference between the first text feature vector and the standard Gaussian distribution text, and the difference is the first text loss. P(x) is the vocabulary frequency corresponding to the first text feature vector and the probability of the number of occurrences of specific medical terms. Q(x) is the vocabulary frequency of the standard Gaussian distribution and the probability of the number of occurrences of specific medical terms.

[0115] Step S2033, extracting a first image feature vector from the first feature vector;

[0116] In this embodiment, the first text feature vector is a vector corresponding to the image portion in the first feature vector, and the first image feature vector is obtained by separating the image-related portion from the first feature vector. The separation operation can be completed by identifying the image-related portion in the first feature vector through a machine learning algorithm, and then performing feature selection and extraction.

[0117] Step S2034, inputting the first image feature vector into a preset image decoder for decoding to obtain a first image decoding vector;

[0118] In this embodiment, the preset image decoder is a picture decoder (Decoder), and the picture decoder (Decoder) performs decoding calculation by integrating a built-in decoder (such as png-decoder and jpeg-decoder), thereby obtaining a first image decoding vector.

[0119] Step S2035, extracting the pathological Gaussian image data in the first pathological text-image pair data, and performing loss estimation calculation based on the first image decoding vector and the pathological Gaussian image data to obtain a first image loss;

[0120] In this embodiment, the first image decoding vector is image data obtained by performing a decoding operation based on the first image feature vector, and the pathological Gaussian image data is image data obtained by performing Gaussian blur processing on the pathological image data. The difference between the first image decoding vector and the pathological Gaussian image data can be compared by the mean square error, and the difference can be quantified as the first image loss.

[0121] Step S2036: sum the first text loss and the first image loss to obtain the first total loss value.

[0122] In this embodiment, the first total loss value is obtained by adding the loss value corresponding to the first text loss and the loss value corresponding to the first image loss.

[0123] This embodiment extracts the first text feature vector from the first feature vector; obtains the dimension corresponding to the first text feature vector, and performs loss estimation calculation based on the dimension corresponding to the first text feature vector and the standard Gaussian distribution to obtain the first text loss; extracts the first image feature vector from the first feature vector; inputs the first image feature vector into a preset image decoder for decoding to obtain a first image decoding vector; extracts the pathological Gaussian image data from the first pathological text-image pair data, and performs loss estimation calculation based on the first image decoding vector and the pathological Gaussian image data to obtain the first image loss; sums the first text loss and the first image loss, thereby effectively obtaining a first total loss value reflecting the text part loss and the image part loss in the model output result, so as to facilitate subsequent optimization and adjustment based on the first total loss value.

[0124] Continue to refer Figure 6 In some optional implementations of this embodiment, step S40 includes the following steps:

[0125] Step S401, obtaining a second data identifier, and extracting Gaussian distributed pathological text data and pathological image data from a database according to the second data identifier;

[0126] In this embodiment, the second data identifier is identification information pointing to the pathological text data and pathological image data of Gaussian distribution. The pathological text data and pathological image data of Gaussian distribution correspond to each other and are queried and acquired from the database through the second data identifier.

[0127] Step S402, associating the Gaussian distributed pathology text data with the pathology image data to obtain the second pathology text-image pair data.

[0128] In this embodiment, the pathological text data with Gaussian distribution and the pathological image data are paired and a data set is created. The paired pathological text data with Gaussian distribution and the pathological image data are saved in the data set, thereby forming a data association to obtain the second pathological text-image pair data.

[0129] This embodiment obtains a second data identifier, extracts Gaussian-distributed pathological text data and pathological image data from a database according to the second data identifier, and associates the Gaussian-distributed pathological text data with the pathological image data, thereby effectively obtaining second pathological text-image pair data that associates and organizes the Gaussian-distributed pathological text data and the pathological image data, so as to facilitate subsequent second feature vector extraction and second total loss value calculation.

[0130] Continue to refer Figure 7 In some optional implementations of this embodiment, step S50 includes the following steps:

[0131] Step S501, inputting the second pathological text-image pair data into the optimized coupling layer to obtain a coupling vector;

[0132] In this embodiment, the optimized coupling layer is a coupling layer obtained after optimizing the first total loss value. The optimized coupling layer is used to convert feature vectors of different modalities into cross-modal feature vectors, and integrate feature vectors from different modalities through specific fusion strategies (such as weighted summation, splicing, attention mechanism), thereby completing feature conversion and obtaining a coupling vector.

[0133] Step S502, inputting the coupling vector into an optimized encoder model for feature encoding to obtain the second feature vector;

[0134] In this embodiment, the optimized encoder model is an encoder model obtained after optimizing the first total loss value. The feature encoding of the optimized encoder model refers to the process of converting the coupling vector into a numerical form that can be understood and processed by the machine learning model. The second feature encoding vector includes a second pathological text vector and a second pathological image vector. Through feature encoding, the second feature encoding vector is effectively obtained to facilitate subsequent analysis and processing.

[0135] Step S503: Perform text loss calculation and image loss calculation on the second feature vector to obtain a second total loss value.

[0136] In this embodiment, text loss calculation refers to calculating the loss value of the part of the text vector corresponding to the second feature vector, and image loss calculation refers to calculating the loss value of the part of the image vector corresponding to the second feature vector. The second total loss value is obtained by the second feature vector based on the text loss calculation and the image loss calculation, reflecting the values ​​of the text part loss and the image part loss in the model output result. The second total loss value can effectively reflect the deviation of the optimized encoder model and the optimized coupling layer in the result output, thereby facilitating the subsequent optimization operation based on the second total loss value.

[0137] In this embodiment, the coupling vector is obtained by inputting the second pathological text-image pair data into the optimized coupling layer; the coupling vector is input into the optimized encoder model for feature encoding to obtain the second feature vector; text loss calculation and image loss calculation are performed on the second feature vector, thereby obtaining a second total loss value of the second feature vector after feature conversion of the optimized coupling layer and feature encoding of the optimized encoder model, so as to facilitate further optimization of the optimized encoder model and the optimized coupling layer according to the second total loss value.

[0138] Continue to refer Figure 8 In some optional implementations of this embodiment, step S503 includes the following steps:

[0139] Step S5031, extracting a second text feature vector from the second feature vector;

[0140] In this embodiment, the second text feature vector is a vector corresponding to the text portion in the second feature vector, and the second text feature vector is obtained by separating the portion related to the text from the second feature vector. The separation operation can be completed by identifying the portion related to the text in the second feature vector through a machine learning algorithm, and then performing feature selection and extraction.

[0141] Step S5032, obtaining the dimension corresponding to the second text feature vector, and performing loss estimation calculation according to the dimension corresponding to the second text feature vector and the standard Gaussian distribution to obtain the second text loss;

[0142] In this embodiment, the dimension of the second text feature vector is obtained, and the dimension includes the vocabulary frequency, sentence length, and the number of occurrences of specific medical terms in the second text feature vector. Using the standard Gaussian distribution (i.e., a normal distribution with a mean of 0 and a variance of 1) as a reference distribution, the second text loss representing the degree of difference between the second text feature vector and the standard Gaussian distribution is obtained by calculating the difference or distance between the second feature vector and the standard Gaussian distribution, and the calculation can be implemented using the KL divergence (Kullback-Leibler Divergence). KL divergence (Kullback-LeiblerDivergence), also known as relative entropy (Relative Entropy), information divergence (Information Divergence), information gain (Information Gain) or KLIC, is an important concept in information theory, which is used to quantify the difference between two probability distributions. The calculation formula of KL divergence is D(P||Q)=ΣP(x)*log(P(x) / Q(x)). In this embodiment, D(P||Q) corresponds to the difference between the second text feature vector and the standard Gaussian distribution text, and the difference is the second text loss. P(x) is the vocabulary frequency corresponding to the second text feature vector and the probability of the number of occurrences of specific medical terms. Q(x) is the vocabulary frequency of the standard Gaussian distribution and the probability of the number of occurrences of specific medical terms.

[0143] Step S5033, extracting a second image feature vector from the second feature vector;

[0144] In this embodiment, the second text feature vector is a vector corresponding to the image portion in the second feature vector, and the second image feature vector is obtained by separating the image-related portion from the second feature vector. The separation operation can be completed by identifying the image-related portion in the second feature vector through a machine learning algorithm, and then performing feature selection and extraction.

[0145] Step S5034, inputting the second image feature vector into a preset image decoder for decoding to obtain a second image decoding vector;

[0146] In this embodiment, the preset image decoder is a picture decoder (Decoder), and the picture decoder (Decoder) performs decoding calculations by integrating built-in decoders (such as png-decoder and jpeg-decoder), thereby obtaining a second image decoding vector.

[0147] Step S5035, extracting the pathological image data in the second pathological text-image pair data, and performing loss estimation calculation based on the second image decoding vector and the pathological Gaussian image data in the second pathological text-image pair data to obtain a second image loss;

[0148] In this embodiment, the second image decoding vector is image data obtained by performing a decoding operation based on the second image feature vector, and the pathological Gaussian image data is image data obtained by performing Gaussian blur processing on the pathological image data. The difference between the first image decoding vector and the pathological Gaussian image data can be compared by the mean square error, and the difference can be quantified as the second image loss.

[0149] Step S5036: sum the second text loss and the second image loss to obtain the second total loss value.

[0150] In this embodiment, the second total loss value is obtained by adding the loss value corresponding to the second text loss and the loss value corresponding to the second image loss.

[0151] This embodiment extracts the second text feature vector from the second feature vector; obtains the dimension corresponding to the second text feature vector, and performs loss estimation calculation based on the dimension corresponding to the second text feature vector and the standard Gaussian distribution to obtain the second text loss; extracts the second image feature vector from the second feature vector; inputs the second image feature vector into a preset image decoder for decoding to obtain a second image decoding vector; extracts the pathological image data from the second pathological text-image pair data, and performs loss estimation calculation based on the second image decoding vector and the pathological Gaussian image data to obtain the second image loss; and sums the second text loss and the second image loss to obtain the second total loss value.

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0153] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0154] Further references Fig. 9 , as a response to the above Figure 1 The present application provides an embodiment of a text-to-image device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0155] like Fig. 9 As shown, the text-to-image device 800 described in this embodiment includes: a first data acquisition module 801, a first loss calculation module 802, a first optimization module 803, a second data acquisition module 804, a second loss calculation module 805, a second optimization module 806, and an image conversion module 807. Among them:

[0156] The first data acquisition module 801 is used to acquire first pathology text-image pair data;

[0157] A first loss calculation module 802 is used to extract vectors from the first pathological text-image pair data according to a pre-trained encoder model and a preset coupling layer, and calculate a loss value according to the extracted first feature vector to obtain a first total loss value;

[0158] A first optimization module 803, configured to optimize the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain an optimized encoder model and an optimized coupling layer;

[0159] The second data acquisition module 804 is used to acquire second pathology text-image pair data;

[0160] A second loss calculation module 805 is used to extract vectors from the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculate a loss value according to the extracted second feature vector to obtain a second total loss value;

[0161] A second optimization module 806, configured to re-optimize the optimized encoder model and the optimized coupling layer according to the second total loss value to obtain an effective encoder model and an effective coupling layer;

[0162] The image conversion module 807 is used to obtain the actual pathological text, input the actual pathological text into the effective encoder model and the effective coupling layer, and perform image decoding to obtain a pathological state image.

[0163] By adopting the above-mentioned device, this embodiment can obtain the first pathological text-image pair data; perform vector extraction on the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculate the loss value according to the extracted first feature vector to obtain the first total loss value; optimize the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain the optimized encoder model and the optimized coupling layer; obtain the second pathological text-image pair data; perform vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculate the loss value according to the extracted second feature vector to obtain the second total loss value; optimize the optimized encoder model and the optimized coupling layer again according to the second total loss value to obtain the effective encoder model and the effective coupling layer; obtain the actual pathological text, input the actual pathological text into the effective encoder model and the effective coupling layer, and perform image decoding to obtain the pathological state image. Thereby effectively realizing the generation of the corresponding pathological state image according to the input pathological text, and improving the accuracy of the generated pathological state image.

[0164] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0165] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 with components 91-93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0166] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0167] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 9. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of the text-to-image method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0168] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run computer-readable instructions stored in the memory 91 or process data, such as computer-readable instructions for running the text-to-image method.

[0169] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0170] By adopting the above-mentioned computer device, this embodiment can obtain the first pathological text-image pair data; perform vector extraction on the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculate the loss value according to the extracted first feature vector to obtain the first total loss value; optimize the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain the optimized encoder model and the optimized coupling layer; obtain the second pathological text-image pair data; perform vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculate the loss value according to the extracted second feature vector to obtain the second total loss value; optimize the optimized encoder model and the optimized coupling layer again according to the second total loss value to obtain the effective encoder model and the effective coupling layer; obtain the actual pathological text, input the actual pathological text into the effective encoder model and the effective coupling layer, and perform image decoding to obtain the pathological state image. Thereby effectively realizing the generation of the corresponding pathological state image according to the input pathological text, and improving the accuracy of the generated pathological state image.

[0171] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the text-to-image method as described above.

[0172] By adopting the above-mentioned computer-readable storage medium, this embodiment can obtain the first pathological text-image pair data; perform vector extraction on the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculate the loss value according to the extracted first feature vector to obtain the first total loss value; optimize the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain the optimized encoder model and the optimized coupling layer; obtain the second pathological text-image pair data; perform vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculate the loss value according to the extracted second feature vector to obtain the second total loss value; optimize the optimized encoder model and the optimized coupling layer again according to the second total loss value to obtain the effective encoder model and the effective coupling layer; obtain the actual pathological text, input the actual pathological text into the effective encoder model and the effective coupling layer, and perform image decoding to obtain the pathological state image. Thereby effectively realizing the generation of the corresponding pathological state image according to the input pathological text, and improving the accuracy of the generated pathological state image.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0174] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for converting text to image, It is characterized in that The steps include: Acquiring first pathological text-image pair data; Performing vector extraction on the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculating the loss value according to the extracted first feature vector to obtain a first total loss value; Optimizing the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain an optimized encoder model and an optimized coupling layer; Acquiring second pathological text-image pair data; Performing vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculating a loss value according to the extracted second feature vector to obtain a second total loss value; Re-optimize the optimized encoder model and the optimized coupling layer according to the second total loss value to obtain an effective encoder model and an effective coupling layer; The actual pathological text is obtained, inputted into the effective encoder model and the effective coupling layer, and image decoding is performed to obtain a pathological state image.

2. The text-to-image method according to claim 1, It is characterized in that The step of obtaining the first pathological text-image pair data specifically includes: Acquire a first data identifier, and extract pathological text data and pathological image data from a database according to the first data identifier; Performing Gaussian blur processing on the pathological image data to obtain pathological Gaussian image data; The pathological text data and the pathological Gaussian image data are associated to obtain the first pathological text-image pair data.

3. The text-to-image method according to claim 1, It is characterized in that The step of extracting vectors from the first pathological text-image pair data according to the pre-trained encoder model and the preset coupling layer, and calculating the loss value according to the extracted first feature vector to obtain the first total loss value specifically includes: Inputting the first pathological text-image pair data into the pre-trained encoder model for feature encoding to obtain a first feature encoding vector; Inputting the first feature coding vector into the preset coupling layer for feature conversion to obtain the first feature vector; Perform text loss calculation and image loss calculation on the first feature vector to obtain a first total loss value.

4. The text-to-image method according to claim 3, It is characterized in that The step of performing text loss calculation and image loss calculation on the first feature vector to obtain a first total loss value specifically includes: Extracting a first text feature vector from the first feature vector; Obtaining a dimension corresponding to the first text feature vector, and performing loss estimation calculation according to the dimension corresponding to the first text feature vector and a standard Gaussian distribution to obtain a first text loss; Extracting a first image feature vector from the first feature vector; Inputting the first image feature vector into a preset image decoder for decoding to obtain a first image decoding vector; Extracting the pathological Gaussian image data in the first pathological text-image pair data, and performing loss estimation calculation based on the first image decoding vector and the pathological Gaussian image data to obtain a first image loss; The first text loss and the first image loss are summed to obtain the first total loss value.

5. The text-to-image method according to claim 1, It is characterized in that The step of obtaining the second pathological text-image pair data specifically includes: Acquire a second data identifier, and extract Gaussian-distributed pathological text data and pathological image data from a database according to the second data identifier; The Gaussian-distributed pathological text data and the pathological image data are associated to obtain the second pathological text-image pair data.

6. The text-to-image method according to claim 1, It is characterized in that The step of extracting vectors from the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and calculating the loss value according to the extracted second feature vector to obtain a second total loss value specifically includes: Inputting the second pathological text-image pair data into the optimized coupling layer to obtain a coupling vector; Inputting the coupling vector into an optimized encoder model for feature encoding to obtain the second feature vector; Perform text loss calculation and image loss calculation on the second feature vector to obtain a second total loss value.

7. The text-to-image method according to claim 6, It is characterized in that The step of performing text loss calculation and image loss calculation on the second feature vector to obtain a second total loss value specifically includes: Extracting a second text feature vector from the second feature vector; Obtaining the dimension corresponding to the second text feature vector, and performing loss estimation calculation according to the dimension corresponding to the second text feature vector and a standard Gaussian distribution to obtain a second text loss; extracting a second image feature vector from the second feature vector; Inputting the second image feature vector into a preset image decoder for decoding to obtain a second image decoding vector; Extracting the pathological image data in the second pathological text-image pair data, and performing loss estimation calculation based on the second image decoding vector and the pathological Gaussian image data in the second pathological text-image pair data to obtain a second image loss; The second text loss and the second image loss are summed to obtain the second total loss value.

8. A text-to-image device, It is characterized in that include: A first data acquisition module, used for acquiring first pathological text-image pair data; A first loss calculation module, used for performing vector extraction on the first pathological text-image pair data according to a pre-trained encoder model and a preset coupling layer, and performing loss value calculation according to the extracted first feature vector to obtain a first total loss value; A first optimization module, used to optimize the pre-trained encoder model and the preset coupling layer according to the first total loss value to obtain an optimized encoder model and an optimized coupling layer; A second data acquisition module, used to acquire second pathology text-image pair data; A second loss calculation module, used for performing vector extraction on the second pathological text-image pair data according to the optimized encoder model and the optimized coupling layer, and performing loss value calculation according to the extracted second feature vector to obtain a second total loss value; A second optimization module, configured to re-optimize the optimized encoder model and the optimized coupling layer according to the second total loss value to obtain an effective encoder model and an effective coupling layer; The image conversion module is used to obtain the actual pathological text, input the actual pathological text into the effective encoder model and the effective coupling layer, and perform image decoding to obtain a pathological state image.

9. A computer device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the text-to-image method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text-to-image method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Image-text generation model training method and device, electronic equipment and medium

    CN117094365A

  • Image description text generation method and device, computer equipment and storage medium

    CN117576689A

  • Training method and device with text image generation network, electronic equipment and medium

    CN118298192A

  • Training method of text recognition model, text recognition method, and apparatus

    US20220415071A1

  • Systems and methods for text-to-image generation using language models

    US20240185035A1