Image processing method, device, apparatus and storage medium

Abstract

The application discloses an image processing method, device, equipment and storage medium. The method comprises the following steps: acquiring a first medical image and staining mode conversion information, the first medical image having a first staining mode, and the staining mode conversion information being used for indicating conversion of the first staining mode into a second staining mode; performing encoding processing on the first medical image to obtain a first image representation; inputting the staining mode conversion information and the first image representation into a noise prediction model to perform noise prediction processing on a preset noise image representation according to the staining mode conversion information and the first image representation, obtain predicted noise, and perform denoising processing on the preset noise image representation according to the predicted noise to obtain a second image representation; and performing decoding processing on the second image representation to obtain a second medical image, the second medical image having the second staining mode. The method combines the first image representation having biological tissue detail information with the staining mode conversion information, and can improve the accuracy of virtual staining.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Histopathological examination is one of the important means of medical diagnosis, mainly involving the observation of human tissue samples under a microscope to analyze tissue structure and cellular characteristics. A key step in this process is the chemical staining of tissue sample sections. Different types of chemical dyes can interact with different components in tissues or cells, making it easier to distinguish different microstructures, abnormal substances, and lesion areas within the tissue.

[0003] Currently, related technologies propose virtual staining methods based on neural network models. For example, in histopathological examinations, it is often necessary to switch between different virtual staining modes, such as switching from hematoxylin-eosin (HE) staining to immunohistochemistry (IHC) staining, or switching from unstained mode to HE staining. For each pair of staining modes, the related technologies employ a separate neural network model.

[0004] Therefore, scenarios involving switching between multiple staining modalities require multiple neural network models, resulting in significant consumption of computational and storage resources. Although diffusion models are widely used in fields such as textual analysis, the complexity of histopathological examination makes it difficult to clearly describe the morphology and arrangement of tissue cells using only text. This information significantly impacts the staining results of slide images, making it challenging to utilize diffusion models to switch between different staining modalities in related technologies. Summary of the Invention

[0005] This application provides an image processing method and related methods, apparatus, devices, and storage media. It can improve the accuracy of a noise prediction model in converting a first medical image with a first staining pattern into a second medical image with a second staining pattern.

[0006] According to one aspect of this application, an image processing method is provided, comprising: acquiring a first medical image and staining pattern conversion information, the first medical image having a first staining pattern, the staining pattern conversion information being used to indicate converting the first staining pattern to a second staining pattern; encoding the first medical image to obtain a first image representation; inputting the staining pattern conversion information and the first image representation into a noise prediction model to perform noise prediction processing on a preset noisy image representation based on the staining pattern conversion information and the first image representation to obtain predicted noise, and performing denoising processing on the preset noisy image representation based on the predicted noise to obtain a second image representation; decoding the second image representation to obtain a second medical image, the second medical image having the second staining pattern; wherein the noise prediction model is trained based on a target loss, the target loss being determined based on a first noise and a second noise, the second noise being noise predicted by the noise prediction model for a noisy sample image representation based on a first sample image representation of a first sample medical image and sample staining pattern conversion information, the first noise being noise added to a second sample image representation of a second sample medical image, and the noisy sample image representation being obtained by adding the first noise to the second sample image representation.

[0007] According to another aspect of this application, an image processing apparatus is provided, comprising: an information acquisition module for acquiring a first medical image and staining pattern conversion information, wherein the first medical image has a first staining pattern, and the staining pattern conversion information is used to indicate converting the first staining pattern to a second staining pattern; an encoding processing module for encoding the first medical image to obtain a first image representation; and a representation generation module for inputting the staining pattern conversion information and the first image representation into a noise prediction model to perform noise prediction processing on a preset noise image representation based on the staining pattern conversion information and the first image representation to obtain predicted noise, and to perform noise prediction processing on the preset noise image representation based on the predicted noise. A noisy image representation is denoised to obtain a second image representation. A decoding processing module is used to decode the second image representation to obtain a second medical image, which has the second staining pattern. The noise prediction model is trained based on a target loss, which is determined based on a first noise and a second noise. The second noise is the noise predicted by the noise prediction model for the noisy sample image representation based on the first sample image representation of the first sample medical image and the sample staining pattern conversion information. The first noise is the noise added to the second sample image representation of the second sample medical image. The noisy sample image representation is obtained by adding the first noise to the second sample image representation.

[0008] Optionally, the representation generation module may include an information acquisition submodule, a noise coding submodule, a parameter determination submodule, an iterative decrementing submodule, and a representation generation submodule.

[0009] The information acquisition submodule is used to acquire the preset noisy image and the denoising processing iteration threshold;

[0010] The noise encoding submodule is used to encode the preset noise image to obtain a preset noise image representation;

[0011] The parameter determination submodule is used to determine the preset noise image representation as the current image representation and the denoising processing iteration threshold as the current denoising iteration value.

[0012] The iterative decrease submodule is used to decrease the current denoising iteration value according to a preset subtrahend;

[0013] The generation submodule is used to input the staining mode conversion information and the first image representation into the noise prediction model to perform noise prediction processing on the current image representation to obtain predicted noise, and to perform denoising processing on the current image representation based on the predicted noise to obtain an updated current image representation. Then, it returns the current denoising iteration value by decreasing it according to a preset subtraction until the current denoising iteration value is one, and determines the current image representation when the current denoising iteration value is one as the second image representation.

[0014] Optionally, the generation submodule may include a time acquisition unit, an information encoding unit, a feature extraction unit, and a noise prediction unit.

[0015] A time acquisition unit is used to acquire time steps and encode the time steps to obtain a time step representation.

[0016] An information encoding unit is used to encode the coloring pattern conversion information to obtain a coloring pattern conversion representation;

[0017] The feature extraction unit is used to perform feature extraction processing based on the staining mode conversion representation, the time step representation, the current image representation, and the first image representation to obtain a conditional control representation;

[0018] The noise prediction unit is used to perform noise prediction processing based on the coloring mode conversion representation, the time step representation, the current image representation, and the conditional control representation to obtain predicted noise.

[0019] Optionally, the feature extraction unit may include a first convolution subunit, an addition subunit, a feature extraction subunit, and a second convolution subunit.

[0020] The first convolutional subunit is used to perform a first convolution process on the first image representation to obtain a convolutional image representation;

[0021] An addition calculation subunit is used to perform an addition calculation on the current image representation and the convolutional image representation to obtain a first intermediate representation;

[0022] The feature extraction subunit is used to perform feature extraction processing on the staining pattern transformation representation, the time step representation, and the first intermediate representation to obtain the second intermediate representation;

[0023] The second convolution subunit is used to perform a second convolution process on the second intermediate representation to obtain the conditional control representation.

[0024] Optionally, the feature extraction subunit can be specifically used for:

[0025] The coloring pattern transformation representation, the time step representation, and the first intermediate representation are subjected to feature extraction processing to obtain an intermediate feature representation;

[0026] Feature extraction processing is performed on the staining pattern transformation representation, the time step representation, and the intermediate feature representation to obtain a second intermediate representation.

[0027] Optionally, the feature extraction subunit can be specifically used for:

[0028] The coloring mode transformation representation and the first intermediate representation are fused to obtain a first prompt image fusion representation. Feature extraction is performed based on the time step representation and the first prompt image fusion representation to obtain a first prompt image feature representation.

[0029] The attention weight of the first image region is calculated based on the feature representation of the first prompt image. Then, the representation weighting process is performed based on the first image region attention weight and the fusion representation of the first prompt image to obtain the intermediate feature representation.

[0030] The coloring mode transformation representation and the intermediate feature representation are fused to obtain a second prompt image fusion representation. Feature extraction is performed based on the time step representation and the second prompt image fusion representation to obtain a second prompt image feature representation.

[0031] The second image region attention weight is calculated based on the feature representation of the second prompt image. The second intermediate representation is obtained by weighting the representation based on the second image region attention weight and the fusion representation of the second prompt image.

[0032] Optionally, the conditional control representation includes a first conditional control representation and a second conditional control representation; the second convolutional subunit may be specifically used for:

[0033] The intermediate feature representation is subjected to a second convolution process to obtain the first conditional control representation;

[0034] The second intermediate representation is subjected to a second convolution process to obtain the second conditional control representation.

[0035] Optionally, the noise prediction unit can be specifically used for:

[0036] Feature extraction processing is performed on the staining mode transformation representation, the time step representation, and the current image representation to obtain a third intermediate representation;

[0037] Feature extraction processing is performed on the staining pattern transformation representation, the time step representation, and the third intermediate representation to obtain a fourth intermediate representation;

[0038] The fourth intermediate representation and the first conditional control representation are added together to obtain the fifth intermediate representation;

[0039] The coloring mode conversion representation, the time step representation, and the fifth intermediate representation are decoded to obtain the sixth intermediate representation;

[0040] The sixth intermediate representation and the second conditional control representation are added together to obtain the predicted noise.

[0041] Optionally, the image processing apparatus may also include a staining quality scoring module, a synonym suggestion determination module, and a slice image update module.

[0042] The staining quality scoring module is used to determine that the second medical image is the current medical image, perform quality detection processing on the current medical image, and obtain the staining quality score corresponding to the current medical image.

[0043] The synonym suggestion determination module is used to obtain synonymous staining pattern conversion information that has a semantic similarity greater than a preset similarity threshold with the staining pattern conversion information when the staining quality score is less than the staining quality threshold, and update the staining pattern conversion information according to the synonymous staining pattern conversion information;

[0044] The slice image update module is used to return the staining pattern conversion information and the first image representation as inputs to the noise prediction model to regenerate a new second medical image until the staining quality score is not less than the staining quality threshold. The current medical image with a staining quality score not less than the staining quality threshold is determined as the second medical image.

[0045] Optionally, the staining pattern conversion information includes text information indicating that a first medical image with a first staining pattern is converted into a second medical image with a second staining pattern, as well as feature description information of the first staining pattern and feature description information of the second staining pattern.

[0046] Optionally, the image processing device may further include a sample information acquisition module, a sample image encoding module, a first noise addition module, a second noise prediction module, a target loss determination module, and a prediction model training module.

[0047] The sample information acquisition module is used to acquire a first sample medical image, a second sample medical image, and sample staining mode conversion information. The sample staining mode conversion information includes text information that converts the first sample medical image of the third staining mode into the second sample medical image of the fourth staining mode.

[0048] The sample image encoding module is used to encode the first sample medical image to obtain a first sample image representation, and to encode the second sample medical image to obtain a second sample image representation;

[0049] The first noise addition module is used to add first noise to the second sample image representation to obtain a noisy sample image representation;

[0050] The second noise prediction module is used to input the sample staining mode conversion information, the first sample image representation and the noisy sample image representation into the noise prediction model to perform noise prediction and obtain the second noise.

[0051] The target loss determination module is used to calculate the target loss based on the error between the first noise and the second noise.

[0052] The prediction model training module is used to iteratively train the noise prediction model based on the target loss until the training result conditions are met, thereby obtaining the trained noise prediction model.

[0053] Optionally, the noise prediction model includes a cue encoder, a time encoder, a noise prediction network, and a conditional control network. The conditional control network is used to input a sample conditional control representation into the noise prediction network to control the noise prediction network to perform noise prediction that conforms to the sample conditional control representation.

[0054] The second noise prediction module can be specifically used for:

[0055] The prompt encoder encodes the sample staining pattern conversion information into a sample staining pattern conversion representation.

[0056] The time step is acquired and encoded into a time step representation using the time encoder.

[0057] The sample staining mode transformation representation, the time step representation, the noisy sample image representation, and the first sample image representation are input into the conditional control network for feature extraction to obtain the sample conditional control representation.

[0058] The sample staining mode transformation representation, the time step representation, the noisy sample image representation, and the sample condition control representation are input into the noise prediction network to predict noise and obtain the second noise.

[0059] Optionally, the prediction model training module can be specifically used for:

[0060] Fix the first weight parameters corresponding to the prompt encoder, the time encoder, and the noise prediction network;

[0061] The second weight parameters corresponding to the conditional control network are iteratively trained based on the target loss to update the second weight parameters until the training termination condition is met, thereby obtaining the trained noise prediction model.

[0062] Optionally, the first noise addition module can be specifically used for:

[0063] A two-dimensional standard Gaussian noise is randomly sampled as the first noise;

[0064] Calculate the first combination coefficient and the first product value represented by the second sample image;

[0065] Calculate the second product of the second combination coefficient and the first noise;

[0066] The first product and the second product are added together to obtain a noisy sample image representation. The first combination coefficient and the second combination coefficient are determined by the random sampling time step.

[0067] According to another aspect of this application, a computer-readable storage medium is provided that stores a computer program, wherein the image processing method described above is executed when the computer program is run by a processor.

[0068] According to another aspect of this application, a computer device is provided, which includes a processor and a memory, the memory storing a computer program that, when invoked by the processor, executes the image processing method described above.

[0069] According to another aspect of this application, a computer program product is provided, which includes a computer program stored in a storage medium; a processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to perform the above-described image processing method.

[0070] While simple text is insufficient to clearly describe the morphology, arrangement, and other biological tissue information of cells, this embodiment encodes the first medical image to be converted to a staining mode, obtaining a first image representation. This first image representation contains more detailed and clearer biological tissue information, including the morphology and arrangement of cells. Combining this first image representation with staining mode conversion information allows them to work together to guide the denoising process of the noise prediction model. The resulting second medical image with the second staining mode thus reflects the specific biological tissue details, such as the morphology and arrangement of cells in the first medical image, improving the accuracy of virtual staining using the second staining mode and overcoming the obstacle of related technologies where raw image processing cannot be accurately used for section staining mode switching. Furthermore, the target loss used by the noise prediction model during training in this embodiment is determined by a first noise and a second noise. The first noise is noise added to the second sample image representation of the second sample medical image. The second noise is noise predicted by the noise prediction model based on the first sample image representation of the first sample medical image and the sample staining mode conversion information. Compared to typical training methods for noise prediction models, this training approach, when predicting the second noise, considers not only textual information but also the first sample image representation converted from the first sample medical image. This improves the reliability of the trained noise prediction model in switching staining modes. Furthermore, since training involves adding the first noise to the second sample image representation and then predicting the second noise based on the result of the noise addition, the target loss is calculated based on the first and second noises. This method enhances the accuracy of the trained noise prediction model in staining mode conversion, thereby improving the accuracy of virtual staining.

[0071] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 A schematic diagram of a system architecture provided in an embodiment of this application is shown.

[0074] Figure 2 A schematic diagram of a system deployment provided by an embodiment of this application is shown.

[0075] Figure 3 The illustration shows an application scenario of an image processing method provided in an embodiment of this application.

[0076] Figure 4 The diagram illustrates an application scenario of another image processing method provided in an embodiment of this application.

[0077] Figure 5 A schematic flowchart of an image processing method provided in an embodiment of this application is shown.

[0078] Figure 6 This illustration shows a schematic diagram of a second image representation generation process provided in an embodiment of this application.

[0079] Figure 7 A flowchart illustrating an iterative generation process for a second image representation provided in an embodiment of this application is shown.

[0080] Figure 8 A schematic diagram of a noise prediction processing method provided in an embodiment of this application is shown.

[0081] Figure 9 The diagram shows a model architecture diagram of a noise prediction model provided in an embodiment of this application.

[0082] Figure 10 A network architecture diagram of a condition control network provided in an embodiment of this application is shown.

[0083] Figure 11 A schematic diagram of the generation process of a second intermediate representation provided in an embodiment of this application is shown.

[0084] Figure 12 This illustration shows a schematic diagram of another second intermediate representation generation process provided by an embodiment of this application.

[0085] Figure 13 A schematic diagram of a prediction noise generation process provided in an embodiment of this application is shown.

[0086] Figure 14 A flowchart illustrating another image processing method provided in an embodiment of this application is shown.

[0087] Figure 15 A flowchart illustrating a staining pattern conversion provided in an embodiment of this application is shown.

[0088] Figure 16 The diagram shows a flowchart illustrating a training method for a noise prediction model provided in an embodiment of this application.

[0089] Figure 17 This is a schematic diagram of the generation process of a noisy sample image representation provided in an embodiment of this application.

[0090] Figure 18 This is a schematic diagram of a second noise prediction process provided in an embodiment of this application.

[0091] Figure 19 This is a block diagram of an image processing apparatus provided in an embodiment of this application.

[0092] Figure 20 This is a module block diagram of a computer device provided in an embodiment of this application.

[0093] Figure 21 This is a block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0094] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The implementation methods described with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0095] In some processes described in the specification, claims, and the foregoing drawings, multiple steps are included in a specific order. However, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not themselves represent any execution order. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order, sequence, or quantity.

[0096] It is worth noting that, in the specific embodiments of this application, data such as first medical images and staining pattern conversion information are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, when an embodiment of this application needs to obtain staining pattern conversion information, separate permission or consent for obtaining the staining pattern conversion information can be obtained through pop-ups or redirection to a confirmation page. After obtaining separate permission or consent, the necessary related data for enabling the embodiments of this application to operate normally can then be obtained.

[0097] Tissue staining is a standard method used in clinical pathology and life science research to examine the structure of tissue cells. Traditional chemical staining methods are based on the principle that staining reagents can interact specifically with different components of tissues or cells. Different types of staining reagents are used to stain different components of tissues or cells to reveal color differences, thereby making it easier to distinguish different microstructures, abnormal substances, and lesions in tissues or cells, which are undetectable in unstained tissues.

[0098] Chemical staining methods require complex sample preparation, specialized laboratory facilities, and highly trained technicians, making them difficult to standardize in resource-constrained environments. Currently, computer vision technology, through training deep neural networks, can digitally generate stained images of tissue sections, providing a virtual staining method as an alternative to chemical staining. This method is often referred to as image-to-image conversion, such as using generative adversarial networks to stain hematoxylin and eosin from tissue section images input to the network.

[0099] Due to the complexity of histopathological examination, it is difficult to clearly describe the morphology and arrangement of tissue cells using only textual descriptions. This information significantly impacts the staining results of tissue sections, making it challenging to use diffusion models to switch between different staining modalities. Furthermore, the virtual staining models trained for these techniques cannot handle the application of multiple staining reagents to tissue sections. For example, performing virtual staining for immunohistochemistry (IHC) and hematoxylin-eosin requires training one or two virtual staining models. To perform three types of virtual staining, three to six models need to be trained. This limits the types of staining that can be applied to specific staining types in tissue sections.

[0100] To address the aforementioned problems, the inventors, through research, proposed the image processing method provided in the embodiments of this application. The system implementation environment and related application scenarios of the method involved in this application will be described below.

[0101] Please see Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The image processing method provided in this application embodiment can be applied in system 100. The data acquisition device 110 is used to acquire tissue slice images as sample medical images. For example, the data acquisition device 110 can acquire a large number of tissue slice images from an image database storing tissue slice images, and preprocess these tissue slice images, such as cropping them to a size suitable for model training. The preprocessed tissue slice images are then used as sample medical images. Furthermore, the data acquisition device 110 can also acquire sample staining pattern conversion information from a prompt database. This sample staining pattern conversion information in the prompt database can be manually written. Then, the data acquisition device 110 can store the sample medical images and sample staining pattern conversion information as training data in database 120 for model training.

[0102] Optionally, the training data maintained in database 120 may not all come from data acquisition device 110; it may also be received from other devices. For example, execution device 140 may also store acquired tissue slice images in database 120 from an open digital tissue slice image library on the Internet. The above description should not be construed as limiting the embodiments of this application. Training device 130 can train the noise prediction model based on the training data in database 120 until the noise prediction model meets the training termination condition, thus obtaining the trained noise prediction model, i.e., the target model 101.

[0103] The training termination conditions may include: the total loss value of the target loss is less than a preset value, the total loss value of the target loss is within a threshold range, or the training iterations have reached a preset number. The target model 101 can be a deep neural network composed of one or more neural networks. For example, the target model 101 can be a trained noise prediction model, whose backbone is a diffusion model, such as the standard Denoising Diffusion Probabilistic Model (DDPM), Latent Diffusion Model (LDM), and Stable Diffusion. Furthermore, the network architecture of the diffusion model can include a U-Net structured neural network or a Swin-Transformer, etc., without limitation.

[0104] The execution device 140 can deploy the target model 101 trained by the training device 130. During the computation and other related processing performed by the processing module 141 of the execution device 140, the execution device 140 can call data, programs, etc., in the data storage system 142 for corresponding computational processing, and store the processing results and instructions obtained from the computational processing into the data storage system 142. For example, when the execution device 140 receives the first medical image and staining pattern conversion information sent by the user device 150, it can call the processing module 141 to run the target model 101 to perform staining pattern conversion on the first medical image according to the staining pattern conversion information, and return the staining pattern conversion result, i.e., the second medical image, to the user device 150. Data transmission between the execution device 140 and the user device 150 can be performed through an input / output (I / O) interface.

[0105] The aforementioned training device 130, execution device 140, and user device 150 can be computer devices such as servers or terminals. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), blockchain, and big data and artificial intelligence platforms. The terminal can be a laptop, tablet, desktop computer, etc.

[0106] It should be noted that, Figure 1 This is merely a schematic diagram of a system architecture provided in this application embodiment. The system architecture and application scenarios described in this application embodiment are for the purpose of more clearly illustrating the technical solutions of this application embodiment and do not constitute a limitation on the technical solutions provided in this application embodiment. For example, Figure 1 The execution device 140 can be used to deploy the target model 101 trained by the training device 130. In other cases, the user device 150 can also directly deploy the target model 101 trained by the training device 130 and perform coloring mode conversion locally.

[0107] Please see Figure 2 , Figure 2 A schematic diagram of a system deployment is shown. Exemplarily, the image processing method provided in this application embodiment can also be deployed in, for example, Figure 2 The system shown is a C / S (Client / Server) architecture system. This system may include servers, gateways, the internet, and devices.

[0108] A server is a computer system that provides certain services to the server-side. Compared to ordinary computer equipment, servers have higher requirements in terms of stability, security, and performance. Figure 2 As shown, server 210 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0109] A gateway is also known as an internetwork connector or protocol converter. For example... Figure 2 As shown, gateway 220 achieves network interconnection at the transport layer via Internet 230, and is a computer system or device that acts as a translator. Between two systems using different communication protocols, data formats, or languages, or even completely different architectures, a gateway is a translator. At the same time, a gateway can also provide filtering and security functions.

[0110] Devices can send requests to servers, requesting access to or use resources, data, or services on the server. Devices typically provide a user interface for interaction with the system, and generate requests in response to user interactions on that interface. For example... Figure 2 As shown, the device can include a tablet computer 240a, a desktop computer 240b, and a laptop computer 240c, and can also include various forms such as a personal digital assistant (PDA). Furthermore, it can be a single device or a collection of multiple devices.

[0111] The device can communicate with the server and exchange data via wired or wireless means. For example... Figure 2 As shown, tablet computer 240a can communicate with the Internet 230 via wireless router 250, and messages sent by desktop computer 240b to server 210 via Internet 230 can be transmitted to the corresponding server 210 via gateway 220. Messages sent by server 210 to laptop computer 240c via gateway 220 can also be transmitted to laptop computer 240c via Internet 230.

[0112] In one implementation, the image processing method and related methods of this application can be implemented partly on a server and partly on a device. For example, the server trains and deploys a trained noise prediction model, and the device can send a first medical image and staining pattern conversion information to the server. Then, after receiving the first medical image and staining pattern conversion information, the server runs the noise prediction model to perform staining pattern conversion on the first medical image according to the staining pattern conversion information, obtains a second medical image after staining pattern conversion, and returns the second medical image to the device.

[0113] It should be noted that, Figure 2 This is merely a schematic diagram of a system deployment provided in this application embodiment. The system deployment scheme described in this application embodiment is only for more clearly illustrating the technical solutions of this application embodiment and does not constitute a limitation on the technical solutions provided in this application embodiment. For example, "device" can refer to one of multiple devices, and this embodiment only uses one as an example. Figure 2 The three devices shown (tablet computer 240a, desktop computer 240b, and laptop computer 240c) are used as examples. Those skilled in the art will recognize that, as system deployment schemes evolve, the technical solutions provided in the embodiments of this invention are equally applicable to similar technical problems.

[0114] The embodiments of this application can be applied in various scenarios, such as Figure 3 The intelligent medical teaching scenario shown and Figure 4 The scenarios shown include those for pathological auxiliary analysis.

[0115] (1) Intelligent medical teaching scenario.

[0116] In pathology classes, teachers can use virtual staining systems to virtually stain tissue slide images, improving teaching efficiency and quality. In some embodiments, the image processing method based on this application can be used to develop a virtual staining system for virtually staining tissue slide images. Figure 3 As shown, the virtual staining system may include a server 311 deployed on a cloud server 310 and a client 321 deployed on a teaching computer 320. The server 311 is primarily responsible for performing complex calculations or logical processing, processing, analyzing, or performing operations on the data requested by the client 321. For example, it may perform virtual staining based on a first medical image and staining pattern conversion information sent by the client 321. The client 321 is primarily responsible for providing an interactive interface, sending requests to the server 311 based on user actions on the interface, and presenting the data.

[0117] like Figure 3As shown, on the client 321 interface, the user can click the image upload box 3211 to upload the image to be stained (e.g., an unstained tissue section image) and write staining prompts in the text editing bar 3213. Optionally, the user can also click button 3214 to select to use the edited prompt template. After completing the upload of the image to be stained and the writing of the staining prompts, the user can click button 3215. In response to the click operation of button 3215, the client 321 can generate a staining request. Then, the teaching computer 320 can send the staining request generated by the client 321 to the server 311 of the cloud server 310. When the server 311 receives the staining request, it can perform image staining based on the image to be stained and the staining prompts in the staining request to obtain the stained target image. Further, the cloud server 310 returns the target stained image to the client 321 of the teaching computer 320, and the client 321 displays the target stained image on the interface.

[0118] (2) Pathological auxiliary analysis scenario.

[0119] Histopathology uses tissue sections to analyze the impact of diseases on the body, including the mechanisms of disease occurrence, lesion types, and progression patterns, revealing structural abnormalities in tissues and cells under disease conditions. Different staining methods can reveal different characteristics of tissues. Therefore, pathologists need to restain tissue sections for comprehensive observation, helping them to fully analyze tissue lesions, which is of great significance for developing personalized treatment plans. After collecting tissue samples, pathologists can obtain the corresponding tissue section images and upload them to a virtual staining system for virtual staining. This virtual staining system can be developed based on the image processing method of this application. Pathologists can complete the virtual staining of tissue section images by editing staining prompts and uploading tissue section images in the editing bar of the virtual staining system.

[0120] like Figure 4In the illustrated example of a pathological analysis scenario for thyroid tissue, after steps such as tissue sampling, embedding and sectioning, and scanning of the sections, the display of the analysis device 400 shows an unstained thyroid tissue section image 410. The pathologist can use the virtual staining system installed in the analysis device 400 to stain the unstained thyroid tissue section image 410. For example, the pathologist can edit the staining prompt "Given an unstained pathological tissue section image, generate a corresponding HE-stained pathological tissue section image," and then the virtual staining system outputs an HE-stained thyroid tissue section image 420. Since HE staining cannot clearly show the characteristics of tumor cells or distinguish between benign and malignant cells for poorly differentiated thyroid cancer, IHC restaining is required to identify specific thyroid cancer markers, such as carcinoembryonic antigen (CEA) and Galectin-3 protein.

[0121] Pathologists can further edit a staining prompt: "Given an HE-stained pathological tissue section image, generate a corresponding IHC-stained pathological tissue section image." The virtual staining system then outputs an IHC-stained thyroid tissue section image 430, allowing pathologists to more easily locate and identify cellular regions expressing these markers, more accurately differentiate thyroid cancer subtypes, and assess tumor differentiation. For example, CEA expression supports the diagnosis of medullary thyroid carcinoma, while high Galectin-3 expression supports the possibility of papillary thyroid carcinoma. Furthermore, pathologists can edit different staining prompts on the virtual staining system as needed to switch between unstained, HE-stained, and IHC-stained thyroid tissue section images without physical restaining, preserving the integrity of the section sample and allowing for rapid switching to different staining modes to analyze other markers.

[0122] It should be noted that, Figure 3 and Figure 4 These are merely illustrative diagrams illustrating two application scenarios provided in this application's embodiments. The application scenarios described in this application's embodiments are only for more clearly illustrating the technical solutions of this application's embodiments and do not constitute a limitation on the technical solutions provided in this application's embodiments. The image processing method proposed in this application can also be applied to other application scenarios requiring virtual tissue staining. As those skilled in the art will understand, with the evolution of application scenarios, the technical solutions provided in the embodiments of this invention are equally applicable to similar technical problems.

[0123] According to one embodiment of this application, an image processing method is provided. This image processing method can be executed by a computer device (e.g., a server, an edge computing device, or other terminal device with certain processing capabilities), which at least has storage, computing, and communication functions. Figure 5 A flowchart illustrating the image processing method provided in this embodiment is shown. Figure 5 As shown, the image processing method may specifically include:

[0124] Step 510: Obtain the first medical image and staining pattern conversion information.

[0125] Virtual staining technology can simulate the effects of chemical staining on tissue slide images using neural network models. The image processing method provided in this application can convert tissue slide images into various staining modes based on staining mode conversion information. Here, staining mode refers to the staining effect of traditional staining methods on tissue slides using algorithms, such as HE staining, IHC staining, and silver staining. By converting staining modes, biological information such as tissue structure under different staining effects can be observed on the same tissue, thereby achieving visual enhancement of biological information from traditional staining on digital images, which is beneficial for pathological analysis and diagnosis. The staining mode conversion information in this application refers to text information used to instruct the conversion of a first medical image with a first staining mode into a second medical image with a second staining mode, such as "Given an IHC-stained tissue slide image, generate the corresponding HE-stained tissue slide image."

[0126] In this embodiment, the first medical image and the second medical image refer to tissue slide images. These tissue slide images are digital images of biological tissue slides obtained through microscopy and digitization techniques. These images are obtained after processing and scanning thin-layer samples cut from biological tissues (such as human, animal, or plant) tissues. The first and second medical images, as tissue slide images, typically have high resolution and rich colors after staining, and can be used in medical diagnosis, scientific research, and teaching applications.

[0127] The first staining pattern refers to the original staining pattern to be converted. Correspondingly, the first medical image is the tissue section image of the original staining pattern. For example, in the staining pattern conversion information example above, "IHC staining" is the first staining pattern, and "IHC stained tissue section image" is the first medical image. The second staining pattern refers to the target staining pattern, which is another staining pattern different from the first staining pattern. Correspondingly, the second medical image is the tissue section image of the target staining pattern. For example, in the staining pattern conversion information example above, "HE staining" is the second staining pattern, and "HE stained tissue section image" is the second medical image.

[0128] As an implementation, in the case where a part of the image processing method as described above is implemented on the terminal and another part is implemented on the server, the acquisition method of the first medical image and the staining pattern conversion information may include downstream task sending. The downstream task refers to the task to which the image processing method of this application is applied. For example, tasks such as virtual tissue staining for intelligent medical teaching and virtual restaining for case-assisted analysis. Take Figure 3 the intelligent medical teaching scenario shown as an example. The server 310 may receive the first medical image: "unstained tissue section image" and the staining pattern conversion information: "perform HE staining on the unstained tissue section image" sent by the teaching computer 320.

[0129] Considering that the image processing method of this application is actually a technology for generating tissue section images of a specific staining pattern using narrative text information, in order to enhance the quality of the generated images, characteristic description information of the first staining pattern and characteristic description information of the second staining pattern may be added to the staining pattern conversion information. That is, in addition to the text information indicating the conversion of the first medical image of the first staining pattern into the second medical image of the second staining pattern, the staining pattern conversion information may also include the characteristic description information of the first staining pattern and the characteristic description information of the second staining pattern. Among them, the characteristic description information refers to the text information used to describe some special attributes of the staining pattern in actual staining applications.

[0130] For example, for the staining pattern conversion information involving "HE staining to IHC staining", the characteristic description information of the HE staining pattern and the characteristic description information of the IHC staining pattern may be added. For example, the characteristic description information of the HE staining pattern: "<HE staining>: The nucleus of red blood cells is stained with hematoxylin, and the cytoplasm and other tissue structures are stained with eosin to blue / violet, making the nucleus appear purplish red, and the cytoplasm and other structures appear pink to red, providing good contrast of cells and tissue structures. HE staining can provide comprehensive tissue morphological information and is suitable for preliminary diagnosis and evaluation of tissue structures, but it cannot provide information about specific molecular markers". The characteristic description information of the IHC staining pattern: "<IHC staining>: Usually, an enzyme-labeled antibody (such as horseradish peroxidase or alkaline phosphatase) is used to bind to a specific antigen, and then a brown color is presented through a chromogenic substrate. The labeled antigen shows specific brown deposits under the microscope, and the background is usually stained blue or other colors to enhance the contrast. IHC staining can provide detailed information about the localization and expression level of specific proteins or antigens and has high specificity. This is very helpful for understanding the molecular mechanism of diseases and performing accurate pathological diagnoses".

[0131] Thus, by adding characteristic descriptions of the first and second staining modes to the staining mode conversion information, the noise prediction model used in the subsequent staining conversion process can capture the differences between the first and second staining modes. Based on these differences, the model can fully learn the different staining knowledge between the two staining modes, thereby improving the accuracy of staining mode conversion for tissue section images.

[0132] Step 520: Encode the first medical image to obtain a first image representation.

[0133] The first image representation refers to the representation of the first medical image in the latent space. The dimension of the first image representation is lower than that of the first medical image. This is because directly inputting a high-dimensional first medical image into a noise prediction model results in a large computational burden and susceptibility to noise interference. Therefore, the first medical image can be encoded first, compressing it from a high-dimensional image to a low-dimensional representation.

[0134] In one implementation, a computer device can encode a first medical image using an image encoder to obtain a first image representation. The image encoder can be any neural network capable of mapping an image from a high-dimensional pixel space to a low-dimensional latent space; this image encoder is a pre-trained neural network. For example, the image encoder can be the encoder part of a variational autoencoder (VAE), a convolutional neural network (CNN), etc., and the network type of the image encoder is not limited here.

[0135] For example, if the first medical image has a resolution of 512×512 pixels, inputting this image into the VAE encoder for encoding will output a feature map with a resolution of 64×64 pixels, which is the representation of the first image. Thus, by compressing the image into a representation in the latent space through the image encoder, the data dimensionality can be effectively reduced, significantly lowering the computational overhead of subsequent staining pattern conversion of tissue slice images, accelerating the staining pattern conversion speed, and allowing the model to more accurately focus on the detailed features of the tissue slice image, such as tissue structure and cell morphology, in the low-dimensional latent space, which helps improve the image quality of the staining pattern conversion.

[0136] Step 530: Input the coloring mode conversion information and the first image representation into the noise prediction model to perform noise prediction processing on the preset noise image representation based on the coloring mode conversion information and the first image representation to obtain predicted noise, and perform denoising processing on the preset noise image representation based on the predicted noise to obtain the second image representation.

[0137] In this embodiment, the noise prediction model is trained based on a target loss, which is determined based on a first noise and a second noise. The second noise is the noise predicted by the noise prediction model for the noisy sample image representation based on the first sample image representation of the first sample medical image and sample staining pattern conversion information. The first noise is the noise added to the second sample image representation of the second sample medical image. The noisy sample image representation is obtained by adding the first noise to the second sample image representation. The specific training process of the noise prediction model will be described in detail in other embodiments below, and will not be repeated here.

[0138] Unlike general technical fields that utilize noise prediction models to generate text-based images (such as product concept image generation, image style transfer, or photo background replacement), the image processing method in this application is primarily applied to the conversion of staining patterns in tissue section images, i.e., generating tissue section images under new staining patterns. Since technical fields using tissue section images (such as pathological analysis and medical research) have very strict requirements for the accuracy of biological information, it is essential to ensure that only the staining pattern changes before and after the staining pattern conversion, while the biological information such as tissue structure (e.g., cell morphology) remains unchanged. Clearly, related technologies that generate target images solely based on textual descriptions cannot be used to convert the staining patterns of tissue section images, especially for generating tissue section images with strict and specific requirements for biological information.

[0139] Therefore, this application proposes that in the denoising process of staining pattern conversion using a noise prediction model, in addition to enabling the noise prediction model to learn how the staining pattern is converted based on the staining pattern conversion information, the noise prediction model also uses the first image representation of the first medical image as a conditional control to ensure that the content of the tissue slice image remains unchanged, that is, to generate a second medical image with the same biological information as the first medical image.

[0140] In some embodiments, the process by which a computer device generates a second image representation of a second medical image from a first image representation of a first medical image using a noise prediction model is a sampling process. This sampling process starts from a purely noisy image representation and gradually denoises the noisy image representation through iterative calculation to generate a second image representation of the second medical image. Figure 6 The generation process of the second image representation is shown, which may include steps 531 to 535.

[0141] Step 531: Obtain the preset noisy image and the denoising processing iteration threshold.

[0142] In this embodiment, the process by which the noise prediction model generates a second image representation of a second medical image from a first image representation of a first medical image is a denoising process. This process can be simply understood as: predicting noise in an image, then removing the predicted noise from that image. After multiple rounds of noise prediction and removal, a denoised image is obtained. The starting point of the denoising process is a pure noise image, i.e., a preset noise image. This pure noise image is a noise value randomly sampled from a probability distribution, which may include Gaussian distribution and uniform distribution, etc. For example, randomly sampling a two-dimensional image from a two-dimensional standard Gaussian distribution means that each pixel value of this two-dimensional image is sampled from an independent standard Gaussian distribution.

[0143] The pre-defined noisy image serves as the initial input data in the noise prediction model's generation process. The denoising iteration threshold is a preset value that specifies the number of noise prediction and denoising operations required during the noise prediction model's processing. Since the generation process represented by the second image involves progressively predicting and denoising the predicted noise, an iterative calculation method can be used to complete this progressive noise prediction and denoising process. The total number of iterations is the denoising iteration threshold. Typically, the denoising iteration threshold can be determined by a maximum time step. This maximum time step can be an experimental value obtained through numerous experiments or a general empirical value. For example, when the maximum time step T = 1000, the total number of iterations is 1000.

[0144] As one implementation method, a computer device can randomly sample noise from a two-dimensional standard Gaussian distribution (also known as a standard normal distribution) as an initial preset noise image x. T ~N(0,I), meaning that each pixel value of the preset noisy image is an independent Gaussian distribution with a mean of 0 and a variance of I of 1. Therefore, the preset noisy image has no explicit image structure or coloring pattern. This initialization step simulates a "completely random" state and is the starting point for the noise prediction model to move from disorder to order and from random noise to a clear image. The preset noisy image has the same size as the second medical image. For example, if the computer ultimately generates an H×W second medical image (image height H, image width W), then the randomly sampled noise from the computer, as the preset noisy image, also needs to be an image of size H×W. This consistency in image size ensures that each denoising process is performed in the correct dimensions throughout the entire generation process, thus generating an image of the required size in the final step.

[0145] Step 532: Encode the preset noise image to obtain a preset noise image representation.

[0146] As an implementation, in order to reduce computing resources and improve computing efficiency, it is also necessary to compress the preset noise image into an image representation. Therefore, the computer device can use an image encoder to encode the preset noise image to obtain a preset noise image representation. For example, the preset noise image x T is input into the encoder of the VAE for encoding to obtain a preset noise image representation z T = E(x T ), where h < H, w < W, and E(·) is the encoder. Optionally, when obtaining the preset noise image, the computer device can also directly randomly sample noise of a preset dimension as the preset noise image representation. The preset dimension is the same as the dimension of the first image representation, so that the encoding of the noise image can be omitted, and the computing resource consumption required for encoding can be reduced during the iterative denoising process.

[0147] Step 533: Determine the preset noise image representation as the current image representation, and determine the denoising processing iteration threshold as the current denoising iteration value.

[0148] In the embodiment of the present application, the current image representation refers to the image representation used as noise prediction by the noise prediction model during each iteration of generating the second image representation, that is, the input of noise prediction. The current denoising iteration value refers to the iteration value corresponding to each iteration process during the iteration process of generating the second image representation. For example, in the first iteration process, the corresponding current denoising iteration value is that the denoising processing iteration threshold is 1000. Then, in the second iteration process, the corresponding current denoising iteration value can be the denoising processing iteration threshold minus one, that is, 999.

[0149] Step 534: Decrease the current denoising iteration value according to a preset subtractor.

[0150] As an implementation, the preset subtractor refers to the subtractor used by the computer device to perform subtraction calculation on the current denoising iteration value every time an iteration process is completed. Usually, the preset subtractor is set to 1. In the first iteration process of generating the second image representation by the computer device, the preset noise image representation can be determined as the current image representation to perform noise prediction processing using the noise prediction model, and denoise the current image representation to obtain an image representation for the second iteration process, and subtract 1 from the current denoising iteration value to obtain a denoising iteration value for the second iteration generation process, and so on. Each iteration generation process has a corresponding current image representation and current denoising iteration value.

[0151] Step 535: Input the coloring mode conversion information and the first image representation into the noise prediction model to perform noise prediction processing on the current image representation to obtain predicted noise, and perform denoising processing on the current image representation based on the predicted noise to obtain the updated current image representation, and return to decrease the current denoising iteration value according to the preset subtraction until the current denoising iteration value is one, and determine the current image representation when the current denoising iteration value is one as the second image representation.

[0152] Please see Figure 7 , Figure 7 The iterative process for generating the second image representation is illustrated, and the computer device can perform actions such as... Figure 7 The steps are as follows: Step a: Obtain the preset noise image; Step b: Initialize the current image table to the preset noise image, and initialize the current denoising iteration value to the denoising processing iteration threshold; Step c: Determine whether the current denoising iteration value is greater than 0; If the current denoising iteration value is greater than 0, execute steps d, e, and f.

[0153] In this process, step d involves inputting the coloring mode conversion information and the first image representation into the noise prediction model to perform noise prediction processing on the current image representation based on the coloring mode conversion information and the first image representation, thereby obtaining predicted noise. Step e involves denoising the current image representation based on the predicted noise, thereby obtaining an updated current image representation. Step f involves decrementing the current denoising iteration value by 1. Further, step f returns to step c. If the current denoising iteration value is not greater than 0, that is, the iteration process is complete, step g is executed: the current image representation is used as the second image representation, and the second image representation is output.

[0154] In some embodiments, the computer device may input coloring mode conversion information and a first image representation into a noise prediction model, and then the noise prediction model performs noise prediction processing on the current image representation based on the coloring mode conversion information and the first image representation to obtain predicted noise. Figure 8 The process of noise prediction processing is shown, which may include steps 5351 to 5354, as detailed below.

[0155] Step 5351: Obtain the time step, encode the time step, and obtain the time step representation;

[0156] Step 5352: Encode the coloring pattern conversion information to obtain the coloring pattern conversion representation;

[0157] Step 5353: Perform feature extraction processing based on the coloring mode transformation representation, time step representation, current image representation, and first image representation to obtain the conditional control representation;

[0158] Step 5354: Perform noise prediction processing based on the coloring mode transformation representation, time step representation, current image representation, and conditional control representation to obtain the predicted noise.

[0159] Since the denoising process using a noise prediction model involves gradually restoring a noisy image to a realistic one, the model needs to know which stage it's at. Therefore, by acquiring time steps, the model can explicitly use this information to guide the gradual progression of noise prediction, thus completing the stepwise denoising process. Considering that stepwise denoising can be implemented iteratively, a corresponding time step is acquired in each iteration of noise prediction and denoising. To ensure the time step guides the model's gradual approach, it is sequentially decreased with each iteration. This sequential decrease ensures that the denoising level for each iteration is smooth and consistent, resulting in a more stable image representation during iterative denoising. When iterative denoising terminates, the final generated second image representation corresponds to a more realistic and natural second medical image.

[0160] As one implementation method, the noise prediction model can progressively denoise in the order of time steps T, T-1, ..., 1, 0. Specifically, at the beginning of the iterative denoising process, the computer device can obtain an initial value as the initial maximum time step. This maximum time step can be set according to the general empirical value of the noise prediction model. To conform to the calculation order of iterative denoising, the maximum time step can be the same as the number of iterations of iterative denoising, so that the time step decreases sequentially as the number of iterations decreases. Because the time step is a specific numerical value and cannot directly participate in the noise prediction processing of the noise prediction model, the computer device may need to encode the time step into a data format that can participate in the noise prediction processing, that is, a time step representation in the latent space. This time step representation is a feature vector that allows the noise prediction model to effectively understand and utilize the time step information.

[0161] Furthermore, after acquiring the time-step representation, the computer device can use a noise prediction model to perform feature extraction processing based on the staining pattern transformation representation, the time-step representation, the current image representation, and the first image representation to obtain a conditional control representation. The conditional control representation refers to the feature vector extracted by the noise prediction model based on the first image representation. This conditional control representation contains biological information related to tissue structures in the first image captured by the noise prediction model, such as cell morphology and arrangement. Consequently, when the noise prediction model uses the conditional control representation to predict noise, this biological information related to tissue structures can constrain the direction of noise removal, guiding the gradual denoising to generate the second image representation. This ensures that the second image representation possesses the biological information related to tissue structures from the first medical image, thus guaranteeing that before and after the staining pattern transformation, only the staining pattern changes in the tissue slice image, while the biological information such as tissue structures (e.g., cell morphology) remains unchanged.

[0162] Please see Figure 9 , Figure 9 A model architecture diagram of a noise prediction model 500 is shown. This noise prediction model 500 may include a cue encoder 504, a time encoder 506, a noise prediction network 580, and a conditional control network 590. The image encoder 502 encodes a first medical image 501 to obtain a first image representation 507, which is then used as input to the conditional control network 590. The cue encoder 504 encodes staining pattern conversion information 503 to obtain a staining pattern conversion representation 5030, which is used as input to the noise prediction network 580 and the conditional control network 590. The time encoder 506 encodes a time step 505 to obtain a time step representation 5050, which is used as input to the noise prediction network 580 and the conditional control network 590. The current image representation 509 is used as input to the noise prediction network 580 and the conditional control network 590. The conditional control representation 5074 output by the conditional control network 590 is used as input to the noise prediction network 580.

[0163] Specifically, the prompt encoder 504 can be any neural network capable of encoding text information, such as the BERT (Bidirectional Encoder Representation from Transformers) network. The temporal encoder 506 can be any neural network capable of encoding digital information, such as a Multilayer Perceptron (MLP). The temporal encoder 506 can also be a calculator using an ensemble coding algorithm, such as a sine / cosine position coding algorithm. The noise prediction network 580 can be any generative neural network capable of generating data through progressive denoising, such as U-Net or Swin-Transformer; no specific limitations are imposed here. The conditional control network 590 can be a sub-network within the noise prediction network 580 used for feature extraction.

[0164] In one implementation, the computer device can input the shading mode conversion representation 5030, the time step representation 5050, the current image representation 509, and the first image representation 507 into the conditional control network 590 for feature extraction processing to obtain the conditional control representation 5074. Further, the computer device can input the shading mode conversion representation 5030, the time step representation 5050, the current image representation 509, and the conditional control representation 5074 into the noise prediction network 580 for noise prediction processing to obtain the predicted noise 589.

[0165] In text-to-image (TPE) scenarios where images are generated from given text, textual prompts alone are insufficient to accurately convey the finer-grained image information, such as complex layouts, shapes, and textures. The technical field of tissue slice images demands extremely high accuracy in biological information. Therefore, it is crucial to ensure that the biological characteristics of the tissue structure (e.g., cells) presented in the tissue slice image remain unchanged before and after staining pattern conversion. To address this, in the denoising process of generating a second image representation using a noise prediction network, this application not only teaches the noise prediction network how the staining pattern is converted based on the staining pattern conversion information, but also uses the conditional control representation generated by the conditional control network based on the first image representation as the conditional input to the noise prediction network.

[0166] In some embodiments, the computer device may input the chromatic mode transformation representation, the time step representation, the current image representation, and the first image representation into a conditional control network for feature extraction processing to obtain a conditional control representation. Exemplarily, the computer device may perform a first convolution processing on the first image representation to obtain a convolutional image representation, and perform an addition calculation on the current image representation and the convolutional image representation to obtain a first intermediate representation. Further, the computer device may perform feature extraction processing on the chromatic mode transformation representation, the time step representation, and the first intermediate representation to obtain a second intermediate representation, and perform a second convolution processing on the second intermediate representation to obtain a conditional control representation.

[0167] In this process, addition is used to ensure that the generated conditional control representation does not deviate from the noise prediction process. The first convolutional process compresses or transforms the image information of the input first image representation. This is done by performing a linear transformation along the channel dimension of the input first image representation without changing its spatial resolution, enabling more efficient extraction of conditional information-related features. This first convolutional process can be performed with a zero-convolutional layer. The second convolutional process integrates the conditional information-related features extracted from the first image representation, performing final compression and shaping to ensure they are used in a suitable data format for noise prediction. The second convolutional process can be performed by one or more zero-convolutional layers, the number of which can be determined by the network structure used in the feature extraction process.

[0168] Figure 10 A network architecture diagram of a conditional control network is shown. Figure 10 As shown, the conditional control network 590 may include a first zero-convolutional layer 591, a feature extraction sub-network 592, and a second zero-convolutional layer 593. The first zero-convolutional layer 591 and the second zero-convolutional layer 593 refer to convolutional layers where the weights and biases are set to zero in the initial training phase. This means that in the initial training phase, the first zero-convolutional layer 591 and the second zero-convolutional layer 593 will not have any impact on the output of the noise prediction network 580, because the outputs of the first zero-convolutional layer 591 and the second zero-convolutional layer 593 are always zero initially. As training progresses, the weights of the first zero-convolutional layer 591 and the second zero-convolutional layer 593 can be gradually learned and updated, thereby gradually introducing conditional control representations into the noise prediction network 580, enabling the noise prediction network 580, as the backbone network, to maintain stable noise prediction performance.

[0169] In one implementation, the computer device can input the first image representation 507 into the first zero-convolutional layer 591 for first convolution processing to obtain a convolutional image representation 5070, and perform addition calculation on the current image representation 509 and the convolutional image representation 5070 to obtain a first intermediate representation 5071. Further, the computer device can input the coloring mode conversion representation 5030, the time step representation 5050, and the first intermediate representation 5071 into the feature extraction sub-network 592 for feature extraction processing to obtain a second intermediate representation 5073. Further, the computer device can input the second intermediate representation 5073 into the second zero-convolutional layer 593 for second convolution processing to obtain a conditional control representation 5074. Simultaneously, the computer device can input the coloring mode conversion representation 5030, the time step representation 5050, and the current image representation 509 into the noise prediction network 580 for noise prediction, and perform addition calculation on the prediction result with the conditional control representation 5074 to obtain predicted noise 589.

[0170] Thus, by inputting the conditional control representation generated from the first image representation of the first medical image into the noise prediction network, the conditional control representation can guide the noise prediction network to predict noise as conditional control information. In turn, when the noise is used to generate the second image representation, the second image representation can accurately maintain the same biological characteristics as the first medical image, with only the staining pattern being converted.

[0171] In this embodiment, the feature extraction subnetwork in the conditional control network is the network part of the noise prediction network used for feature extraction. Thus, a feature extraction subnetwork with the same feature extraction performance as the noise prediction network allows the noise prediction network to learn how to predict noise that meets the requirements for generating the second image representation based on the conditional control representation input to the conditional control network without compromising its original prediction capabilities. For example, the network architecture of the noise prediction network can be a Stable Diffusion (SD) architecture. The SD structure includes an SD Encoder, an SD Middle, and an SD Decoder. Since the network part used for feature extraction in the SD is the encoder (SD Encoder) and the intermediate feature extractor (SD Middle), the feature extraction subnetwork in the conditional control network can correspondingly be configured with an encoder (SD Encoder) and an intermediate feature extractor (SD Middle).

[0172] For example, Figure 11 A schematic diagram illustrating the generation process of a second intermediate representation is shown. For example... Figure 11As shown, the feature extraction sub-network 592 may include an encoder 5921 and an intermediate feature extractor 5922. The encoder 5921 and the intermediate feature extractor 5922 may refer to feature extraction blocks. Each feature extraction block may include multiple neural networks, each neural network includes multiple neural layers, and each neural layer is a downsampling layer used for feature extraction. For example, the encoder 5921 may be an SD Encoder, and the intermediate feature extractor 5922 may be an SD Middle.

[0173] In one implementation, the computer device can perform feature extraction processing on the chromatic pattern conversion representation, the time step representation, and the first intermediate representation to obtain an intermediate feature representation; and perform feature extraction processing on the chromatic pattern conversion representation, the time step representation, and the intermediate feature representation to obtain a second intermediate representation. For example, the computer device can input the chromatic pattern conversion representation 5050, the time step representation 5030, and the first intermediate representation 5071 into the encoder 5921 for feature extraction processing to obtain an intermediate feature representation 5072. Further, the computer device can input the chromatic pattern conversion representation 5050, the time step representation 5030, and the intermediate feature representation 5072 into the intermediate feature extractor 5922 for feature extraction processing to obtain a second intermediate representation 5073.

[0174] Considering that the features of tissue slice images are typically more subtle than those in macroscopic visual tasks such as face recognition and image classification, it is necessary to capture more microscopic tissue structures at the pixel level, such as cell morphology and nucleoplasmic distribution. This is because the extraction of biological features from tissue slice images, especially those related to the diversity and heterogeneity of pathological states, often exhibits irregular and discontinuous changes, such as in blood vessels and cytoplasm. Therefore, this application proposes an adaptive feature extraction method based on an attention mechanism at different scales. Specifically, since the staining pattern conversion information in this application contains characteristic descriptions of the staining pattern, including the stained tissue structure information corresponding to the staining pattern affinity, such as "hematoxylin stains the cell nucleus red, eosin stains the cytoplasm blue / purple, and other tissue structures, making the cell nucleus appear purplish-red, and the cytoplasm and other structures appear pink to red."

[0175] Motivated by this, this application proposes to introduce an adaptive tissue region attention mechanism based on staining pattern transformation information. This attention mechanism can fuse staining pattern transformation information with the image information of the first medical image, and further guide the feature extraction subnetwork of the conditional control network to focus on feature extraction of specific tissue regions in the tissue slice image, thereby increasing attention to key tissue structure features in the first medical image and making the second image representation have more accurate key tissue structure features of the first medical image.

[0176] As another implementation, the computer device can perform representation fusion processing on the coloring mode transformation representation and the first intermediate representation to obtain a first prompt image fusion representation; perform feature extraction processing on the time step representation and the first prompt image fusion representation to obtain a first prompt image feature representation; calculate a first image region attention weight based on the first prompt image feature representation; and perform representation weighting processing on the first image region attention weight and the first prompt image fusion representation to obtain an intermediate feature representation.

[0177] Furthermore, the computer device can perform representation fusion processing on the coloring pattern transformation representation and the intermediate feature representation to obtain a second prompt image fusion representation; perform feature extraction processing based on the time step representation and the second prompt image fusion representation to obtain a second prompt image feature representation; calculate the second image region attention weight based on the second prompt image feature representation; and perform representation weighting processing based on the second image region attention weight and the second prompt image fusion representation to obtain a second intermediate representation.

[0178] For example, Figure 12 The diagram illustrates another process for generating a second intermediate representation. The computer device can perform the following operations: perform representation fusion processing on the chromatic mode transformation representation 5050 and the first intermediate representation 5071 to obtain a first prompt image fusion representation 5075. Input the time step representation 5030 and the first prompt image fusion representation 5075 into the encoder 5921 for feature extraction processing to obtain a first prompt image feature representation. Calculate the first image region attention weight based on the first prompt image feature representation. Perform representation weighting processing based on the first image region attention weight and the first prompt image fusion representation to obtain an intermediate feature representation 5072. Further, perform representation fusion processing on the chromatic mode transformation representation 5050 and the intermediate feature representation 5072 to obtain a second prompt image fusion representation 5076. The time step representation 5030 and the fusion representation of the second prompt image 5076 are input into the intermediate feature extractor 5922 for feature extraction processing to obtain the feature representation of the second prompt image. The attention weight of the second image region is calculated based on the feature representation of the second prompt image. The representation weight is then weighted based on the attention weight of the second image region and the fusion representation of the second prompt image to obtain the second intermediate representation 5073.

[0179] Taking the calculation of the attention weight of the first image region as an example, the coloring mode transformation representation and the first intermediate representation are fused to obtain the fused representation of the first cue image. The specific calculation formula is as follows:

[0180] f′ i =F i ·F text

[0181] Among them, F textThis refers to the coloring pattern transformation representation, since the first intermediate representation F mid In reality, it's a feature map, which is divided into n image regions. Each image region can be represented by its corresponding image region, F. i Let represent the image region corresponding to the i-th image region. F′ i This refers to the image region representation F i The corresponding prompt image fusion representation, the first prompt image fusion representation

[0182] Furthermore, the image region attention weights are calculated based on the fusion representation of the cue images corresponding to each image region. The specific calculation formula is as follows:

[0183] α i =softmax(H(F′) i ))

[0184] Where, α i H(·) refers to the image region attention weight corresponding to the i-th image region. H(·) refers to the encoder, such as SDEncoder. By calculating the image region attention weight, the feature extraction subnetwork can allocate attention to special stained regions (such as cytoplasm) indicated by the staining pattern transformation representation.

[0185] Furthermore, based on the image region attention weights and the fused representation of the cue image, a representation weighting process is performed to obtain the intermediate feature representation. The specific calculation formula is as follows:

[0186]

[0187] Here, F refers to the intermediate feature representation. By combining the image region attention weights calculated from the staining pattern transformation representation and the first intermediate representation, the feature extraction subnetwork can focus on a specific region on the tissue slice image for feature extraction based on the image region attention weights. This specific region is the specific tissue structure corresponding to the staining pattern captured from the staining pattern transformation representation. Thus, based on this specific region, a conditional control representation with more accurate biological characteristics of the first medical image can be generated. Furthermore, this conditional control representation is used to guide the noise prediction network in the process of predicting noise, thereby enabling the predicted noise to more accurately maintain the same biological characteristics as the first medical image when used to generate the second image representation, with only the staining pattern transformation occurring.

[0188] As one implementation, when the feature extraction subnetwork of the conditional control network includes an encoder (e.g., SDEncoder) and an intermediate feature extractor (e.g., SD Middle), corresponding zero-convolution layers can be set for the encoder and the intermediate feature extractor respectively for convolution processing. For example, a computer device can perform a second convolution processing on the intermediate feature representation (generated by the encoder of the feature extraction subnetwork) to obtain a first conditional control representation; and perform a second convolution processing on the second intermediate representation (generated by the intermediate feature extractor of the feature extraction subnetwork) to obtain a second conditional control representation.

[0189] Furthermore, the computer device can generate predicted noise based on the first conditional control representation and the second conditional control representation through a noise prediction network. Specifically, feature extraction processing is performed on the coloring mode transformation representation, the time step representation, and the current image representation to obtain a third intermediate representation; feature extraction processing is performed on the coloring mode transformation representation, the time step representation, and the third intermediate representation to obtain a fourth intermediate representation; the fourth intermediate representation and the first conditional control representation are added together to obtain a fifth intermediate representation; the coloring mode transformation representation, the time step representation, and the fifth intermediate representation are decoded to obtain a sixth intermediate representation; the sixth intermediate representation and the second conditional control representation are added together to obtain the predicted noise.

[0190] The third intermediate representation extracts low-dimensional features from the current image representation, reducing the spatial dimensionality of the data and improving the computational efficiency of the model. The fourth intermediate representation extracts high-dimensional features based on the third intermediate representation. Its role is to combine low-dimensional features to capture global information in the current image representation, enabling the network to better learn the relationships between different scales and contexts. Addition combines the extracted intermediate representations with the conditional control representation. This allows the noise prediction network of the noise prediction model to use the conditional control representation to predict noise. The biological information related to tissue structure contained in this conditional control representation can constrain the direction of noise removal, guiding the gradual denoising to generate the second image representation, ensuring that the second image representation contains the biological information related to tissue structure from the first medical image. Decoding is used to gradually restore the spatial resolution of the intermediate representation, allowing the output predicted noise to participate in the denoising calculation.

[0191] For example, Figure 13 A schematic diagram of the prediction noise generation process is shown, such as... Figure 13As shown, the conditional control network 590 includes a zero-convolutional layer 597, an encoder 5921, an intermediate feature extractor 5922, a zero-convolutional layer 598, and a zero-convolutional layer 599. The zero-convolutional layer 597 performs a first convolution on the first image representation 507 to obtain a convolutional image representation, and adds the current image representation 509 to the convolutional image representation. The zero-convolutional layer 598 performs a second convolution on the intermediate feature representation generated by the intermediate feature extractor 5922 to obtain a first conditional control representation. The zero-convolutional layer 599 performs a second convolution on the second intermediate representation generated by the encoder 5921 to obtain a second conditional control representation.

[0192] like Figure 13 As shown, the noise prediction network 580 includes an encoder 581, an intermediate feature extractor 582, and a decoder 583, which can be an SD Decoder. The encoder 581 performs feature extraction processing on the tinting mode transformation representation 5050, the time step representation 5030, and the current image representation 509 to obtain a third intermediate representation. The intermediate feature extractor 582 performs feature extraction processing on the tinting mode transformation representation 5050, the time step representation 5030, and the third intermediate representation to obtain a fourth intermediate representation. Then, the fourth intermediate representation is added to the first conditional control representation to obtain a fifth intermediate representation. The decoder 583 decodes the tinting mode transformation representation 5050, the time step representation 5030, and the fifth intermediate representation to obtain a sixth intermediate representation. Then, the sixth intermediate representation is added to the second conditional control representation to obtain the predicted noise 589.

[0193] Furthermore, after obtaining the predicted noise 589, the current image representation 509 can be denoised based on the predicted noise 589 to obtain the updated current image representation. The current denoising iteration value is then decremented according to a preset subtraction factor to determine whether the current denoising iteration value is one. If it is not one, the noise prediction continues to be performed to update the current image representation until the current denoising iteration value is one. At this point, the current image representation is determined as the second image representation.

[0194] Step 540: Decode the second image representation to obtain the second medical image.

[0195] Since the second image representation is a low-dimensional image representation in the latent space, not an image in the pixel space, it needs to be decoded to generate a second medical image in the pixel space, which has a second staining pattern. Optionally, after obtaining the second image representation, staining pattern transformation information with the same meaning but slightly different expression can be written to regenerate the second image representation. This process can be repeated multiple times, and the multiple generated second image representations can be averaged. Then, the averaged second image representation is decoded to obtain the second image, thereby enhancing the generation quality of the tissue section image.

[0196] In one implementation, a computer device can use an image decoder to decode a second image representation to obtain a second medical image. The decoding of the second image representation by the image decoder is essentially an upsampling process. The image decoder can be any neural network model capable of mapping the image representation from a low-dimensional latent space to a high-dimensional pixel space. This image decoder can be a pre-trained neural network. For example, the image decoder can be the decoder part of a variational autoencoder.

[0197] This embodiment encodes the first medical image to be stained to obtain a first image representation. This first image representation contains more detailed and clearer biological tissue information, including the morphology and arrangement of tissue cells. Combining this first image representation with the staining mode conversion information allows them to work together to guide the denoising process of the noise prediction model. The resulting second medical image with the second staining mode thus reflects the specific biological tissue details, such as the morphology and arrangement of tissue cells in the first medical image. This improves the accuracy of virtual staining using the second staining mode and overcomes the obstacle of related technologies where raw image processing cannot be accurately used for section staining mode switching.

[0198] Since the first image representation is the encoded representation of the first medical image in the feature space, and this first image representation possesses biological features such as the tissue structure of the first medical image, in the denoising process of generating a second image representation of the second staining pattern from a preset noisy image representation using a noise prediction model, using the first image representation as input to the noise prediction model can guide the noise predicted by the noise prediction model in the process of generating the second image representation. This allows the second image representation to accurately maintain the same biological features as the first medical image. Therefore, decoding the second image representation can yield a second medical image that has been converted from the first staining pattern to the second staining pattern while retaining its biological features. Furthermore, because the staining pattern conversion information serves as arbitrary textual information indicating the conversion of the staining pattern of the tissue slice image, only one noise prediction model needs to be trained to input different staining pattern conversion information to achieve the conversion of multiple staining patterns. Compared to methods that require multiple models to achieve the conversion of multiple staining patterns, this effectively reduces the consumption of computational and storage resources.

[0199] Considering that generative noise prediction models are subject to some randomness and uncertainty when providing input natural language prompts, resulting in unstable generation results, in this embodiment, when performing staining pattern conversion on the first medical image using staining pattern conversion information, multiple second image representations can be generated using staining pattern conversion information with the same meaning but different expressions, so that the final generated second medical image can meet the staining quality requirements. This image processing method can be executed by a computer device (e.g., a server, edge computing device, or other terminal device with certain processing capabilities) after step 540 of the above embodiment. The computer device has at least storage, computing, and communication functions. Figure 14 A flowchart illustrating the image processing method provided in this embodiment is shown. Figure 14 As shown, the image processing method may specifically include:

[0200] Step 550: Determine the second medical image as the current medical image, perform quality detection processing on the current medical image, and obtain the staining quality score corresponding to the current medical image.

[0201] By generating a second medical image through multiple iterations, the final generated second medical image can meet the staining quality requirements. The current tissue section image refers to the image whose staining quality needs to be tested. Quality detection processing is to ensure the accuracy and usability of the virtual staining effect, especially before and after the conversion of the staining mode in this application, it is even more important that the biological characteristics of the tissue structure (e.g., cells) presented in the tissue section image remain unchanged. Therefore, a staining quality score can be calculated from multiple quantitative indicators.

[0202] In some embodiments, the staining quality score of the current tissue section image can be determined by a weighted summation of at least the image's color fidelity, contrast, and tissue structure consistency. Color fidelity: The colors of the virtual stained image should be close to those of traditional stained images, especially in characteristic areas of pathological interest. This can be quantified as color similarity to traditional stained images (e.g., through color difference or color similarity). Contrast: The image contrast should be sufficiently high to make the tissue structure clearly visible. Tissue structure consistency reflects the degree of consistency between the second medical image and the first medical image in terms of tissue structure; therefore, the weight corresponding to tissue structure consistency is greater than the weight corresponding to other quantitative indicators.

[0203] As one implementation method, the computer device can acquire a traditional stained section image under the staining mode corresponding to the current tissue section image, and calculate the color similarity between the current tissue section image and the traditional stained section image as color fidelity. Simultaneously, it can calculate the contrast corresponding to the current tissue section image using methods for calculating image contrast equivalents such as global contrast, local contrast, or local standard deviation. Further, the computer device can extract image texture features from the current tissue section image and the first medical image using methods such as gray-level co-occurrence matrix statistical features and local binary patterns to calculate image texture similarity and obtain tissue structure consistency. Further, the computer device can perform a weighted sum based on color fidelity, contrast, and tissue structure consistency to obtain the staining quality score corresponding to the current tissue section image.

[0204] Step 560: When the staining quality score is less than the staining quality threshold, obtain synonymous staining pattern conversion information that has a semantic similarity greater than a preset similarity threshold with the staining pattern conversion information, and update the staining pattern conversion information according to the synonymous staining pattern conversion information.

[0205] The staining quality threshold is the minimum staining quality score required to determine whether the current tissue section image meets the required staining quality standard. If the staining quality score is less than the staining quality threshold, the current tissue section image does not meet the minimum staining quality score, and the staining quality is substandard. Therefore, the staining mode needs to be changed again.

[0206] As one implementation method, a large language model can be used to generate multiple similar coloring pattern transformation pieces that are semantically similar to the coloring pattern transformation piece. The semantic similarity between each similar coloring pattern transformation piece and the coloring pattern transformation piece is then calculated. Furthermore, similar coloring pattern transformation pieces with a semantic similarity greater than a preset similarity threshold are designated as synonymous coloring pattern transformation pieces, and the coloring pattern transformation piece is updated to this synonymous piece. For example, the semantic similarity calculation can utilize similarity calculation algorithms such as cosine similarity to perform similarity calculations on the transformation information representations of the two transformation pieces to be calculated.

[0207] Step 570: Return the staining pattern conversion information and the first image representation to the noise prediction model to regenerate a new second medical image until the staining quality score is not less than the staining quality threshold. The current medical image with a staining quality score not less than the staining quality threshold is determined as the second medical image.

[0208] As one implementation, after updating the staining pattern conversion information, the encoding process of the first medical image can be returned to obtain a first image representation, in order to regenerate a new second medical image. For example, as shown... Figure 15 The flowchart shown illustrates the staining pattern conversion process, which a computer device can execute as follows: Figure 15 The steps are as follows: Step 001: Obtain the first medical image and staining pattern conversion information; Step 002: Encode the first medical image to obtain a first image representation; Step 003: Input the staining pattern conversion information and the first image representation into a noise prediction model to generate a second image representation, and decode the second image representation to obtain a second medical image; Step 004: Determine that the second medical image is the current tissue slice image, perform quality detection processing on the current tissue slice image, and obtain the staining quality score corresponding to the current tissue slice image; Step 005: Determine that the staining quality score is less than the staining quality threshold; if the staining quality score is less than the staining quality threshold, then execute step 006: Update the staining pattern conversion information; otherwise, execute step 007: Output the current tissue slice image as the second medical image. The specific execution content of steps 001 to 007 can be referred to steps 510 to 570 in the above embodiment, and will not be repeated here.

[0209] Thus, after converting the first medical image of the second staining mode into a second medical image of the second staining mode, staining quality can be detected in the second medical image. If the second medical image does not meet the staining quality requirements, multiple representations of the second image can be generated using staining mode conversion information with the same meaning but different expressions. This ensures that the final generated second medical image meets the staining quality requirements, thereby improving the accuracy of tissue section image conversion under different staining modes and consequently improving the precision of virtual staining. Furthermore, because the staining mode conversion information is arbitrary textual information indicating the staining type conversion of the tissue section image, multiple staining mode conversions can be achieved by inputting different staining mode conversion information into a single noise prediction model. Compared to methods that require multiple dedicated staining models to implement their respective staining mode conversions, this effectively reduces the computational and storage resource consumption for staining mode conversion of tissue section images.

[0210] According to one embodiment of this application, a method for training a noise prediction model is provided. This method can be executed by a computer device (e.g., a server, edge computing device, or other terminal device with certain processing capabilities) before step 530 of the above embodiment. This computer device must at least have storage, computing, and communication functions. Figure 16 A flowchart illustrating the training method of the noise prediction model provided in this embodiment is shown. Figure 16 As shown, the method may specifically include:

[0211] Step 610: Obtain the first sample medical image, the second sample medical image, and sample staining pattern conversion information.

[0212] In this embodiment, the first and second sample medical images refer to tissue section images. The sample staining mode conversion information includes text information indicating that the first sample medical image with a third staining mode is converted to the second sample medical image with a fourth staining mode. Furthermore, in addition to the text information instructing the conversion of the first sample medical image with a third staining mode to the second sample medical image with a fourth staining mode, the sample staining mode conversion information may also include characteristic description information of the third and fourth staining modes. For specific principles and implementation methods, please refer to step 510 in the above embodiments, which will not be repeated here. The third staining mode refers to the original staining mode to be converted during model training; correspondingly, the first sample medical image is the tissue section image of this original staining mode. The fourth staining mode refers to the target staining mode during model training, which is a different staining mode from the third staining mode; correspondingly, the second sample medical image is the tissue section image of this target staining mode.

[0213] In one implementation, a computer device can randomly retrieve a pair of tissue slide images with different staining patterns from a database storing training data, and randomly designate one of the tissue slide images as a first sample medical image x and the other as a second sample medical image y. Further, based on the third staining pattern corresponding to the first sample medical image x and the fourth staining pattern corresponding to the second sample medical image y, sample staining pattern conversion information c that can be used to indicate a conversion from the third staining pattern to the fourth staining pattern is retrieved from a prompt information database.

[0214] By randomly selecting the first and second sample medical images, along with the corresponding sample staining pattern conversion information, the noise prediction model is trained to switch staining patterns. This allows the trained noise prediction model to switch between multiple staining patterns based on different input staining pattern conversion information. Compared to methods that require multiple models to achieve multiple staining pattern conversions, this method effectively reduces the consumption of computing and storage resources, both during the model inference and training phases.

[0215] Step 620: Encode the first sample medical image to obtain the first sample image representation, and encode the second sample medical image to obtain the second sample image representation.

[0216] In one implementation, a computer device can encode a first sample medical image x using an image encoder to obtain a first sample image representation z. x The second sample medical image y is encoded using an image encoder to obtain the second sample image representation z. y .

[0217] Step 630: Add first noise to the second sample image representation to obtain a noisy sample image representation.

[0218] In the embodiments of this application, the training process of the noise prediction model includes a forward process and a backward process. The forward process is the process of gradually adding noise to the image representation until the image representation becomes completely noisy, while the backward process is the denoising process, which requires the noise prediction model to be able to accurately predict the noise added in the forward process, that is, the first noise.

[0219] In one implementation, a computer device can randomly sample a first noise based on a probability distribution and add the first noise to a second sample image representation, thereby enabling the noise prediction model to perform noise prediction during model training. Figure 17 The diagram illustrates the process of generating a noisy sample image representation, which may include steps 631 to 634.

[0220] Step 631: Randomly sample a two-dimensional standard Gaussian noise as the first noise;

[0221] For example, a computer device can randomly sample a two-dimensional standard Gaussian distribution ∈ ~N(0,I), that is, Gaussian noise as the first noise ∈.

[0222] Step 632: Calculate the first combination coefficient and the first product value of the second sample image representation;

[0223] The first combination coefficient is the parameter α. t square root Parameter α t It is related to the time step t and is usually determined by a decreasing scheduling function. As the time step t increases, the parameter α... t It will gradually decrease. Computer equipment can measure the first combination coefficient. The second sample image represents z y Perform multiplication to obtain the first product value.

[0224] Step 633: Calculate the second product of the second combination coefficient and the first noise;

[0225] The second combination coefficient is 1 and the parameter α. t square root of the difference Computer equipment can combine the second coefficients The second product is obtained by multiplying the first noise ∈.

[0226] Step 634: Perform addition calculation on the first product value and the second product value to obtain the noisy sample image representation.

[0227] For example, the first combination coefficient and the second combination coefficient are determined by the time step of random sampling, and the computer device can obtain the noisy sample image representation by adding the first product value and the second product value. That is, the image representation after adding noise at time step t.

[0228] Step 640: Input the sample staining mode conversion information, the first sample image representation, and the noisy sample image representation into the noise prediction model to predict the noise and obtain the second noise.

[0229] In one implementation, the noise prediction model includes a cue encoder, a time encoder, a noise prediction network, and a conditional control network. The conditional control network is used to input sample conditional control representations into the noise prediction network to control the noise prediction network to perform noise predictions that conform to the sample conditional control representations. Specifically, the computer device can encode sample staining pattern transformation information into a sample staining pattern transformation representation through the cue encoder. Further, the computer device can acquire a time step and encode the time step into a time step representation through the time encoder. The sample staining pattern transformation representation, the time step representation, the noisy sample image representation, and the first sample image representation are then input into the conditional control network for feature extraction to obtain the sample conditional control representation. Further, the computer device can input the sample staining pattern transformation representation, the time step representation, the noisy sample image representation, and the sample conditional control representation into the noise prediction network for noise prediction to obtain the second noise.

[0230] Figure 18 A schematic diagram of the prediction process for the second noise is shown. (For example...) Figure 18 As shown, the noise prediction model 600 includes a cue encoder 602, a temporal encoder 603, a noise prediction network 670, and a conditional control network 680. The noise prediction network 670 is a U-Net structure network, including an encoder 671, an intermediate feature extractor 672, and a decoder 673. The conditional control network 680 includes a zero-convolutional layer 681, an encoder 682, an intermediate feature extractor 683, a zero-convolutional layer 684, and a zero-convolutional layer 685. The encoder 682 and intermediate feature extractor 683 of the conditional control network 680 have the same structure as the encoder 671 and intermediate feature extractor 672 of the noise prediction network 670. Specifically, encoders 671 and 682 can be SD Encoders, intermediate feature extractors 672 and 683 can be SD Middlers, and the decoder 673 can be an SD Decoder.

[0231] For example, the sample staining mode transformation information can be encoded into a sample staining mode transformation representation by the prompt encoder 602, and the time step can be encoded into a time step representation by the time encoder 603. The sample staining mode transformation representation, the time step representation, the noisy sample image representation 604, and the first sample image representation 601 are input into the conditional control network 680 for feature extraction to obtain the first sample conditional control representation and the second sample conditional control representation.

[0232] Specifically, the zero-convolutional layer 681 performs a first convolution on the first sample image representation 601 to obtain a sample convolutional image representation, and performs an addition calculation on the noisy sample image representation 604 and the sample convolutional image representation to obtain a first sample intermediate representation. The encoder 671 performs feature extraction processing on the sample staining mode transformation representation, the time step representation, and the first sample intermediate representation to obtain a sample intermediate feature representation. The intermediate feature extractor 672 performs feature extraction processing on the sample staining mode transformation representation, the time step representation, and the sample intermediate feature representation to obtain a second sample intermediate representation. Furthermore, the zero-convolutional layer 684 performs a second convolution on the sample intermediate feature representation to obtain a first sample conditional control representation, and the zero-convolutional layer 685 performs a second convolution on the second sample intermediate representation to obtain a second sample conditional control representation.

[0233] Furthermore, the noise prediction network 670 inputs the sample staining mode transformation representation, time step representation, noisy sample image representation 604, first sample condition control representation, and second sample condition control representation into the noise prediction network to perform noise prediction, thereby obtaining the second noise 678.

[0234] Specifically, encoder 671 performs feature extraction processing on the sample staining mode transformation representation, time step representation, and noisy sample image representation 604 to obtain a third intermediate sample representation. Intermediate feature extractor 672 performs feature extraction processing on the sample staining mode transformation representation, time step representation, and third intermediate sample representation to obtain a fourth intermediate sample representation. Then, it performs addition calculation on the fourth intermediate sample representation and the first sample conditional control representation to obtain a fifth intermediate sample representation. Decoder 673 decodes the sample staining mode transformation representation, time step representation, and fifth intermediate sample representation to obtain a sixth intermediate sample representation. Further, it performs addition calculation on the sixth intermediate sample representation and the second sample conditional control representation to obtain second noise 678.

[0235] It should be noted that, Figure 18 The diagram illustrates the prediction process for the second noise, which is merely one method for predicting the second noise provided in this embodiment. Figure 18 The model architecture of the noise prediction model 600 described is only for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. For example, Figure 18 The way the conditional control network 680 generates the first sample conditional control representation and the second sample conditional control representation can be replaced by a method based on... Figure 12 The generation method of the conditional control representation of the adaptive organizational region attention mechanism described herein is not limited here.

[0236] Step 650: Calculate the target loss based on the error between the first noise and the second noise.

[0237] The target loss refers to the loss function used to train the noise prediction model of this application. By calculating the target loss, the noise prediction model can learn to predict noise step by step during the training process, so that the predicted noise (i.e., the second noise) becomes more and more similar to the real noise.

[0238] As one implementation method, the mean square error is calculated based on the first noise and the second noise, and this mean square error is used as the target loss. The calculation formula is as follows:

[0239] L = E ε(x)ε(y),∈～N(0,1) [‖∈-∈ θ (z t ,z x ,c,t)‖ 2 ]

[0240] Where L is the target loss, ε(·) is the image encoder used to encode the first and second sample medical images, ∈~N(0,1) is a standard Gaussian distribution, ∈ is the first noise, and v θ The noise prediction model transforms information c based on sample staining pattern, time step t, and the first sample image representation z. x And noisy sample image representation z t The predicted second noise.

[0241] Step 660: Iteratively train the noise prediction model based on the target loss until the training result conditions are met, and obtain the trained noise prediction model.

[0242] Since the noise prediction network is a pre-trained network, it has already been trained with a large amount of data and possesses the ability to predict noise. Fixing its weight parameters can prevent the loss of this ability during training. Furthermore, the cue encoder and time encoder are general-purpose, pre-trained neural networks and do not need to participate in training. Therefore, only the conditional control network needs to be trained.

[0243] In one implementation, the computer device can fix the first weight parameters corresponding to the prompt encoder, the time encoder, and the noise prediction network. Further, iteratively train the second weight parameters corresponding to the conditional control network according to the target loss. That is, iteratively execute the above steps 610 to 650 to update the second weight parameters until the training termination condition is met, and obtain the trained noise prediction model.

[0244] As another implementation method, if the training effect of the noise prediction model is not ideal during the training process, for example, the target loss changes abnormally, including slow decrease, oscillation, or failure to converge, or the image quality of the tissue slice image generated after denoising based on the second noise does not meet the staining quality requirements, then it is possible not to fix the first weight parameters corresponding to the cue encoder, the time encoder, and the noise prediction network, and to directly iterate the training of the first weight parameters corresponding to the cue encoder, the time encoder, and the noise prediction network and the second weight parameters corresponding to the conditional control network during the model training process.

[0245] The training termination conditions can include: the total loss value of the target loss is less than a preset value, the total loss value of the target loss is within a threshold range, or the number of training iterations reaches a preset number. For example, in this application, an iteration threshold can be set for iterative training of the noise prediction model, which can be determined experimentally based on empirical values. Optionally, an optimizer can be used to optimize the target loss, and the learning rate, batch size, and epoch can be set based on experimental experience.

[0246] The technical solution in this application can be implemented based on the PyTorch deep learning framework. The optimizer can be AdamW, and the learning rate is set to 10. -4 The model is trained until convergence. On the hardware side, computer devices such as NVIDIA A100, V100, Ascend 910B, or other similar or higher-performance graphics processing units (GPUs) can meet the requirements.

[0247] This application uses VQ-VAE to construct the latent space and map the samples generated in the latent space back to the pixel space of the original image. The downsampling rate of the latent space representation relative to the original image is a crucial consideration. If the downsampling rate is too low, it will not significantly improve computational resource utilization and training efficiency; conversely, it will lead to severe loss of tissue structural details in tissue slice images, failing to guarantee the conversion quality of staining patterns. In practice, a suitable balance can be achieved by setting the downsampling rate between 4 and 16.

[0248] Since the first sample image representation is the encoded representation of the first sample medical image in the feature space, and this first sample image representation possesses biological features such as the tissue structure of the first sample medical image, in the denoising process of generating the second sample image representation of the fourth staining mode from the noisy sample image representation using the noise prediction model, using the first sample image representation as input to the noise prediction model can guide the noise predicted by the noise prediction model in the process of generating the second sample image representation. This allows the second sample image representation to accurately maintain the same biological features as the first sample medical image. Therefore, decoding the second sample image representation can obtain a second sample medical image that is converted from the third staining mode to the fourth staining mode while retaining the same biological features. Thus, the accuracy of the staining model conversion is improved, thereby improving the precision of virtual staining. Simultaneously, by randomly selecting the first and second sample medical images, along with the corresponding sample staining mode conversion information, to train the noise prediction model for staining mode switching, a single trained noise prediction model can achieve conversions of multiple staining modes based on different input staining mode conversion information. Compared to methods that require multiple models to achieve multiple staining mode conversions, this effectively reduces the consumption of computational and storage resources.

[0249] Please see Figure 19 This diagram illustrates a structural block diagram of an image processing apparatus 700 provided in an embodiment of this application. The image processing apparatus 700 may include:

[0250] The information acquisition module 710 is used to acquire a first medical image and staining pattern conversion information, wherein the first medical image has a first staining pattern and the staining pattern conversion information is used to indicate the conversion of the first staining pattern to a second staining pattern conversion information.

[0251] The encoding processing module 720 is used to encode the first medical image to obtain a first image representation;

[0252] The generation module 730 is used to input the coloring mode conversion information and the first image representation into the noise prediction model, so as to perform noise prediction processing on the preset noise image representation according to the coloring mode conversion information and the first image representation to obtain predicted noise, and perform denoising processing on the preset noise image representation according to the predicted noise to obtain the second image representation coloring mode conversion information.

[0253] The decoding processing module 740 is used to decode the second image representation to obtain a second medical image, the second medical image having the second staining pattern;

[0254] The noise prediction model is trained based on a target loss, which is determined based on a first noise and a second noise. The second noise is the noise predicted by the noise prediction model for the noisy sample image representation based on the first sample image representation and sample staining pattern conversion information of the first sample medical image. The first noise is the noise added to the second sample image representation of the second sample medical image. The staining pattern conversion information of the noisy sample image representation is obtained after adding the first noise to the second sample image representation.

[0255] In some embodiments, the representation generation module 730 may include an information acquisition submodule, a noise coding submodule, a parameter determination submodule, an iterative decrementing submodule, and a representation generation submodule.

[0256] The information acquisition submodule is used to acquire the preset noisy image and the denoising processing iteration threshold;

[0257] The noise encoding submodule is used to encode the preset noise image to obtain a preset noise image representation;

[0258] The parameter determination submodule is used to determine the preset noise image representation as the current image representation and the denoising processing iteration threshold as the current denoising iteration value.

[0259] The iterative decrease submodule is used to decrease the current denoising iteration value according to a preset subtrahend;

[0260] The generation submodule is used to input the coloring mode conversion information and the first image representation into the noise prediction model to perform noise prediction processing on the current image representation to obtain predicted noise, and to perform denoising processing on the current image representation based on the predicted noise to obtain an updated current image representation. Then, it returns the current denoising iteration value by decreasing it according to a preset subtraction until the current denoising iteration value is one, and determines the current image representation when the current denoising iteration value is one as the second image representation coloring mode conversion information.

[0261] In some embodiments, the generation submodule may include a time acquisition unit, an information encoding unit, a feature extraction unit, and a noise prediction unit.

[0262] A time acquisition unit is used to acquire time steps and encode the time steps to obtain a time step representation.

[0263] An information encoding unit is used to encode the coloring mode conversion information to obtain coloring mode conversion representation coloring mode conversion information;

[0264] The feature extraction unit is used to perform feature extraction processing based on the staining mode conversion representation, the time step representation, the current image representation, and the first image representation to obtain a conditional control representation;

[0265] The noise prediction unit is used to perform noise prediction processing based on the coloring mode conversion representation, the time step representation, the current image representation, and the conditional control representation to obtain predicted noise.

[0266] In some embodiments, the feature extraction unit may include a first convolution subunit, an addition subunit, a feature extraction subunit, and a second convolution subunit.

[0267] The first convolutional subunit is used to perform a first convolution process on the first image representation to obtain a convolutional image representation;

[0268] An addition calculation subunit is used to perform an addition calculation on the current image representation and the convolutional image representation to obtain a first intermediate representation;

[0269] The feature extraction subunit is used to perform feature extraction processing on the staining pattern transformation representation, the time step representation, and the first intermediate representation to obtain the second intermediate representation;

[0270] The second convolution subunit is used to perform a second convolution process on the second intermediate representation to obtain the conditional control representation.

[0271] In some embodiments, the feature extraction subunit may be specifically used to: perform feature extraction processing on the coloring pattern transformation representation, the time step representation, and the first intermediate representation to obtain an intermediate feature representation; and perform feature extraction processing on the coloring pattern transformation representation, the time step representation, and the intermediate feature representation to obtain a second intermediate representation.

[0272] In some embodiments, the feature extraction subunit may be specifically used for: performing representation fusion processing on the coloring mode transformation representation and the first intermediate representation to obtain a first prompt image fusion representation; performing feature extraction processing on the time step representation and the first prompt image fusion representation to obtain a first prompt image feature representation; calculating a first image region attention weight based on the first prompt image feature representation; performing representation weighting processing on the first image region attention weight and the first prompt image fusion representation to obtain an intermediate feature representation; performing representation fusion processing on the coloring mode transformation representation and the intermediate feature representation to obtain a second prompt image fusion representation; performing feature extraction processing on the time step representation and the second prompt image fusion representation to obtain a second prompt image feature representation; calculating a second image region attention weight based on the second prompt image feature representation; performing representation weighting processing on the second image region attention weight and the second prompt image fusion representation to obtain a second intermediate representation.

[0273] In some embodiments, the conditional control representation includes a first conditional control representation and a second conditional control representation; the second convolutional subunit may be specifically used to: perform a second convolutional process on the intermediate feature representation to obtain the first conditional control representation; and perform a second convolutional process on the second intermediate representation to obtain the second conditional control representation.

[0274] In some embodiments, the noise prediction unit may be specifically configured to: perform feature extraction processing on the coloring mode conversion representation, the time step representation, and the current image representation to obtain a third intermediate representation; perform feature extraction processing on the coloring mode conversion representation, the time step representation, and the third intermediate representation to obtain a fourth intermediate representation; perform addition calculation on the fourth intermediate representation and the first conditional control representation to obtain a fifth intermediate representation; perform decoding processing on the coloring mode conversion representation, the time step representation, and the fifth intermediate representation to obtain a sixth intermediate representation; and perform addition calculation on the sixth intermediate representation and the second conditional control representation to obtain predicted noise.

[0275] In some embodiments, the image processing apparatus 700 may further include a staining quality scoring module, a synonym suggestion determination module, and a slice image update module.

[0276] The staining quality scoring module is used to determine that the second medical image is the current medical image, perform quality detection processing on the current medical image, and obtain the staining quality score corresponding to the current medical image.

[0277] The synonym suggestion determination module is used to obtain synonymous staining pattern conversion information that has a semantic similarity greater than a preset similarity threshold with the staining pattern conversion information when the staining quality score is less than the staining quality threshold, and update the staining pattern conversion information according to the synonymous staining pattern conversion information;

[0278] The slice image update module is used to return the staining mode conversion information and the first image representation as inputs to the noise prediction model to regenerate a new second medical image until the staining quality score is not less than the staining quality threshold. The current medical image with a staining quality score not less than the staining quality threshold is determined as the staining mode conversion information of the second medical image.

[0279] In some embodiments, the staining pattern conversion information includes text information indicating that a first medical image with a first staining pattern is converted into a second medical image with a second staining pattern, as well as feature description information of the first staining pattern and feature description information of the second staining pattern.

[0280] In some embodiments, the image processing apparatus 700 may further include a sample information acquisition module, a sample image encoding module, a first noise addition module, a second noise prediction module, a target loss determination module, and a prediction model training module.

[0281] The sample information acquisition module is used to acquire a first sample medical image, a second sample medical image, and sample staining mode conversion information. The sample staining mode conversion information includes text information about converting the first sample medical image of the third staining mode into the second sample medical image of the fourth staining mode.

[0282] The sample image encoding module is used to encode the first sample medical image to obtain a first sample image representation, and to encode the second sample medical image to obtain a second sample image representation;

[0283] The first noise addition module is used to add first noise to the second sample image representation to obtain a noisy sample image representation;

[0284] The second noise prediction module is used to input the sample staining mode conversion information, the first sample image representation and the noisy sample image representation into the noise prediction model to perform noise prediction and obtain the second noise staining mode conversion information.

[0285] The target loss determination module is used to calculate the target loss based on the error between the first noise and the second noise.

[0286] The prediction model training module is used to iteratively train the noise prediction model based on the target loss until the training result conditions are met, thereby obtaining the trained noise prediction model.

[0287] In some embodiments, the noise prediction model includes a cue encoder, a time encoder, a noise prediction network, and a conditional control network, wherein the conditional control network is used to input a sample conditional control representation to the noise prediction network to control the noise prediction network to perform noise prediction that conforms to the sample conditional control representation;

[0288] The second noise prediction module can be specifically used for:

[0289] The prompt encoder encodes the sample staining mode conversion information into a sample staining mode conversion representation;

[0290] The time step is acquired and encoded into a time step representation using the time encoder.

[0291] The sample staining mode transformation representation, the time step representation, the noisy sample image representation, and the first sample image representation are input into the conditional control network for feature extraction to obtain the sample conditional control representation.

[0292] The sample staining mode transformation representation, the time step representation, the noisy sample image representation, and the sample condition control representation are input into the noise prediction network to predict noise and obtain the second noise.

[0293] In some embodiments, the prediction model training module may be specifically used to: fix the first weight parameters corresponding to the cue encoder, the time encoder, and the noise prediction network; iteratively train the second weight parameters corresponding to the conditional control network according to the target loss to update the second weight parameters until the training termination condition is met, thereby obtaining the trained noise prediction model.

[0294] In some embodiments, the first noise addition module may be specifically used to: randomly sample a two-dimensional standard Gaussian noise as the first noise; calculate a first combination coefficient and a first product of the second sample image representation; calculate a second combination coefficient and a second product of the first noise; perform addition calculation on the first product and the second product to obtain a noisy sample image representation, wherein the first combination coefficient and the second combination coefficient are determined by the random sampling time step.

[0295] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the described devices, modules, etc., can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0296] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0297] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0298] This embodiment encodes the first medical image to be stained to obtain a first image representation. This first image representation contains more detailed and clearer biological tissue information, including the morphology and arrangement of tissue cells. Combining this first image representation with the staining mode conversion information allows them to work together to guide the denoising process of the noise prediction model. The resulting second medical image with the second staining mode thus reflects the specific biological tissue details, such as the morphology and arrangement of tissue cells in the first medical image. This improves the accuracy of virtual staining using the second staining mode and overcomes the obstacle of related technologies where raw image processing cannot be accurately used for section staining mode switching. Because the first image representation is the encoded representation of the first medical image in the feature space, and this first image representation possesses biological features such as the tissue structure of the first medical image, in the denoising process of generating a second image representation of the second staining pattern using a noise prediction model from a pre-set noise image representation, using the first image representation as input to the noise prediction model can guide the noise predicted by the noise prediction model in the process of generating the second image representation. This ensures that the second image representation accurately maintains the same biological features as the first medical image. Therefore, decoding the second image representation can yield a second medical image that has been converted from the first staining pattern to the second staining pattern while retaining the same biological features. Furthermore, because the staining pattern conversion information serves as arbitrary textual information indicating the conversion of the staining pattern of the tissue slice image, only one noise prediction model needs to be trained to input different staining pattern conversion information to achieve the conversion of multiple staining patterns. Compared to methods that require multiple models to achieve the conversion of multiple staining patterns, this effectively reduces the consumption of computational and storage resources.

[0299] like Figure 20As shown in the figure, this application embodiment also provides a computer device 800, which includes a processor 810, a memory 820, a power supply 830, and an input unit 840. The memory 820 stores a computer program, which, when called by the processor 810, can execute the various method steps provided in the above embodiments. Those skilled in the art will understand that the structure of the computer device shown in the figures does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0300] The processor 810 may include one or more processing cores. The processor 810 connects to various parts of the entire battery management system using various interfaces and lines. It executes instructions, programs, instruction sets, or program sets stored in the memory 820, calls data stored in the memory 820, performs various functions and processes data within the battery management system, and performs various functions and processes data within the computer device, thereby providing overall control of the computer device. Optionally, the processor 810 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 810 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 810 and may be implemented separately using a communication chip.

[0301] The memory 820 may include random access memory (RAM) or read-only memory (ROM). The memory 820 can be used to store instructions, programs, instruction sets, or program assemblies. The memory 820 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created during the use of the computer device (such as phonebook and audio / video data). Accordingly, the memory 820 may also include a memory controller to provide the processor 810 with access to the memory 820.

[0302] The power supply 830 can be logically connected to the processor 810 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 830 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0303] The input unit 840 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0304] Although not shown, the computer device 800 may also include an output unit, a display unit, etc., which will not be described in detail here. Specifically, in the embodiments of this application, the processor 810 of the computer device loads the executable files corresponding to the processes of one or more computer programs into the memory 820 according to the following instructions, and the processor 810 runs the data stored in the memory 820, such as telephone books and audio and video data, thereby implementing the various method steps provided in the foregoing embodiments.

[0305] like Figure 21 As shown, this application embodiment also provides a computer-readable storage medium 900, which stores a computer program 910. The computer program 910 can be invoked by the processor of a computer device to execute various method steps provided in this application embodiment.

[0306] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media include non-volatile computer-readable storage medium. The computer-readable storage medium 900 has storage space for a computer program 910 that performs any of the method steps in the above embodiments. These computer programs 910 can be read from or written to one or more computer program products. The computer program 910 can be compressed in an appropriate form.

[0307] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the various method steps provided in the above embodiments.

[0308] The terms "comprising" and "including," and any variations thereof, in the specification and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus. The terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined objective, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) may be used to implement one or more modules or units. Furthermore, each module or unit may be part of a larger module or unit that includes the functionality of that module or unit.

[0309] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0310] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0311] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0312] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0313] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. It should also be understood that the various implementation methods provided in the embodiments of this application can be arbitrarily combined to achieve different technical effects.

[0314] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first medical image and staining pattern conversion information, wherein the first medical image has a first staining pattern, and the staining pattern conversion information is used to indicate the conversion of the first staining pattern to a second staining pattern; The first medical image is encoded to obtain a first image representation; The coloring mode conversion information and the first image representation are input into the noise prediction model to perform noise prediction processing on the preset noise image representation based on the coloring mode conversion information and the first image representation to obtain predicted noise. Then, the preset noise image representation is denoised based on the predicted noise to obtain the second image representation. The second image representation is decoded to obtain a second medical image, which has the second staining pattern. The noise prediction model is trained based on a target loss, which is determined based on a first noise and a second noise. The second noise is the noise predicted by the noise prediction model for the noisy sample image representation based on the first sample image representation of the first sample medical image and the sample staining pattern conversion information. The first noise is the noise added to the second sample image representation of the second sample medical image. The noisy sample image representation is obtained by adding the first noise to the second sample image representation.

2. The method according to claim 1, characterized in that, The step of inputting the staining mode conversion information and the first image representation into a noise prediction model to perform noise prediction processing on a preset noise image representation based on the staining mode conversion information and the first image representation to obtain predicted noise, and then performing denoising processing on the preset noise image representation based on the predicted noise to obtain a second image representation, includes: Obtain the preset noisy image and the denoising processing iteration threshold; The preset noise image is encoded to obtain a preset noise image representation; The preset noise image representation is determined as the current image representation, and the denoising processing iteration threshold is determined as the current denoising iteration value; The current denoising iteration value is decreased according to a preset subtraction factor; The coloring mode conversion information and the first image representation are input into the noise prediction model to perform noise prediction processing on the current image representation to obtain predicted noise. The current image representation is then denoised based on the predicted noise to obtain an updated current image representation. The current denoising iteration value is then decremented according to a preset subtraction factor until the current denoising iteration value is one. The current image representation with the current denoising iteration value of one is then determined as the second image representation.

3. The method according to claim 2, characterized in that, The step of performing noise prediction processing on the current image representation to obtain predicted noise includes: Obtain the time step, encode the time step, and obtain the time step representation; The coloring pattern conversion information is encoded to obtain the coloring pattern conversion representation; Feature extraction processing is performed based on the staining mode transformation representation, the time step representation, the current image representation, and the first image representation to obtain a conditional control representation; Noise prediction processing is performed based on the staining mode transformation representation, the time step representation, the current image representation, and the conditional control representation to obtain predicted noise.

4. The method according to claim 3, characterized in that, The step of performing feature extraction processing based on the staining mode transformation representation, the time step representation, the current image representation, and the first image representation to obtain a conditional control representation includes: The first image representation is subjected to a first convolution process to obtain a convolutional image representation; The current image representation and the convolutional image representation are added together to obtain a first intermediate representation; Feature extraction processing is performed on the staining pattern transformation representation, the time step representation, and the first intermediate representation to obtain the second intermediate representation; The second intermediate representation is subjected to a second convolution process to obtain the conditional control representation.

5. The method according to claim 4, characterized in that, The step of performing feature extraction processing on the staining pattern transformation representation, the time step representation, and the first intermediate representation to obtain the second intermediate representation includes: The coloring pattern transformation representation, the time step representation, and the first intermediate representation are subjected to feature extraction processing to obtain an intermediate feature representation; Feature extraction processing is performed on the staining pattern transformation representation, the time step representation, and the intermediate feature representation to obtain a second intermediate representation.

6. The method according to claim 4, characterized in that, The step of performing feature extraction processing on the staining pattern transformation representation, the time step representation, and the first intermediate representation to obtain the second intermediate representation includes: The coloring mode transformation representation and the first intermediate representation are fused to obtain a first prompt image fusion representation. Feature extraction is performed based on the time step representation and the first prompt image fusion representation to obtain a first prompt image feature representation. The attention weight of the first image region is calculated based on the feature representation of the first prompt image. Then, the representation weighting process is performed based on the first image region attention weight and the fusion representation of the first prompt image to obtain the intermediate feature representation. The coloring mode transformation representation and the intermediate feature representation are fused to obtain a second prompt image fusion representation. Feature extraction is performed based on the time step representation and the second prompt image fusion representation to obtain a second prompt image feature representation. The second image region attention weight is calculated based on the feature representation of the second prompt image. The second intermediate representation is obtained by weighting the representation based on the second image region attention weight and the fusion representation of the second prompt image.

7. The method according to claim 5 or 6, characterized in that, The conditional control representation includes a first conditional control representation and a second conditional control representation; The second convolution process performed on the second intermediate representation to obtain the conditional control representation includes: The intermediate feature representation is subjected to a second convolution process to obtain the first conditional control representation; The second intermediate representation is subjected to a second convolution process to obtain the second conditional control representation.

8. The method according to claim 7, characterized in that, The step of performing noise prediction processing based on the staining mode transformation representation, the time step representation, the current image representation, and the conditional control representation to obtain predicted noise includes: Feature extraction processing is performed on the staining mode transformation representation, the time step representation, and the current image representation to obtain a third intermediate representation; Feature extraction processing is performed on the staining pattern transformation representation, the time step representation, and the third intermediate representation to obtain a fourth intermediate representation; The fourth intermediate representation and the first conditional control representation are added together to obtain the fifth intermediate representation; The coloring mode conversion representation, the time step representation, and the fifth intermediate representation are decoded to obtain the sixth intermediate representation; The sixth intermediate representation and the second conditional control representation are added together to obtain the predicted noise.

9. The method according to any one of claims 1 to 8, characterized in that, After decoding the second image representation to obtain the second medical image, the process further includes: The second medical image is identified as the current medical image. The current medical image is then subjected to quality detection processing to obtain the staining quality score corresponding to the current medical image. When the staining quality score is less than the staining quality threshold, acquire synonymous staining pattern conversion information that has a semantic similarity greater than a preset similarity threshold with the staining pattern conversion information, and update the staining pattern conversion information according to the synonymous staining pattern conversion information; The staining pattern conversion information and the first image representation are input into the noise prediction model to regenerate a new second medical image until the staining quality score is not less than the staining quality threshold. The current medical image with a staining quality score not less than the staining quality threshold is then determined as the second medical image.

10. The method according to any one of claims 1 to 9, characterized in that, The staining pattern conversion information includes text information indicating that a first medical image with a first staining pattern is converted into a second medical image with a second staining pattern, as well as feature description information of the first staining pattern and feature description information of the second staining pattern.

11. The method according to any one of claims 1 to 10, characterized in that, The training steps for the noise prediction model are as follows: Acquire a first sample medical image, a second sample medical image, and sample staining pattern conversion information, wherein the sample staining pattern conversion information includes text information that converts the first sample medical image of the third staining pattern into the second sample medical image of the fourth staining pattern; The first sample medical image is encoded to obtain a first sample image representation, and the second sample medical image is encoded to obtain a second sample image representation; Add first noise to the second sample image representation to obtain a noisy sample image representation; The sample staining mode conversion information, the first sample image representation, and the noisy sample image representation are input into the noise prediction model to predict noise and obtain the second noise. The target loss is calculated based on the error between the first noise and the second noise. The noise prediction model is iteratively trained based on the target loss until the training result conditions are met, thus obtaining the trained noise prediction model.

12. The method according to claim 11, characterized in that, The noise prediction model includes a cue encoder, a time encoder, a noise prediction network, and a conditional control network. The conditional control network is used to input a sample conditional control representation into the noise prediction network to control the noise prediction network to perform noise prediction that conforms to the sample conditional control representation. The step of inputting the sample staining pattern conversion information, the first sample image representation, and the noisy sample image representation into a noise prediction model to predict noise and obtain the second noise includes: The prompt encoder encodes the sample staining pattern conversion information into a sample staining pattern conversion representation. The time step is acquired and encoded into a time step representation using the time encoder. The sample staining mode transformation representation, the time step representation, the noisy sample image representation, and the first sample image representation are input into the conditional control network for feature extraction to obtain the sample conditional control representation. The sample staining pattern transformation representation, the time step representation, the noisy sample image representation, and the sample condition control representation are input into the noise prediction network to predict noise and obtain the second noise.

13. The method according to claim 12, characterized in that, The step of iteratively training the noise prediction model based on the target loss until the training result conditions are met to obtain the trained noise prediction model includes: Fix the first weight parameters corresponding to the prompt encoder, the time encoder, and the noise prediction network; The second weight parameters corresponding to the conditional control network are iteratively trained based on the target loss to update the second weight parameters until the training termination condition is met, thereby obtaining the trained noise prediction model.

14. The method according to any one of claims 11 to 13, characterized in that, The step of adding first noise to the second sample image representation to obtain a noisy sample image representation includes: A two-dimensional standard Gaussian noise is randomly sampled as the first noise; Calculate the first combination coefficient and the first product value represented by the second sample image; Calculate the second product of the second combination coefficient and the first noise; The first product and the second product are added together to obtain a noisy sample image representation. The first combination coefficient and the second combination coefficient are determined by the random sampling time step.

15. An image processing method apparatus, characterized in that, The device includes: An information acquisition module is used to acquire a first medical image and staining pattern conversion information, wherein the first medical image has a first staining pattern and the staining pattern conversion information is used to indicate the conversion of the first staining pattern to a second staining pattern. The encoding processing module is used to encode the first medical image to obtain a first image representation; The representation generation module is used to input the staining mode conversion information and the first image representation into the noise prediction model, so as to perform noise prediction processing on the preset noise image representation according to the staining mode conversion information and the first image representation to obtain predicted noise, and perform denoising processing on the preset noise image representation according to the predicted noise to obtain a second image representation; A decoding processing module is used to decode the second image representation to obtain a second medical image, the second medical image having the second staining pattern; The noise prediction model is trained based on a target loss, which is determined based on a first noise and a second noise. The second noise is the noise predicted by the noise prediction model for the noisy sample image representation based on the first sample image representation of the first sample medical image and the sample staining pattern conversion information. The first noise is the noise added to the second sample image representation of the second sample medical image. The noisy sample image representation is obtained by adding the first noise to the second sample image representation.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1 to 14.

17. A computer device, characterized in that, include: Memory; A processor, wherein a computer program is stored in the memory, and when executed by the processor, the computer program implements the image processing method as described in any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 14.

Images