Image conversion device and method with channel flexibility

Abstract

An image conversion method according to one embodiment of the present invention may include: a step of generating a channel image by separating the channels of a bright field image; a step of generating an input embedding by encoding the channel image through an encoder; a step of generating a latent space by processing the input embedding through a self-attention mechanism; a step of generating an output embedding by performing decoding attention based on the latent space; and a step of generating a fluorescence image based on the output embedding through a decoder.

Description

Technology Field

[0001] The present invention relates to an image conversion device and method, and is an image conversion technology that converts a bright field image into a fluorescent image using a deep learning-based in silico method. Background Technology

[0002] Bright Field Imaging is a widely used technique for observing the structure of cells or tissues. In this method, an image is formed based on the degree of absorption, reflection, and refraction of light as it passes through a sample, providing structural information such as the shape of cells. Meanwhile, Fluorescence Imaging involves attaching fluorescent substances to specific molecules or cellular structures to visually represent the location or quantity of those substances. This allows for the acquisition of detailed biological information, such as specific proteins or organelles, making it an essential tool for research and diagnosis.

[0003] Brightfield imaging offers the advantage of non-invasively observing the structures of cells and tissues, but it has limitations in visualizing the precise location or quantity of specific molecules or cellular organelles. On the other hand, fluorescence imaging provides this detailed information, but it presents challenges such as cell damage caused by the use of fluorescent dyes, long imaging times, and high costs. In particular, the use of multiple types of fluorescent dyes can lead to greater cell damage or distortion of the results. For these reasons, technologies to convert brightfield images into fluorescence images have been researched. This conversion technology allows for the acquisition of detailed fluorescence images without the use of fluorescent dyes, which can significantly improve the efficiency of research and clinical trials.

[0004] However, a problem arises when the number of channels in the input image (bright field) differs from the number of channels in the output image (fluorescence image) during the process of converting between bright field and fluorescence images. While bright field images can consist of data with multiple channels by collecting images at various depths along the z-axis, the number of channels in fluorescence images is variable depending on the specific molecule or structure being observed. Since existing deep learning models are trained assuming a fixed number of channels, they have limitations in processing data with new channel counts. To solve this problem, there is a need for technology that can flexibly respond even in environments where the number of input and output channels fluctuates. The problem to be solved

[0005] To solve the aforementioned problems, the present invention aims to provide a deep learning-based image conversion device and method having channel flexibility that can operate stably even in an environment where the number of input and output channels varies while converting from a bright field image to a fluorescence image. means of solving the problem

[0006] An image conversion method according to one embodiment of the present invention may include: a step of generating a channel image by separating the channels of a bright field image; a step of generating an input embedding by encoding the channel image through an encoder; a step of generating a latent space by processing the input embedding through a self-attention mechanism; a step of generating an output embedding by performing decoding attention based on the latent space; and a step of generating a fluorescence image based on the output embedding through a decoder.

[0007] In addition, according to one embodiment, the step of generating a latent space may be characterized by using input embeddings as keys and values ​​of a self-attention mechanism.

[0008] In addition, according to one embodiment, the step of generating a latent space may be characterized by using the initialized latent space as a query for a self-attention mechanism.

[0009] In addition, according to one embodiment, the step of generating a latent space may be characterized in that the initialization of the latent space is performed only in the first step of the self-attention mechanism, and thereafter, the updated value is used to generate the final latent space.

[0010] In addition, according to one embodiment, the step of generating output embeddings may be characterized by using a latent space as a key and a value.

[0011] In addition, according to one embodiment, the step of generating output embeddings may be characterized by using the target average image of each channel of the fluorescence image as a query.

[0012] In addition, according to one embodiment, the image conversion method may further include the step of matching the feature distribution of a bright field image and the feature distribution of a fluorescence image based on an optimal transport plan.

[0013] In addition, according to one embodiment, the feature distribution of the fluorescence image may be characterized by being extracted from the fluorescence image through an encoder.

[0014] In addition, according to one embodiment, the target encoder may be characterized by using a pre-trained model that can reconstruct itself through VQGAN.

[0015] In addition, according to one embodiment, the matching step may be characterized by obtaining an optimal transport plan through the Sinkhorn algorithm.

[0016] An image conversion device according to one embodiment of the present invention may include a channel image generation module that separates the channels of a bright field image to generate a channel image, an input embedding module that encodes the channel image through an encoder to generate an input embedding, a latent space generation module that processes the input embedding through a self-attention mechanism to generate a latent space, an output embedding module that performs decoding attention based on the latent space to generate an output embedding, and a fluorescence image generation module that generates a fluorescence image based on the output embedding through a decoder.

[0017] In addition, according to one embodiment, the latent space generation module may be characterized by using input embeddings as keys and values ​​of a self-attention mechanism.

[0018] In addition, according to one embodiment, the latent space generation module may be characterized by using the initialized latent space as a query for the self-attention mechanism.

[0019] In addition, according to one embodiment, the latent space generation module may be characterized in that the initialization of the latent space is performed only in the first step of the self-attention mechanism, and thereafter, the updated value is used to generate the final latent space.

[0020] In addition, according to one embodiment, the output embedding module may be characterized by using a latent space as a key and a value.

[0021] In addition, according to one embodiment, the output embedding module may be characterized by using the target average image of each channel of the fluorescence image as a query.

[0022] In addition, according to one embodiment, the image conversion device may further include a matching module that matches the feature distribution of a bright field image and the feature distribution of a fluorescence image based on an optimal transport plan.

[0023] In addition, according to one embodiment, the feature distribution of the fluorescence image may be characterized by being extracted from the fluorescence image through an encoder.

[0024] In addition, according to one embodiment, the target encoder may be characterized by using a pre-trained model that can reconstruct itself through VQGAN.

[0025] In addition, according to one embodiment, the matching module may be characterized by obtaining an optimal transport plan through the Sinkhorn algorithm. Effects of the invention

[0026] According to the image conversion apparatus and method of the present invention, it is possible to convert a bright field image with various channel numbers into a fluorescence image, thereby enabling the utilization of fluorescence imaging information while solving problems such as cell damage, high cost, and time consumption caused by the use of fluorescent dyes.

[0027] In addition, the image conversion device and method of the present invention enable the utilization of a model previously trained on new data, thereby significantly improving the efficiency of research and clinical application.

[0028] In addition, according to the image conversion device and method of the present invention, a bright field image and a fluorescent image can be directly matched to enable more accurate conversion. Brief explanation of the drawing

[0029] FIG. 1 is a block diagram of an image conversion device according to one embodiment. Figure 2 is a schematic diagram of an image conversion process according to one embodiment. FIG. 3 is a flowchart of an image conversion method according to one embodiment. Specific details for implementing the invention

[0030] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0031] The terms used in this specification will be briefly explained, and the invention will be described in detail.

[0032] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0033] Throughout the specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "part," "module," and "unit" used in the specification refer to a unit that processes at least one function or operation and may be implemented as software, hardware components such as FPGAs or ASICs, or a combination of software and hardware. However, the terms "part," "module," and "unit" are not limited to software or hardware. "Part," "module," and "unit" may be configured to reside in an addressable storage medium or configured to run one or more processors. Accordingly, as an example, terms such as "part," "module," and "unit" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0034] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. In addition, parts unrelated to the description are omitted in the drawings to clearly explain the invention. Terms including ordinal numbers, such as “first,” “second,” etc., may be used to describe various components, but the components are not limited by these terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term “and / or” includes a combination of multiple related items or any one of the multiple related items.

[0035] The various systems, devices, storage media, modules, and mechanisms described in this specification may be implemented in a dedicated computing device, or in one or more computing chips of one or more dedicated computing devices.

[0037] Hereinafter, the image conversion device (1) and method of the present invention will be described.

[0038] The image conversion device (1) and method of the present invention are related to a technology for converting a bright field image into a fluorescent image using a deep learning-based in silico method, and can flexibly process various channels.

[0040] The image conversion device (1) of the present invention will be described below with reference to the drawings.

[0041] FIG. 1 is a block diagram of an image conversion device (1) of the present invention according to one embodiment.

[0042] Referring to FIG. 1, the image conversion device (1) of the present invention may include a collection unit (10), a storage unit (20), an output unit (30), and a processor (40). In addition to the configurations described above, the image conversion device (1) of the present invention may include various configurations necessary to implement image conversion without limitation. For example, the image conversion device (1) may include a communication unit (not shown) for wired / wireless communication with an external device (e.g., a shooting device and a server). Here, wired communication may include, for example, USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), RS-232 (Recommended Standard 232), or POTS (Plain Old Telephone Service), and wireless communication may include Wifi (Wireless Fidelity), BT (Bluetooth), NFC (Near Field Communication), GPS (Global Positioning System), cellular communication (e.g., LTE, LTE-A, CDMA, WCDMA, UMTS, WiBro, or GSM, etc.).

[0043] The collection unit (10) can collect various data and / or commands required for image conversion in the image conversion device (1) of the present invention. The collection unit (10) can collect data related to user settings or image data, etc., that are input through various devices. For example, the collection unit (10) can collect user input information input through input devices such as a keyboard, mouse, or touchscreen, or image data including bright field images and fluorescent images from a server or a shooting device.

[0044] The bright field image collected by the collection unit (10) is an image that can be used as input data for image conversion into a fluorescent image, represents the structure of a cell or tissue, and may be an image that includes multiple channels. The bright field image may be an image taken at multiple depths in the z-axis direction and may represent data obtained under various observation conditions.

[0045] The fluorescent image collected by the collection unit (10) is an image that can be used as target data for image conversion, and may be an image that visualizes a specific molecule or cell structure. The image conversion device (1) of the present invention must learn a deep learning model in the process of converting a bright field image into a fluorescent image, and the fluorescent image can be used as target data in this learning.

[0046] Brightfield images and fluorescence images may contain varying numbers of channels. A channel can refer to a component containing data that reflects various depths and / or observation conditions within each image. In other words, each channel may represent a specific part, structure, or observed property of the image, and the channel may contain information regarding these. The number of channels in brightfield images and fluorescence images may differ, which has acted as a factor making mutual image conversion difficult in conventional technology.

[0047] According to one embodiment, the bright field image utilized in the image conversion device (1) of the present invention may include images captured along the z-axis. That is, the channels may be divided according to the data obtained at each depth, which is acquired by capturing images at various depths of cells or tissues along the z-axis.

[0048] According to another embodiment, the bright field image may include images taken under various observation conditions. That is, channels may be divided according to each data obtained by taking images of the same sample under different conditions (e.g., wavelength of light, illumination intensity, etc.).

[0049] According to one embodiment, the fluorescent image utilized in the image conversion device (1) of the present invention may include an image visualizing intracellular molecules, proteins, or structural details using a specific fluorescent dye. Accordingly, the channel may be divided according to the molecules or structures of the cell and / or tissue being observed.

[0050] The collection unit (10) can collect various data and / or commands without limitation in addition to the examples described above.

[0051] The storage unit (20) can receive and store programs, data, and / or commands used in the image conversion device (1) of the present invention from the collection unit (10) or the processor (40), and can also transmit the stored data to the processor (40) or the output unit (30) as needed. For example, the storage unit (20) can receive and store bright field images collected by the collection unit (10), embeddings generated by the processor (40), latent spaces, and fluorescence images, and can transmit them to the processor (40) or the output unit (30) as needed.

[0052] The storage unit (20) may be implemented as at least one of a main memory and an auxiliary memory. The main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM, for example, and the auxiliary memory may be implemented based on a device capable of storing data permanently or semi-permanently, such as a flash memory device (Solid State Drive (SSD), etc.), an SD (Secure Digital) card, a hard disk drive (HDD), a compact disc, a DVD, or a laser disc.

[0053] The output unit (30) can output data that is computed and output by the processor (40) to the outside. The output unit (30) receives data generated from the storage unit (20) and / or the processor (40) and can notify the user or transmit it to another external device. For example, the output unit (30) can transmit a fluorescent image generated by the processor (40) to an external output device so that it can be output in the form of an image. Here, the output device may include a display, an image output terminal, etc., but is not limited thereto.

[0054] The processor (40) can receive data and / or commands from the collection unit (10) or the storage unit (20), and calculate or process them to produce a result. Additionally, the processor (40) can transmit the calculated result to the storage unit (20) or the output unit (30). The processor (40) can perform image conversion by receiving and processing the received data and / or commands. The processor (40) can perform the above calculation or processing by including a plurality of modules. Accordingly, the processor (40) can perform the operations described in the following embodiments, and the operations described as being performed by the device and / or module in the following embodiments can be considered as being performed by the processor (40) unless otherwise specified.

[0055] The processor (40) can perform machine learning based on an artificial intelligence model. The processor (40) can write data to the storage unit (20) or read data stored in the storage unit (20), and in particular, can process data according to predefined operation rules or an artificial intelligence model by executing a program stored in the storage unit (20). Functions related to the artificial intelligence model according to the present invention can be operated through the processor (40) and the storage unit (20). The processor (40) can control the processing of input data according to predefined operation rules or an artificial intelligence model stored in the storage unit (20).

[0056] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model is trained using a number of training data by a learning algorithm, thereby creating predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0057] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0058] The processor (40) may include one or more electronic devices capable of performing various computational and control processing, such as a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller unit (MCU), an application processor (AP), an electronic control unit (ECU), a microprocessor (Micom), or other such devices. These processing or control devices may be implemented, for example, by using one or more semiconductor chips, circuits, or related components alone or in combination. The processor (40) may perform processing in the image conversion process described later through a plurality of modules implemented in the above devices.

[0059] In addition, according to one embodiment, the processor (40) of the present invention may be designed with a structure specialized for processing an artificial intelligence model. The artificial intelligence model may be composed of various modules capable of effectively processing and converting data, including an encoder and a decoder, and each module may perform the role of analyzing and processing input data and converting it into output data.

[0060] Figure 2 is a schematic diagram of an image conversion process according to one embodiment.

[0061] Referring to FIG. 2, the processor (40) of the present invention can generate a channel image by separating the channels of a bright field image, and can generate an input embedding by encoding the channel image through an encoder. Additionally, the processor (40) can generate a latent space by processing the input embedding through a self-attention mechanism, and can generate an output embedding by performing decoding attention based on the latent space. Additionally, the processor (40) can generate a fluorescence image based on the output embedding through a decoder.

[0062] Referring again to FIG. 1, the processor (40) of the present invention may include a channel image generation module (410), an input embedding module (420), a latent space generation module (430), an output embedding module (440), a fluorescence image generation module (450), and a matching module (460).

[0063] The channel image generation module (410) can generate channel images by separating the channels of a bright field image. The channel image generation module (410) can receive a bright field image and generate multiple channel images by separating its various channels. For example, if the bright field image can be separated into channels based on being captured at various depths along the z-axis, the channel image generation module (410) can generate individual channel images by separating the image according to the z-axis depth. According to another embodiment, the channel image generation module (410) can generate channel images by separating the bright field image based on channels according to observation conditions. That is, the channel image generation module (410) can receive a bright field image as input, separate each channel to generate channel images, and perform the task of separating channels according to the depth of the z-axis or dividing images captured under various conditions into individual channels.

[0064] The input embedding module (420) can generate input embeddings by encoding the channel image through an encoder. An embedding is a method of representing high-dimensional data in a low-dimensional space, meaning that key information and features are compressed and converted into a vector form. Therefore, the input embedding here may refer to a vector that represents the important features of the channel image in a low-dimensional space by compressing them through an encoder.

[0065] The input embedding module (420) receives channel images and can encode them through an encoder. The encoder can convert the channel images from high-dimensional data into low-dimensional embedding vectors, called input embeddings, through a neural network. In this process, the encoder can extract important patterns or features from each channel image and convert them into numerically represented vectors. For example, features such as cell structure, specific patterns, and color information may be included in the input embeddings.

[0066] The input embedding module (420) can generate different input embeddings by individually encoding each channel image. Therefore, the input embedding module (420) can flexibly respond to data with a variable number of channels.

[0067] The latent space generation module (430) can generate a latent space by processing input embeddings through a self-attention mechanism. The latent space generation module (430) can receive input embeddings and process them through a self-attention mechanism to finally generate a latent space. The latent space can include information about similarities and relationships between each embedding by compressing and expressing important features of the input embeddings. Since the latent space generation module can process multiple input embeddings and embed them into a single latent space, it can flexibly respond regardless of the number of channels of the input image.

[0068] A self-attention mechanism is a mechanism that evaluates how related each element within input data is to one another. It enables data elements to focus on more important information through their interactions with other data elements. In other words, through computation, the self-attention mechanism calculates the relationship between each input embedding vector and other embedding vectors, and can evaluate the importance of each embedding vector. The self-attention mechanism can utilize three elements: query, key, and value. The query represents the information a specific embedding seeks, the key represents the characteristics of each embedding, and the value represents the actual information corresponding to those characteristics. The self-attention mechanism calculates the similarity between the query and the key, and can select and emphasize values ​​based on their importance.

[0069] The latent space generation module (430) can calculate how image features of each channel are related to each other within the input embedding by applying a self-attention mechanism. For example, it can evaluate how related one channel image feature is to another channel image feature and emphasize important features. The latent space generation module (430) can determine how related each vector of the input embedding is to other vectors through query, key, and value relationships, and thereby reconstruct the input embedding to generate a latent space.

[0070] The latent space generated by the latent space generation module (430) is a result reflecting the interaction between input embeddings and can be data analyzing the relationship between each channel. The latent space can play a key role in the process of converting into a fluorescent image through the decoding attention described later. Since the latent space can reflect the mutual correlation between input embeddings, it can form a space that reflects the correlation between images rather than a simple data summary.

[0071] According to one embodiment, the latent space generation module (430) can use the input embeddings as keys and values ​​of the self-attention mechanism, and the initialized latent space (latent variable of FIG. 2) can be used as a query of the self-attention mechanism.

[0072] According to an embodiment, the latent space generation module (430) can use input embeddings as keys and values ​​for a self-attention mechanism. By using input embeddings as keys and values, the latent space generation module (430) can determine how related each channel image is to other channel images. Since each input embedding is a vector that describes the features of the underlying channel image, using the input embedding as a key can serve as a criterion for calculating similarity with other embeddings. Additionally, the latent space generation module (430) can extract input embeddings that are determined to have high similarity through the key and use them as values.

[0073] Additionally, according to the embodiment, the latent space generation module (430) can use the initialized latent space as a query for the self-attention mechanism. Here, the initialized latent space may refer to the latent space at the starting stage, where learned data is not reflected. The initialized latent space is a value initialized at the first stage of learning, and may have the same meaning as a latent variable that is set to an arbitrary value or a default value before the model learns. The latent space generation module (430) can use the initialized latent space as a query that serves as a criterion for finding important information in the input embedding, and since the initialized latent space does not contain specific information, it can only set a criterion for what information to look for.

[0074] According to one embodiment, the potential space generation module (430) can generate a final potential space by initializing the potential space only in the first step of the self-attention mechanism and subsequently using the updated value. The potential space initialization can be performed only in the first step of the self-attention mechanism. The initialized potential space may be a value arbitrarily set by the user or a pre-set value, in a state where specific information has not been learned.

[0075] The latent space generation module (430) can use input embeddings as key and value to enable each input embedding to describe the features of the corresponding channel image and to participate in the process of selecting important information based on similarity, and can also use the initialized latent space as a query to set criteria for what to look for in the input embeddings. As a result, the latent space generation module (430) can generate a sophisticated latent space that reflects the interrelationships between input embeddings, and the latent space reflects important information between channel images and can be used as key basic data for the process of converting to a fluorescent image.

[0076] The output embedding module (440) can generate output embeddings by performing decoding attention based on a latent space. The output embedding module (440) can selectively extract necessary information from the latent space through decoding attention, extract features suitable for each channel of the output image, and finally generate output embeddings based thereon.

[0077] Decoding attention is one of the attention mechanisms, a method of selectively reflecting important information when converting input information into an output data format. Decoding attention calculates how relevant a specific part of the input information is to the output and can generate an output based on that importance. Decoding attention can use three components of the attention mechanism of query, key, and value, just as described in the self-attention mechanism above. The output embedding module (440) evaluates the relationship between the input data and the output based on the three components of decoding attention to selectively highlight information necessary for the final output and remove unnecessary information.

[0078] The output embedding module (440) can generate an output embedding as a result of decoding attention. The output embedding is a vector in which information extracted from the latent space is reconstructed to fit the output format. That is, it may be a vector summarizing the important features of the data to be output, serving as an intermediate form for important information in the latent space to be converted into a final output image through a decoding process. The output embedding reflects the important features that a specific channel of each image must represent, and based on this, a final fluorescent image can be generated. The output embedding is a form in which necessary information is selectively extracted from the latent space through a decoding attention mechanism and expressed as a low-dimensional vector, which can be utilized to generate the fluorescent image described later.

[0079] The output embedding module (440) calculates the similarity between the query and the key through decoding attention and can determine which information in the latent space is most important in the output fluorescence image. Information with high similarity can be selected from the values ​​and extracted as an output embedding. That is, the output embedding module (440) can synthesize the evaluated information to generate an output embedding that includes only the information to be converted into a fluorescence image. In this process, a vector can be generated that reflects only the important features related to the query among the various features of the latent space.

[0080] According to one embodiment, the output embedding module (440) can use the latent space as a key and value and can use the target average image of each channel of the fluorescence image as a query.

[0081] According to an embodiment, the output embedding module (440) can use the latent space as the key and value of the decoding attention. By using the latent space as the key and value, the output embedding module (440) can calculate and evaluate the similarity of how related each element of the query and the latent space is, and important information based on the similarity calculation can be selected as the value to generate the final output embedding.

[0082] Additionally, according to an embodiment, the output embedding module (440) may use the target average image of each channel as a query. Here, the target average image of each channel may represent the target state and ideal average shape that each channel intends to represent during the process of generating a fluorescence image. For example, the target average image may represent a representative image representing a specific molecule or cell structure to be expressed in each channel of the fluorescence image. According to one embodiment, the target average image of each channel may be derived from an input actual fluorescence image or a fluorescence image generated by computation, or it may be an image generated by reflecting average characteristics derived from various data through a learning process. However, the method of setting the target average image is not limited to the above example, and it may be set by methods such as arbitrarily setting it by the user.

[0083] The output embedding module (440) can use a target average image as a query to find an output embedding that has features suitable for the target in the latent space through decoding attention. For example, if a specific fluorescent image channel visualizes a specific protein, the target average image of that channel represents the ideal structure and location of the protein, and the output embedding module (440) can use this as a query to find important information suitable for the target in the latent space through the calculation of similarity between the query and the key, and generate an output embedding with this as a value.

[0084] The fluorescence image generation module (450) can generate a fluorescence image based on an output embedding through a decoder. The fluorescence image generation module (450) receives the output embedding and inputs it into the decoder, thereby enabling the decoder to generate a fluorescence image based on it. The decoder can convert the detailed features of the image (e.g., cell structure) into an image format that can visually represent the vector information included in the output embedding.

[0085] According to an embodiment, the fluorescent image generation module (450) includes a plurality of decoders, and by using each decoder separately, a plurality of fluorescent images can be generated.

[0086] The matching module (460) can match the feature distribution of the bright field image and the feature distribution of the fluorescence image based on the Optimal Transport Plan.

[0087] The feature distributions of brightfield images and fluorescence images can refer to the form in which information from each channel is distributed in vector space. The feature distribution of brightfield images can be represented as information from each channel (e.g., cell density, structure by depth, etc.) distributed in vector space, while the feature distribution of fluorescence images indicates how structural features or biological information are expressed in each channel, thereby allowing for the determination of the location and quantity of specific substances or structures. Each feature distribution may serve as a summary of the key structural information found in the brightfield and fluorescence images.

[0088] The optimal transport plan can be calculated through the Optimal Transport (OT) problem. The Optimal Transport problem is a mathematical concept that deals with the problem of minimizing the distance (or cost) between two probability distributions; it is a method for finding the way to minimize the cost when moving a substance from one distribution to another. The Optimal Transport plan can represent the matching relationship between two distributions and can present the optimal path and the associated cost for moving from one point to another.

[0089] The matching module (460) can match the feature distribution of the bright field image to the feature distribution of the fluorescence image using an optimal transport plan. Through this, the matching module (460) can efficiently match features between the two images to perform more precise image transformation.

[0090] The matching module (460) can generate an input embedding that creates a latent space and distributes it within a specific space, and can extract this as a feature distribution of a bright field image. Additionally, the matching module (460) can separate the generated fluorescence image by channel and extract target features from each channel through a target encoder, and extract the distribution formed by the target features as a feature distribution of the fluorescence image.

[0091] According to one embodiment, the target encoder may use a pre-trained model to reconstruct itself through VQGAN.

[0092] According to one embodiment, the matching module (460) can obtain an optimal transport plan through the Sinkhorn algorithm. An objective function for learning a target encoder using the optimal transport plan obtained by the Sinkhorn algorithm can be defined as shown in the following mathematical formula 1.

[0093]

[0094] Here, L ot is the objective function for encoder training, consisting of the optimal transport plan Γ and the cost matrix , and the Frobenius inner product including the hyperparameter λ (<·> F Includes ).

[0095] In addition, the Kullback-Leibler divergence (KL divergence) term (D φ By using ), the feature distribution of the bright field image and the feature distribution of the fluorescence image can be induced to be close. The matching module (460) can obtain an optimal transport plan through calculations using this Sinkhorn algorithm and enable more accurate image transformation.

[0096] In addition, β is a hyperparameter that controls the value of the Kullback-Leibler divergence term, and P can represent the probability distribution of the input embedding and Q can represent the probability distribution of the target embedding.

[0097] Additionally, when the matching module (460) obtains an optimal transport plan through calculations using the Sinkhorn algorithm, the matching module (460) obtains an MSE loss (L between the converted fluorescence image and the target) recon The decoder can be trained by calculating ). In this case, the final objective function for model training is as follows Equation 2.

[0098]

[0099] The image conversion method of the present invention will be described below with reference to the drawings.

[0100] FIG. 3 is a flowchart of an image conversion method of the present invention according to one embodiment.

[0101] Referring to FIG. 3, the image conversion method of the present invention may include a step (S310) of generating a channel image by separating the channels of a bright field image. Additionally, the image conversion method may include a step (S320) of generating an input embedding by encoding the channel image through an encoder. Additionally, the image conversion method may include a step (S330) of generating a latent space by processing the input embedding through a self-attention mechanism. Additionally, the image conversion method may include a step (S340) of generating an output embedding by performing decoding attention based on the latent space. Additionally, the image conversion method may include a step (S350) of generating a fluorescence image based on the output embedding through a decoder.

[0102] In addition, according to one embodiment, the step of generating a latent space may be characterized by using input embeddings as keys and values ​​of a self-attention mechanism.

[0103] In addition, according to one embodiment, the step of generating a latent space may be characterized by using the initialized latent space as a query for a self-attention mechanism.

[0104] In addition, according to one embodiment, the step of generating a latent space may be characterized in that the initialization of the latent space is performed only in the first step of the self-attention mechanism, and thereafter, the updated value is used to generate the final latent space.

[0105] In addition, according to one embodiment, the step of generating output embeddings may be characterized by using a latent space as a key and a value.

[0106] In addition, according to one embodiment, the step of generating output embeddings may be characterized by using the target average image of each channel of the fluorescence image as a query.

[0107] In addition, according to one embodiment, the image conversion method may further include the step of matching the feature distribution of a bright field image and the feature distribution of a fluorescence image based on an optimal transport plan.

[0108] In addition, according to one embodiment, the feature distribution of the fluorescence image may be characterized by being extracted from the fluorescence image through an encoder.

[0109] In addition, according to one embodiment, the target encoder may be characterized by using a pre-trained model that can reconstruct itself through VQGAN.

[0110] In addition, according to one embodiment, the matching step may be characterized by obtaining an optimal transport plan through the Sinkhorn algorithm.

[0112] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential characteristics of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the invention.

Claims

Claim 1 An image conversion method comprising: a step of generating a channel image by separating the channels of a bright field image; a step of generating an input embedding by encoding the channel image through an encoder; a step of generating a latent space by processing the input embedding through a self-attention mechanism; a step of generating an output embedding by performing decoding attention based on the latent space; a step of generating a fluorescence image based on the output embedding through a decoder; and a step of matching the feature distribution of the bright field image with the feature distribution of the fluorescence image; wherein the step of generating the output embedding is characterized by using the latent space as a key and value and using the target average image of each channel of the fluorescence image as a query. Claim 2 An image conversion method according to claim 1, wherein the step of generating the latent space is characterized by using the input embedding as the key and value of the self-attention mechanism. Claim 3 An image transformation method according to paragraph 2, wherein the step of generating the potential space is characterized by using the initialized potential space as a query of the self-attention mechanism. Claim 4 An image transformation method according to claim 3, wherein the step of generating the potential space is characterized in that the initialization of the potential space is performed only in the first step of the self-attention mechanism, and thereafter, the updated value is used to generate the final potential space. Claim 5 delete Claim 6 delete Claim 7 An image conversion method according to claim 1, further comprising the step of matching the feature distribution of the bright field image and the feature distribution of the fluorescence image based on an optimal transport plan. Claim 8 An image conversion method according to claim 7, wherein the feature distribution of the fluorescent image is extracted from the fluorescent image through a target encoder. Claim 9 An image conversion method according to claim 8, wherein the target encoder uses a model that has been pre-trained to reconstruct itself through VQGAN. Claim 10 An image conversion method according to claim 9, wherein the matching step is characterized by obtaining the optimal transport plan through the Sinkhorn algorithm. Claim 11 An image conversion device comprising: a channel image generation module that separates the channels of a bright field image to generate a channel image; an input embedding module that encodes the channel image through an encoder to generate an input embedding; a latent space generation module that processes the input embedding through a self-attention mechanism to generate a latent space; an output embedding module that performs decoding attention based on the latent space to generate an output embedding; a fluorescence image generation module that generates a fluorescence image based on the output embedding through a decoder; and a matching module that matches the feature distribution of the bright field image with the feature distribution of the fluorescence image; wherein the output embedding module uses the latent space as a key and value and uses the target average image of each channel of the fluorescence image as a query. Claim 12 An image conversion device according to claim 11, wherein the latent space generation module uses the input embedding as the key and value of the self-attention mechanism. Claim 13 An image conversion device according to claim 12, wherein the latent space generation module uses the initialized latent space as a query for the self-attention mechanism. Claim 14 An image conversion device according to claim 13, wherein the potential space generation module is characterized in that the initialization of the potential space is performed only in the first step of the self-attention mechanism, and thereafter, the updated value is used to generate the final potential space. Claim 15 delete Claim 16 delete Claim 17 An image conversion device according to claim 11, further comprising a matching module that matches the feature distribution of the bright field image and the feature distribution of the fluorescence image based on an optimal transport plan. Claim 18 An image conversion device according to claim 17, characterized in that the feature distribution of the fluorescent image is extracted from the fluorescent image through a target encoder. Claim 19 An image conversion device according to claim 18, wherein the target encoder uses a pre-trained model capable of reconstructing itself through VQGAN. Claim 20 An image conversion device according to claim 19, wherein the matching module obtains the optimal transport plan through the Sinkhorn algorithm.

Images