Image processing method and device using reinforcement learning

Abstract

The present disclosure relates to an image processing method performed by one or more processors. The image processing method according to the present disclosure may perform image compression in consideration of information imbalance and appropriately restore a compressed image.

Description

Image processing method and device using reinforcement learning

[0001] The present disclosure relates to image compression and restoration techniques, and more particularly, to a method for efficiently compressing and restoring high-resolution medical images and videos using machine learning techniques.

[0002]

[0003] Digital pathology systems are technologies that convert, store, and analyze pathological samples into digital images. These systems replace traditional microscopic diagnostics, increase accuracy and efficiency, and facilitate remote collaboration. Digital image processing and analysis provide automated diagnostic assistance, contributing to pathology research and education.

[0004] Digital pathology systems utilize massive pathology images. Because these massive images require high storage costs, their storage requires compression. Conventional pathology image compression techniques compress the entire image without considering information imbalances within the image, leading to problems such as inability to properly restore compressed pathology images.

[0005] Therefore, there is a need in the art for a technology to appropriately compress and restore pathological images.

[0006] The present invention was conducted as part of the Information and Communications Broadcasting Innovation Talent Development Project of the National Information Society Agency (Project Identification Number: 1711193663, Subproject Number: 2020-0-01819-004, Research Project Name: ICT Masterpiece Talent Development (Korea University), Host Organization: Korea University Industry-Academic Cooperation Foundation, Research Period: 2020.07.01 ~ 2029.12.31), and the Digital Pathology-Based Cancer Specialized AI Analysis Solution Development (R&D) Project of the Korea Health Industry Development Institute (Project Identification Number: 1465038729, Subproject Number: HI21C0940010023, Research Project Name: Development of a Digital Pathology Data Creation and Management System for Artificial Intelligence Software Development, Host Organization: Catholic University Industry-Academic Cooperation Foundation, Research Period: 2021.04.01 ~ This is derived from research conducted as part of the (2025.12.31)

[0007] Meanwhile, the Korean government, which is the subject of the task, has no property interest in any aspect of the present invention.

[0008]

[0009] The present disclosure seeks to provide an effective compression technique that takes into account information imbalance contained within a large pathological image.

[0010] The problems to be solved by the present disclosure are not limited to the problems described above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0011]

[0012] An image processing method performed by one or more processors according to one embodiment of the present disclosure, comprising: an image compression step; wherein the image compression step comprises: a compression information determination step of determining compression information for each of one or more original image patches constituting an original image; and a step of performing compression on the original image based on the compression information for each of the one or more original image patches; wherein the compression information comprises at least one of whether or not to compress each of the one or more original image patches or a compression ratio; and wherein the compression information determination step comprises: a step of determining at least one of whether or not to compress a first original image patch constituting at least a portion of the original image or a compression ratio; and a step of determining a second original image patch constituting at least a portion of the original image as an uncompressed patch; and may be characterized by using a compression information determination agent based on machine learning.

[0013] In addition, the compression information decision agent is characterized in that it is learned using a compression information decision agent learning step based on reinforcement learning; and the compression information decision agent learning step may include a step of generating a first restored image patch from first patch compression data that compresses the first original image patch using a machine learning-based decoder module; a step of calculating an information restoration level value of the first restored image patch; and a step of calculating a reward function value using the compression information for the first original image patch and the information restoration level value of the first restored image patch.

[0014] In addition, the first patch compression data may be characterized by including a first original image patch representation vector that encodes the first original image patch using a machine learning-based encoder module.

[0015] In addition, the step of calculating the information restoration level value of the first restored image patch may include the steps of: generating a first masking image patch representing the first type data included in the first original image patch; generating a second masking image patch representing the first type data included in the first restored image patch; and calculating a similarity between the first masking image patch and the second masking image patch.

[0016] Additionally, the original image patch and the restored image patch may be characterized in that they are cell images, and the first type data is a cell nucleus.

[0017] Additionally, the above similarity may be characterized by using a Dice coefficient.

[0018] In addition, the reward function may be characterized in that the higher the compression ratio of the first original image patch and the higher the similarity between the first masking image and the second masking image, the greater the reward.

[0019] In addition, the image restoration step may further include a step of generating a restoration image from original image compression data obtained by compressing the original image using a machine learning-based decoder; and the original image compression data may include first patch compression data obtained by compressing the first original image patch using a machine learning-based encoder; and the second original image patch.

[0020] In addition, the step of generating a restored image from the original image compression data may include the step of generating a first restored image patch from the first patch compression data using the decoder; and the step of combining the first restored image patch and the second original image patch.

[0021] An image processing device according to one aspect of the present disclosure comprises: a processor; and a memory; wherein the processor compresses an image, wherein the processor determines compression information for each of one or more original image patches constituting an original image, and performs compression on the original image based on the compression information for each of the one or more original image patches, wherein the compression information includes at least one of whether or not to compress each of the one or more original image patches or a compression ratio, and wherein the processor determines the compression information includes determining at least one of whether or not to compress a first original image patch constituting at least a portion of the original image or a compression ratio, and determining a second original image patch constituting at least a portion of the original image as an uncompressed patch, and may be characterized by using a compression information determination agent based on machine learning.

[0022] A computer program stored on a computer-readable recording medium according to one aspect of the present disclosure, wherein the computer program includes one or more operations executable by a processor, the operations including: an image compression step; wherein the image compression step includes: a compression information determination step of determining compression information for each of one or more original image patches constituting an original image; and a step of performing compression on the original image based on the compression information for each of the one or more original image patches; wherein the compression information includes at least one of whether or not to compress each of the one or more original image patches or a compression ratio; and wherein the compression information determination step includes: a step of determining at least one of whether or not to compress or a compression ratio for a first original image patch constituting at least a portion of the original image; and a step of determining a second original image patch constituting at least a portion of the original image as an uncompressed patch; and may be characterized by using a compression information determination agent based on machine learning.

[0023]

[0024] The image processing method according to the present disclosure can perform image compression while considering information imbalance and ensure that the compressed image is appropriately restored.

[0025] The effects according to the present disclosure are not limited to the effects described above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0026]

[0027] Figure 1 is a conceptual diagram illustrating a large pathological image according to one aspect of the present disclosure.

[0028] FIG. 2 is a conceptual diagram illustrating an image processing method according to one aspect of the present disclosure.

[0029] Figure 3 is a conceptual diagram illustrating an image processing process according to one aspect of the present disclosure.

[0030] FIG. 4 is a chart showing the performance of an image processing method according to one aspect of the present disclosure.

[0031] FIG. 5 is a diagram showing compression-related performance indicators of an image processing method according to one aspect of the present disclosure.

[0032] FIG. 6 is a block diagram illustrating a computing device for performing an image processing method according to one aspect of the present disclosure.

[0033]

[0034] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by the exemplary embodiments. Unless otherwise defined, all terms (including technical and scientific terms) used in this specification shall be used with meanings that can be commonly understood by those of ordinary skill in the technical field to which this disclosure pertains. However, this may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc.

[0035] Additionally, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless explicitly and specifically defined otherwise. In certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should be defined based on their meaning and the overall content of this disclosure, rather than simply their names.

[0036] Throughout this specification, when a part is said to "include" a certain component, this does not mean that other components may be included, but rather that other components may be excluded, unless specifically stated otherwise. Furthermore, the singular forms used herein also include plural forms unless specifically stated otherwise. Furthermore, the expression "at least one of a, b, and / or c" used throughout this specification can encompass "a alone," "b alone," "c alone," "a and b," "a and c," "b and c," or "all of a, b, and c."

[0037] Meanwhile, terms such as "first and / or second" used in this specification may be used to describe various components, but are only used to distinguish one component from another and are not intended to be limited to the components referred to by those terms. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and the second component may also be referred to as the first component.

[0038] In addition, terms such as “unit”, “module”, etc. described in this specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or a combination of hardware and software. In addition, embodiments of the present disclosure in this specification may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware or / and software configurations that execute specific functions. For example, embodiments of the present disclosure may employ direct circuit configurations such as memory, processing, logic, look-up tables, etc. that may execute various functions under the control of one or more microprocessors or other control devices.

[0039] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In describing the embodiments, descriptions of technical details that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to convey the gist of the present invention more clearly without obscuring unnecessary explanation. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically depicted. Furthermore, the size of each component does not entirely reflect the actual size. Throughout this specification, the same reference numerals may refer to the same or corresponding components.

[0040] The graph neural network learning method according to the present disclosure is described in detail with reference to the drawings below.

[0041] Figure 1 is a conceptual diagram illustrating a large pathological image according to one aspect of the present disclosure.

[0042] Referring to FIG. 1, a large pathological image (100) may refer to an image of a specific organ, tissue, and / or cell taken using various medical and bio-related devices including a microscope.

[0043] In some embodiments, the large-scale pathology image (100) may include image information for one or more cells. Preferably, one or more cells included in the large-scale pathology image (100) are stained to facilitate observation and imaging.

[0044] The amount of information contained in different regions of the image constituting the giant pathology image (100) may vary. For example, the first region (110) of the giant pathology image (100) is expressed in bold, which may indicate that a large number of cells are densely concentrated in the first region (110). On the other hand, the second region (120) of the giant pathology image (100) is expressed in relatively light, which may indicate that a smaller number of cells exist in the second region (120) than in the first region (110).

[0045] Accordingly, when compression is performed on the first region (110), a relatively greater loss of information may occur compared to when compression is performed on the second region (120). On the other hand, since the amount of information expressed by the second region (120) is relatively smaller than that of the first region (110), when compression is performed on the second region (120), a lesser loss of information may occur compared to when compression is performed on the first region (110).

[0046]

[0047] FIG. 2 is a conceptual diagram illustrating an image processing method according to one aspect of the present disclosure.

[0048] Referring to FIG. 2, an image compression and restoration method according to some embodiments of the present disclosure may include an encoder (320) for compressing an original image patch (210), a decoder (330) for generating a restored image patch (220) from original image compression data that compresses the original image, and an image masking module (400) for generating a first masking image patch (210a) from the original image patch (210) and a second masking image patch (220a) from the restored image patch (220).

[0049] In some embodiments of the present disclosure, the original image patch (210) may refer to a large pathological image before compression. In one embodiment, the original image patch (210) may be an image captured using a medical device or observation device, such as a microscope and / or an endoscope, of a specific organ, tissue, and / or multiple cells. As will be described in detail later in FIG. 3, the original image patch (210) may be segmented into multiple image patches. As described above in FIG. 1, each of the multiple segmented image patches may contain different amounts of information.

[0050] In some embodiments, the encoder (320) and the decoder (330) may be used together to form a backbone (300). In one embodiment, the backbone (300) may be a module using software, hardware, and / or a combination thereof for implementing the compression of an original image and the generation of a restored image. In one embodiment, the backbone (300) may perform machine learning-based image compression and restoration using the encoder (320) and the decoder (330). Preferably, the backbone (300) may include additional modules in addition to the encoder (320) and the decoder (330). As an example, the backbone (300) may include a compression information determination agent that determines an image patch on which compression is to be performed among a plurality of image patches constituting the original image patch (210), and a module for training the compression information determination agent. In one embodiment, the backbone (300) may be implemented using various structures such as an autoencoder and / or a generative adversarial neural network. In a preferred example, the backbone (300) can be implemented utilizing various structures such as an autoencoder (AE), an ESRGAN, a Vector Quantized Variational Autoencoder (VQVAE), and / or a Variational Autoencoder (VAE).

[0051] In one embodiment, the backbone (300) may include a compression information determination agent (310). In some embodiments, the compression information determination agent (310) may be a module implemented using software, hardware, and / or a combination thereof that generates compression information by determining compression information for each of one or more image patches constituting the original image patch (210). In one embodiment, the compression information determined by the compression information determination agent (310) for the image patches may include whether the image patches are compressed and / or a compression ratio. For example, whether or not to be compressed may be determined for each patch. Furthermore, in one embodiment, compression ratio information may be included in addition to the compression ratio information for the image patches. In this case, the compression ratio may be determined as a value between 0 and 1 for each patch. For example, if the compression ratio is 0.9, the compressed data obtained by compressing the image patches may have only 10% of the capacity of the original image patches. In another embodiment, only information regarding whether or not to be compressed may be determined for the image patches. Furthermore, the compression ratio of the entire image can be determined based on the compression and / or compression ratio information for individual image patches.

[0052] In some embodiments, the compressed information decision agent (310) may be constructed using machine learning. In one embodiment, the compressed information decision agent (310) may be constructed using a neural network including one or more neurons. Preferably, the compressed information decision agent (310) may be trained using reinforcement learning. In one embodiment, a reward function for training the compressed information decision agent (310) may be determined based on the compressed information for the image patch and the information restoration level of the restored image patch.

[0053] In some embodiments, the information restoration level of a restored image patch may be determined by comparing it with an original image patch. In one embodiment, the information restoration level of a restored image patch may be determined using the similarity between the images. For example, the information restoration level of a restored image patch may be determined using the similarity between the original image patch and the restored image patch. As another example, the information restoration level of a restored image patch may be determined using masking images processed to express only specific types of information for the original image patch and the restored image patch. In this case, the information restoration level value may be determined using the similarity between the masking images.

[0054] That is, the similarity between images may be the similarity between the original image patch (210) and the restored image patch (220). As another example, the similarity between images may be the similarity between the first masking image patch (210a) and the second masking image patch (220a).

[0055] In one embodiment, the masking image patch may be data for representing a specific type of information included in the original image patch (210) or the restored image patch (220). For example, the masking image patch may be an image including a segmentation result or a segmentation result for the location of a cell within the original image patch (210) and / or the restored image patch (220).

[0056] At this time, as will be described later, the metric for measuring similarity can utilize the similarity of both one-dimensional and multi-dimensional variables. For example, to measure similarity, one-dimensional variable similarity such as Tanimoto coefficient, Dice coefficient, and Jaccard coefficient, and / or multi-dimensional variable similarity including Euclidean distance and Pearson correlation coefficient can be used. Alternatively, to measure image similarity, metrics such as Mean Squared Error (MSE), Peak Signal to Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM) between images can be used. Furthermore, images can be converted into vectors to measure similarity between images.

[0057] Preferably, the reward function can be designed so that the compression information decision agent (310) performs compression only on image patches with minimal information loss due to compression. For example, the reward function can be designed to have a higher value as more image patches are compressed, as the compression ratio for the image patches is higher, or as the similarity between images is higher. Preferably, the reward function can be designed as in Equations 1 to 3 below.

[0058]

[0059]

[0060]

[0061] In the above mathematical expressions 1 to 3, Reward is a reward function, Whether to compress the image patch, is a similarity between image patches, which may represent the information restoration level value of the restored image patch. In the above mathematical expression 3, the Dice coefficient is used as an example, is a penalty term due to information loss due to compression, and λ is a hyperparameter due to the penalty term.

[0062] Referring to mathematical expressions 1 to 3, the compression information decision agent (310) receives a reward when compressing an image patch, but may receive a penalty in the reward function value depending on the degree to which the restored image patch (220) and the original image patch (210) are not similar (i.e., depending on the degree of information loss of the restored image patch (220)). Accordingly, the higher the information restoration level value of the restored image patch (220) (i.e., the higher the similarity between the restored image patch (220) and the original image patch (210) or the higher the similarity between the second masking image patch (220a) and the first masking image patch (210a)), the higher the reward function value may be.

[0063] More preferably, the compression information determination agent (310) according to some embodiments of the present disclosure may utilize compression level information for the original image in order to avoid compressing all image patches constituting the original image. The compression level information for the original image may refer to the ratio of image patches constituting the original image on which compression is performed, and in this case, may be different from the compression ratio included in the compression information for the image patches.

[0064] In some embodiments of the present disclosure, the encoder (320) may be a module implemented using software, hardware, and / or a combination thereof for compressing the raw image patch (210) to generate raw image compression data. In one embodiment, the encoder (320) may be constructed using machine learning. Preferably, the encoder (320) may be a module for converting at least a portion of the raw image patch (210) into a fixed-size vector using a neural network. For example, the encoder (320) may be a neural network for encoding each image patch, at least a portion of the image patches constituting the raw image patch (210), into a fixed-size vector. The operation of the encoder (320) is described in detail below with reference to FIG. 3 .

[0065] In some embodiments of the present disclosure, the decoder (330) may be a module implemented using software, hardware, and / or a combination thereof to generate a restored image patch (220) using the original image compression data generated by the encoder (320). In one embodiment, the decoder (330) may be constructed using machine learning. Preferably, the decoder (330) may generate the restored image patch (220) from the original image compression data using a neural network. The operation of the decoder (330) is described in detail below with reference to FIG. 3 .

[0066] In some embodiments of the present disclosure, the encoder (320) and decoder (330) included in the backbone (300) may be trained preferentially before the training of the compression information determination agent (310). In one embodiment, the training of the encoder (320) and decoder (330) may be performed independently and / or separately from the training of the compression information determination agent (310). Preferably, the training of the encoder (320) and decoder (330) may be performed using reconstruction loss, etc. resulting from the compression and reconstruction of an image. Furthermore, the training of the compression information determination agent (310) may be performed after the training of the encoder (320) and decoder (330) is completed.

[0067] Referring back to FIG. 2, the restored image patch (220) according to some embodiments of the present disclosure may be an image reconstructed by the backbone (300) and / or the decoder (330). In one embodiment, the restored image patch (220) may differ from the original image patch (210) in at least some portions. At this time, an information restoration level value of the restored image patch (220) with respect to the original image patch (210) may be calculated. In one embodiment, the information restoration level value of the restored image patch (220) may be determined by calculating a similarity using the original image patch (210) and the restored image patch (220) as they are. For example, the similarity may be calculated using the Mean Squared Error (MSE), the Peak Signal to Noise Ratio (PSNR), and / or the Structural Similarity Index Measure (SSIM) between the original image patch (210) and the restored image patch (220). In another embodiment, image vectors of an original image patch (210) and a restored image patch (220) may be generated, and similarity may be measured using distance and / or cosine similarity between the generated image vectors. More preferably, only specific types of information included in the original image patch (210) and the restored image patch (220) may be filtered to extract a new image, and then the similarity between the extracted images may be calculated, and the information restoration level value of the restored image patch (220) may be determined based on this. In addition, as will be described later, the similarity between a masking image patch expressing specific types of information included in the original image patch (210) and the restored image patch (220) may be calculated, and the information restoration level value of the restored image patch (220) may be determined based on this.

[0068] Referring back to FIG. 2 in this regard, an image masking module (400) may be utilized to perform an image compression and restoration method according to some embodiments of the present disclosure. In some embodiments, the image masking module (400) may be a module for generating a first masking image patch (210a) from an original image patch (210) and a second masking image patch (220a) from a restored image patch (220).

[0069] In one embodiment, the first masking image patch (210a) and the second masking image patch (220a) may be images that represent only specific types of information and / or data among the information included in the original image patch (210) and the restored image patch (220). In one embodiment, the specific types of information represented in the masking images may be various intracellular structures, including cell nuclei and / or mitochondria. In one embodiment, the masking image patches may be images that include segmentation results or segmentation results for the locations of cells, intracellular structures, and / or cell nuclei within the original image patch (210) and / or the restored image patch (220).

[0070] In some embodiments, the image masking module (400) may be implemented using machine learning. Preferably, the image masking module (400) may be implemented using a machine learning module for identifying intracellular structures from cell images. For example, the image masking module (400) may be implemented using a machine learning module for identifying cell nuclei and / or cellular structures, such as HoverNet.

[0071] Referring back to FIG. 2, the first masking image patch (210a) and the second masking image (222) may be images generated by the image masking module (400). At this time, the first masking image patch (210a) may be associated with the original image patch (210), and the second masking image patch (220a) may be associated with the restored image patch (220). In one embodiment, the similarity between the first masking image patch (210a) and the second masking image patch (220a) may be calculated.

[0072] In one embodiment, techniques such as the MSE, PSNR, and / or SSIM described above may be used to compute the similarity between the first masking image patch (210a) and the second masking image patch (220a). In another embodiment, the similarity between the first masking image patch (210a) and the second masking image patch (220a) may be measured using the similarity (e.g., distance and / or cosine similarity, etc.) between the image vectors encoding the first masking image patch (210a) and the second masking image patch (220a). In another embodiment, a technique for measuring the similarity between information included in an image may be used. In one embodiment, the Tanimoto coefficient, the Dice coefficient, and / or the Jaccard coefficient may be used to measure the similarity between the first masking image patch (210a) and the second masking image patch (220a). That is, the similarity between the first masking image patch (210a) and the second masking image patch (220a) can be calculated depending on whether a cell nucleus is expressed in the first masking image patch (210a) and the second masking image patch (220a) in a specific area (e.g., pixel) within the image.

[0073] The similarity calculated by the above-described similarity calculation technique can be determined as the information restoration level value of the restored image patch (220). In addition, the above-described similarity calculation technique is exemplary, and various techniques for calculating similarity between data can be used to determine the information restoration level value.

[0074] Figure 3 is a conceptual diagram illustrating an image processing process according to one aspect of the present disclosure.

[0075] Referring to FIG. 3, an original image (100) according to some embodiments of the present disclosure may be divided into a plurality of original image patches (210) through tessellation. In some embodiments, each of the plurality of divided original image patches (210) may have its compression status and / or compression ratio determined by a compression information determination agent as described above with reference to FIG. 2. At this time, among the original image patches, an image patch determined to be compressed by the compression information determination agent may be determined as a first original image patch (211), and an image patch determined not to be compressed (i.e., an uncompressed patch) may be determined as a second original image patch (212).

[0076] Referring back to FIG. 3, an image processing method according to some embodiments of the present disclosure may compress a first raw image patch (211) for which compression has been determined. At this time, as described above with reference to FIG. 2, compression of the first raw image patch (211) may be performed using an encoder. At this time, the encoder may be a module that reduces the size of the first raw image patch (211) or converts the first raw image patch (211) into a first raw image patch representation vector (211a). At this time, the first raw image patch representation vector (211a) generated by the encoder may be a vector having a fixed size.

[0077] In some embodiments, compression by the encoder may not be performed on the second raw image patch (212) for which it is determined that compression will not be performed. Accordingly, the second raw image patch (212) may be combined with the restored image patch generated from the first raw image patch representation vector (211a) in a later restoration step.

[0078] In some embodiments, a restored image patch (220) may be generated from a first raw image patch representation vector (211a). A decoder may be used to generate the restored image patch (220). In one embodiment, the decoder may be constructed using machine learning. Preferably, the encoder and decoder may be implemented based on structures such as an autoencoder, VQVAE, VAE, GAN, or ESRGAN.

[0079] An image processing method according to some embodiments may generate a restored image by combining one or more restored image patches (220) generated by a decoder and one or more second original image patches (212a).

[0080] FIG. 4 is a chart showing the performance of an image processing method according to one aspect of the present disclosure.

[0081] Referring to the chart disclosed in part a on the upper left of FIG. 4, it can be seen that the image compression and restoration method according to some embodiments of the present disclosure shows high performance indices in terms of PSNR, SSIM and / or LPIPS (Learned Perceptual Image Patch Similarity) for image patches and / or whole slide images (WSI) when the structure of the backbone is selected as ESRGAN and / or VQVAE.

[0082] In addition, the chart disclosed in part b in the upper right of FIG. 4 is the result of conducting a Turing test on images restored by the image compression and restoration method according to the present disclosure, which was conducted on five experts. In this Turing test, restored images generated by the image processing method according to the present disclosure were shown to the experts using ESRGAN as a backbone. Referring to the chart, the probability that the experts could distinguish between the restored images and the original images was 56%, which is almost the same level of accuracy as the experts' random selection, which may mean that it was difficult to distinguish a significant number of restored images from the original images. In other words, this may mean that information loss due to image compression was minimized, thereby improving the quality of the restored images.

[0083] The multiple photographs disclosed in section c at the bottom of Fig. 4 are drawings comparing an original image patch and a restored image patch. Referring to this section, it can be confirmed that when the backbone is used as an ESRGAN and / or VQVAE structure, a restored image with a level almost similar to the original image can be obtained.

[0084] Additionally, Table 1 below shows image compression performance according to an image processing method according to one aspect of the present invention.

[0085]

[0086] In the above Table 1, Original WSI means the original image, and the table below shows the capacity (GB) and compression ratio (CR) of compressed data for the TCGA breast cancer dataset and the Camelyon16 dataset when ESRGAN is used as is, when image compression is performed according to the image processing method of the present invention with ESRGAN as the backbone (for hyperparameters 0.1, 0.25, 0.5, 0.75, and 1.0, respectively), and when image compression is performed according to the baseline method according to the technique disclosed in Faghani, S., Codipilly, DC, Moassefi, M., Iyer, PG, Erickson, BJ: Optimizing storage and computational efficiency: An efficient algorithm for whole slide image size reduction. Mayo Clinic Proceedings: Digital Health 1(3), 419-424 (2023).

[0087] The baseline technique simply removes the background of the original image, while the ESRGAN baseline technique compresses all image patches. In contrast, the present invention selectively compresses only some image patches, and excludes image patches that require no information loss. Considering these differences, image compression according to the present invention can have the advantage of minimizing information loss while achieving a sufficiently high compression ratio.

[0088] FIG. 5 is a diagram showing compression-related performance indicators of an image processing method according to one aspect of the present disclosure.

[0089] Figure 5 is a diagram illustrating the cost savings in pathological image storage by the image processing method according to the present disclosure. Figure 5 represents simulated data assuming that the amount of data collected is sampled from a normal distribution, that the amount of data increases by 5% annually on average, and that the data is stored in the cloud for 10 years at $22.53 per terabyte. The compression ratio of the image compression method according to the present invention is assumed to be 3.77%, and the compression ratio of the baseline technique is assumed to be 10%.

[0090] Referring to Figure 5, the baseline technique can store pathological images at approximately 10 times lower cost compared to storing the original image during the data storage period. Conversely, when image compression is performed according to the image compression technique of the present invention, pathological images can be stored at 26 times lower cost compared to the original image.

[0091]

[0092] FIG. 6 is a block diagram illustrating a computing device for performing an image processing method according to one aspect of the present disclosure.

[0093] The computing device (800) may include a memory (810), a processor (820), a communication module (830), and an input / output interface (840), and as illustrated in FIG. 6, the computing device (800) may be configured to communicate information and / or data via a network using the communication module (830).

[0094] The memory (810) may include any non-transitory computer-readable recording medium. In one embodiment, the memory (810) may include a non-volatile mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, a non-volatile mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the computing device (800) as a separate permanent storage device distinct from the memory. In addition, the memory (810) may store an operating system and at least one program code.

[0095] These software components may be loaded from a computer-readable recording medium separate from the memory (810). This separate computer-readable recording medium may include a recording medium directly connectable to the computing device (800), for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. As another example, the software components may be loaded into the memory (810) via a communication module (830) other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (810) based on a computer program that is installed by files provided by developers or a file distribution system that distributes installation files of applications via the communication module (830).

[0096] Here, the program may include program commands, data files, and data structures, either singly or in combination. The program may be designed and produced using machine language code or high-level language code. The program may be specifically designed to implement the aforementioned malicious application detection system, or may be implemented using various functions or definitions that are known and available to those skilled in the art of computer software. The program for implementing the aforementioned malicious application detection system may be recorded on a processor-readable recording medium.

[0097] The memory can store a program that performs the operations described above and the operations described below, and the processor can execute the program stored in the memory. In the case where there are multiple processors and memories, they can be integrated into a single chip or provided in physically separate locations. The memory can include volatile memory such as static random access memory (S-RAM) and dynamic random access memory (DRAM) for temporarily storing data. In addition, the memory can include non-volatile memory such as read-only memory (ROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM) for long-term storage of control programs and control data.

[0098]

[0099] The processor (820) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to another user terminal (not shown) or another external system via the memory (810) or the communication module (830).

[0100] A processor may include various logic circuits and arithmetic circuits, process data according to a program provided from memory, and generate control signals based on the processing results. The memory and processor may be implemented as separate chips. Alternatively, the memory and processor may be implemented as a single chip.

[0101] The processor (820) may include one or more processors. In this case, the one or more processors may be homogeneous or heterogeneous processors. For example, the processor (820) may include heterogeneous processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or a neural processing unit (NPU). For convenience of explanation, one or more homogeneous or heterogeneous processors may be referred to as a "processor" in this specification.

[0102] The communication module (830) may provide a configuration or function for a user terminal (not shown) and a computing device (800) to communicate with each other via a network, and may provide a configuration or function for the computing device (800) to communicate with an external system (e.g., a separate cloud system, etc.). For example, control signals, commands, data, etc. provided under the control of the processor (820) of the computing device (800) may be transmitted to the user terminal and / or the external system via the communication module (830) and the network via the communication module of the user terminal and / or the external system.

[0103] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0104] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0105] The wireless communication module may include a wireless communication interface including an antenna and a transmitter for transmitting a Wi-Fi signal. Furthermore, the wireless communication module may further include a Wi-Fi signal conversion module that modulates a digital control signal output from the control unit through the wireless communication interface into an analog wireless signal under the control of the control unit.

[0106] The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving Wi-Fi signals. Furthermore, the wireless communication module may further include a Wi-Fi signal conversion module for demodulating an analog wireless signal received through the wireless communication interface into a digital control signal.

[0107] The short-range communication module is for short-range communication, and can support short-range communication using at least one of Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0108] Additionally, the input / output interface (840) of the computing device (800) may be a means for interfacing with a device (not shown) for input or output that may be connected to the computing device (800) or that the computing device (800) may include.

[0109] The input unit is configured to receive information and data including audio information (or signals), text, etc. from a network or a user, and may include at least one microphone and at least one user input unit. Data collected from the input unit may be analyzed and processed into a user control command.

[0110] The user input unit is for receiving information from the user, and when information is input through the user input unit, the control unit can control the operation of the device to correspond to the input information. The user input unit may include a hardware physical key (e.g., a button located on at least one of the front, rear, and side of the device, a dome switch, a jog wheel, a jog switch, etc.) and a software touch key. As an example, the touch key may be a virtual key, a soft key, or a visual key displayed on a touch screen type display unit through software processing, or may be a touch key disposed on a part other than the touch screen. Meanwhile, the virtual key or visual key may have various forms and be displayed on the touch screen, and may be, for example, formed of a graphic, text, an icon, a video, or a combination thereof.

[0111] Although the input / output interface (840) is illustrated as a separate element from the processor (820) in FIG. 6, the present invention is not limited thereto, and the input / output interface (840) may be configured to be included in the processor (820). The computing device (800) may include more components than those illustrated in FIG. 6. However, there is no need to explicitly illustrate most conventional components.

[0112] The processor (820) of the computing device (800) may be configured to manage, process and / or store information and / or data received from multiple user terminals and / or multiple external systems.

[0113] The above-described methods and / or various embodiments may be realized by digital electronic circuits, computer hardware, firmware, software, and / or a combination thereof. Various embodiments of the present disclosure may be implemented as a computer program that is executed by a data processing device, for example, one or more programmable processors and / or one or more computing devices, or stored on a computer-readable recording medium and / or a computer-readable recording medium. The above-described computer program may be written in any form of programming language, including a compiled language or an interpreted language, and may be distributed in any form, such as a standalone program, a module, a subroutine, etc. The computer program may be distributed through a single computing device, multiple computing devices connected through the same network, and / or multiple computing devices distributed to be connected through multiple different networks.

[0114] The methods and / or various embodiments described above may be performed by one or more processors configured to execute one or more computer programs that process, store, and / or manage any function, function, etc. by operating on the basis of input data or generating output data. For example, the methods and / or various embodiments of the present disclosure may be performed by special purpose logic circuits such as Field Programmable Gate Arrays (FPGAs) or Application Specific Integrated Circuits (ASICs), and an apparatus and / or system for performing the methods and / or embodiments of the present disclosure may be implemented as special purpose logic circuits such as FPGAs or ASICs.

[0115] The one or more processors executing the computer program may include a general-purpose or special-purpose microprocessor and / or one or more processors of any type of digital computing device. The processor may receive instructions and / or data from each of read-only memory and random-access memory, or may receive instructions and / or data from the read-only memory and the random-access memory. In the present invention, components of a computing device performing the methods and / or embodiments may include one or more processors for executing instructions, and one or more memory devices for storing instructions and / or data.

[0116] According to one embodiment, the computing device can transmit and receive data to and from one or more mass storage devices for storing data. For example, the computing device can receive and / or transfer data from a magnetic disc or an optical disc. A computer-readable storage medium suitable for storing instructions and / or data associated with a computer program may include, but is not limited to, any form of non-volatile memory, including semiconductor memory devices such as Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), flash memory devices, and the like. For example, the computer-readable storage medium may include a magnetic disk such as an internal hard disk or a removable disk, a magneto-optical disk, a CD-ROM, and a DVD-ROM disk.

[0117] To provide interaction with a user, a computing device may include, but is not limited to, a display device (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), etc.) for providing or displaying information to the user, and a pointing device (e.g., a keyboard, a mouse, a trackball, etc.) for allowing the user to provide input and / or commands to the computing device. That is, the computing device may further include any other types of devices for providing interaction with the user. For example, the computing device may provide any form of sensory feedback to the user, including visual feedback, auditory feedback, and / or tactile feedback, for interaction with the user. In this regard, the user may provide input to the computing device through various gestures, such as visual, vocal, or motion.

[0118] In the present invention, various embodiments may be implemented in a computing system that includes backend components (e.g., a data server), middleware components (e.g., an application server), and / or front-end components. In this case, the components may be interconnected via any form or medium of digital data communication, such as a communications network. For example, the communications network may include a Local Area Network (LAN), a Wide Area Network (WAN), etc.

[0119] A computing device based on the exemplary embodiments described herein may be implemented using hardware and / or software configured to interact with a user, including a user device, a user interface (UI) device, a user terminal, or a client device. For example, the computing device may include a portable computing device, such as a laptop computer. Additionally or alternatively, the computing device may include, but is not limited to, Personal Digital Assistants (PDAs), tablet PCs, game consoles, wearable devices, Internet of Things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, and the like. The computing device may further include other types of devices configured to interact with a user. Furthermore, the computing device may include a portable communication device (e.g., a mobile phone, a smart phone, a cordless cellular phone, etc.) suitable for wireless communication over a network, such as a mobile communication network. The computing device may be configured to communicate wirelessly with a network server using wireless communication technologies and / or protocols, such as Radio Frequency (RF), Microwave Frequency (MWF), and / or Infrared Ray Frequency (IRF).

[0120] In embodiments according to the present disclosure, functions related to artificial intelligence may be implemented through a processor and memory. In this case, the processor may be any one of a general-purpose processor such as a CPU (Center Processing Unit), an AP (Application Processor), a DSP (Digital Signal Processor), a graphics-only processor such as a GPU (Graphics Processing Unit), a VPU (Vision Processing Unit), and an AI-only processor such as an NPU (Neural Network Processing Unit). The processor may process input data according to predefined operation rules or AI models stored in the memory. Alternatively, if the processor is an AI-only processor, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model. In some embodiments according to the present disclosure, functions related to artificial intelligence may be implemented through a plurality of processors.

[0121] In embodiments of the present disclosure, predefined operating rules or artificial intelligence models may be configured to perform machine learning. Here, "configured to perform machine learning" means that the predefined operating rules or artificial intelligence models are trained using a learning algorithm and a plurality of learning data sets to perform a desired characteristic (or purpose). This learning may be performed within the device itself implementing the artificial intelligence according to the present disclosure, or may be performed through a separate server and / or system.

[0122] Artificial intelligence models can be implemented as neural networks (or artificial neural networks) and operate based on statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to a general model in which artificial neurons (nodes) form a network by combining synapses, and through learning, the strength of the synaptic connections changes, thereby achieving problem-solving capabilities. A neural network can be composed of multiple neural network layers. For example, a neural network can include an input layer, a hidden layer, and an output layer. Each of the multiple neural network layers can include at least one node and at least one weight, and can perform neural network operations through operations between the computational results of the previous (precious) layer and the weights. At least one weight of the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, at least one weight can be updated during the learning process to reduce or minimize the loss or cost values ​​obtained from the artificial intelligence model. Neural networks can infer the desired outcome from arbitrary input.

[0123] The learning methods of artificial intelligence models can be categorized into supervised learning, where input and output data are provided as training data, and the correct answer (output data) corresponding to the problem (input data) is determined, unsupervised learning, where only input data is provided without output data, and the correct answer (output data) corresponding to the problem (input data) is not determined, and reinforcement learning, where a reward is given whenever an action is taken in the current state, and learning progresses in the direction of maximizing this reward. Alternatively, they can be categorized according to the architecture, which is the structure of the learning model.

[0124] In an embodiment of the present disclosure, the artificial intelligence model is a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, etc., R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, At least one of various artificial intelligence structures and algorithms, such as Recommendation, Data Creation, etc., may be used, and the above-described examples are merely listing examples of artificial intelligence structures and algorithms used according to embodiments of the present disclosure, and do not limit the artificial intelligence structures and algorithms used according to embodiments of the present disclosure.

[0125] Meanwhile, the embodiments disclosed in this specification may be implemented in the form of a recording medium that stores computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium. The computer-readable recording medium may include any type of recording medium that stores instructions that can be deciphered by a computer. Examples thereof include ROM, RAM, magnetic tape, magnetic disk, flash memory, and optical data storage devices.

[0126] The above-described embodiments are specific examples for implementing the present disclosure. In addition to the above-described embodiments, the present disclosure will also include embodiments that can be easily modified or modified. Furthermore, the present disclosure will also include techniques that can be easily modified and implemented using the above-described embodiments.

[0127] The various embodiments of the present invention, including specific structural and functional details, are exemplary. Therefore, the embodiments of the present disclosure are not limited to those described above and may be implemented in various other forms. Furthermore, the terminology used herein is intended to describe certain embodiments and is not intended to limit the embodiments. For example, singular terms and the above may be interpreted to include plural forms, unless the context clearly dictates otherwise.

[0128] In the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which they pertain. Furthermore, commonly used terms, such as terms defined in dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology.

[0129] While the present invention has been described in connection with certain embodiments herein, various modifications and variations can be made without departing from the scope of the present disclosure, which would be apparent to those skilled in the art. Furthermore, such modifications and variations are intended to fall within the scope of the claims appended to this specification.

Claims

1. An image processing method performed by one or more processors, including an image compression step; The above image compression step is, A compression information determination step for determining compression information for each of one or more original image patches constituting the original image; and A step of performing compression on the original image based on compression information for each of the one or more original image patches; The compression information is characterized in that it includes at least one of whether or not each of the one or more original image patches is compressed or a compression ratio, The above compression information determination step is: A step of determining at least one of whether to compress or a compression ratio for a first original image patch constituting at least a portion of the original image; and A step of determining a second original image patch constituting at least a portion of the original image as an uncompressed patch; characterized by using a machine learning-based compressed information decision agent, Image processing methods.

2. In paragraph 1, The above compression information decision agent, It is characterized in that it is learned using a compression information decision agent learning step based on reinforcement learning; The above compressed information decision agent learning step is, A step of generating a first restored image patch from first patch compression data that compresses the first original image patch using a machine learning-based decoder module; A step of calculating an information restoration level value of the first restoration image patch; and A step of calculating a compensation function value using the compression information for the first original image patch and the information restoration level value of the first restored image patch; Image processing methods.

3. In paragraph 2, The above first patch compression data is, A first original image patch representation vector encoding the first original image patch using a machine learning-based encoder module; characterized in that it includes; Image processing methods.

4. In paragraph 2, The step of calculating the information restoration level value of the above first restoration image is: A step of generating a first masking image patch representing first type data included in the first original image patch; generating a second masking image patch representing the first type data included in the first restoration image patch; and A step of calculating a similarity between the first masking image patch and the second masking image patch; Image processing methods.

5. In paragraph 4, The above original image patch and the above restored image patch are cell images, The first type of data is characterized in that it is the location of a cell structure. Image processing methods.

6. In paragraph 4, The above similarity is, Characterized by using dice coefficients, Image processing methods.

7. In paragraph 2, The reward function is, The higher the information restoration level value of the first restoration image patch, the greater the reward. Image processing methods.

8. In paragraph 1, further comprising an image restoration step; The above image restoration step is, A step of generating a restored image from compressed original image data using a machine learning-based decoder; The above original image compression data is, First patch compression data obtained by compressing the first original image patch using a machine learning-based encoder; and characterized in that it comprises the second original image patch; Image processing methods.

9. In paragraph 8, The step of generating a restored image from the above original image compression data is: A step of generating a first restored image patch from the first patch compression data using the decoder; comprising a step of combining the first restoration image patch and the second original image patch; Image processing methods.

10. As an image processing device, processor; and memory; including; The above processor, Compress the image, The above processor compresses the image, Determining compression information for each of one or more original image patches constituting the original image, and performing compression on the original image based on the compression information for each of the one or more original image patches, The compression information is characterized in that it includes at least one of whether or not each of the one or more original image patches is compressed or a compression ratio, The above processor determines the compression information, Determining at least one of whether to compress or a compression ratio for a first original image patch constituting at least a portion of the original image, and determining a second original image patch constituting at least a portion of the original image as an uncompressed patch, characterized by using a machine learning-based compressed information decision agent, Image processing device.

11. A computer program stored in a computer-readable recording medium, wherein the computer program includes one or more operations executable by a processor, the operations being: including an image compression step; The above image compression step is, A compression information determination step for determining compression information for each of one or more original image patches constituting the original image; and A step of performing compression on the original image based on compression information for each of the one or more original image patches; The compression information is characterized in that it includes at least one of whether or not each of the one or more original image patches is compressed or a compression ratio, The above compression information determination step is: A step of determining at least one of whether to compress or a compression ratio for a first original image patch constituting at least a portion of the original image; and A step of determining a second original image patch constituting at least a portion of the original image as an uncompressed patch; characterized by using a machine learning-based compressed information decision agent, A computer program stored on a computer-readable recording medium.

Images