Method and system for recognizing context of digital slide image

Abstract

One aspect of the present invention comprises the steps of: receiving an original image; selecting a range of context to be analyzed from the original image; converting the range of context into a low-capacity image by using log polar coordinate image transformation or nonlinear polar coordinate image transformation; and extracting image characteristics by inputting the low-capacity image into a vision transformer.

Description

Method and system for context recognition of digital slide images

[0001] The present invention was made under the support of the Ministry of Health and Welfare under the project identification number 1465038793 and project number HA23C0142000023. The research management specialized institution of the project is the National Cancer Center, the research project name is "Cancer Conquest Promotion Research and Development Project", the research project name is "Customized Rare Cancer Using Pathogenomic Data and Artificial Intelligence", the main institution is Samsung Seoul Hospital, and the research period is from 2023.04.01 to 2025.12.31.

[0002] The present invention relates to a method and system for context recognition of a digital slide image, and more particularly, to a method and system for context recognition of a digital slide image for analyzing a digital slide image using artificial intelligence.

[0003] Histological diagnosis of pathology specimens requires a pathologist directly viewing tissue slides. However, pathological diagnoses, which are determined by the pathologist's subjective judgment, suffer from low reproducibility and inter-pathologist agreement. To address these issues, the adoption of digital slides using high-resolution digital scanners has been increasing in the pathology diagnostic process, and various image analysis programs utilizing artificial intelligence are being developed.

[0004] Most image analysis programs that utilize artificial intelligence crop WSI (Whole Slide Imaging) into smaller image patches and analyze them to derive results. However, since WSI is very large (approximately 1 to 4 GB per page) and has a high resolution (over 100,000 X 100,000 pixels), it is difficult to properly consider other surrounding contexts with a narrow area of ​​image patches. Accordingly, research is being conducted on multiple magnification learning (MIL) and graph networks that can analyze images by considering the context of the surrounding images.

[0005] However, both of these methods utilize cropping, which divides the image into small patches. This makes it difficult to fully consider the surrounding context and requires significant computation. Furthermore, many studies utilize convolutional neural networks (CNNs) to extract image features. However, unlike real-world tissue slides, these methods are sensitive to rotational transformations, leading to inconsistent results.

[0006] In this regard, U.S. Patent No. 10055551 discloses a system and method for quantitative analysis of tissue pathology images using a multi-classifier ensemble scheme. The prior art is characterized by preprocessing a digitized image, a biopsy region of interest, or an entire biopsy slide, and describing the image segments by forming feature vectors based on at least one of tissue, color, or morphological tissue characteristics, and classifying the image segments using one or more classification methods in a first classification step.

[0007] While various methods for analyzing pathological images have been proposed, they still suffer from the problem of being sensitive to rotational transformations. Therefore, there is a need for a context recognition method and system for digital slide images that utilizes a contextual image representation method that is rotationally invariant and reflects the surrounding context, replacing the traditional image cropping method.

[0008] The present invention aims to provide a method and system for recognizing context of a digital slide image, to which a contextual image expression method that is invariant to rotational transformation and can reflect the surrounding context is applied.

[0009] In order to achieve the above object, the present invention is characterized by including the steps of: receiving an original image; selecting a range of context to be analyzed from the original image; converting the range of context into a low-capacity image using log polar coordinate image transformation or nonlinear polar coordinate image transformation; and inputting the low-capacity image into a vision transformer to extract image characteristics.

[0010] Preferably, the step of selecting the range of the context may select a circular area having a radius of d pixels in the original image as the range of the context.

[0011] Preferably, the step of converting into the low-capacity image may include a step of converting the original image (x,y) into a log polar coordinate image (x´,y´).

[0012] Preferably, the step of converting to the low-capacity image may further include a step of adding a hyperparameter to the log polar coordinate image to adjust the center-to-periphery image reflection ratio.

[0013] Preferably, the step of converting into the low-capacity image may further include a step of approximating the x-coordinate (x´) of the log polar coordinate image through Maclaurin Series Expansion.

[0014] Preferably, the step of converting into the low-capacity image may include a step of converting the original image (x,y) into a non-linear polar coordinate image (x´,y´).

[0015] Preferably, the nonlinear polar coordinate image is defined by the following [Relationship 1], where f(0)=0, f(d)=1, 0 <x<d 일 때 f´(x) > 0, and 0 <x<d 일 때 f″(x) <0을 만족할 수 있다.

[0016] [Relationship 1]

[0017]

[0018] (Here, d represents the maximum radius of the original image, and N represents the length of one side of the transformed image)

[0019] Preferably, the vision transformer may be defined such that the token (H(z)) of the image patch z located at the pth horizontal and qth vertical position of the low-capacity image can be defined by the following [Relationship 2].

[0020] [Relationship 2]

[0021]

[0022] (Here, is the feature embedding of CNN (Convolutional Neural Network) for the image, means the position embedding of z)

[0023] Preferably, the vision transformer is of the above [Relationship 2] go is set to, go is set to, go can be set to .

[0024] In addition, the present invention is characterized by including an image receiving unit that receives an original image; a range selection unit that selects a range of context to be analyzed from the original image; a transformation unit that transforms the range of context into a low-capacity image using log polar coordinate image transformation or nonlinear polar coordinate image transformation; and a feature extraction unit that inputs the low-capacity image into a vision transformer having tokens defined by a CNN feature embedding, a function regarding relative distance, and a function regarding polar coordinate distance, thereby extracting image features.

[0025] The present invention has the advantage of being seamless and applicable to general situations compared to existing image patch-based context recognition techniques.

[0026] In addition, the present invention has an advantage in that it can provide a representation method that is invariant to rotational transformation by introducing polar coordinates since the context of an actual tissue slide is invariant to rotational transformation.

[0027] In addition, the present invention has the advantage of being able to determine the degree of reflection of the center or periphery using a logarithmic function and various nonlinear functions.

[0028] In addition, the present invention has the advantage of excellent calculation speed and memory efficiency because it first converts a wide area image into a small image and then performs calculations.

[0029] Figure 1 shows a flowchart of a method for recognizing context of a digital slide image according to an embodiment of the present invention.

[0030] Figure 2 shows a flowchart of steps for converting into a low-capacity image according to an embodiment of the present invention.

[0031] Figure 3 shows a schematic diagram of log polar coordinate image transformation according to an embodiment of the present invention.

[0032] Figure 4 shows the change in the conversion equation according to the change in the hyperparameter p value according to an embodiment of the present invention.

[0033] FIG. 5 shows an example of a nonlinear polar coordinate image transformation image transformation function according to an embodiment of the present invention.

[0034] Fig. 6 shows a token configuration schematic diagram of a vision transformer according to an embodiment of the present invention.

[0035] FIG. 7 shows a configuration diagram of a context recognition system for a digital slide image according to an embodiment of the present invention.

[0036] Step of receiving the original image;

[0037] A step of selecting the scope of the context to be analyzed from the above original image;

[0038] A step of converting the above context range into a low-capacity image using polar coordinate image transformation; and

[0039] A step of inputting the above low-capacity image into a vision transformer to extract image characteristics;

[0040] A method for context recognition of a digital slide image including:

[0041] Hereinafter, the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by the exemplary embodiments. The same reference numerals in each drawing indicate components that perform substantially the same functions.

[0042] The purpose and effects of the present invention can be naturally understood or made clearer by the following description, and the purpose and effects of the present invention are not limited solely by the following description. Furthermore, in describing the present invention, if a detailed description of known technologies related to the present invention is deemed to unnecessarily obscure the gist of the present invention, such detailed description will be omitted.

[0043] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the description of the invention, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0044] While terms like "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0045] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0046] When interpreting components, even if there is no explicit description, it is interpreted as including the margin of error. When describing temporal relationships, for example, when temporal continuity is described with phrases such as "after," "following," "next to," or "before," this also includes cases where the relationship is not continuous, unless "immediately" or "directly" is used.

[0047] Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.

[0048] Fig. 1 illustrates a flowchart of a method for recognizing context of a digital slide image according to an embodiment of the present invention. Referring to Fig. 1, the method for recognizing context of a digital slide image may include a step of receiving an original image (S110), a step of selecting a range of context (S120), a step of converting into a low-capacity image (S130), and a step of extracting image characteristics (S140).

[0049] A method for recognizing the context of a digital slide image can perform log polar image transformation and non-linear polar image transformation to effectively summarize a large amount of digital pathological tissue slide images used for tissue diagnosis.

[0050] The context recognition method of a digital slide image can effectively express a large-capacity digital pathology tissue slide image as a single low-capacity image by transforming a wider image area around a target area in a whole slide image (WSI) converted from a glass slide produced for tissue diagnosis through a digital scanner into a log-polar coordinate image transform or a nonlinear polar coordinate image transform that generalizes the log-polar coordinate image transform.

[0051] The method for recognizing the context of a digital slide image can be broadly divided into log-polar coordinate image transformation or non-linear polar coordinate image transformation and extracting image features using a Vision Transformer. The method for recognizing the context of a digital slide image does not use the existing image patch cropping method to obtain surrounding context information about the original image, but directly transforms the context image surrounding the original image to create a new image that reflects the surrounding context, and performs feature extraction on the transformed image to obtain a representation of the context of the original image.

[0052] Compared to existing image patch-based context recognition techniques, the context recognition method for digital slide images is seamless and applicable to general situations. Since the context of actual tissue slides is rotationally invariant, the context recognition method for digital slide images can provide a rotationally invariant representation by introducing polar coordinates. The context recognition method for digital slide images can determine the degree of reflection of the center or periphery using logarithmic functions and various nonlinear functions. The context recognition method for digital slide images first converts a large area image into a small image and then performs calculations, resulting in superior computational speed and memory efficiency.

[0053] The step of receiving an original image (S110) may receive the original image. The step of receiving an original image (S110) may receive the original image and transmit it to the step of selecting a context range (S120). The step of receiving an original image (S110) may transmit the original image to the step of selecting a context range (S120) using a wireless or wired network, and when using a wireless network, a broad mobile communication network such as CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), or WiFi may be used.

[0054] The step of selecting the context range (S120) can select the context range to be analyzed from the original image. The step of selecting the context range (S120) can appropriately select the context range from the image by considering the image characteristics to be extracted, computational speed, computational load, etc.

[0055] The step of selecting a context range (S120) may select a circular area with a radius of d pixels in the original image as the context range. Here, the center of the selected context range may be the center of the circular area with a radius of d pixels.

[0056] Fig. 2 illustrates a flowchart of a step (S130) of converting into a low-capacity image according to an embodiment of the present invention. Referring to Fig. 2, in one embodiment, the step (S130) of converting into a low-capacity image may include a step (S131) ​​of converting into a polar coordinate image, a step (S132) of adjusting a center-to-periphery image reflection ratio, and a step (S133) of approximation.

[0057] The step of converting into a low-capacity image (S130) can convert the above context range into a low-capacity image using polar coordinate image transformation. Specifically, the step of converting into a low-capacity image (S130) can use log polar coordinate image transformation or nonlinear polar coordinate image transformation as the polar coordinate image transformation. In the step of converting into a low-capacity image (S130), the low-capacity image that has been converted into a log polar coordinate image transformation or a nonlinear polar coordinate image transformation can be transformed into a square image of a specified size, and the transformed image can be input to a vision transformer to extract a feature vector.

[0058] FIG. 3 is a schematic diagram of a log polar coordinate image transformation according to an embodiment of the present invention. Referring to FIG. 3, the step of transforming into a polar coordinate image (S131) ​​can transform an original image (x, y) into a log polar coordinate image (x´, y´). The step of transforming into a polar coordinate image (S131) ​​can use a log polar coordinate image transformation to transform into a log polar coordinate image. The log polar coordinate image transformation can effectively reduce a large-capacity image continuously from a high magnification to a low magnification based on the center through a log transformation, and can make an image that has undergone a log transformation invariant to rotation transformation through a polar coordinate image transformation.

[0059] The step of converting to a polar coordinate image (S131) ​​is when the original image is P and the log polar coordinate image is Q, Log polar coordinate transformation formula for ( ) is defined as in [Mathematical Formula 1] below.

[0060] [Mathematical Formula 1]

[0061]

[0062] Here, d is the maximum radius of the original image P, N is the length of one side of the transformed image Q, and π(3.14...) is the circumference ratio. N can be determined as shown in [Table 1] below depending on the type of vision transformer (ViT) used.

[0063] ViTNOriginal ViT224ViT-L (Large)384ViT-H (Huge)512

[0064] Figure 4 shows the change of the transformation equation according to the change of the hyperparameter p value according to an embodiment of the present invention. Referring to Figure 4, the step (S132) of adjusting the center-to-peripheral image reflection ratio can adjust the center-to-peripheral image reflection ratio by adding a hyperparameter to the log polar coordinate image. The adjusted transformation equation ( ) is as shown in [Mathematical Formula 2] below.

[0065] [Equation 2]

[0066]

[0067] (Here, d represents the maximum radius of the original image P, and N represents the length of one side of the transformed image Q)

[0068] The approximation step (S133) can approximate the x-coordinate (x´) of the log polar coordinate image through Maclaurin Series Expansion. The Maclaurin Series Expansion is It can be expressed as follows, and the x-coordinate (x´) of the log polar coordinate image can be approximated as in [Mathematical Formula 3] below.

[0069] [Equation 3]

[0070]

[0071] In another embodiment, the step of converting into a low-capacity image (S130) may include a step of converting the original image (x, y) into a nonlinear polar coordinate image (x´, y´). The step of converting into a low-capacity image (S130) may use a nonlinear polar coordinate image transformation to convert into a nonlinear polar coordinate image. The nonlinear polar coordinate image transformation is an image transformation using a monotonically increasing nonlinear function similar to the log polar coordinate image transformation, and can enable more effective image feature expression by reflecting the region of interest or magnification as desired by the user. Examples of the nonlinear polar coordinate image transformation include the cycloid curve transformation and the arctangent curve transformation.

[0072] Nonlinear polar coordinate image transformation is similar to log polar coordinate image transformation, but it can mean any transformation that uses a function that satisfies certain conditions instead of a logarithmic function. Nonlinear polar coordinate image transformation is when the original image is P and the transformed image is Q, Nonlinear polar coordinate transformation equation for ( ) is as follows [Relationship 1].

[0073] [Relationship 1]

[0074]

[0075] (Here, d represents the maximum radius of the original image, and N represents the length of one side of the transformed image)

[0076] That is, the nonlinear polar coordinate image is defined by the following [Relationship 1], where f(0)=0, f(d)=1, 0 <x<d 일 때 f´(x) > 0, and 0 <x<d 일 때 f″(x) <0을 만족할 수 있다. 즉, 변환함수 f(x)가 위 4가지 조건을 만족할 때, 변환 S를 비선형 극좌표 변환 이미지로 정의할 수 있다.

[0077] Fig. 5 shows an example of a nonlinear polar coordinate image transformation image transformation function according to an embodiment of the present invention. Referring to Fig. 5, an example of a transformation function f(x) that satisfies the conditions of nonlinear polar coordinate image transformation is an irrational function ( ), elliptic function( ), arctan curve (4 ) etc.

[0078] The step of extracting image features (S140) can input a low-capacity image into a vision transformer to extract image features. The vision transformer used in the step of extracting image features (S140) can be a context vision transformer (CViT). The context vision transformer has a similar structure to the existing vision transformer, but differs in the input image and positional embedding. The input image of the context vision transformer is an image that has undergone log polar transformation or nonlinear log polar transformation, and can have a size of N*N pixels. The CLS token (Special Classification token) of the context vision transformer can be a feature vector of the image, i.e., a context representation.

[0079] Fig. 6 is a schematic diagram of a token configuration of a vision transformer according to an embodiment of the present invention. Referring to Fig. 6, the vision transformer may be defined as follows [Relationship 2] in which a token (H(z)) of an image patch z located at the pth horizontal and qth vertical position of a low-capacity image is defined. Specifically, when the vision transformer of the present invention is a context vision transformer and the input image of the context vision transformer is divided into n horizontal and n vertical regions, the pth horizontal and qth vertical region ( ) The token H(z) of the image patch z is as follows [Relationship 2].

[0080] [Relationship 2]

[0081]

[0082] (Here, is the feature embedding of CNN (Convolutional Neural Network) for the image, means the position embedding of z)

[0083] The vision transformer is [Relationship 2] go is set to, go is set to, go can be set to .

[0084] is a function of the horizontal position p, which represents the relative distance from the origin on the entire image. That is, all image patches with horizontal position p have the same value, regardless of their vertical position q. As the horizontal position p increases, the distance on the actual image also increases rapidly, so it can be set to an exponential function in the case of log polar coordinate image transformation. can be set using its inverse function relationship for other nonlinear polar coordinate transformations. When using log polar coordinate transformation, can be set as shown in [Mathematical Formula 4] below.

[0085] [Equation 4]

[0086]

[0087] (Here, if p=1 , if p=n am)

[0088] and is a function of the vertical position q, which means the angle in the polar coordinates of the entire image. That is, image patches with the same vertical position q, regardless of their horizontal position p, are each and It has value. and Since the vertical position q should have similar values ​​when it is 1 and when it is n, and can be set as a function as in [Mathematical Formula 5] below.

[0089] [Equation 5]

[0090]

[0091] Fig. 7 illustrates a configuration diagram of a context recognition system (1000) for a digital slide image according to an embodiment of the present invention. Referring to Fig. 7, the context recognition system (1000) for a digital slide image may include an image receiving unit (110), a range selection unit (120), a conversion unit (130), and a feature extraction unit (140).

[0092] The image receiving unit (110) can receive an original image. The image receiving unit (110) can perform the step (S110) of receiving the above-described original image.

[0093] The range selection unit (120) can select the range of context to be analyzed from the original image. The range selection unit (120) can perform the step (S120) of selecting the range of context described above.

[0094] The conversion unit (130) can convert the above context range into a low-capacity image using log polar coordinate image conversion or nonlinear polar coordinate image conversion. The conversion unit (130) can perform the step (S130) of converting into a low-capacity image as described above.

[0095] The feature extraction unit (140) can extract image features by inputting the low-capacity image into a vision transformer having tokens defined by a CNN (Convolutional Neural Network) feature embedding, a function regarding relative distance, and a function regarding polar coordinate distance. The feature extraction unit (140) can perform the step (S140) of extracting the image features described above.

[0096] While the present invention has been described in detail through representative examples above, those skilled in the art will understand that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims described below but also by all changes or modifications derived from the claims and equivalent concepts.

[0097] The present invention aims to provide a method and system for recognizing the context of a digital slide image, to which a contextual image expression method that is invariant to rotational transformation and can reflect the surrounding context is applied.

Claims

1. Step of receiving the original image; A step of selecting the scope of the context to be analyzed from the above original image; A step of converting the above context range into a low-capacity image using polar coordinate image transformation; and A step of inputting the above low-capacity image into a vision transformer to extract image characteristics; A method for context recognition of a digital slide image including:

2. In paragraph 1, The step of selecting the scope of the above context is: A method for recognizing context in a digital slide image, characterized in that a circular area having a radius of d pixels in the original image is selected as the scope of the context.

3. In paragraph 1, The step of converting to the above low-capacity image is: A method for context recognition of a digital slide image, characterized in that it comprises a step of converting the original image (x,y) into a log polar coordinate image (x´,y´).

4. In paragraph 3, The step of converting to the above low-capacity image is: A method for context recognition of a digital slide image, characterized in that it further includes a step of adjusting the center-to-periphery image reflection ratio by adding a hyperparameter to the above log polar coordinate image.

5. In paragraph 4, The step of converting to the above low-capacity image is: A method for recognizing context of a digital slide image, characterized in that it further includes a step of approximating the x-coordinate (x´) of the above log polar coordinate image through Maclaurin Series Expansion.

6. In paragraph 1, The step of converting to the above low-capacity image is: A method for context recognition of a digital slide image, characterized in that it comprises a step of converting the original image (x,y) into a nonlinear polar coordinate image (x´,y´).

7. In paragraph 6, The above nonlinear polar coordinate image is, It is defined as [Relationship 1] below, f(0)=0, f(d)=1, 0 <x<d 일 때 f´(x) > 0, and 0 <x<d 일 때 f″(x) <0을 만족하는 것을 특징으로 하는 디지털 슬라이드 이미지의 문맥 인식 방법. [Relationship 1] (Here, d represents the maximum radius of the original image, and N represents the length of one side of the transformed image) 8. In paragraph 1, The above vision transformer, A method for recognizing context of a digital slide image, characterized in that the token (H(z)) of the image patch z located at the pth horizontal and qth vertical position of the low-capacity image is defined by the following [Relationship 2]. [Relationship 2] (Here, is the feature embedding of CNN (Convolutional Neural Network) for the image, means the position embedding of z) 9. In paragraph 8, The above vision transformer, In the above [Relationship 2] go is set to, go is set to, go A method for recognizing the context of a digital slide image, characterized in that it is set to .

10. Image receiving unit for receiving the original image; A range selection unit for selecting the range of context to be analyzed from the original image above; A transformation unit that transforms the range of the above context into a low-capacity image using a log polar coordinate image transformation or a nonlinear polar coordinate image transformation; and A feature extraction unit for extracting image features by inputting the low-capacity image into a vision transformer having tokens defined by a CNN feature embedding, a function regarding relative distance, and a function regarding polar distance; A context recognition system for digital slide images containing .

Images