Lipid droplet detection device, lipid droplet detection method and program

Abstract

To appropriately extract a lipid droplet candidate without using a teacher image.SOLUTION: A lipid droplet detection device includes an original image acquisition unit, an R image creation unit, a G image creation unit, a cell nucleus image extraction unit, a foreground binary image creation unit, a background binary image creation unit, and a lipid droplet candidate extraction unit for positioning a foreground binary image and a background binary image, creating an AND image corresponding to a rank by performing an AND operation, and cutting out a square circumscribing a lipid droplet included in the AND image as a lipid droplet candidate area. The lipid droplet candidate extraction unit determines the size and the rank of the square by performing reinforcement learning. In the reinforcement learning, it is given as a reward of the reinforcement learning that which route, on an environment of the reinforcement learning with a horizontal axis set as the size of the square and with a vertical axis set as the rank, the lipid droplet detection device as an agent of the reinforcement learning passes through to make multiple lipid droplet candidate areas cut out by the lipid droplet candidate extraction unit mutually overlap more.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a lipid droplet detection device, a lipid droplet detection method, and a program. [Background technology]

[0002] Lipid droplets contained in raw pathological tissue images obtained using optical microscopes, objective lenses, cameras, etc. are complex in size, shade, shape, etc., so even experienced pathologists may find it difficult and time-consuming to determine whether or not lipid droplets are present. In addition, in formalin-fixed paraffin-embedded (FFPE) specimens that are typically prepared for pathological analysis, lipid droplets are removed during the specimen preparation process. Therefore, frozen tissues must be used to detect lipid droplets pathologically, but it is difficult to prepare and store frozen tissues in daily diagnosis. In addition, there is a need to develop technologies that can support pathologists by analyzing raw pathological tissue images that contain many complex images and by detecting lipid droplets even in FFPE.

[0003] Patent Document 1 describes a pathology diagnostic device that identifies the range of a lesion contained in an image of a diagnostic target site. Patent Document 1 also describes that even pathologists with a certain level of experience may have different findings when distinguishing between balloon-like hepatocytes and lipid droplet-containing hepatocytes contained in a liver biopsy image. In the technology described in Patent Document 1, whether or not a hepatocyte contains lipid droplets is input as correct answer data, and a verification model (learning model) is trained.

[0004] The technology described in Patent Document 1 and other documents requires a large amount of learning and training images, but when targeting rare diseases or rare lesions, it is difficult to prepare a large number of training images, making it difficult to apply methods that require a large amount of learning. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-112407 A Summary of the Invention [Problem to be solved by the invention]

[0006] In view of the above, the present invention aims to provide a lipid droplet detection device, a lipid droplet detection method, and a program that can appropriately extract lipid droplet candidates without using a teacher image. [Means for solving the problem]

[0007] According to one aspect of the present invention, there is provided an image processing apparatus including: an original image acquisition unit for acquiring an original pathological tissue image; an R image creation unit for creating an R image from the original pathological tissue image acquired by the original image acquisition unit; a G image creation unit for creating a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced, from the original pathological tissue image acquired by the original image acquisition unit; a cell nucleus image extraction unit for creating a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei contained in the original pathological tissue image, from the R image created by the R image creation unit; and a third monochrome binary image obtained by performing a process of converting black pixels contained in the second monochrome binary image to white pixels and a process of converting white pixels contained in the second monochrome binary image to black pixels according to a rank that is the number of white pixels present in a filter region formed by a plurality of pixels included in the second monochrome binary image, and a foreground binary image is created by subtracting the first monochrome binary image created by the cell nucleus image extraction unit from the third monochrome binary image. a background binary image creation unit that creates a background binary image in which the fat droplets are extracted as white regions from the G image created by the G image creation unit, and a fat droplet candidate extraction unit that creates an AND image according to the rank by performing an AND operation by aligning the foreground binary image created by the foreground binary image creation unit and the background binary image created by the background binary image creation unit, and cuts out a square circumscribing the fat droplet included in the AND image as a fat droplet candidate region, The extraction unit determines the size of the square and the rank by performing reinforcement learning, and in the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through which route in the reinforcement learning environment, the lipid droplet candidate regions cut out by the lipid droplet candidate extraction unit will overlap more with each other. This is given as a reward for the reinforcement learning, and the Q value of the reinforcement learning is updated based on the reward of the reinforcement learning.The movement start position and movement end position of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower limit and upper limit of the size of the square.

[0008] The lipid droplet detection device of one aspect of the present invention may include a discrimination unit that determines whether the lipid droplet is included in the square cut out by the lipid droplet candidate extraction unit.

[0009] In one embodiment of the lipid droplet detection device of the present invention, the foreground binary image creation unit creates the third monochrome binary image according to the rank determined by performing the reinforcement learning, and the background binary image creation unit creates the background binary image regardless of the rank determined by performing the reinforcement learning.

[0010] In the lipid droplet detection device according to one embodiment of the present invention, the foreground binary image creation unit creates the second monochrome binary image by applying an error diffusion method to the G image created by the G image creation unit, converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 3 or more white pixels to white pixels, and converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 3 white pixels to black pixels, thereby creating the third monochrome binary image of "rank 3", converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 4 or more white pixels to white pixels, and converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 4 white pixels to black pixels. the third monochrome binary image of "Rank 4" may be created by converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 5 or more white pixels to white pixels and converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 5 white pixels to black pixels, to create the third monochrome binary image of "Rank 5"; and the third monochrome binary image of "Rank 6" may be created by converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 6 or more white pixels to white pixels and converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 6 white pixels to black pixels.

[0011] In one embodiment of the lipid droplet detection device of the present invention, the start position of movement of the reinforcement learning agent in the environment may be a position in the environment where the size of the square is 2500 pixels (50 pixels x 50 pixels) and the rank is the "Rank 6", and the end position of movement of the reinforcement learning agent in the environment may be a position in the environment where the size of the square is 25 pixels (5 pixels x 5 pixels) and the rank is the "Rank 3".

[0012] In one embodiment of the lipid droplet detection device of the present invention, the position of the reinforcement learning agent in the environment at each time may correspond to the state of the reinforcement learning, and the movement of the reinforcement learning agent in the environment may correspond to the reinforcement learning action.

[0013] In the lipid droplet detection device according to one aspect of the present invention, penalty areas may be provided in three locations in the environment.

[0014] In one embodiment of the lipid droplet detection device of the present invention, the background binary image creation unit may perform adaptive binarization processing on the G image created by the G image creation unit using a 225-pixel (15 pixels x 15 pixels) average filter, apply a 9-pixel (3 pixels x 3 pixels) average filter, and perform opening processing to create the background binary image.

[0015] One aspect of the present invention includes an original image acquisition step in which the lipid droplet detection device acquires an original pathological tissue image, an R image creation step in which the lipid droplet detection device creates an R image from the original pathological tissue image acquired in the original image acquisition step, a G image creation step in which the lipid droplet detection device creates a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are emphasized, from the original pathological tissue image acquired in the original image acquisition step, a cell nucleus image extraction step in which the lipid droplet detection device creates a first monochrome binary image, which is a monochrome binary image in which cell nuclei contained in the original pathological tissue image are extracted, from the R image created in the R image creation step, and a process in which the lipid droplet detection device creates a second monochrome binary image from the G image created in the G image creation step, and converts black pixels included in the second monochrome binary image to white pixels according to a rank that is the number of white pixels present in a filter region composed of a plurality of pixels included in the second monochrome binary image, and a process in which white pixels included in the second monochrome binary image are converted to black pixels. A foreground binary image creation step in which the fat droplet detection device creates a foreground binary image by subtracting the first monochrome binary image created in the cell nucleus image extraction step from the third monochrome binary image, and a background binary image creation step in which the fat droplet detection device creates a background binary image in which the fat droplets are extracted as white regions from the G image created in the G image creation step, and a fat droplet candidate extraction step in which the fat droplet detection device aligns the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step and performs an AND operation to create an AND image according to the rank, and cuts out a square circumscribing the fat droplet included in the AND image as a fat droplet candidate region, wherein in the fat droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, and in the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, andThis is a lipid droplet detection method in which the route through which the lipid droplet detection device as an agent of the reinforcement learning passes through the reinforcement learning environment, in which the other of the horizontal axis and the vertical axis is set as the rank, which results in the plurality of lipid droplet candidate regions cut out in the lipid droplet candidate extraction step overlapping with each other more is given as a reward for the reinforcement learning, the Q value of the reinforcement learning is updated based on the reward for the reinforcement learning, and the movement start position and movement end position of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower limit and upper limit of the size of the square.

[0016] One aspect of the present invention is a method for obtaining a pathological tissue image by a computer, comprising: an original image obtaining step of obtaining an original pathological tissue image in a computer; an R image creating step of creating an R image from the original pathological tissue image obtained in the original image obtaining step; a G image creating step of creating a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced, from the original pathological tissue image obtained in the original image obtaining step; a cell nucleus image extracting step of creating a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei contained in the original pathological tissue image, from the R image created in the R image creating step; and a second monochrome binary image from the G image created in the G image creating step, and performing a process of converting black pixels included in the second monochrome binary image to white pixels and a process of converting white pixels included in the second monochrome binary image to black pixels according to a rank, which is the number of white pixels present in a filter region formed by a plurality of pixels included in the second monochrome binary image. A foreground binary image creation step of creating a foreground binary image by subtracting the first monochrome binary image created in the cell nucleus image extraction step from the third monochrome binary image, a background binary image creation step of creating a background binary image in which the fat droplets are extracted as white regions from the G image created in the G image creation step, and a fat droplet candidate extraction step of creating an AND image according to the rank by aligning the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step and performing an AND operation, and cutting out a square circumscribing the fat droplet included in the AND image as a fat droplet candidate region, wherein in the fat droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, and in the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, andA route through the reinforcement learning environment, in which the other of the horizontal axis and the vertical axis is set as the rank, along which the lipid droplet detection device as the agent of the reinforcement learning should pass so that the lipid droplet candidate regions cut out in the lipid droplet candidate extraction step overlap with each other more is given as a reward for the reinforcement learning, a Q value of the reinforcement learning is updated based on the reward for the reinforcement learning, and the movement start position and movement end position of the agent of the reinforcement learning in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower limit and upper limit of the size of the square. Effect of the Invention

[0017] According to the present invention, lipid droplet candidates can be appropriately extracted without using a training image. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing an example of a lipid droplet detection device 1 according to a first embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of weighting in the error diffusion method applied to a pixel of interest indicated by "*." [Diagram 3] 10 is a diagram showing an example of a first monochrome binary image (binary cell nucleus image) created by cell nucleus image extraction section 14. FIG. [Figure 4] 10A to 10C are diagrams showing examples of foreground binary images of “Rank 3” to “Rank 6” created by foreground binary image creation section 15. FIG. [Diagram 5] FIG. 13 is a diagram showing an example of an AND image of “Rank 3” to “Rank 6” created by the lipid droplet candidate extraction unit 17. [Figure 6] 1 is a diagram for illustrating an example of the relationship between the lipid droplets and squares cut out as lipid droplet candidate regions by the lipid droplet candidate extraction unit 17. FIG. [Figure 7] FIG. 1 is a diagram showing an example of a reinforcement learning environment defined (set) in the reinforcement learning performed by the lipid droplet candidate extraction unit 17. [Figure 8] FIG. 4 is a diagram showing an example of a discrimination result of lipid droplets by the discrimination unit 18. [Figure 9] 4 is a flowchart for explaining an example of processing executed by the lipid droplet detection device 1 of the first embodiment. [Figure 10] 1 is a scatter plot matrix showing the relationship between lipid droplet image characteristics and normal image characteristics. [Figure 11] FIG. 13 shows images including lipid droplets detected by the discrimination unit 18 and evaluated by two pathologists. [Figure 12] FIG. 13 is a diagram showing an example of AND images of "Rank 3" to "Rank 6" created by the lipid droplet candidate extraction unit 17 of the modified example of the lipid droplet detection device 1 of the first embodiment. [Figure 13] FIG. 13 is a diagram showing the average Q value for each episode in an example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, with reference to the drawings, embodiments of the lipid droplet detection device, lipid droplet detection method, and program of the present invention will be described.

[0020] First Embodiment FIG. 1 is a diagram showing an example of a lipid droplet detection device 1 according to the first embodiment. 1, the lipid droplet detection device 1 is configured by, for example, a computer having a communication interface (input / output interface), a memory, and a processor. The communication interface (input / output interface) has an interface circuit for connecting the lipid droplet detection device 1 to, for example, a camera that captures the original pathological tissue image. In another example, the lipid droplet detection device 1 may be connected via a communication interface (input / output interface) to a storage device (eg, a server, a computer, etc.) in which data of the original pathological tissue image is stored.

[0021] 1, the memory of the lipid droplet detection device 1 stores programs and various data used in the processing executed by the processor. The processor has a function as an original image acquisition unit 11, a function as an R image creation unit 12, a function as a G image creation unit 13, a function as a cell nucleus image extraction unit 14, a function as a foreground binary image creation unit 15, a function as a background binary image creation unit 16, a function as a lipid droplet candidate extraction unit 17, and a function as a discrimination unit 18.

[0022] The original image acquisition unit 11 acquires (reads) an original image (color image) of a pathological tissue obtained using, for example, an optical microscope, an objective lens, a camera, and the like. The R image creation unit 12 creates an R image from the original pathological tissue image acquired by the original image acquisition unit 11 by using, for example, a technique described in the website indicated by the URL below. https: / / potesara-tips.com / separating-colors /

[0023] The G image creation unit 13 creates a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image acquired by the original image acquisition unit 11 are emphasized, by using, for example, a technique described on the website indicated by the above URL.

[0024] The cell nucleus image extraction section 14 creates a first monochrome binary image, which is a monochrome binary image in which cell nuclei included in the original pathological tissue image are extracted from the R image created by the R image creation section 12. In detail, the cell nucleus image extraction unit 14 creates a binary dot image from the R image created by the R image creation unit 12 using pseudo-gradation based on an error diffusion method, which is a technology described, for example, in the website indicated by the URL below. https: / / zenn.dev / baroqueengine / books / a19140f2d9fc1a / viewer / 531f1d

[0025] FIG. 2 is a diagram showing an example of weights in the error diffusion method applied to a pixel of interest indicated by "*."

[0026] Next, the cell nucleus image extraction unit 14 applies a median filter (more specifically, a filter consisting of 9 pixels (3 pixels x 3 pixels)), which is a technique described, for example, in the website indicated at the URL below, to the binary point image. https: / / www.frontier.maxell.co.jp / blog / posts / 17.html

[0027] Next, the cell nucleus image extraction unit 14 performs black and white inversion processing on the image obtained by applying the median filter. Finally, the cell nucleus image extraction unit 14 performs a closing process, which is a technique described, for example, in the website indicated by the URL below, on the image obtained by the black and white inversion process, to create the above-mentioned first monochrome binary image, which is a binary cell nucleus image. https: / / www.frontier.maxell.co.jp / blog / posts / 22.html

[0028] FIG. 3 is a diagram showing an example of a first monochrome binary image (binary cell nucleus image) created by the cell nucleus image extraction section 14. As shown in FIG. In detail, Figure 3(A) shows an example of a portion (more specifically, one-sixth of the entire original pathological tissue image) of the original pathological tissue image (color image) acquired by the original image acquisition unit 11, and Figure 3(B) shows a first monochrome binary image (binary cell nucleus image) created by the cell nucleus image extraction unit 14 from the original pathological tissue image (color image) shown in Figure 3(A).

[0029] In the example shown in Figure 1, the foreground binary image creation unit 15 creates a second monochrome binary image by applying the above-mentioned error diffusion method to the G image (fat droplet enhanced image) created by the G image creation unit 13. Next, foreground binary image creation unit 15 applies a filter area consisting of a plurality of pixels (e.g., 9 pixels (3 pixels x 3 pixels)) to the second monochrome binary image. In detail, foreground binary image creation unit 15 defines the number of white pixels existing in the filter area (9 pixels (3 pixels x 3 pixels)) as a "rank." Next, foreground binary image creation unit 15 creates a third monochrome binary image by performing a process of converting black pixels included in the second monochrome binary image to white pixels and a process of converting white pixels included in the second monochrome binary image to black pixels according to the rank. In detail, foreground binary image creation unit 15 performs a process of converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where white pixels of rank or higher exist into white pixels, and also performs a process of converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where white pixels of rank lower exist into black pixels, thereby creating a third monochrome binary image.

[0030] Specifically, foreground binary image creation unit 15 creates a third monochrome binary image of "Rank 3" by converting all pixels in a filter area (9 pixels (3 pixels x 3 pixels)) where three or more white pixels exist to white pixels, and by converting all pixels in a filter area (9 pixels (3 pixels x 3 pixels)) where less than three white pixels exist to black pixels. In addition, foreground binary image creation unit 15 creates a third monochrome binary image of "Rank 4" by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where four or more white pixels exist to white pixels, and by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where less than four white pixels exist to black pixels. Furthermore, foreground binary image creation unit 15 creates a third monochrome binary image of "Rank 5" by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where 5 or more white pixels exist to white pixels, and by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where less than 5 white pixels exist to black pixels. In addition, foreground binary image creation unit 15 creates a third monochrome binary image of "Rank 6" by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where six or more white pixels exist to white pixels, and by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where less than six white pixels exist to black pixels.

[0031] Furthermore, the foreground binary image creating section 15 creates a foreground binary image by subtracting the first monochrome binary image (see FIG. 3(B)) created by the cell nucleus image extracting section 14 from the third monochrome binary image. Specifically, the foreground binary image creation unit 15 creates a foreground binary image of "Rank 3" by subtracting the first monochrome binary image (see Figure 3 (B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "Rank 3". In addition, the foreground binary image creation unit 15 creates a foreground binary image of "Rank 4" by subtracting the first monochrome binary image (see Figure 3 (B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "Rank 4". Furthermore, the foreground binary image creation unit 15 creates a foreground binary image of "Rank 5" by subtracting the first monochrome binary image (see Figure 3 (B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "Rank 5". In addition, the foreground binary image creation unit 15 creates a foreground binary image of "Rank 6" by subtracting the first monochrome binary image (see Figure 3 (B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "Rank 6".

[0032] FIG. 4 is a diagram showing examples of foreground binary images of “Rank 3” to “Rank 6” created by foreground binary image creation section 15. As shown in FIG. In detail, Figure 4(A) shows a foreground binary image of "Rank 6" created by foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 4(B) shows a foreground binary image of "Rank 5" created by foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 4(C) shows a foreground binary image of "Rank 4" created by foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A), and Figure 4(D) shows a foreground binary image of "Rank 3" created by foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A).

[0033] In the example shown in FIG. 1, the background binary image creating unit 16 creates a background binary image by extracting lipid droplets as white regions from the G image (lipid droplet enhanced image) created by the G image creating unit 13. In detail, the background binary image creation unit 16 applies an average filter (averaging filter) of 225 pixels (15 pixels x 15 pixels) to the G image created by the G image creation unit 13 by using the technology described on the website indicated at the URL below, and performs adaptive binary processing. https: / / www.frontier.maxell.co.jp / blog / posts / 14.html https: / / www.frontier.maxell.co.jp / blog / posts / 19.html

[0034] Next, the background binary image creation unit 16 applies an average value filter (averaging filter) of 9 pixels (3 pixels x 3 pixels) and performs an opening process, which is a technique described, for example, in the website indicated by the URL below, to create a background binary image in which fat droplets are extracted as white regions. https: / / www.frontier.maxell.co.jp / blog / posts / 22.html

[0035] In the example shown in FIG. 1, the fat droplet candidate extraction unit 17 aligns the foreground binary image (see FIG. 4) created by the foreground binary image creation unit 15 with the background binary image created by the background binary image creation unit 16 and performs an AND operation to create an AND image according to the rank ("Rank 3" to "Rank 6"). That is, when the brightness of a pixel on the background binary image at the same coordinates (same position) as a pixel (white pixel) with a brightness of 255 on the foreground binary image is 255, the brightness of the pixel on the AND image at that coordinate (position) becomes 255. The brightness of a pixel on the AND image at the same coordinates (same position) as a pixel on the foreground binary image with a brightness of less than 255 becomes zero. That is, a pixel on the AND image at the same coordinates (same position) as a pixel on the foreground binary image with a brightness of less than 255 becomes a black pixel. In addition, the brightness of a pixel on the AND image at the same coordinates (same position) as a pixel on the background binary image with a brightness of less than 255 also becomes zero. In other words, pixels in the AND image that have the same coordinates (same position) as pixels in the foreground binary image whose luminance is less than 255 will also become black pixels.

[0036] FIG. 5 is a diagram showing an example of an AND image of “Rank 3” to “Rank 6” created by the lipid droplet candidate extraction unit 17. In detail, Figure 5(A) shows an AND image of "Rank 6" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 5(B) shows an AND image of "Rank 5" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 5(C) shows an AND image of "Rank 4" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A), and Figure 5(D) shows an AND image of "Rank 3" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A).

[0037] Furthermore, the lipid droplet candidate extraction unit 17 cuts out a square circumscribing the lipid droplet included in the AND image (see FIG. 5) as a lipid droplet candidate region.

[0038] FIG. 6 is a diagram for explaining an example of the relationship between the squares cut out as lipid droplet candidate regions by the lipid droplet candidate extraction unit 17 and lipid droplets. In the example shown in Fig. 6, the square located in the upper left of Fig. 6 (the square indicated by "The previously extracted region") is first cut out by the lipid droplet candidate extraction unit 17, and then the square located in the lower right of Fig. 6 (the square indicated by "The newly extracted region") is cut out by the lipid droplet candidate extraction unit 17. In addition, the lower right portion of the square located in the upper left of Fig. 6 and the upper left portion of the square located in the lower right of Fig. 6 form an overlapping portion (the portion indicated by "The overlaps region") that overlaps each other. In the example shown in FIG. 6, the square located in the upper left of FIG. 6 circumscribes the largest lipid droplet in the center of FIG. 6, and the square located in the lower right of FIG. 6 circumscribes the second largest lipid droplet on the right side of FIG. 6. The square located in the upper left of Figure 6 includes the entire largest lipid droplet in the center of Figure 6, the entire smallest lipid droplet on the left side of Figure 6, and part of the second largest lipid droplet on the right side of Figure 6. The square located at the bottom right of Figure 6 includes the entire second largest lipid droplet on the right side of Figure 6 and part of the largest lipid droplet in the center of Figure 6.

[0039] In the example shown in FIG. 1, the lipid droplet candidate extraction unit 17 determines the size of the square (see FIG. 6) and the above-mentioned rank ("Rank 3" to "Rank 6") by performing reinforcement learning. In detail, in the example shown in FIG. 1 (an example of the lipid droplet detection device 1 of the first embodiment), the lipid droplet candidate extraction unit 17 performs Q-learning as reinforcement learning.

[0040] FIG. 7 is a diagram showing an example of a reinforcement learning environment defined (set) in the reinforcement learning performed by the lipid droplet candidate extraction unit 17. In the example shown in Fig. 7, the horizontal axis of the reinforcement learning environment is set as the size of the square (see Fig. 6) (more specifically, the area of ​​the square (number of pixels)), and the vertical axis of the reinforcement learning environment is set as the above-mentioned ranks ("Rank 3" to "Rank 6"). In Fig. 7, the numbers "0" to "39" indicate indexes. In the example shown in Figure 7, in the reinforcement learning performed by the lipid droplet candidate extraction unit 17, the reward for the reinforcement learning is given based on which route in the reinforcement learning environment shown in Figure 7 the lipid droplet detection device 1, as the reinforcement learning agent, should take to ensure that the multiple lipid droplet candidate regions (squares (see Figure 6)) cut out by the lipid droplet candidate extraction unit 17 overlap with each other more.

[0041] In the example shown in FIG. 7, the movement start position of the reinforcement learning agent (fat droplet detection device 1) in the reinforcement learning environment is set to index "9" in the upper right corner of the reinforcement learning environment shown in FIG. 7. In other words, the movement start position of the reinforcement learning agent (fat droplet detection device 1) in the reinforcement learning environment is a position in the reinforcement learning environment where the size of the square is 2500 pixels (50 pixels x 50 pixels) and the rank is "Rank 6". The movement start position of the reinforcement learning agent (fat droplet detection device 1) in the direction of the horizontal axis of the reinforcement learning environment (left and right direction in FIG. 7) corresponds to the upper limit of the size of the square (2500 pixels (50 pixels x 50 pixels)). In the example shown in FIG. 7, the movement end position (goal) of the reinforcement learning agent (fat droplet detection device 1) is set to the index "30" in the lower left corner of the reinforcement learning environment. In other words, the movement end position of the reinforcement learning agent (fat droplet detection device 1) in the reinforcement learning environment is a position in the reinforcement learning environment where the size of the square is 25 pixels (5 pixels x 5 pixels) and the rank is "Rank 3". The movement end position of the reinforcement learning agent (fat droplet detection device 1) in the direction of the horizontal axis of the reinforcement learning environment (left and right direction in FIG. 7) corresponds to the lower limit of the square size (25 pixels (5 pixels x 5 pixels)). Furthermore, in the example shown in FIG. 7, as shown by the arrows in FIG. 7, the reinforcement learning agent (fat droplet detection device 1) is set to be movable leftward or downward in the reinforcement learning environment in FIG. 7. The position of the reinforcement learning agent (fat droplet detection device 1) in the reinforcement learning environment at each time t is determined by the "state S t The movement of the reinforcement learning agent (the lipid droplet detection device 1) in the reinforcement learning environment (the leftward movement in FIG. 7 or the downward movement in FIG. 7) corresponds to "action a" in the reinforcement learning.

[0042] In the example shown in FIG. 7, the size of the square on the horizontal axis of the reinforcement learning environment (area of ​​the square (number of pixels)) and the index have the relationship shown in the following formula. In other words, the size of the square is the square obtained by dividing the index by 10, multiplying the remainder by 5, adding 5, and squaring the result. Specifically, the square size "1600" is the square obtained by dividing the index "17" by 10, multiplying the remainder by 5, adding 5, and squaring the result. Square size = (5 × (index%10) + 5) 2

[0043] In the example shown in Fig. 7, the rank on the vertical axis of the reinforcement learning environment and the index have the relationship shown in the following formula. In other words, the rank is calculated by dividing the index by 10, discarding the decimal point, and subtracting the result from 6. To be more specific, the rank is calculated by dividing the index "17" by 10, discarding the decimal point, and subtracting the result from 6. Rank=6-index / 10

[0044] As described above, in the example shown in FIG. 7, in the reinforcement learning performed by the lipid droplet candidate extraction unit 17, the route in the reinforcement learning environment through which the lipid droplet detection device 1 as an agent of the reinforcement learning passes through, which causes the plurality of lipid droplet candidate regions (squares (see FIG. 6)) cut out by the lipid droplet candidate extraction unit 17 to overlap with each other more, is given as a reward for the reinforcement learning. Therefore, as shown in FIG. 6, when the largest lipid droplet in the center and the second largest lipid droplet on the right side are close to each other, not only does the square located in the upper left of FIG. 6 include the largest lipid droplet in the center, but also includes a part of the second largest lipid droplet on the right side. In other words, the second largest lipid droplet in FIG. 6 is not only cut out (extracted) by the square located in the lower right of FIG. 6, but also cut out (extracted) with a part overlapping with the square located in the upper left of FIG. 6. In detail, in the examples shown in Figures 6 and 7, the reward for reinforcement learning is set to the percentage of the area of ​​the square located in the lower right of Figure 6 (the square shown in "The newly extracted region") that overlaps with the square located in the upper left of Figure 6 that was initially extracted (the square shown in "The previously extracted region"), and the area of ​​the square shown in "The overlaps region" that overlaps with the square located in the lower right of Figure 6 (the square shown in "The newly extracted region").

[0045] In the example shown in Fig. 1, the Q value of the reinforcement learning is updated based on the reward of the reinforcement learning. In detail, the Q value of the reinforcement learning is updated based on the reward of the reinforcement learning, etc., using the following formula.

[0046]

number

[0047] In the above equation, r represents the reward, α represents the learning rate, and γ represents the discount rate. In the example shown in Figure 1, the ε-greedy method (ε: probability of randomly selecting an action) is used to select “action a” in reinforcement learning.

[0048] In the example shown in Fig. 7, penalty areas are set in three locations in the reinforcement learning environment. Specifically, index "8", index "29", and index "38" are set as penalty areas. 7, reinforcement learning actions are performed a maximum of 12 times in one episode. In the reinforcement learning actions, when the position of the reinforcement learning agent (fat droplet detection device 1) goes outside the reinforcement learning environment or enters a penalty area, the reinforcement learning reward is set to "-1" and the position of the reinforcement learning agent (fat droplet detection device 1) is returned to the movement start position (index "9"). 1 and 7, the above-mentioned episode is repeated multiple times to repeat the reinforcement learning. Furthermore, every time the reinforcement learning is performed, pixels having edges with low luminance values ​​(small differences in luminance values) are deleted from the background binary image by, for example, the background binary image creation unit 16 or the like.

[0049] In the example shown in FIG. 1, the discrimination unit 18 discriminates whether or not a lipid droplet (see FIG. 6) is included in a square (see FIG. 6) cut out by the lipid droplet candidate extraction unit 17. In detail, the discrimination unit 18 discriminates whether or not a lipid droplet is included in a square (see FIG. 8, for example) area on the pathological tissue original image (pathological tissue original image acquired by the original image acquisition unit 11) corresponding to the lipid droplet candidate area (square) cut out by the lipid droplet candidate extraction unit 17 by using a model obtained by performing learning using teacher data, which is a data set of a pathological tissue original image for learning and a label indicating whether or not lipid droplets are included in the pathological tissue original image for learning. Specifically, the discrimination unit 18 detects lipid droplets from the square (lipid droplet candidate area) cut out by the lipid droplet candidate extraction unit 17 using a random forest with parameters optimized by grid search.

[0050] FIG. 8 is a diagram showing an example of the discrimination result of the lipid droplet by the discrimination unit 18. In detail, FIG. 8(A) shows the discrimination result by the discrimination unit 18 as to whether or not lipid droplets are included in the square cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 6" (see FIG. 5(A)). Specifically, FIG. 8(A) shows the lipid droplets detected by the discrimination unit 18 from the square (lipid droplet candidate region) cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 6" (see FIG. 5(A)) surrounded by a square. FIG. 8(B) shows the discrimination result by the discrimination unit 18 as to whether or not lipid droplets are included in the square cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 5" (see FIG. 5(B)). Specifically, FIG. 8(B) shows the lipid droplets detected by the discrimination unit 18 from the square (lipid droplet candidate region) cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 5" (see FIG. 5(B)) surrounded by a square. FIG. 8(C) shows the result of discrimination by the discrimination unit 18 as to whether or not fat droplets are included in the square cut out by the fat droplet candidate extraction unit 17 from the AND image of "rank 4" (see FIG. 5(C)). Specifically, FIG. 8(C) shows the fat droplets detected by the discrimination unit 18 from the square (fat droplet candidate region) cut out by the fat droplet candidate extraction unit 17 from the AND image of "rank 4" (see FIG. 5(C)) surrounded by a square. FIG. 8(D) shows the result of discrimination by the discrimination unit 18 as to whether or not fat droplets are included in the square cut out by the fat droplet candidate extraction unit 17 from the AND image of "rank 3" (see FIG. 5(D)). Specifically, FIG. 8(D) shows the fat droplets detected by the discrimination unit 18 from the square (fat droplet candidate region) cut out by the fat droplet candidate extraction unit 17 from the AND image of "rank 3" (see FIG. 5(D)) surrounded by a square. FIG. 8(E) shows the overall discrimination result by the discrimination unit 18. Specifically, FIG. 8(E) shows lipid droplets detected by the discrimination unit 18 from among all the squares (lipid droplet candidate regions) cut out by the lipid droplet candidate extraction unit 17, surrounded by a rectangle.

[0051] 1, foreground binary image creation unit 15 creates a third monochrome binary image according to the rank determined by performing reinforcement learning, while background binary image creation unit 16 creates a background binary image regardless of the rank determined by performing reinforcement learning.

[0052] FIG. 9 is a flowchart for explaining an example of processing executed by the lipid droplet detection device 1 of the first embodiment. In the example shown in FIG. 9, in step S1, the original image acquisition unit 11 acquires an original pathological tissue image (color image). In step S2, the R image creation unit 12 creates an R image from the original pathological tissue image acquired in step S1. In step S3, the G image creating unit 13 creates a G image (fat droplet enhanced image) from the original pathological tissue image acquired in step S1. In step S4, the cell nucleus image extraction unit 14 extracts cell nuclei from the R image created in step S2, and creates a first monochrome binary image. In step S5, foreground binary image creation unit 15 creates foreground binary images of "Rank 3" to "Rank 6". In step S6, the background binary image creation unit 16 creates a background binary image from the G image (fat droplet enhanced image) created in step S3. In step S7, the lipid droplet candidate extraction unit 17 aligns the foreground binary images of "Rank 3" to "Rank 6" created in step S5 with the background binary image created in step S6 to create an AND image of "Rank 3" to "Rank 6", and cuts out squares circumscribing lipid droplets included in the AND image as lipid droplet candidate regions. In addition, the lipid droplet candidate extraction unit 17 determines the size and rank ("Rank 3" to "Rank 6") of the squares by performing reinforcement learning. In step S8, the discriminator 18 discriminates whether or not fat droplets are included in the square cut out in step S7. In step S9, the lipid droplet detection device 1 outputs a lipid droplet-detected image, which is an image determined to include lipid droplets in step S8.

[0053] [Example] The inventors conducted experiments (such as evaluation of the lipid droplet detection device 1 of the first embodiment) using 10 human liver tissue images (five normal images (original pathological tissue images not containing lipid droplets) and five lipid droplet images (original pathological tissue images containing lipid droplets)) taken at Oita University School of Medicine. Three lipid droplet images and two normal images were used as learning data (teaching data) used for training the discrimination unit 18. The discrimination unit 18 was evaluated using the remaining five images (three normal images and two lipid droplet images). In the experiments, two pathologists with more than 10 years of clinical experience independently evaluated the images in which lipid droplets were detected.

[0054] (1) Experiment 1 The inventors compared the f-values ​​of k-nearest neighbors, support vector machines (SVM) and random forests (RF) using 10-fold cross-validation. Table 1 shows the comparison results of f-values ​​using 10-fold cross-validation of k-nearest neighbors, support vector machines and random forests as classifiers. In detail, each f-value in Table 1 is the average score for each image.

[0055] [Table 1]

[0056] The inventors used the area ratio p defined by the following formula as a feature used for classification. In the formula, a represents the area of ​​the binarized object (fat droplet) (not the "action" of the reinforcement learning described above), and D represents the area of ​​the detection region (a square circumscribing the fat droplet).

[0057]

number

[0058] In addition, the inventors used symmetry q defined by the following formula as a feature used for classification: In the formula, s represents the area of ​​overlap between the binarized object (fat droplet) and the binarized object (fat droplet) flipped left and right.

[0059]

number

[0060] Furthermore, the inventors used the contrast ratio c defined by the following formula as a feature used for classification: In the formula below, r (not the "reward" of the reinforcement learning described above) represents the average brightness value of the R image in the detection region (the square circumscribing the lipid droplet), and g represents the average brightness value of the G image in the detection region (the square circumscribing the lipid droplet).

[0061]

number

[0062] In addition, the inventors used a variance ratio v defined by the following formula as a feature used for classification.

[0063]

number

[0064] (2) Experiment 2 Furthermore, the inventors performed a dimensionality reduction of the features using principal component analysis. Two pathologists also evaluated the degree of lipid droplet detection. Table 2 shows the effect of dimensionality reduction of the features using principal component analysis. The execution time of this method was an average of 481.8 [sec] per image using Google Colaboratory.

[0065] [Table 2]

[0066] The inventors provided 160 examples from a total of 200 data sets consisting of normal images (images of non-fat droplets that are not fat droplets) and fat droplet images (the ratio of normal images to fat droplet images is 1:1) to the learning of each classifier. However, the detected fat droplet images do not necessarily show a typical circular shape, and fat droplet images may be connected to normal images or may be in contact with each other. In addition, the structure of non-fat droplets such as vascular cavities other than fat droplets may be similar to the structure of fat droplets. For this reason, it is thought that the boundary separating the features of fat droplet images and the features of normal images spans both.

[0067] Figure 10 is a scatter plot matrix showing the relationship between the characteristics of lipid droplet images (shown as "Lipid droplet" in Figure 10) and the characteristics of normal images (shown as "Normal" in Figure 10). The present inventors performed a performance evaluation of the classifier using an f-value, which decreases when there are an extremely large number of false positives or false negatives. Furthermore, as a result of the pathologist's evaluation, the classifier using random forest showed the fewest false positives and false negatives and the highest f-value compared to other classifiers. This is thought to be because the classifier using random forest was able to detect lipid droplets from lipid droplet candidates by drawing a boundary line perpendicular to the axis in the feature space compared to other classifiers. This indicates that there is a threshold value that can simply separate the distribution of lipid droplet images from the distribution of normal images to a certain extent. The characteristics of lipid droplets detected by the discrimination unit 18 were high area ratio and symmetry, and low contrast ratio and variance ratio.

[0068] FIG. 11 shows images including lipid droplets detected by the discrimination unit 18 and evaluated by two pathologists. In detail, FIG. 11(A) shows a part (one-sixth of the entire original pathological tissue image) of the same pathological tissue original image (color image) as FIG. 3(A). FIG. 11(B) shows the discrimination result by the discrimination unit 18 for the original pathological tissue image shown in FIG. 11(A). In detail, FIG. 11(B) shows an image output as a lipid droplet detection image from the lipid droplet detection device 1 after cutting out a square (lipid droplet candidate region) by the lipid droplet candidate extraction unit 17 and discriminating by the discrimination unit 18 for the original pathological tissue image shown in FIG. 11(A). As a result of two pathologists evaluating the area surrounded by the square in the image shown in Fig. 11(B), it was found that the square area contained sufficient lipid droplets that should be detected by the lipid droplet detection device 1. In reality, pathologists do not make diagnoses based on lipid droplets alone, but also consider other morphological information, so the lipid droplet detection image output from the lipid droplet detection device 1 is considered to be clinically usable.

[0069] As shown in Table 2, as a result of dimensionality reduction using principal component analysis, there was no difference in the evaluation by the mixing matrix. However, among the lipid droplet images that could not be sufficiently separated from normal images, some could not be identified by dimensionality reduction (i.e., some were not detected as lipid droplets by the discrimination unit 18). Therefore, in the lipid droplet detection device 1 of the first embodiment, classification is performed using a simple small number of features that do not require dimensionality reduction, but in the future, in order to further improve generalization performance and accuracy, it is considered necessary to separate lipid droplet images detected by the discrimination unit 18 from normal images, add features such as circularity, and increase the number of training data in the modified example of the lipid droplet detection device 1 of the first embodiment.

[0070] Adaptive binarization is an extremely powerful process that can extract even faint images because a binarization threshold is set for each pixel, but the multiple lipid droplet images contained in the original pathological tissue image each have differences in size, shade, etc. For this reason, adaptive binarization using only the same filter size does not necessarily allow sufficient separation of the boundary between lipid droplets and the surrounding area. However, if the lipid droplet image can be binarized using an appropriate filter size according to its size, the true boundary is believed to be near the boundary obtained by adaptive binarization. Therefore, in the foreground binary image creation unit 15 of the lipid droplet detection device 1 of the first embodiment, a binary point image (second monochrome binary image) obtained by applying an error diffusion method to the G image obtained from the original pathological tissue image is used to express the shading information of the luminance value of the G image as a density of dots (for example, a high rank indicates a white and high density G image, and a low rank indicates a light density part), and the lipid droplet image of each rank seen in the foreground binary image and the boundary around it (the size of the lipid droplet) are captured using shading. If it is a common part between the foreground binary image and the background binary image of each rank obtained by the adaptive binarization process, it may be a boundary according to the size of the lipid droplet image. In the lipid droplet detection device 1 of the first embodiment, even for lipid droplets that are difficult to separate from the surroundings by only the adaptive binarization process, the lipid droplet candidate extraction unit 17 cuts out the lipid droplet candidate region (square) and the discrimination unit 18 discriminates, so that lipid droplets can be detected.

[0071] In the lipid droplet detection device 1 of the first embodiment, the overlap of lipid droplet candidate regions cut out by the lipid droplet candidate extraction unit 17 in order to cover all lipid droplet candidates contained in the original pathological tissue image is given as a reward for reinforcement learning. However, when only the overlap of lipid droplet candidate regions cut out by the lipid droplet candidate extraction unit 17 is evaluated, there is a risk that images (normal images) of things other than lipid droplets (e.g., vascular cavities, etc.) and noise present around the lipid droplet images may be included in the lipid droplet candidate regions. Therefore, in a modified example of the lipid droplet detection device 1 of the first embodiment, it is determined that weak edges observed in the lipid droplet enhanced image (G image) may be noise (even a faint lipid droplet edge has the same strength as a cell nucleus edge), and the lipid droplet candidate regions cut out by the lipid droplet candidate extraction unit 17 are controlled by using edge information for noise removal.

[0072] FIG. 12 is a diagram showing an example of AND images of "Rank 3" to "Rank 6" created by the lipid droplet candidate extraction unit 17 of the modified example of the lipid droplet detection device 1 of the first embodiment. In detail, Figure 12(A) shows an AND image of "rank 6" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment, Figure 12(B) shows an AND image of "rank 5" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment, Figure 12(C) shows an AND image of "rank 4" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment, and Figure 12(D) shows an AND image of "rank 3" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment. In the example shown in Fig. 12, the lipid droplet candidate extraction unit 17 of the modified example of the lipid droplet detection device 1 of the first embodiment removed the bottom 10% of the edge intensity as noise. As a result, as shown in Fig. 12, small normal image parts such as cell nuclei and gaps near the lipid droplet image are not included in the lipid droplet candidate region cut out by the lipid droplet candidate extraction unit 17, and noise removal using edges was effective in extracting lipid droplet candidates.

[0073] As described above, in the example shown in FIG. 1 (one example of the lipid droplet detection device 1 of the first embodiment), the lipid droplet candidate extraction unit 17 performs Q-learning as reinforcement learning. In the Q-learning performed by the lipid droplet candidate extraction unit 17, the size of the lipid droplet image corresponds to the size of the square (lipid droplet candidate region) cut out by the lipid droplet candidate extraction unit 17, and the shading of the lipid droplet image corresponds to the rank ("Rank 3" to "Rank 6"), and each of the lipid droplet images having different sizes and shading is captured by combining the size and rank of the square (lipid droplet candidate region). In other words, the lipid droplet candidate extraction unit 17 can appropriately cut out the lipid droplet candidate region (square) including each of the lipid droplet images having different sizes by changing the size of the square (lipid droplet candidate region) by performing Q-learning. In addition, the lipid droplet candidate extraction unit 17 can appropriately cut out the lipid droplet candidate region (square) including each of the lipid droplet images having different shading according to the rank (shading of the lipid droplet image) determined by performing Q-learning. In the example shown in FIG. 7 described above, the downward movement of the agent (fat droplet detection device 1) on the environment shown in FIG. 7 corresponds to a lowering of the rank (for example, the size of the object (fat droplet) on the foreground binary image shown in FIG. 4 increases). The left-right position of the agent (fat droplet detection device 1) on the environment shown in FIG. 7 corresponds to the size of the fat droplet candidate region (square) cut out by the fat droplet candidate extraction unit 17. Each time the agent (fat droplet detection device 1) moves leftward on the environment shown in FIG. 7, the size of the fat droplet candidate region (square) cut out by the fat droplet candidate extraction unit 17 is determined. When the agent (fat droplet detection device 1) moves leftward multiple times in succession on the environment shown in FIG. 7 without moving downward on the environment shown in FIG. 7, the sizes of multiple fat droplet candidate regions (square) are determined (set) for one rank.

[0074] The size range (upper and lower limits) of the lipid droplet candidate region (square) corresponding to each rank is determined by the agent (lipid droplet detection device 1) moving from the movement start position to the movement end position (goal) in the environment shown in Fig. 7. The size range of the lipid droplet candidate region (square) corresponding to each rank can also be determined by using the index range at the same rank. However, in the example shown in Fig. 7 where the penalty area is set at index "8", the agent (fat droplet detection device 1) cannot move leftward in the environment shown in Fig. 7 at rank "6", so the size of the fat droplet candidate region (square) at rank "6" cut out by the fat droplet candidate extraction unit 17 is determined (set) to be "2500" or more. As a result, small-sized fat droplets are not detected at rank "6". In this way, by repeating the interaction between the environment and the agent, the lipid droplet detection device 1 of the first embodiment is able to learn according to the image. This is thought to be because even if there is no special design for lipid droplet extraction, "emergence" occurs from simple interactions and is brought to the surface.

[0075] Although the lipid droplet candidate extraction unit 17 can extract lipid droplet candidate regions (squares) without setting penalty areas, in the example shown in Fig. 7, penalty areas are set in three places in the environment shown in Fig. 7 for the purpose of shortening the learning time and avoiding instability of learning. In the early stage of learning, the agent often enters the penalty areas, but little by little the agent begins to take actions to avoid the penalty areas and reaches the goal.

[0076] FIG. 13 is a diagram showing the average Q value for each episode in the examples. In reinforcement learning, the agent that takes the optimal action in each state (index) is adopted, so that the average Q value converges to a constant value of the image and stabilizes after 200 episodes. The parameters used in the embodiment are ε=0.1, α=0.2, and γ=0.9, and the number of learnings is 100 in consideration of processing time. In the example shown in FIG. 7, the agent's action is only to move leftward or downward in FIG. 7, so the agent does not pass through the environmental indexes "0" to "7" and "39" shown in FIG. 7, and the environmental indexes "0" to "7" and "39" shown in FIG. 7 are missing. When the original pathological tissue image (color image) shown in FIG. 3(A) is acquired by the original image acquisition unit 11, the images corresponding to the indexes "0" to "7" shown in FIG. 7 become noise representing a part of the lipid droplet image seen in the foreground binary image of rank "6". Therefore, in the example shown in FIG. 7, the images corresponding to the indexes "0" to "7" shown in FIG. 7 are excluded from the detection target area. On the other hand, lipid droplets can be detected if they pass index "39." However, since it is not possible to classify them according to rank, in the example shown in Figure 7, the image corresponding to index "39" shown in Figure 7 was also excluded from the detection target area. In the embodiment, it was possible to detect lipid droplets contained in original pathological tissue images (color images) by performing reinforcement learning and using a small amount of data, without using a large amount of data as in deep learning.

[0077] As described above, in the lipid droplet detection device 1 of the first embodiment, it is possible to detect lipid droplet images seen in original pathological tissue images in a hierarchical manner by optimizing the combination of size and rank of lipid droplet candidate regions (squares) using reinforcement learning.

[0078] As mentioned above, there is a need for the development of technology that can analyze original pathological tissue images containing complex images and support pathologists. According to the lipid droplet detection device 1 of the first embodiment, in order to determine whether or not lipid droplets are included in the image shown in Fig. 11(A), for example, a pathologist does not need to check the presence or absence of lipid droplets throughout the entire image shown in Fig. 11(A), but only needs to check whether or not lipid droplets are present within the multiple squares shown in Fig. 11(B). In other words, according to the lipid droplet detection device 1 of the first embodiment, the burden on the pathologist can be significantly reduced, and the pathologist can be supported.

[0079] <Second embodiment> The lipid droplet detection device 1 of the second embodiment is configured similarly to the lipid droplet detection device 1 of the first embodiment shown in FIG. 1, except for the points described below.

[0080] As described above, in the example shown in Figure 1 (an example of the lipid droplet detection device 1 of the first embodiment), the lipid droplet candidate extraction unit 17 determines the size and rank ("Rank 3" to "Rank 6") of the lipid droplet candidate region (square) by performing Q-learning as reinforcement learning. On the other hand, in an example of the lipid droplet detection device 1 of the second embodiment, the lipid droplet candidate extraction unit 17 determines the size and rank ("Rank 3" to "Rank 6") of the lipid droplet candidate region (square) by performing reinforcement learning other than Q-learning (e.g., Sarsa or Monte Carlo method).

[0081] As described above, the lipid droplet detection device, lipid droplet detection method, and program of the present invention have been described with reference to the drawings, but the lipid droplet detection device, lipid droplet detection method, and program of the present invention are not limited to the above-mentioned embodiments, and can be appropriately modified within the scope of the present invention. The configurations of each example of the above-mentioned embodiments may be appropriately combined. In each of the above-mentioned embodiments, the processing performed in the lipid droplet detection device 1 has been described as software processing performed by executing a program, but the processing performed in the lipid droplet detection device 1 may be processing performed by hardware. Alternatively, the processing performed in the lipid droplet detection device 1 may be processing that combines both software and hardware. [Explanation of symbols]

[0082] 1... lipid droplet detection device, 11... original image acquisition unit, 12... R image creation unit, 13... G image creation unit, 14... cell nucleus image extraction unit, 15... foreground binary image creation unit, 16... background binary image creation unit, 17... lipid droplet candidate extraction unit, 18... discrimination unit

Claims

1. an original image acquisition unit for acquiring an original pathological tissue image; an R image creation unit that creates an R image from the original pathological tissue image acquired by the original image acquisition unit; A G image creation unit that creates a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the pathological tissue original image acquired by the original image acquisition unit are enhanced, a cell nucleus image extraction unit that creates a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei included in the original pathological tissue image, from the R image created by the R image creation unit; a foreground binary image creation unit that creates a second monochrome binary image from the G image created by the G image creation unit, creates a third monochrome binary image by performing a process of converting black pixels included in the second monochrome binary image to white pixels and a process of converting white pixels included in the second monochrome binary image to black pixels in accordance with a rank that is the number of white pixels present in a filter area formed by a plurality of pixels included in the second monochrome binary image, and creates a foreground binary image by subtracting the first monochrome binary image created by the cell nucleus image extraction unit from the third monochrome binary image; A background binary image creation unit that creates a background binary image in which the fat droplets are extracted as white regions from the G image created by the G image creation unit; A fat droplet detection device comprising: a fat droplet candidate extraction unit that performs an AND operation by aligning the foreground binary image created by the foreground binary image creation unit and the background binary image created by the background binary image creation unit to create an AND image according to the rank; and cuts out a square circumscribing the fat droplet included in the AND image as a fat droplet candidate region. The lipid droplet candidate extraction unit determines the size of the square and the rank by performing reinforcement learning, In the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through which route in the reinforcement learning environment, the lipid droplet candidate regions cut out by the lipid droplet candidate extraction unit will overlap more with each other. The Q value of the reinforcement learning is updated based on the reward of the reinforcement learning. A lipid droplet detection device, wherein the start and end positions of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower and upper limits of the size of the square.

2. The lipid droplet detection device according to claim 1 , further comprising a discrimination unit that determines whether the lipid droplet is included in the square cut out by the lipid droplet candidate extraction unit.

3. the foreground binary image creation unit creates the third monochrome binary image according to the rank determined by performing the reinforcement learning; The lipid droplet detection device according to claim 1 or 2, wherein the background binary image creation unit creates the background binary image regardless of the rank determined by performing the reinforcement learning.

4. The foreground binary image creation unit creating the second monochrome binary image by applying an error diffusion method to the G image created by the G image creation unit; creating the third monochrome binary image of "Rank 3" by converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having three or more white pixels into white pixels, and converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having less than three white pixels into black pixels; creating the third monochrome binary image of "Rank 4" by converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having four or more white pixels into white pixels, and converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having less than four white pixels into black pixels; creating the third monochrome binary image of "Rank 5" by converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having five or more white pixels into white pixels, and converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having less than five white pixels into black pixels; All pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 6 or more white pixels are converted to white pixels, and all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 6 white pixels are converted to black pixels. The third monochrome binary image of "rank 6" is created by converting the pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 6 white pixels to black pixels. The fat droplet detection device according to claim 3.

5. the movement start position of the reinforcement learning agent in the environment is a position in the environment where the size of the square is 2500 pixels (50 pixels x 50 pixels) and the rank is the "Rank 6", The end position of the movement of the reinforcement learning agent in the environment is a position in the environment where the size of the square is 25 pixels (5 pixels x 5 pixels) and the rank is the "rank 3". The lipid droplet detection device according to claim 4.

6. the position of the agent in the environment at each time corresponds to a state of the reinforcement learning; The lipid droplet detection device according to claim 5 , wherein the movement of the reinforcement learning agent through the environment corresponds to the reinforcement learning behavior.

7. The lipid droplet detection device according to claim 6 , wherein penalty areas are provided in three locations in the environment.

8. The background binary image creation unit performs adaptive binarization processing on the G image created by the G image creation unit using a 225-pixel (15 pixels x 15 pixels) average filter, applies a 9-pixel (3 pixels x 3 pixels) average filter, and performs opening processing to create the background binary image. The fat droplet detection device of claim 1 or 2.

9. An original image acquisition step in which the lipid droplet detection device acquires an original pathological tissue image; An R image creation step in which the lipid droplet detection device creates an R image from the pathological tissue original image acquired in the original image acquisition step; A G image creation step in which the lipid droplet detection device creates a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced from the original pathological tissue image acquired in the original image acquisition step; A cell nucleus image extraction step in which the lipid droplet detection device creates a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei contained in the original pathological tissue image from the R image created in the R image creation step; A foreground binary image creation step in which the lipid droplet detection device creates a second monochrome binary image from the G image created in the G image creation step, converts black pixels included in the second monochrome binary image to white pixels according to a rank that is the number of white pixels present in a filter area formed by a plurality of pixels included in the second monochrome binary image, and converts white pixels included in the second monochrome binary image to black pixels to create a third monochrome binary image obtained by performing the above process, and creates a foreground binary image by subtracting the first monochrome binary image created in the cell nucleus image extraction step from the third monochrome binary image; A background binary image creation step in which the lipid droplet detection device creates a background binary image in which the lipid droplets are extracted as white regions from the G image created in the G image creation step; The fat droplet detection device aligns the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step to create an AND image according to the rank by performing an AND operation, and cuts out a square circumscribing the fat droplet included in the AND image as a fat droplet candidate region. A fat droplet detection method comprising: In the lipid droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, In the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through a route in the reinforcement learning environment, the lipid droplet candidate regions cut out in the lipid droplet candidate extraction step will overlap more with each other. The Q value of the reinforcement learning is updated based on the reward of the reinforcement learning. A lipid droplet detection method, wherein the start and end positions of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower and upper limits of the size of the square.

10. To your computer An original image acquisition step of acquiring an original pathological tissue image; an R image creating step of creating an R image from the original pathological tissue image acquired in the original image acquiring step; A G image creation step of creating a G image which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced from the original pathological tissue image acquired in the original image acquisition step; a cell nucleus image extraction step of creating a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei included in the original pathological tissue image, from the R image created in the R image creation step; a foreground binary image creating step of creating a second monochrome binary image from the G image created in the G image creating step, converting black pixels included in the second monochrome binary image to white pixels and converting white pixels included in the second monochrome binary image to black pixels according to a rank that is the number of white pixels present in a filter area formed by a plurality of pixels included in the second monochrome binary image to create a third monochrome binary image, and creating a foreground binary image by subtracting the first monochrome binary image created in the cell nucleus image extracting step from the third monochrome binary image; A background binary image creation step of creating a background binary image in which the fat droplets are extracted as white regions from the G image created in the G image creation step; A program for executing a fat droplet candidate extraction step of performing an AND operation by aligning the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step to create an AND image according to the rank, and cutting out a square circumscribing the fat droplet included in the AND image as a fat droplet candidate region, In the lipid droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, In the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through a route in the reinforcement learning environment, the lipid droplet candidate regions cut out in the lipid droplet candidate extraction step will overlap more with each other. The Q value of the reinforcement learning is updated based on the reward of the reinforcement learning. A program in which the start and end movement positions of the reinforcement learning agent in one direction along the horizontal and vertical axes of the environment correspond to the lower and upper limits of the size of the square.

Images