Lipid droplet detection device, lipid droplet detection method and program

Abstract

To appropriately extract a lipid droplet candidate without using a teacher image.SOLUTION: A lipid droplet detection device includes an original image acquisition unit, an R image creation unit, a G image creation unit, a cell nucleus image extraction unit, a foreground binary image creation unit, a background binary image creation unit, and a lipid droplet candidate extraction unit for positioning a foreground binary image and a background binary image, creating an AND image corresponding to a rank by performing an AND operation, and cutting out a square circumscribing a lipid droplet included in the AND image as a lipid droplet candidate area. The lipid droplet candidate extraction unit determines the size and the rank of the square by performing reinforcement learning. In the reinforcement learning, it is given as a reward of the reinforcement learning that which route, on an environment of the reinforcement learning with a horizontal axis set as the size of the square and with a vertical axis set as the rank, the lipid droplet detection device as an agent of the reinforcement learning passes through to make multiple lipid droplet candidate areas cut out by the lipid droplet candidate extraction unit mutually overlap more.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a lipid droplet detection device, a lipid droplet detection method, and a program. [Background technology]

[0002] Lipid droplets contained in raw pathological tissue images obtained using optical microscopes, objective lenses, cameras, etc., are complex in size, intensity, and shape, making it difficult and time-consuming for even experienced pathologists to determine whether or not they are present. Furthermore, in formalin-fixed, paraffin-embedded (FFPE) specimens typically prepared for pathological analysis, lipid droplets are removed during the preparation process. Therefore, frozen tissue is required to detect lipid droplets pathologically, but preparing and storing frozen tissue is difficult for routine diagnostics. Furthermore, there is a need for the development of technologies that can support pathologists in analyzing raw pathological tissue images, which often contain complex images, and that can detect lipid droplets even in FFPE specimens.

[0003] Patent Document 1 describes a pathology diagnostic device that identifies the range of a lesion contained in an image of a diagnostic target site. Patent Document 1 also describes that even pathologists with a certain level of experience may have different findings when distinguishing between balloon-like hepatocytes and lipid droplet-containing hepatocytes contained in a liver biopsy image. In the technology described in Patent Document 1, whether or not a hepatocyte is lipid droplet-containing is input as correct answer data, and a verification model (learning model) is trained.

[0004] The technology described in Patent Document 1 and elsewhere requires a large amount of training and training images, but when targeting rare diseases or rare lesions, it is difficult to prepare a large number of training images, making it difficult to apply methods that require a large amount of training. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-112407 Summary of the Invention [Problem to be solved by the invention]

[0006] In view of the above, the present invention aims to provide a lipid droplet detection device, a lipid droplet detection method, and a program that can appropriately extract lipid droplet candidates without using a training image. [Means for solving the problem]

[0007] One aspect of the present invention includes an original image acquisition unit that acquires an original pathological tissue image, an R image creation unit that creates an R image from the original pathological tissue image acquired by the original image acquisition unit, a G image creation unit that creates a G image that is a lipid droplet-enhanced image in which lipid droplets included in the original pathological tissue image are enhanced from the original pathological tissue image acquired by the original image acquisition unit, a cell nucleus image extraction unit that creates a first monochrome binary image that is a monochrome binary image obtained by extracting cell nuclei included in the original pathological tissue image from the R image created by the R image creation unit, and a third monochrome binary image obtained by performing a process of converting black pixels included in the second monochrome binary image to white pixels and a process of converting white pixels included in the second monochrome binary image to black pixels according to a rank that is the number of white pixels present in a filter region formed by a plurality of pixels included in the second monochrome binary image, and a foreground binary image that is obtained by subtracting the first monochrome binary image created by the cell nucleus image extraction unit from the third monochrome binary image. a background binary image creation unit that creates a background binary image in which the lipid droplets are extracted as white regions from the G image created by the G image creation unit; and a lipid droplet candidate extraction unit that creates an AND image according to the rank by performing an AND operation after aligning the foreground binary image created by the foreground binary image creation unit with the background binary image created by the background binary image creation unit, and cuts out a square circumscribing the lipid droplet included in the AND image as a lipid droplet candidate region, wherein the lipid droplet candidate extraction unit The extraction unit determines the size of the square and the rank by performing reinforcement learning, and in the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through which route in the reinforcement learning environment, the plurality of lipid droplet candidate regions cut out by the lipid droplet candidate extraction unit will overlap more with each other. The Q value of the reinforcement learning is updated based on the reward of the reinforcement learning.The movement start position and movement end position of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower limit and upper limit of the size of the square.

[0008] The lipid droplet detection device according to one aspect of the present invention may include a discrimination unit that determines whether the lipid droplet is included in the square cut out by the lipid droplet candidate extraction unit.

[0009] In one embodiment of the lipid droplet detection device of the present invention, the foreground binary image creation unit may create the third monochrome binary image according to the rank determined by performing the reinforcement learning, and the background binary image creation unit may create the background binary image regardless of the rank determined by performing the reinforcement learning.

[0010] In one aspect of the lipid droplet detection device of the present invention, the foreground binary image creation unit creates the second monochrome binary image by applying an error diffusion method to the G image created by the G image creation unit, and converts all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 3 or more white pixels into white pixels, and converts all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 3 white pixels into black pixels, thereby creating the third monochrome binary image of "rank 3", and converts all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 4 or more white pixels into white pixels, and converts all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 4 white pixels into black pixels. The third monochrome binary image of "Rank 4" may be created by converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 5 or more white pixels to white pixels and converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 5 white pixels to black pixels, to create the third monochrome binary image of "Rank 5", and the third monochrome binary image of "Rank 6" may be created by converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 6 or more white pixels to white pixels and converting all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 6 white pixels to black pixels.

[0011] In one embodiment of the lipid droplet detection device of the present invention, the start position of the movement of the reinforcement learning agent in the environment may be a position in the environment where the size of the square is 2500 pixels (50 pixels x 50 pixels) and the rank is "Rank 6", and the end position of the movement of the reinforcement learning agent in the environment may be a position in the environment where the size of the square is 25 pixels (5 pixels x 5 pixels) and the rank is "Rank 3".

[0012] In one aspect of the lipid droplet detection device of the present invention, the position of the reinforcement learning agent in the environment at each time corresponds to the reinforcement learning state, and the movement of the reinforcement learning agent in the environment corresponds to the reinforcement learning behavior.

[0013] In the lipid droplet detection device according to one aspect of the present invention, penalty areas may be set in three locations in the environment.

[0014] In one embodiment of the lipid droplet detection device of the present invention, the background binary image creation unit may perform adaptive binarization processing on the G image created by the G image creation unit using a 225-pixel (15 pixels x 15 pixels) average filter, apply a 9-pixel (3 pixels x 3 pixels) average filter, and perform opening processing to create the background binary image.

[0015] One aspect of the present invention includes an original image acquisition step in which a lipid droplet detection device acquires an original pathological tissue image; an R image creation step in which the lipid droplet detection device creates an R image from the original pathological tissue image acquired in the original image acquisition step; a G image creation step in which the lipid droplet detection device creates a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are emphasized, from the original pathological tissue image acquired in the original image acquisition step; a cell nucleus image extraction step in which the lipid droplet detection device creates a first monochrome binary image, which is a monochrome binary image in which cell nuclei contained in the original pathological tissue image are extracted, from the R image created in the R image creation step; and a process in which the lipid droplet detection device creates a second monochrome binary image from the G image created in the G image creation step, and converts black pixels included in the second monochrome binary image to white pixels according to a rank, which is the number of white pixels present in a filter region formed by a plurality of pixels included in the second monochrome binary image, and converts white pixels included in the second monochrome binary image to black pixels. A foreground binary image creation step in which a third monochrome binary image is created by performing the above process, and a foreground binary image is created by subtracting the first monochrome binary image created in the cell nucleus image extraction step from the third monochrome binary image, and the lipid droplet detection device creates a background binary image in which the lipid droplets are extracted as white regions from the G image created in the G image creation step, and the lipid droplet detection device aligns the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step and performs an AND operation to create an AND image according to the rank, and cuts out a square circumscribing the lipid droplet included in the AND image as a lipid droplet candidate region. In the lipid droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, and in the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, andThis is a lipid droplet detection method in which the route through which the lipid droplet detection device as the reinforcement learning agent passes in the reinforcement learning environment, in which the other of the horizontal axis and the vertical axis is set as the rank, results in the most overlapping of the multiple lipid droplet candidate regions extracted in the lipid droplet candidate extraction step, and is given as a reward for the reinforcement learning, the Q value of the reinforcement learning is updated based on the reward for the reinforcement learning, and the movement start position and movement end position of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower and upper limits of the size of the square.

[0016] One aspect of the present invention is a method for obtaining an original pathological tissue image by a computer, comprising: an original image acquisition step of acquiring an original pathological tissue image in the original image acquisition step; an R image creation step of creating an R image from the original pathological tissue image acquired in the original image acquisition step; a G image creation step of creating a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced, from the original pathological tissue image acquired in the original image acquisition step; a cell nucleus image extraction step of creating a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei contained in the original pathological tissue image, from the R image created in the R image creation step; and a second monochrome binary image from the G image created in the G image creation step, and performing a process of converting black pixels contained in the second monochrome binary image to white pixels and a process of converting white pixels contained in the second monochrome binary image to black pixels according to a rank, which is the number of white pixels present in a filter region formed by a plurality of pixels included in the second monochrome binary image. A foreground binary image creation step of creating a foreground binary image by subtracting the first monochrome binary image created in the cell nucleus image extraction step from the third monochrome binary image, and a background binary image creation step of creating a background binary image in which the lipid droplets are extracted as white regions from the G image created in the G image creation step, and a lipid droplet candidate extraction step of performing an AND operation by aligning the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step to create an AND image according to the rank, and cutting out a square circumscribing the lipid droplet included in the AND image as a lipid droplet candidate region, wherein in the lipid droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, and in the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, andThe other of the horizontal axis and the vertical axis is set as the rank, and the route along which the lipid droplet detection device as the reinforcement learning agent passes through in the reinforcement learning environment so that the plurality of lipid droplet candidate regions cut out in the lipid droplet candidate extraction step overlap with each other is given as a reward for the reinforcement learning, the Q value of the reinforcement learning is updated based on the reward for the reinforcement learning, and the movement start position and movement end position of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower limit and upper limit of the size of the square. [Effects of the Invention]

[0017] According to the present invention, lipid droplet candidates can be appropriately extracted without using training images. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram showing an example of a lipid droplet detection device 1 according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing an example of weighting in the error diffusion method applied to a pixel of interest indicated by "*." [Figure 3] 10 is a diagram showing an example of a first monochrome binary image (binary cell nucleus image) created by the cell nucleus image extraction unit 14. FIG. [Figure 4] 10A and 10B are diagrams showing examples of foreground binary images of "Rank 3" to "Rank 6" created by foreground binary image creating section 15. FIG. [Figure 5] FIG. 10 is a diagram showing an example of AND images of "Rank 3" to "Rank 6" created by the lipid droplet candidate extraction unit 17. [Figure 6] FIG. 10 is a diagram for schematically explaining an example of the relationship between the lipid droplets and squares cut out as lipid droplet candidate regions by the lipid droplet candidate extraction unit 17. [Figure 7] FIG. 1 is a diagram showing an example of a reinforcement learning environment defined (set) in reinforcement learning performed by the lipid droplet candidate extraction unit 17. [Figure 8] FIG. 10 is a diagram showing an example of the results of discrimination of lipid droplets by the discrimination unit 18. [Figure 9] 4 is a flowchart for explaining an example of processing executed by the lipid droplet detection device 1 of the first embodiment. [Figure 10] 1 is a scatter plot matrix showing the relationship between the characteristics of lipid droplet images and the characteristics of normal images. [Figure 11] FIG. 10 shows images including lipid droplets detected by the discrimination unit 18 and evaluated by two pathologists. [Figure 12] FIG. 10 is a diagram showing an example of AND images of "Rank 3" to "Rank 6" created by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment. [Figure 13] FIG. 10 is a diagram showing the average Q value for each episode in an example. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, with reference to the drawings, embodiments of the lipid droplet detection device, lipid droplet detection method, and program of the present invention will be described.

[0020] First Embodiment FIG. 1 is a diagram showing an example of a lipid droplet detection device 1 according to the first embodiment. 1, the lipid droplet detection device 1 is configured by, for example, a computer including a communication interface (input / output interface), a memory, and a processor. The communication interface (input / output interface) has an interface circuit for connecting the lipid droplet detection device 1 to, for example, a camera that captures the original pathological tissue image. In another example, the lipid droplet detection device 1 may be connected via a communication interface (input / output interface) to a storage device (for example, a server, computer, etc.) in which data of the original pathological tissue image is stored.

[0021] 1, the memory of the lipid droplet detection device 1 stores programs and various data used in the processing executed by the processor. The processor has a function as an original image acquisition unit 11, a function as an R image creation unit 12, a function as a G image creation unit 13, a function as a cell nucleus image extraction unit 14, a function as a foreground binary image creation unit 15, a function as a background binary image creation unit 16, a function as a lipid droplet candidate extraction unit 17, and a function as a discrimination unit 18.

[0022] The original image acquisition unit 11 acquires (reads) an original image (color image) of a pathological tissue obtained using, for example, an optical microscope, an objective lens, a camera, and the like. The R image creating unit 12 creates an R image from the original pathological tissue image acquired by the original image acquiring unit 11 by using, for example, the technology described in the website indicated by the URL below. https: / / potesara-tips.com / separating-colors /

[0023] The G image creation unit 13 creates a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are emphasized, from the original pathological tissue image acquired by the original image acquisition unit 11, by using, for example, a technique described on the website indicated by the above URL.

[0024] Cell nucleus image extraction unit 14 creates a first monochrome binary image, which is a monochrome binary image in which cell nuclei included in the original pathological tissue image are extracted from the R image created by R image creation unit 12. In detail, the cell nucleus image extraction unit 14 creates a binary dot image from the R image created by the R image creation unit 12 using pseudo-gradation based on the error diffusion method, which is a technology described, for example, on the website indicated by the URL below. https: / / zenn.dev / baroqueengine / books / a19140f2d9fc1a / viewer / 531f1d

[0025] FIG. 2 is a diagram showing an example of weights in the error diffusion method applied to pixels of interest indicated by "*."

[0026] Next, the cell nucleus image extraction unit 14 applies a median filter (more specifically, a filter consisting of 9 pixels (3 pixels x 3 pixels)), which is a technique described, for example, in the website indicated by the URL below, to the binary point image. https: / / www.frontier.maxell.co.jp / blog / posts / 17.html

[0027] Next, the cell nucleus image extraction unit 14 performs black-and-white inversion processing on the image obtained by applying the median filter. Finally, the cell nucleus image extraction unit 14 performs a closing process on the image obtained by the black and white inversion process, which is a technique described, for example, in the website indicated by the URL below, to create the above-mentioned first monochrome binary image, which is a binary cell nucleus image. https: / / www.frontier.maxell.co.jp / blog / posts / 22.html

[0028] FIG. 3 is a diagram showing an example of a first monochrome binary image (binary cell nucleus image) created by the cell nucleus image extraction unit 14. As shown in FIG. In detail, Figure 3(A) shows an example of a portion (more specifically, one-sixth of the entire original pathological tissue image) of the original pathological tissue image (color image) acquired by the original image acquisition unit 11, and Figure 3(B) shows a first monochrome binary image (binary cell nucleus image) created by the cell nucleus image extraction unit 14 from the original pathological tissue image (color image) shown in Figure 3(A).

[0029] In the example shown in Figure 1, the foreground binary image creation unit 15 creates a second monochrome binary image by applying the above-mentioned error diffusion method to the G image (fat droplet enhanced image) created by the G image creation unit 13. Next, foreground binary image creation unit 15 applies a filter area consisting of a plurality of pixels (for example, 9 pixels (3 pixels x 3 pixels)) to the second monochrome binary image. In detail, foreground binary image creation unit 15 defines the number of white pixels present in the filter area (9 pixels (3 pixels x 3 pixels)) as a "rank." Next, the foreground binary image creation unit 15 creates a third monochrome binary image by performing a process of converting black pixels contained in the second monochrome binary image to white pixels and a process of converting white pixels contained in the second monochrome binary image to black pixels according to the rank. In detail, the foreground binary image creation unit 15 performs a process of converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where white pixels of rank or higher exist into white pixels, and also performs a process of converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where white pixels of rank lower exist into black pixels, thereby creating a third monochrome binary image.

[0030] Specifically, the foreground binary image creation unit 15 creates a third monochrome binary image of "Rank 3" by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where three or more white pixels exist into white pixels, and converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where less than three white pixels exist into black pixels. In addition, the foreground binary image creation unit 15 creates a third monochrome binary image of "rank 4" by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where four or more white pixels exist into white pixels, and converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where less than four white pixels exist into black pixels. Furthermore, the foreground binary image creation unit 15 creates a third monochrome binary image of "Rank 5" by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where five or more white pixels exist into white pixels, and converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where less than five white pixels exist into black pixels. In addition, the foreground binary image creation unit 15 creates a third monochrome binary image of "Rank 6" by converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where six or more white pixels exist into white pixels, and converting all pixels in the filter area (9 pixels (3 pixels x 3 pixels)) where less than six white pixels exist into black pixels.

[0031] Furthermore, the foreground binary image creating unit 15 creates a foreground binary image by subtracting the first monochrome binary image (see FIG. 3(B)) created by the cell nucleus image extracting unit 14 from the third monochrome binary image. Specifically, the foreground binary image creation unit 15 creates a foreground binary image of "Rank 3" by subtracting the first monochrome binary image (see Figure 3(B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "Rank 3". In addition, the foreground binary image creation unit 15 creates a foreground binary image of "rank 4" by subtracting the first monochrome binary image (see Figure 3(B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "rank 4". Furthermore, the foreground binary image creation unit 15 creates a foreground binary image of "Rank 5" by subtracting the first monochrome binary image (see Figure 3(B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "Rank 5". In addition, the foreground binary image creation unit 15 creates a foreground binary image of "Rank 6" by subtracting the first monochrome binary image (see Figure 3(B)) created by the cell nucleus image extraction unit 14 from the third monochrome binary image of "Rank 6".

[0032] FIG. 4 is a diagram showing examples of foreground binary images of "Rank 3" to "Rank 6" created by foreground binary image creating unit 15. As shown in FIG. In detail, Figure 4(A) shows a foreground binary image of "Rank 6" created by the foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 4(B) shows a foreground binary image of "Rank 5" created by the foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 4(C) shows a foreground binary image of "Rank 4" created by the foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A), and Figure 4(D) shows a foreground binary image of "Rank 3" created by the foreground binary image creation unit 15 from the original pathological tissue image (color image) shown in Figure 3(A).

[0033] In the example shown in FIG. 1, the background binary image creating unit 16 creates a background binary image in which lipid droplets are extracted as white regions from the G image (lipid droplet-enhanced image) created by the G image creating unit 13. In detail, the background binary image creation unit 16 applies a 225 pixel (15 pixels x 15 pixels) average filter (averaging filter) to the G image created by the G image creation unit 13 by using the technology described on the website indicated by the URL below, and performs adaptive binarization processing. https: / / www.frontier.maxell.co.jp / blog / posts / 14.html https: / / www.frontier.maxell.co.jp / blog / posts / 19.html

[0034] Next, the background binary image creation unit 16 applies a 9-pixel (3 pixels x 3 pixels) average filter (averaging filter) and performs opening processing, which is a technique described, for example, in the website indicated by the URL below, to create a background binary image in which fat droplets are extracted as white regions. https: / / www.frontier.maxell.co.jp / blog / posts / 22.html

[0035] In the example shown in FIG. 1, the lipid droplet candidate extraction unit 17 aligns the foreground binary image (see FIG. 4) created by the foreground binary image creation unit 15 with the background binary image created by the background binary image creation unit 16 and performs an AND operation to create an AND image according to the rank ("Rank 3" to "Rank 6"). That is, if the brightness of a pixel on the background binary image at the same coordinates (same position) as a pixel (white pixel) with a brightness of 255 on the foreground binary image is 255, the brightness of the pixel on the AND image at that coordinate (position) will be 255. The brightness of a pixel on the AND image at the same coordinates (same position) as a pixel on the foreground binary image with a brightness of less than 255 will be zero. That is, a pixel on the AND image at the same coordinates (same position) as a pixel on the foreground binary image with a brightness of less than 255 will be a black pixel. In addition, the brightness of a pixel on the AND image at the same coordinates (same position) as a pixel on the background binary image with a brightness of less than 255 will also be zero. In other words, pixels in the AND image that have the same coordinates (same positions) as pixels in the foreground binary image with a brightness of less than 255 also become black pixels.

[0036] FIG. 5 is a diagram showing an example of AND images of "Rank 3" to "Rank 6" created by the lipid droplet candidate extraction unit 17. In detail, Figure 5(A) shows an AND image of "Rank 6" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 5(B) shows an AND image of "Rank 5" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A), Figure 5(C) shows an AND image of "Rank 4" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A), and Figure 5(D) shows an AND image of "Rank 3" created by the lipid droplet candidate extraction unit 17 from the original pathological tissue image (color image) shown in Figure 3(A).

[0037] Furthermore, the lipid droplet candidate extraction unit 17 cuts out a square circumscribing the lipid droplet contained in the AND image (see FIG. 5) as a lipid droplet candidate region.

[0038] FIG. 6 is a diagram for schematically explaining an example of the relationship between the squares cut out as lipid droplet candidate regions by the lipid droplet candidate extraction unit 17 and the lipid droplets. In the example shown in Figure 6, the square located in the upper left of Figure 6 (the square indicated by "The previously extracted region") is first cut out by the lipid droplet candidate extraction unit 17, and then the square located in the lower right of Figure 6 (the square indicated by "The newly extracted region") is cut out by the lipid droplet candidate extraction unit 17. In addition, the lower right part of the square located in the upper left of Figure 6 and the upper left part of the square located in the lower right of Figure 6 form an overlapping region (the part indicated by "The overlaps region") where they overlap each other. In the example shown in Figure 6, the square located in the upper left of Figure 6 circumscribes the largest lipid droplet in the center of Figure 6, and the square located in the lower right of Figure 6 circumscribes the second largest lipid droplet on the right side of Figure 6. The square in the upper left of Figure 6 includes the entire largest lipid droplet in the center of Figure 6, the entire smallest lipid droplet on the left side of Figure 6, and part of the second largest lipid droplet on the right side of Figure 6. The square located at the bottom right of Figure 6 includes the entire second largest lipid droplet on the right side of Figure 6 and part of the largest lipid droplet in the center of Figure 6.

[0039] In the example shown in FIG. 1, the lipid droplet candidate extraction unit 17 determines the size of the square (see FIG. 6) and the above-mentioned rank ("Rank 3" to "Rank 6") by performing reinforcement learning. In detail, in the example shown in FIG. 1 (an example of the lipid droplet detection device 1 of the first embodiment), the lipid droplet candidate extraction unit 17 performs Q-learning as reinforcement learning.

[0040] FIG. 7 is a diagram showing an example of a reinforcement learning environment defined (set) in the reinforcement learning performed by the lipid droplet candidate extraction unit 17. In the example shown in Figure 7, the horizontal axis of the reinforcement learning environment is set as the size of the square (see Figure 6) (more specifically, the area (number of pixels) of the square), and the vertical axis of the reinforcement learning environment is set as the above-mentioned rank ("Rank 3" to "Rank 6"). In Figure 7, the numbers "0" to "39" indicate indexes. In the example shown in Figure 7, in the reinforcement learning performed by the lipid droplet candidate extraction unit 17, the reward for reinforcement learning is given based on which route in the reinforcement learning environment shown in Figure 7 the lipid droplet detection device 1, as the reinforcement learning agent, should take to ensure that the multiple lipid droplet candidate regions (squares (see Figure 6)) cut out by the lipid droplet candidate extraction unit 17 overlap more with each other.

[0041] In the example shown in FIG. 7, the movement start position of the reinforcement learning agent (lipid droplet detection device 1) in the reinforcement learning environment is set to index "9" in the upper right corner of the reinforcement learning environment shown in FIG. 7. In other words, the movement start position of the reinforcement learning agent (lipid droplet detection device 1) in the reinforcement learning environment is a position in the reinforcement learning environment where the size of the square is 2500 pixels (50 pixels x 50 pixels) and the rank is "rank 6." The movement start position of the reinforcement learning agent (lipid droplet detection device 1) in the direction of the horizontal axis of the reinforcement learning environment (left and right direction in FIG. 7) corresponds to the upper limit of the square size (2500 pixels (50 pixels x 50 pixels)). In the example shown in FIG. 7, the movement end position (goal) of the reinforcement learning agent (lipid droplet detection device 1) is set to the index "30" in the lower left corner of the reinforcement learning environment. In other words, the movement end position of the reinforcement learning agent (lipid droplet detection device 1) in the reinforcement learning environment is a position in the reinforcement learning environment where the square size is 25 pixels (5 pixels x 5 pixels) and the rank is "rank 3." The movement end position of the reinforcement learning agent (lipid droplet detection device 1) in the direction of the horizontal axis of the reinforcement learning environment (left and right direction in FIG. 7) corresponds to the lower limit of the square size (25 pixels (5 pixels x 5 pixels)). Furthermore, in the example shown in FIG. 7, as shown by the arrows in FIG. 7, the reinforcement learning agent (lipid droplet detection device 1) is set to be able to move leftward or downward in the reinforcement learning environment. The position of the reinforcement learning agent (lipid droplet detection device 1) in the reinforcement learning environment at each time t is determined by the "state S t The movement of the reinforcement learning agent (lipid droplet detection device 1) in the reinforcement learning environment (movement to the left in FIG. 7 or movement downward in FIG. 7) corresponds to "action a" in reinforcement learning.

[0042] In the example shown in Figure 7, the size of the square on the horizontal axis of the reinforcement learning environment (area of ​​the square (number of pixels)) and the index have the relationship shown in the following formula. In other words, the size of the square is the square obtained by dividing the index by 10, multiplying the remainder by 5, adding 5, and squaring the result. Specifically, the square size "1600" is the square obtained by dividing the index "17" by 10, multiplying the remainder by 5, adding 5, and squaring the result. Square size = (5 × (index%10) + 5) 2

[0043] In the example shown in Figure 7, the rank and index on the vertical axis of the reinforcement learning environment have the relationship shown in the following formula. In other words, the rank is calculated by dividing the index by 10, discarding the decimal point, and subtracting the result from 6. Specifically, the rank "5" is calculated by dividing the index "17" by 10, discarding the decimal point, and subtracting the result from 6. Rank=6-index / 10

[0044] As described above, in the example shown in FIG. 7, in the reinforcement learning performed by the lipid droplet candidate extraction unit 17, the route in the reinforcement learning environment through which the lipid droplet detection device 1 as the reinforcement learning agent passes through results in the most overlapping of the multiple lipid droplet candidate regions (squares (see FIG. 6)) extracted by the lipid droplet candidate extraction unit 17. Therefore, as shown in FIG. 6, when the largest lipid droplet in the center and the second largest lipid droplet on the right are close to each other, the square located in the upper left of FIG. 6 not only includes the largest lipid droplet in the center, but also includes a portion of the second largest lipid droplet on the right. In other words, the second largest lipid droplet in FIG. 6 is not only extracted (cut out) by the square located in the lower right of FIG. 6, but is also cut out (extracted) with a portion overlapping with the square located in the upper left of FIG. 6. In detail, in the examples shown in Figures 6 and 7, the reinforcement learning reward is set to the proportion of the area of ​​the square located in the lower right of Figure 6 (the square indicated as "The newly extracted region") that overlaps with the initially extracted square located in the upper left of Figure 6 (the square indicated as "The previously extracted region") (the area indicated as "The overlaps region").

[0045] In the example shown in Figure 1, the Q value of reinforcement learning is updated based on the reward of reinforcement learning. In detail, the Q value of reinforcement learning is updated using the following formula based on the reward of reinforcement learning, etc.

[0046]

number

[0047] In the above equation, r represents the reward, α represents the learning rate, and γ represents the discount rate. In the example shown in Figure 1, the ε-greedy method (ε: probability of randomly selecting an action) is used to select "action a" in reinforcement learning.

[0048] In the example shown in Figure 7, penalty areas are set at three locations in the reinforcement learning environment. Specifically, index "8", index "29", and index "38" are set as penalty areas. 7, reinforcement learning actions are performed a maximum of 12 times in one episode. In the reinforcement learning actions, if the position of the reinforcement learning agent (lipid droplet detection device 1) goes outside the reinforcement learning environment or enters a penalty area, the reinforcement learning reward is set to "-1", and the position of the reinforcement learning agent (lipid droplet detection device 1) is returned to the movement start position (index "9"). 1 and 7, the reinforcement learning is repeated by repeating the above-mentioned episodes multiple times. Furthermore, each time reinforcement learning is performed, pixels having edges with low brightness values ​​(small differences in brightness values) are deleted from the background binary image by, for example, the background binary image creation unit 16.

[0049] In the example shown in FIG. 1, the discrimination unit 18 determines whether or not a lipid droplet (see FIG. 6) is contained in a square cut out by the lipid droplet candidate extraction unit 17. In detail, the discrimination unit 18 determines whether or not a lipid droplet is contained in a square region (see FIG. 8, for example) on the original pathological tissue image (original pathological tissue image acquired by the original image acquisition unit 11) corresponding to the lipid droplet candidate region (square) cut out by the lipid droplet candidate extraction unit 17, for example, by using a model obtained by performing learning using training data, which is a data set of original pathological tissue images for learning and labels indicating whether or not the original pathological tissue images for learning contain lipid droplets. Specifically, the discrimination unit 18 detects lipid droplets from the square (lipid droplet candidate region) cut out by the lipid droplet candidate extraction unit 17 using a random forest with parameters optimized by grid search.

[0050] FIG. 8 is a diagram showing an example of the results of lipid droplet discrimination by the discrimination unit 18. In detail, FIG. 8(A) shows the results of discrimination by the discrimination unit 18 as to whether lipid droplets are contained in squares cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 6" (see FIG. 5(A)). Specifically, FIG. 8(A) shows, surrounded by squares, lipid droplets detected by the discrimination unit 18 from the squares (lipid droplet candidate regions) cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 6" (see FIG. 5(A)). FIG. 8(B) shows the results of discrimination by the discrimination unit 18 as to whether lipid droplets are contained in squares cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 5" (see FIG. 5(B)). Specifically, FIG. 8(B) shows, surrounded by squares, lipid droplets detected by the discrimination unit 18 from the squares (lipid droplet candidate regions) cut out by the lipid droplet candidate extraction unit 17 from the AND image of "rank 5" (see FIG. 5(B)). 8(C) shows the result of discrimination by the discrimination unit 18 as to whether or not lipid droplets are contained in the squares cut out by the lipid droplet candidate extraction unit 17 from the "Rank 4" AND image (see FIG. 5(C)). Specifically, FIG. 8(C) shows the lipid droplets detected by the discrimination unit 18 from the squares (lipid droplet candidate regions) cut out by the lipid droplet candidate extraction unit 17 from the "Rank 4" AND image (see FIG. 5(C)) surrounded by squares. FIG. 8(D) shows the result of discrimination by the discrimination unit 18 as to whether or not lipid droplets are contained in the squares cut out by the lipid droplet candidate extraction unit 17 from the "Rank 3" AND image (see FIG. 5(D)). Specifically, FIG. 8(D) shows the lipid droplets detected by the discrimination unit 18 from the squares (lipid droplet candidate regions) cut out by the lipid droplet candidate extraction unit 17 from the "Rank 3" AND image (see FIG. 5(D)) surrounded by squares. FIG. 8(E) shows the overall discrimination result by the discrimination unit 18. Specifically, FIG. 8(E) shows the lipid droplets detected by the discrimination unit 18 from all the squares (lipid droplet candidate regions) cut out by the lipid droplet candidate extraction unit 17, surrounded by a rectangle.

[0051] 1, foreground binary image creation unit 15 creates a third monochrome binary image according to the rank determined by reinforcement learning, while background binary image creation unit 16 creates a background binary image regardless of the rank determined by reinforcement learning.

[0052] FIG. 9 is a flowchart for explaining an example of processing executed by the lipid droplet detection device 1 of the first embodiment. In the example shown in FIG. 9, in step S1, the original image acquisition unit 11 acquires an original pathological tissue image (color image). In step S2, the R image creating unit 12 creates an R image from the original pathological tissue image acquired in step S1. In step S3, the G image creating unit 13 creates a G image (a lipid droplet-enhanced image) from the original pathological tissue image acquired in step S1. In step S4, the cell nucleus image extraction unit 14 extracts cell nuclei from the R image created in step S2, and creates a first monochrome binary image. In step S5, the foreground binary image creating unit 15 creates foreground binary images of "Rank 3" to "Rank 6". In step S6, the background binary image creating unit 16 creates a background binary image from the G image (fat droplet enhanced image) created in step S3. In step S7, the lipid droplet candidate extraction unit 17 aligns the foreground binary images of "Rank 3" to "Rank 6" created in step S5 with the background binary image created in step S6 to create an AND image of "Rank 3" to "Rank 6", and cuts out squares circumscribing the lipid droplets included in the AND image as lipid droplet candidate regions. In addition, the lipid droplet candidate extraction unit 17 determines the size and rank ("Rank 3" to "Rank 6") of the squares by performing reinforcement learning. In step S8, the discriminator 18 discriminates whether or not the square cut out in step S7 contains lipid droplets. In step S9, the lipid droplet detection device 1 outputs a lipid droplet-detected image, which is an image determined to contain lipid droplets in step S8.

[0053] [Example] The inventors conducted experiments (such as evaluation of the lipid droplet detection device 1 of the first embodiment) using 10 images of human liver tissue taken at Oita University School of Medicine (five normal images (original pathological tissue images without lipid droplets) and five lipid droplet images (original pathological tissue images containing lipid droplets)). Three lipid droplet images and two normal images were used as learning data (teaching data) used for training the discrimination unit 18. The remaining five images (three normal images and two lipid droplet images) were used to evaluate the discrimination unit 18. In the experiments, two pathologists with more than 10 years of clinical experience independently evaluated the images in which lipid droplets were detected.

[0054] (1) Experiment 1 The inventors compared the f-values ​​of k-nearest neighbors, support vector machines (SVMs), and random forests (RFs) using 10-fold cross-validation. Table 1 shows the comparison results of f-values ​​using 10-fold cross-validation for k-nearest neighbors, support vector machines, and random forests as classifiers. Specifically, each f-value in Table 1 is the average score for each image.

[0055] [Table 1]

[0056] The inventors used the area ratio p defined by the following formula as a feature used for classification: In the formula below, a represents the area of ​​the binarized object (fat droplet) (not the "action" of the reinforcement learning described above), and D represents the area of ​​the detection region (the square circumscribing the fat droplet).

[0057]

number

[0058] Furthermore, the inventors used symmetry q, defined by the following formula, as a feature used for classification: In the formula below, s represents the area where the binarized object (fat droplet) overlaps with the binarized object (fat droplet) mirrored horizontally.

[0059]

number

[0060] Furthermore, the inventors used the contrast ratio c defined by the following formula as a feature used for classification: In the formula below, r (not the "reward" of the reinforcement learning described above) represents the average brightness value of the R image within the detection region (the square circumscribing the lipid droplet), and g represents the average brightness value of the G image within the detection region (the square circumscribing the lipid droplet).

[0061]

number

[0062] Furthermore, the inventors used a variance ratio v defined by the following equation as a feature used for classification.

[0063]

number

[0064] (2) Experiment 2 Furthermore, the inventors performed a dimensionality reduction of the features using principal component analysis. Two pathologists also evaluated the degree of lipid droplet detection. Table 2 shows the effect of dimensionality reduction of the features using principal component analysis. The execution time of this method was an average of 481.8 seconds per image using Google Colaboratory.

[0065] [Table 2]

[0066] The inventors used 160 examples from a total of 200 datasets consisting of normal images (images of non-lipid droplets, i.e., images that are not lipid droplets) and lipid droplet images (the ratio of normal images to lipid droplet images was 1:1) for training each classifier. However, the detected lipid droplet images do not necessarily exhibit a typical circular shape; lipid droplet images may be connected to normal images or may be in contact with each other. In addition, the structure of non-lipid droplets, such as vascular cavities other than lipid droplets, may be similar to the structure of lipid droplets. For this reason, it is thought that the boundary separating the characteristics of lipid droplet images from those of normal images spans both.

[0067] Figure 10 is a scatter plot matrix showing the relationship between the characteristics of lipid droplet images (shown as "Lipid droplet" in Figure 10) and the characteristics of normal images (shown as "Normal" in Figure 10). The inventors evaluated the performance of the classifier using the f-value, which decreases when there are extremely many false positives or false negatives. Furthermore, pathologists' evaluations showed that the random forest classifier had the fewest false positives and false negatives and the highest f-value compared to other classifiers. This is thought to be because the random forest classifier was able to detect lipid droplets from lipid droplet candidates by drawing a boundary line perpendicular to the axis in the feature space, compared to other classifiers. This indicates that there is a threshold that can simply separate the distribution of lipid droplet images from the distribution of normal images to a certain extent. The characteristics of lipid droplets detected by the discrimination unit 18 were high area ratio and symmetry, and low contrast ratio and variance ratio.

[0068] FIG. 11 shows images containing lipid droplets detected by the discrimination unit 18 and evaluated by two pathologists. In detail, FIG. 11(A) shows a portion (one-sixth of the entire original pathological tissue image) of the same original pathological tissue image (color image) as FIG. 3(A). FIG. 11(B) shows the discrimination result by the discrimination unit 18 for the original pathological tissue image shown in FIG. 11(A). In detail, FIG. 11(B) shows an image output as a lipid droplet detection image from the lipid droplet detection device 1 after the lipid droplet candidate extraction unit 17 cuts out a square (lipid droplet candidate region) from the original pathological tissue image shown in FIG. 11(A) and the discrimination unit 18 discriminates the image. Two pathologists evaluated the area surrounded by the square in the image shown in Figure 11(B), and found that the square area contained a sufficient number of lipid droplets that should be detected by the lipid droplet detection device 1. In reality, pathologists do not make diagnoses based on lipid droplets alone, but also on other morphological information, so the lipid droplet detection image output from the lipid droplet detection device 1 is considered to be clinically usable.

[0069] As shown in Table 2, the dimension reduction using principal component analysis resulted in no difference in the evaluation using the mixing matrix. However, among the lipid droplet images that were not sufficiently separated from normal images, some could not be identified even after dimension reduction (i.e., some were not detected as lipid droplets by the discrimination unit 18). Therefore, in the lipid droplet detection device 1 of the first embodiment, classification is performed using a simple small number of features that do not require dimension reduction. However, in order to further improve generalization performance and accuracy in the future, in modified versions of the lipid droplet detection device 1 of the first embodiment, it is considered necessary to separate the lipid droplet images detected by the discrimination unit 18 from normal images, add features such as circularity, and increase the number of training data.

[0070] Adaptive binarization is an extremely effective process that can extract even faint images because a binarization threshold is set for each pixel. However, the multiple lipid droplet images contained in the original pathological tissue image each have differences in size, density, etc. For this reason, adaptive binarization using only the same filter size does not necessarily ensure sufficient separation of the boundary between the lipid droplet and its surroundings. However, if binarization can be performed using an appropriate filter size according to the size of the lipid droplet image, the true boundary is likely to be near the boundary obtained by adaptive binarization. Therefore, the foreground binary image creation unit 15 of the lipid droplet detection device 1 of the first embodiment uses a binary point image (second monochrome binary image) obtained by applying error diffusion to the G image obtained from the original pathological tissue image, and expresses the shading information of the brightness value of the G image as dot density (for example, high ranks indicate white, dark G image densities, and low ranks indicate lighter densities). The lipid droplet images of each rank seen in the foreground binary image and their surrounding boundaries (lipid droplet size) are captured using shading. If there is a common area between the foreground binary image and the background binary image of each rank obtained by adaptive binarization processing, there is a possibility that this is a boundary corresponding to the size of the lipid droplet image. In the lipid droplet detection device 1 of the first embodiment, even for lipid droplets that are difficult to separate from their surroundings using adaptive binarization processing alone, the lipid droplet candidate extraction unit 17 cuts out lipid droplet candidate regions (squares), and the discrimination unit 18 performs discrimination, thereby enabling lipid droplet detection.

[0071] In the lipid droplet detection device 1 of the first embodiment, the overlap of lipid droplet candidate regions extracted by the lipid droplet candidate extraction unit 17 to cover all lipid droplet candidates contained in the original pathological tissue image is given as a reward for reinforcement learning. However, when evaluating only the overlap of lipid droplet candidate regions extracted by the lipid droplet candidate extraction unit 17, there is a risk that images (normal images) of things other than lipid droplets (e.g., vascular cavities) and even noise present around the lipid droplet images may be included in the lipid droplet candidate regions. Therefore, in a modified example of the lipid droplet detection device 1 of the first embodiment, weak edges observed in the lipid droplet-enhanced image (G image) are determined to be likely noise (even faint lipid droplet edges have the same strength as cell nucleus edges), and the edge information is used for noise removal to control the lipid droplet candidate regions extracted by the lipid droplet candidate extraction unit 17.

[0072] FIG. 12 is a diagram showing an example of AND images of "Rank 3" to "Rank 6" created by the lipid droplet candidate extraction unit 17 of the modified example of the lipid droplet detection device 1 of the first embodiment. In detail, Figure 12(A) shows an AND image of "rank 6" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment, Figure 12(B) shows an AND image of "rank 5" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment, Figure 12(C) shows an AND image of "rank 4" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment, and Figure 12(D) shows an AND image of "rank 3" created by removing noise by the lipid droplet candidate extraction unit 17 of a modified example of the lipid droplet detection device 1 of the first embodiment. In the example shown in Fig. 12, the lipid droplet candidate extraction unit 17 of the modified example of the lipid droplet detection device 1 of the first embodiment removed the bottom 10% of the edge intensity as noise. As a result, as shown in Fig. 12, small normal image parts such as cell nuclei and gaps near the lipid droplet image are not included in the lipid droplet candidate region cut out by the lipid droplet candidate extraction unit 17, and noise removal using edges was effective in extracting lipid droplet candidates.

[0073] As described above, in the example shown in FIG. 1 (one example of the lipid droplet detection device 1 of the first embodiment), the lipid droplet candidate extraction unit 17 performs Q-learning as reinforcement learning. In the Q-learning performed by the lipid droplet candidate extraction unit 17, the size of the lipid droplet image corresponds to the size of the square (lipid droplet candidate region) cut out by the lipid droplet candidate extraction unit 17, and the shading of the lipid droplet image corresponds to the rank ("Rank 3" to "Rank 6"). By combining the size and rank of the square (lipid droplet candidate region), each of multiple lipid droplet images of different sizes and shading can be captured. In other words, the lipid droplet candidate extraction unit 17 performs Q-learning to change the size of the square (lipid droplet candidate region), thereby appropriately cutting out lipid droplet candidate regions (squares) including each of multiple lipid droplet images of different sizes. Furthermore, the lipid droplet candidate extraction unit 17 can appropriately cut out lipid droplet candidate regions (squares) including each of multiple lipid droplet images of different shading according to the rank (shading of the lipid droplet image) determined by performing Q-learning. In the example shown in Figure 7 described above, downward movement of the agent (lipid droplet detection device 1) in the environment shown in Figure 7 corresponds to a lowering of the rank (for example, an increase in the size of the object (lipid droplet) in the foreground binary image shown in Figure 4). The left-right position of the agent (lipid droplet detection device 1) in the environment shown in Figure 7 corresponds to the size of the lipid droplet candidate region (square) cut out by the lipid droplet candidate extraction unit 17. Each time the agent (lipid droplet detection device 1) moves leftward in the environment shown in Figure 7, the size of the lipid droplet candidate region (square) cut out by the lipid droplet candidate extraction unit 17 is determined. If the agent (lipid droplet detection device 1) moves leftward multiple times in succession in the environment shown in Figure 7 without moving downward in the environment shown in Figure 7, the sizes of multiple lipid droplet candidate regions (square) are determined (set) for one rank.

[0074] The range (upper and lower limits) of the size of the lipid droplet candidate region (square) according to each rank is determined by the agent (lipid droplet detection device 1) moving from the start position to the end position (goal) in the environment shown in Figure 7. The range of the size of the lipid droplet candidate region (square) according to each rank can also be determined by using the range of indices at the same rank. However, in the example shown in Figure 7 where the penalty area is set at index "8", the agent (lipid droplet detection device 1) cannot move leftward in the environment shown in Figure 7 at rank "6", so the size of the lipid droplet candidate region (square) at rank "6" cut out by the lipid droplet candidate extraction unit 17 is determined (set) to be "2500" or more. As a result, small lipid droplets are not detected at rank "6". In this way, repeated interactions between the environment and the agent enable image-based learning in the lipid droplet detection device 1 of the first embodiment. This suggests that even without a special blueprint for lipid droplet extraction, "emergence" occurs from simple interactions and is brought to the surface.

[0075] Although the lipid droplet candidate extraction unit 17 can extract lipid droplet candidate regions (squares) without setting penalty areas, in the example shown in Fig. 7, penalty areas are set in three locations in the environment shown in Fig. 7 for the purposes of shortening the learning time and avoiding instability of learning. In the early stages of learning, the agent often enters the penalty areas, but gradually the agent begins to take actions that avoid the penalty areas and eventually reaches the goal.

[0076] FIG. 13 is a diagram showing the average Q value for each episode in the example. In reinforcement learning, the agent's optimal behavior in each state (index) is adopted, and after 200 episodes, the average Q value converges to a constant value for that image and stabilizes. The parameters used in the example were ε = 0.1, α = 0.2, and γ = 0.9, and the number of learning iterations was set to 100, taking processing time into consideration. In the example shown in FIG. 7, the agent's behavior is limited to movement to the left or downward in FIG. 7, so the agent does not pass through the environmental indices "0" to "7" and "39" shown in FIG. 7, and the environmental indices "0" to "7" and "39" shown in FIG. 7 are missing. When the original pathological tissue image (color image) shown in FIG. 3(A) is acquired by the original image acquisition unit 11, the images corresponding to the indices "0" to "7" shown in FIG. 7 become noise representing a portion of the lipid droplet image seen in the foreground binary image of rank "6." Therefore, in the example shown in FIG. 7, the images corresponding to the indices "0" to "7" shown in FIG. 7 were excluded from the detection target area. On the other hand, although lipid droplets can be detected by passing through index "39," it is not possible to classify them according to rank. Therefore, in the example shown in Figure 7, the image corresponding to index "39" shown in Figure 7 was also excluded from the detection target area. In the examples, it was possible to detect lipid droplets contained in original pathological tissue images (color images) by performing reinforcement learning and using a small amount of data, without using large amounts of data as in deep learning.

[0077] As described above, the lipid droplet detection device 1 of the first embodiment makes it possible to detect lipid droplet images seen in original pathological tissue images in a hierarchical manner by optimizing the combination of size and rank of lipid droplet candidate regions (squares) using reinforcement learning.

[0078] As mentioned above, there is a need for the development of technology that can analyze original pathological tissue images containing complex images and support pathologists. According to the lipid droplet detection device 1 of the first embodiment, in order to determine whether or not lipid droplets are contained in the image shown in Fig. 11(A), for example, a pathologist does not need to check the presence or absence of lipid droplets throughout the entire image shown in Fig. 11(A), but can simply check whether or not lipid droplets are present within the multiple squares shown in Fig. 11(B). In other words, according to the lipid droplet detection device 1 of the first embodiment, the burden on the pathologist can be significantly reduced, and support for the pathologist is possible.

[0079] Second Embodiment The lipid droplet detection device 1 of the second embodiment is configured in the same manner as the lipid droplet detection device 1 of the first embodiment shown in FIG. 1, except for the points that will be described later.

[0080] As described above, in the example shown in Figure 1 (an example of the lipid droplet detection device 1 of the first embodiment), the lipid droplet candidate extraction unit 17 determines the size and rank ("Rank 3" to "Rank 6") of the lipid droplet candidate region (square) by performing Q-learning as reinforcement learning. On the other hand, in an example of the lipid droplet detection device 1 of the second embodiment, the lipid droplet candidate extraction unit 17 determines the size and rank ("Rank 3" to "Rank 6") of the lipid droplet candidate region (square) by performing reinforcement learning other than Q-learning (e.g., Sarsa or Monte Carlo methods).

[0081] As described above, the lipid droplet detection device, lipid droplet detection method, and program of the present invention have been described with reference to the drawings, but the lipid droplet detection device, lipid droplet detection method, and program of the present invention are not limited to the above-described embodiments, and can be modified as appropriate within the scope of the present invention. The configurations of the examples of the above-described embodiments may be combined as appropriate. In each example of the above-described embodiment, the processing performed in the lipid droplet detection device 1 has been described as software processing performed by executing a program, but the processing performed in the lipid droplet detection device 1 may be processing performed by hardware. Alternatively, the processing performed in the lipid droplet detection device 1 may be processing that combines both software and hardware. [Explanation of symbols]

[0082] 1... lipid droplet detection device, 11... original image acquisition unit, 12... R image creation unit, 13... G image creation unit, 14... cell nucleus image extraction unit, 15... foreground binary image creation unit, 16... background binary image creation unit, 17... lipid droplet candidate extraction unit, 18... discrimination unit

Claims

1. an original image acquisition unit for acquiring an original pathological tissue image; an R image creation unit that creates an R image from the original pathological tissue image acquired by the original image acquisition unit; A G image creation unit that creates a G image, which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced, from the original pathological tissue image acquired by the original image acquisition unit; a cell nucleus image extraction unit that creates a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei included in the original pathological tissue image, from the R image created by the R image creation unit; and a foreground binary image creation unit that creates a second monochrome binary image from the G image created by the G image creation unit, converts black pixels included in the second monochrome binary image to white pixels and converts white pixels included in the second monochrome binary image to black pixels according to a rank that is the number of white pixels present in a filter area formed by a plurality of pixels included in the second monochrome binary image, to create a third monochrome binary image, and creates a foreground binary image by subtracting the first monochrome binary image created by the cell nucleus image extraction unit from the third monochrome binary image; A background binary image creation unit that creates a background binary image in which the lipid droplets are extracted as white regions from the G image created by the G image creation unit; A lipid droplet detection device comprising: a lipid droplet candidate extraction unit that performs an AND operation by aligning the foreground binary image created by the foreground binary image creation unit with the background binary image created by the background binary image creation unit to create an AND image according to the rank; and a lipid droplet candidate extraction unit that cuts out a square circumscribing the lipid droplet included in the AND image as a lipid droplet candidate region. The lipid droplet candidate extraction unit determines the size of the square and the rank by performing reinforcement learning, In the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through which route in the reinforcement learning environment, the plurality of lipid droplet candidate regions cut out by the lipid droplet candidate extraction unit will overlap more with each other. The Q value of the reinforcement learning is updated based on the reward of the reinforcement learning. The lipid droplet detection device, wherein the start and end positions of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower and upper limits of the size of the square.

2. The lipid droplet detection device according to claim 1, further comprising a discrimination unit that determines whether the lipid droplet is included in the square cut out by the lipid droplet candidate extraction unit.

3. the foreground binary image creation unit creates the third monochrome binary image according to the rank determined by performing the reinforcement learning; The lipid droplet detection device according to claim 1 or 2, wherein the background binary image creation unit creates the background binary image regardless of the rank determined by performing the reinforcement learning.

4. The foreground binary image creation unit creating the second monochrome binary image by applying an error diffusion method to the G image created by the G image creation unit; converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) where three or more white pixels exist into white pixels, and converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) where less than three white pixels exist into black pixels, thereby creating the third monochrome binary image of "Rank 3"; converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having four or more white pixels to white pixels, and converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having less than four white pixels to black pixels, thereby creating the third monochrome binary image of "rank 4"; converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having five or more white pixels to white pixels, and converting all pixels in the filter area consisting of nine pixels (3 pixels x 3 pixels) and having less than five white pixels to black pixels, thereby creating the third monochrome binary image of "rank 5"; All pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having 6 or more white pixels are converted to white pixels, and all pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 6 white pixels are converted to black pixels. The third monochrome binary image of "rank 6" is created by converting the pixels in the filter area consisting of 9 pixels (3 pixels x 3 pixels) and having less than 6 white pixels to black pixels. The lipid droplet detection device according to claim 3.

5. the movement start position of the reinforcement learning agent in the environment is a position in the environment where the size of the square is 2500 pixels (50 pixels x 50 pixels) and the rank is "Rank 6", The movement end position of the reinforcement learning agent in the environment is a position in the environment where the size of the square is 25 pixels (5 pixels x 5 pixels) and the rank is "rank 3". The lipid droplet detection device according to claim 4.

6. the position of the reinforcement learning agent in the environment at each time corresponds to a state of the reinforcement learning; The lipid droplet detection device according to claim 5 , wherein the movement of the reinforcement learning agent through the environment corresponds to the reinforcement learning behavior.

7. The lipid droplet detection device according to claim 6, wherein penalty areas are set in three locations in the environment.

8. The background binary image creation unit performs adaptive binarization processing on the G image created by the G image creation unit using a 225-pixel (15 pixels x 15 pixels) average value filter, applies a 9-pixel (3 pixels x 3 pixels) average value filter, and performs opening processing. The lipid droplet detection device according to claim 1 or 2, wherein the background binary image is created by performing the following.

9. An original image acquisition step in which the lipid droplet detection device acquires an original pathological tissue image; An R image creation step in which the lipid droplet detection device creates an R image from the pathological tissue original image acquired in the original image acquisition step; A G image creation step in which the lipid droplet detection device creates a G image which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced from the original pathological tissue image acquired in the original image acquisition step; A cell nucleus image extraction step in which the lipid droplet detection device creates a first monochrome binary image, which is a monochrome binary image in which cell nuclei contained in the original pathological tissue image are extracted from the R image created in the R image creation step; The lipid droplet detection device creates a second monochrome binary image from the G image created in the G image creation step, and converts black pixels included in the second monochrome binary image to white pixels according to a rank that is the number of white pixels present in a filter area composed of a plurality of pixels included in the second monochrome binary image, and converts white pixels included in the second monochrome binary image to black pixels. A foreground binary image creation step is performed by subtracting the first monochrome binary image created in the cell nucleus image extraction step from the third monochrome binary image to create a foreground binary image. A background binary image creation step in which the lipid droplet detection device creates a background binary image in which the lipid droplets are extracted as white areas from the G image created in the G image creation step; The lipid droplet detection device aligns the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step to perform an AND operation to create an AND image according to the rank, and a lipid droplet candidate extraction step of cutting out a square circumscribing the lipid droplet included in the AND image as a lipid droplet candidate region. In the lipid droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, In the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through which route in the reinforcement learning environment, the plurality of lipid droplet candidate regions cut out in the lipid droplet candidate extraction step will overlap more with each other. The Q value of the reinforcement learning is updated based on the reward of the reinforcement learning. The lipid droplet detection method, wherein the start and end positions of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower and upper limits of the size of the square.

10. To the computer an original image acquisition step of acquiring an original pathological tissue image; an R image creation step of creating an R image from the original pathological tissue image acquired in the original image acquisition step; A G image creation step of creating a G image which is a lipid droplet-enhanced image in which lipid droplets contained in the original pathological tissue image are enhanced from the original pathological tissue image acquired in the original image acquisition step; a cell nucleus image extraction step of creating a first monochrome binary image, which is a monochrome binary image obtained by extracting cell nuclei included in the original pathological tissue image, from the R image created in the R image creation step; a foreground binary image creation step of creating a second monochrome binary image from the G image created in the G image creation step, converting black pixels included in the second monochrome binary image to white pixels and converting white pixels included in the second monochrome binary image to black pixels according to a rank that is the number of white pixels present in a filter area formed by a plurality of pixels included in the second monochrome binary image, to create a third monochrome binary image, and creating a foreground binary image by subtracting the first monochrome binary image created in the cell nucleus image extraction step from the third monochrome binary image; A background binary image creation step of creating a background binary image in which the lipid droplets are extracted as white regions from the G image created in the G image creation step; A program for executing a fat droplet candidate extraction step in which an AND image according to the rank is created by performing an AND operation by aligning the foreground binary image created in the foreground binary image creation step with the background binary image created in the background binary image creation step, and a square circumscribing the fat droplet included in the AND image is cut out as a fat droplet candidate region. In the lipid droplet candidate extraction step, the size of the square and the rank are determined by performing reinforcement learning, In the reinforcement learning, one of the horizontal axis and the vertical axis is set as the size of the square, and the other of the horizontal axis and the vertical axis is set as the rank. If the lipid droplet detection device as an agent of the reinforcement learning passes through which route in the reinforcement learning environment, the plurality of lipid droplet candidate regions cut out in the lipid droplet candidate extraction step will overlap more with each other. The Q value of the reinforcement learning is updated based on the reward of the reinforcement learning. A program in which the movement start position and movement end position of the reinforcement learning agent in one direction of the horizontal axis and the vertical axis of the environment correspond to the lower limit and upper limit of the size of the square.

Images