Color conversion device, color conversion method, program, training device, image judgment device, and color conversion system

Abstract

The color conversion device disclosed herein includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and a conversion unit that converts a captured image that reflects color characteristics specific to the second imaging device into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

Description

[Technical Field]

[0001] The present disclosure relates to a color conversion device, a color conversion method, a program, a training device, an image assessment device, a machine learning model, and a color conversion system. [Background technology]

[0002] It is common to capture an image of a subject using an imaging device and generate an image. Even when images are generated using the same imaging device, the color tone of the generated image may change if the conditions at the time of capture, such as lighting, change. For this reason, there is a demand for a technology that can align the color tones of multiple images.

[0003] Patent Document 1 discloses a color correction device that performs color correction on image data of biological tissues obtained by imaging with a medical imaging device, to make the image suitable for display on a color monitor. Patent Document 2 discloses an endoscope device that records a color chart during an examination and corrects the color image during the next examination based on the color difference between a temporary color chart generated during the next examination and the color chart used during the previous examination. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 166697 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-54758 Summary of the Invention [Problem to be solved by the invention]

[0005] When an image is generated by capturing an image of a subject using an imaging device, the generated image generally reflects color characteristics specific to the imaging device. As a result, even if images of the same subject are captured using different imaging devices, the color tones may differ.

[0006] The present disclosure aims to provide a color conversion device, a color conversion method, a program, a training device, an image assessment device, and a machine learning model that can correct images generated by different imaging devices and bring color tones closer together. [Means for solving the problem]

[0007] A color conversion device according to one embodiment of the present disclosure includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and a conversion unit that converts a captured image that reflects color characteristics specific to the second imaging device into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

[0008] A color conversion method according to one embodiment of the present disclosure is a color conversion method executed by a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and the color conversion method acquires a captured image expressed in the second color space, converts the captured image into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

[0009] A program according to one embodiment of the present disclosure is stored on a non-transitory storage medium, and is executed by a computer of a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space. The program causes the computer to execute the following steps: acquiring a captured image expressed in the second color space; converting the captured image into a reference image expressed in the reference color space according to the second conversion rule; and converting the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

[0010] A training device according to one embodiment of the present disclosure includes an acquisition unit that acquires a training dataset including training images expressed in a first color space and annotations to be judged, and a training unit that uses the training dataset to train a machine learning model that judges images expressed in the first color space, wherein the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in the second color space into an image expressed in the reference color space, and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

[0011] An image assessment device according to one embodiment of the present disclosure includes an acquisition unit that acquires an image expressed in a first color space, and a assessment unit that assesses an image expressed in the first color space using a trained machine learning model that assesses images expressed in the first color space, wherein the trained machine learning model is trained using a training dataset that includes training images expressed in the first color space and an annotation to be assessed, and the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in a second color space into an image expressed in a reference color space, and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

[0012] A machine learning model according to one embodiment of the present disclosure is a machine learning model that causes a computer to acquire an image expressed in a first color space and judge the image expressed in the first color space, wherein the machine learning model is trained using a training dataset that includes training images expressed in the first color space and an annotation to be judged, and the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in the second color space into an image expressed in the reference color space, and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

[0013] A color conversion system according to one embodiment of the present disclosure is a color conversion system comprising the above-described color conversion device and a conversion rule generation device, wherein the conversion rule generation device generates the inverse conversion rule and the second conversion rule, and the color conversion device acquires the generated inverse conversion rule and the second conversion rule.

[0014] A color transformation system according to one embodiment of the present disclosure is a color transformation system comprising the above-mentioned color transformation device, a standard transformation rule generation device, and an inverse transformation rule generation device, wherein the transformation rule generation device generates the first transformation rule and the second transformation rule, the inverse transformation rule generation device generates the inverse transformation rule of the first transformation rule, and the color transformation device acquires the generated inverse transformation rule and the second transformation rule.

[0015] A color conversion device according to one embodiment of the present disclosure includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image that reflects color characteristics specific to a first imaging device into a reference image that is not dependent on the color characteristics of a specific imaging device, and a second conversion rule that converts an image that reflects color characteristics specific to a second imaging device into the reference image, and a conversion unit that converts the captured image that reflects color characteristics specific to the second imaging device into the reference image according to the second conversion rule, and converts the reference image into a converted image having color characteristics specific to the first imaging device according to the inverse conversion rule. [Effects of the Invention]

[0016] Images produced by different imaging devices can be corrected to bring the colors closer together. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a first example of a color conversion system. [Figure 2] FIG. 1 is a diagram showing an example of the functional configuration of a color conversion device in a first example. [Figure 3] FIG. 1 is a diagram showing an example of a functional configuration of an imaging device; [Figure 4] 1 is a flowchart illustrating an example of the operation of a color conversion device during a process for generating a conversion rule. [Figure 5] 1 is a flowchart illustrating an example of the operation of a color conversion device when performing color conversion processing. [Figure 6] FIG. 10 is a diagram showing an example of the configuration of a fourth modified example of a color conversion system. [Figure 7]FIG. 13 is a diagram showing an example of the functional configuration of a color conversion device according to Modification 4. [Figure 8] FIG. 1 is a diagram showing an example of a functional configuration of a conversion rule generation device. [Figure 9] FIG. 13 is a diagram showing an example of the configuration of a color conversion system according to a fifth modified example. [Figure 10] FIG. 1 is a diagram showing an example of the functional configuration of a standard conversion rule generation device. [Figure 11] FIG. 1 is a diagram showing an example of the functional configuration of an inverse conversion rule generation device. [Figure 12] FIG. 10 is a diagram showing an example of the configuration of a second example of a color conversion system. [Figure 13] FIG. 1 is a diagram showing an example of the functional configuration of a training device. [Figure 14] Flowchart for explaining an example of the operation of the training device [Figure 15] A flowchart for explaining an example of the operation of a color conversion system including a training device. [Figure 16] FIG. 10 is a diagram showing an example of the configuration of a third example of a color conversion system. [Figure 17] FIG. 1 is a diagram showing an example of a functional configuration of an image assessment device. [Figure 18] Flowchart for explaining an example of operation of the image assessment device [Figure 19] A flowchart for explaining an example of the operation of a color conversion system including an image determination device. [Figure 20] FIG. 10 is a diagram showing an example of the hardware configuration of each device. DETAILED DESCRIPTION OF THE INVENTION

[0018] <Summary> Multiple images may be generated using multiple image capture devices. Each image capture device has its own unique color characteristics, and images of the same subject may be generated with different color tones.

[0019] When an observer views such multiple images, differences in color tone between the images can affect the observation. For example, if an observer is accustomed to the color tone of images generated by a specific imaging device, the observer may feel uncomfortable when viewing images generated by another imaging device due to the difference in color tone.

[0020] The present disclosure relates to a color conversion device that performs color conversion processing to bring the color tone of an image generated by an imaging device other than a specific imaging device, among images generated by multiple imaging devices, closer to the color tone of an image generated by the specific imaging device.

[0021] The color conversion device of the present disclosure can be applied to, for example, the following cases. Pathological diagnosis is generally performed in which an observer such as a pathologist examines a pathological image created by collecting tissue from a subject to determine the presence or absence of a lesion and its type. A pathological image is, for example, an image obtained by taking a magnified photograph of a specimen obtained by collecting, fixing, staining, and thinly slicing a portion of the subject's tissue. A pathological image is generated by photographing the specimen using an imaging device with a magnification function, such as a microscope, placed in a hospital.

[0022] When an observer observes a pathology image and makes a diagnosis, the color or intensity of the stained tissue can be an important clue. When an observer makes a diagnosis using pathology images generated by various imaging devices, the color tone of the pathology image may change due to the color characteristics specific to the imaging device, which may adversely affect the diagnosis.

[0023] One way to resolve this situation is to correct the image generated by the imaging device to, for example, an image with the original color tone of the stained specimen, so that the color tone specific to the imaging device is not reflected in the image observed by the observer.

[0024] However, observers who have observed many pathological images with color tones specific to a particular imaging device may find the corrected color tones unnatural. Here, a specific imaging device refers to an imaging device that generates images that the observer has observed many times in the past. In such cases, it becomes difficult for the observer to make a diagnosis based on their past experience. It is desirable for the annotation work, in which the observer adds diagnostic information to images, to be performed using color tones that the observer is familiar with. Therefore, there is a concern that annotation work using images with unfamiliar color tones due to correction will significantly reduce work efficiency.

[0025] In addition, pathology image analysis AI (Artificial Intelligence) has been developed to assist observers in diagnosing new pathology images by using a trained machine learning model that has been trained using a large number of pathology images and the label data corresponding to each pathology image.

[0026] In such pathology image analysis AI, there are two methods for training machine learning models: one is to use pathology images of various color tones, and the other is to uniformly train on pathology images of one color tone. These methods are used separately or in combination depending on the purpose of the pathology image analysis AI and the images used. For example, training on pathology images of various color tones is thought to improve the robustness of the trained machine learning model. On the other hand, training on uniformly trained pathology images of one color tone is thought to improve learning efficiency.

[0027] In consideration of such cases, the color conversion device disclosed herein converts images of various color tones generated by various imaging devices into images of a specific color tone based on color characteristics unique to a specific imaging device. Here, the specific color tone may be a color tone dependent on a specific imaging device that generates images that an observer has observed many times in the past. Alternatively, the specific color tone may be a color tone dependent on a specific imaging device that generates images that a machine learning model of an image analysis AI has learned many times in the past. This allows observers who have observed many images of specific color tones dependent on a specific imaging device to make observations and diagnoses by utilizing their past knowledge. Furthermore, by correcting images of various color tones into images of specific color tones dependent on a specific imaging device and using them as training data for a machine learning model of an image analysis AI, accurate diagnostic assistance can be provided for new pathological images generated by a specific imaging device.

[0028] Although the following description will be directed to an apparatus and method for handling pathological images, the present disclosure is not limited thereto. The apparatus and method of the present disclosure can handle various images other than pathological images, as long as they are digitized images.

[0029] The color conversion device of the present disclosure and a color conversion system including the color conversion device will be described in detail below with specific examples.

[0030] <First example> First, a first example of a color conversion system including a color conversion device 10 according to an embodiment of the present disclosure will be described.

[0031] [Configuration example] 1 is a diagram showing an example of the configuration of a first example of a color conversion system. The color conversion system 100 according to the first example includes a color conversion device 10, a plurality of imaging devices 20, and an observation device 30.

[0032] Each of the multiple imaging devices 20 has a camera that captures an image of a subject and generates a digital image. More specifically, the multiple imaging devices 20 are devices that generate pathological images. In the present disclosure, the number of multiple imaging devices is not particularly limited, but the following description will be given of a case where the multiple imaging devices 20 include a first imaging device 20_1 and a second imaging device 20_2, as shown in FIG. 1 .

[0033] In the following description, the first imaging device 20_1 and the second imaging device 20_2 may be collectively referred to as imaging device 20, or either the first imaging device 20_1 or the second imaging device 20_2 may be simply referred to as imaging device 20.

[0034] The first imaging device 20_1 generates an image expressed in a first color space. The second imaging device 20_2 generates an image expressed in a second color space. In this embodiment, the first color space or the second color space may each be an RGB color space. In the following description, the image generated by the first imaging device 20_1 will be referred to as a first image, and the image generated by the second imaging device 20_2 will be referred to as a second image.

[0035] The first image reflects color characteristics specific to the first imaging device. The second image reflects color characteristics specific to the second imaging device. The first image and the second image have different color tones. In this way, each imaging device included in the multiple imaging devices 20 generates an image reflecting a different color characteristic.

[0036] In the following description, it is assumed that the first imaging device 20_1 is a specific imaging device, that is, an imaging device that generates images that the observer has observed many times in the past.

[0037] The color conversion device 10 does not perform color conversion processing on the first image acquired from the first imaging device 20_1 among the multiple imaging devices 20, but performs color conversion processing on the second image acquired from the second imaging device 20_2. In this case, the color conversion processing is processing for converting an image reflecting color characteristics specific to the second imaging device 20_2 into an image reflecting color characteristics specific to the first imaging device 20_1. In the following description, an image subjected to color conversion processing by the color conversion device 10 will be referred to as a converted image.

[0038] The observation device 30 is a device for an observer to observe a pathological image generated by the first imaging device 20_1 or the second imaging device 20_2 and perform a diagnosis, etc. The observation device 30 is, for example, a type of computer equipped with a display device, such as a PC (Personal Computer) or a tablet terminal.

[0039] The observation device 30 acquires and displays a first image from the first imaging device 20_1. The observation device 30 also acquires and displays a converted image obtained by performing color conversion processing on the second image generated by the second imaging device 20_2 from the color conversion device 10. This allows the observer to observe an image that reflects color characteristics specific to the first imaging device 20_1, regardless of which imaging device among the multiple imaging devices 20 has generated the image.

[0040] Fig. 2 is a diagram showing an example of the functional configuration of the color conversion device 10 in the first example. As shown in Fig. 2, the color conversion device 10 includes a communication unit 11, a storage unit 12, and a control unit 13. The color conversion device 10 is a type of computer, such as a PC, a workstation, or a tablet terminal.

[0041] The communication unit 11 transmits and receives various types of information to and from the imaging device 20 or the observation device 30 via wired or wireless communication.

[0042] The storage unit 12 stores various types of information necessary for the color conversion process performed by the color conversion device 10. The information stored in the storage unit 12 includes, for example, information about the conversion rules used in the color conversion process, and images before and after the color conversion process is performed. The storage unit 12 has a conversion rule storage unit 121 that stores various conversion rules.

[0043] The control unit 13 controls the operation of the color conversion device 10. The control unit 13 includes an acquisition unit 131, a conversion rule generation unit 132, and a conversion unit 133.

[0044] 1, the acquisition unit 131 acquires the second image received by the communication unit 11 from the second imaging device 20_2.

[0045] The conversion rule generation unit 132 generates a conversion rule to be used in the color conversion process. The conversion rule is, for example, an ICC (International Color Consortium) profile, which indicates the correspondence between the color values ​​of an image before conversion and the color values ​​of an image after conversion.

[0046] More specifically, the conversion rule generating unit 132 first generates an ICC profile that converts an image generated by the image capturing device 20 into a reference image that does not depend on the color characteristics of the image capturing device 20 .

[0047] In this embodiment, the reference image is an image that is expressed in a reference color space and does not depend on the color characteristics of the image capture device 20. The reference color space is, for example, the well-known L * a * b * The reference color space may be a color space or an XYZ color space. The reference color space may be called a profile connection space. In the example shown in FIG. 1 , the conversion rule generation unit 132 generates a first conversion rule for converting an image generated by the first imaging device 20_1 and reflecting color characteristics specific to the first imaging device 20_1 into a reference image, and a second conversion rule for converting an image generated by the second imaging device 20_2 and reflecting color characteristics specific to the second imaging device 20_2 into the reference image.

[0048] The first conversion rule is a rule for converting an image expressed in a first color space and reflecting color characteristics specific to the first imaging device 20_1 into a reference image expressed in a reference color space and independent of the color characteristics of the imaging device 20. The conversion rule generation unit 132 determines the first conversion rule based on the correspondence between the color values ​​of each pixel in the first chart image and the colorimetric values, using a first chart image acquired by the first imaging device 20_1 photographing a first color chart and colorimetric values ​​of the first color chart acquired in advance. Alternatively, if the original color of an object appearing in the image reflecting color characteristics specific to the first imaging device 20_1 is known, the conversion rule generation unit 132 may generate the first conversion rule so as to match or approximate the color value of each pixel in the reference image to the original color of the object. Alternatively, the conversion rule generation unit 132 may generate the first conversion rule using the color values ​​of each color patch in the first chart image and colorimetric values ​​of each color patch of the first color chart acquired in advance.

[0049] The first conversion rule is generated in the form of a look-up table (LUT), for example. Alternatively, the first conversion rule may be generated in the form of a matrix, for example. Table 1 shows an example of the first conversion rule expressed in the LUT format. In Table 1, the values ​​(r1, g1, b1) indicated by RGB on the left side of Table 1 and the values ​​(x, g, b1) indicated by XYZ in the same row are 1, y1, z1) correspond to each other. [Table 1]

[0050] The second conversion rule is a rule for converting an image expressed in the second color space and reflecting color characteristics specific to the second imaging device 20_2 into a reference image expressed in the reference color space and independent of the color characteristics of the imaging device 20. The conversion rule generation unit 132 determines the second conversion rule based on the correspondence between the color values ​​of each pixel in the second chart image and the colorimetric values, using a second chart image acquired by the second imaging device 20_2 photographing a second color chart and colorimetric values ​​of the second color chart acquired in advance. Alternatively, when the original color of an object appearing in the image reflecting color characteristics specific to the second imaging device 20_2 is known, the conversion rule generation unit 132 may generate the second conversion rule so as to match the color value of each pixel in the reference image to the original color of the object.

[0051] Like the first transformation rule, the second transformation rule may be generated in the form of an LUT or a determinant, for example.

[0052] The first chart image and the second chart image may be prepared in advance by, for example, an administrator of the color conversion system 100, or may be generated by the first imaging device 20_1 and the second imaging device 20_2 as necessary. The colorimetric values ​​of each color patch of the first color chart included in the first chart image and the colorimetric values ​​of each color patch of the second color chart included in the second chart image are known. Before the conversion rule generation unit 132 generates a conversion rule, the color conversion device 10 receives the color values ​​of each color patch in the first chart image and the colorimetric values ​​of each color patch of the first color chart included in the first chart image.

[0053] The first chart image and the second chart image may be different images. More specifically, the first color chart and the second color chart may include different color patches, for example, color patches of different colors.

[0054] The first color chart and the second color chart may each include at least red, green, blue, and white color patches.

[0055] More specifically, the red color patches included in the first and second color charts have a transmission spectrum with a peak wavelength of 620 nm to 680 nm and a half-width of 100 nm or less. The green color patches included in the first and second color charts have a transmission spectrum with a peak wavelength of 510 nm to 540 nm and a half-width of 80 nm or less. The blue color patches included in the first and second color charts have a transmission spectrum with a peak wavelength of 450 nm to 472 nm and a half-width of 80 nm or less. The white color patches included in the first and second color charts can be made of a base material that is not colored with a dye or pigment. Examples of base material materials include glass, film, acrylic resin, polycarbonate, gelatin, and paper.

[0056] By including such color patches in the first color chart and the second color chart, the coordinates of the color patches of each color on the xy chromaticity diagram can be made to match or approximate the coordinates of each color defined in the BT.2020 standard, thereby enabling suitable color calibration to be performed when the BT.2020 standard is applied to the imaging device 20.

[0057] The colors of red, green, and blue included in the first color patch are not limited to the above examples, and may be in the following ranges on the xy chromaticity diagram, for example.

[0058] The color of the red color patch included in the first color patch may be within the area surrounded by coordinates (0.735, 0.265), (0.627, 0.372) on the xy chromaticity diagram, the curve of the blackbody locus BL where the color deviation is -0.01 between color temperatures of 1200 K or more and 6000 K or less, and (0.520, 0.174). Note that the blackbody locus refers to the locus on the chromaticity diagram that shows how a completely black object changes with absolute temperature.

[0059] The color of the green color patch included in the first color patch may be within the area surrounded by the coordinates (0.005, 0.587), (0.014, 0.750), (0.074, 0.834), (0.155, 0.806), and (0.345, 0.652) on the xy chromaticity diagram, and the curve portion of the blackbody locus BL where the color deviation is +0.02 between color temperatures of 6000 K or more and 7000 K or less.

[0060] The color of the blue color patch included in the first color patch may be within the area surrounded by the coordinates on the xy chromaticity diagram: (0.174, 0.005), (0.131, 0.046), (0.073, 0.185), the straight line connecting the points on the blackbody locus BL where the color deviation is +0.02 and -0.01 at a color temperature of 15000K, the curve portion on the blackbody locus BL where the color deviation is -0.01 at color temperatures from 15000K to 7000K, and (0.280, 0.054).

[0061] By using a chart image including such color patches, it is possible to generate a conversion rule that can accommodate a wide color gamut.

[0062] Furthermore, as an example of a color chart other than those described above, for example, the following color chart may be adopted to make the pathological images handled by the color conversion system 100 easier to see. Such a first color chart and second color chart each include at least a first color patch, and the coordinate point of the first color patch is within a triangular area formed by connecting three points on the xy chromaticity diagram, namely, (0.200, 0.070), (0.450, 0.280), and (0.310, 0.340) with straight lines.

[0063] Only one of the first color chart and the second color chart may include such a first color patch. According to this configuration, when a pathology image reflecting the color characteristics of the first image capture device 20_1 is significantly different (has a strong peculiarity) from a stained actual specimen, and a pathology image reflecting the color characteristics of the second image capture device 20_2 is not significantly different (has a weak peculiarity) from the actual specimen, by using the first color patch for the first color chart, it is possible to obtain an effect that a highly accurate conversion coefficient can be generated for the first image capture device with a strong peculiarity, and a conversion coefficient can be efficiently generated for the second image capture device with a weak peculiarity.

[0064] If the pathology image is an image of cells stained with hematoxylin and eosin (HE), the coordinate points of the first color patch can be set within the above range to bring the color of the color patch closer to the characteristic color of the pathology image. This makes it possible to generate a first conversion rule or a second conversion rule that enables conversion to bring the color closer to the characteristic color of the pathology image. Therefore, the first conversion rule or the second conversion rule can perform color correction so that the converted image is closer to the color of the actual object (stained cells).

[0065] The coordinate points of the first color patch may be within a pentagonal area formed by connecting five points (0.310, 0.340), (0.350, 0.300), (0.350, 0.200), (0.260, 0.150), and (0.280, 0.250) in that order on the xy chromaticity diagram. Having the coordinate points of the first color patch within this pentagonal area allows for more accurate color correction.

[0066] The coordinate points of the first color patch may be within a rectangular area formed by connecting four points (0.311, 0.334), (0.347, 0.210), (0.270, 0.162), and (0.292, 0.216) on the xy chromaticity diagram with straight lines. Having the coordinate points of the first color patch within this rectangular area allows for even more accurate color correction.

[0067] The first color chart and the second color chart may further include a second color patch that is a different color from the first color patch. In this case, the coordinate points of both the first color patch and the second color patch may be within any of the above-mentioned regions on the xy chromaticity diagram. By having the coordinate points of multiple color patches within any of the above-mentioned regions, it is possible to generate a first conversion rule or a second conversion rule that can perform color correction on an image having a color tone characteristic of a pathology image.

[0068] Furthermore, the conversion rule generation unit 132 generates an inverse conversion rule for performing an inverse conversion of the first conversion rule based on the first conversion rule generated as described above. The inverse conversion rule is a rule for converting a reference image that is expressed in a reference color space and does not depend on the color characteristics of the imaging device into an image that reflects color characteristics specific to the first imaging device 20_1. The inverse conversion rule can be created, for example, by the following method. Note that the following describes a case where the reference color space is an XYZ color space and the conversion rule is expressed in LUT format.

[0069] The inverse transformation rule in LUT format can be created, for example, by at least the following three steps: a first step of preparing a set of grid points in a reference color space, a second step of determining the relationship between each grid point in the set prepared in the first step and the grid points in the reference color space in the LUT, which is the first transformation rule for transforming from the first color space to the reference color space, and a third step of determining, from the relationship determined in the second step, grid points in the first color space that correspond to each grid point in the set prepared in the first step.

[0070] An example of a method for creating an inverse transformation rule in the LUT format will be described in detail below. First, the transformation rule generation unit 132 creates a set of lattice points in the XYZ color space. In the following, an example is shown in which the definition described in ISO 15076-1:2010 is used as the LUT format. In this example, a three-dimensional LUT format is used as the LUT format. A three-dimensional LUT is defined as (X i ,Y j ,Z k) When i, j, and k are specified, the converted RGB values ​​corresponding to each specified value are defined. Table 2 shows an example of a set of lattice points in the XYZ color space. Table 2 shows the case where the range that XYZ can take is divided equally for each of X, Y, and Z. If the range that XYZ can take is divided into 8 equal parts, the number of lattice points will be 9 to the power of 3. min and max indicate the minimum and maximum lattice point numbers, respectively, in the range that XYZ can take. If it is divided into 8 equal parts, min is 0 and max is 8. [Table 2]

[0071] X min , Y min , Z min The values ​​of may be preset to a predetermined value, for example, 0. max , Y max , Z max The values ​​of X may be preset to a predetermined value, for example 128. min , Y min , Z min The value of X may be the minimum value in the XYZ color space that can be obtained when each color value in the RGB color space is changed from the minimum value (for example, 0) to the maximum value (for example, 255). max , Y max , Z max The value may be the maximum value in XYZ that can be obtained when each color value in the RGB color space is changed from the minimum value (for example, 0) to the maximum value (for example, 255).

[0072] Next, the conversion rule generation unit 132 plots the color values ​​of all lattice points in the XYZ color space that correspond to all lattice points in the RGB color space in the LUT of the first conversion rule in the XYZ color space, and creates a solid in the XYZ space that corresponds to the RGB color space.

[0073] Next, the conversion rule generation unit 132 checks where each grid point in the XYZ color space shown in Table 1 is located on the solid body. The conversion rule generation unit 132 performs interpolation as needed based on the correspondence between each grid point in the XYZ color space and a position on the solid body. The interpolation can be performed using a known method, such as a spline function or linear interpolation. If a grid point in the XYZ color space shown in Table 1 is located outside the solid body, the conversion rule generation unit 132 may replace the grid point so that it is located on the outermost or innermost part of the solid body. Examples of replacement methods include replacing the grid point with a point that minimizes the color difference or a point with the same hue. Alternatively, the grid point may be replaced with the outermost grid point in the solid body that minimizes the color difference with a grid point in the XYZ color space shown in Table 1. Alternatively, the grid point may be replaced with the outermost grid point in the solid body that has the same hue as a grid point in the XYZ color space shown in Table 1.

[0074] This allows the conversion rule generation unit 132 to generate an LUT that converts the grid points in the XYZ color space shown in Table 1 into grid points in the RGB color space, that is, an inverse conversion rule.

[0075] Although the method for creating an inverse conversion rule by conversion rule generation unit 132 has been specifically described above, the method for creating an inverse conversion rule in the present disclosure is not limited to the above-described method. For example, if the first conversion rule is expressed as a determinant (e.g., 3 × 3), conversion rule generation unit 132 can create an inverse conversion rule by calculating its inverse matrix.

[0076] Before the color conversion process is performed by conversion unit 133, conversion rule generation unit 132 may generate the first conversion rule, the second conversion rule, and the inverse conversion rule and store them in conversion rule storage unit 121 of storage unit 12. Alternatively, conversion rule generation unit 132 may perform the color conversion process and the process of generating a new conversion rule in parallel or successively.

[0077] The conversion unit 133 performs color conversion processing on the second image acquired from the second imaging device 20_2 using the first conversion rule, the second conversion rule, and the inverse conversion rule.

[0078] As described above, the color conversion process is a process of converting the second image, which is expressed in the second color space and reflects color characteristics specific to the second imaging device 20_2, into an image, which is expressed in the first color space and reflects color characteristics specific to the first imaging device 20_1. The conversion unit 133 first converts the second image acquired from the second imaging device 20_2 into a reference image, which is expressed in the reference color space and does not depend on the color characteristics of the imaging device 20, using the second conversion rule. Then, the conversion unit 133 converts the reference image into a converted image, which is expressed in the first color space and has color characteristics specific to the first imaging device 20_1, using the inverse conversion rule.

[0079] The converted image converted by the color conversion process is output to the observation device 30 via the communication unit 11. As a result, as described above, the observation device 30 acquires the first image generated by the first imaging device 20_1 and the converted image. The observer can observe the first image or the converted image displayed by the observation device 30 and make a diagnosis. Here, both the first image and the converted image have color characteristics specific to the first imaging device 20_1. Therefore, for an observer who has observed many first images captured by the first imaging device 20_1 in the past, the first image or the converted image displayed on the observation device 30 are images with familiar color tones because they have the color characteristics of the first imaging device 20_1. In this way, the color conversion system 100 can convert an image reflecting color characteristics different from those specific to a specific imaging device 20 into an image reflecting the color characteristics specific to the specific imaging device 20 so that an observer who is accustomed to seeing images reflecting the color characteristics specific to a specific imaging device 20 can easily observe the image.

[0080] [Example of operation] Next, a description will be given of an example of operation in the first configuration example of the color conversion system 100. Fig. 4 is a flowchart for explaining an example of operation of the conversion rule generation process in the color conversion system 100.

[0081] In step S1, the first imaging device 20_1 and the second imaging device 20_2 capture images of a first color chart and a second color chart, respectively, which are prepared in advance, to generate a first chart image and a second chart image, respectively. The generated first chart image and second chart image are acquired by the color conversion device 10.

[0082] In step S2, the conversion rule generation unit 132 of the color conversion device 10 generates a first conversion rule based on the first chart image and a second conversion rule based on the second chart image. The above-described methods may be appropriately adopted as the method for generating each conversion rule.

[0083] In step S3, the conversion rule generation unit 132 generates an inverse conversion rule based on the first conversion rule. The method for generating the inverse conversion rule may be any of the methods described above. The conversion rule and inverse conversion rule generated in steps S2 and S3 are stored in the conversion rule storage unit 121 of the color conversion device 10 and used in the color conversion process described below.

[0084] Next, a description will be given of an example of the operation of the color conversion process in the color conversion system 100. Fig. 5 is a flowchart for explaining an example of the operation of the color conversion process in the color conversion system 100.

[0085] In step S11, the acquisition unit 131 of the color conversion device 10 acquires a second image from the second imaging device 20_2.

[0086] In step S12, the conversion unit 133 of the color conversion device 10 converts the second image into a reference image using the second conversion rule.

[0087] In step S13, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image using the inverse conversion rule. The converted image generated in step S13 is sent to the observation device 30.

[0088] In step S14, the observation device 30 displays the converted image. As a result, even if the object to be observed is photographed by the second imaging device 20_2, the observer can observe the converted image as an image having color characteristics specific to the first imaging device 20_1. As a result, for an observer who is accustomed to observing images reflecting color characteristics specific to the first imaging device 20_1, observation is much easier than simply observing an image generated by the second imaging device 20_2 and reflecting color characteristics specific to the second imaging device 20_2.

[0089] <Modification of the First Example> (Modification 1) In the first example described above, the multiple imaging devices 20 include the first imaging device 20_1 and the second imaging device 20_2. In the present disclosure, the number of imaging devices 20 is not particularly limited, and the multiple imaging devices 20 may include imaging devices other than the first imaging device 20_1 and the second imaging device 20_2.

[0090] When the multiple imaging devices in the color conversion system include three or more imaging devices, the color conversion device acquires images of imaging devices other than a specific imaging device (first imaging device 20_1 in the example shown in FIG. 1 ) and performs color conversion processing on each image. In this case, before performing the color conversion processing, the color conversion device generates a conversion rule for converting an image generated by each imaging device into a reference image and an inverse conversion rule for converting the reference image into an image reflecting color characteristics specific to the specific imaging device. The color conversion device first converts an image generated by an imaging device other than the specific imaging device into the reference image using the conversion rule, and then uses the inverse conversion rule for converting the reference image into an image reflecting color characteristics specific to the specific imaging device, thereby generating a converted image having color characteristics specific to the specific imaging device.

[0091] When the multiple imaging devices in the color conversion system include three or more imaging devices, an imaging device other than the specific imaging device is an example of a second imaging device of the present disclosure, and a color space that represents an image generated by an imaging device other than the specific imaging device is an example of a second color space of the present disclosure.

[0092] (Variation 2) Furthermore, in the first example described above, the color conversion device 10 performs color conversion processing on the second image generated by the second imaging device 20_2. However, in the present disclosure, for example, an observation device may have the function of a color conversion device and perform color conversion processing. In this case, the observation device acquires the first image and the second image from the first imaging device and the second imaging device, respectively, and does not perform color conversion processing on the first image, but performs color conversion processing on the second image before displaying it. This allows an observer to observe the first image or the converted image displayed on the observation device and make a diagnosis, etc.

[0093] (Variation 3) In addition, in this disclosure, among the multiple imaging devices, an imaging device other than a specific imaging device, for example, a second imaging device, may have the function of a color conversion device and perform color conversion processing. In this case, the first imaging device directly transmits the generated first image to the observation device, and the second imaging device performs color conversion processing on the generated second image and then transmits the converted image to the observation device. This allows the observer to perform diagnosis, etc. by observing the first image or the converted image displayed on the observation device. In this case, the imaging device is a type of computer having a processor for performing color conversion processing.

[0094] The second image capturing device may obtain an inverse transformation rule for transforming a reference image into a transformed image from an external device, or may obtain the first chart image and information on the colorimetric values ​​of the first color chart and create the inverse transformation rule based on these.

[0095] Because pathological images tend to have large data volumes, the color conversion process may require a relatively large amount of processing power or time. By having an imaging device other than the specific imaging device perform the color conversion process, the processing power or processing time required for the color conversion process can be distributed among multiple imaging devices. Therefore, having an imaging device other than the specific imaging device perform the color conversion process is particularly effective when, for example, there are a relatively large number of imaging devices.

[0096] (Variation 4) In the first example described above, the color conversion device 10 has the conversion rule generation unit 132, which generates various conversion rules. However, in the present disclosure, the color conversion device 10 may acquire conversion rules from an external device. In this case, the color conversion device 10 does not need to have the conversion rule generation unit 132.

[0097] 6 is a diagram showing an example of the configuration of a color conversion system according to Modification 4. A color conversion system 100A according to Modification 4 includes a color conversion device 10A, a plurality of imaging devices 20, an observation device 30, and a conversion rule generation device 60.

[0098] Fig. 7 is a diagram showing an example of the functional configuration of a color conversion device 10A in Modification 4. As shown in Fig. 7, color conversion device 10A according to Modification 4 differs from color conversion device 10 of the first example shown in Fig. 2 in that it does not have a conversion rule generation unit. The functions and operations of each component of color conversion device 10A shown in Fig. 7 are similar to the functions and operations of each component described in the first example above.

[0099] 8 is a diagram showing an example of the functional configuration of conversion rule generating device 60. As shown in FIG. 8, conversion rule generating device 60 includes a communication unit 61, an acquisition unit 62, and a conversion rule generating unit 63.

[0100] The communication unit 61 transmits and receives various types of information to and from the color conversion device 10A and the multiple imaging devices 20 via wired or wireless communication.

[0101] The acquisition unit 62 acquires the first chart image and the second chart image for generating each conversion rule.

[0102] The conversion rule generation unit 63 generates a first conversion rule and a second conversion rule, and further generates an inverse conversion rule of the first conversion rule. The method of generating each conversion rule by the conversion rule generation unit 63 may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

[0103] The second transformation rule and the inverse transformation rule generated by the transformation rule generation device 60 are transmitted to the color transformation device 10A. The color transformation device 10A uses the second transformation rule and the inverse transformation rule acquired from the transformation rule generation device 60 to perform the color transformation process described in the first example above.

[0104] As explained above, in Modification 4, an example has been described in which the transformation rule generation device 60 external to the color transformation device 10A generates the second transformation rule and the inverse transformation rule, and the color transformation device 10A acquires the second transformation rule and the inverse transformation rule from the transformation rule generation device 60 and performs the color transformation process. According to Modification 4, the color transformation device 10A does not need to have the function of generating each transformation rule. This makes it possible to reduce the manufacturing costs and operating costs of the color transformation device 10A.

[0105] (Variation 5) In the above-described fourth modification, the conversion rule generation device 60 generates the first conversion rule and the second conversion rule, and generates the inverse conversion rule of the first conversion rule. In the present fifth modification, an example will be described in which a device for generating the first conversion rule and the second conversion rule and a device for generating the inverse conversion rule of the first conversion rule are provided separately.

[0106] 9 is a diagram showing an example of the configuration of a color transformation system according to Modification 5. A color transformation system 100B according to Modification 5 includes a color transformation device 10A, a plurality of imaging devices 20, an observation device 30, a standard transformation rule generation device 70, and an inverse transformation rule generation device 80.

[0107] The configuration and operation of color conversion device 10A are the same as those of the fourth modification.

[0108] 10 is a diagram showing an example of the functional configuration of the standard conversion rule generation device 70. As shown in FIG. 10, the standard conversion rule generation device 70 includes a communication unit 71, an acquisition unit 72, and a standard conversion rule generation unit 73.

[0109] The communication unit 71 transmits and receives various information to and from the plurality of image capture devices 20 and the inverse conversion rule generation device 80 via wired or wireless communication.

[0110] The acquisition unit 72 acquires the first chart image and the second chart image for generating each conversion rule.

[0111] The standard conversion rule generation unit 73 generates the first conversion rule and the second conversion rule. The method of generating the first conversion rule and the second conversion rule by the conversion rule generation unit 63 may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

[0112] The first conversion rule generated by the standard conversion rule generation device 70 is transmitted to the inverse conversion rule generation device 80. The second conversion rule generated by the standard conversion rule generation device 70 may be transmitted to the color conversion device 10A via the inverse conversion rule generation device 80, or may be transmitted to the color conversion device 10A without passing through the inverse conversion rule generation device 80.

[0113] 11 is a diagram showing an example of the functional configuration of the inverse conversion rule generating device 80. As shown in FIG. 11, the inverse conversion rule generating device 80 includes a communication unit 81, an acquisition unit 82, and an inverse conversion rule generating unit 83.

[0114] The communication unit 81 transmits and receives various information to and from the color conversion device 10A and the standard conversion rule generation device 70 via wired or wireless communication.

[0115] The acquiring unit 72 acquires at least the first conversion rule from the standard conversion rule generating device 70.

[0116] The inverse conversion rule generation unit 83 generates an inverse conversion rule of the first conversion rule based on the first conversion rule. The method of generating the inverse conversion rule by the inverse conversion rule generation unit 83 may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

[0117] The inverse transformation rules generated by the inverse transformation rule generation device 80 are transmitted to the color transformation device 10A. The color transformation device 10A then performs the color transformation process described in the first example above using the acquired second transformation rule and inverse transformation rule.

[0118] As explained above, in Modification 5, an example has been described in which a standard conversion rule generation device 70 external to color conversion device 10A generates the first conversion rule and the second conversion rule, an inverse conversion rule generation device 80 external to color conversion device 10A generates the inverse conversion rule of the first conversion rule, and color conversion device 10A acquires the second conversion rule and the inverse conversion rule to perform color conversion processing. According to Modification 5, color conversion device 10A does not need to have the function of generating each conversion rule. This allows for reductions in manufacturing and operating costs of color conversion device 10A.

[0119] (Variation 6) In the present disclosure, the multiple imaging devices may belong to the same facility, such as the same medical institution (such as a hospital or clinic) or the same research institute, or may belong to different facilities. Furthermore, the facilities to which the imaging devices belong may be established in different countries.

[0120] Because the number of pathologists who can make accurate diagnoses on pathological images is limited, such pathologists tend to receive a large number of requests for pathological image diagnoses from multiple hospitals, etc. Without the color conversion device of the present disclosure, the requested pathologist would have to make a diagnosis using pathological images in unfamiliar color tones. However, with a color conversion system including the color conversion device of the present disclosure, even in such cases, the pathologist can make a diagnosis using pathological images in familiar color tones.

[0121] (Variation 7) In the first example described above, an example of operation was described in which the color conversion device 10 performs the process of generating a conversion rule before the color conversion process. As described above, in the present disclosure, the color conversion device may perform the color conversion process and the process of generating a new conversion rule in parallel or consecutively.

[0122] For example, a color conversion device acquires, from a new imaging device other than a specific imaging device, a chart image generated by the imaging device by capturing an image of a color chart, information regarding the colorimetric values ​​of the color chart, and an image generated by the imaging device by capturing an object. Based on the chart image and the information regarding the colorimetric values, the color conversion device generates a conversion rule for converting an image that reflects color characteristics specific to the new imaging device into a reference image, and generates the reference image using the acquired image and the conversion rule. The color conversion device then converts the reference image using a previously generated inverse conversion rule for converting the reference image into an image that reflects color characteristics specific to the specific imaging device, thereby generating a converted image. In this way, when acquiring an image from a new imaging device that generates an image for which a conversion rule for converting to a reference image has not been generated, the conversion rule generation process and the color conversion process may be performed in parallel or consecutively.

[0123] For example, suppose a pathologist collects images generated by multiple imaging devices from multiple hospitals and performs diagnoses using these images. In this case, the pathologist may receive a request from a new hospital to perform a diagnosis using images. In such a case, by having the color conversion device acquire the chart image and the observation image generated by a new imaging device from the new hospital at approximately the same time, a converted image with a color tone familiar to the observer can be generated without any problems, even if a conversion rule for converting an image that reflects the color characteristics specific to the imaging device into a reference image has not been generated in advance.

[0124] <Second example> Next, a second example of a color conversion system including the color conversion device 10 according to an embodiment of the present disclosure will be described.

[0125] [Configuration example] 12 is a diagram showing an example of the configuration of a second example of a color conversion system. A color conversion system 200 according to the second example includes a color conversion device 10, multiple image capture devices 20, and a training device 40. The color conversion device 10 and the multiple image capture devices 20 are the same as those described in the first example above.

[0126] The training device 40 uses images (pathological images) acquired directly from the multiple imaging devices 20 or via the color conversion device 10 as training images to train a machine learning model to be trained. The machine learning model to be trained is a machine learning model that causes an image assessment device 50, described below, to acquire images that are expressed in a first color space and reflect color characteristics specific to the first imaging device 20_1, and assess the images that are expressed in the first color space and reflect color characteristics specific to the first imaging device 20_1. The training device 40 is a type of computer, such as a PC, a workstation, or a tablet terminal.

[0127] Determining an image expressed in the first color space and reflecting color characteristics specific to the first image capture device 20_1 means performing a determination process that helps an observer make a diagnosis using a pathology image, such as determining the presence or absence of a lesion in the pathology image or the location or type of the lesion. In order to train a machine learning model that can perform such a determination process, the training device 40 trains the machine learning model to be trained using a training dataset that includes training images expressed in the first color space and reflecting color characteristics specific to the first image capture device 20_1 and label data of the training images.

[0128] The label data of the training images includes, for example, annotations of the target of determination included in the training images. The target of determination is, for example, whether or not the pathology image includes a lesion. In this case, the label data includes, for example, mask data that conceals the lesion in the training image. Alternatively, the target of determination may be, for example, what type of tissue a site of interest included in the pathology image is classified into. Alternatively, if the pathology image includes a lesion, the target of determination may be, for example, what disease the lesion is caused by. The annotation is, for example, correct answer data indicating whether or not the pathology image includes a lesion (i.e., lesion segmentation), what type of tissue a site of interest included in the training image is classified into, or what disease the lesion included in the training image is caused by.

[0129] 13 is a diagram showing an example of the functional configuration of training device 40. Training device 40 includes a communication unit 41, an operation unit 42, an acquisition unit 43, a training unit 44, and a storage unit 45.

[0130] The communication unit 41 transmits and receives various types of information to and from the color conversion device 10 or the imaging device 20 via wired or wireless communication.

[0131] The operation unit 42 accepts an operation to add annotations to the first image and the converted image. The operation to add annotations to the first image and the converted image is performed by, for example, a pathologist.

[0132] The acquisition unit 43 acquires a training data set including training images expressed in a first color space and reflecting color characteristics specific to the first image capture device 20_1, and annotations to be determined. The training images include the first image received from the first image capture device 20_1 and the converted image received from the color conversion device 10.

[0133] The storage unit 45 stores a machine learning model 46 to be trained.

[0134] The training unit 44 uses the training data set to train a machine learning model 46 to be trained, which is to judge an image that is expressed in the first color space and reflects color characteristics specific to the first image capture device 20_1. By performing the training process using a sufficient number of images, it is possible to generate a trained machine learning model that can perform judgment processing with high accuracy.

[0135] Thus, in a second example of a color conversion system including a color conversion device according to the present disclosure, the training device performs training using images that reflect color characteristics specific to a specific imaging device, and the color conversion device converts images generated by an imaging device other than the specific imaging device into converted images that have color characteristics specific to the specific imaging device. This increases the number of images with specific color tones included in the training data. A trained machine learning model trained using such converted images can effectively assist pathologists in making accurate diagnoses of pathological images.

[0136] In the second example, as in the first example, the number of imaging devices is not limited to two. If sufficient processing capacity can be ensured, for example, the training device may have the function of a color conversion device and be able to perform conversion rule generation processing and color conversion processing.

[0137] [Example of operation of training device 40] FIG. 14 is a flowchart for explaining an example of the operation of the training process in the training device 40.

[0138] In step S21, the acquisition unit 43 of the training apparatus 40 acquires the first image transmitted from the first imaging device 20_1. The number of first images acquired by the training apparatus 40 may be multiple.

[0139] In step S22, the acquisition unit 43 acquires the converted image transmitted from the color conversion device 10. The training device 40 may acquire a plurality of converted images. The order in which steps S21 and S22 are performed may be reversed. Furthermore, steps S21 and S22 may be performed in parallel.

[0140] In step S23, the operation unit 42 accepts an input operation of an annotation by a pathologist or the like.

[0141] In step S24, the training unit 44 trains the machine learning model 46 to be trained based on the first image, the converted image, and the annotations.

[0142] The operations from step S21 to step S24 may be repeatedly performed.

[0143] With this operation, even if multiple imaging devices 20 generate images that reflect their own unique color characteristics, the training device 40 can train the machine learning model 46 to be trained using images with specific color tones. This eliminates the need for correction processing or setting of training parameters related to the color characteristics unique to each of the multiple imaging devices 20, making it possible to generate a trained machine learning model that can make accurate judgments with little effort.

[0144] 14, the training device 40 receives an input operation for an annotation via the operation unit 42, but the present disclosure is not limited to this. For example, an apparatus other than the training device may generate the annotation. In this case, the acquisition unit of the training device may acquire the annotation together with the first image and the converted image from an external apparatus.

[0145] In addition, in the present disclosure, the training device may perform a certain amount of training on the machine learning model to be trained by supervised learning using annotations, and then perform training by unsupervised learning using the first image and the converted image that do not have annotations. In this case, the effort required to prepare annotations can be reduced.

[0146] In the present disclosure, the training images may include a second image received from the second imaging device 20_2 and a converted image received from the color conversion device. The converted image is an image having color characteristics specific to the first imaging device 20_1. By including images that reflect the color characteristics of different imaging devices, i.e., images with different color tones, in the training images, a highly robust trained machine learning model can be generated.

[0147] [Example of operation of color conversion system 200] FIG. 15 is a flowchart for explaining an example of the operation of the color conversion system 200 including the training device 40.

[0148] In step S31, the color conversion device 10 acquires a second image from the second imaging device 20_2.

[0149] In step S32, the color conversion device 10 converts the second image into the reference image using the second conversion rule.

[0150] In step S33, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image that reflects color characteristics specific to the first imaging device 20_1, using the inverse conversion rule. The converted image generated in step S33 is transmitted to the training device 40.

[0151] In step S34, the training device 40 executes the training process described with reference to FIG.

[0152] As described above, in the color conversion system 200, the color conversion device 10 converts an image generated by the second imaging device 20_2 into a converted image having color characteristics specific to the first imaging device 20_1, and the training device 40 performs training using the converted image. This allows the training device 40 to easily increase the number of images with specific color tones included in the training data. Furthermore, a trained machine learning model trained using such converted images can suitably assist pathologists in making accurate diagnoses of pathological images.

[0153] <Third example> Next, a third example of a color conversion system including the color conversion device 10 according to an embodiment of the present disclosure will be described.

[0154] [Configuration example] 16 is a diagram showing an example of the configuration of a third example of a color conversion system. A color conversion system 300 according to the third example includes a color conversion device 10, multiple image capture devices 20, and an image assessment device 50. The color conversion device 10 and the multiple image capture devices 20 are the same as those described in the first example above.

[0155] The image assessment device 50 performs assessment processing to receive an output of an assessment result by inputting images (pathological images) acquired directly from the multiple imaging devices 20 or via the color conversion device 10 into a trained machine learning model. The trained machine learning model is a machine learning model generated by the training device 40 performing the training processing in the second example described above. In other words, the trained machine learning model is a machine learning model trained using images expressed in a first color space and reflecting color characteristics specific to the first imaging device 20_1. The image assessment device 50 is a type of computer such as a PC, a workstation, or a tablet terminal.

[0156] 17 is a diagram showing an example of the functional configuration of the image assessment device 50. The image assessment device 50 includes a communication unit 51, an acquisition unit 52, a assessment unit 53, and a storage unit .

[0157] The communication unit 51 transmits and receives various types of information to and from the color conversion device 10 or the imaging device 20 via wired or wireless communication.

[0158] The acquiring unit 52 acquires the first image from the first imaging device 20_1 and the converted image from the color conversion device 10. Both the first image and the converted image are images in which color characteristics specific to the first imaging device 20_1 are reflected.

[0159] The memory unit 54 stores a trained machine learning model 55.

[0160] The determination unit 53 determines an image that is expressed in the first color space and reflects color characteristics specific to the first imaging device 20_1, using the trained machine learning model 55. More specifically, the determination unit 53 inputs at least one of the newly acquired first image and the converted image into the trained machine learning model 55, and generates a determination result based on the output result. The determination result is output to a device used by a pathologist or the like to assist in diagnosis, such as a display device not shown in FIG. 16 .

[0161] Thus, in a third example of a color conversion system having a color conversion device according to the present disclosure, the image assessment device can accurately assess new images that reflect color characteristics specific to a specific imaging device using a trained machine learning model that has been trained using images that reflect color characteristics specific to a specific imaging device.

[0162] In the third example, as in the first or second example, the number of imaging devices is not limited to two. In addition, if sufficient processing capacity can be ensured, for example, the image assessment device may have the function of a color conversion device and be able to perform conversion rule generation processing and color conversion processing.

[0163] [Example of operation of image assessment device 50] FIG. 18 is a flowchart for explaining an example of the operation of the image determination process in the image determination device 50.

[0164] In step S41, the acquisition unit 52 of the image assessment device 50 acquires the first image transmitted from the first imaging device 20_1. The number of first images acquired by the image assessment device 50 may be multiple.

[0165] In step S42, the acquisition unit 52 acquires the converted image transmitted from the color conversion device 10. The image assessment device 50 may acquire multiple converted images. The order in which steps S41 and S42 are performed may be reversed. Furthermore, steps S41 and S42 may be performed in parallel.

[0166] In step S43, the determination unit 53 inputs, to the trained machine learning model 55, at least one of the first image acquired in step S41 and the converted image acquired in step S42.

[0167] In step S44, the determination unit 53 generates a determination result based on the output result from the trained machine learning model 55. The determination result may be the output result itself, or may be processed in some way. An example of processing is when the output result is text information, and the determination result is displayed in the form of a table or diagram so that a pathologist can easily understand the content at a glance.

[0168] By performing such an operation, it is possible to perform a highly accurate determination process on an input image that reflects color characteristics specific to a particular imaging device 20.

[0169] [Example of operation of color conversion system 300] FIG. 19 is a flowchart for explaining an example of the operation of the color conversion system 300 including the image assessment device 50.

[0170] In step S51, the color conversion device 10 acquires a second image from the second imaging device 20_2.

[0171] In step S52, the color conversion device 10 converts the second image into a reference image using the second conversion rule.

[0172] In step S53, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image having color characteristics specific to the first imaging device 20_1 using the inverse conversion rule. The converted image generated in step S53 is transmitted to the training device 40.

[0173] In step S54, the image assessment device 50 executes the image assessment process described with reference to FIG.

[0174] In this way, in the color conversion system 300, the image assessment device 50 can accurately assess a new image that reflects the color characteristics specific to the first imaging device 20_1 using a trained machine learning model that has been trained using an image that reflects the color characteristics specific to the first imaging device 20_1.

[0175] <Example of hardware configuration> Each of the components in the above-described embodiments has been described as a type of computer. Below, an example of a computer hardware configuration for realizing the functions of each of these components will be described.

[0176] FIG. 20 is a diagram showing an example of the hardware configuration of a computer for realizing the functions of each component of the color conversion system.

[0177] The computer 1000 comprises an input device 1001, an output device 1002, a CPU 1003, a ROM (Read Only Memory) 1004, a RAM (Random Access Memory) 1005, a storage device 1006, a reading device 1007 that reads information from various recording media, and a transmission / reception device 1008, and each part is connected by a bus 1009.

[0178] The reading device 1007 reads a program for realizing the functions of each component of the color conversion system from a non-transitory storage medium on which the program is recorded, and stores the program in the storage device 1006. Alternatively, the transmitting / receiving device 1008 communicates with a system device connected to the network, and stores a program downloaded from the system device in the storage device 1006.

[0179] The CPU 1003 copies the program stored in the storage device 1006 to the RAM 1005, and sequentially reads and executes instructions contained in the program from the RAM 1005, thereby realizing the functions of each component.

[0180] The disclosures of the specification, drawings and abstract contained in Japanese Patent Application No. 2024-049772, filed on March 26, 2024, are incorporated herein by reference in their entirety. [Industrial Applicability]

[0181] The present disclosure is useful for a color conversion device that performs color conversion processing on an image. [Explanation of symbols]

[0182] 100, 100A, 100B, 200, 300 color conversion system 10,10A color conversion device 11 Communications Department 12 Storage section 121 Conversion rule storage section 13 Control Unit 131 Acquisition Department 132 Conversion rule generation unit 133 Conversion Unit 20 Imaging device 20_1 First imaging device 20_2 Second imaging device 30 Observation equipment 40 Training equipment 41 Communications Department 42 Operation section 43 Acquisition Department 44 Training Department 45 Storage section 46 Machine learning models to train 50 Image judgment device 51 Communications Department 52 Acquisition Department 53 Judgment section 54 Memory section 55 pre-trained machine learning models 60 Conversion rule generator 61 Communications Department 62 Acquisition Department 63 Conversion rule generation unit 70 Standard conversion rule generator 71 Communications Department 72 Acquisition Department 73 Standard conversion rule generation unit 80 Inverse transformation rule generator 81 Communications Department 82 Acquisition Department 83 Inverse conversion rule generation unit

Claims

1. a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space; a conversion unit that converts the captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with the second conversion rule, and converts the reference image into a converted image expressed in the first color space in accordance with the inverse conversion rule; and the first conversion rule is determined based on a first chart image acquired by a first imaging device photographing a first color chart and colorimetric values ​​of the first color chart; the second conversion rule is determined based on a second chart image acquired by photographing a second color chart with a second imaging device and colorimetric values ​​of the second color chart; Color conversion device.

2. the image expressed in the first color space and the converted image expressed in the first color space are images that reflect color characteristics of a first imaging device; The color conversion device according to claim 1 .

3. the first color chart and the second color chart each include a different color patch; The color conversion device according to claim 1 .

4. the first color chart and the second color chart each include at least red, green, blue, and white color patches; The color conversion device according to claim 1 .

5. The reference color space is L ＊ a ＊ b ＊ color space or XYZ color space, The color conversion device according to claim 1 .

6. the first color space and the second color space are RGB color spaces; The color conversion device according to claim 1 .

7. the first conversion rule includes a first look-up table indicating a correspondence relationship between coordinate values ​​in the first color space and coordinate values ​​in the reference color space; the second conversion rule includes a second lookup table indicating a correspondence relationship between coordinate values ​​in the second color space and coordinate values ​​in the reference color space; The color conversion device according to claim 1 .

8. The captured image is a pathological image. The color conversion device according to claim 1 .

9. the first color chart for determining the first conversion rule and the second color chart for determining the second conversion rule each include at least a first color patch; The coordinate point of the first color patch is within a triangular area formed by connecting three points (0.200, 0.070), (0.450, 0.280), and (0.310, 0.340) with straight lines on the xy chromaticity diagram.

9. The color conversion device according to claim 8.

10. 1. A color conversion method executed by a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, A process of acquiring a captured image expressed in the second color space; a process of converting the captured image into a reference image expressed in the reference color space in accordance with the second conversion rule; and converting the reference image into a converted image expressed in the first color space in accordance with the inverse conversion rule; the first conversion rule is determined based on a first chart image acquired by a first imaging device photographing a first color chart and colorimetric values ​​of the first color chart; the second conversion rule is determined based on a second chart image acquired by photographing a second color chart with a second imaging device and colorimetric values ​​of the second color chart; Color conversion method.

11. A program executed by a computer of a color conversion device that stores an inverse conversion rule of a first conversion rule for converting an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule for converting an image expressed in a second color space into an image expressed in the reference color space, acquiring a captured image expressed in the second color space; a step of converting the captured image into a reference image expressed in the reference color space in accordance with the second conversion rule; transforming the reference image into a transformed image expressed in the first color space according to the inverse transformation rule; causing the computer to execute the first conversion rule is determined based on a first chart image acquired by a first imaging device photographing a first color chart and colorimetric values ​​of the first color chart; the second conversion rule is determined based on a second chart image acquired by photographing a second color chart with a second imaging device and colorimetric values ​​of the second color chart; program.

12. an acquisition unit that acquires a training dataset including training images expressed in a first color space and annotations to be determined; a training unit that uses the training dataset to train a machine learning model that determines images expressed in the first color space; and The training images are converting the captured image expressed in the second color space into a reference image expressed in the reference color space according to a second conversion rule for converting an image expressed in the second color space into an image expressed in the reference color space; converting the reference image into a converted image expressed in the first color space according to an inverse conversion rule of a first conversion rule for converting an image expressed in the first color space into an image expressed in the reference color space; This training device is generated by

13. the captured image is a pathological image, The machine learning model outputs a determination result in response to an input of the pathological image, the determination target of which is whether or not the pathological image includes a lesion.

13. The training device of claim 12.

14. the captured image is a pathological image, The machine learning model outputs a determination result in response to an input of the pathological image, the determination target being a tissue into which a site of interest included in the pathological image is classified.

13. The training device of claim 12.

15. a determination unit that determines an image expressed in a first color space using a trained machine learning model that determines an image expressed in the first color space; and the trained machine learning model is trained using a training dataset including training images represented in the first color space and annotations to be determined; The training images are converting the captured image expressed in the second color space into a reference image expressed in the reference color space according to a second conversion rule for converting an image expressed in the second color space into an image expressed in the reference color space; converting the reference image into a converted image expressed in the first color space according to an inverse conversion rule of a first conversion rule for converting an image expressed in the first color space into an image expressed in the reference color space; An image assessment device that is generated by the above.

16. determining an image represented in a first color space; A machine learning model that causes a computer to execute the machine learning model is trained using a training dataset including training images represented in the first color space and annotations to be determined; The training images are converting the captured image expressed in the second color space into a reference image expressed in the reference color space according to a second conversion rule for converting an image expressed in the second color space into an image expressed in the reference color space; converting the reference image into a converted image expressed in the first color space according to an inverse conversion rule of a first conversion rule for converting an image expressed in the first color space into an image expressed in the reference color space; The machine learning model is generated by

17. A color conversion system comprising the color conversion device according to claim 1 and a conversion rule generation device, the conversion rule generation device generates the inverse conversion rule and the second conversion rule; the color transformation device acquires the generated inverse transformation rule and the second transformation rule; Color conversion system.

18. A color transformation system comprising the color transformation device according to claim 1, a standard transformation rule generation device, and an inverse transformation rule generation device, the standard conversion rule generation device generates the first conversion rule and the second conversion rule; the inverse conversion rule generation device generates the inverse conversion rule of the first conversion rule; the color transformation device acquires the generated inverse transformation rule and the second transformation rule; Color conversion system.

19. a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image that reflects color characteristics specific to a first imaging device into a reference image that does not depend on color characteristics of a specific imaging device, and a second conversion rule that converts an image that reflects color characteristics specific to a second imaging device into the reference image; a conversion unit that converts a captured image in which color characteristics specific to the second imaging device are reflected into the reference image in accordance with the second conversion rule, and converts the reference image into a converted image having color characteristics specific to the first imaging device in accordance with the inverse conversion rule; A color conversion device comprising:

Images