Color conversion device, color conversion method, program, training device, image determination device, machine learning model, and color conversion system

Abstract

A color conversion device according to the present disclosure includes: a conversion rule storage unit for storing an inverse conversion rule to a first conversion rule for converting an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule for converting an image expressed in a second color space into an image expressed in the reference color space; and a conversion unit for converting a captured image in which the color characteristic peculiar to a second imaging device is reflected according to the second conversion rule into a reference image expressed in the reference color space, and converting the reference image into a conversion image expressed in the first color space according to the inverse conversion rule.

Description

Color conversion device, color conversion method, program, training device, image judgment device, machine learning model, and color conversion system

[0001] The present disclosure relates to a color conversion device, a color conversion method, a program, a training device, an image assessment device, a machine learning model, and a color conversion system.

[0002] It is common to capture an image of a subject using an imaging device and generate an image. Even when images are generated using the same imaging device, the color tone of the generated image may change if the conditions at the time of capture, such as lighting, change. For this reason, there is a demand for a technology that can align the color tones of multiple images.

[0003] Patent Document 1 discloses a color correction device that performs color correction on image data of biological tissues obtained by imaging with a medical imaging device, to make the image suitable for display on a color monitor. Patent Document 2 discloses an endoscopic device that records a color chart during an examination and corrects the color image during the next examination based on the color difference between a temporary color chart generated during the next examination and the color chart used during the previous examination.

[0004] International Publication No. 2020 / 166697 Japanese Patent Application Laid-Open No. 2016-54758

[0005] When an image is generated by capturing an image of a subject using an imaging device, the generated image generally reflects color characteristics specific to the imaging device. As a result, even if images of the same subject are captured using different imaging devices, the color tones may differ.

[0006] The present disclosure aims to provide a color conversion device, a color conversion method, a program, a training device, an image assessment device, and a machine learning model that can correct images generated by different imaging devices and bring color tones closer together.

[0007] A color conversion device according to one aspect of the present disclosure includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and a conversion unit that converts a captured image that reflects color characteristics specific to the second imaging device into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

[0008] A color conversion method according to one aspect of the present disclosure is a color conversion method executed by a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and the color conversion method acquires a captured image expressed in the second color space, converts the captured image into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

[0009] A program according to one aspect of the present disclosure is stored on a non-transitory storage medium, and is executed by a computer of a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and causes the computer to execute the following steps: acquiring a captured image expressed in the second color space; converting the captured image into a reference image expressed in the reference color space in accordance with the second conversion rule; and converting the reference image into a converted image expressed in the first color space in accordance with the inverse conversion rule.

[0010] A training device according to one aspect of the present disclosure includes an acquisition unit that acquires a training dataset including training images expressed in a first color space and annotations to be judged, and a training unit that uses the training dataset to train a machine learning model that judges images expressed in the first color space, wherein the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second transformation rule that converts an image expressed in the second color space into an image expressed in the reference color space, and converting the reference image into a transformed image expressed in the first color space in accordance with an inverse transformation rule of the first transformation rule that converts an image expressed in the first color space into an image expressed in the reference color space.

[0011] An image assessment device according to one aspect of the present disclosure includes an acquisition unit that acquires an image expressed in a first color space, and a assessment unit that assesses an image expressed in the first color space using a trained machine learning model that assesses images expressed in the first color space, wherein the trained machine learning model is trained using a training dataset that includes training images expressed in the first color space and an annotation to be assessed, and the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in a second color space into an image expressed in a reference color space, and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

[0012] A machine learning model according to one aspect of the present disclosure is a machine learning model that causes a computer to acquire an image expressed in a first color space and judge the image expressed in the first color space, wherein the machine learning model is trained using a training dataset that includes training images expressed in the first color space and an annotation to be judged, and the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

[0013] A color conversion system according to one aspect of the present disclosure is a color conversion system comprising the above-described color conversion device and a conversion rule generation device, wherein the conversion rule generation device generates the inverse conversion rule and the second conversion rule, and the color conversion device acquires the generated inverse conversion rule and the second conversion rule.

[0014] A color transformation system according to one aspect of the present disclosure is a color transformation system comprising the above-described color transformation device, a standard transformation rule generation device, and an inverse transformation rule generation device, wherein the transformation rule generation device generates the first transformation rule and the second transformation rule, the inverse transformation rule generation device generates the inverse transformation rule of the first transformation rule, and the color transformation device acquires the generated inverse transformation rule and the second transformation rule.

[0015] A color conversion device according to one aspect of the present disclosure includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image that reflects color characteristics specific to a first imaging device into a reference image that is not dependent on the color characteristics of a specific imaging device, and a second conversion rule that converts an image that reflects color characteristics specific to a second imaging device into the reference image, and a conversion unit that converts the captured image that reflects color characteristics specific to the second imaging device into the reference image according to the second conversion rule, and converts the reference image into a converted image having color characteristics specific to the first imaging device according to the inverse conversion rule.

[0016] Images produced by different imaging devices can be corrected to bring the colors closer together.

[0017] FIG. 1 is a diagram showing an example of the configuration of a first example of a color conversion system. FIG. 2 is a diagram showing an example of the functional configuration of a color conversion device in the first example. FIG. 3 is a diagram showing an example of the functional configuration of an imaging device. A flowchart for explaining an example of the operation of a color conversion device during a process of generating a conversion rule. A flowchart for explaining an example of the operation of a color conversion device during execution of a color conversion process. FIG. 4 is a diagram showing an example of the configuration of a fourth modified example of a color conversion system. FIG. 5 is a diagram showing an example of the functional configuration of a conversion rule generation device. FIG. 6 is a diagram showing an example of the configuration of a fifth modified example of a color conversion system.

[0018] <Overview> Multiple images may be generated using multiple image capture devices. Each image capture device has its own unique color characteristics, and even when capturing the same subject, images with different color tones may be generated.

[0019] When an observer views such multiple images, differences in color tone between the images can affect the observation. For example, if an observer is accustomed to the color tone of images generated by a specific imaging device, the observer may feel uncomfortable when viewing images generated by another imaging device due to the difference in color tone.

[0020] The present disclosure relates to a color conversion device that performs color conversion processing to bring the color tone of an image generated by an imaging device other than a specific imaging device, among images generated by multiple imaging devices, closer to the color tone of an image generated by the specific imaging device.

[0021] The color conversion device of the present disclosure can be applied to, for example, the following cases. Pathological diagnosis is generally performed in which an observer such as a pathologist examines a pathological image created by collecting tissue from a subject to determine the presence or absence of a lesion and its type. A pathological image is, for example, an image obtained by taking a magnified photograph of a specimen obtained by collecting, fixing, staining, and thinly slicing a portion of the subject's tissue. A pathological image is generated by photographing the specimen using an imaging device with a magnification function, such as a microscope, placed in a hospital.

[0022] When an observer observes a pathology image and makes a diagnosis, the color or intensity of the stained tissue can be an important clue. When an observer makes a diagnosis using pathology images generated by various imaging devices, the color tone of the pathology image may change due to the color characteristics specific to the imaging device, which may adversely affect the diagnosis.

[0023] One way to resolve this situation is to correct the image generated by the imaging device to, for example, an image with the original color tone of the stained specimen, so that the color tone specific to the imaging device is not reflected in the image observed by the observer.

[0024] However, observers who have observed many pathological images with color tones specific to a particular imaging device may find the corrected color tones unnatural. Here, a specific imaging device refers to an imaging device that generates images that the observer has observed many times in the past. In such cases, it becomes difficult for the observer to make a diagnosis based on their past experience. It is desirable for the annotation work, in which the observer adds diagnostic information to images, to be performed using color tones that the observer is familiar with. Therefore, there is a concern that annotation work using images with unfamiliar color tones due to correction will significantly reduce work efficiency.

[0025] In addition, pathology image analysis AI (Artificial Intelligence) has been developed that uses a trained machine learning model trained using a large number of pathology images and label data corresponding to each pathology image to assist observers in diagnosing new pathology images.

[0026] In such pathology image analysis AI, training of a machine learning model can be performed using pathology images of various color tones, or by uniformly training on pathology images of one color tone. These methods are used separately or in combination depending on the purpose of use of the pathology image analysis AI and the images used. For example, training using pathology images of various color tones is thought to improve the robustness of the trained machine learning model. On the other hand, training by uniformly training on pathology images of one color tone is thought to improve learning efficiency.

[0027] In consideration of such cases, the color conversion device disclosed herein converts images of various color tones generated by various imaging devices into images of a specific color tone based on color characteristics unique to a specific imaging device. Here, the specific color tone may be a color tone dependent on a specific imaging device that generates images that an observer has observed many times in the past. Alternatively, the specific color tone may be a color tone dependent on a specific imaging device that generates images that an image analysis AI machine learning model has learned many times in the past. This allows observers who have observed many images of specific color tones dependent on a specific imaging device to make observations and diagnoses by utilizing their past knowledge. Furthermore, by correcting images of various color tones into images of specific color tones dependent on a specific imaging device and using them as training data for the image analysis AI machine learning model, accurate diagnostic assistance can be provided for new pathological images generated by a specific imaging device.

[0028] Although the following description will be directed to an apparatus and method for handling pathological images, the present disclosure is not limited thereto. The apparatus and method of the present disclosure can handle various images other than pathological images, as long as they are digitized images.

[0029] The color conversion device of the present disclosure and a color conversion system including the color conversion device will be described in detail below with specific examples.

[0030] First Example First, a first example of a color conversion system including a color conversion device 10 according to an embodiment of the present disclosure will be described.

[0031] 1 is a diagram showing an example of the configuration of a first example of a color conversion system. The color conversion system 100 according to the first example includes a color conversion device 10, a plurality of image capture devices 20, and an observation device 30.

[0032] Each of the imaging devices 20 has a camera for capturing an image of a subject and generates a digital image. More specifically, the imaging devices 20 are devices for generating pathological images. In the present disclosure, the number of imaging devices is not particularly limited. However, in the following description, a case will be described in which the imaging devices 20 include a first imaging device 20_1 and a second imaging device 20_2, as shown in FIG. 1 .

[0033] In the following description, the first imaging device 20_1 and the second imaging device 20_2 may be collectively referred to as imaging device 20, or either the first imaging device 20_1 or the second imaging device 20_2 may be simply referred to as imaging device 20.

[0034] The first imaging device 20_1 generates an image expressed in a first color space. The second imaging device 20_2 generates an image expressed in a second color space. In this embodiment, the first color space or the second color space may each be an RGB color space. In the following description, the image generated by the first imaging device 20_1 will be referred to as a first image, and the image generated by the second imaging device 20_2 will be referred to as a second image.

[0035] The first image reflects color characteristics specific to the first imaging device. The second image reflects color characteristics specific to the second imaging device. The first image and the second image have different color tones. In this way, each imaging device included in the multiple imaging devices 20 generates an image reflecting a different color characteristic.

[0036] In the following description, it is assumed that the first imaging device 20_1 is a specific imaging device, that is, an imaging device that generates images that the observer has observed many times in the past.

[0037] The color conversion device 10 does not perform color conversion processing on the first image acquired from the first imaging device 20_1 among the multiple imaging devices 20, but performs color conversion processing on the second image acquired from the second imaging device 20_2. In this case, the color conversion processing is processing to convert an image reflecting color characteristics specific to the second imaging device 20_2 into an image reflecting color characteristics specific to the first imaging device 20_1. In the following description, an image that has undergone color conversion processing by the color conversion device 10 will be referred to as a converted image.

[0038] The observation device 30 is a device that allows an observer to observe a pathological image generated by the first imaging device 20_1 or the second imaging device 20_2 and perform a diagnosis, etc. The observation device 30 is, for example, a type of computer equipped with a display device, such as a PC (Personal Computer) or a tablet terminal.

[0039] The observation device 30 acquires and displays a first image from the first imaging device 20_1. The observation device 30 also acquires and displays a converted image obtained by performing color conversion processing on the second image generated by the second imaging device 20_2 from the color conversion device 10. This allows the observer to observe an image that reflects the color characteristics specific to the first imaging device 20_1, regardless of which imaging device among the multiple imaging devices 20 generated the image.

[0040] Fig. 2 is a diagram showing an example of the functional configuration of the color conversion device 10 in the first example. As shown in Fig. 2, the color conversion device 10 includes a communication unit 11, a storage unit 12, and a control unit 13. The color conversion device 10 is a type of computer, such as a PC, a workstation, or a tablet terminal.

[0041] The communication unit 11 transmits and receives various types of information to and from the imaging device 20 or the observation device 30 via wired or wireless communication.

[0042] The storage unit 12 stores various types of information necessary for the color conversion process performed by the color conversion device 10. The information stored in the storage unit 12 includes, for example, information about the conversion rules used in the color conversion process, and images before and after the color conversion process is performed. The storage unit 12 has a conversion rule storage unit 121 that stores various conversion rules.

[0043] The control unit 13 controls the operation of the color conversion device 10. The control unit 13 includes an acquisition unit 131, a conversion rule generation unit 132, and a conversion unit 133.

[0044] The acquisition unit 131 acquires an image to be subjected to color conversion processing. In the example shown in Fig. 1, the acquisition unit 131 acquires the second image received from the second imaging device 20_2 by the communication unit 11.

[0045] The conversion rule generation unit 132 generates a conversion rule to be used in the color conversion process. The conversion rule is, for example, an ICC (International Color Consortium) profile, which indicates the correspondence between the color values ​​of an image before conversion and the color values ​​of an image after conversion.

[0046] More specifically, the conversion rule generating unit 132 first generates an ICC profile that converts an image generated by the image capturing device 20 into a reference image that does not depend on the color characteristics of the image capturing device 20 .

[0047] In this embodiment, the reference image is an image that is expressed in a reference color space and does not depend on the color characteristics of the image capture device 20. The reference color space is, for example, a known L ＊ a ＊ b ＊ 1 , the conversion rule generation unit 132 generates a first conversion rule for converting an image generated by the first imaging device 20_1, which reflects color characteristics specific to the first imaging device 20_1, into a reference image, and a second conversion rule for converting an image generated by the second imaging device 20_2, which reflects color characteristics specific to the second imaging device 20_2, into a reference image.

[0048] The first conversion rule is a rule for converting an image expressed in a first color space and reflecting color characteristics specific to the first imaging device 20_1 into a reference image expressed in a reference color space and independent of the color characteristics of the imaging device 20. The conversion rule generation unit 132 determines the first conversion rule based on the correspondence between the color values ​​of each pixel in the first chart image and the colorimetric values, using a first chart image acquired by the first imaging device 20_1 photographing a first color chart and previously acquired colorimetric values ​​of the first color chart. Alternatively, if the original color of an object appearing in the image reflecting color characteristics specific to the first imaging device 20_1 is known, the conversion rule generation unit 132 may generate the first conversion rule so as to match or approximate the color value of each pixel in the reference image to the original color of the object. Alternatively, the conversion rule generation unit 132 may generate the first conversion rule using the color values ​​of each color patch in the first chart image and previously acquired colorimetric values ​​of each color patch of the first color chart.

[0049] The first conversion rule may be generated in the form of, for example, a look-up table (LUT). Alternatively, the first conversion rule may be generated in the form of, for example, a matrix. Table 1 shows an example of the first conversion rule expressed in the LUT format. In Table 1, the values ​​(r １ , g １ , b １ ) and the value (x １， y １ , z １ ) are shown to correspond to each other.

[0050] The second conversion rule is a rule for converting an image expressed in the second color space and reflecting color characteristics specific to the second imaging device 20_2 into a reference image expressed in a reference color space and independent of the color characteristics of the imaging device 20. The conversion rule generation unit 132 determines the second conversion rule based on the correspondence between the color values ​​of each pixel in the second chart image and the colorimetric values, using a second chart image acquired by the second imaging device 20_2 photographing a second color chart and colorimetric values ​​of the second color chart acquired in advance. Alternatively, when the original color of an object appearing in the image reflecting color characteristics specific to the second imaging device 20_2 is known, the conversion rule generation unit 132 may generate the second conversion rule so as to match the color value of each pixel in the reference image to the original color of the object.

[0051] Like the first transformation rule, the second transformation rule may be generated in, for example, an LUT format or a determinant format.

[0052] The first chart image and the second chart image may be prepared in advance by, for example, an administrator of the color conversion system 100, or may be generated by the first imaging device 20_1 and the second imaging device 20_2 as necessary. The colorimetric values ​​of each color patch of the first color chart included in the first chart image and the colorimetric values ​​of each color patch of the second color chart included in the second chart image are known. Before the conversion rule generation unit 132 generates a conversion rule, the color conversion device 10 receives the color values ​​of each color patch in the first chart image and the colorimetric values ​​of each color patch of the first color chart included in the first chart image.

[0053] The first chart image and the second chart image may be different images. More specifically, the first color chart and the second color chart may include different color patches, for example, color patches of different colors.

[0054] The first color chart and the second color chart may each include at least red, green, blue, and white color patches.

[0055] More specifically, the red color patches included in the first color chart and the second color chart have a transmission spectrum with a peak wavelength of 620 nm to 680 nm and a half-width of 100 nm or less. The green color patches included in the first color chart and the second color chart have a transmission spectrum with a peak wavelength of 510 nm to 540 nm and a half-width of 80 nm or less. The blue color patches included in the first color chart and the second color chart have a transmission spectrum with a peak wavelength of 450 nm to 472 nm and a half-width of 80 nm or less. The white color patches included in the first color chart and the second color chart can be made of a base material that is not colored with a dye or pigment. Examples of the base material include glass, film, acrylic resin, polycarbonate, gelatin, and paper.

[0056] By including such color patches in the first color chart and the second color chart, the coordinates of the color patches of each color on the xy chromaticity diagram can be made to match or approximate the coordinates of each color defined in the BT.2020 standard. This makes it possible to perform suitable color calibration when the BT.2020 standard is applied to the imaging device 20.

[0057] The red, green, and blue colors included in the first color patch are not limited to the above example, and may be in the following ranges on the xy chromaticity diagram, for example.

[0058] The color of the red color patch included in the first color patch may be within the area surrounded by (0.735, 0.265), (0.627, 0.372) and (0.520, 0.174) in coordinates on the xy chromaticity diagram, the curve portion of the blackbody locus BL where the color deviation is −0.01 between color temperatures of 1200 K or more and 6000 K or less. The blackbody locus refers to the locus on the chromaticity diagram that depicts the state of a completely black object as it changes with absolute temperature.

[0059] The color of the green color patch included in the first color patch may be within the area surrounded by the coordinates on the xy chromaticity diagram: (0.005, 0.587), (0.014, 0.750), (0.074, 0.834), (0.155, 0.806), (0.345, 0.652), and the curved portion of the blackbody locus BL where the color deviation is +0.02 between color temperatures of 6000 K or more and 7000 K or less.

[0060] The color of the blue color patch included in the first color patch may be within the area surrounded by the coordinates on the xy chromaticity diagram: (0.174, 0.005), (0.131, 0.046), (0.073, 0.185), the straight line connecting the points on the blackbody locus BL where the color deviation is +0.02 and the points on the blackbody locus BL where the color temperature is 15000 K and the color deviation is -0.01, the curved portion on the blackbody locus BL where the color temperature is from 15000 K to 7000 K and the color deviation is -0.01, and (0.280, 0.054).

[0061] By using a chart image including such color patches, it is possible to generate a conversion rule that can accommodate a wide color gamut.

[0062] Furthermore, as an example of a color chart other than those described above, for example, the following color chart may be adopted to make the pathological images handled by the color conversion system 100 easier to see. Such a first color chart and second color chart each include at least a first color patch, and the coordinate point of the first color patch is within a triangular area formed by connecting three points, (0.200, 0.070), (0.450, 0.280), and (0.310, 0.340), on the xy chromaticity diagram with straight lines.

[0063] Only one of the first color chart and the second color chart may include such a first color patch. With this configuration, if the pathology image reflecting the color characteristics of the first image capture device 20_1 deviates significantly (has a strong peculiarity) from the stained actual specimen, and the pathology image reflecting the color characteristics of the second image capture device 20_2 does not deviate significantly from the actual specimen (has a weak peculiarity), by using the first color patch for the first color chart, it is possible to create highly accurate conversion coefficients for the first image capture device with the strong peculiarity, and to create efficient conversion coefficients for the second image capture device with the weak peculiarity.

[0064] When the pathology image is an image of cells stained with hematoxylin-eosin (HE), the coordinate points of the first color patch can be set within the above range to bring the color of the color patch closer to a color characteristic of the pathology image. This makes it possible to generate a first conversion rule or a second conversion rule that enables conversion to bring the color closer to a color characteristic of the pathology image. Therefore, the first conversion rule or the second conversion rule can be used to perform color correction so that the converted image becomes closer to the color of the actual object (stained cells).

[0065] The coordinate points of the first color patch may be within a pentagonal area formed by connecting five points (0.310, 0.340), (0.350, 0.300), (0.350, 0.200), (0.260, 0.150), and (0.280, 0.250) in this order on the xy chromaticity diagram. Having the coordinate points of the first color patch within this pentagonal area allows for more accurate color correction.

[0066] The coordinate points of the first color patch may be within a quadrangular area formed by connecting four points (0.311, 0.334), (0.347, 0.210), (0.270, 0.162), and (0.292, 0.216) on the xy chromaticity diagram with straight lines. Having the coordinate points of the first color patch within this quadrangular area allows for even more accurate color correction.

[0067] The first color chart and the second color chart may further include a second color patch that is a different color from the first color patch. In this case, the coordinate points of both the first color patch and the second color patch may be within any of the above-mentioned regions on the xy chromaticity diagram. By having the coordinate points of multiple color patches within any of the above-mentioned regions, it is possible to generate a first conversion rule or a second conversion rule that can perform color correction on an image having a color tone characteristic of a pathology image.

[0068] Furthermore, the conversion rule generation unit 132 generates an inverse conversion rule for performing an inverse conversion of the first conversion rule based on the first conversion rule generated as described above. The inverse conversion rule is a rule for converting a reference image that is expressed in a reference color space and does not depend on the color characteristics of the imaging device into an image that reflects color characteristics specific to the first imaging device 20_1. The inverse conversion rule can be created, for example, by the following method. Note that the following describes a case where the reference color space is the XYZ color space and the conversion rule is expressed in LUT format.

[0069] The inverse transformation rule in LUT format can be created, for example, by at least the following three steps: a first step of preparing a set of grid points in a reference color space, a second step of determining the relationship between each grid point in the set prepared in the first step and the grid points in the reference color space in the LUT, which is the first transformation rule for transforming from the first color space to the reference color space, and a third step of determining, from the relationship determined in the second step, grid points in the first color space that correspond to each grid point in the set prepared in the first step.

[0070] An example of a method for creating an inverse transformation rule in the LUT format will be described in detail below. First, the transformation rule generation unit 132 creates a set of lattice points in the XYZ color space. In the following, an example is shown in which the definition described in ISO 15076-1:2010 is used as the LUT format. In this example, a three-dimensional LUT format is used as the LUT format. A three-dimensional LUT is defined as (X ｉ , Y ｊ , Z ｋ) When i, j, and k are specified, the converted RGB values ​​corresponding to each specified value are defined. Table 2 shows an example of a set of lattice points in the XYZ color space. Table 2 shows a case where the ranges that XYZ can take are equally divided for each of X, Y, and Z. If the ranges that XYZ can take are each divided into eight equal parts, the number of lattice points will be 9 to the power of 3. min and max indicate the minimum and maximum lattice point numbers, respectively, in the ranges that XYZ can take. If divided into eight equal parts, min is 0 and max is 8.

[0071] X ｍｉｎ , Y ｍｉｎ , Z ｍｉｎ The values ​​of may be preset to a predetermined value, for example, 0. ｍａｘ , Y ｍａｘ , Z ｍａｘ The values ​​of X may be preset to a predetermined value, for example 128. ｍｉｎ , Y ｍｉｎ , Z ｍｉｎ The value of X may be the minimum value in the XYZ color space that can be obtained when each color value in the RGB color space is changed from the minimum value (for example, 0) to the maximum value (for example, 255). ｍａｘ , Y ｍａｘ , Z ｍａｘ The value of may be the maximum value in XYZ that can be obtained when each color value in the RGB color space is changed from the minimum value (for example, 0) to the maximum value (for example, 255).

[0072] Next, the conversion rule generation unit 132 plots the color values ​​of all grid points in the XYZ color space that correspond to all grid points in the RGB color space in the LUT of the first conversion rule in the XYZ color space, and creates a solid in the XYZ space that corresponds to the RGB color space.

[0073] Next, the conversion rule generation unit 132 checks where each grid point in the XYZ color space shown in Table 1 is located on the solid body. The conversion rule generation unit 132 performs interpolation as needed based on the correspondence between each grid point in the XYZ color space and a position on the solid body. The interpolation can be performed using a known method, such as a spline function or linear interpolation. If a grid point in the XYZ color space shown in Table 1 is located outside the solid body, the conversion rule generation unit 132 may replace the grid point so that it is located on the outermost or innermost part of the solid body. Examples of replacement methods include replacing the grid point with a point that minimizes the color difference or a point with the same hue. Alternatively, the grid point may be replaced with the outermost grid point in the solid body that minimizes the color difference with a grid point in the XYZ color space shown in Table 1. Alternatively, the grid point may be replaced with the outermost grid point in the solid body that has the same hue as a grid point in the XYZ color space shown in Table 1.

[0074] This allows the conversion rule generating unit 132 to generate an LUT that converts the grid points of the XYZ color space shown in Table 1 into grid points of the RGB color space, that is, an inverse conversion rule.

[0075] Although the method for creating an inverse transformation rule by the transformation rule generation unit 132 has been specifically described above, the method for creating an inverse transformation rule in the present disclosure is not limited to the above-described method. For example, if the first transformation rule is expressed as a determinant (e.g., 3 × 3), the transformation rule generation unit 132 can create an inverse transformation rule by calculating its inverse matrix.

[0076] Before the color conversion process is performed by conversion unit 133, conversion rule generation unit 132 may generate the first conversion rule, the second conversion rule, and the inverse conversion rule and store them in conversion rule storage unit 121 of storage unit 12. Alternatively, conversion rule generation unit 132 may perform the color conversion process and the process of generating a new conversion rule in parallel or successively.

[0077] The conversion unit 133 performs color conversion processing on the second image acquired from the second imaging device 20_2 using the first conversion rule, the second conversion rule, and the inverse conversion rule.

[0078] As described above, the color conversion process is a process of converting the second image, which is expressed in the second color space and reflects color characteristics specific to the second imaging device 20_2, into an image, which is expressed in the first color space and reflects color characteristics specific to the first imaging device 20_1. The conversion unit 133 first uses the second conversion rule to convert the second image acquired from the second imaging device 20_2 into a reference image, which is expressed in the reference color space and does not depend on the color characteristics of the imaging device 20. Then, the conversion unit 133 uses the inverse conversion rule to convert the reference image into a converted image, which is expressed in the first color space and has color characteristics specific to the first imaging device 20_1.

[0079] The converted image converted by the color conversion process is output to the observation device 30 via the communication unit 11. As a result, as described above, the observation device 30 acquires the first image generated by the first imaging device 20_1 and the converted image. The observer can observe the first image or the converted image displayed by the observation device 30 and make a diagnosis. Here, both the first image and the converted image have color characteristics specific to the first imaging device 20_1. Therefore, for an observer who has observed many first images captured by the first imaging device 20_1 in the past, the first image or the converted image displayed on the observation device 30 are images with familiar color tones because they have the color characteristics of the first imaging device 20_1. In this way, the color conversion system 100 can convert an image reflecting color characteristics different from those specific to a specific imaging device 20 into an image reflecting the color characteristics specific to a specific imaging device 20, so that an observer who is accustomed to seeing images reflecting the color characteristics specific to a specific imaging device 20 can easily observe the image.

[0080] [Operation Example] Next, a description will be given of an operation example in the first configuration example of the color conversion system 100. Fig. 4 is a flowchart for explaining an operation example of the conversion rule generation process in the color conversion system 100.

[0081] In step S1, the first imaging device 20_1 and the second imaging device 20_2 capture images of a first color chart and a second color chart, respectively, which are prepared in advance, to generate a first chart image and a second chart image, respectively. The generated first chart image and second chart image are acquired by the color conversion device 10.

[0082] In step S2, the conversion rule generation unit 132 of the color conversion device 10 generates a first conversion rule based on the first chart image and a second conversion rule based on the second chart image. The above-described methods may be appropriately employed to generate each conversion rule.

[0083] In step S3, the conversion rule generation unit 132 generates an inverse conversion rule based on the first conversion rule. The inverse conversion rule may be generated by any of the methods described above. The conversion rules and inverse conversion rules generated in steps S2 and S3 are stored in the conversion rule storage unit 121 of the color conversion device 10 and used in the color conversion process described below.

[0084] Next, a description will be given of an example of the operation of the color conversion process in the color conversion system 100. FIG.

[0085] In step S11, the acquisition unit 131 of the color conversion device 10 acquires a second image from the second imaging device 20_2.

[0086] In step S12, the conversion unit 133 of the color conversion device 10 converts the second image into a reference image using the second conversion rule.

[0087] In step S13, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image using the inverse conversion rule. The converted image generated in step S13 is sent to the observation device 30.

[0088] In step S14, the observation device 30 displays the converted image. As a result, even if the object to be observed is photographed by the second imaging device 20_2, the observer can observe the converted image as an image having color characteristics specific to the first imaging device 20_1. This makes it much easier for an observer who is accustomed to observing images reflecting color characteristics specific to the first imaging device 20_1 to observe the converted image than if the observer simply observed an image generated by the second imaging device 20_2 reflecting color characteristics specific to the second imaging device 20_2.

[0089] <Modifications of the First Example> (Modification 1) In the first example described above, the case where the multiple imaging devices 20 include the first imaging device 20_1 and the second imaging device 20_2 has been described. In the present disclosure, the number of imaging devices 20 is not particularly limited, and the multiple imaging devices 20 may include imaging devices other than the first imaging device 20_1 and the second imaging device 20_2.

[0090] When the multiple imaging devices in the color conversion system include three or more imaging devices, the color conversion device acquires images of imaging devices other than a specific imaging device (the first imaging device 20_1 in the example shown in FIG. 1 ) and performs color conversion processing on each image. In this case, before performing the color conversion processing, the color conversion device generates a conversion rule for converting an image generated by each imaging device into a reference image and an inverse conversion rule for converting the reference image into an image reflecting color characteristics specific to the specific imaging device. The color conversion device first uses the conversion rule to convert an image generated by an imaging device other than the specific imaging device into the reference image, and then uses the inverse conversion rule to convert the reference image into an image reflecting color characteristics specific to the specific imaging device, thereby generating a converted image having color characteristics specific to the specific imaging device.

[0091] When the multiple imaging devices in the color conversion system include three or more imaging devices, an imaging device other than the specific imaging device is an example of a second imaging device of the present disclosure, and a color space that represents an image generated by an imaging device other than the specific imaging device is an example of a second color space of the present disclosure.

[0092] (Variation 2) In the first example described above, the color conversion device 10 performs color conversion processing on the second image generated by the second imaging device 20_2. However, in the present disclosure, for example, an observation device may have the functionality of a color conversion device and perform color conversion processing. In this case, the observation device acquires the first image and the second image from the first imaging device and the second imaging device, respectively, and does not perform color conversion processing on the first image, but performs color conversion processing on the second image before displaying it. This allows an observer to perform a diagnosis by observing the first image or the converted image displayed on the observation device.

[0093] (Variation 3) In addition, in the present disclosure, among the multiple imaging devices, an imaging device other than a specific imaging device, for example, a second imaging device, may have the function of a color conversion device and perform color conversion processing. In this case, the first imaging device directly transmits the generated first image to the observation device, and the second imaging device performs color conversion processing on the generated second image and then transmits the converted image to the observation device. This allows an observer to perform diagnosis, etc. by observing the first image or the converted image displayed on the observation device. In this case, the imaging device is a type of computer having a processor for executing color conversion processing.

[0094] The second image capturing device may acquire from an external device an inverse transformation rule for transforming the reference image into the transformed image, or may acquire the first chart image and information on the colorimetric values ​​of the first color chart and create the inverse transformation rule based on these.

[0095] Because pathological images tend to have large data volumes, the color conversion process may require a relatively large amount of processing power or time. By having an imaging device other than the specific imaging device perform the color conversion process, the processing power or processing time required for the color conversion process can be distributed among multiple imaging devices. Therefore, having an imaging device other than the specific imaging device perform the color conversion process is particularly effective when, for example, there are a relatively large number of imaging devices.

[0096] (Variation 4) In the first example described above, the color conversion device 10 has the conversion rule generation unit 132, which generates various conversion rules. However, in the present disclosure, the color conversion device 10 may acquire conversion rules from an external device. In this case, the color conversion device 10 does not need to have the conversion rule generation unit 132.

[0097] 6 is a diagram showing an example of the configuration of a color conversion system according to Modification 4. A color conversion system 100A according to Modification 4 includes a color conversion device 10A, a plurality of image capture devices 20, an observation device 30, and a conversion rule generation device 60.

[0098] Fig. 7 is a diagram showing an example of the functional configuration of a color conversion device 10A in Modification 4. As shown in Fig. 7, color conversion device 10A according to Modification 4 differs from color conversion device 10 of the first example shown in Fig. 2 in that it does not have a conversion rule generation unit. The functions and operations of each component of color conversion device 10A shown in Fig. 7 are similar to the functions and operations of each component described in the first example above.

[0099] 8 is a diagram showing an example of the functional configuration of conversion rule generating device 60. As shown in FIG. 8, conversion rule generating device 60 includes a communication unit 61, an acquisition unit 62, and a conversion rule generating unit 63.

[0100] The communication unit 61 transmits and receives various types of information to and from the color conversion device 10A and the plurality of image capture devices 20 via wired or wireless communication.

[0101] The acquisition unit 62 acquires a first chart image and a second chart image for generating each conversion rule.

[0102] The conversion rule generation unit 63 generates a first conversion rule and a second conversion rule, and further generates an inverse conversion rule of the first conversion rule. The method of generating each conversion rule by the conversion rule generation unit 63 may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

[0103] The second transformation rule and the inverse transformation rule generated by the transformation rule generation device 60 are transmitted to the color transformation device 10A. The color transformation device 10A can then use the second transformation rule and the inverse transformation rule acquired from the transformation rule generation device 60 to perform the color transformation process described in the first example above.

[0104] As described above, in Modification 4, an example has been described in which the transformation rule generation device 60 external to the color transformation device 10A generates the second transformation rule and the inverse transformation rule, and the color transformation device 10A acquires the second transformation rule and the inverse transformation rule from the transformation rule generation device 60 and performs the color transformation process. According to Modification 4, the color transformation device 10A does not need to have the function of generating each transformation rule. This makes it possible to reduce the manufacturing costs and operating costs of the color transformation device 10A.

[0105] (Variation 5) In the above Variation 4, the conversion rule generation device 60 generates the first conversion rule and the second conversion rule, and generates the inverse conversion rule of the first conversion rule. In this Variation 5, an example will be described in which a device for generating the first conversion rule and the second conversion rule and a device for generating the inverse conversion rule of the first conversion rule are provided separately.

[0106] 9 is a diagram showing an example of the configuration of a color transformation system according to Modification 5. A color transformation system 100B according to Modification 5 includes a color transformation device 10A, a plurality of imaging devices 20, an observation device 30, a standard transformation rule generation device 70, and an inverse transformation rule generation device 80.

[0107] The configuration and operation of the color conversion device 10A are the same as those of the fourth modification.

[0108] 10 is a diagram showing an example of the functional configuration of standard conversion rule generation device 70. As shown in FIG. 10, standard conversion rule generation device 70 includes a communication unit 71, an acquisition unit 72, and a standard conversion rule generation unit 73.

[0109] The communication section 71 transmits and receives various information to and from the plurality of image capture devices 20 and the inverse conversion rule generating device 80 via wired or wireless communication.

[0110] The acquisition unit 72 acquires a first chart image and a second chart image for generating each conversion rule.

[0111] The standard conversion rule generation unit 73 generates the first conversion rule and the second conversion rule. The method of generating the first conversion rule and the second conversion rule by the conversion rule generation unit 63 may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

[0112] The first transformation rule generated by the standard transformation rule generation device 70 is transmitted to the inverse transformation rule generation device 80. The second transformation rule generated by the standard transformation rule generation device 70 may be transmitted to the color transformation device 10A via the inverse transformation rule generation device 80, or may be transmitted to the color transformation device 10A without passing through the inverse transformation rule generation device 80.

[0113] 11 is a diagram showing an example of the functional configuration of the inverse conversion rule generating device 80. As shown in FIG. 11, the inverse conversion rule generating device 80 includes a communication unit 81, an acquisition unit 82, and an inverse conversion rule generating unit 83.

[0114] The communication section 81 transmits and receives various information to and from the color conversion device 10A and the standard conversion rule generation device 70 via wired or wireless communication.

[0115] The acquiring unit 72 acquires at least the first conversion rule from the standard conversion rule generating device 70 .

[0116] The inverse conversion rule generation unit 83 generates an inverse conversion rule of the first conversion rule based on the first conversion rule. The method of generating the inverse conversion rule by the inverse conversion rule generation unit 83 may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

[0117] The inverse conversion rules generated by the inverse conversion rule generation device 80 are transmitted to the color conversion device 10A. The color conversion device 10A then performs the color conversion process described in the first example above using the acquired second conversion rule and inverse conversion rule.

[0118] As explained above, in Variation 5, an example has been described in which a standard transformation rule generation device 70 external to the color transformation device 10A generates the first transformation rule and the second transformation rule, an inverse transformation rule generation device 80 external to the color transformation device 10A generates an inverse transformation rule of the first transformation rule, and the color transformation device 10A acquires the second transformation rule and the inverse transformation rule and performs color transformation processing. According to Variation 5, the color transformation device 10A does not need to have the function of generating each transformation rule. This makes it possible to reduce the manufacturing and operating costs of the color transformation device 10A.

[0119] (Variation 6) In the present disclosure, multiple imaging devices may belong to the same facility, such as the same medical institution (such as a hospital or a clinic) or the same research institute, or may belong to different facilities. Furthermore, the facilities to which the imaging devices belong may be established in different countries.

[0120] Because the number of pathologists who can make accurate diagnoses on pathological images is limited, such pathologists tend to receive a large number of requests for pathological image diagnoses from multiple hospitals, etc. Without the color conversion device of the present disclosure, the requested pathologist would have to make a diagnosis using pathological images in unfamiliar color tones. However, with a color conversion system including the color conversion device of the present disclosure, even in such cases, the pathologist can make a diagnosis using pathological images in familiar color tones.

[0121] (Variation 7) In the first example described above, an example of operation was described in which the color conversion device 10 performs the process of generating a conversion rule before the color conversion process. As described above, in the present disclosure, the color conversion device may perform the color conversion process and the process of generating a new conversion rule in parallel or consecutively.

[0122] For example, a color conversion device acquires, from a new imaging device other than a specific imaging device, a chart image generated by the imaging device by capturing an image of a color chart, information on the colorimetric values ​​of the color chart, and an image generated by the imaging device by capturing an object. Based on the chart image and the information on the colorimetric values, the color conversion device generates a conversion rule for converting an image that reflects color characteristics specific to the new imaging device into a reference image, and generates the reference image using the acquired image and the conversion rule. The color conversion device then converts the reference image using a previously generated inverse conversion rule for converting the reference image into an image that reflects color characteristics specific to the specific imaging device, to generate a converted image. In this way, when acquiring an image from a new imaging device that generates an image for which a conversion rule for converting to a reference image has not been generated, the conversion rule generation process and the color conversion process may be performed in parallel or consecutively.

[0123] For example, suppose a pathologist collects images generated by multiple imaging devices from multiple hospitals and performs diagnoses using these images. In this case, the pathologist may receive a request from a new hospital to perform a diagnosis using images. In such a case, by having the color conversion device acquire the chart image and the observation image generated by a new imaging device from the new hospital at approximately the same time, a converted image with a color tone familiar to the observer can be generated without any problems, even if a conversion rule for converting an image that reflects the color characteristics specific to the imaging device into a reference image has not been generated in advance.

[0124] Second Example Next, a second example of a color conversion system including the color conversion device 10 according to an embodiment of the present disclosure will be described.

[0125] 12 is a diagram showing an example of the configuration of a second example of a color conversion system. A color conversion system 200 according to the second example includes a color conversion device 10, multiple image capture devices 20, and a training device 40. The color conversion device 10 and the multiple image capture devices 20 are the same as those described in the first example above.

[0126] The training device 40 trains a machine learning model to be trained using images (pathological images) acquired directly from the multiple imaging devices 20 or via the color conversion device 10 as training images. The machine learning model to be trained is a machine learning model that causes an image assessment device 50, described below, to acquire images that are expressed in a first color space and reflect color characteristics specific to the first imaging device 20_1, and assess the images that are expressed in the first color space and reflect color characteristics specific to the first imaging device 20_1. The training device 40 is a type of computer, such as a PC, a workstation, or a tablet terminal.

[0127] Determining an image expressed in the first color space and reflecting color characteristics specific to the first image capture device 20_1 means performing a determination process that helps an observer make a diagnosis using a pathology image, such as determining the presence or absence of a lesion in the pathology image or the location or type of the lesion. In order to train a machine learning model that can perform such a determination process, the training device 40 trains the machine learning model to be trained using a training dataset that includes training images expressed in the first color space and reflecting color characteristics specific to the first image capture device 20_1 and label data of the training images.

[0128] The label data of the training images includes, for example, annotations of the target of determination included in the training images. The target of determination is, for example, whether or not the pathology image includes a lesion. In this case, the label data includes, for example, mask data that obscures the lesion in the training image. Alternatively, the target of determination may be, for example, the type of tissue to which a site of interest included in the pathology image is classified. Alternatively, the target of determination may be, for example, the type of disease caused by the lesion if the pathology image includes a lesion. The annotation is, for example, correct answer data indicating whether or not the pathology image includes a lesion (i.e., lesion segmentation), the type of tissue to which the site of interest included in the training image is classified, or the type of disease caused by the lesion included in the training image.

[0129] 13 is a diagram showing an example of the functional configuration of the training device 40. The training device 40 includes a communication unit 41, an operation unit 42, an acquisition unit 43, a training unit 44, and a storage unit 45.

[0130] The communication unit 41 transmits and receives various types of information to and from the color conversion device 10 or the imaging device 20 via wired or wireless communication.

[0131] The operation unit 42 receives an operation to add annotations to the first image and the converted image. The operation to add annotations to the first image and the converted image is performed by, for example, a pathologist.

[0132] The acquisition unit 43 acquires a training data set including training images expressed in a first color space and reflecting color characteristics specific to the first image capture device 20_1, and annotations to be determined. The training images include the first image received from the first image capture device 20_1 and the converted image received from the color conversion device 10.

[0133] The storage unit 45 stores a machine learning model 46 to be trained.

[0134] The training unit 44 uses the training data set to train a machine learning model 46 to be trained, which is used to judge images that are expressed in the first color space and that reflect color characteristics specific to the first image capture device 20_1. By performing the training process using a sufficient number of images, it is possible to generate a trained machine learning model that can perform judgment processing with high accuracy.

[0135] Thus, in a second example of a color conversion system including a color conversion device according to the present disclosure, the training device performs training using images that reflect color characteristics specific to a specific imaging device, and the color conversion device converts images generated by an imaging device other than the specific imaging device into converted images that have color characteristics specific to the specific imaging device. This increases the number of images with specific color tones included in the training data. A trained machine learning model trained using such converted images can effectively assist pathologists in making accurate diagnoses of pathological images.

[0136] In the second example, as in the first example, the number of image capture devices is not limited to two. Furthermore, if sufficient processing capacity can be ensured, for example, the training device may have the function of a color conversion device and be able to perform the conversion rule generation process and the color conversion process.

[0137] [Example of Operation of Training Apparatus 40] FIG. 14 is a flowchart for explaining an example of operation of training processing in the training apparatus 40.

[0138] In step S21, the acquisition unit 43 of the training apparatus 40 acquires the first image transmitted from the first imaging device 20_1. The training apparatus 40 may acquire a plurality of first images.

[0139] In step S22, the acquisition unit 43 acquires the converted image transmitted from the color conversion device 10. The training device 40 may acquire multiple converted images. The order in which steps S21 and S22 are performed may be reversed. Furthermore, steps S21 and S22 may be performed in parallel.

[0140] In step S23, the operation unit 42 accepts an input operation of an annotation by a pathologist or the like.

[0141] In step S24, the training unit 44 trains the machine learning model 46 to be trained based on the first image, the converted image, and the annotations.

[0142] The operations from step S21 to step S24 may be repeatedly performed.

[0143] With this operation, even if multiple imaging devices 20 generate images that reflect their own unique color characteristics, the training device 40 can train the machine learning model 46 to be trained using images with specific color tones. This eliminates the need for correction processing or setting of training parameters related to the color characteristics unique to each of the multiple imaging devices 20, making it possible to generate a trained machine learning model that can make accurate judgments with little effort.

[0144] 14 , the training device 40 receives an input operation for the annotation via the operation unit 42, but the present disclosure is not limited to this. For example, an apparatus other than the training device may generate the annotation. In this case, the acquisition unit of the training device may acquire the annotation together with the first image and the converted image from an external apparatus.

[0145] In addition, in the present disclosure, the training device may perform a certain amount of training on the machine learning model to be trained by supervised learning using annotations, and then perform training by unsupervised learning using the first image and the converted image that do not have annotations, which reduces the effort required to prepare annotations.

[0146] In the present disclosure, the training images may include a second image received from the second imaging device 20_2 and a converted image received from the color conversion device. The converted image is an image having color characteristics specific to the first imaging device 20_1. By including images that reflect the color characteristics of different imaging devices, i.e., images with different color tones, in the training images, a highly robust trained machine learning model can be generated.

[0147] [Example of Operation of Color Conversion System 200] FIG. 15 is a flowchart for explaining an example of operation of color conversion system 200 including training device 40.

[0148] In step S31, the color conversion device 10 acquires a second image from the second imaging device 20_2.

[0149] In step S32, the color conversion device 10 converts the second image into a reference image using the second conversion rule.

[0150] In step S33, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image that reflects color characteristics specific to the first imaging device 20_1 using the inverse conversion rule. The converted image generated in step S33 is transmitted to the training device 40.

[0151] In step S34, the training device 40 executes the training process described with reference to FIG.

[0152] In this way, in the color conversion system 200, the color conversion device 10 converts the image generated by the second imaging device 20_2 into a converted image having color characteristics specific to the first imaging device 20_1, and the training device 40 performs training using the converted image. This makes it easy to increase the number of images with specific color tones included in the training data in the training device 40. Furthermore, a trained machine learning model trained using such converted images can effectively assist pathologists in making accurate diagnoses of pathological images.

[0153] <Third Example> Next, a third example of a color conversion system including the color conversion device 10 according to an embodiment of the present disclosure will be described.

[0154] 16 is a diagram showing an example of the configuration of a third example of a color conversion system. A color conversion system 300 according to the third example includes a color conversion device 10, a plurality of image capture devices 20, and an image assessment device 50. The color conversion device 10 and the plurality of image capture devices 20 are the same as those described in the first example above.

[0155] The image assessment device 50 performs a determination process in which images (pathological images) acquired directly from the multiple imaging devices 20 or via the color conversion device 10 are input into a trained machine learning model, and a determination result is output. The trained machine learning model is a machine learning model generated by the training device 40 performing the training process in the second example described above. In other words, the trained machine learning model is a machine learning model trained using images expressed in a first color space and reflecting color characteristics specific to the first imaging device 20_1. The image assessment device 50 is a type of computer, such as a PC, a workstation, or a tablet terminal.

[0156] 17 is a diagram showing an example of the functional configuration of the image assessment device 50. The image assessment device 50 includes a communication unit 51, an acquisition unit 52, a determination unit 53, and a storage unit 54.

[0157] The communication unit 51 transmits and receives various types of information to and from the color conversion device 10 or the imaging device 20 via wired or wireless communication.

[0158] The acquisition unit 52 acquires the first image from the first imaging device 20_1 and the converted image from the color conversion device 10. Both the first image and the converted image are images that reflect color characteristics specific to the first imaging device 20_1.

[0159] The storage unit 54 stores a trained machine learning model 55 .

[0160] The determination unit 53 determines an image that is expressed in the first color space and reflects color characteristics specific to the first imaging device 20_1, using the trained machine learning model 55. More specifically, the determination unit 53 inputs at least one of the newly acquired first image and the converted image into the trained machine learning model 55, and generates a determination result based on the output result. The determination result is output to a device used by a pathologist or the like to assist in diagnosis, such as a display device not shown in FIG. 16 .

[0161] Thus, in a third example of a color conversion system having a color conversion device according to the present disclosure, the image assessment device can accurately assess new images that reflect color characteristics specific to a specific imaging device using a trained machine learning model that has been trained using images that reflect color characteristics specific to a specific imaging device.

[0162] In the third example, as in the first or second example, the number of imaging devices is not limited to 2. Furthermore, if sufficient processing capacity can be ensured, for example, the image assessment device may have the function of a color conversion device and be able to perform the conversion rule generation process and the color conversion process.

[0163] [Example of Operation of Image Assessment Device 50] FIG. 18 is a flowchart for explaining an example of operation of image assessment processing in the image assessment device 50.

[0164] In step S41, the acquisition unit 52 of the image assessment device 50 acquires the first image transmitted from the first imaging device 20_1. The image assessment device 50 may acquire a plurality of first images.

[0165] In step S42, the acquisition unit 52 acquires the converted image transmitted from the color conversion device 10. The image assessment device 50 may acquire multiple converted images. The order in which steps S41 and S42 are performed may be reversed. Furthermore, steps S41 and S42 may be performed in parallel.

[0166] In step S43, the determination unit 53 inputs at least one of the first image acquired in step S41 and the converted image acquired in step S42 into the trained machine learning model 55.

[0167] In step S44, the determination unit 53 generates a determination result based on the output result from the trained machine learning model 55. The determination result may be the output result itself, or may be processed in some way. An example of processing is when the output result is text information, and the determination result is displayed in the form of a table or diagram so that a pathologist can easily understand the content at a glance.

[0168] By performing such an operation, it is possible to perform a highly accurate determination process on an input image that reflects color characteristics specific to a particular imaging device 20 .

[0169] [Example of Operation of Color Conversion System 300] FIG. 19 is a flowchart for explaining an example of operation of the color conversion system 300 including the image assessment device 50.

[0170] In step S51, the color conversion device 10 acquires a second image from the second imaging device 20_2.

[0171] In step S52, the color conversion device 10 converts the second image into a reference image using the second conversion rule.

[0172] In step S53, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image having color characteristics specific to the first imaging device 20_1 using the inverse conversion rule. The converted image generated in step S53 is transmitted to the training device 40.

[0173] In step S54, the image assessment device 50 executes the image assessment process described with reference to FIG.

[0174] In this way, in the color conversion system 300, the image assessment device 50 can accurately assess new images that reflect color characteristics specific to the first imaging device 20_1 using a trained machine learning model that has been trained using images that reflect color characteristics specific to the first imaging device 20_1.

[0175] <Example of Hardware Configuration> Each of the components in the above-described embodiment has been described as a type of computer. Below, an example of a hardware configuration of a computer for realizing the functions of each of these components will be described.

[0176] FIG. 20 is a diagram showing an example of the hardware configuration of a computer for realizing the functions of each component of the color conversion system.

[0177] The computer 1000 includes an input device 1001, an output device 1002, a CPU 1003, a ROM (Read Only Memory) 1004, a RAM (Random Access Memory) 1005, a storage device 1006, a reading device 1007 that reads information from various recording media, and a transmission / reception device 1008, and each part is connected by a bus 1009.

[0178] The reading device 1007 reads a program for realizing the functions of each component of the color conversion system from a non-transitory storage medium on which the program is recorded, and stores the program in the storage device 1006. Alternatively, the transmitting / receiving device 1008 communicates with a system device connected to a network, and stores a program downloaded from the system device in the storage device 1006.

[0179] The CPU 1003 then copies the program stored in the storage device 1006 to the RAM 1005, and sequentially reads and executes instructions contained in the program from the RAM 1005, thereby realizing the functions of each component.

[0180] The disclosures of the specification, drawings and abstract contained in Japanese Patent Application No. 2024-049772, filed on March 26, 2024, are incorporated herein by reference in their entirety.

[0181] The present disclosure is useful for a color conversion device that performs color conversion processing on an image.

[0182] 100, 100A, 100B, 200, 300 Color conversion system 10, 10A Color conversion device 11 Communication unit 12 Memory unit 121 Conversion rule storage unit 13 Control unit 131 Acquisition unit 132 Conversion rule generation unit 133 Conversion unit 20 Imaging device 20_1 First imaging device 20_2 Second imaging device 30 Observation device 40 Training device 41 Communication unit 42 Operation unit 43 Acquisition unit 44 Training unit 45 Memory unit 46 Machine learning model to be trained 50 Image judgment device 51 Communication unit 52 Acquisition unit 53 Judgment unit 54 Memory unit 55 Trained machine learning model 60 Conversion rule generation device 61 Communication unit 62 Acquisition unit 63 Conversion rule generation unit 70 Standard conversion rule generation device 71 Communication unit 72 Acquisition unit 73 Standard conversion rule generation unit 80 Inverse conversion rule generation device 81 Communication unit 82 Acquisition unit 83 Inverse conversion rule generation unit

Claims

1. A color conversion device having: a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space; and a conversion unit that converts a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with the second conversion rule, and converts the reference image into a converted image expressed in the first color space in accordance with the inverse conversion rule.

2. The color conversion device according to claim 1, wherein the image expressed in the first color space and the converted image expressed in the first color space are images that reflect the color characteristics of the first imaging device.

3. The color conversion device according to claim 1, wherein the first conversion rule is determined based on a first chart image obtained by a first imaging device photographing a first color chart and the colorimetric values ​​of the first color chart, and the second conversion rule is determined based on a second chart image obtained by a second imaging device photographing a second color chart and the colorimetric values ​​of the second color chart.

4. The color conversion device according to claim 3, wherein the first color chart and the second color chart each include different color patches.

5. The color conversion device according to claim 3, wherein the first color chart and the second color chart each include color patches of at least red, green, blue, and white.

6. The reference color space is L ＊ a ＊ b ＊ The color conversion device according to claim 1 , wherein the color space is a RGB color space or an XYZ color space.

7. The color conversion device according to claim 1, wherein the first color space and the second color space are RGB color spaces.

8. A color conversion device as described in claim 3, wherein the first conversion rule includes a first lookup table indicating the correspondence between coordinate values ​​in the first color space and coordinate values ​​in the reference color space, and the second conversion rule includes a second lookup table indicating the correspondence between coordinate values ​​in the second color space and coordinate values ​​in the reference color space.

9. The color conversion device according to claim 1, wherein the captured image is a pathological image.

10. The color conversion device according to claim 9, wherein the first color chart for determining the first conversion rule and the second color chart for determining the second conversion rule each include at least a first color patch, and the coordinate point of the first color patch is within a triangular area on an xy chromaticity diagram, the coordinate points being (0.200, 0.070), (0.450, 0.280), and (0.310, 0.340) connected by straight lines.

11. A color conversion method executed by a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, the color conversion method comprising the steps of: acquiring a captured image expressed in the second color space; converting the captured image into a reference image expressed in the reference color space according to the second conversion rule; and converting the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

12. A program executed by a computer of a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, the program causing the computer to execute the following steps: acquiring a captured image expressed in the second color space; converting the captured image into a reference image expressed in the reference color space according to the second conversion rule; and converting the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

13. A training device comprising: an acquisition unit that acquires a training dataset including training images expressed in a first color space and annotations to be judged; and a training unit that uses the training dataset to train a machine learning model that judges images expressed in the first color space, wherein the training images are generated by: converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second transformation rule that converts an image expressed in the second color space into an image expressed in the reference color space; and converting the reference image into a transformed image expressed in the first color space in accordance with an inverse transformation rule of the first transformation rule that converts an image expressed in the first color space into an image expressed in the reference color space.

14. The training device described in claim 13, wherein the captured image is a pathological image, and the machine learning model outputs a judgment result in response to an input of the pathological image, with the judgment target being whether or not the pathological image contains a lesion.

15. The training device described in claim 13, wherein the captured image is a pathological image, and the machine learning model outputs a judgment result in response to the input of the pathological image, with the tissue into which the area of ​​interest contained in the pathological image is classified as the judgment target.

16. An image judgment device comprising: a judgment unit that judges images expressed in a first color space using a trained machine learning model that judges images expressed in the first color space, wherein the trained machine learning model has been trained using a training dataset including training images expressed in the first color space and annotations to be judged, and the training images are generated by: converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in the second color space into an image expressed in the reference color space; and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

17. A machine learning model that causes a computer to execute the steps of: determining an image expressed in a first color space; wherein the machine learning model is trained using a training dataset including training images expressed in the first color space and an annotation to be determined; and the training images are generated by: converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in the second color space into an image expressed in the reference color space; and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

18. A color transformation system comprising the color transformation device according to claim 1 and a transformation rule generation device, wherein the transformation rule generation device generates the inverse transformation rule and the second transformation rule, and the color transformation device acquires the generated inverse transformation rule and the second transformation rule.

19. A color transformation system comprising the color transformation device of claim 1, a standard transformation rule generation device, and an inverse transformation rule generation device, wherein the standard transformation rule generation device generates the first transformation rule and the second transformation rule, the inverse transformation rule generation device generates the inverse transformation rule of the first transformation rule, and the color transformation device obtains the generated inverse transformation rule and second transformation rule.

20. A color conversion device comprising: a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image reflecting color characteristics specific to a first imaging device into a reference image that is not dependent on the color characteristics of a specific imaging device, and a second conversion rule that converts an image reflecting color characteristics specific to a second imaging device into the reference image; and a conversion unit that converts the captured image reflecting color characteristics specific to the second imaging device into the reference image according to the second conversion rule, and converts the reference image into a converted image having color characteristics specific to the first imaging device according to the inverse conversion rule.

Images