Image processing device, image recognition system, and image processing method

Abstract

To automatically generate a color / brightness corrector for generating a reconstruction image by suppressing the difference in color and brightness between images, and realize high-precision recognition of an object in the reconstruction image without changing a recognizer.SOLUTION: In an image processing device 1, a correction unit for generating a reconstruction image from input images in accordance with a setting parameter is realized by one or more processors or modularized hardware executing a program, the setting parameter is adjusted so as to make a first reconstruction image generated from an input first image by the correction unit close to a second image of the same target as the first image with a color tone different from that of the first image, and a target image is input to the correction unit to generate a reconstruction image of the target image. In generation of the reconstruction image of the target image by the correction unit, a feature amount is extracted from the input target image in accordance with the setting parameter, the extracted feature amount is corrected, and the reconstruction image of the target image is generated from the corrected feature amount.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing technique, and more particularly to an image processing technique for suppressing differences in color and brightness of a captured image and accurately detecting a specific object contained in the image. [Background technology]

[0002] In recent years, image recognition technology using machine learning and the like has been studied. Technology for equalizing the color tone and brightness of an image has also been studied. For example, Patent Document 1 proposes a technology for correcting color unevenness in an image. In Patent Document 1, a color averaging filter is used as a parameter used in image processing to reduce color unevenness in an image representing a biological material. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 150194 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when reducing color unevenness in an image using a color averaging filter as in Patent Document 1, color unevenness may remain or color artifacts may occur in the corrected image, so that color unevenness in the image is not necessarily reduced. Also, in Patent Document 1, when an object in an image after color unevenness correction is identified by a classifier, there is a problem that the classification accuracy of the classifier decreases. [Means for solving the problem]

[0005] An image processing system of one embodiment of the present invention includes one or more processors and one or more memories, the one or more memories store a corrector that generates a reconstructed image from an input image in accordance with setting parameters, and the setting parameters are adjusted so that a first reconstructed image generated by the corrector from an input first image and a second image of the same object but with a different color tone as the first image approach each other, the processor inputs a target image to the corrector to generate a reconstructed image of the target image, and the corrector extracts features from the input target image in accordance with the setting parameters, corrects the extracted features, and generates the reconstructed image of the target image from the corrected features.

[0006] A learning system of one embodiment of the present invention includes one or more processors and one or more memories, wherein the one or more memories store a corrector that generates a reconstructed image from an input image according to setting parameters, and the one or more processors perform the adjustment of the setting parameters, and in the adjustment, perturb a second image to generate a first image, input the first image to the corrector to generate the first reconstructed image, input the second image to the corrector to generate a second reconstructed image, and adjust the setting parameters so that a cost calculated from a predetermined loss is reduced, and the loss includes a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

[0007] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. Also, aspects of the present invention may be realized and realized by the elements and combinations of various elements and aspects of the following detailed description and the appended claims.

[0008] It should be understood that the descriptions in this specification are exemplary and illustrative only and are not intended to limit the scope or application of the present invention in any manner. Effect of the Invention

[0009] According to the present invention, even when there are color and brightness differences between images, a reconstructed image is created that suppresses the color and brightness differences between the images, and further, machine learning is performed to automatically generate a color and brightness corrector that generates a reconstructed image in which the color and brightness differences of the target image are suppressed. This makes it possible to identify objects (e.g., abnormal tissue) in an image with high accuracy without changing the classifier. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram showing functions of the image processing device during learning according to the first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing device according to a first embodiment or an image processing device according to a second embodiment. [Diagram 3] FIG. 2 is a block diagram showing a color / brightness corrector during learning according to the first embodiment. [Figure 4] FIG. 2 is a block diagram showing functions of the image processing device at the time of evaluation according to the first embodiment. [Figure 5A] 11 is a diagram for explaining an example of the operation of a color / brightness feature extraction unit. FIG. [Figure 5B] 11 is a diagram for explaining an example of the operation of a color / brightness feature extraction unit. FIG. [Figure 6A] 11 is a diagram for explaining an example of the operation of a color / brightness feature correction unit. FIG. [Figure 6B] 11 is a diagram for explaining an example of the operation of a color / brightness feature correction unit. FIG. [Figure 7] 10 is a diagram for explaining an example of the operation of an image reconstruction unit. FIG. [Figure 8] 10 is a diagram for explaining an example of the operation of an image reconstruction unit. FIG. [Figure 9] FIG. 4 is a block diagram showing a color / brightness corrector during evaluation according to the first embodiment. [Figure 10] 11 is a diagram for explaining an example of an operation of a feature extraction unit; FIG. [Figure 11] 11 is a diagram for explaining an example of an operation of a feature extraction unit; FIG. [Figure 12]10A and 10B are diagrams for explaining an example of the operation of a recognition unit. [Figure 13] 10A and 10B are diagrams for explaining an example of the operation of a recognition unit. [Figure 14] 11 is a diagram for explaining an example of an operation of an output unit; FIG. [Figure 15] 11 is a diagram for explaining an example of an operation of an output unit; FIG. [Figure 16A] 11 is a diagram for explaining an example of an operation of an output unit; FIG. [Figure 16B] 11 is a diagram for explaining an example of an operation of an output unit; FIG. [Figure 17] 4 is a flowchart for explaining the overall operation during learning of the image processing device according to the first embodiment. [Figure 18] 5 is a flowchart for explaining the overall operation during evaluation of the image processing device according to the first embodiment. [Figure 19] FIG. 11 is a block diagram showing functions of an image processing device during learning according to a second embodiment. [Figure 20A] FIG. 11 is a block diagram showing a color / brightness corrector during learning according to the second embodiment. [Figure 20B] FIG. 11 is a block diagram showing a color / brightness corrector during learning according to the second embodiment. [Figure 21A] 11 is a diagram for explaining an example of the operation of a color / brightness feature extraction / correction unit. FIG. [Figure 21B] 11 is a diagram for explaining an example of the operation of a color / brightness feature extraction / correction unit. FIG. [Figure 22A] 11 is a diagram for explaining an example of the operation of a color / brightness feature extraction / correction unit. FIG. [Figure 22B] 11 is a diagram for explaining an example of the operation of a color / brightness feature extraction / correction unit. FIG. [Diagram 23] 10 is a flowchart for explaining the overall operation during learning of an image processing device 2 according to a second embodiment. [Figure 24] FIG. 11 is a block diagram showing functions of an image processing device at the time of evaluation according to the second embodiment. [Diagram 25]10 is a flowchart for explaining the overall operation during evaluation of the image processing device according to the second embodiment. [Figure 26] FIG. 1 is a diagram showing a schematic configuration of an image processing system equipped with an image processing device. [Figure 27] FIG. 1 is a diagram showing a schematic configuration of an image processing system equipped with an image processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be indicated by the same numerals. Note that the accompanying drawings show specific embodiments and implementation examples according to the principles of the present invention, but these are for understanding the present invention and are by no means used to interpret the present invention in a limiting manner.

[0012] In the present embodiment, the present invention has been described in sufficient detail for those skilled in the art to practice the present invention, but it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present invention. Therefore, the following description should not be interpreted as being limited thereto.

[0013] Additionally, embodiments of the present invention may be implemented in software running on a general purpose computer, in specialized hardware, or a combination of software and hardware, as described below.

[0014] In the following, each process in an embodiment of the present invention will be explained with "each processing unit as a program (e.g., an identification unit, etc.)" as the subject (acting entity); however, since a program is executed by a processor (CPU, etc.) to perform the defined processes using memory and a communication port (communication control device), the explanation may also be made with the processor as the subject.

[0015] In one embodiment of the present specification, a color and brightness corrector that suppresses the color and brightness difference between images is created during learning, and during evaluation, the color and brightness corrector is used to create a reconstructed image in which the color and brightness difference of the target image is suppressed, and objects in the reconstructed image are detected. This allows for highly accurate identification of objects (e.g., abnormal tissues) in an image without changing the classifier.

[0016] In one embodiment of the present specification, during learning, a perturbed image is automatically created from a target image while updating parameters of a color / brightness corrector composed of a color / brightness feature extraction unit, a color / brightness feature correction unit, and an image reconstruction unit. When evaluating an image, a reconstructed image in which the color / brightness difference of the evaluation image is suppressed is created using the color / brightness corrector created during learning, and feature amounts of the reconstructed image are extracted for classification, thereby providing an image processing method that realizes highly accurate classification of objects in an image.

[0017] An image processing device according to an embodiment of the present specification executes, during machine learning, a process of extracting a feature amount of color and brightness difference between images, a process of correcting the feature amount of color and brightness difference, a process of creating a reconstructed image from the corrected feature amount, and a process of generating a color and brightness corrector that performs machine learning to automatically generate a reconstructed image in which color and brightness differences are suppressed from a target image. During evaluation, the image processing device executes a process of automatically generating a reconstructed image in which color and brightness differences are suppressed from a target image using the color and brightness corrector created during machine learning, extracting a feature amount of the reconstructed image, and identifying an object in the reconstructed image from the calculated feature amount of the reconstructed image.

[0018] (1) First embodiment <Functional configuration of the classifier generation device> FIG. 1 is a block diagram showing a functional configuration of an image processing device during machine learning according to an embodiment of the present invention. The image processing device 1 includes an input unit 10, a perturbed image creation unit 11, a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, an image reconstruction unit 14, an output unit 15, a control unit 91, and a memory 90. The image processing device 1 may be implemented in an image acquisition device, or may be implemented in a server connected to the image acquisition device via a network, as described later (third and fourth embodiments). The image processing device 1 may be implemented in one or more computers, or may be implemented in a cloud. In this way, the image processing device 1 may be implemented in a system including one or more processors and one or more memories (storage devices).

[0019] In the image processing device 1, the perturbed image creation unit 11, the color / brightness feature extraction unit 12, the color / brightness feature correction unit 13, the image reconstruction unit 14, and the output unit 15 may be realized by one or more processors that execute a program, or may be realized by modularized hardware.

[0020] Image data is input to the input unit 10. For example, the input unit 10 may acquire coded still image data, such as JPG, JPEG2000, PNG, or BMP format, WSI, or the like captured at a predetermined time interval by imaging means such as a camera built into the image processing device 1, and use the image as an input image. The input unit 10 may also extract still image data of frames at a predetermined interval from moving image data in a format such as MotionJPEG, MPEG, H.264, or HD / SDI, and use the image as an input image.

[0021] The input unit 10 may also use an image acquired by an imaging means via a bus, a network, or the like as an input image. The input unit 10 may also use an image already stored in a removable recording medium as an input image. The image input from the input unit 10 is output to the perturbation image creation unit 11 and the color / brightness feature extraction unit 12.

[0022] The perturbed image creation unit 11 creates a perturbed image by perturbing the values ​​of the input image. As a perturbation method, a perturbed image is created by perturbing the γ value, R value, B value, G value, etc. In order to prevent artifacts from occurring, a range for perturbing each value is set in advance, and values ​​are randomly determined within the range to create multiple perturbed images. Machine learning uses one or more input images and multiple perturbed images generated from each input image.

[0023] The color / brightness feature extraction unit 12 calculates color / brightness feature amounts (values ​​of each parameter) and output values ​​of each layer of a network (e.g., a Convolutional Neural Network) that constitutes the color / brightness feature extraction unit 12 from the input image and the perturbed image.

[0024] FIG. 3 is an example of a block diagram showing the functional configuration of a color / brightness corrector during machine learning. The color / brightness feature extractor 12 corresponds to the encoders 311 and 312 in FIG. 3. The encoder 311 extracts a global feature G1 from the input image 301. The encoder 312 extracts a local feature L1 from the input image 301. The encoder 312 extracts a local feature L2 from the perturbed image 302. The global feature is a feature that represents the color and brightness of the entire image, and is, for example, a 1x1x several hundred dimensional feature. The size of the global feature is smaller than the size of the image. On the other hand, the local feature is a feature that represents the shape in the image and is a feature of the same size as the image. In this example, features of two sizes are extracted, but features of three or more different sizes may be extracted. By extracting features of different sizes, the color and brightness of the image can be corrected more appropriately. The type of feature may be one.

[0025] The color / brightness feature correction unit 13 is composed of an adjustment unit 321, an adjustment unit 322, a normalization unit 331, and a normalization unit 332. The adjustment unit 321 corrects the global feature G1 and the global feature G2 so that the corrected global feature G1 and the global feature G2 approach each other, thereby obtaining the global feature G11 and the global feature G21, respectively. The adjustment unit 322 corrects the local feature L1 and the local feature L2 so that the local feature L1 and the local feature L2 approach each other, thereby obtaining the local feature L11 and the local feature L21, respectively. The normalization unit 331 normalizes the global feature G11 by the local feature L11, thereby calculating the normalized feature. The normalization unit 332 normalizes the global feature G21 by the local feature L21, thereby calculating the normalized feature.

[0026] 3, and generates a reconstructed image 351 from the normalized feature amount obtained by the color / brightness feature correction unit 13. Also, generates a reconstructed image 352 from the normalized feature amount obtained by the color / brightness feature correction unit 13. When learning a color / brightness corrector using machine learning, for example, loss values ​​are calculated and learning is performed so that the input image 301 and the reconstructed image 351, the input image 301 and the reconstructed image 352, the global feature amount G11 and the global feature amount G21, and the local feature amount L11 and the local feature amount L21 approach each other, and encoders 311 and 312, adjustment units 321 and 322, normalization units 331 and 332, and decoder 340 are created.

[0027] The output unit 15 displays the reconstructed image and the loss values, cost values, etc. output by the created color and brightness corrector on an output device (such as a display), and stores in memory 90 information on the reconstructed image, the color and brightness corrector, and values ​​calculated by each unit from the color and brightness feature extraction unit 12 to the image reconstruction unit 14.

[0028] The control unit 91 is realized by a processor, and is connected to each element in the image processing device 1. The operation of each element of the image processing device 1 is performed by the autonomous operation of each of the above-mentioned components or by instructions from the control unit 91.

[0029] FIG. 4 is a block diagram showing a functional configuration of the image processing device 1 at the time of evaluation according to an embodiment of the present specification. The image processing device 1 has an input unit 10, a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, an image reconstruction unit 14, a feature extraction unit 16, a discrimination unit 17, an output unit 15, a control unit 91, and a memory 90. The image processing device 1 may be implemented in an image processing device in an image acquisition device, or may be implemented in a server connected to the image acquisition device via a network, as described later (third and fourth embodiments). The image processing device 1 may be implemented in one or more computers, and may be implemented in a cloud. In this way, the image processing device 1 may be implemented in a system including one or more processors and one or more memories (storage devices).

[0030] In the image processing device 1, the color / brightness feature extraction unit 12, color / brightness feature correction unit 13, image reconstruction unit 14, output unit 15, feature extraction unit 16, and classification unit 17 may be realized by a processor that executes a program, or may be realized by modularized hardware.

[0031] The following describes each processing unit that differs from that in FIG. The parameter values ​​of the color / brightness corrector created during learning are read from the memory 90, and the parameters of the color / brightness feature extraction unit 12, the color / brightness feature correction unit 13, and the image reconstruction unit 14 are set.

[0032] For an image input from an input unit 10, a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, and an image reconstruction unit 14 are used to generate a reconstructed image in which color / brightness differences are suppressed.

[0033] Machine learning is performed in advance to identify objects in the input image as objects, for example, to identify normal tissues and cells as normal tissues and cells, and to identify abnormal tissues and cells in the input image as abnormal tissues and cells, and a classifier (each parameter of the feature extraction unit 16 and the classification unit 17 required for classification (filter coefficients, offset values, etc.)) is created from the training image.

[0034] The values ​​of each parameter (filter coefficient, offset value, etc.) of a classifier network (e.g., a Convolutional Neural Network, etc.) consisting of a feature extraction unit 16 and a classification unit 17 that were created in advance through machine learning are read from the memory 90, and each parameter of the feature extraction unit 16 and the classification unit 17 is set.

[0035] The feature extraction unit 16 calculates feature amounts from the input image and output values ​​of each layer of a network (e.g., a Convolutional Neural Network) constituting the feature extraction unit. The classification unit 17 receives the output value of the final layer of the network of the feature extraction unit 16 and calculates the classification result of the classifier. The output unit 15 displays the classification result on an output device (such as a display) and stores the classification result in memory 90. The control unit 91 is realized by a processor and is connected to each element in the image processing device 1. The operation of each element of the image processing device 1 is performed by the autonomous operation of each of the above-mentioned components or by instructions from the control unit 91.

[0036] In this manner, in the image processing device 1 of this embodiment, during machine learning, a perturbed image is created using a learning image, while each parameter of a color / brightness corrector consisting of a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, and an image reconstruction unit 14 is updated, and a color / brightness corrector that suppresses color / brightness differences in an image is created. Furthermore, during evaluation, each parameter of the color / brightness corrector and the classifier created in advance is read, a reconstructed image in which the color / brightness differences in the evaluation image are suppressed using the color / brightness corrector, and the reconstructed image is classified using the classifier, thereby making it possible to identify objects (e.g., abnormal tissue, etc.) in an image with high accuracy.

[0037] <Hardware configuration of image processing device> 2 is a diagram showing an example of a hardware configuration of an image processing device 1 according to an embodiment of the present invention. The image processing device 1 has a CPU (processor) 201 that executes various programs, a memory 202 that stores various programs, a storage device (corresponding to the memory 90) 203 that stores various data, an output device 204 for outputting a reconstructed image, a color / brightness corrector, a classification result, a loss value during machine learning, and the like, an input device 205 for inputting user instructions, images, and the like, and a communication device 206 for communicating with other devices, which are interconnected by a bus 207.

[0038] The CPU 201 reads and executes various programs from the memory 202 as necessary. The memory 202 stores, as programs, an input unit 10, a perturbed image creation unit 11, a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, an image reconstruction unit 14, a feature extraction unit 16, a classification unit 17, and an output unit 15.

[0039] The auxiliary storage device 203 stores training images, evaluation images, parameters of the color and brightness correctors and reconstructed images generated by the color and brightness feature extraction unit 12, the color and brightness feature correction unit 13, and the image reconstruction unit 14, parameters (filter coefficients, offset values, etc.) of the classifiers generated by the feature extraction unit 16 and the classification unit 17, classification results and classification values, display information generated by the output unit 15, etc.

[0040] The output device 204 is composed of devices such as a display, a printer, and a speaker. For example, the output device 204 displays data generated by the output unit 15 on a display screen. The input device 205 is composed of devices such as a keyboard, a mouse, and a microphone. Instructions by a user (including the decision to input each image) are input to the image processing device 1 by the input device 205.

[0041] The communication device 206 is not an essential component of the image processing device 1, and in the case where a personal computer or the like connected to the image acquisition device includes a communication device, the image processing device 1 does not need to have the communication device 206. The communication device 206 performs an operation of receiving data (including images, color and brightness correctors, etc.) transmitted from other devices (e.g., servers) connected via a network, and storing the data in the storage device 203.

[0042] The image processing device of this embodiment uses a perturbed image during learning to extract and correct color and brightness features, and creates a color and brightness corrector that generates a reconstructed image in which color and brightness differences are suppressed. Furthermore, during evaluation, a reconstructed image is generated from the evaluation image using the color and brightness corrector, and the reconstructed image is classified using a classifier, making it possible to classify objects (e.g., abnormal tissue, etc.) in the image with high accuracy.

[0043] <Configuration and operation of each part> The configuration and operation of each element during machine learning will be explained in detail below. (i) Perturbation image creation unit 11 A perturbed image is created by perturbing the values ​​of the input image. The perturbation method involves perturbing the gamma value, R value, B value, G value (tone level), etc. to create a perturbed image. For example, a perturbed image is created by multiplying the gamma value by 1.4, adding 3 to the R value over the entire image, and subtracting 5 from the B value over the entire image. To prevent artifacts from occurring, a range for perturbing each value (for example, a range of 0.5 to 1.5 for the gamma value) is set in advance, and a value is randomly determined within that range to create a perturbed image.

[0044] In this way, by changing certain values ​​of the input image, an image of the same object as the input image but with a different color tone can be generated. The color tone of some or all areas of the generated image is different from that of the input image. Perturbation can efficiently generate many different variations of images. Images with different color tones are images with different colors and brightness. Images with a different color tone from the input image may be generated in a different way from the perturbation image, which is an image that artificially adds a small disturbance or shift to the input image. For example, they can be generated by changing the dye solution or illumination light.

[0045] (ii) Color and brightness feature extraction unit 12 The color and brightness features of the input image are calculated. As an example, an example of calculating color and brightness features (global features, local features) is shown in Figures 5A and 5B. CNN in Figures 5A and 5B stands for Convolutional Neural Network.

[0046] For example, as shown in Fig. 5A, the feature extractor G5 obtains global feature quantities FGi of color and brightness of the input image 301 or the perturbed image 302 from the input image 301 or the perturbed image 302 using the following formula (1). Also, as shown in Fig. 5B, the feature extractor L5 obtains local feature quantities FLi of color and brightness of the input image 301 or the perturbed image 302 from the input image 301 or the perturbed image 302 using the following formula (2).

[0047] The filter coefficient wgj shown in equation (1) is a coefficient calculated by machine learning or the like, for example, so that the reconstructed image 351 reconstructed from the input image 301 approaches the input image 301, and so that the reconstructed image 352 reconstructed from the perturbed image 302 approaches the input image 301.

[0048] In formula (1), pgj is a pixel value, bgi is an offset value, q is the number of filter coefficients, and h is a nonlinear function. As shown in Fig. 5A, formula (1) is used to calculate the calculation results of each filter from the upper left to the lower right of the target image to calculate the feature amount fgi of an arbitrary filter i. For example, the matrix of the feature amount fgi calculated by the feature extractor G5 is set as the global feature amount FGi of the input image 301 or the perturbed image 302.

[0049]

number

[0050] The filter coefficient wlj shown in equation (2) is a coefficient calculated by machine learning or the like so that the reconstructed image 351 reconstructed from the input image 301 approaches the input image 301, and so that the reconstructed image 352 reconstructed from the perturbed image 302 approaches the input image 301.

[0051] In formula (2), plj is a pixel value, bli is an offset value, r is the number of filter coefficients, and h is a nonlinear function. As shown in Fig. 5B, formula (2) is used to calculate the calculation results of each filter from the upper left to the lower right of the target image, thereby calculating the feature amount fli of an arbitrary filter i. For example, the matrix of the feature amount fli calculated by the feature extractor L5 is set as the local feature amount FLi of the input image 301 or the perturbed image 302.

[0052]

number

[0053] (iii) Color and brightness characteristic correction unit 13 The color / brightness feature correction unit 13 is composed of adjustment units 321 and 322 and normalization units 331 and 332. The adjustment unit 321 obtains a filter coefficient wdj shown in Equation (3) by machine learning or the like so that the corrected feature amount of the global feature amount FGi (global feature amount G1) of the input image 301 and the corrected feature amount of the global feature amount FGi (global feature amount G2) of the perturbed image 302 approach each other.

[0054] In formula (3), pdj is the value of the global feature, bdi is the offset value, s1 is the number of filter coefficients, and h is a nonlinear function. As shown in FIG. 6A, formula (3) is used to calculate the calculation results of each filter for the global features G1 and G2, thereby calculating the feature fdi of an arbitrary filter i. For example, the matrix of the feature fdi calculated from the global feature G1 by the feature extractor D1 is set as the global feature G11, and the matrix of the feature fdi calculated from the global feature G2 by the feature extractor D1 is set as the global feature G21. Alternatively, the matrix obtained by adding the global feature G1 and the correction feature fdi calculated from the global feature G1 by the feature extractor D1 is set as the global feature G11, and the matrix obtained by adding the global feature G2 and the correction feature fdi calculated from the global feature G2 by the feature extractor D1 is set as the global feature G21.

[0055]

number

[0056] The adjustment unit 322 obtains a filter coefficient wej shown in Equation (4) by machine learning or the like so that the corrected feature amount of the local feature amount FLi (local feature amount L1) of the input image 301 and the corrected feature amount of the local feature amount FLi (local feature amount L2) of the perturbed image 302 approach each other.

[0057] In formula (4), pej denotes the value of the local feature, bei denotes the offset value, s2 denotes the number of filter coefficients, and h denotes a nonlinear function. As shown in FIG. 6B, formula (4) is used to obtain the calculation results of each filter for the local features L1 and L2, thereby obtaining the feature fei of an arbitrary filter i. For example, the matrix of the feature fei obtained from the local feature L1 by the feature extractor D2 is defined as the local feature L11, and the matrix of the feature fei obtained from the local feature L2 by the feature extractor D2 is defined as the local feature L21. Alternatively, the matrix obtained by adding the local feature L1 and the corrected feature fei obtained from the local feature L1 by the feature extractor D2 is defined as the local feature L11, and the matrix obtained by adding the local feature L2 and the feature fei obtained from the local feature L2 by the feature extractor D2 is defined as the local feature L21.

[0058]

number

[0059] The normalization unit 331 normalizes the global feature amount G11 as shown in formula (5). In formula (5), FG indicates the global feature amount G11, and FG max indicates the maximum value of the global feature G11. max indicates the maximum value of the local feature L11. Therefore, the global feature G11 is corrected to the global feature G111 using formula (5).

[0060]

number

[0061] The normalization unit 332 normalizes the global feature G21 as shown in formula (5). In formula (5), FG indicates the global feature G21, and FG max indicates the maximum value of the global feature G21. max indicates the maximum value of the local feature L21. Therefore, the global feature G21 is corrected to the global feature G211 using formula (5).

[0062] The global feature G111 and the local feature L11 are combined into a set to form a normalized feature NF1. The global feature G211 and the local feature L21 are combined into a set to form a normalized feature NF2. (iv) Image reconstruction unit 14

[0063] The image reconstructing unit 14 uses the decoder 340 to create a reconstructed image 351 from the normalized feature NF1 and a reconstructed image 352 from the normalized feature NF2. As shown in Fig. 7 and formula (6), x, which is the global feature G111, is copied from the upper left to the lower right so that the vertical and horizontal sizes become the same as those of the local feature L11 to create a global feature G1111, and the global feature G1111 and the local feature L11 are concatenated to create a corrected color / brightness feature CF1.

[0064] As shown in FIG. 7 and Equation (6), the global feature G211 x is multiplied by the vertical H y and horizontal W y Then, the global feature G2111 and the local feature L21 are concatenated to create the corrected color and brightness feature CF2.

[0065]

number

[0066] The decoder 340 obtains the filter coefficient wpj shown in formula (7) by machine learning or the like so that the input image 301 and the reconstructed image 351 reconstructed from the corrected color and brightness feature CF1 approach each other. In formula (7), ppj is the value of the corrected color and brightness feature CF1, bpi is the offset value, t1 is the number of filter coefficients, and h is a nonlinear function. As shown in FIG. 8, the feature fpi of an arbitrary filter i is obtained by obtaining the calculation result of each filter for the corrected color and brightness feature CF1 using formula (7). For example, the reconstructed image 351 is created from the matrix of the feature fpi obtained by the feature extractor U1 from the corrected color and brightness feature CF1. For example, the reconstructed image 351 may be created by multiplying the feature fpi by 255.

[0067]

number

[0068] Further, the decoder 340 obtains the filter coefficients wpj shown in the formula (7) by machine learning or the like so that the input image and the reconstructed image 2 reconstructed from the corrected color / brightness feature value CF2 approach each other. In equation (7), ppj is the value of the corrected color / brightness feature CF2, bpi is the offset value, t1 is the number of filter coefficients, and h is a nonlinear function. As shown in FIG. 8, equation (7) is used to calculate the calculation results of each filter for the corrected color / brightness feature CF2 to calculate the feature fpi of an arbitrary filter i. For example, a reconstructed image 2 is created from the matrix of feature fpi calculated by feature extractor U1 from the corrected color / brightness feature CF2. For example, a reconstructed image 352 may be created by multiplying feature fpi by 255.

[0069] When creating a color / brightness corrector using machine learning, for example, the parameters (filter coefficients, offset values, etc.) from equations (1) to (7) are calculated so that the overall cost shown in equation (12), which combines the four loss values ​​shown in equations (8) to (11), is reduced. MSE is the mean square error, Input is the input image 301, Output1 is the reconstructed image 351, Output2 is the reconstructed image 352, FeatureG11 is the global feature G11, FeatureG21 is the global feature G21, FeatureL11 is the local feature L11, and FeatureL21 is the local feature L21.

[0070]

number

[0071]

number

[0072]

number

[0073]

number

[0074]

number

[0075] (v) Output section 15 When creating a color / brightness corrector during machine learning, the output unit 15 displays, as an example, the input image, the perturbed image, reconstructed images 1 and 2 of the input image and the perturbed image, each loss value, and the cost value in a GUI (graphical user interface) shown in FIG.

[0076] The example in Figure 14 displays the input image, the perturbed image, the reconstructed image 1, the reconstructed image 2, four loss values ​​(0.09, 0.08, 0.06, 0.03), and the cost value 0.26.

[0077] The output unit 15 stores in memory 90 the input image (learning image, etc.), the parameters (filter coefficients, weights, offset values, etc.) of the color / brightness corrector consisting of the color / brightness feature extraction unit 12, the color / brightness feature correction unit 13 and the image reconstruction unit 14, the reconstructed images 1 and 2, the display information generated by the output unit 15, etc.

[0078] <Configuration and operation of each part> The configuration and operation of each element during evaluation will be described in detail below. (i) Input unit 10 An input image for evaluation (target image) 301 is output to the color / brightness feature extraction unit 12 .

[0079] (ii) Color and brightness feature extraction unit 12 A global feature amount FGi of the input image 301 and a local feature amount FLi of the input image are calculated using the color / brightness feature extraction unit 12 of the color / brightness corrector shown in FIG. 9, which is created during machine learning.

[0080] (iii) Color and brightness characteristic correction unit 13 Using the color / brightness feature correction unit 13 of the color / brightness corrector shown in Fig. 9 created during machine learning, the adjustment unit 321 calculates a global feature G11 from the global feature G1. The adjustment unit 322 calculates a local feature L11 from the local feature L1. The normalization unit 331 corrects the global feature G11 to a global feature G111. The global feature G111 and the local feature L11 are combined into a set to form a normalized feature NF1.

[0081] (iv) Image reconstruction unit 14 A reconstructed image 351 is generated from the normalized feature amount NF1 using the decoder 340 of the image reconstruction unit 14 of the color / brightness corrector shown in FIG. 9, which is generated during machine learning.

[0082] (v) Feature Extraction Unit 16 The feature amount of the reconstructed image 351 is calculated. An example of calculating the feature amount is shown in Fig. 10. CNN in Fig. 10 stands for Convolutional Neural Network.

[0083] For example, the feature extractor A1 is used to obtain the feature amount FAi of the object (for example, tissue, cell, etc.) of the reconstructed image 351 from the reconstructed image 351 according to equation (13).

[0084] The filter coefficient wj shown in equation (13) is a coefficient obtained by machine learning or the like so as to distinguish objects other than the object to be detected from objects other than the object to be detected (e.g., normal tissue or normal cells from normal tissue or normal cells, etc.) and to distinguish objects to be detected from objects to be detected (e.g., abnormal tissue or abnormal cells from abnormal tissue or abnormal cells, etc.).

[0085] In equation (13), pj is a pixel value of the reconstructed image 1, bi is an offset value, m is the number of filter coefficients, and h is a nonlinear function. As shown in Fig. 11, by using equation (13) to calculate the calculation results of each filter 401 from the upper left to the lower right of an input image 301, which is a target image (e.g., a pathological tissue image), a feature value fi of an arbitrary filter i is calculated. For example, the matrix of feature values ​​fi calculated by feature extractor A1 is set as feature value FAi of input image 301. A method for creating feature extractor A1 will be described in the classification unit 17 described later.

[0086]

number

[0087] (vi) Identification unit 17 As shown in FIG. 12, the discrimination unit 17 uses the feature amount FAi (matrix f) of the feature extractor A1 obtained by the feature extraction unit 16 to calculate a value of likelihood of an object to be detected (likelihood of abnormal tissue, likelihood of a lesion, etc.) by the logistic regression process in the logistic regression layer 501 according to equation (14), and judges whether an object (e.g., tissue or cell) in the input image 301 is an object to be detected (abnormal tissue, etc.) or an object other than an object to be detected (normal tissue, etc.). In equation (14), w indicates a weight matrix, b indicates an offset value, g indicates a nonlinear function, and y indicates a discrimination result, and the weight of w and the offset value of b are obtained by machine learning using a learning image in advance. In addition, the target image is discriminated using the weight of w and the offset value of b obtained during learning, and the discrimination result y is calculated.

[0088]

number

[0089] The identification unit 17 learns the features of the object using the feature extraction unit 16 by, for example, a known machine learning technique so that, if an object in the input reconstructed image 351 is other than the object to be detected, it determines that the object is other than the object to be detected by, for example, logistic regression processing in the logistic regression layer 501 according to equation (14).

[0090] The reconstructed image 351 is, for example, an HE stained image, and the objects to be detected are, for example, abnormal tissues and abnormal cells, and the objects other than the objects to be detected are, for example, normal tissues and normal cells.

[0091] If an object in an input image is an object to be detected, the object feature is learned by the feature extraction unit 16 so that the object is determined to be an object to be detected by logistic regression processing. For example, a Convolutional Neural Network may be used as a machine learning technique.

[0092] 13, during learning, the identification unit 17 creates a feature extractor A1 that calculates a feature amount fi (assumed to be FAi) of the reconstructed image 351 by machine learning, so as to determine an object to be detected as an object to be detected and to determine an object other than the object to be detected as an object other than the object to be detected, using the reconstructed image 351. The feature extractor A1 is created using the feature extraction unit 16 according to equations (13) and (14).

[0093] During learning, the classification unit 17 repeatedly performs the feature extraction unit 16 and the classification unit 17 using a plurality of learning images to obtain the weight w, the filter coefficient wj, and the offset values ​​b and bi shown in equations (13) and (14), and creates a feature extractor A1 that calculates the feature amount FAi of the reconstructed image 351 from the reconstructed image 351 using the feature extraction unit 16. The classification unit 17 also calculates the classification result of the reconstructed image 351 using the feature extractor A1.

[0094] The discrimination unit 17 stores the determined weight w, filter coefficient wj, and offset values ​​b and bi in the memory 90. (vii) Output section 15

[0095] The output unit 15 uses the classification result y obtained by the classifier to display the result of the object-likeness determination (for example, lesion-likeness determination) on a GUI (Graphical User Interface) shown in FIG. 16A or FIG. 16B, for example.

[0096] Fig. 16A is an example of the case of the breast, showing the classification results of non-tumor and tumor (abnormal tissue, abnormal cells, etc.). In the example of Fig. 16A, the discrimination unit 17 classifies the input reconstructed image 1 of the breast as including a tumor, which is an abnormal tissue / cell, and calculates the object-likeliness value of the tumor to be 0.89.

[0097] Fig. 16B is a diagram showing an example of a display when the image button is pressed in Fig. 16A, which shows an example of displaying an input image, a reconstructed image obtained by reconstructing the input image using a color / brightness corrector, and a classification score.

[0098] Furthermore, when an input image (target image) 301 is input from the input unit 10 and the output unit 15 displays the classification results of each classifier, if an object in the image is determined to be an object (e.g., abnormal tissue or abnormal cell), the output unit 15 may draw a detection frame 521 in the input target image 301 to indicate the location of the object to be detected (e.g., a location suspected of being abnormal tissue or abnormal cell) as shown in Fig. 15. On the other hand, if the object is determined to be normal tissue or normal cell, the input target image may be displayed as is without drawing the detection frame 521 on the input target image. Furthermore, as shown in Fig. 15, the result 522 of the determined object-likeliness (e.g., tumor) is displayed.

[0099] The output unit 15 stores in a memory 90 the input image, the parameters (filter coefficients, weights, offset values, etc.) of the color / brightness corrector consisting of the color / brightness feature extraction unit 12, the color / brightness feature correction unit 13 and the image reconstruction unit 14, the reconstructed image, the parameters (filter coefficients, weights, offset values, etc.) of the classifier consisting of the feature extraction unit 16 and the classification unit 17, the classification results, the classification values, the display information generated by the output unit 15, etc.

[0100] <Processing procedure of image processing device> 17 is a flowchart for explaining the operation of the image processing device 1 according to the embodiment during machine learning. In the following, each processing unit (the input unit 10, the color / brightness feature extraction unit 12, etc.) will be described as the main operating unit, but the description may be rephrased so that the CPU 201 is the main operating unit and executes each processing unit as a program.

[0101] (i) Step 1701 The input unit 10 receives a learning image, and outputs the input image to the perturbed image creation unit 11 and the color / brightness feature extraction unit 12 .

[0102] (ii) Step 1702 The perturbed image creating unit 11 creates a perturbed image while randomly perturbing each value (R value, G value, B value, etc.) of the image, the γ value, etc.

[0103] (iii) Step 1703 The color / brightness feature extraction unit 12 uses machine learning to obtain global features FGi of multiple learning input images 301 using a filter according to the above-mentioned formula (1), and creates a feature extractor G5 by repeatedly updating the parameters of the feature extractor. For the feature FGi, a filter coefficient wgj and an offset value bgi are obtained. Thus, a global feature G1 is obtained from the input image 301, and a global feature G2 is obtained from the perturbed image 302.

[0104] Furthermore, the color / brightness feature extraction unit 12 uses machine learning to obtain a local feature amount FLi of the input image 301 using a filter according to the above-mentioned formula (2), and creates a feature extractor L5. For the feature amount FLi, a filter coefficient wlj and an offset value bli are obtained. Thus, a local feature amount L1 is obtained from the input image 301, and a local feature amount L2 is obtained from the perturbed image 302.

[0105] (iv) Step 1704 The adjustment unit 321 of the color / brightness feature correction unit 13 uses a filter to obtain a feature FDi from the global features G1 and G2 by machine learning using the above-mentioned formula (3), and creates a feature extractor D1. For the feature FDi, a filter coefficient wdj and an offset value bdi are obtained. Therefore, a global feature G11 is obtained from the global feature G1, and a global feature G21 is obtained from the global feature G2.

[0106] Furthermore, the adjustment unit 322 of the color / brightness feature correction unit 13 uses a filter to obtain a feature amount FEi from the local feature amounts L1 and L2 by machine learning using the above-mentioned formula (4), and creates a feature extractor D2. For the feature amount FEi, a filter coefficient wej and an offset value bei are obtained. Thus, a local feature amount L11 is obtained from the local feature amount L1, and a local feature amount L21 is obtained from the local feature amount L2.

[0107] (v) Step 1705 The normalization unit 331 of the color / brightness feature corrector 13 corrects the global feature G11 to a global feature G111 by the above-mentioned equation (5). Also, the normalization unit 332 of the color / brightness feature corrector 13 corrects the global feature G21 to a global feature G211 by the above-mentioned equation (5).

[0108] (vi) Step 1706 The image reconstruction unit 14 creates a global feature G1111 from the global feature G111 using the above formula (6), concatenates the global feature G1111 and the local feature L11, and creates a corrected color / brightness feature CF1. Also, the image reconstruction unit 14 creates a global feature G2111 from the global feature G211 using the above formula (6), concatenates the global feature G2111 and the local feature L21, and creates a corrected color / brightness feature CF2. The image reconstruction unit 14 also uses machine learning to obtain a feature amount FPi from the corrected color / brightness feature amount CF1 using a filter according to the above-mentioned formula (7), and creates a feature extractor U1. For the feature amount FPi, a filter coefficient wpj and an offset value bpi are obtained. Thus, a reconstructed image 351 is created from the corrected color / brightness feature amount CF1.

[0109] The image reconstruction unit 14 also uses machine learning to obtain a feature amount FPi from the corrected color / brightness feature amount CF2 using a filter according to the above-mentioned formula (7), and creates a feature extractor U1. For the feature amount FPi, a filter coefficient wpj and an offset value bpi are obtained. Thus, a reconstructed image 352 is created from the corrected color / brightness feature amount CF2.

[0110] (vii) Step 1707 The color / brightness feature extraction unit 12, color / brightness feature correction unit 13, and image reconstruction unit 14 calculate the overall cost of equation (12) using equations (8) to (11) above, and use machine learning to update each parameter (each filter coefficient, each offset value) in equations (1) to (7) above so as to reduce the cost, thereby creating a color / brightness corrector.

[0111] (viii) Step 1708 The output unit 15 stores in the memory 90 each of the parameters (each filter coefficient, each offset value, etc.) in equations (1) to (7).

[0112] 18 is a flowchart for explaining the operation at the time of evaluation of the image processing device 1 according to the embodiment. In the following, each processing unit (input unit 10, color / brightness feature extraction unit 12, etc.) will be described as the main operating unit, but it may be interpreted as the CPU 201 being the main operating unit, and the CPU 201 executing each processing unit as a program.

[0113] (i) Step 1801 The input unit 10 receives an evaluation image (also called an input image or a target image) and outputs the evaluation image to a color / brightness feature extraction unit 12 .

[0114] (ii) Step 1802 The color / brightness feature extraction unit 12 reads the parameters (filter coefficients, offset values) of equations (1) and (2) from the memory, and calculates a global feature amount G1 and a local feature amount L1.

[0115] (iii) Step 1803 The color / brightness feature correction unit 13 reads each parameter (each filter coefficient, each offset value) of equations (3) and (4) from the memory, and obtains a global feature G11 from the global feature G1 and a local feature L11 from the local feature L1.

[0116] (iv) Step 1804 The color / brightness feature correction unit 13 reads the parameters (filter coefficient, offset value) of the formula (5) from the memory, and corrects the global feature G11 to a global feature G111.

[0117] (v) Step 1805 The image reconstruction unit 14 creates a global feature G1111 from the global feature G111 using the above-mentioned equation (6), and concatenates the global feature G1111 and the local feature L11 to create a corrected color / brightness feature CF1.

[0118] Moreover, the image reconstructing unit 14 reads the parameters (filter coefficient, offset value) of equation (7) from the memory, and creates a reconstructed image 351 from the corrected color / brightness feature value CF1.

[0119] (vi) Step 1806 The feature extraction unit 16 reads the parameters (filter coefficient, offset value) of equation (13) from the memory, and obtains the feature amount FAi of the object (for example, tissue, cell, etc.) of the reconstructed image 351 using a filter according to the above equation (13).

[0120] (vii) Step 1807 The discrimination unit 17 reads the parameters (weight w, offset value b, etc.) of equation (14) from the memory, obtains a discrimination result from the feature amount FAi according to equation (14), and calculates a discrimination value of the object-likeness (e.g., lesion-likeness, etc.) of the object in the reconstructed image 351. The parameters read in each of the above steps have been adjusted by learning as described above.

[0121] (viii) Step 1808 The output unit 15 stores the calculated identification result and identification value in the memory 90.

[0122] According to the first embodiment, a color / brightness corrector is created that extracts and corrects color / brightness features while using a perturbed image during machine learning, and generates a reconstructed image in which color / brightness differences are suppressed. Furthermore, during evaluation, a reconstructed image is generated from an evaluation image using the color / brightness corrector, and the reconstructed image is identified using a classifier, making it possible to identify objects (e.g., abnormal tissue, etc.) in an image with high accuracy. Note that the color / brightness corrector learning and evaluation processes of the first embodiment may be executed by different computer systems, that is, the evaluation processing system and the learning system.

[0123] (2) Second embodiment As shown in Fig. 19, an image processing device 2 for machine learning according to the second embodiment includes many of the same components as the image processing device 1 of the first embodiment in Fig. 1, but unlike the first embodiment, does not include a color / brightness feature extraction unit 12 and a color / brightness feature correction unit 13, but includes a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24. Therefore, the following mainly describes the components that are different from Fig. 1.

[0124] The image processing device 2 of this embodiment extracts color and brightness features and corrects the color and brightness features in each of the two encoders during machine learning, and creates color and brightness correctors in the decoder that create reconstructed images for the input image and the perturbed image from the corrected features. During evaluation, a reconstructed image is generated from the evaluation image using the color and brightness correctors, and the reconstructed image is classified using a classifier, thereby identifying objects (e.g., abnormal tissue) in the image with high accuracy.

[0125] <Configuration and operation of each part> Below, we will explain in detail the configuration and operation of each element during machine learning that differs from Figure 1. (i) Color and brightness feature extraction and correction unit 18 20A and 20B show a configuration example of the color / brightness feature extraction / correction unit 18. The color / brightness feature extraction / correction unit 18 includes an encoder 71 and an encoder 72. The encoder 71 includes encoders 711 to 71M, and adjustment units 731 to 73M. The encoder 72 includes encoders 721 to 72M, and adjustment units 741 to 74M, where M represents any natural number equal to or greater than 2. The encoders 71 and 72 include normalization units 751 and 752, respectively.

[0126] The color / brightness feature extraction / correction unit 18 obtains the color / brightness feature of the input image, and then corrects the color / brightness feature. An example of obtaining color / brightness feature (global feature, local feature) is shown in Figs. 20A and 20B. The encoder and decoder in Figs. 20A and 20B can use CNN.

[0127] For example, the encoder 711 obtains a global feature G711 from the input image 301, and obtains a global feature G721 from the perturbed image 302. The encoder 71M obtains a global feature G71M from the global feature of the input image 301 from the previous stage. The encoder 71M obtains a global feature G72M from the global feature of the perturbed image 302 from the previous stage.

[0128] The encoder 721 obtains a local feature L711 from the input image 301, and obtains a local feature L721 from the perturbed image 302. The encoder 72M obtains a local feature L71M from the local feature of the input image 301 from the previous stage. The encoder 72M obtains a local feature L72M from the local feature of the perturbed image 302 from the previous stage.

[0129] There are two methods: one is to add the output of the adjustment unit 73N and the global feature G71N to generate the global feature G711N for the next stage, and the other is to use the output of the adjustment unit 73N as the corrected global feature G711N as is (without adding the global feature G71N). In either method, the output of the adjustment unit 73N becomes the corrected global feature G711N. The same explanation is also applied to the global feature G72N, and further to the adjustment unit 74N and the local feature L71N or L72N.

[0130] As shown in Fig. 21A, for example, an encoder 71N uses a feature extractor GN to obtain a global feature FGNi of color and brightness from an input image, a perturbed image, or a global feature according to equation (15), where N is any natural number from 1 to M. As shown in Fig. 21B, an encoder 72N uses a feature extractor LN to obtain a local feature FLNi of color and brightness from an input image, a perturbed image, or a local feature according to equation (16).

[0131] The filter coefficient wgnj shown in equation (15) is a coefficient calculated by machine learning or the like so that the reconstructed image reconstructed from the input image approaches the input image, and so that the reconstructed image reconstructed from the perturbed image approaches the input image.

[0132] In formula (15), pgnj is a pixel value, bgni is an offset value, q is the number of filter coefficients, and h is a nonlinear function. As shown in Fig. 21A, by using formula (15) to obtain the calculation results of each filter from the upper left to the lower right of the target image, the feature amount fgni of an arbitrary filter i is obtained. For example, the matrix of the feature amount fgni obtained by the feature extractor GN is set as the global feature amount FGNi of the input image, the perturbed image, or the global feature amount.

[0133]

number

[0134] The filter coefficient wlnj shown in equation (16) is a coefficient calculated by machine learning or the like so that the reconstructed image reconstructed from the input image approaches the input image, and so that the reconstructed image reconstructed from the perturbed image approaches the input image.

[0135] In formula (16), plnj is a pixel value, blni is an offset value, r is the number of filter coefficients, and h is a nonlinear function. As shown in Fig. 21B, by using formula (16) to obtain the calculation results of each filter from the upper left to the lower right of the target image, the feature amount flni of an arbitrary filter i is obtained. For example, the matrix of the feature amount flni obtained by the feature extractor LN is set as the local feature amount FLNi of the input image, the perturbed image, or the local feature amount.

[0136]

number

[0137] Next, the adjustment unit 73N obtains the filter coefficients wdnj shown in equation (17) by machine learning or the like so that the global feature amount FGNi (global feature amount G71N) of the input image and the global feature amount FGNi (global feature amount G72N) of the perturbed image approach each other.

[0138] In formula (17), pdnj is the value of the global feature, bdni is the offset value, s1 is the number of filter coefficients, and h is a nonlinear function. As shown in Fig. 22A, formula (17) is used to calculate the calculation results of each filter for global features G71N and G72N to calculate the feature fdni of an arbitrary filter i. For example, the matrix of the feature fdni calculated from the global feature G71N by the feature extractor DN1 is set as the global feature G711N, and the matrix of the feature fdni calculated from the global feature G72N by the feature extractor DN1 is set as the global feature G721N.

[0139]

number

[0140] The adjustment unit 74N obtains the filter coefficient wenj shown in equation (18) by machine learning or the like so that the local feature amount FLNi (local feature amount L71N) of the input image and the local feature amount FLNi (local feature amount L72N) of the perturbed image approach each other.

[0141] In formula (18), penj is the value of the local feature, beni is the offset value, s2 is the number of filter coefficients, and h is a nonlinear function. As shown in Fig. 22B, formula (18) is used to find the calculation results of each filter for local features L71N and L72N to find the feature feni of an arbitrary filter i. For example, the matrix of feature feni found from local feature L71N by feature extractor DN2 is local feature L711N, and the matrix of feature feni found from local feature L72N by feature extractor DN2 is local feature L721N.

[0142]

number

[0143] Using equations (15) and (17), the above process is performed by all of the encoders 71N and the adjustment units 73N, the extraction and correction of the global features are repeated, and finally, the global features G711M and the global features G721M are obtained. Also, using equations (16) and (18), the above process is performed by all of the encoders 72N and the adjustment units 74N, the extraction and correction of the local features are repeated, and finally, the local features L711M and the local features L721M are obtained.

[0144] The normalization unit 751 normalizes the global feature G711M corrected by the adjustment unit 73M as shown in equation (19). In equation (19), FG indicates the global feature G711M, and FG max indicates the maximum value of the global feature G711M. Also, FL max indicates the maximum value of the local feature L711M. Therefore, the global feature G711M is corrected to the global feature G7111M using equation (19).

[0145]

number

[0146] The normalization unit 752 normalizes the global feature G721M corrected by the adjustment unit 73M as shown in equation (19). In equation (19), FG indicates the global feature G721M, and FG max indicates the maximum value of the global feature G721M. max indicates the maximum value of the local feature L721M. Therefore, the global feature G721M is corrected to the global feature G7211M using equation (19).

[0147] The global feature G7111M and the local feature L711M are combined into a set to form a normalized feature NM1. The global feature G7211M and the local feature L721M are combined into a set to form a normalized feature NM2.

[0148] (ii) Image reconstruction unit 24 The image reconstructing unit 24 uses a decoder to create a reconstructed image 351 from the normalized feature NM1 and a reconstructed image 352 from the normalized feature NM2. As shown in Fig. 7 and formula (6), x, which is the global feature G7111M, is copied from the upper left to the lower right so that the vertical and horizontal sizes become the same as those of the local feature L711M to create a global feature G71111M, and the global feature G71111M and the local feature L711M are concatenated to create a corrected color / brightness feature CFM1.

[0149] As shown in FIG. 7 and Equation (6), the global feature G7211M x is multiplied by the vertical H y and horizontal W y Then, global feature G72111M and local feature L721M are concatenated to create corrected color and brightness feature CFM2.

[0150] The decoder is similar to the image reconstruction unit 14 in the first embodiment. The decoder creates a reconstructed image 351 from a matrix of feature amounts fpi obtained by a feature extractor U1 from the corrected color / brightness feature amounts CFM1. The decoder also obtains a filter coefficient wpj shown in equation (7) by machine learning or the like so that the input image 301 and the reconstructed image 351 reconstructed from the corrected color / brightness feature amounts CFM1 approach each other.

[0151] The decoder creates a reconstructed image 352 from the matrix of features fpi obtained by the feature extractor U1 from the corrected color / brightness features CFM2. The decoder also obtains the filter coefficients wqj shown in equation (7) by machine learning or the like so that the input image 301 and the reconstructed image 352 reconstructed from the corrected color / brightness features CFM2 approach each other.

[0152] <Hardware configuration of image processing device> An example of the hardware configuration of the image processing device 2 according to this embodiment has a configuration similar to that shown in FIG. 2 , but differs from the image processing device 1 according to the first embodiment in that a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24 are included in a memory 202, instead of the color / brightness feature extraction unit 12 and the color / brightness feature correction unit 13.

[0153] The storage device 203 of the image processing device 2 stores the parameters of the color / brightness corrector generated by the color / brightness feature extraction / correction unit 18 and the image reconstruction unit 24, the reconstructed image, and the like.

[0154] <Processing procedure of image processing device> 23 is a flowchart for explaining the operation of the image processing device 2 according to this embodiment during machine learning. In the following, each processing unit (the input unit 10, the color / brightness feature extraction / correction unit 18, etc.) will be described as the main operating unit, but the description may be rephrased so that the CPU 201 is the main operating unit and executes each processing unit as a program.

[0155] (i) Step 2301 The input unit 10 receives a learning image, and outputs the input image to the perturbed image creation unit 11 and the color / brightness feature extraction / correction unit 18 .

[0156] (ii) Step 2302 The perturbed image creating unit 11 creates a perturbed image while randomly perturbing each value (R value, G value, B value, etc.) of the image, the γ value, etc.

[0157] (iii) Step 2303 The color / brightness feature extraction / correction unit 18 uses a filter to obtain a global feature FGNi from the input image, the perturbed image, or the global features G711N and G721N of the previous stage by machine learning according to the above-mentioned formula (15), and creates a feature extractor GN. For the global feature FGNi, a filter coefficient wgnj and an offset value bgni are obtained. Therefore, the current global feature G71N is obtained from the input image or the global feature G711N of the previous stage, and the current global feature G72N is obtained from the perturbed image or the global feature G721N of the previous stage.

[0158] Furthermore, the color / brightness feature extraction / correction unit 18 uses a filter to obtain local feature amounts FLNi from the input image, perturbed image, or previous local feature amounts L711N and L721N by machine learning according to the above-mentioned formula (16), and creates a feature extractor LN. For the local feature amounts FLNi, a filter coefficient wlnj and an offset value blni are obtained. Thus, the current local feature amount L71N is obtained from the input image or previous local feature amount L711N, and the current local feature amount L72N is obtained from the perturbed image, local feature amount L72N, or previous local feature amount L721N.

[0159] (iv) Step 2304 The adjustment unit 73N of the color / brightness feature extraction / correction unit 18 uses a filter to obtain feature FDNi from global feature G71N and G72N by machine learning using the above-mentioned formula (17), and creates a feature extractor DN1. For feature FDNi, a filter coefficient wdnj and an offset value bdni are obtained. Therefore, the current global feature G711N is obtained from the global feature G71N, and the current global feature G721N is obtained from the global feature G72N.

[0160] Furthermore, the adjustment unit 74N of the color / brightness feature extraction / correction unit 18 uses a filter to obtain a feature amount FENi from the local feature amounts L71N and L72N by machine learning using the above-mentioned formula (18), and creates a feature extractor DN2. For the feature amount FENi, a filter coefficient wenj and an offset value beni are obtained. Thus, the current local feature amount L711N is obtained from the local feature amount L71N, and the current local feature amount L721N is obtained from the local feature amount L72N.

[0161] (v) Step 2305 If the color / brightness feature extraction / correction unit 18 has already calculated the global feature amounts G711N, G721N and the local feature amounts L711N, L721N for all adjustment units 73N, 74N, the process proceeds to step 2306. Otherwise, the process returns to step 2303.

[0162] (vi) Step 2306 A normalization unit 751 in the color / brightness feature extraction / correction unit 18 corrects the global feature G711N to a global feature G7111N by the above-mentioned equation (19). Also, a normalization unit 752 in the color / brightness feature extraction / correction unit 18 corrects the global feature G721N to a global feature G7211N by the above-mentioned equation (19).

[0163] (vii) Step 2307 The image reconstruction unit 24 creates a global feature G71111N from the global feature G7111N using the above formula (6), concatenates the global feature G71111N and the local feature L711N, and creates a corrected color / brightness feature CFM1. Also, the image reconstruction unit 24 creates a global feature G72111N from the global feature G7211N using the above formula (6), and concatenates the global feature G72111N and the local feature L721N, and creates a corrected color / brightness feature CFM2.

[0164] The image reconstruction unit 24 also uses machine learning to obtain a feature amount FPi from the corrected color / brightness feature amount CFM1 using a filter according to the above-mentioned formula (7), and creates a feature extractor U1. For the feature amount FPi, a filter coefficient wpj and an offset value bpi are obtained. Thus, a reconstructed image 351 is created from the corrected color / brightness feature amount CFM1.

[0165] The image reconstruction unit 24 also uses machine learning to obtain a feature amount FPi from the corrected color / brightness feature amount CFM2 using a filter according to the above-mentioned formula (7), and creates a feature extractor U1. For the feature amount FPi, a filter coefficient wpj and an offset value bpi are obtained. Thus, a reconstructed image 352 is created from the corrected color / brightness feature amount CFM2.

[0166] (viii) Step 2308 The color / brightness feature extraction / correction unit 18 and the image reconstruction unit 24 calculate the overall cost of equation (12) using equations (8) to (11) above, and update the parameters (filter coefficients, offset values) of equations (15) to (19) and equation (7) above using machine learning to reduce the cost, thereby creating a color / brightness corrector. Note that in equations (10) and (11) above, Feature G11 indicates global feature G711N, Feature G21 indicates global feature G721N, Feature L11 indicates local feature L711N, and Feature L21 indicates local feature L721N.

[0167] (ix) Step 2309 The output unit 15 stores the parameters (each filter coefficient, each offset value, etc.) of equations (15) to (19) and equation (7) in the memory 90.

[0168] As shown in Fig. 24, the image processing device 2 at the time of evaluation according to the second embodiment includes many of the same components as the image processing device 1 of Fig. 4 of the first embodiment, but unlike the first embodiment, does not include the color / brightness feature extraction unit 12 and the color / brightness feature correction unit 13, but includes a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24. Therefore, here, the components different from Fig. 4 will be described.

[0169] The parameter values ​​of the color / brightness corrector created during learning are read from the memory 90, and the parameters of the color / brightness feature extraction / correction unit 18 and image reconstruction unit 24 are set.

[0170] For an image input from an input unit 10, a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24 are used to generate a reconstructed image in which color / brightness differences are suppressed.

[0171] 25 is a flowchart for explaining the operation of the image processing device 2 according to this embodiment at the time of evaluation. In the following, each processing unit (the input unit 10, the color / brightness feature extraction / correction unit 18, etc.) will be described as the main operating unit, but it may be interpreted as the CPU 201 being the main operating unit, and the CPU 201 executing each processing unit as a program.

[0172] (i) Step 2501 The input unit 10 receives an evaluation image, and outputs the evaluation image to a color / brightness feature extraction / correction unit 18 .

[0173] (ii) Step 2502 The color / brightness feature extraction / correction unit 18 reads the parameters (filter coefficients, offset values) of equations (15) and (16) from the memory, and calculates each global feature amount G71N and each local feature amount L71N.

[0174] (iii) Step 2503 The color / brightness feature extraction / correction unit 18 reads each parameter (each filter coefficient, each offset value) of equations (17) and (18) from the memory, and an adjustment unit 73N calculates a global feature G711N from the global feature G71N, and an adjustment unit 74N calculates a local feature L711N from the local feature L71N.

[0175] (iv) Step 2504 The color / brightness feature extraction / correction unit 18 reads the parameters (filter coefficient, offset value) of equation (19) from the memory, and corrects the global feature G711N to the global feature G7111N.

[0176] (v) Step 2505 The image reconstructing unit 24 creates a global feature G71111N from the global feature G7111N using the above-mentioned equation (6), and concatenates the global feature G71111N and the local feature L711N to create a corrected color / brightness feature CFM1.

[0177] Moreover, the image reconstructing unit 24 reads the parameters (filter coefficient, offset value) of equation (7) from the memory, and creates a reconstructed image 351 from the corrected color / brightness feature amount CFM1.

[0178] (vi) Step 2506 The feature extraction unit 16 reads the parameters (filter coefficient, offset value) of equation (13) from the memory, and obtains the feature amount FAi of the object (for example, tissue, cell, etc.) of the reconstructed image 351 using a filter according to the above equation (13).

[0179] (vii) Step 2507 The classification unit 17 reads the parameters of equation (14) (weight w, offset value b, etc.) from the memory, obtains a classification result from the feature amount FAi using equation (14), and calculates a classification value of the object-likeliness (e.g., lesion-likeliness) of the object in the reconstructed image 351.

[0180] (viii) Step 2508 The output unit 15 stores the calculated identification result and identification value in the memory 90.

[0181] According to the second embodiment, a color and brightness corrector is created that extracts and corrects color and brightness features for each layer while using a perturbation image during machine learning, and generates a reconstructed image in which color and brightness differences are suppressed. Furthermore, during evaluation, a reconstructed image is generated from an evaluation image using the color and brightness corrector, and the reconstructed image is identified using a classifier, making it possible to identify objects (e.g., abnormal tissue, etc.) in an image with high accuracy. Note that the color and brightness corrector learning and evaluation processes of the second embodiment may be executed by different computer systems, that is, the evaluation processing system and the learning system.

[0182] (3) Third embodiment 26 is a functional block diagram showing a configuration of an image processing system 2600 according to a third embodiment of the present specification. The image processing system 2600 includes a server 2603 and an image acquisition device 2605. The image processing system 2600 is also an image identification system.

[0183] The image acquisition device 2605 is, for example, a device such as a virtual slide device or a personal computer equipped with a camera, and has an imaging unit 2601 that captures image data, and a display unit 2604 that displays a reconstructed image / classification result transmitted from the server 2603. Although not shown, the image acquisition device 2605 has a communication device that transmits image data to the server 2603 and receives data transmitted from the server 2603.

[0184] The server 2603 has a storage unit 2602 that creates a color and brightness corrector for image data (input image) transmitted from an image acquisition device 2605 using the image processing device 1 or 2 according to the first or second embodiment, obtains a reconstructed image and classification results using the created color and brightness corrector, and stores the reconstructed image and classification results, etc. output from the image processing device 1 or 2.

[0185] Although not shown, the server 2603 has a communication device that receives image data transmitted from the image acquisition device 2605 and transmits data such as reconstructed images and identification results to the image acquisition device 2605.

[0186] The image processing device 1 or 2 uses a color / brightness corrector to generate a reconstructed image of the image captured by the imaging unit 2601, and classifies objects (e.g., tissues, cells, etc.) in the reconstructed image into the presence or absence of an object to be detected (e.g., abnormal tissue or abnormal cells (e.g., cancer), etc.). The display unit 2604 displays the reconstructed image and data such as the identification results transmitted from the server 2603 on the display screen of the image acquisition device 2605. The image acquisition device 2605 may be a regenerative medicine device or an iPS cell culture device having an imaging unit, or an MRI or ultrasound imaging device, etc.

[0187] According to the third embodiment, a color / brightness corrector obtained by the image processing device 1 or 2 is used to generate a reconstructed image of an image transmitted from a facility or the like at a different location, and objects (e.g., tissues, cells, etc.) in the reconstructed image are accurately classified as to whether they are objects to be detected (abnormal tissues, abnormal cells, etc.) using each parameter of the classifier, and data such as the reconstructed image and the classification results are transmitted to a facility or the like at a different location, and the reconstructed image and the classification results are displayed on a display unit of an image acquisition device in the facility or the like, thereby making it possible to provide an image processing system.

[0188] (4) Fourth embodiment 27 is a functional block diagram showing the configuration of an image processing system 2700 according to the fourth embodiment of the present invention. The image processing system 2700 includes a server 2703 and an image acquisition device 2705. The image processing system 2700 is also an image identification system.

[0189] The image acquisition device 2705 is, for example, a device such as a virtual slide device or a personal computer equipped with a camera, and includes an imaging unit 2701 for capturing image data, a storage unit 2704 for storing a color / brightness corrector transmitted from the server 2703, and an image processing device 1 or 2 for reading the color / brightness corrector transmitted from the server 2703, generating a reconstructed image of an image newly captured by the imaging unit 2701 of the image acquisition device 2705, and classifying objects (e.g., tissues, cells, etc.) in the reconstructed image as objects to be detected (e.g., abnormal tissues, abnormal cells, etc.) using a classifier of the image processing device 1 or 2 according to the first or second embodiment of the present invention. The image acquisition device 2705 includes a communication device (not shown) for transmitting image data to the server 2703 and receiving data transmitted from the server 2703.

[0190] The server 2703 has a storage unit 2702 which creates a color and brightness corrector from the image processing device 1 or 2 according to the first or second embodiment of the present invention for image data transmitted from the image acquisition device 2705, and stores the color and brightness corrector output from the image processing device 1 or 2. The server 2703 also accepts input designation of an input image or a teacher image by an operator. The server 2703 also presents each output (reconstructed image, classification result, etc.) of the image processing device 1 or 2 to the operator. Although not shown, the server 2703 has a communication device which receives image data transmitted from the image acquisition device 2705 and transmits a color and brightness corrector to the image acquisition device 2705.

[0191] The image processing device 1 or 2 generates a reconstructed image from an image captured by the imaging unit 2701, performs machine learning on objects (e.g., tissues, cells, etc.) in the reconstructed image to determine whether they are objects to be detected (e.g., normal tissues and cells are normal tissues and cells, and abnormal tissues and cells are abnormal tissues and cells, etc.), and creates a classifier that calculates the features of objects (e.g., tissues, cells, etc.) in images of facilities in different locations, etc. The storage unit 2704 stores the color and brightness corrector and the like transmitted from the server 2703 .

[0192] The image processing device 1 or 2 in the image acquisition device 2705 reads a color / brightness corrector, a classifier, etc. from the storage unit 2704, and uses the color / brightness corrector to generate a reconstructed image of an image newly captured by the imaging unit 2701 of the image acquisition device 2705.The image processing device 1 or 2 uses the classifier to classify objects (e.g., tissues, cells, etc.) in the reconstructed image as objects to be detected (e.g., abnormal tissues, abnormal cells, etc.) and displays data such as the reconstructed image and the classification results on the display screen of the output device 204 of the image processing device 1 or 2.

[0193] As the image acquisition device 2705, a regenerative medicine device having an imaging unit, an iPS cell culture device, an MRI or ultrasonic imaging device, or the like may be used.

[0194] According to the fourth embodiment, machine learning is performed on images transmitted from facilities at different locations to suppress color and brightness differences between images, a color and brightness corrector is created, the color and brightness corrector is transmitted to a facility at a different location, the color and brightness corrector is read in an image acquisition device at the facility, and a reconstructed image of the newly captured image is generated. Then, a classifier created by machine learning is read in to classify objects to be detected (e.g., normal tissues and cells as normal tissues and cells, abnormal tissues and cells as abnormal tissues and cells, etc.), and objects (e.g., tissues, cells, etc.) in the reconstructed image are classified as objects to be detected (e.g., abnormal tissues, abnormal cells, etc.), thereby making it possible to provide an image processing system.

[0195] The above-described embodiments can be modified as follows. In the perturbed image creation unit 11, perturbation may be performed on a color system other than RGB, such as HSV or HLS, or minute noise may be added, with the same effect.

[0196] Although a method of normalizing a global feature using the maximum value of a local feature has been described as an example of a normalization method in the normalization section of color / brightness feature correction section 13, a normalization method other than that described may be used, such as normalizing a local feature using the maximum value of a global feature, or normalizing the length of a feature vector at each coordinate of the global feature and the local feature, and the same effect may be obtained. Also, although normalization is performed using two normalization sections in color / brightness feature correction section 13, a normalization section other than two may be used and the same effect may be obtained.

[0197] In the image reconstruction units 14 and 24, the loss value is calculated using MSE as the loss function, but other loss functions such as cross entropy error may be used with the same effect. Also, the cost is calculated using four loss values, but the cost may be calculated using five loss values ​​including the loss value of the classification result and the correct answer, with the same effect.

[0198] In the feature extraction unit 16, a plurality of feature amounts are obtained using filters by machine learning, but other feature amounts such as HOG may also be used with the same effect. Furthermore, in the identification unit 17, negative log likelihood is used as the loss function, but squared error, hinge loss, or the like may also be used with the same effect.

[0199] The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a system or device, and a computer (or a CPU or MPU) of the system or device reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0200] Also, an operating system (OS) running on a computer may perform all or a part of the actual processing based on the instructions of the program code, and the functions of the above-mentioned embodiments may be realized by the processing. Furthermore, after the program code read from a storage medium is written into a memory on a computer, a CPU of the computer may perform all or a part of the actual processing based on the instructions of the program code, and the functions of the above-mentioned embodiments may be realized by the processing.

[0201] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a hard disk or memory of the system or device, or in a storage medium such as a CD-RW or CD-R, so that when used, the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage means or storage medium.

[0202] Finally, the processes and techniques described herein are not inherently related to any particular apparatus, but may be implemented by any suitable combination of components. Moreover, various types of general-purpose devices may be used in accordance with the methods described herein. It may be beneficial to construct a dedicated apparatus to perform the steps of the methods described herein. Also, various inventions may be formed by suitable combinations of multiple components disclosed in the embodiments. For example, some components may be omitted from all components shown in the embodiments. Furthermore, components across different embodiments may be combined as appropriate. The present invention has been described with reference to specific examples, which are in all respects illustrative rather than limiting. Those skilled in the art will recognize that there are numerous combinations of hardware, software, and firmware suitable for implementing the present invention. For example, the described software may be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Java, and the like.

[0203] Furthermore, in the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all of the control lines and information lines in the product are necessarily shown. All of the components may be connected to each other.

[0204] In addition, other implementations of the invention will be apparent to those of ordinary skill in the art from consideration of the specification and embodiments of the invention disclosed herein. Various aspects and / or components of the described embodiments may be used alone or in any combination. [Explanation of symbols]

[0205] 1. Image processing device 2. Image Processing Device 10 Input section 11. Perturbation image creation section 12. Color and brightness feature extraction section 13 Color and brightness characteristic correction section 14. Image reconstruction unit 15 Output section 16. Feature extraction section 17. Identification unit 18 Color and brightness feature extraction and correction section 24. Image reconstruction unit 91 Control section 2600···Image Processing System 2700···Image Processing System

Claims

1. An image processing device, one or more processors; one or more memories; the one or more memories store a corrector that generates a reconstructed image from an input image according to setting parameters; the setting parameters are adjusted so that a first reconstructed image generated by the corrector from the input first image and a second image of the same object but different color tone from the first image are closer to each other; the processor inputs an image of the object to the corrector to generate a reconstructed image of the object; The corrector performs the following in accordance with the setting parameters: Extracting features from the input target image; correcting the extracted feature amount; an image processing device that generates the reconstructed image of the target image from the corrected feature amount;

2. 2. The image processing device according to claim 1, The image processing device, wherein the first image is a perturbed image generated from the second image.

3. 2. The image processing device according to claim 1, The corrector is extracting a plurality of types of feature amounts of different sizes including the feature amount from the input target image; correcting each of the plurality of types of extracted feature amounts; an image processing device that generates the reconstructed image of the target image from the corrected feature amounts of the plurality of types;

4. 2. The image processing device according to claim 1, The one or more processors: extracting features from the reconstructed image of the target image; An image processing device that obtains an identification result of the reconstructed image of the target image from the feature amount extracted from the reconstructed image of the target image.

5. 2. The image processing device according to claim 1, The one or more processors perform the adjustment of the configuration parameters, wherein the adjustment comprises: perturbing the second image to generate the first image; inputting the first image into the corrector to generate the first reconstructed image; inputting the second image into the corrector to generate a second reconstructed image; adjusting the setting parameters so that a cost calculated from a predetermined loss becomes smaller; The image processing device, wherein the loss includes a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

6. 6. The image processing device according to claim 5, The image processing device, wherein the loss includes a difference between a feature amount obtained by correcting a feature amount extracted from the first image and a feature amount obtained by correcting a feature amount extracted from the second image.

7. 4. The image processing device according to claim 3, The one or more processors perform the adjustment of the configuration parameters, wherein the adjustment comprises: perturbing the second image to generate the first image; inputting the first image into the corrector to generate the first reconstructed image; inputting the second image into the corrector to generate a second reconstructed image; adjusting the setting parameters so that a cost calculated from a predetermined loss becomes smaller; The loss is a difference between the first reconstructed image and the second image; a difference between the second reconstructed image and the second image; a difference between the same type of corrected feature quantity and a feature quantity obtained by correcting each of the plurality of types of feature quantities extracted from the first image and a feature quantity obtained by correcting each of the plurality of types of feature quantities extracted from the second image.

8. 6. The image processing device according to claim 5, The one or more processors display the loss, the first reconstructed image, and the second reconstructed image.

9. An image processing device, one or more processors; one or more memories; the one or more memories store a corrector that generates a reconstructed image from an input image according to setting parameters; The one or more processors perform the adjustment of the configuration parameters, wherein the adjustment comprises: perturbing the second image to generate the first image; inputting the first image into the corrector to generate the first reconstructed image; inputting the second image into the corrector to generate a second reconstructed image; adjusting the setting parameters so that a cost calculated from a predetermined loss becomes smaller; The image processing device, wherein the loss includes a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

10. 10. The image processing device according to claim 9, The image processing device, wherein the loss includes a difference between a feature amount obtained by correcting a feature amount extracted from the first image and a feature amount obtained by correcting a feature amount extracted from the second image.

11. 1. An image identification system, comprising: an image capture device including an imaging device that captures an image of an object and generates an image; The image processing device according to claim 4, the image acquisition device transmits the target image to the image processing device; the image processing device transmits the reconstructed image of the target image and the identification result to the image acquisition device; The image acquisition device displays the received reconstructed image and the classification result on a display device.

12. 1. An image identification system, comprising: an image capture device that captures image data; a first image processing device that is the image processing device according to claim 9; The image acquisition device includes the image processing device according to claim 4, the image acquisition device transmits the image data to the first image processing device; the first image processing device generates the corrector using the received image data, stores the corrector in a memory, and transmits the corrector to the image acquisition device; The image acquisition device stores the received corrector in a memory; An image identification system in which the image processing device within the image acquisition device uses the corrector to generate a reconstructed image of the target image and calculate the identification result, and displays the target image, the reconstructed image of the target image, and the identification result on a display device.

13. An image processing method by an apparatus, comprising: The apparatus stores a corrector for generating a reconstructed image from an input image according to set parameters; the setting parameters are adjusted so that a first reconstructed image generated by the corrector from the input first image and a second image of the same object but different color tone from the first image are closer to each other; The image processing method includes: the apparatus inputs an image of the object to the corrector to generate a reconstructed image of the object image; The generation of the reconstructed image of the target image by the corrector is performed in accordance with the setting parameters, Extracting features from the input target image; correcting the extracted feature amount; generating the reconstructed image of the target image from the corrected feature amount;

14. 14. The image processing method according to claim 13, The image processing method, wherein the device calculates an identification result of the reconstructed image of the input image from feature amounts extracted from the reconstructed image of the input image.

15. An image processing method by an apparatus, comprising: The apparatus stores a corrector for generating a reconstructed image from an input image according to set parameters; The image processing method includes: the apparatus inputs an image of the object to the corrector to generate a reconstructed image of the object image; The generation of the reconstructed image of the target image by the corrector is performed in accordance with the setting parameters, perturbing the second image to generate the first image; inputting the first image into the corrector to generate a first reconstructed image; inputting the second image into the corrector to generate a second reconstructed image; adjusting the setting parameters so that a cost calculated from a predetermined loss becomes smaller; An image processing method, wherein the loss includes a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

Images