Image processing device, image identification system, and image processing method

Abstract

This apparatus stores a corrector for generating a reconstructed image from an input image in accordance with a setting parameter. The setting parameter is adjusted so that a first reconstructed image generated by the corrector from an inputted first image and a second image having the same subject as the first image but a different color tone from the first image approach each other. The device inputs a subject image to the corrector to generate a reconstructed image of the subject image. In generation of the reconstructed image of the subject image by the corrector, a feature amount is extracted from the inputted subject image in accordance with the setting parameter, the extracted feature amount is corrected, and a reconstructed image of the subject image is generated from the corrected feature amount.

Description

Image processing device, image identification system, and image processing method Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2023-130362, filed on August 9, 2023, the contents of which are incorporated herein by reference.

[0002] The present invention relates to an image processing technique, and more particularly to an image processing technique for suppressing differences in color and brightness of a captured image and accurately detecting a specific object contained in the image.

[0003] In recent years, image recognition techniques using machine learning and the like have been studied. Techniques for equalizing the color and brightness of an image have also been studied. For example, Patent Document 1 proposes a technique for correcting color unevenness in an image. Patent Document 1 uses a color averaging filter as a parameter used in image processing to reduce color unevenness in an image representing a biological material.

[0004] International Publication No. 2017 / 150194

[0005] However, when color unevenness in an image is reduced using a color averaging filter as in Patent Document 1, color unevenness may remain or color artifacts may occur in the corrected image, resulting in a problem that color unevenness in the image is not necessarily reduced. Furthermore, Patent Document 1 also has a problem that when an object in an image after color unevenness correction is identified by a classifier, the classification accuracy of the classifier decreases.

[0006] An image processing system according to one aspect of the present invention includes one or more processors and one or more memories, wherein the one or more memories store a corrector that generates a reconstructed image from an input image according to setting parameters, and the setting parameters are adjusted so that a first reconstructed image generated by the corrector from an input first image approaches a second image of the same subject but with a different color tone from the first image, and the processor inputs a target image to the corrector to generate a reconstructed image of the target image, and the corrector extracts features from the input target image according to the setting parameters, corrects the extracted features, and generates the reconstructed image of the target image from the corrected features.

[0007] A learning system according to one aspect of the present invention includes one or more processors and one or more memories, wherein the one or more memories store a corrector that generates a reconstructed image from an input image according to setting parameters, and the one or more processors perform the adjustment of the setting parameters, and in the adjustment, perturb a second image to generate a first image, input the first image to the corrector to generate the first reconstructed image, input the second image to the corrector to generate a second reconstructed image, and adjust the setting parameters so that a cost calculated from a predetermined loss is reduced, wherein the loss includes a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

[0008] Further features related to the present invention will become apparent from the description and accompanying drawings of this specification, and aspects of the present invention may be realized and realized by the elements and combinations of various elements and aspects set forth in the following detailed description and the appended claims.

[0009] It should be understood that the descriptions in this specification are exemplary and illustrative only and are not intended to limit the scope or application of the present invention in any way.

[0010] According to the present invention, even when there are differences in color and brightness between images, a reconstructed image is created that suppresses the differences in color and brightness between the images, and further, machine learning is performed to automatically generate a color and brightness corrector that generates a reconstructed image in which the differences in color and brightness of the target image are suppressed. This makes it possible to identify objects (e.g., abnormal tissue) in an image with high accuracy without changing the classifier.

[0011] 1 is a block diagram showing functions of an image processing device during learning according to the first embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of the image processing device according to the first embodiment or the image processing device according to the second embodiment. FIG. 3 is a block diagram showing a color and brightness corrector during learning according to the first embodiment. FIG. 4 is a block diagram showing functions of an image processing device during evaluation according to the first embodiment. FIG. 5 is a diagram for explaining an example of the operation of a color and brightness feature extraction unit. FIG. 6 is a diagram for explaining an example of the operation of the color and brightness feature extraction unit. FIG. 7 is a diagram for explaining an example of the operation of the color and brightness feature correction unit. FIG. 8 is a diagram for explaining an example of the operation of the color and brightness feature correction unit. FIG. 9 is a diagram for explaining an example of the operation of an image reconstruction unit. FIG. 10 is a block diagram showing a color and brightness corrector during evaluation according to the first embodiment. FIG. 11 is a diagram for explaining an example of the operation of a feature extraction unit. FIG. 12 is a diagram for explaining an example of the operation of a classification unit. FIG. 13 is a diagram for explaining an example of the operation of the classification unit. FIG. 14 is a diagram for explaining an example of the operation of an output unit. FIG. 15 is a diagram for explaining an example of the operation of the output unit. FIG. 16 is a flowchart for explaining the overall operation of the image processing device according to the first embodiment during learning. FIG. 17 is a flowchart for explaining the overall operation of the image processing device according to the first embodiment during evaluation. FIG. 1 is a block diagram showing functions of an image processing device during learning according to a second embodiment. FIG. 2 is a block diagram showing a color and brightness corrector during learning according to a second embodiment. FIG. 3 is a block diagram showing a color and brightness corrector during learning according to a second embodiment. FIG. 4 is a diagram showing an example of operation of a color and brightness feature extraction and correction unit. FIG. 5 is a diagram showing an example of operation of the color and brightness feature extraction and correction unit. FIG. 6 is a diagram showing an example of operation of the color and brightness feature extraction and correction unit. FIG. 7 is a flowchart showing the overall operation during learning of an image processing device 2 according to a second embodiment. FIG. 8 is a block diagram showing functions of an image processing device during evaluation according to the second embodiment. FIG. 9 is a flowchart showing the overall operation during evaluation of an image processing device according to a second embodiment.1 is a diagram illustrating a schematic configuration of an image processing system equipped with an image processing device.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show specific embodiments and implementation examples in accordance with the principles of the present invention, but these are for understanding the present invention and are by no means to be used to interpret the present invention in a limiting manner.

[0013] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to practice the present invention, it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical concept of the present invention. Therefore, the following description should not be interpreted as being limited thereto.

[0014] Additionally, embodiments of the present invention may be implemented in software running on a general purpose computer, in specialized hardware, or a combination of software and hardware, as described below.

[0015] In the following, each process in an embodiment of the present invention will be explained using "each processing unit as a program (e.g., an identification unit, etc.)" as the subject (acting entity); however, since a program is executed by a processor (e.g., a CPU) to perform the specified process using memory and a communication port (communication control device), the explanation may also use the processor as the subject.

[0016] In one embodiment of the present specification, a color and brightness corrector that suppresses differences in color and brightness between images is created during training, and during evaluation, the color and brightness corrector is used to create a reconstructed image in which the color and brightness differences in the target image are suppressed, and objects in the reconstructed image are detected. This enables highly accurate identification of objects (e.g., abnormal tissue) in the image without changing the classifier.

[0017] According to an embodiment of the present specification, during learning, a perturbed image is automatically created from a target image while updating parameters of a color / brightness corrector composed of a color / brightness feature extraction unit, a color / brightness feature correction unit, and an image reconstruction unit. When evaluating an image, the color / brightness corrector created during learning is used to create a reconstructed image in which color / brightness differences in the evaluation image are suppressed, and feature amounts of the reconstructed image are extracted for identification, thereby providing an image processing method that achieves high-accuracy identification of objects in an image.

[0018] An image processing device according to an embodiment of the present specification performs the following processes during machine learning: extracting features of color and brightness differences between images, correcting the features of color and brightness differences, creating a reconstructed image from the corrected features, and generating a color and brightness corrector that performs machine learning to automatically generate a reconstructed image from a target image in which color and brightness differences have been suppressed. During evaluation, the device performs the following processes: automatically generating a reconstructed image from the target image in which color and brightness differences have been suppressed using the color and brightness corrector created during machine learning, extracting features of the reconstructed image, and identifying objects in the reconstructed image from the calculated features of the reconstructed image.

[0019] (1) First Embodiment <Functional Configuration of Classifier Generating Device> FIG. 1 is a block diagram showing the functional configuration of an image processing device during machine learning according to an embodiment of the present invention. The image processing device 1 includes an input unit 10, a perturbed image creating unit 11, a color / brightness feature extracting unit 12, a color / brightness feature correcting unit 13, an image reconstructing unit 14, an output unit 15, a control unit 91, and a memory 90. The image processing device 1 may be implemented within an image acquisition device, or, as described later (third and fourth embodiments), may be implemented within a server connected to the image acquisition device via a network. The image processing device 1 may be implemented on one or more computers, or may be implemented in a cloud. In this way, the image processing device 1 may be implemented within a system including one or more processors and one or more memories (storage devices).

[0020] In the image processing device 1, the perturbed image creation unit 11, the color / brightness feature extraction unit 12, the color / brightness feature correction unit 13, the image reconstruction unit 14, and the output unit 15 may be realized by one or more processors that execute a program, or may be realized by modularized hardware.

[0021] Image data is input to the input unit 10. For example, the input unit 10 may acquire encoded still image data, WSI, or the like, captured at predetermined time intervals by an imaging unit such as a camera built into the image processing device 1, in a format such as JPG, JPEG2000, PNG, or BMP, and use the acquired images as input images. Alternatively, the input unit 10 may extract still image data of frames at predetermined intervals from moving image data in a format such as MotionJPEG, MPEG, H.264, or HD / SDI, and use the extracted images as input images.

[0022] The input unit 10 may also use an image acquired by an imaging means via a bus, a network, etc. as an input image. The input unit 10 may also use an image already stored in a removable recording medium as an input image. The image input from the input unit 10 is output to the perturbed image creation unit 11 and the color / brightness feature extraction unit 12.

[0023] The perturbed image creation unit 11 creates a perturbed image by perturbing the values ​​of an input image. The perturbation method involves perturbing the gamma value, R value, B value, G value, etc. To prevent artifacts from occurring, a range for perturbing each value is set in advance, and values ​​are randomly determined within that range to create multiple perturbed images. Machine learning uses one or more input images and multiple perturbed images generated from each input image.

[0024] The color / brightness feature extraction unit 12 calculates color / brightness feature amounts (values ​​of each parameter) and output values ​​of each layer of the network (e.g., Convolutional Neural Network) that constitutes the color / brightness feature extraction unit 12 from the input image and the perturbed image.

[0025] FIG. 3 is an example block diagram showing the functional configuration of a color / brightness corrector during machine learning. The color / brightness feature extraction unit 12 corresponds to the encoders 311 and 312 in FIG. 3 . The encoder 311 extracts a global feature G1 from the input image 301. The encoder 312 extracts a global feature G2 from the perturbed image 302. The encoder 312 extracts a local feature L1 from the input image 301. The encoder 312 extracts a local feature L2 from the perturbed image 302. The global feature is a feature representing the color and brightness of the entire image, for example, a 1x1x several hundred-dimensional feature. The size of the global feature is smaller than the size of the image. On the other hand, the local feature is a feature representing the shape within the image and is the same size as the image. In this example, feature values ​​of two sizes are extracted, but feature values ​​of three or more different sizes may be extracted. Extracting feature values ​​of different sizes enables more appropriate color / brightness correction of the image. Note that the number of feature types may be one.

[0026] The color / brightness feature correction unit 13 is composed of an adjustment unit 321, an adjustment unit 322, a normalization unit 331, and a normalization unit 332. The adjustment unit 321 corrects the global feature G1 and the global feature G2 so that the corrected global feature G1 and the global feature G2 approach each other, thereby obtaining a global feature G11 and a global feature G21, respectively. The adjustment unit 322 corrects the local feature L1 and the local feature L2 so that the local feature L1 and the local feature L2 approach each other, thereby obtaining a local feature L11 and a local feature L21, respectively. The normalization unit 331 normalizes the global feature G11 using the local feature L11 to calculate a normalized feature. The normalization unit 332 normalizes the global feature G21 using the local feature L21 to calculate a normalized feature.

[0027] The image reconstruction unit 14 corresponds to the decoder 340 in Fig. 3 and generates a reconstructed image 351 from the normalized feature amounts obtained by the color / brightness feature correction unit 13. It also generates a reconstructed image 352 from the normalized feature amounts obtained by the color / brightness feature correction unit 13. When training the color / brightness corrector using machine learning, for example, loss values ​​are calculated and training is performed so that the input image 301 and the reconstructed image 351, the input image 301 and the reconstructed image 352, the global feature amount G11 and the global feature amount G21, and the local feature amount L11 and the local feature amount L21 approach each other, and encoders 311 and 312, adjustment units 321 and 322, normalization units 331 and 332, and decoder 340 are created.

[0028] The output unit 15 displays the reconstructed image and the loss values ​​and cost values ​​output by the created color / brightness corrector on an output device (such as a display), and stores information such as the reconstructed image, the color / brightness corrector, and values ​​calculated by each unit from the color / brightness feature extraction unit 12 to the image reconstruction unit 14 in memory 90.

[0029] The control unit 91 is realized by a processor and is connected to each element in the image processing device 1. The operation of each element of the image processing device 1 is performed by the autonomous operation of each of the above-mentioned components or by instructions from the control unit 91.

[0030] FIG. 4 is a block diagram showing the functional configuration of an image processing device 1 during evaluation according to an embodiment of the present specification. The image processing device 1 includes an input unit 10, a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, an image reconstruction unit 14, a feature extraction unit 16, a classification unit 17, an output unit 15, a control unit 91, and a memory 90. The image processing device 1 may be implemented within an image processing device within an image acquisition device, or, as described later (third and fourth embodiments), may be implemented within a server connected to the image acquisition device via a network. The image processing device 1 may be implemented on one or more computers, or may be implemented in a cloud. In this manner, the image processing device 1 may be implemented within a system including one or more processors and one or more memories (storage devices).

[0031] In the image processing device 1, the color / brightness feature extraction unit 12, color / brightness feature correction unit 13, image reconstruction unit 14, output unit 15, feature extraction unit 16, and identification unit 17 may be realized by a processor that executes a program, or may be realized by modularized hardware.

[0032] The following describes each processing unit that differs from that in Fig. 1. The parameter values ​​of the color and brightness corrector created during learning are read from memory 90, and the parameters of color and brightness feature extraction unit 12, color and brightness feature correction unit 13, and image reconstruction unit 14 are set.

[0033] For an image input from an input unit 10, a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, and an image reconstruction unit 14 are used to generate a reconstructed image in which color / brightness differences are suppressed.

[0034] Machine learning is performed in advance to identify objects in the input image as objects, for example, to identify normal tissues and cells as normal tissues and cells, and to identify abnormal tissues and cells in the input image as abnormal tissues and cells, and a classifier (each parameter (filter coefficient, offset value, etc.) of the feature extraction unit 16 and the classifier 17 required for identification) is created from the training image.

[0035] The values ​​of each parameter (filter coefficient, offset value, etc.) of a classifier network (e.g., a convolutional neural network, etc.) consisting of a feature extraction unit 16 and a classification unit 17 created in advance through machine learning are read from the memory 90, and each parameter of the feature extraction unit 16 and the classification unit 17 is set.

[0036] The feature extraction unit 16 calculates feature amounts from the input image and output values ​​of each layer of a network (e.g., a convolutional neural network) that constitutes the feature extraction unit. The classification unit 17 receives the output value of the final layer of the network of the feature extraction unit 16 and calculates the classification result of the classifier. The output unit 15 displays the classification result on an output device (a display, etc.) and stores the classification result in memory 90. The control unit 91 is realized by a processor and is connected to each element in the image processing device 1. The operation of each element of the image processing device 1 is performed by the autonomous operation of each of the above-mentioned components or by instructions from the control unit 91.

[0037] As described above, the image processing device 1 of this embodiment creates a perturbed image using the training image during machine learning, while updating the parameters of the color / brightness corrector consisting of the color / brightness feature extraction unit 12, color / brightness feature correction unit 13, and image reconstruction unit 14, to create a color / brightness corrector that suppresses differences in color / brightness of the image. Furthermore, during evaluation, the parameters of the color / brightness corrector and classifier created in advance are read, the color / brightness corrector is used to create a reconstructed image in which differences in color / brightness of the evaluation image are suppressed, and the classifier is used to classify the reconstructed image, thereby making it possible to identify objects (e.g., abnormal tissue) in the image with high accuracy.

[0038] 2 is a diagram showing an example of the hardware configuration of an image processing device 1 according to an embodiment of the present invention. The image processing device 1 includes a CPU (processor) 201 that executes various programs, a memory 202 that stores various programs, a storage device (corresponding to the memory 90) 203 that stores various data, an output device 204 that outputs reconstructed images, color and brightness correctors, classification results, loss values ​​during machine learning, etc., an input device 205 that inputs user instructions, images, etc., and a communication device 206 that communicates with other devices, all of which are connected to each other by a bus 207.

[0039] The CPU 201 reads and executes various programs from the memory 202 as necessary. The memory 202 stores, as programs, an input unit 10, a perturbed image creation unit 11, a color / brightness feature extraction unit 12, a color / brightness feature correction unit 13, an image reconstruction unit 14, a feature extraction unit 16, a recognition unit 17, and an output unit 15.

[0040] The auxiliary storage device 203 stores the training image, the evaluation image, the parameters and reconstructed images of the color and brightness corrector generated by the color and brightness feature extraction unit 12, the color and brightness feature correction unit 13, and the image reconstruction unit 14, the parameters (filter coefficients, offset values, etc.) of the classifier generated by the feature extraction unit 16 and the classification unit 17, the classification results and classification values, and the display information generated by the output unit 15.

[0041] The output device 204 is composed of devices such as a display, a printer, and a speaker. For example, the output device 204 displays data generated by the output unit 15 on a display screen. The input device 205 is composed of devices such as a keyboard, a mouse, and a microphone. Instructions from the user (including the determination of input of each image) are input to the image processing device 1 via the input device 205.

[0042] The communication device 206 is not an essential component of the image processing device 1, and if a personal computer or the like connected to the image acquisition device includes a communication device, the image processing device 1 does not need to have the communication device 206. The communication device 206 receives data (including images, color and brightness correctors, etc.) transmitted from another device (e.g., a server) connected via a network, and stores the data in the storage device 203.

[0043] The image processing device of this embodiment uses perturbed images during learning to extract and correct color and brightness features, and creates a color and brightness corrector that generates a reconstructed image in which color and brightness differences are suppressed. Furthermore, during evaluation, the color and brightness corrector is used to generate a reconstructed image from the evaluation image, and a classifier is used to classify the reconstructed image, thereby enabling high-accuracy identification of objects (e.g., abnormal tissue) in the image.

[0044] <Configuration and Operation of Each Unit> The configuration and operation of each element during machine learning will be described in detail below. (i) Perturbed Image Creation Unit 11 This unit creates a perturbed image by perturbing the values ​​of an input image. As a perturbation method, the gamma value, R value, B value, G value (gradation level), etc. are perturbed to create a perturbed image. For example, a perturbed image is created by multiplying the gamma value by 1.4, adding 3 to the R value for the entire image, and subtracting 5 from the B value for the entire image. To prevent artifacts from occurring, a range for perturbing each value (for example, a range from 0.5 to 1.5 for the gamma value) is set in advance, and a value is randomly determined within that range to create a perturbed image.

[0045] In this way, by changing specific values ​​in the input image, it is possible to generate an image of the same subject as the input image but with a different color tone. The color tone of some or all areas of the generated image differs from that of the input image. Perturbation allows for efficient generation of many different variations of images. Images with different color tones are images with different colors and brightness. Images with a different color tone from the input image may be generated by a method other than a perturbed image, which is an image that is artificially perturbed or shifted by a small amount. For example, they can be generated by changing the dye solution or illumination light.

[0046] (ii) Color / Brightness Feature Extraction Unit 12: This unit extracts color / brightness features of an input image. As an example, an example of extracting color / brightness features (global features and local features) is shown in FIGS. 5A and 5B. CNN in FIGS. 5A and 5B stands for Convolutional Neural Network.

[0047] For example, as shown in Fig. 5A, the feature extractor G5 calculates a global feature FGi of color and brightness of the input image 301 or the perturbed image 302 from the input image 301 or the perturbed image 302 using the following formula (1). Also, as shown in Fig. 5B, the feature extractor L5 calculates a local feature FLi of color and brightness of the input image 301 or the perturbed image 302 from the input image 301 or the perturbed image 302 using the following formula (2).

[0048] The filter coefficient wgj shown in equation (1) is a coefficient calculated by machine learning or the like, for example, so that the reconstructed image 351 reconstructed from the input image 301 approaches the input image 301, and so that the reconstructed image 352 reconstructed from the perturbed image 302 approaches the input image 301.

[0049] In equation (1), pgj represents a pixel value, bgi represents an offset value, q represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 5A , equation (1) is used to calculate the calculation results of each filter from the upper left to the lower right of the target image to calculate the feature value fgi of an arbitrary filter i. For example, the matrix of the feature values ​​fgi calculated by the feature extractor G5 is set as the global feature value FGi of the input image 301 or the perturbed image 302.

[0050]

[0051] The filter coefficient wlj shown in equation (2) is a coefficient obtained by machine learning or the like so that the reconstructed image 351 reconstructed from the input image 301 approaches the input image 301, and so that the reconstructed image 352 reconstructed from the perturbed image 302 approaches the input image 301.

[0052] In equation (2), p, j represents a pixel value, b, i represents an offset value, r represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 5B , equation (2) is used to calculate the calculation results of each filter from the upper left to the lower right of the target image, thereby calculating the feature value f for an arbitrary filter i. For example, the matrix of the feature value f obtained by the feature extractor L5 is set as the local feature value F of the input image 301 or the perturbed image 302.

[0053]

[0054] (iii) Color / Brightness Feature Correction Unit 13 The color / brightness feature correction unit 13 is composed of adjustment units 321 and 322 and normalization units 331 and 332. The adjustment unit 321 obtains the filter coefficient wdj shown in Equation (3) by machine learning or the like so that the corrected feature amount of the global feature amount FGi (global feature amount G1) of the input image 301 and the corrected feature amount of the global feature amount FGi (global feature amount G2) of the perturbed image 302 approach each other.

[0055] In equation (3), pdj represents the value of the global feature, bdi represents the offset value, s1 represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 6A , equation (3) is used to calculate the calculation results of each filter for global features G1 and G2, thereby calculating the feature fdi of an arbitrary filter i. For example, the matrix of feature fdi calculated from global feature G1 by feature extractor D1 is defined as global feature G11, and the matrix of feature fdi calculated from global feature G2 by feature extractor D1 is defined as global feature G21. Alternatively, the matrix obtained by adding together global feature G1 and correction feature fdi calculated from global feature G1 by feature extractor D1 is defined as global feature G11, and the matrix obtained by adding together global feature G2 and correction feature fdi calculated from global feature G2 by feature extractor D1 is defined as global feature G21.

[0056]

[0057] The adjustment unit 322 obtains the filter coefficient wej shown in Equation (4) by machine learning or the like so that the corrected feature amount of the local feature amount FLi (local feature amount L1) of the input image 301 and the corrected feature amount of the local feature amount FLi (local feature amount L2) of the perturbed image 302 become closer to each other.

[0058] In equation (4), p e j represents the value of the local feature, b e i represents the offset value, s2 represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 6B , equation (4) is used to calculate the calculation results of each filter for local features L1 and L2, thereby calculating the feature fe i of an arbitrary filter i. For example, the matrix of feature fe i calculated from local feature L1 by feature extractor D2 is defined as local feature L11, and the matrix of feature fe i calculated from local feature L2 by feature extractor D2 is defined as local feature L21. Alternatively, the matrix obtained by adding the local feature L1 and the corrected feature fe i calculated from local feature L1 by feature extractor D2 is defined as local feature L11, and the matrix obtained by adding the local feature L2 and the feature fe i calculated from local feature L2 by feature extractor D2 is defined as local feature L21.

[0059]

[0060] The normalization unit 331 normalizes the global feature G11 as shown in equation (5). In equation (5), FG represents the global feature G11, FGmax represents the maximum value of the global feature G11, and FLmax represents the maximum value of the local feature L11. Therefore, equation (5) is used to correct the global feature G11 to the global feature G111.

[0061]

[0062] The normalization unit 332 normalizes the global feature G21 as shown in equation (5). In equation (5), FG represents the global feature G21, FGmax represents the maximum value of the global feature G21, and FLmax represents the maximum value of the local feature L21. Therefore, equation (5) is used to correct the global feature G21 to the global feature G211.

[0063] The global feature G111 and the local feature L11 are combined into a set to form a normalized feature NF1, and the global feature G211 and the local feature L21 are combined into a set to form a normalized feature NF2.

[0064] (iv) Image Reconstruction Unit 14 The image reconstruction unit 14 uses the decoder 340 to create a reconstructed image 351 from the normalized feature NF1 and a reconstructed image 352 from the normalized feature NF2. As shown in Fig. 7 and equation (6), x, which is the global feature G111, is copied from the top left to the bottom right so that the vertical and horizontal sizes become the same as those of the local feature L11 to create a global feature G1111, and the global feature G1111 and the local feature L11 are concatenated to create a corrected color / brightness feature CF1.

[0065] Furthermore, as shown in FIG. 7 and equation (6), x, which is the global feature G211, is copied from the top left to the bottom right so that the size becomes the same as the vertical length Hy and horizontal length Wy of the local feature L21, thereby creating a global feature G2111, and the global feature G2111 and the local feature L21 are concatenated to create a corrected color / brightness feature CF2.

[0066]

[0067] The decoder 340 calculates the filter coefficients wpj shown in Equation (7) by machine learning or the like so that the input image 301 and the reconstructed image 351 reconstructed from the corrected color / brightness feature CF1 are closer to each other. In Equation (7), ppj represents the value of the corrected color / brightness feature CF1, bpi represents the offset value, t1 represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 8 , Equation (7) is used to calculate the calculation results of each filter for the corrected color / brightness feature CF1, thereby calculating the feature fpi of an arbitrary filter i. For example, the reconstructed image 351 is created from a matrix of the feature fpi calculated by the feature extractor U1 from the corrected color / brightness feature CF1. For example, the reconstructed image 351 may be created by multiplying the feature fpi by 255.

[0068]

[0069] The decoder 340 also uses machine learning or other methods to determine the filter coefficients wpj shown in equation (7) so that the input image and the reconstructed image 2 reconstructed from the corrected color and brightness feature CF2 are closer to each other. In equation (7), ppj represents the value of the corrected color and brightness feature CF2, bpi represents the offset value, t1 represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 8 , equation (7) is used to determine the calculation results of each filter for the corrected color and brightness feature CF2, thereby determining the feature fpi of an arbitrary filter i. For example, the reconstructed image 2 is created from the matrix of feature fpi calculated by the feature extractor U1 from the corrected color and brightness feature CF2. For example, the reconstructed image 352 may be created by multiplying the feature fpi by 255.

[0070] When creating a color / brightness corrector using machine learning, for example, the parameters (filter coefficients, offset values, etc.) of equations (1) to (7) are calculated so as to reduce the overall cost shown in equation (12), which combines the four loss values ​​shown in equations (8) to (11). MSE indicates mean square error, Input indicates the input image 301, Output1 indicates the reconstructed image 351, Output2 indicates the reconstructed image 352, Feature G11 indicates the global feature G11, Feature G21 indicates the global feature G21, Feature L11 indicates the local feature L11, and Feature L21 indicates the local feature L21.

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] (v) Output Unit 15 When creating a color / brightness corrector during machine learning, the output unit 15 displays, as an example, an input image, a perturbed image, reconstructed images 1 and 2 of the input image and the perturbed image, each loss value, and a cost value in a GUI (graphical user interface) shown in FIG. 14 .

[0077] The example in FIG. 14 displays the input image, the perturbed image, reconstructed image 1, reconstructed image 2, four loss values ​​(0.09, 0.08, 0.06, 0.03), and a cost value of 0.26.

[0078] The output unit 15 stores in memory 90 the input image (learning image, etc.), the parameters (filter coefficients, weights, offset values, etc.) of the color / brightness corrector consisting of the color / brightness feature extraction unit 12, the color / brightness feature correction unit 13, and the image reconstruction unit 14, the reconstructed images 1 and 2, the display information generated by the output unit 15, etc.

[0079] <Configuration and Operation of Each Unit> The configuration and operation of each element during evaluation will be described in detail below. (i) Input Unit 10 Outputs an input image (target image) 301 for evaluation to the color and brightness feature extraction unit 12.

[0080] (ii) Color / Brightness Feature Extraction Unit 12 The color / brightness feature extraction unit 12 of the color / brightness corrector shown in FIG. 9, which is created during machine learning, is used to calculate the global feature FGi of the input image 301 and the local feature FLi of the input image.

[0081] (iii) Color / Brightness Feature Correction Unit 13 Using the color / brightness feature correction unit 13 of the color / brightness corrector shown in FIG. 9 created during machine learning, the adjustment unit 321 calculates a global feature G11 from the global feature G1. The adjustment unit 322 calculates a local feature L11 from the local feature L1. The normalization unit 331 corrects the global feature G11 to a global feature G111. The global feature G111 and the local feature L11 are combined into a set to form a normalized feature NF1.

[0082] (iv) Image Reconstruction Unit 14 A reconstructed image 351 is created from the normalized feature NF1 using the decoder 340 of the image reconstruction unit 14 of the color and brightness corrector shown in FIG. 9, which is created during machine learning.

[0083] (v) Feature Extraction Unit 16: The feature extraction unit 16 obtains feature amounts of the reconstructed image 351. An example of obtaining feature amounts is shown in Fig. 10. CNN in Fig. 10 stands for Convolutional Neural Network.

[0084] For example, the feature extractor A1 is used to obtain the feature amount FAi of the object (for example, tissue, cell, etc.) of the reconstructed image 351 from the reconstructed image 351 according to equation (13).

[0085] The filter coefficient wj shown in equation (13) is a coefficient obtained by machine learning or the like so as to distinguish objects other than those to be detected from those to be detected (for example, normal tissue or normal cells from normal tissue or normal cells, etc.), and to distinguish objects to be detected from objects to be detected (for example, abnormal tissue or abnormal cells from abnormal tissue or abnormal cells, etc.).

[0086] In equation (13), pj represents a pixel value of the reconstructed image 1, bi represents an offset value, m represents the number of filter coefficients, and h represents a nonlinear function. As shown in Fig. 11, equation (13) is used to calculate the calculation results of each filter 401 for an input image 301, which is a target image (e.g., a pathological tissue image), from the upper left to the lower right, to calculate a feature value fi of an arbitrary filter i. For example, the matrix of feature values ​​fi calculated by feature extractor A1 is set as feature value FAi of input image 301. A method for creating feature extractor A1 will be described in the identification unit 17, which will be described later.

[0087]

[0088] (vi) Classification Unit 17 As shown in FIG. 12 , the classification unit 17 uses the feature quantities FAi (matrix f) of the feature extractor A1 obtained by the feature extraction unit 16 to calculate a value of likelihood of an object to be detected (likelihood of abnormal tissue, likelihood of a lesion, etc.) according to equation (14) through logistic regression processing in the logistic regression layer 501, and determines whether an object (e.g., tissue or cell) in the input image 301 is an object to be detected (such as abnormal tissue) or an object other than an object to be detected (such as normal tissue). In equation (14), w represents a weight matrix, b represents an offset value, g represents a nonlinear function, and y represents a classification result. The weight of w and the offset value of b are calculated in advance using training images through machine learning. The target image is classified using the weight of w and the offset value of b calculated during training, and the classification result y is calculated.

[0089]

[0090] The identification unit 17 learns the features of the object using the feature extraction unit 16, for example, by known machine learning techniques, so that if an object in the input reconstructed image 351 is other than the object to be detected, it determines that the object is other than the object to be detected by logistic regression processing in the logistic regression layer 501, for example, according to equation (14).

[0091] The reconstructed image 351 is, for example, an HE stained image, and the objects to be detected are, for example, abnormal tissues and abnormal cells, and the objects other than the objects to be detected are, for example, normal tissues and normal cells.

[0092] Furthermore, if an object in an input image is an object to be detected, the feature extraction unit 16 is used to learn the feature quantities of the object so that the object is determined to be an object to be detected by logistic regression processing. For example, a convolutional neural network may be used as a machine learning technique.

[0093] 13, during learning, the identification unit 17 uses machine learning to create a feature extractor A1 that calculates a feature amount fi (referred to as FAi) of the reconstructed image 351 using the reconstructed image 351 to determine an object to be detected as an object to be detected and to determine an object other than the object to be detected as an object other than the object to be detected. The feature extractor A1 is created using the feature extraction unit 16 according to equations (13) and (14).

[0094] During learning, the classification unit 17 repeatedly performs the feature extraction by the feature extraction unit 16 and the classification unit 17 using a plurality of learning images to determine the weight w, the filter coefficient wj, and the offset values ​​b and bi shown in equations (13) and (14), and creates a feature extractor A1 that calculates the feature amount FAi of the reconstructed image 351 from the reconstructed image 351 using the feature extraction unit 16. The classification unit 17 also calculates the classification result of the reconstructed image 351 using the feature extractor A1.

[0095] The discrimination unit 17 stores the weight w, filter coefficient wj, and offset values ​​b and bi thus determined in the memory 90.

[0096] (vii) Output Unit 15 The output unit 15 uses the classification result y obtained by the classifier to display the result of the object-likeness determination (e.g., lesion-likeness determination) on a GUI (graphical user interface) shown in, for example, FIG. 16A or FIG. 16B .

[0097] 16A is an example of the case of a breast, showing the classification results of non-tumor and tumor (abnormal tissue, abnormal cells, etc.). In the example of Fig. 16A, the identification unit 17 classifies the input reconstructed image 1 of the breast as including a tumor, which is an abnormal tissue / cell, and calculates the object-likeness value of the tumor to be 0.89.

[0098] Fig. 16B is a diagram showing an example of a display when the image button is pressed in Fig. 16A, showing an example of a display of an input image, a reconstructed image obtained by reconstructing the input image using a color and brightness corrector, and an identification score.

[0099] Furthermore, when an input image (target image) 301 is input from the input unit 10 and the output unit 15 displays the classification results of each classifier, if an object in the image is determined to be an object (e.g., abnormal tissue or abnormal cells), the output unit 15 may draw a detection frame 521 within the input target image 301 to indicate the location of the object to be detected (e.g., a location suspected of being abnormal tissue or abnormal cells), as shown in Fig. 15. On the other hand, if the object is determined to be normal tissue or normal cells, the output unit 15 may display the input target image as is without drawing the detection frame 521 on the input target image. Furthermore, as shown in Fig. 15, the output unit 15 displays the determined object-likeness result 522 (e.g., a tumor).

[0100] The output unit 15 stores in memory 90 the input image, the parameters (filter coefficients, weights, offset values, etc.) of the color / brightness corrector consisting of the color / brightness feature extraction unit 12, the color / brightness feature correction unit 13, and the image reconstruction unit 14, the reconstructed image, the parameters (filter coefficients, weights, offset values, etc.) of the classifier consisting of the feature extraction unit 16 and the classification unit 17, the classification results, the classification values, display information generated by the output unit 15, etc.

[0101] 17 is a flowchart for explaining the operation of the image processing device 1 during machine learning according to the embodiment. In the following, each processing unit (input unit 10, color / brightness feature extraction unit 12, etc.) will be described as the main operating unit, but it may also be interpreted as the CPU 201 being the main operating unit, and the CPU 201 executing each processing unit as a program.

[0102] (i) Step 1701 The input unit 10 receives a learning image and outputs the input image to the perturbed image creation unit 11 and the color / brightness feature extraction unit 12.

[0103] (ii) Step 1702 The perturbed image generating unit 11 generates a perturbed image by randomly perturbing each value (R value, G value, B value, etc.) of the image, the γ value, etc.

[0104] (iii) Step 1703: The color / brightness feature extraction unit 12 uses machine learning to find global features FGi of multiple learning input images 301 using a filter according to the above-mentioned formula (1), and creates a feature extractor G5 by repeatedly updating the parameters of the feature extractor. For the feature FGi, the filter coefficients wgj and offset values ​​bgi are found. Therefore, a global feature G1 is found from the input image 301, and a global feature G2 is found from the perturbed image 302.

[0105] Furthermore, the color / brightness feature extraction unit 12 uses machine learning to obtain a local feature FLi of the input image 301 using a filter according to the above-mentioned equation (2), and creates a feature extractor L5. For the feature FLi, a filter coefficient wlj and an offset value bli are obtained. Therefore, a local feature L1 is obtained from the input image 301, and a local feature L2 is obtained from the perturbed image 302.

[0106] (iv) Step 1704: The adjustment unit 321 of the color and brightness feature correction unit 13 uses machine learning to obtain a feature FDi from the global feature values ​​G1 and G2 using a filter according to the above-mentioned equation (3), and creates a feature extractor D1. For the feature FDi, a filter coefficient wdj and an offset value bdi are obtained. Therefore, a global feature value G11 is obtained from the global feature value G1, and a global feature value G21 is obtained from the global feature value G2.

[0107] Furthermore, the adjustment unit 322 of the color / brightness feature correction unit 13 uses machine learning to obtain a feature FEi from the local feature amounts L1 and L2 using a filter according to the above-mentioned equation (4), and creates a feature extractor D2. For the feature FEi, a filter coefficient wej and an offset value bei are obtained. Accordingly, a local feature L11 is obtained from the local feature amount L1, and a local feature L21 is obtained from the local feature amount L2.

[0108] (v) Step 1705: The normalization unit 331 of the color / brightness feature corrector 13 corrects the global feature G11 to a global feature G111 by the above-mentioned formula (5). Also, the normalization unit 332 of the color / brightness feature corrector 13 corrects the global feature G21 to a global feature G211 by the above-mentioned formula (5).

[0109] (vi) Step 1706: The image reconstruction unit 14 generates a global feature G1111 from the global feature G111 using the above-described formula (6), concatenates the global feature G1111 with the local feature L11, and generates a corrected color / brightness feature CF1. The image reconstruction unit 14 also generates a global feature G2111 from the global feature G211 using the above-described formula (6), concatenates the global feature G2111 with the local feature L21, and generates a corrected color / brightness feature CF2. The image reconstruction unit 14 also uses machine learning to obtain a feature FPi from the corrected color / brightness feature CF1 using a filter using the above-described formula (7), and generates a feature extractor U1. For the feature FPi, the filter coefficient wpj and offset value bpi are calculated. Thus, a reconstructed image 351 is generated from the corrected color / brightness feature CF1.

[0110] The image reconstructor 14 also uses machine learning to obtain the feature FPi from the corrected color and brightness feature CF2 using a filter according to the above-mentioned equation (7), thereby creating a feature extractor U1. For the feature FPi, the filter coefficient wpj and offset value bpi are obtained. Therefore, a reconstructed image 352 is created from the corrected color and brightness feature CF2.

[0111] (vii) Step 1707: The color / brightness feature extraction unit 12, the color / brightness feature correction unit 13, and the image reconstruction unit 14 calculate the overall cost of equation (12) using equations (8) to (11) above, and update the parameters (filter coefficients, offset values) of equations (1) to (7) above using machine learning to reduce the cost, thereby creating a color / brightness corrector.

[0112] (viii) Step 1708 The output unit 15 stores the parameters (filter coefficients, offset values, etc.) of equations (1) to (7) in the memory 90.

[0113] 18 is a flowchart for explaining the operation during evaluation of the image processing device 1 according to the embodiment. In the following, each processing unit (input unit 10, color / brightness feature extraction unit 12, etc.) will be described as the main operating unit, but it may also be interpreted as the CPU 201 being the main operating unit, and the CPU 201 executing each processing unit as a program.

[0114] (i) Step 1801 The input unit 10 receives an evaluation image (also called an input image or a target image) and outputs the evaluation image to the color and brightness feature extraction unit 12.

[0115] (ii) Step 1802 The color and brightness feature extraction unit 12 reads the parameters (filter coefficients, offset values) of equations (1) and (2) from the memory, and calculates the global feature amount G1 and the local feature amount L1.

[0116] (iii) Step 1803 The color and brightness feature correction unit 13 reads each parameter (each filter coefficient, each offset value) of equations (3) and (4) from the memory, and obtains a global feature G11 from the global feature G1 and a local feature L11 from the local feature L1.

[0117] (iv) Step 1804 The color and brightness feature corrector 13 reads the parameters (filter coefficient, offset value) of equation (5) from the memory, and corrects the global feature G11 to a global feature G111.

[0118] (v) Step 1805 The image reconstructing unit 14 generates a global feature G1111 from the global feature G111 using the above-mentioned equation (6), and concatenates the global feature G1111 and the local feature L11 to generate a corrected color / brightness feature CF1.

[0119] Furthermore, the image reconstructing unit 14 reads the parameters (filter coefficients, offset values) of equation (7) from the memory, and creates a reconstructed image 351 from the corrected color / brightness feature value CF1.

[0120] (vi) Step 1806 The feature extraction unit 16 reads the parameters (filter coefficient, offset value) of equation (13) from the memory, and obtains the feature amount FAi of the object (for example, tissue, cell, etc.) in the reconstructed image 351 using a filter according to the above equation (13).

[0121] (vii) Step 1807 The classifier 17 reads the parameters (weight w, offset value b, etc.) of equation (14) from the memory, obtains a classification result from the feature amount FAi using equation (14), and calculates a classification value indicating the object-likelihood (e.g., lesion-likelihood) of the object in the reconstructed image 351. The parameters read in each of the above steps have been adjusted by learning as described above.

[0122] (viii) Step 1808 The output unit 15 stores the calculated discrimination result and discrimination value in the memory 90.

[0123] According to the first embodiment, a color and brightness corrector is created that extracts and corrects color and brightness features using perturbed images during machine learning to generate a reconstructed image in which color and brightness differences are suppressed. Furthermore, during evaluation, a reconstructed image is generated from the evaluation image using the color and brightness corrector, and the reconstructed image is identified using a classifier, making it possible to identify objects (e.g., abnormal tissue) in the image with high accuracy. Note that the color and brightness corrector learning and evaluation processes of the first embodiment may be executed by different computer systems, i.e., the evaluation processing system and the learning system.

[0124] (2) Second Embodiment As shown in Fig. 19, an image processing device 2 for machine learning according to the second embodiment includes many of the same components as the image processing device 1 of the first embodiment in Fig. 1, but unlike the first embodiment, does not include the color / brightness feature extraction unit 12 and the color / brightness feature correction unit 13, but includes a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24. Therefore, the following description will mainly focus on the components that are different from those in Fig. 1.

[0125] The image processing device 2 of this embodiment extracts and corrects color and brightness features in each of the two encoders during machine learning, and then creates color and brightness correctors in the decoder that create reconstructed images of the input image and the perturbed image from the corrected features. During evaluation, the color and brightness correctors are used to generate reconstructed images from the evaluation images, and the classifier is used to classify the reconstructed images, thereby enabling highly accurate identification of objects (e.g., abnormal tissue) in the images.

[0126] <Configuration and Operation of Each Unit> Below, the configuration and operation of each element during machine learning that differs from that shown in FIG. 1 will be described in detail. (i) Color / Brightness Feature Extraction / Correction Unit 18 FIGS. 20A and 20B show an example configuration of the color / brightness feature extraction / correction unit 18. The color / brightness feature extraction / correction unit 18 includes an encoder 71 and an encoder 72. The encoder 71 includes encoders 711 to 71M and adjustment units 731 to 73M. The encoder 72 includes encoders 721 to 72M and adjustment units 741 to 74M. Here, M represents any natural number equal to or greater than 2. The encoders 71 and 72 include normalization units 751 and 752, respectively.

[0127] The color / brightness feature extraction / correction unit 18 calculates the color / brightness feature of the input image and then corrects the color / brightness feature. Examples of calculating color / brightness feature (global feature, local feature) are shown in FIGS. 20A and 20B. The encoder and decoder in FIGS. 20A and 20B can use CNN.

[0128] For example, the encoder 711 obtains a global feature G711 from the input image 301, and obtains a global feature G721 from the perturbed image 302. The encoder 71M obtains a global feature G71M from the global feature of the input image 301 from the previous stage. The encoder 71M obtains a global feature G72M from the global feature of the perturbed image 302 from the previous stage.

[0129] The encoder 721 obtains a local feature L711 from the input image 301, and obtains a local feature L721 from the perturbed image 302. The encoder 72M obtains a local feature L71M from the local feature of the input image 301 from the previous stage. The encoder 72M obtains a local feature L72M from the local feature of the perturbed image 302 from the previous stage.

[0130] There are two methods: one is to add the output of the adjustment unit 73N to the global feature G71N to generate the global feature G711N for the next stage, and the other is to use the output of the adjustment unit 73N as the corrected global feature G711N directly (without adding the global feature G71N). In either method, the output of the adjustment unit 73N becomes the corrected global feature G711N. The same explanation applies to the global feature G72N, and also to the adjustment unit 74N and the local feature L71N or L72N.

[0131] 21A , for example, an encoder 71N uses a feature extractor GN to obtain global color / brightness features FGNi from an input image, a perturbed image, or a global feature using equation (15), where N is a natural number from 1 to M. As shown in FIG. 21B , an encoder 72N uses a feature extractor LN to obtain local color / brightness features FLNi from an input image, a perturbed image, or a local feature using equation (16).

[0132] The filter coefficient wgnj shown in equation (15) is a coefficient obtained by machine learning or the like so that the reconstructed image reconstructed from the input image approaches the input image, and so that the reconstructed image reconstructed from the perturbed image approaches the input image.

[0133] In equation (15), pgnj represents a pixel value, bgni represents an offset value, q represents the number of filter coefficients, and h represents a nonlinear function. As shown in Fig. 21A, equation (15) is used to calculate the calculation results of each filter from the upper left to the lower right of the target image to calculate a feature value fgni for an arbitrary filter i. For example, the matrix of feature values ​​fgni calculated by the feature extractor GN is set as the global feature value FGNi of the input image, the perturbed image, or the global feature value.

[0134]

[0135] The filter coefficient wlnj shown in equation (16) is a coefficient obtained by machine learning or the like so that the reconstructed image reconstructed from the input image approaches the input image, and so that the reconstructed image reconstructed from the perturbed image approaches the input image.

[0136] In equation (16), p1nj represents a pixel value, blni represents an offset value, r represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 21B , equation (16) is used to calculate the calculation results of each filter from the upper left to the lower right of the target image, thereby calculating the feature value flni of an arbitrary filter i. For example, the matrix of the feature value flni calculated by the feature extractor LN is set as the local feature value FLNi of the input image, the perturbed image, or the local feature value.

[0137]

[0138] Next, the adjustment unit 73N obtains the filter coefficient wdnj shown in equation (17) by machine learning or the like so that the global feature FGNi (global feature G71N) of the input image and the global feature FGNi (global feature G72N) of the perturbed image approach each other.

[0139] In equation (17), pdnj represents the value of the global feature, bdni represents the offset value, s1 represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 22A , equation (17) is used to calculate the calculation results of each filter for global features G71N and G72N, thereby calculating the feature fdni of an arbitrary filter i. For example, the matrix of feature fdni calculated from global feature G71N by feature extractor DN1 is defined as global feature G711N, and the matrix of feature fdni calculated from global feature G72N by feature extractor DN1 is defined as global feature G721N.

[0140]

[0141] The adjustment unit 74N obtains the filter coefficient wenj shown in equation (18) by machine learning or the like so that the local feature amount FLNi (local feature amount L71N) of the input image and the local feature amount FLNi (local feature amount L72N) of the perturbed image approach each other.

[0142] In equation (18), penj represents the value of the local feature, beni represents the offset value, s2 represents the number of filter coefficients, and h represents a nonlinear function. As shown in FIG. 22B , equation (18) is used to calculate the calculation results of each filter for the local features L71N and L72N, thereby calculating the feature feni of an arbitrary filter i. For example, the matrix of the feature feni calculated from the local feature L71N by the feature extractor DN2 is defined as local feature L711N, and the matrix of the feature feni calculated from the local feature L72N by the feature extractor DN2 is defined as local feature L721N.

[0143]

[0144] Using equations (15) and (17), the above process is performed by all encoders 71N and adjustment units 73N, and extraction and correction of global features are repeated to finally obtain global feature amounts G711M and global feature amounts G721M. Also, using equations (16) and (18), the above process is performed by all encoders 72N and adjustment units 74N, and extraction and correction of local features are repeated to finally obtain local feature amounts L711M and local feature amounts L721M.

[0145] The normalization unit 751 normalizes the global feature G711M corrected by the adjustment unit 73M as shown in equation (19). In equation (19), FG represents the global feature G711M, FGmax represents the maximum value of the global feature G711M, and FLmax represents the maximum value of the local feature L711M. Therefore, equation (19) is used to correct the global feature G711M to the global feature G711M.

[0146]

[0147] The normalization unit 752 normalizes the global feature G721M corrected by the adjustment unit 73M as shown in equation (19). In equation (19), FG represents the global feature G721M, FGmax represents the maximum value of the global feature G721M, and FLmax represents the maximum value of the local feature L721M. Therefore, equation (19) is used to correct the global feature G721M to the global feature G7211M.

[0148] The global feature G7111M and the local feature L711M are combined into a set to form a normalized feature NM1. Furthermore, the global feature G7211M and the local feature L721M are combined into a set to form a normalized feature NM2.

[0149] (ii) Image Reconstruction Unit 24 The image reconstruction unit 24 uses a decoder to create a reconstructed image 351 from the normalized feature NM1 and a reconstructed image 352 from the normalized feature NM2. As shown in Fig. 7 and equation (6), x, which is the global feature G7111M, is copied from the top left to the bottom right so that the vertical and horizontal sizes become the same as those of the local feature L711M to create a global feature G71111M, and the global feature G71111M and the local feature L711M are concatenated to create a corrected color / brightness feature CFM1.

[0150] Furthermore, as shown in Figure 7 and equation (6), x, which is the global feature G7211M, is copied from the top left to the bottom right so that the size is the same as the vertical direction Hy and the horizontal direction Wy of the local feature L721M to create a global feature G72111M, and the global feature G72111M and the local feature L721M are concatenated to create a corrected color / brightness feature CFM2.

[0151] The decoder is the same as the image reconstructor 14 of the first embodiment. The decoder creates a reconstructed image 351 from a matrix of feature values ​​fpi obtained by the feature extractor U1 from the corrected color and brightness feature values ​​CFM1. The decoder also obtains the filter coefficients wpj shown in equation (7) by machine learning or the like so that the input image 301 and the reconstructed image 351 reconstructed from the corrected color and brightness feature values ​​CFM1 approach each other.

[0152] The decoder creates a reconstructed image 352 from a matrix of feature values ​​fpi obtained by the feature extractor U1 from the corrected color and brightness feature values ​​CFM2. The decoder also obtains the filter coefficients wqj shown in equation (7) by machine learning or the like so that the input image 301 and the reconstructed image 352 reconstructed from the corrected color and brightness feature values ​​CFM2 approach each other.

[0153] <Hardware configuration of image processing device> An example of the hardware configuration of the image processing device 2 according to this embodiment has the same configuration as that shown in FIG. 2 , but differs from the image processing device 1 according to the first embodiment in that the image processing device 2 includes a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24 in the memory 202, instead of the color / brightness feature extraction unit 12 and the color / brightness feature correction unit 13.

[0154] The storage device 203 of the image processing device 2 stores the parameters of the color and brightness corrector generated by the color and brightness feature extraction and correction unit 18 and the image reconstruction unit 24, the reconstructed image, and the like.

[0155] 23 is a flowchart for explaining the operation of the image processing device 2 during machine learning according to this embodiment. In the following, each processing unit (such as the input unit 10 and the color / brightness feature extraction / correction unit 18) will be described as the main operating unit, but this may also be interpreted as the CPU 201 being the main operating unit, and the CPU 201 executing each processing unit as a program.

[0156] (i) Step 2301 The input unit 10 receives a learning image and outputs the input image to the perturbed image creation unit 11 and the color / brightness feature extraction / correction unit 18 .

[0157] (ii) Step 2302 The perturbed image generating unit 11 generates a perturbed image by randomly perturbing each value (R value, G value, B value, etc.) of the image, the γ value, etc.

[0158] (iii) Step 2303: The color / brightness feature extraction / correction unit 18 uses machine learning to obtain global features FGNi from the input image, perturbed image, or previous global features G711N and G721N using a filter according to the above-mentioned equation (15), and creates a feature extractor GN. For the global features FGNi, a filter coefficient wgnj and an offset value bgni are calculated. Therefore, the current global feature G71N is calculated from the input image or previous global feature G711N, and the current global feature G72N is calculated from the perturbed image or previous global feature G721N.

[0159] Furthermore, the color / brightness feature extraction / correction unit 18 uses a filter based on machine learning to determine a local feature FLNi from the input image, the perturbed image, or the previous local feature amounts L711N and L721N using the above-described formula (16), thereby creating a feature extractor LN. A filter coefficient wlnj and an offset value blni are determined for the local feature FLNi. Therefore, the current local feature L71N is determined from the input image or the previous local feature amount L711N, and the current local feature L72N is determined from the perturbed image, the local feature amount L72N, or the previous local feature amount L721N.

[0160] (iv) Step 2304: The adjustment unit 73N of the color / brightness feature extraction / correction unit 18 uses machine learning to obtain a feature FDNi from the global feature quantities G71N and G72N using a filter according to the above-mentioned equation (17), and creates a feature extractor DN1. For the feature FDNi, a filter coefficient wdnj and an offset value bdni are obtained. Therefore, the current global feature quantity G711N is obtained from the global feature quantity G71N, and the current global feature quantity G721N is obtained from the global feature quantity G72N.

[0161] Furthermore, the adjustment unit 74N of the color / brightness feature extraction / correction unit 18 uses machine learning to obtain a feature FENi from the local feature amounts L71N and L72N using a filter according to the above-mentioned equation (18), and creates a feature extractor DN2. For the feature FENi, a filter coefficient wenj and an offset value beni are obtained. Therefore, a current local feature amount L711N is obtained from the local feature amount L71N, and a current local feature amount L721N is obtained from the local feature amount L72N.

[0162] (v) Step 2305: If the color / brightness feature extraction / correction unit 18 has already calculated the global feature amounts G711N, G721N and the local feature amounts L711N, L721N for all adjustment units 73N, 74N, the process proceeds to step 2306. Otherwise, the process returns to step 2303.

[0163] (vi) Step 2306: The normalization unit 751 of the color / brightness feature extraction / correction unit 18 corrects the global feature G711N to a global feature G7111N using the above-mentioned formula (19). Also, the normalization unit 752 of the color / brightness feature extraction / correction unit 18 corrects the global feature G721N to a global feature G7211N using the above-mentioned formula (19).

[0164] (vii) Step 2307 The image reconstructing unit 24 generates a global feature G71111N from the global feature G7111N using the above-mentioned formula (6), concatenates the global feature G71111N and the local feature L711N, and generates a corrected color / brightness feature CFM1. Also, the image reconstructing unit 24 generates a global feature G72111N from the global feature G7211N using the above-mentioned formula (6), and concatenates the global feature G72111N and the local feature L721N, and generates a corrected color / brightness feature CFM2.

[0165] The image reconstructor 24 also uses machine learning to obtain the feature FPi from the corrected color and brightness feature CFM1 using a filter according to the above-mentioned equation (7), thereby creating a feature extractor U1. For the feature FPi, the filter coefficient wpj and offset value bpi are obtained. Therefore, a reconstructed image 351 is created from the corrected color and brightness feature CFM1.

[0166] The image reconstructor 24 also uses machine learning to obtain the feature FPi from the corrected color and brightness feature CFM2 using a filter according to the above-mentioned equation (7), thereby creating a feature extractor U1. For the feature FPi, the filter coefficient wpj and offset value bpi are obtained. Therefore, a reconstructed image 352 is created from the corrected color and brightness feature CFM2.

[0167] (viii) Step 2308: The color / brightness feature extraction / correction unit 18 and the image reconstruction unit 24 calculate the overall cost of equation (12) using equations (8) to (11) above, and update the parameters (filter coefficients, offset values) of equations (15) to (19) above and equation (7) above using machine learning to reduce cost, thereby creating a color / brightness corrector. Note that in equations (10) and (11) above, Feature G11 represents the global feature G711N, Feature G21 represents the global feature G721N, Feature L11 represents the local feature L711N, and Feature L21 represents the local feature L721N.

[0168] (ix) Step 2309 The output unit 15 stores the parameters (filter coefficients, offset values, etc.) of equations (15) to (19) and equation (7) in the memory 90.

[0169] As shown in Fig. 24, the image processing device 2 at the time of evaluation according to the second embodiment includes many of the same components as the image processing device 1 of the first embodiment in Fig. 4, but unlike the first embodiment, does not include the color / brightness feature extraction unit 12 and the color / brightness feature correction unit 13, but includes a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24. Therefore, only the components different from Fig. 4 will be described here.

[0170] The parameter values ​​of the color and brightness corrector created during learning are read from the memory 90, and the parameters of the color and brightness feature extraction and correction unit 18 and the image reconstruction unit 24 are set.

[0171] For an image input from an input unit 10, a color / brightness feature extraction / correction unit 18 and an image reconstruction unit 24 are used to generate a reconstructed image in which color / brightness differences are suppressed.

[0172] 25 is a flowchart for explaining the operation during evaluation of the image processing device 2 according to this embodiment. In the following, each processing unit (input unit 10, color / brightness feature extraction / correction unit 18, etc.) will be described as the main operating unit, but it may also be interpreted as the CPU 201 being the main operating unit, and the CPU 201 executing each processing unit as a program.

[0173] (i) Step 2501 The input unit 10 receives an evaluation image and outputs the evaluation image to the color and brightness feature extraction and correction unit 18 .

[0174] (ii) Step 2502 The color and brightness feature extraction and correction unit 18 reads the parameters (filter coefficients, offset values) of equations (15) and (16) from the memory, and calculates the global feature amount G71N and the local feature amount L71N.

[0175] (iii) Step 2503 The color / brightness feature extraction / correction unit 18 reads each parameter (each filter coefficient, each offset value) of equations (17) and (18) from the memory, and the adjustment unit 73N calculates a global feature G711N from the global feature G71N, and the adjustment unit 74N calculates a local feature L711N from the local feature L71N.

[0176] (iv) Step 2504 The color and brightness feature extraction and correction unit 18 reads the parameters (filter coefficient, offset value) of equation (19) from the memory, and corrects the global feature G711N to the global feature G7111N.

[0177] (v) Step 2505 The image reconstructing unit 24 generates a global feature G71111N from the global feature G7111N using the above-mentioned equation (6), and concatenates the global feature G71111N and the local feature L711N to generate a corrected color / brightness feature CFM1.

[0178] Furthermore, the image reconstructing unit 24 reads the parameters (filter coefficients, offset values) of equation (7) from the memory, and creates a reconstructed image 351 from the corrected color / brightness feature value CFM1.

[0179] (vi) Step 2506 The feature extraction unit 16 reads the parameters (filter coefficient, offset value) of equation (13) from the memory, and obtains the feature amount FAi of the object (for example, tissue, cell, etc.) in the reconstructed image 351 using a filter according to the above equation (13).

[0180] (vii) Step 2507 The classifier 17 reads the parameters (weight w, offset value b, etc.) of equation (14) from the memory, obtains a classification result from the feature amount FAi using equation (14), and calculates a classification value indicating the object-likelihood (e.g., lesion-likelihood) of the object in the reconstructed image 351.

[0181] (viii) Step 2508 The output unit 15 stores the calculated identification result and identification value in the memory 90.

[0182] According to the second embodiment, a color and brightness corrector is created that extracts and corrects color and brightness features for each layer while using perturbed images during machine learning, and generates a reconstructed image in which color and brightness differences are suppressed. Furthermore, during evaluation, a reconstructed image is generated from the evaluation image using the color and brightness corrector, and the reconstructed image is identified using a classifier, making it possible to identify objects (e.g., abnormal tissue) in the image with high accuracy. Note that the color and brightness corrector learning and evaluation processes of the second embodiment may be executed by different computer systems, i.e., the evaluation processing system and the learning system.

[0183] (3) Third Embodiment Fig. 26 is a functional block diagram showing the configuration of an image processing system 2600 according to a third embodiment of the present specification. The image processing system 2600 includes a server 2603 and an image acquisition device 2605. The image processing system 2600 also functions as an image identification system.

[0184] The image acquisition device 2605 is, for example, a device such as a virtual slide device or a personal computer equipped with a camera, and has an imaging unit 2601 that captures image data and a display unit 2604 that displays the reconstructed image / identification results transmitted from the server 2603. Although not shown, the image acquisition device 2605 also has a communication device that transmits image data to the server 2603 and receives data transmitted from the server 2603.

[0185] The server 2603 has a storage unit 2602 that creates a color / brightness corrector using the image processing device 1 or 2 according to the first or second embodiment for image data (input image) transmitted from the image acquisition device 2605, obtains a reconstructed image and identification results using the created color / brightness corrector, and stores the reconstructed image and identification results output from the image processing device 1 or 2.

[0186] Although not shown, the server 2603 has a communication device that receives image data transmitted from the image acquisition device 2605 and transmits data such as reconstructed images and identification results to the image acquisition device 2605.

[0187] The image processing device 1 or 2 uses a color / brightness corrector to generate a reconstructed image of the image captured by the imaging unit 2601, and classifies objects (e.g., tissues, cells, etc.) in the reconstructed image to determine whether or not they contain objects to be detected (e.g., abnormal tissues or abnormal cells (e.g., cancer)). The display unit 2604 displays data such as the reconstructed image and the identification results transmitted from the server 2603 on the display screen of the image acquisition device 2605. The image acquisition device 2605 may be a regenerative medicine device or an iPS cell culture device having an imaging unit, or an MRI or ultrasound imaging device, etc.

[0188] According to the third embodiment, a color / brightness corrector determined by the image processing device 1 or 2 is used to generate a reconstructed image of an image transmitted from a facility or the like at a different location, and objects (e.g., tissues, cells, etc.) in the reconstructed image are accurately classified as objects to be detected (abnormal tissues, abnormal cells, etc.) using each parameter of the classifier, and data such as the reconstructed image and the classification results are transmitted to a facility or the like at a different location, and the reconstructed image and the data such as the classification results are displayed on the display unit of an image acquisition device at that facility or the like, thereby making it possible to provide an image processing system.

[0189] (4) Fourth Embodiment Fig. 27 is a functional block diagram showing the configuration of an image processing system 2700 according to a fourth embodiment of the present invention. The image processing system 2700 includes a server 2703 and an image acquisition device 2705. The image processing system 2700 also functions as an image identification system.

[0190] The image acquisition device 2705 is, for example, a device such as a virtual slide device or a personal computer equipped with a camera, and includes an imaging unit 2701 that captures image data, a storage unit 2704 that stores color and brightness correctors transmitted from the server 2703, and an image processing device 1 or 2 that reads the color and brightness correctors transmitted from the server 2703, generates a reconstructed image of the image newly captured by the imaging unit 2701 of the image acquisition device 2705, and classifies objects (e.g., tissues, cells, etc.) in the reconstructed image as objects to be detected (e.g., abnormal tissues, abnormal cells, etc.) using the classifier of the image processing device 1 or 2 according to the first or second embodiment of the present invention. Note that the image acquisition device 2705 also includes a communication device (not shown) that transmits image data to the server 2703 and receives data transmitted from the server 2703.

[0191] The server 2703 has a storage unit 2702 that creates a color and brightness corrector from the image processing device 1 or 2 according to the first or second embodiment of the present invention for image data transmitted from the image acquisition device 2705 and stores the color and brightness corrector output from the image processing device 1 or 2. The server 2703 also accepts input designation of an input image or a teacher image from an operator. The server 2703 also presents each output (reconstructed image, classification result, etc.) of the image processing device 1 or 2 to the operator. Although not shown, the server 2703 also has a communication device that receives image data transmitted from the image acquisition device 2705 and transmits the color and brightness corrector to the image acquisition device 2705.

[0192] The image processing device 1 or 2 generates a reconstructed image from an image captured by the imaging unit 2701, performs machine learning on objects (e.g., tissues, cells, etc.) in the reconstructed image to determine whether they are objects to be detected (e.g., normal tissues and cells are normal tissues and cells, and abnormal tissues and cells are abnormal tissues and cells, etc.), and creates a classifier that calculates feature quantities of objects (e.g., tissues, cells, etc.) in images of facilities, etc., located in different locations. The storage unit 2704 stores color and brightness correctors, etc., transmitted from the server 2703.

[0193] The image processing device 1 or 2 in the image acquisition device 2705 reads a color / brightness corrector, a classifier, etc. from the storage unit 2704, and uses the color / brightness corrector to generate a reconstructed image of an image newly captured by the imaging unit 2701 of the image acquisition device 2705.The image processing device 1 or 2 uses the classifier to classify objects (e.g., tissues, cells, etc.) in the reconstructed image as objects to be detected (e.g., abnormal tissues, abnormal cells, etc.), and displays data such as the reconstructed image and the classification results on the display screen of the output device 204 of the image processing device 1 or 2.

[0194] The image acquisition device 2705 may be a regenerative medicine device or an iPS cell culture device having an imaging unit, or an MRI or ultrasonic imaging device.

[0195] According to the fourth embodiment, machine learning is performed on images transmitted from facilities at different locations to create a color / brightness corrector or the like so as to reduce differences in color and brightness between the images, the color / brightness corrector or the like is transmitted to a facility at a different location, the color / brightness corrector is loaded into an image acquisition device at the facility, and a reconstructed image of the newly captured image is generated. A classifier created by performing machine learning to classify objects to be detected (e.g., normal tissues and cells as normal tissues and cells, abnormal tissues and cells as abnormal tissues and cells, etc.) is then loaded, and objects (e.g., tissues, cells, etc.) in the reconstructed image are classified as objects to be detected (e.g., abnormal tissues, abnormal cells, etc.), thereby providing an image processing system.

[0196] The above-described embodiments can be modified as follows: In the perturbed image creation unit 11, perturbation may be performed on a color system other than RGB, such as HSV or HLS, or minute noise may be added, with the same effect.

[0197] Although the normalization method of the normalization unit of the color / brightness feature correction unit 13 has been described above as an example of a normalization method in which global features are normalized using the maximum value of local features, normalization methods other than those described above may also be used, with similar effects, such as normalizing local features using the maximum value of global features or normalizing the lengths of feature vectors at each coordinate of the global feature and local feature. Furthermore, although normalization is performed using two normalization units in the color / brightness feature correction unit 13, normalization may be performed using a number other than two, with similar effects.

[0198] In the image reconstruction units 14 and 24, the loss value is calculated using MSE as the loss function, but other loss functions such as cross entropy error may be used, with the same effect. Also, although the cost is calculated using four loss values, the cost may be calculated using five loss values, including the loss values ​​of the classification result and the correct value, with the same effect.

[0199] Although the feature extraction unit 16 uses filters based on machine learning to obtain multiple feature quantities, other feature quantities such as HOG may be used with similar effects. Furthermore, the identification unit 17 uses negative log likelihood as a loss function, but squared error, hinge loss, or the like may be used with similar effects.

[0200] The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a system or device, and the computer (or CPU or MPU) of the system or device reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.

[0201] Furthermore, an operating system (OS) running on a computer may perform some or all of the actual processing based on instructions in the program code, and the functions of the above-described embodiments may be realized by this processing.Furthermore, after the program code is read from a storage medium and written to memory on the computer, a CPU of the computer may perform some or all of the actual processing based on instructions in the program code, and the functions of the above-described embodiments may be realized by this processing.

[0202] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a hard disk or memory of the system or device, or in a storage medium such as a CD-RW or CD-R, so that when used, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage means or storage medium.

[0203] Finally, the processes and techniques described herein are not inherently related to any particular apparatus but may be implemented by any suitable combination of components. Furthermore, various types of general-purpose devices can be used in accordance with the methods described herein. It may also be beneficial to construct specialized apparatus to perform the method steps described herein. Various inventions can also be formed by suitable combinations of multiple components disclosed in the embodiments. For example, some components may be omitted from all of the components shown in the embodiments. Furthermore, components from different embodiments may be combined as appropriate. While the present invention has been described with reference to specific examples, these are intended in all respects to be illustrative and not limiting. Those skilled in the art will recognize that there are numerous combinations of hardware, software, and firmware suitable for practicing the present invention. For example, the described software can be implemented in a wide variety of programming or scripting languages, including assembler, C / C++, Perl, Shell, PHP, Java, etc.

[0204] Furthermore, in the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0205] Additionally, other implementations of the invention will be apparent to those skilled in the art from consideration of the specification and embodiments of the invention disclosed herein. Various aspects and / or components of the described embodiments may be used alone or in any combination.

Claims

1. An image processing device comprising: one or more processors; and one or more memories, wherein the one or more memories store a corrector that generates a reconstructed image from an input image in accordance with setting parameters, and the setting parameters are adjusted so that a first reconstructed image generated by the corrector from an input first image and a second image of the same object but with a different color tone from the first image approach each other, wherein the processor inputs a target image to the corrector and generates a reconstructed image of the target image, and the corrector extracts features from the input target image in accordance with the setting parameters, corrects the extracted features, and generates the reconstructed image of the target image from the corrected features.

2. An image processing device according to claim 1, wherein the first image is a perturbed image generated from the second image.

3. An image processing device as described in claim 1, wherein the corrector generates multiple types of feature amounts of different sizes including the feature amount from the input target image, corrects each of the multiple types of extracted feature amounts, and generates the reconstructed image of the target image from the multiple types of corrected feature amounts.

4. An image processing device as described in claim 1, wherein the one or more processors extract features from the reconstructed image of the target image, and determine an identification result for the reconstructed image of the target image from the features extracted from the reconstructed image of the target image.

5. An image processing device as described in claim 1, wherein the one or more processors perform the adjustment of the setting parameters, and in the adjustment, perturb the second image to generate the first image, input the first image to the corrector to generate the first reconstructed image, input the second image to the corrector to generate a second reconstructed image, and adjust the setting parameters so that a cost calculated from a predetermined loss is reduced, and the loss includes a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

6. An image processing device according to claim 5, wherein the loss includes a difference between a feature amount obtained by correcting a feature amount extracted from the first image and a feature amount obtained by correcting a feature amount extracted from the second image.

7. An image processing device as described in claim 3, wherein the one or more processors perform the adjustment of the setting parameters, and in the adjustment, perturb the second image to generate the first image, input the first image to the corrector to generate the first reconstructed image, input the second image to the corrector to generate a second reconstructed image, and adjust the setting parameters so that a cost calculated from a predetermined loss is reduced, and the loss includes: a difference between the first reconstructed image and the second image, a difference between the second reconstructed image and the second image, and a difference in corrected features of the same type between features obtained by correcting each of the multiple types of features extracted from the first image and features obtained by correcting each of the multiple types of features extracted from the second image.

8. An image processing device according to claim 5, wherein the one or more processors display the loss, the first reconstructed image, and the second reconstructed image.

9. An image processing device comprising: one or more processors; and one or more memories, wherein the one or more memories store a corrector that generates a reconstructed image from an input image in accordance with setting parameters, wherein the one or more processors perform the adjustment of the setting parameters, and in the adjustment, the following are performed: perturbing a second image to generate a first image; inputting the first image to the corrector to generate the first reconstructed image; inputting the second image to the corrector to generate a second reconstructed image; adjusting the setting parameters so that a cost calculated from a predetermined loss is reduced, wherein the loss includes a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

10. An image processing device as described in claim 9, wherein the loss includes a difference between a feature amount obtained by correcting a feature amount extracted from the first image and a feature amount obtained by correcting a feature amount extracted from the second image.

11. An image identification system comprising: an image acquisition device including an imaging device that photographs an object and generates an image; and an image processing device as described in claim 4, wherein the image acquisition device transmits the object image to the image processing device, the image processing device transmits a reconstructed image of the object image and the identification result to the image acquisition device, and the image acquisition device displays the received reconstructed image and the identification result on a display device.

12. An image identification system comprising: an image acquisition device that captures image data; and a first image processing device which is the image processing device described in claim 9; the image acquisition device comprises the image processing device described in claim 4; the image acquisition device transmits the image data to the first image processing device; the first image processing device generates the corrector using the received image data, stores the corrector in a memory, and transmits the corrector to the image acquisition device; the image acquisition device stores the received corrector in a memory; and the image processing device within the image acquisition device generates a reconstructed image of the target image using the corrector and calculates the identification result, and displays the target image, the reconstructed image of the target image, and the identification result on a display device.

13. An image processing method using an apparatus, wherein the apparatus stores a corrector that generates a reconstructed image from an input image in accordance with setting parameters, and the setting parameters are adjusted so that a first reconstructed image generated by the corrector from an input first image and a second image of the same object but with a different color tone from the first image approach each other, and the image processing method comprises the apparatus inputting a target image to the corrector to generate a reconstructed image of the target image, and the corrector generates the reconstructed image of the target image in accordance with the setting parameters, extracting features from the input target image, correcting the extracted features, and generating the reconstructed image of the target image from the corrected features.

14. An image processing method according to claim 13, further comprising the step of: calculating an identification result of the reconstructed image of the input image from a feature quantity extracted from the reconstructed image of the input image.

15. An image processing method using an apparatus, the apparatus storing a corrector that generates a reconstructed image from an input image in accordance with setting parameters, the image processing method comprising the steps of: inputting a target image to the corrector to generate a reconstructed image of the target image; the corrector generating the reconstructed image of the target image by perturbing a second image in accordance with the setting parameters, inputting the first image to the corrector to generate the first reconstructed image, and inputting the second image to the corrector to generate a second reconstructed image; and adjusting the setting parameters so that a cost calculated from a predetermined loss is reduced, the loss including a difference between the first reconstructed image and the second image and a difference between the second reconstructed image and the second image.

Images