Image processing device, image processing method, and program

By using the inference and display processing of the image processing device, a visual drawing is generated, which solves the problem of low likelihood of the detailed investigation area in the image and improves the efficiency of the operation.

CN112955929BActive Publication Date: 2024-11-08KK TOSHIBA +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980073242.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-11-07
Filing Date
2019-11-05
Publication Date
2024-11-08
Estimated Expiration
2039-11-05

AI Technical Summary

Technical Problem

In existing technologies, the classification results of detailed inspection areas within images have low likelihood, resulting in a significant amount of time being spent on detailed inspection work.

Method used

An image processing device is used to perform region classification, multidimensional data transformation and display processing through an inference processing unit, a dimension transformation processing unit and a display processing unit, respectively, to generate and display a drawing with color or pattern, so as to facilitate the identification and selection of specific regions.

Benefits of technology

It improves the efficiency of detailed inspection areas within images, quickly identifying and confirming areas with low likelihood through visualization, and reducing manual operation time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112955929B_ABST
    Figure CN112955929B_ABST
Patent Text Reader

Abstract

The present invention relates to an image processing device, an image processing method and a program. The image processing device of an embodiment has an inference processing unit, a dimensional conversion processing unit and a display processing unit. The inference processing unit obtains a classification result obtained by classifying multiple regions set in an image into any one of multiple predetermined classes. The dimensional conversion processing unit converts multidimensional data corresponding to multiple regions set in the image into low-dimensional data. The display processing unit displays a display image including one or more regions and a drawing diagram having multiple drawing points in which colors or patterns corresponding to the classification results are added to the low-dimensional data in each of the multiple regions, and, when there is an instruction for selecting a selected point from each of the drawn points, performs an identification display for making the selected point and the region corresponding to the selected point in the display image identifiable.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to an image processing device, an image processing method, and a program. Background Art

[0002] Conventionally, there is known a method for setting a plurality of regions in an image, and classifying each of the plurality of regions into any one of a plurality of predetermined classes to obtain a classification result.

[0003] However, according to the conventionally known method, there is a problem that it takes a lot of time to identify a part that needs to be carefully examined in the image, such as a region with a low likelihood of the above-mentioned classification result.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2018-18313

[0007] Non-patent literature

[0008] Non-patent document 1: Kiyuna Asaharu, "e-Pathologist: Pathological image analysis system based on artificial intelligence", [online], June 30, 2017, J-STAGE, Journal of the Society for Medical Image Information, Vol. 34, No. 2, 2017, pp. 105-108, [retrieved on October 29, 2017], Internet<URL:https: / / www.jstage.jst.go.jp / article / mii / 34 / 2 / 34_105 / _pdf / -char / ja>

[0009] An object of the embodiments is to provide an image processing device, an image processing method, and a program capable of improving the work efficiency in the work of specifying a part requiring detailed examination in an image. Summary of the invention

[0010] Means for solving problems

[0011] The image processing device of the embodiment includes an inference processing unit, a dimensionality conversion processing unit, and a display processing unit. The inference processing unit is configured to perform processing for obtaining a classification result obtained by classifying a plurality of regions set in a processing target image into any one of a plurality of predetermined classes. The dimensionality conversion processing unit is configured to perform processing for obtaining multidimensional data corresponding to the plurality of regions set in the processing target image, and to perform processing for converting the multidimensional data into low-dimensional data. The display processing unit is configured to perform drawing for making a display image including one or more of the plurality of regions, and a plurality of drawing points generated by adding different colors or patterns corresponding to the classification result to the low-dimensional data in each of the plurality of regions. Figure 1 and displaying it on a display device, and, in the presence of an instruction for selecting a desired drawing point as a selection point from among the drawing points included in the drawing, performing processing related to an identification display for making the selection point and the area corresponding to the selection point in the display image visually identifiable. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a diagram showing an example of the configuration of an image processing system including the image processing device according to the embodiment.

[0013] Figure 2 It is a diagram showing an example of the configuration of the image processing device according to the first embodiment.

[0014] Figure 3A This is a flowchart showing a part of a specific example of processing performed in the image processing device according to the first embodiment.

[0015] Figure 3B Yes means Figure 3A Flow chart of the subsequent processing of .

[0016] Figure 4 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0017] Figure 5 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0018] Figure 6 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0019] Figure 7 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0020] Figure 8 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0021] Fig. 9 It is used to illustrate Figure 8 A diagram of a two-dimensional distribution map included in the analysis support image of the display image.

[0022] Fig.10 is a diagram showing an example of the configuration of an image processing device according to the second embodiment.

[0023] Fig.11A This is a flowchart showing a part of a specific example of processing performed in the image processing device according to the second embodiment.

[0024] Fig. 11B Yes means Fig.11A Flow chart of the subsequent processing of .

[0025] Fig. 12A This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0026] Fig. 12B This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0027] Fig.13A This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0028] Fig. 13B This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0029] Fig.14A This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0030] Fig. 14B This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0031] Fig.15A It is used to illustrate Fig.14A A diagram of a two-dimensional distribution map included in the analysis support image of the display image.

[0032] Fig. 15B It is used to illustrate Fig. 14B A diagram of a two-dimensional distribution map included in the analysis support image of the display image. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments will be described with reference to the drawings.

[0034] (First Embodiment)

[0035] Figures 1 to 9 This invention relates to the first embodiment.

[0036] like Figure 1 As shown, the image processing system 101 includes an imaging device 1 , an image processing device 2 , an input I / F (interface) 3 , and a display device 4 . Figure 1 It is a diagram showing an example of the configuration of an image processing system including the image processing device according to the embodiment.

[0037] The imaging device 1 is configured to include, for example, an image sensor. The imaging device 1 is configured to capture an image of a subject and obtain an image, and output the obtained image to the image processing device 2. In addition, in the present embodiment, the description is made assuming that the ID number of the subject captured by the imaging device 1 is preliminarily added as additional information to the image obtained by capturing the subject.

[0038] The image processing device 2 is configured as, for example, a computer having one or more processors 201 and a storage medium 202. In addition, the image processing device 2 is configured to generate a display image by processing the image output from the camera device 1, and output the generated display image to the display device 4. In addition, the image processing device 2 is configured to detect an instruction output from the input I / F 3, generate a display image corresponding to the detected instruction, and output it to the display device 4.

[0039] The input I / F3 is configured to include an input device that can be operated by a user, such as a mouse, a keyboard, and a touch panel. In addition, the input I / F3 is configured to be able to give an instruction for setting the magnification of a main image (described later) included in the display image displayed by the display device 4 to a desired magnification. In addition, the input I / F3 is configured to be able to give an instruction for setting a focus position, which is equivalent to the center point when the main image is displayed in the display screen of the display device 4, and is equivalent to the reference point when the magnification of the main image is changed. In addition, the input I / F3 is configured to be able to give an instruction for selecting a desired position in the main image displayed by the display device 4 as a selected position. In addition, the input I / F3 is configured to be able to give an instruction for selecting a desired drawing point from each drawing point included in a two-dimensional distribution diagram (described later) displayed by the display device 4 as a selected point.

[0040] The display device 4 is configured to include, for example, a liquid crystal monitor, etc. The display device 4 is configured to be able to display a display image and display information output from the image processing device 2 .

[0041] like Figure 2 As shown, the image processing device 2 includes, for example, a pre-processing unit 21 , an image recording unit 22 , an inference processing unit 23 , a dimensional conversion processing unit 24 , a processing result storage unit 25 , and a display processing unit 26 . Figure 2 It is a diagram showing an example of the configuration of the image processing device according to the first embodiment.

[0042] The preprocessing unit 21 is configured to generate a plurality of images having different resolutions by performing a conversion process for converting the resolution of the image output from the imaging device 1. The preprocessing unit 21 is configured to output the plurality of images generated as described above to the image recording unit 22.

[0043] The image recording unit 22 is configured to record the plurality of images output from the pre-processing unit 21 in accordance with the ID numbers of the subjects.

[0044] In addition, the following description assumes the following situation: P images having N different resolutions R1, R2, ..., RN are generated by the pre-processing unit 21, and the generated P images are recorded in the image recording unit 22 according to the ID number of the subject. In addition, the following description assumes that the resolutions of the P images generated by the pre-processing unit 21 satisfy the relationship of R1<R2<...<RN. In addition, in this embodiment, at least one image of each resolution R1 to RN is included in the P images.

[0045] The inference processing unit 23 is configured to read an image GRN having the highest resolution RN from among the P images recorded in the image recording unit 22 and to set S (S≧2) regions in the read image GRN.

[0046] In addition, in this embodiment, the description is made assuming that the S regions set by the inference processing unit 23 and the dimensional conversion processing unit 24 are rectangular regions. In addition, in this embodiment, the description is made assuming that the inference processing unit 23 and the dimensional conversion processing unit 24 set the S regions in the same manner.

[0047] The inference processing unit 23 is configured to perform processing for obtaining class classification information corresponding to information indicating a classification result obtained by classification into any one of a plurality of predetermined classes, and processing for obtaining likelihood information corresponding to information indicating a calculation result of likelihood of the classification result, for each of the S regions set in the image GRN. In addition, the inference processing unit 23 is configured to output the class classification information and likelihood information corresponding to each of the S regions set in the image GRN to the processing result storage unit 25 in association with the ID number of the subject.

[0048] That is, the inference processing unit 23 is configured to perform the following processing: a process for obtaining a classification result of classifying a plurality of regions set in the image GRN (processing target image) into any one of a plurality of predetermined classes; and a process for calculating the likelihood of the classification result in each of the plurality of regions.

[0049] In addition, in the present embodiment, the inference processing unit 23 is constituted by a multi-layer neural network having an input layer, a hidden layer, and an output layer, for example, as a statistical model constituted by machine learning. In addition, in the present embodiment, the inference processing unit 23 is constituted as an inference model obtained by learning each coupling coefficient (weight) contained in the above-mentioned neural network by deep learning (machine learning). Supervised learning is performed in the inference processing unit 23. Then, in the inference processing unit 23 of the present embodiment, for example, when multidimensional data such as pixel values ​​or feature vectors obtained from each pixel contained in one of the S regions set in the image GRN is input as input data to the input layer of the neural network, the output layer of the neural network outputs output data indicating the classification result obtained by classifying the one region into any one of the specified multiple classes and the likelihood of the classification result of the one region. In addition, in the inference processing unit 23, classification is performed based on the size of the likelihood, but as an example of the likelihood calculated by the above-mentioned statistical model, the output value of each node in the output layer of the neural network is used as the likelihood (the likelihood is not calculated by another process in particular). In general, in the learning of the recognition function using a neural network, the node output of the output layer is learned in such a way that the correct answer category 1 and the incorrect answer category 0. During reasoning, the following state is shown: the closer the output is to 1, the more likely the category assigned to the node will appear (the higher the likelihood), and conversely, the closer it is to 0, the less likely it will appear (the lower the likelihood). Likelihood is a value generated for each type of class. When it is a binary classification problem as described later, it is 0.8 for normal and 0.2 for abnormal (the total value is 1), and it can be inferred normally as a classification result.

[0050] The dimension conversion processing unit 24 includes a multi-dimensional data acquisition unit 24A and a low-dimensional data conversion unit 24B.

[0051] The multi-dimensional data acquisition unit 24A is configured to read the image GRN from the P images recorded in the image recording unit 22 and set S regions in the read image GRN.

[0052] The multidimensional data acquisition unit 24A is configured to perform processing for acquiring multidimensional data corresponding to the S regions set in the image GRN. Specifically, the multidimensional data acquisition unit 24A is configured to, for example, perform processing for acquiring the pixel value of each pixel included in one of the S regions, respectively, for the S regions set in the image GRN. Alternatively, the multidimensional data acquisition unit 24A is configured to, for example, perform processing for calculating a feature vector corresponding to the pixel value of each pixel included in one of the S regions, respectively, for the S regions set in the image GRN.

[0053] The low-dimensional data conversion unit 24B is configured to perform a process for converting the multidimensional data obtained as a result of the processing by the multidimensional data acquisition unit 24A into low-dimensional data. Specifically, the low-dimensional data conversion unit 24B is configured to, for example, respectively perform a process for converting each pixel value obtained from one of the S regions into two-dimensional data for each of the S regions set in the image GRN. Alternatively, the low-dimensional data conversion unit 24B is configured to, for example, respectively perform a process for converting a feature vector calculated in one of the S regions into two-dimensional data for each of the S regions set in the image GRN.

[0054] The low-dimensional data conversion unit 24B is configured to output information in which two-dimensional data acquired in each of the S regions set in the image GRN is associated with the ID number of the subject to the processing result storage unit 25 .

[0055] That is, the dimensional conversion processing unit 24 is configured to perform processing for obtaining multidimensional data corresponding to a plurality of regions set in the image GRN (processing target image), and to perform processing for converting the multidimensional data into low-dimensional data. In this embodiment, a clustering method can also be used in the low-dimensional conversion processing unit 24.

[0056] In addition, in the present embodiment, the dimension conversion processing unit 24 may also be configured, for example, with a neural network configured using an autoencoder. In addition, the dimension conversion processing unit 24 of the present embodiment may also be configured to input the multidimensional data obtained from the image GRN into the above-mentioned neural network, and extract two-dimensional data corresponding to the multidimensional data from the hidden layer of the neural network. In addition, the dimension conversion processing unit 24 of the present embodiment may also be configured, for example, to perform processing for obtaining the likelihood and reconstruction error corresponding to the multidimensional data as two-dimensional data based on the output data obtained by inputting the multidimensional data obtained from the image GRN into the above-mentioned neural network. Unsupervised learning is performed in the dimension conversion processing unit 24.

[0057] The processing result storage unit 25 is configured to store the information output from the inference processing unit 23 and the information output from the dimensional conversion processing unit 24 for each ID number of the subject. That is, in the processing result storage unit 25, the class classification information indicating the classification result obtained by classifying the S regions set in the image GRN into any one of a plurality of predetermined classes, the likelihood information corresponding to the class classification information of each of the S regions, and the two-dimensional data obtained in each of the S regions are stored in a state associated with the ID number of the subject.

[0058] The display processing unit 26 includes an information acquisition unit 26A, a display image generation unit 26B, a display information generation unit 26C, and a display control unit 26D.

[0059] The information acquisition unit 26A is configured to perform a process for detecting an instruction given through the input I / F 3 and acquiring information used in the processes of the display image generation unit 26B, the display information generation unit 26C, and the display control unit 26D based on the detected instruction. Specifically, the information acquisition unit 26A is configured to perform a process for acquiring information indicating the magnification of the main image displayed on the display device 4, information indicating the focus position set in the main image, information indicating the selected position selected in the main image, and information indicating the selected point selected from each plotted point included in the two-dimensional distribution map displayed on the display device 4 together with the main image, based on the detection result of the instruction given through the input I / F 3.

[0060] The display image generation unit 26B is configured to generate a main image corresponding to the magnification and the focus position included in the information obtained by the information acquisition unit 26A using at least one of the P images recorded by the image recording unit 22. That is, the main image generated by the display image generation unit 26B is an image in which the focus position included in the information obtained by the information acquisition unit 26A coincides with the center of the display screen and has the magnification included in the information, and is displayed on the display device 4. In addition, the display image generation unit 26B is configured to generate a sub-image by, for example, reducing the image GRN. In addition, the display image generation unit 26B is configured to generate a display image by overlapping the sub-image in a predetermined area in the main image. In addition, the display image generation unit 26B is configured to determine the display range of the main image displayed on the display device 4 from the image GRN based on the magnification and the focus position included in the information obtained by the information acquisition unit 26A, and overlap a rectangular frame representing the determined display range on the sub-image.

[0061] The display information generating unit 26C is configured to generate display information corresponding to the information read from the processing result storage unit 25. Specifically, the display information generating unit 26C is configured to generate a two-dimensional distribution graph equivalent to a plotted graph obtained by plotting S two-dimensional data included in the information read from the processing result storage unit 25 as coordinates of a plane coordinate system. In addition, the display information generating unit 26C is configured to add different colors or patterns corresponding to the class classification information included in the information read from the processing result storage unit 25 to the S plot points included in the two-dimensional distribution graph generated as described above. In addition, the display information generating unit 26C is configured to generate visual information for visually representing the class classification information and likelihood information equivalent to the inference result included in the information read from the processing result storage unit 25 in the main image generated by the display image generating unit 26B.

[0062] The display control unit 26D is configured to perform processing for displaying the display image generated by the display image generation unit 26B on the display device 4. In addition, when it is detected that the user has issued an instruction in the input I / F 3 to display analysis support information corresponding to the information for supporting the analysis of the main image on the display device 4, the display control unit 26D is configured to perform processing for synthesizing and displaying the analysis support image including the two-dimensional distribution map generated by the display information generation unit 26C and the display image currently displayed on the display device 4. In addition, when it is detected that the input I / F 3 has issued an instruction to display the inference result obtained by the inference processing unit 23 on the display device 4, the display control unit 26D is configured to perform processing for synthesizing and displaying the visual information generated by the display information generation unit 26C and the display image (main image) currently displayed on the display device 4. In addition, the display control unit 26D is configured to perform processing for changing the display mode of the main image and the two-dimensional distribution map currently displayed on the display device 4 according to the selection position included in the information acquired by the information acquisition unit 26A (described later). Furthermore, the display control unit 26D is configured to perform processing (described later) for changing the display form of the main image and the two-dimensional distribution map currently displayed on the display device 4 according to the selection point included in the information acquired by the information acquisition unit 26A.

[0063] That is, the display processing unit 26 is configured to perform processing for displaying a display image (main image) including one or more regions among the S regions set in the image GRN, and a plotting diagram (two-dimensional distribution diagram) having a plurality of plotting points generated by adding different colors or patterns corresponding to the classification results obtained by the inference processing unit 23 to the low-dimensional data (two-dimensional data) in each of the S regions, on the display device 4. In addition, the display processing unit 26 is configured to perform processing for generating a two-dimensional distribution diagram of a plurality of plotting points generated by adding colors or patterns corresponding to the classification results obtained by the inference processing unit 23 to the two-dimensional data in each of the S regions set in the image GRN. In addition, the display processing unit 26 is configured to perform processing for synthesizing visual information corresponding to the classification results and likelihood obtained by the inference processing unit 23 with the display image (main image) and displaying it on the display device 4.

[0064] In addition, in the present embodiment, the functions of each part of the image processing device 2 may also be implemented by hardware including a processor 201 and a storage medium 202. Alternatively, in the present embodiment, for example, a program for implementing the functions of each part of the image processing device 2 may be stored in the storage medium 202, and the processor 201 reads the program and executes it. Examples of computer-readable storage media 202 include optical disks such as CD-ROMs, phase-change optical disks such as DVD-ROMs, optical magnetic disks such as MO (Magnet Optical) or MD (Mini Disk), magnetic disks such as floppy disks (registered trademark) and removable hard disks, and memory cards such as CF cards (registered trademark), SM cards, SD memory cards, or memory sticks. In addition, as storage media, hardware devices such as integrated circuits (IC chips, etc.) specially designed and constructed for the purpose of the present invention are also included.

[0065] Next, refer to Figure 3A as well as Figure 3B The effects of this embodiment are described below. Figure 3A This is a flowchart showing a part of a specific example of processing performed in the image processing device according to the first embodiment. Figure 3B Yes means Figure 3A Flow chart of the subsequent processing of .

[0066] In addition, in the present embodiment, the following case is described as an example: an image obtained by photographing a semiconductor wafer and a plurality of chips formed on the semiconductor wafer as a subject is output from the imaging device 1 to the image processing device 2, and P images corresponding to the image are generated by the pre-processing unit 21. In addition, in the present embodiment, it is assumed that the following case is described: the inference processing unit 23 performs processing for obtaining class classification information, and the class classification information indicates the classification result of the S regions set in the image GRN as either of the two classes of "normal" and "abnormal". In addition, in the present embodiment, it is assumed that the following case is described: the inference processing unit 23 performs processing for obtaining likelihood information, and the likelihood information indicates the calculation result of the likelihood of the classification result in each region classified as the "normal" class (hereinafter also referred to as the normal region), and the calculation result of the likelihood of the classification result in each region classified as the "abnormal" class (hereinafter also referred to as the abnormal region).

[0067] The display image generating unit 26B performs processing for generating a display image having a main image and a sub-image using at least one of the P images recorded by the image recording unit 22 based on the magnification and the focus position included in the information acquired by the information acquiring unit 26A ( Figure 3A Step S1).

[0068] Specifically, the display image generation unit 26B performs processing for generating a display image using the image GRN read from the image recording unit 22 in a default state corresponding to a state in which no instructions regarding the setting of the magnification and the focus position are given. Then, through such processing, for example, Figure 4 The display image DGA including the main image MGA and the sub-image SGA as shown is displayed on the display device 4 . Figure 4 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0069] The main image MGA corresponds to an image in which the entire area of ​​the image GRN is set as the display range. In addition, the sub-image SGA corresponds to an image obtained by reducing the image GRN. Figure 4 In the display image DGA, a rectangular frame FGA indicating that the entire area of ​​the image GRN is the display range of the main image MGA is superimposed on the outermost portion of the sub-image SGA.

[0070] The display image generation unit 26B performs the following determination processing: determining whether the magnification ratio or the focus position included in the information acquired by the information acquisition unit 26A is Figure 3A Whether the status of step S1 has been changed ( Figure 3A Step S2).

[0071] The display image generating unit 26B obtains the magnification ratio or the position of interest relative to the information obtained by the information obtaining unit 26A. Figure 3A If the result of determination in step S1 is that the state of step S1 has been changed (S2: Yes), a process for generating a display image corresponding to the changed magnification and focus position is performed ( Figure 3A Step S3) In addition, the display image generating unit 26B obtains the magnification ratio and the focus position relative to the information obtained by the information obtaining unit 26A. Figure 3A If the status of step S1 has not changed (S2: No), skip Figure 3A The processing of step S3.

[0072] Here, in the Figure 3A In the case of the processing of step S3, for example, Figure 5 The display image DGB including the main image MGB and the sub-image SGA as shown is displayed on the display device 4 . Figure 5 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0073] The main image MGB is equivalent to an image obtained by enlarging the main image MGA according to the magnification ratio included in the information acquired by the information acquisition unit 26A while setting the vicinity of the central portion of the main image MGA as the focus position. In addition, in the display image DGB, a rectangular frame FGB indicating that the vicinity of the central portion of the image GRN is the display range of the main image MGB is overlapped inside the sub-image SGA.

[0074] The display control unit 26D is Figure 3A After the processing of step S2 or step S3, a determination process is performed as to whether an instruction for displaying the inference result obtained by the inference processing unit 23 on the display device 4 has been given in the input I / F 3 ( Figure 3A Step S4).

[0075] The display control unit 26D performs processing for synthesizing and displaying the visual information generated by the display information generating unit 26C, that is, the visual information corresponding to the class classification information and likelihood information stored in the processing result storing unit 25, with the display image currently displayed by the display device 4, when a judgment result (S4: Yes) is obtained that an instruction for displaying the inference result obtained by the inference processing unit 23 on the display device 4 is performed (S4: Yes). Figure 3A In addition, if the display control unit 26D obtains a determination result that no instruction for displaying the inference result obtained by the inference processing unit 23 on the display device 4 is given (S4: No), the display control unit 26D skips the step S5. Figure 3A The processing of step S5 is performed instead of the processing of step S6.

[0076] Here, when the display image DGB is displayed on the display device 4, Figure 3A In the case of the processing of step S5, for example, Figure 6 The display image DGC including the main image MGC and the sub-image SGA as shown is displayed on the display device 4 . Figure 6 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0077] In the main image MGC, grid lines GLA indicating the setting state of the S areas set in the image GRN are drawn with dotted lines.

[0078] The main image MGC shows, among the inference results obtained in each of the S regions, inference results corresponding to the four regions divided by the grid line GLA.

[0079] Specifically, the main image MGC shows that, for example, one region AX of the four regions divided by the grid lines GLA where a hatching pattern overlaps is an abnormal region, and three regions other than the region AX where the hatching pattern is not overlapped are normal regions.

[0080] In the present embodiment, for example, the density of the hatching pattern of the region AX superimposed on the main image MGC may be changed according to the magnitude of the likelihood indicated by the likelihood information read from the processing result storage unit 25 .

[0081] In the present embodiment, for example, the normal region and the abnormal region included in the main image MGC may be colored with different colors based on the class classification information read from the processing result storage unit 25. In the present embodiment, for example, the concentration of the color in which the normal region of the main image MGC is colored and the concentration of the color in which the abnormal region of the main image MGC is colored may be changed respectively based on the magnitude of the likelihood indicated by the likelihood information read from the processing result storage unit 25.

[0082] According to the present embodiment, for example, an instruction for displaying only the inference result corresponding to the normal area in the main image may be made in the input I / F 3. In addition, according to the present embodiment, for example, an instruction for displaying only the inference result corresponding to the abnormal area in the main image may be made in the input I / F 3.

[0083] According to this embodiment, for example, Figure 3A In step S5, processing corresponding to the result of the determination obtained by threshold determination of the likelihood indicated by the likelihood information read from the processing result storage unit 25 may be performed. Specifically, according to the present embodiment, for example, the display control unit 26D may perform processing for synthesizing visual information only on the regions among the S regions set in the main image, where the likelihood indicated by the likelihood information read from the processing result storage unit 25 is lower than the prescribed threshold. Alternatively, according to the present embodiment, for example, the display control unit 26D may perform processing for synthesizing visual information only on the regions among the S regions set in the main image, where the likelihood indicated by the likelihood information read from the processing result storage unit 25 is higher than the prescribed threshold.

[0084] The display control unit 26D performs Figure 3A After the processing of step S4 or step S5, a determination process is performed as to whether an instruction for displaying the analysis support information on the display device 4 has been given in the input I / F 3 ( Figure 3A Step S6).

[0085] The display control unit 26D performs processing for synthesizing and displaying the analysis support image including the two-dimensional distribution map generated by the display information generating unit 26C and the display image currently being displayed on the display device 4, when the determination result (S6: Yes) is obtained that the instruction for displaying the analysis support information on the display device 4 has been issued, ( Figure 3A In addition, if the display control unit 26D obtains a determination result that no instruction for displaying the analysis support information on the display device 4 has been issued (S6: No), the display control unit 26D skips the step Figure 3A The series of processing is completed by performing the processing after step S7.

[0086] Here, when the display image DGC is displayed on the display device 4, Figure 3A In the case of the processing of step S7, for example, Figure 7 The display device 4 displays the display image DGD in which the analysis support image AGA is superimposed on the display image DGC as shown. Figure 7 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment.

[0087] The analysis support image AGA includes a two-dimensional distribution map SPA having S plotted points. In the two-dimensional distribution map SPA, each plotted point corresponding to a normal region is drawn as a hollow point, and each plotted point corresponding to an abnormal region is drawn as a black point.

[0088] Furthermore, according to the present embodiment, the normal region included in the main image MGC and the plotted points corresponding to the normal region included in the two-dimensional distribution map SPA can be colored with the same color system. Furthermore, according to the present embodiment, the abnormal region included in the main image MGC and the plotted points corresponding to the abnormal region included in the two-dimensional distribution map SPA can also be colored with the same color system.

[0089] The display control unit 26D is Figure 3A After the processing of step S7, a determination process is performed based on the information acquired by the information acquisition unit 26A as to whether an instruction for selecting a desired position in the main image as a selected position has been given in the input I / F 3 ( Figure 3B Step S8).

[0090] If the display control unit 26D determines that no instruction has been given to select the desired position in the main image as the selected position (S8: No), it continues to perform the following steps. Figure 3BFurthermore, when the display control unit 26D obtains a determination result that an instruction for selecting a desired position in the main image as a selected position has been performed (S8: Yes), the display control unit 26D performs processing related to identification display for visually identifying a selected area in the main image including the selected position and plotted points corresponding to the selected area included in the two-dimensional distribution map displayed together with the main image ( Figure 3B After step S9), a series of processing is completed.

[0091] Specifically, for example, in Figure 7 When an arbitrary position in the area AX in the main image MGC is selected as the selected position, processing related to identification display is performed for visually identifying the area AX and the plotted points PX corresponding to the area AX included in the two-dimensional distribution map SPA displayed together with the main image MGC. Then, according to such processing, for example, Figure 8 as well as Fig. 9 As shown in FIG. 1 , a frame is added around the area AX in the main image MGD displayed in the display image DGE and around the plot point PX in the two-dimensional distribution diagram SPB included in the analysis support image AGB displayed in the display image DGE. Fig. 9 In the two-dimensional distribution map SPB of Figure 7 Same shading pattern. Figure 8 This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the first embodiment. Fig. 9 It is used to illustrate Figure 8 A diagram of a two-dimensional distribution map included in the analysis support image of the display image.

[0092] The display control unit 26D performs the following determination processing: based on the information acquired by the information acquisition unit 26A, it is determined whether an instruction for selecting a desired plot point as a selected point from among the plot points included in the two-dimensional distribution diagram displayed together with the main image has been issued in the input I / F 3 ( Figure 3B Step S10).

[0093] The display control unit 26D, when the result of determination that the instruction for selecting a desired plot point from among the plot points included in the two-dimensional distribution graph displayed together with the main image as the selected point is not performed (S10: No), ends the series of processing. In addition, when the result of determination that the instruction for selecting a desired plot point from among the plot points included in the two-dimensional distribution graph displayed together with the main image as the selected point is performed (S10: Yes), the display control unit 26D performs processing related to identification display for visually identifying the selected point and the area corresponding to the selected point in the main image (S10: Yes). Figure 3B After step S11), a series of processing is completed.

[0094] Specifically, for example, in Figure 7 When a plot point PX in the two-dimensional distribution graph SPA is selected as a selected point, processing related to identification display for visually identifying the plot point PX and the area AX corresponding to the plot point PX in the main image MGC is performed. Then, according to such processing, for example, Figure 8 as well as Fig. 9 As shown, frames are added around the plot point PX in the two-dimensional distribution map SPB included in the analysis support image AGB displayed in the display image DGE and around the area AX in the main image MGD displayed in the display image DGE.

[0095] In addition, the display control unit 26D of this embodiment may also be Figure 3B In step S9 and / or step S11, for example, a process is performed to reduce the brightness of each area other than area AX in the main image MGC. In addition, the display control unit 26D of this embodiment may be, for example, Figure 3B In step S9, processing is performed to enlarge and display the area AX in the main image MGC.

[0096] As described above, according to the present embodiment, for example, a portion where the likelihood of a classification result obtained by classifying a plurality of regions set in an image into any one of a plurality of predetermined classes is low can be found in the two-dimensional distribution map, and the plotted points corresponding to the portion in the two-dimensional distribution map and the plotted points in the two-dimensional distribution map can be made to correspond to the portions in the two-dimensional distribution map. Figure 1 The area corresponding to the part in the main image displayed can be easily visually recognized. Therefore, according to this embodiment, the work efficiency in the work of specifying the part that needs to be carefully examined in the image can be improved.

[0097] In addition, according to the present embodiment, for example, when an instruction for correcting a classification result displayed as a plurality of plotted points in a two-dimensional distribution diagram or visual information in a main image is given in the input I / F3, class classification information corresponding to the corrected classification result corrected according to the instruction is stored in the processing result storage unit 25.

[0098] In addition, according to the present embodiment, for example, the low-dimensional data conversion unit 24B may perform processing for converting the multidimensional data acquired by the multidimensional data acquisition unit 24A into one-dimensional data, and the display information generation unit 26C may perform processing for generating a drawing graph in which the one-dimensional data is plotted as coordinates on a straight line.

[0099] In addition, according to the present embodiment, for example, the low-dimensional data conversion unit 24B may perform processing for converting the multidimensional data acquired by the multidimensional data acquisition unit 24A into three-dimensional data, and the display information generation unit 26C may perform processing for generating a drawing in which the three-dimensional data is plotted as coordinates of a spatial coordinate system.

[0100] Furthermore, according to the present embodiment, for example, the display information generating unit 26C may be configured to perform processing for generating a plotting diagram different from the two-dimensional distribution diagram using S two-dimensional data included in the information read from the processing result storage unit 25 .

[0101] (Second Embodiment)

[0102] Figures 10 to 15B This relates to the second embodiment.

[0103] In the present embodiment, detailed description of portions having the same configuration as that of the first embodiment is appropriately omitted, and portions having a different configuration from that of the first embodiment are mainly described.

[0104] The image processing system 101 of this embodiment is to replace Figure 2 The image processing device 2 shown has Fig.10 The image processing device 2A shown in the figure is constructed. Fig.10 It is a diagram showing an example of the configuration of an image processing device according to the second embodiment.

[0105] The image processing device 2A is configured as, for example, a computer (not shown) having one or more processors 201 and a storage medium 202. In addition, the image processing device 2A is configured to generate a display image by processing the image output from the camera device 1, and output the generated display image to the display device 4. In addition, the image processing device 2A is configured to detect an instruction output from the input I / F 3, generate a display image corresponding to the detected instruction, and output it to the display device 4.

[0106] For example, Fig.10 As shown, the image processing device 2A includes a pre-processing unit 21 , an image recording unit 22 , an inference processing unit 33 , a dimensionality conversion processing unit 34 , and a display processing unit 36 ​​.

[0107] The inference processing unit 33 is configured to set T (T≧2) regions for the main image output from the display processing unit 36 ​​.

[0108] In addition, in the present embodiment, the T value corresponding to the number of regions to be processed by the inference processing unit 33 and the dimension conversion processing unit 34 may be set to a value sufficiently smaller than the S value corresponding to the number of regions to be processed by the inference processing unit 23 and the dimension conversion processing unit 24 of the first embodiment. In addition, in the present embodiment, the T regions set by the processing of the inference processing unit 33 and the dimension conversion processing unit 34 are explained as rectangular regions. In addition, in the present embodiment, the setting methods of the T regions by the inference processing unit 33 and the dimension conversion processing unit 34 are explained as being the same.

[0109] The inference processing unit 33 is configured to perform processing for obtaining class classification information corresponding to information indicating a classification result of being classified into any one of a plurality of predetermined classes, and processing for obtaining class classification information corresponding to information indicating a calculation result of the likelihood of the classification result, respectively, for T regions set in the main image output from the display processing unit 36. In addition, the inference processing unit 33 is configured to output the class classification information and likelihood information obtained by the above processing to the display processing unit 36. Specifically, the inference processing unit 33 is configured, for example, as a statistical model formed by machine learning, using a multilayer neural network similar to the inference processing unit 23 of the first embodiment.

[0110] That is, the inference processing unit 33 is configured to perform the following processing: a process for obtaining a classification result in which a plurality of regions set in a main image (processing object image) output from the display processing unit 36 ​​are respectively classified into any one of a plurality of prescribed classes; and a process for calculating the likelihood of the classification result in each of the plurality of regions.

[0111] The dimensionality conversion processing unit 34 includes a multi-dimensional data acquisition unit 34A and a low-dimensional data conversion unit 34B.

[0112] The multi-dimensional data acquisition unit 34A is configured to set T regions for the main image output from the display processing unit 36 ​​.

[0113] The multidimensional data acquisition unit 34A is configured to perform processing for acquiring multidimensional data corresponding to each of the T regions set in the main image output from the display processing unit 36. Specifically, the multidimensional data acquisition unit 34A is configured to perform processing such as, for example, a process for acquiring the pixel value of each pixel included in one of the T regions set in the main image output from the display processing unit 36. Alternatively, the multidimensional data acquisition unit 34A is configured to perform processing such as, for example, a process for calculating a feature vector corresponding to the pixel value of each pixel included in one of the T regions set in the main image output from the display processing unit 36.

[0114] The low-dimensional data conversion unit 34B is configured to perform a process for converting the multidimensional data obtained as a result of the process of the multidimensional data acquisition unit 34A into low-dimensional data. Specifically, the low-dimensional data conversion unit 34B is configured to, for example, perform a process of converting each pixel value obtained from one of the T regions set in the main image output from the display processing unit 36 ​​into two-dimensional data. Alternatively, the low-dimensional data conversion unit 34B is configured to, for example, perform a process of converting a feature vector calculated in one of the T regions into two-dimensional data for each of the T regions set in the main image output from the display processing unit 36.

[0115] The low-dimensional data conversion unit 34B is configured to output the two-dimensional data obtained by the above-mentioned processing to the display processing unit 36 ​​.

[0116] That is, the dimensional conversion processing unit 34 is configured to perform processing for acquiring multidimensional data corresponding to a plurality of regions set in the main image (processing target image) output from the display processing unit 36 ​​and to perform processing for converting the multidimensional data into low-dimensional data.

[0117] The display processing unit 36 ​​includes an information acquisition unit 36A, a display image generation unit 36B, a display information generation unit 36C, and a display control unit 36D.

[0118] The information acquisition unit 36A is configured to perform a process for detecting an instruction given through the input I / F 3 and acquiring information used in the processes of the display image generation unit 36B, the display information generation unit 36C, and the display control unit 36D based on the detected instruction. Specifically, the information acquisition unit 36A is configured to perform a process for acquiring information indicating the magnification of the main image displayed on the display device 4, information indicating the focus position set in the main image, information indicating the selected position selected in the main image, and information indicating the selected point selected from each plotted point included in the two-dimensional distribution map displayed on the display device 4 together with the main image, based on the detection result of the instruction given through the input I / F 3.

[0119] The display image generation unit 36B is configured to generate a main image corresponding to the magnification and the focus position included in the information obtained by the information acquisition unit 36A using at least one of the P images recorded by the image recording unit 22. That is, the main image generated by the display image generation unit 36B is an image in which the focus position included in the information obtained by the information acquisition unit 36A coincides with the center of the display screen and is displayed on the display device 4 as an image with the magnification included in the information. In addition, the display image generation unit 36B is configured to output the main image generated as described above to the inference processing unit 33 and the dimensional conversion processing unit 34. In addition, the display image generation unit 36B is configured to generate a sub-image by, for example, reducing the image GRN. In addition, the display image generation unit 36B is configured to generate a display image by overlapping a sub-image in a specified area within the main image. Furthermore, the display image generating unit 26B is configured to determine the display range of the main image displayed on the display device 4 from the image GRN based on the magnification and the focus position included in the information acquired by the information acquiring unit 36A, and to superimpose a rectangular frame representing the determined display range on the sub-image.

[0120] The display information generating unit 36C is configured to generate display information corresponding to the inference result output from the inference processing unit 33 and the two-dimensional data output from the dimension conversion processing unit 34. Specifically, the display information generating unit 36C is configured to generate a two-dimensional distribution graph in which T two-dimensional data output from the dimension conversion processing unit 34 are plotted as coordinates of a plane coordinate system. In addition, the display information generating unit 36C is configured to add different colors or patterns corresponding to the class classification information output from the inference processing unit 33 to each of the T plotted points included in the two-dimensional distribution graph generated as described above. In addition, the display information generating unit 36C is configured to generate visual information for visually displaying the class classification information and likelihood information corresponding to the inference result output from the inference processing unit 33 in the main image generated by the display image generating unit 36B.

[0121] The display control unit 36D is configured to perform processing for displaying the display image generated by the display image generation unit 36B on the display device 4. In addition, the display control unit 36D is configured to perform processing for synthesizing and displaying the analysis support image including the two-dimensional distribution map generated by the display information generation unit 36C and the display image currently displayed on the display device 4 when it is detected that the user has issued an instruction to display the analysis support information corresponding to the information for supporting the analysis of the main image on the display device 4 through the input I / F 3. In addition, the display control unit 36D is configured to perform processing for synthesizing and displaying the visual information generated by the display information generation unit 36C and the display image (main image) currently displayed on the display device 4 when it is detected that the inference result obtained by the inference processing unit 33 has been issued through the input I / F 3. In addition, the display control unit 36D is configured to perform processing for changing the display mode of the main image and the two-dimensional distribution map currently displayed on the display device 4 according to the selection position included in the information acquired by the information acquisition unit 36A (described later). Furthermore, the display control unit 36D is configured to perform a process (described later) of changing the display form of the main image and the two-dimensional distribution map being displayed on the display device 4 according to the selection point included in the information acquired by the information acquisition unit 36A.

[0122] That is, the display processing unit 36 ​​is configured to perform processing for displaying the display image (main image) including the T regions set in the main image and the plotting diagram (two-dimensional distribution diagram) having a plurality of plotting points generated by adding different colors or patterns corresponding to the classification results obtained by the inference processing unit 33 to the low-dimensional data (two-dimensional data) in each of the T regions on the display device 4. In addition, the display processing unit 36 ​​is configured to perform processing for generating a two-dimensional distribution diagram having a plurality of plotting points generated by adding colors or patterns corresponding to the classification results obtained by the inference processing unit 33 to the two-dimensional data in each of the T regions set in the main image. In addition, the display processing unit 36 ​​is configured to perform processing for synthesizing the visual information corresponding to the classification results and likelihood obtained by the inference processing unit 33 with the display image (main image) and displaying them on the display device 4.

[0123] In addition, in the present embodiment, the functions of each part of the image processing device 2A can be implemented by hardware including a processor 201 and a storage medium 202. Alternatively, in the present embodiment, for example, a program for implementing the functions of each part of the image processing device 2A is stored in the storage medium 202, and the processor 201 reads the program and executes it. As a computer-readable storage medium 202, there can be listed optical disks such as CD-ROM, phase change optical disks such as DVD-ROM, optical magnetic disks such as MO (Magnet Optical) and MD (Mini Disk), magnetic disks such as floppy disks (registered trademark) and removable hard disks, and memory cards such as CF cards (registered trademark), SM cards, SD memory cards, and memory sticks. In addition, as a storage medium, hardware devices such as integrated circuits (IC chips, etc.) specially designed and constructed for the purpose of the present invention are also included.

[0124] Next, refer to Fig.11A as well as Fig. 11B The effects of this embodiment are described below. Fig.11A This is a flowchart showing a part of a specific example of processing performed in the image processing device according to the second embodiment. Fig. 11B Yes means Fig.11A Flow chart of the subsequent processing of .

[0125] The display image generating unit 36B performs processing for generating a display image having a main image and a sub-image using at least one of the P images recorded by the image recording unit 22 based on the magnification and the focus position included in the information acquired by the information acquiring unit 36A ( Fig.11A In addition, the display image generation unit 36B will Fig.11A The main image included in the display image generated in step S21 is output to the inference processing unit 33 and the dimensionality conversion processing unit 34.

[0126] Specifically, the display image generating unit 36B generates, for example, in a default state corresponding to a state in which no instruction regarding the setting of the magnification ratio and the focus position is given. Figure 4 The display image DGA includes the main image MGA and the sub-image SGA as shown. The display image generation unit 36B outputs the main image MGA included in the display image DGA to the inference processing unit 33 and the dimensionality conversion processing unit 34 .

[0127] The inference processing unit 33 Fig.11AIn step S21, T regions are set from the main image output by the display processing unit 36. In addition, the inference processing unit 33 performs the following processing: processing for obtaining class classification information and likelihood information as inference results, wherein the class classification information indicates the result of classifying the T regions set in the main image into any one of a plurality of predetermined classes, and the likelihood information indicates the calculation result of the likelihood of the classification result in each of the T regions ( Fig.11A In addition, the inference processing unit 33 will Fig.11A The inference result (class classification information and likelihood information) obtained in step S22 is output to the display processing unit 36.

[0128] In addition, in the present embodiment, the description is made assuming that the inference processing unit 33 performs processing for obtaining class classification information indicating the classification results of the T regions set in the main image output from the display processing unit 36 ​​as either of the two classes of "normal" and "abnormal". In addition, in the present embodiment, the description is made assuming that the inference processing unit 33 performs processing for obtaining likelihood information indicating the calculation results of the likelihood of the classification results in each normal region and the calculation results of the likelihood of the classification results in each abnormal region.

[0129] The dimension conversion processing unit 34 (multi-dimensional data acquisition unit 34A) Fig.11A In step S21, T regions are set from the main image output by the display processing unit 36. In addition, the dimensional conversion processing unit 34 (multidimensional data acquisition unit 34A and low-dimensional data conversion unit 34B) performs processing for acquiring multidimensional data corresponding to the T regions set in the above-mentioned main image, and performs processing for converting the multidimensional data to acquire two-dimensional data ( Fig.11A Step S23). In addition, the dimension conversion processing unit 34 (low-dimensional data conversion unit 34B) will Fig.11A The two-dimensional data obtained in step S23 is output to the display processing unit 36.

[0130] The display image generation unit 36B performs the following determination processing: the magnification ratio or the focus position included in the information acquired by the information acquisition unit 36A is relative to the Fig.11A Whether the state of step S21 has changed ( Fig.11A Step S24).

[0131] The display image generating unit 36B obtains the magnification ratio or the focus position relative to the information obtained by the information obtaining unit 36A. Fig.11A If the state of step S21 is changed (S24: Yes), the system returns to Fig.11A The processing is performed in step S21, thereby generating a display image corresponding to the changed magnification and focus position, and outputting the main image included in the generated display image to the inference processing unit 33 and the dimensional conversion processing unit 34.

[0132] Specifically, the display image generating unit 36B is, for example, Figure 4 When the magnification or the focus position is changed while the display image DGA is being displayed on the display device 4, the generated Figure 5 The display image DGB includes the main image MGB and the sub-image SGA as shown. The display image generation unit 36B outputs the main image MGB included in the display image DGB to the inference processing unit 33 and the dimensionality conversion processing unit 34.

[0133] According to the above-described processing, each time the magnification or the focus position in the main image is changed, the inference processing unit 33 performs: a process of setting T regions in the main image; a process of obtaining a classification result obtained by classifying the T regions into any one of a plurality of predetermined classes; and a process of calculating the likelihood of the classification result in each of the T regions. In addition, according to the above-described processing, each time the magnification or the focus position in the main image is changed, the dimension conversion processing unit 34 performs: a process of setting T regions in the main image; a process of obtaining multidimensional data corresponding to each of the T regions; and a process of converting the multidimensional data to obtain two-dimensional data.

[0134] The display control unit 36D detects that the magnification ratio and the focus position relative to the image are included in the information acquired by the information acquisition unit 36A. Fig.11A If the state of step S21 is not changed (S24: No), a determination process is performed as to whether an instruction for displaying the inference result obtained by the inference processing unit 33 on the display device 4 has been issued in the input I / F 3 ( Fig.11A Step S25).

[0135] The display control unit 36D performs processing for synthesizing and displaying the visual information generated by the display information generating unit 36C and the display image currently being displayed on the display device 4, when a determination result (S25: Yes) is obtained that an instruction for displaying the inference result obtained by the inference processing unit 33 on the display device 4 has been issued (S25: Yes). Fig.11A In addition, if the display control unit 36D obtains a determination result indicating that no instruction has been given to display the inference result obtained by the inference processing unit 33 on the display device 4 (S25: No), the display control unit 36D skips the step S26. Fig.11A The processing of step S26 is replaced by the processing of step S27.

[0136] Here, when Figure 4 The display image DGA is displayed on the display device 4. Fig.11A In the case of the processing of step S26, for example, Fig. 12A The display image DGK including the main image MGK and the sub-image SGA as shown is displayed on the display device 4 . Fig. 12A This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0137] In the main image MGK, grid lines GLK indicating the setting states of T areas are drawn with dotted lines.

[0138] In the main image MGK, inference results corresponding to T regions are shown.

[0139] Specifically, the main image MGK shows, for example, that among the 16 regions divided by the grid lines GLK, two regions including the region AY superimposed with a hatching pattern are abnormal regions, and 14 regions not superimposed with the hatching pattern are normal regions.

[0140] In addition, when Figure 5 The display image DGB is displayed on the display device 4. Fig.11A In the case of the processing of step S26, for example, Fig. 12B The display image DGP including the main image MGP and the sub-image SGA as shown is displayed on the display device 4 . Fig. 12B This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0141] In the main image MGP, grid lines GLP indicating the setting states of T areas are drawn with dotted lines.

[0142] In the main image MGP, inference results corresponding to T regions are shown.

[0143] Specifically, the main image MGP shows, for example, that among the 16 areas divided by the grid lines GLP, three areas including the area AZ overlapped with a hatching pattern are abnormal areas, and 13 areas not overlapped with the hatching pattern are normal areas.

[0144] The display control unit 36D performs Fig.11A After the processing of step S25 or step S26, a determination process is performed as to whether an instruction for displaying the analysis support information on the display device 4 has been given in the input I / F 3 ( Fig.11A Step S27).

[0145] The display control unit 36D performs processing for synthesizing and displaying the analysis support image including the two-dimensional distribution map generated by the display information generating unit 26C and the display image currently being displayed on the display device 4, when the determination result is obtained that the instruction for displaying the analysis support information on the display device 4 has been issued (S27: Yes). Fig.11A In addition, if the display control unit 36D obtains a determination result that no instruction for displaying the analysis support information on the display device 4 has been issued (S27: No), the display control unit 36D skips the step Fig.11A The series of processing is completed by performing the processing after step S28.

[0146] Here, when Fig. 12A The display image DGK is displayed on the display device 4. Fig.11A In the case of the processing of step S28, for example, Fig.13A The display device 4 displays the display image DGL in which the analysis support image AGK is superimposed on the display image DGK as shown. Fig.13A This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0147] The analysis support image AGK includes a two-dimensional distribution map SPK having T plotted points. In the two-dimensional distribution map SPK, each plotted point corresponding to a normal region is plotted as a hollow point, and each plotted point corresponding to an abnormal region is plotted as a blackened point.

[0148] In addition, when Fig. 12B The display image DGP is displayed on the display device 4. Fig.11A In the case of the processing of step S28, for example, Fig. 13B The display device 4 displays the display image DGQ in which the analysis support image AGP is superimposed on the display image DGP as shown. Fig. 13B This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment.

[0149] The analysis support image AGP includes a two-dimensional distribution map SPP having T plotted points. In the two-dimensional distribution map SPP, each plotted point corresponding to a normal region is plotted as a hollow point, and each plotted point corresponding to an abnormal region is plotted as a blackened point.

[0150] That is, according to the above-described processing, the display content of the inference result displayed on the main image and the display content of the two-dimensional distribution map displayed together with the main image can be dynamically changed according to the magnification and the focus position set in the main image.

[0151] The display control unit 36D performs Fig.11A After the processing of step S28, a determination process is performed based on the information acquired by the information acquisition unit 36A as to whether an instruction for selecting a desired position in the main image as a selected position has been given in the input I / F 3 ( Fig. 11B Step S29).

[0152] If the display control unit 36D determines that no instruction has been given to select the desired position in the main image as the selected position (S29: No), it continues to perform the following steps. Fig. 11B In addition, when the display control unit 36D obtains a judgment result that an instruction for selecting a desired position in the main image as a selected position has been performed (S29: Yes), the display control unit 36D performs processing related to identification display for visually identifying a selected area including the selected position in the main image and plotted points corresponding to the selected area included in the two-dimensional distribution map displayed together with the main image ( Fig. 11B After step S30), a series of processing is completed.

[0153] Specifically, for example, in Fig.13A When an arbitrary position in the area AY in the main image MGK is selected as a selected position, processing related to identification display is performed for visually identifying the area AY and the plotted points PY corresponding to the area AY included in the two-dimensional distribution map SPK displayed together with the main image MGK. Then, according to such processing, for example, Fig.14A as well as Fig.15A As shown in FIG. 1 , a frame is added around the area AY in the main image MGL displayed by the display image DGM and around the plot point PY in the two-dimensional distribution map SPL included in the analysis support image AGL displayed by the display image DGM. Fig.15A In the two-dimensional distribution map SPL of Fig.13A Same shading pattern. Fig.14A This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment. Fig.15A It is used to illustrate Fig.14A A diagram of a two-dimensional distribution map included in the analysis support image of the display image.

[0154] In addition, for example, in Fig. 13BWhen an arbitrary position in the area AZ in the main image MGP is selected as a selected position, processing related to identification display for visually identifying the area AZ and the plotted points PZ corresponding to the area AZ included in the two-dimensional distribution map SPP displayed together with the main image MGP is performed. Then, according to such processing, for example, Fig. 14B as well as Fig. 15B As shown in FIG. 1 , a frame is added around the area AZ in the main image MGQ displayed by the display image DGR and around the plot point PZ in the two-dimensional distribution graph SPQ included in the analysis support image AGQ displayed by the display image DGR. Fig. 15B In the two-dimensional distribution map SPQ of the abnormal area, for the convenience of illustration, each plotted point corresponding to the abnormal area is assigned Fig. 13B Same shading pattern. Fig. 14B This is a diagram showing an example of a display image displayed according to the processing performed by the image processing device according to the second embodiment. Fig. 15B It is used to illustrate Fig. 14B A diagram of a two-dimensional distribution map included in the analysis support image of the display image.

[0155] The display control unit 36D performs a determination process based on the information acquired by the information acquisition unit 36A to determine whether an instruction for selecting a desired plot point as a selected point from among the plot points included in the two-dimensional distribution diagram displayed together with the main image has been issued at the input I / F 3. Fig. 11B Step S31).

[0156] The display control unit 36D, when the result of determination that the instruction for selecting a desired plot point from among the plot points included in the two-dimensional distribution diagram displayed together with the main image as the selected point is not performed (S31: No), ends the series of processing. In addition, when the result of determination that the instruction for selecting a desired plot point from among the plot points included in the two-dimensional distribution diagram displayed together with the main image as the selected point is performed (S31: Yes), the display control unit 36D performs processing related to identification display for visually identifying the selected point and the area corresponding to the selected point in the main image (S31: Yes). Fig. 11B After step S32), a series of processing is completed.

[0157] Specifically, for example, in Fig.13A When a plot point PY in the two-dimensional distribution graph SPK is selected as a selected point, processing related to identification display for making the plot point PY and the area AY corresponding to the plot point PY in the main image MGK visually identifiable is performed. Then, according to such processing, for example, Fig.14A as well as Fig.15AAs shown, a frame is added around the plot point PY in the two-dimensional distribution map SPL included in the analysis support image AGL displayed by the display image DGM and around the area AY in the main image MGL displayed by the display image DGM.

[0158] In addition, for example, in Fig. 13B When a plot point PZ in the two-dimensional distribution map SPP is selected as a selected point, processing related to identification display for making the plot point PZ and the area AZ corresponding to the plot point PZ in the main image MGQ visually identifiable is performed. Then, according to such processing, for example, Fig. 14B as well as Fig. 15B As shown, frames are added around the plot point PZ in the two-dimensional distribution map SPQ included in the analysis support image AGQ displayed by the display image DGR and around the area AZ in the main image MGQ displayed by the display image DGR.

[0159] In addition, in the present embodiment, for example, the inference processing unit 33 and the dimensionality conversion processing unit 34 may additionally set U areas having the same shape and size as the T areas set in the main image output from the display control unit 36D around the T areas.

[0160] Then, according to the above-mentioned configuration, for example, Fig.11A In step S22, the inference processing unit 33 performs the following processing: obtaining the classification information indicating the classification result of each of the (T+U) regions into any one of the predetermined multiple classes, and the likelihood information indicating the calculation result of the likelihood of the classification result in each of the (T+U) regions. In addition, according to the above-mentioned configuration, for example, in Fig.11A In step S23, the dimension conversion processing unit 34 performs the following processing: to obtain multidimensional data corresponding to each of the (T+U) regions, and to convert the multidimensional data to obtain two-dimensional data. Therefore, according to the above-mentioned configuration, for example, the frequency of processing by the inference processing unit 33 and the dimension conversion processing unit 34 in accordance with the change of the focus position in the main image can be reduced.

[0161] As described above, according to the present embodiment, for example, a portion where the likelihood of a classification result obtained by classifying a plurality of regions set in an image into any one of a plurality of predetermined classes is low can be found in the two-dimensional distribution map, and a plot point corresponding to the portion in the two-dimensional distribution map and a plot point corresponding to the two-dimensional distribution map can be made to be Figure 1 The area corresponding to the part in the main image displayed can be easily visually recognized. Therefore, according to this embodiment, the work efficiency in the work of confirming the part that needs to be carefully examined in the image can be improved.

[0162] In addition, the embodiments described above are not limited to application to images in the industrial field obtained by photographing a semiconductor chip and multiple chips formed on the semiconductor chip as the subject, and are also generally applicable to images in the medical field by photographing biological tissues, etc. as the subject.

[0163] The embodiments of the present invention have been described, but these embodiments are shown as examples and are not intended to limit the scope of the present invention. Specifically, for example, it is also allowed that different colors or patterns corresponding to the class classification information contained in the information read from the processing result storage unit 25 are not added. The reason is that for the inference result obtained by the inference processing unit 23, even if no color or pattern is added, sometimes the area where the feature quantity is close can be known based on the distribution of the set (concentration) of the inference result. In addition, the dimension conversion processing unit 34 can also be a clustering result equivalent to unsupervised learning, and dimensional conversion is not necessarily performed. These new embodiments can be implemented in various other ways, and various omissions, substitutions and changes can be made without departing from the scope of the subject matter of the invention. These embodiments and their modifications are also included in the scope and subject matter of the invention, and are included in the invention described in the claims and their equivalent scope.

[0164] This application is filed based on Japanese Patent Application No. 2018-209738 filed in Japan on November 7, 2018 as a priority claim, and the above-mentioned disclosure is cited in the description and claims of this application.

Claims

1. An image processing device, characterized in that: have: an inference processing unit configured to perform processing for obtaining a classification result obtained by classifying each of a plurality of regions set in a processing target image into any one of a plurality of predetermined classes; a dimensionality conversion processing unit configured to perform processing for acquiring multidimensional data corresponding to each of the plurality of regions set in the processing target image, and to perform processing for converting the multidimensional data into low-dimensional data; as well as The display processing unit performs processing for displaying a display image including one or more of the plurality of regions and a plotting diagram having a plurality of plotting points generated by adding different colors or patterns corresponding to the classification results to the low-dimensional data in each of the plurality of regions on a display device, and, in the case where there is an instruction for selecting a desired plotting point as a selection point from among the plotting points included in the plotting diagram, performs identification display processing for visually identifying the selection point and the region corresponding to the selection point in the display image, The inference processing unit is configured using a statistical model formed by machine learning, and further performs processing for calculating the classification results of each of the plurality of regions based on likelihood calculated by the statistical model. The display processing unit further performs a process for synthesizing the visual information corresponding to the classification result and the likelihood with the display image and displaying the synthesized image on the display device.

2. The image processing device according to claim 1, wherein: The display processing unit performs processing for visually identifying a selection area including the selection position in the multiple areas and drawing points corresponding to the selection area included in the drawing when there is an instruction for selecting the desired position within the display image as the selection position.

3. The image processing device according to claim 1, wherein: The device further includes a storage unit configured to store the corrected classification result when there is an instruction to correct the classification result displayed as the plurality of plot points in the plotted graph or the visual information in the display image.

4. The image processing device according to claim 1, wherein: The dimensional conversion processing unit performs processing for converting the multidimensional data into two-dimensional data. The display processing unit performs processing for generating a two-dimensional distribution map having the plurality of plot points generated by adding a color or a pattern corresponding to the classification result to the two-dimensional data in each of the plurality of regions.

5. An image processing method, characterized in that: The inference processing unit performs processing for obtaining a classification result by classifying each of a plurality of regions set in the processing target image into any one of a plurality of predetermined classes. The dimensional conversion processing unit performs processing for obtaining multidimensional data corresponding to the plurality of regions set in the processing target image, and performs processing for converting the multidimensional data into low-dimensional data. The display processing unit performs processing for displaying on the display device a display image including one or more of the plurality of regions and a drawing having a plurality of drawing points generated by adding different colors or patterns corresponding to the classification results to the low-dimensional data in each of the plurality of regions, The display processing unit performs a recognition display process for visually recognizing the selected point and the area corresponding to the selected point in the display image when there is an instruction for selecting a desired drawn point from among the drawn points included in the drawn image as a selected point. The inference processing unit is configured using a statistical model formed by machine learning, and further performs processing for calculating the classification results of each of the plurality of regions based on likelihood calculated by the statistical model. The display processing unit further performs a process for synthesizing the visual information corresponding to the classification result and the likelihood with the display image and displaying the synthesized image on the display device.

6. A computer-readable storage medium storing a program, characterized in that: The program causes the computer to execute: Processing for obtaining classification results in which a plurality of regions set in the processing target image are respectively classified into any one of a plurality of predetermined classes; Processing for obtaining multi-dimensional data corresponding to each of the plurality of regions set in the processing target image; Processing for converting the above multi-dimensional data into low-dimensional data; Processing for displaying on a display device a display image including one or more of the plurality of regions and a plotted graph having a plurality of plotted points generated by adding different colors or patterns corresponding to the classification results to the low-dimensional data in each of the plurality of regions; When there is an instruction for selecting a desired drawing point from among the drawing points included in the drawing as a selected point, a process for making the selected point and a region corresponding to the selected point in the display image visually identifiable. A process for calculating the classification result of each of the plurality of regions based on likelihood calculated by a statistical model constructed by machine learning; as well as A process for synthesizing the visual information corresponding to the classification result and the likelihood with the display image and displaying the result on the display device.