Position coordinate detection device, sample processing device, and sample inspection device

Abstract

In order to enable teaching data to be created easily and position coordinates to be detected with high reliability, this position coordinate detection device is constituted by comprising: a deep learning model having been trained using training image data in which the position coordinate to be detected is specified and teaching image data in which a pixel group exhibiting a shape independent of the subject of the training image data is disposed at a position relative to the position coordinate to be detected; a position coordinate calculation unit for calculating a position coordinate using the inference image data outputted from the deep learning model; and a reliability calculation unit for calculating reliability using the global pixel group information of the inference image data outputted from the deep learning model.

Description

[Technical field]

[0001] The present invention relates to a position coordinate detection device using machine learning technology. , sample processing device and sample inspection device Regarding. [Background technology]

[0002] In recent years, machine learning technology called deep learning has been used for image recognition processing. Among them, methods for detecting specific position coordinates of an object in an image include the methods described in Non-Patent Document 1, Non-Patent Document 2, Non-Patent Document 3, and Non-Patent Document 4.

[0003] In the above conventional technology, a deep learning network is trained using training image data in which the position coordinates to be detected are specified and a two-dimensional Gaussian heat map centered on the position coordinates to be detected as teacher data (correct answer data), and then a new image is inferred using the trained deep learning network (called a trained model), thereby detecting the position coordinates to be detected in the new image. Since the inference result output by the trained model is a two-dimensional Gaussian heat map, the coordinates of the peak value are output as the position coordinates to be detected. In addition, the peak value itself represents the reliability (certainty) of the inferred position coordinates.

[0004] The above conventional technology has the advantage that a two-dimensional Gaussian-type heat map can be automatically generated if the position coordinates to be detected are specified, making it easy to create training data. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Jonathan Tompson, Arjun Jain, Yann LeCun, Christoph Bregler, “Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation”, Annual Conference on Neural Information Processing System, 2014. [Non-Patent Document 2] Jonathan Tompson, Ross Gorosin, Arjun Jain, Yann LeCun, Christoph Bregler, “Efficient Object Localization Using Convolutional Networks“, Computer Vision and Pattern Recognition, 2015. [Non-Patent Document 3] Hei Law, Jia Deng, “CornerNet: Detecting Objects as Paired Keypoints“, European Conference on Computer Vision, 2018. [Non-Patent Document 4] Xingyi Zhou, Dequan Wang, Philipp Kr¨ahenb¨uhl, “Objects as Points“, Internet<https: / / arxiv.org / pdf / 1904.07850.pdf> , 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] When using position coordinates to automate the operation of equipment, such as in the case of industrial products, it is necessary to automatically determine whether the position coordinates inferred by the trained model are correct or not. In this case, it is possible to use the reliability.

[0007] However, in conventional technology, the peak value of the inferred heat map indicates the position coordinate to be detected and also indicates the reliability. Therefore, if this peak value is incorrect, there is a possibility that the incorrect position coordinate or reliability will be judged to be the correct position coordinate or reliability and operation will continue.

[0008] It is necessary to use a different scale for the reliability than the peak value of the inferred heat map, or to calculate the position coordinates and the reliability using different scales.

[0009] SUMMARY OF THE PRESENT EMBODIMENTS The present invention aims to solve the above-mentioned problems of the conventional technology and to provide a position coordinate detection device and method capable of achieving position detection with high detection performance simply by providing an image and the coordinates to be detected. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems of the conventional technology, a position coordinate detection device according to the present invention comprises: A deep learning model for semantic segmentation trained using a combination of training image data in which a position coordinate to be detected on an image capturing an object is specified, and teacher image data in which a group of pixels having the same pixel value and exhibiting a circular or polygonal shape centered on the position coordinate to be detected is arranged at a position relative to the object at the position coordinate to be detected on the training image data; a position coordinate calculation unit including a process of calculating a connected component and a center of gravity of the connected component of the inference image data, which calculates the desired position coordinate in the image of a new object whose position coordinate is to be determined by inputting the image of the new object whose position coordinate is to be determined into the deep learning model and using inference image data obtained; and a reliability calculation unit which calculates the reliability of the position coordinate calculated by the position coordinate calculation unit using information on the pixel group of the inference image data output from the deep learning model. The present invention is configured to include the following. Effect of the Invention

[0011] According to the present invention, since the reliability can be calculated using a scale different from the peak value of the heat map, it is possible to determine the possibility that the peak value is incorrect. Furthermore, when calculating the position coordinate or reliability from the shape of the pixel group of the inference data, the pixel group independent of the shape of the subject is used as the teacher data, so that the position coordinate or reliability can be calculated without depending on the shape of the subject. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of the configuration of a position coordinate detection device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of training image data and teacher image data in the first embodiment. [Diagram 3]1A and 1B are diagrams showing an example of image data of an inference target and inference image data which is an inference result in the first embodiment. [Figure 4] 1A and 1B are diagrams showing an example of image data of an inference target and inference image data which is an inference result in the first embodiment. [Diagram 5] 4 is a flowchart showing a position coordinate detection program according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing an example of the configuration of a position coordinate detection device according to a second embodiment. [Figure 7] FIG. 11 is a diagram showing an example of training image data and teacher image data in the second embodiment. [Figure 8] 13 is a diagram showing an example of image data of an inference target and inference image data which is an inference result in the second embodiment. FIG. [Figure 9] 13 is a diagram showing an example of image data of an inference target and inference image data which is an inference result in the second embodiment. FIG. [Figure 10] 10 is a flowchart showing a position coordinate detection program according to the second embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of a position coordinate detection device according to a third embodiment. [Figure 12] FIG. 13 is a diagram showing an example of inference image data which is an inference result in the third embodiment. [Figure 13] 13 is a flowchart showing a position coordinate detection program according to the third embodiment. [Figure 14] FIG. 11 is a block diagram showing a configuration example of a sample processing device according to a fourth embodiment. [Figure 15] 15 is a flowchart showing an example of the operation of the sample processing apparatus of FIG. 14 according to the fourth embodiment. [Figure 16] FIG. 13 is a block diagram showing a configuration example of a sample inspection device according to a fifth embodiment. [Figure 17] 17 is a flowchart showing an example of the operation of the sample inspection device of FIG. 16 according to the fifth embodiment. [Figure 18] FIG. 13 is a block diagram showing an example of the configuration of a position coordinate detecting device according to a modified example of the second embodiment. [Figure 19]FIG. 13 is a diagram showing an example of training image data and teacher image data in a modified example of the second embodiment. [Figure 20] 13 is a flowchart showing a position detection program according to a modified example of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention relates to a position coordinate detection device that includes a deep learning model trained using training image data in which the position coordinates to be detected are specified and teacher image data in which a group of pixels having a shape independent of the subject of the training image data are arranged at a position relative to the position coordinates to be detected, a position coordinate calculation unit that calculates position coordinates using inference image data output from the deep learning model, and a reliability calculation unit that calculates reliability using global information on the group of pixels of the inference image data output from the deep learning model.

[0014] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the elements and combinations described in the embodiments are necessarily essential to the solution of the invention. Also, suitable combinations of the elements described in the embodiments are included in the embodiments disclosed in the present application. EXAMPLES

[0015] A first embodiment of the present invention will be described with reference to FIGS. 1 is a block diagram showing an example of the configuration of a position coordinate detection device 100 according to an embodiment of Example 1. The position coordinate detection device 100 includes a deep learning model 110, a position coordinate calculation unit 120, and a reliability calculation unit 130. The position coordinate detection device 100 is configured by hardware or a software program provided in a part of a computer system (not shown).

[0016] The deep learning model 110 is a trained model trained in advance using multiple pairs of training image data and teacher data. The deep learning model 110 is a fully-convolutional encoder-decoder network as shown in the prior art documents (Non-Patent Documents 1 to 4). An example of the training image data and teacher data will be described with reference to FIG. 2.

[0017] Image 210 in Fig. 2(a) is training image data (hereinafter referred to as training image data 210) containing a subject 201. The training image data 210 is a diagram created for explanation purposes and is not related to images actually output by the device. An x mark 202 on the training image data 210 represents the position coordinates to be detected. The x mark 202 does not exist on the actual image.

[0018] An image 220 in FIG. 2(b) is teacher data (hereinafter referred to as teacher data 220) in which a heat map 204 is formed with the position coordinates of a cross mark 203 to be detected as its center. The position coordinates of the cross mark 203 in the teacher data 220 are the same as the position coordinates of the cross mark 202 in the training image data 210. The cross mark 203 also does not exist in the teacher data 220. The heat map 204 is composed of a group of pixels in which the pixel value at the center is high (e.g., 255) and the pixel value decreases with increasing distance from the center. The distribution of pixel values ​​may be a two-dimensional Gaussian type or a cone type.

[0019] A plurality of pairs of training image data (a) and teacher data (b) as shown in FIG. 2 are prepared, and a deep learning model 110 is trained.

[0020] The behavior of the trained deep learning model 110 will be explained using FIG. The image 310 in FIG. 3(a) is an image in which a subject 301 is captured. The image 310 is a diagram created for explanation and is not related to an image output by an actual device. The cross mark 302 on the image 310 represents the position coordinates to be detected. The cross mark 302 does not exist on the actual image. The image 320 in FIG. 3(b) is an inferred image (hereinafter, referred to as the inferred image 320) in which a heat map 304 is formed around the position coordinates 303, which are the same coordinates as the position coordinates of the cross mark 302 to be detected in the image 310. If the deep learning model 110 is a well-trained model, when the image 310 is input to the deep learning model 110, an inferred image 320 is output.

[0021] The position coordinate calculation unit 120 calculates position coordinates from an inferred image 320 inferred by the deep learning model 110. Specifically, the position coordinate calculation unit 120 detects a peak (position coordinates 303) in the heat map 304 by obtaining a peak with a maximum pixel value from the inferred image 320, and outputs the coordinates.

[0022] The reliability calculation unit 130 calculates reliability from the shape of the heat map 304 formed at the peak position output by the position coordinate calculation unit 120 from the inferred image 320 inferred by the deep learning model 110. A specific calculation method may be a known technique for quantifying the difference in shape, such as a method for quantifying the isotropy of the heat map, a method for calculating the difference with the heat map used in the teacher data, or a method for calculating the correlation value with the heat map used in the teacher data, or a combination of such techniques.

[0023] Next, the behavior of the position coordinate detection device 100 when the degree of difficulty in detecting the position coordinates is high will be described with reference to FIG.

[0024] An image 410 in FIG. 4(a) is an image in which a subject 401 is captured. Backgrounds 411 and 412 with different contrasts and noise 413 are captured in the image 410. The image 410 is a diagram created for explanation purposes and is not related to an image output by an actual device. An image 420 in FIG. 4(b) shows an example of an inferred image (hereinafter, referred to as an inferred image 420) inferred by providing the image 410 to the deep learning model 110. Two heat maps, a heat map 403 and a heat map 404, are formed in the inferred image 420. Here, it is assumed that the pixel value of the peak of the heat map 403 is greater than the pixel value of the peak of the heat map 404. It is also assumed that the heat map 403 has a lower isotropy than the heat map 404.

[0025] The position coordinate calculation unit 120 detects a peak in the heat map 403 by finding the peak with the maximum pixel value from the inference image 420, and outputs the coordinates of the peak.

[0026] The reliability calculation unit 130 calculates the reliability from the shape of the heat map 403 formed at the peak position output by the position coordinate calculation unit 120 from the inferred image 420. Since the heat map 403 has a lower isotropy than the heat map 204 used in the teacher data 220, a low reliability is output. This indicates that the position coordinates output by the position coordinate calculation unit 120 are not appropriate.

[0027] In the above embodiment, only inappropriate position coordinates are output, but multiple position coordinates and their corresponding reliabilities may be output, and appropriate position coordinates may be selected by a post-stage process (not shown). In addition, although the configuration of the position coordinate detection device 100 is changed, the output of the reliability calculation unit 130 may be input to the position coordinate calculation unit 120 to output the position coordinate with the highest reliability.

[0028] FIG. 5 is a flowchart of a position coordinate detection program representing one embodiment of the present invention, which is executed using a computer system (not shown) incorporating the position coordinate detection device 100 described in FIG.

[0029] First, the deep learning model 110 is trained using a plurality of pairs of training image data and teacher data (S501). Examples of training image data and teacher data are shown in FIG. 2(a) as training image data 210 and FIG. 2(b) as teacher data 220, respectively.

[0030] Next, the image for which the position coordinates are to be detected is input to the trained deep learning model to obtain an inferred image (S502). An example of the image for which the position coordinates are to be detected is shown in image 310 in FIG. 3(a), and an example of the inferred image is shown in inferred image 320 in FIG. 3(b).

[0031] From the obtained inferred image, a maximum of N peak coordinates whose pixel values ​​are equal to or greater than a specific threshold value are obtained (S503). The specific threshold value and N are set in advance.

[0032] Next, the reliability is calculated from the shape of the heat map corresponding to the obtained peak coordinates (S504). Examples of the heat map are shown in 403 and 404 in FIG. 4(b). Specific calculation methods may be known techniques for quantifying differences in shape, such as a method for quantifying the isotropy of the heat map, a method for calculating the difference with the heat map used in the training data, or a method for calculating the correlation value with the heat map used in the training data, or a combination of such techniques.

[0033] Finally, the peak coordinates with the highest reliability are obtained and output (S505). At this time, information on the reliability corresponding to the output peak coordinates may be output together. The destination of the output may be a control unit (control unit that controls an external device) or a display screen of a computer (not shown), or may be an external processing device connected to a computer (not shown).

[0034] According to this embodiment, the reliability is calculated using information other than the peak value of the heat map, so that when a peak position is erroneously detected, the output information can indicate that this is an erroneous detection. EXAMPLES

[0035] A second embodiment of the present invention will be described with reference to FIGS. 6 is a block diagram showing an example of the configuration of a position coordinate detection device 600 according to the second embodiment. The position coordinate detection device 600 includes a deep learning model 610, a position coordinate calculation unit 620, and a reliability calculation unit 630. The position coordinate detection device 600 is configured by hardware or a software program provided in a part of a computer system (not shown).

[0036] The deep learning model 610 is a trained model that has been trained in advance using multiple pairs of training image data and teacher data. The deep learning model 610 can perform semantic segmentation. In other words, it calculates the background likelihood and object likelihood for each pixel and outputs the label corresponding to the largest of these as the pixel value. It can also be said that it classifies each pixel as background or object. It can also be said that the greater the ratio of the maximum value to the total value of the background likelihood and object likelihood for each class (or the value of the softmax function), the higher the likelihood. An example of training image data and teacher data will be explained using FIG. 7.

[0037] The training image data 210 in Fig. 7(a) is the same image as the training image data 210 in Fig. 2 and is training image data. The image 720 in Fig. 7(b) is teacher data (hereinafter referred to as teacher data 720) in which a circular figure 704 is formed with the position coordinates of a cross mark 703 to be detected as its center. The cross mark 703 in the teacher data 720 has the same coordinates as the cross mark 203 in the training image data 210 shown in Fig. 2 in the first embodiment. The cross mark 703, like the cross mark 203, does not exist in the teacher data 720.

[0038] The circular figure 704 is made up of a group of pixels where all pixel values ​​are the same value (e.g., 255). On the other hand, the parts other than the circular figure (called the background) have the same value (e.g., 0) different from the circular figure. The pixel values ​​of the circular figure correspond to the position coordinate labels, and the pixel values ​​of the background correspond to the background labels. In this way, the fact that the teacher data 720 in this embodiment is displayed as a binarized image differs from the fact that the teacher data 220 in the first embodiment is displayed as a multi-value image. A group of pixels where pixels are connected, such as the circular figure 704, is also called a blob in the field of image processing.

[0039] A plurality of pairs of training image data as shown in (a) of FIG. 7 and teacher data as shown in (b) are prepared, and a deep learning model 610 is trained.

[0040] The behavior of the trained deep learning model 610 will be explained using FIG. 8. The image 310 in FIG. 8(a) is the same as the image 310 in FIG. 3(a). The image 820 in FIG. 8(b) is an inferred image (hereinafter, referred to as the inferred image 820) in which a blob 804 is formed with a center of gravity at a position coordinate 803, which is the same coordinate as the position coordinate of the x mark 302 to be detected in the image 310. When the deep learning model 610 is a well-trained model, when the image 310 is input to the deep learning model 610, the deep learning model 610 outputs a position coordinate label for pixels within the same radius as the circle 704, centered on the position coordinate 803, which is the correct position coordinate, and outputs a background label for other pixels. That is, the inferred image 820 is output from the deep learning model 610.

[0041] The position coordinate calculation unit 620 calculates position coordinates from the inferred image 820 inferred by the deep learning model 610. Specifically, blobs whose area is equal to or greater than a specific threshold value are found from the inferred image 820, the centroid coordinates of the found blobs are found, and the centroid coordinates are output. The detection of blobs and the calculation of centroid coordinates may be performed by a known method. Furthermore, the selection may be prioritized using the likelihood of the centroid coordinates or the average likelihood of the entire blob. The likelihood value may be output in association with the corresponding centroid coordinates.

[0042] The reliability calculation unit 630 calculates reliability from the shape of the blob 804 formed at the center of gravity output by the position coordinate calculation unit 620 from the inferred image 820 inferred by the deep learning model 610. Specific calculation methods may be known techniques for quantifying differences in shape, such as a method for quantifying the circularity of the blob, a method for calculating a correlation value with a blob used in training data, or a method for converting the blob into polar coordinates from the center of gravity to quantify the unevenness of the boundary, or a combination of such techniques.

[0043] Next, the behavior of the position coordinate detection device 600 when the degree of difficulty in detecting the position coordinates is high will be described with reference to FIG.

[0044] Image 410 in FIG. 9(a) is the same image as image 410 in FIG. 4(a). Image 920 in FIG. 9(b) shows an example of an inferred image (hereinafter, referred to as inferred image 920) inferred by providing image 410 to deep learning model 610. Two blobs, blob 903 and blob 904, are formed in inferred image 920. Here, blob 903 has a higher accuracy of centroid coordinates than blob 904. Also, blob 903 has a lower circularity than blob 904.

[0045] The position coordinate calculation unit 620 finds blobs whose area is equal to or greater than a specific threshold value from the inferred image 920 inferred by the deep learning model 610, finds the centroid coordinates of the found blobs, and outputs the centroid coordinates.

[0046] The reliability calculation unit 630 calculates reliability from the shape of the blob 903 formed at the center of gravity position output by the position coordinate calculation unit 620 in the inferred image 920. Since the blob 903 has a low circularity, a low reliability is output. This indicates that the position coordinates output by the position coordinate calculation unit 620 are not appropriate. In the above embodiment, only inappropriate position coordinates are output, but multiple position coordinates and their corresponding reliabilities may be output, and appropriate position coordinates may be selected by a post-stage process (not shown). In addition, although the configuration of the position coordinate detection device is changed, the output of the reliability calculation unit may be input to the position coordinate calculation unit to output the position coordinate with the highest reliability.

[0047] FIG. 10 is a flowchart of a position coordinate detection program representing one embodiment of the present invention, which is executed using a computer system (not shown) incorporating the position coordinate detection device 600 described in FIG.

[0048] First, a deep learning model is trained using multiple pairs of training image data and teacher data (S1001). Examples of training image data and teacher data are shown in training image data 210 in FIG. 7(a) and teacher data 720 in FIG. 7(b), respectively.

[0049] Next, the image for which the position coordinates are to be detected is input to the trained deep learning model 610 to obtain an inferred image (S1002). An example of the image for which the position coordinates are to be detected is shown as image 310 in FIG. 8(a), and an example of the inferred image is shown as inferred image 820 in FIG. 8(b).

[0050] From the obtained inferred image, a maximum of N blobs whose area is equal to or larger than a specific threshold are obtained, and the positions of the centers of gravity of the blobs are obtained (S1003). The specific threshold and N are set in advance.

[0051] The reliability is calculated from the shape of the blob corresponding to the calculated center of gravity coordinates (S1004). A specific calculation method may be a known technique for quantifying differences in shape, such as a method for quantifying the circularity of the blob, a method for calculating a correlation value with a blob used in training data, or a method for converting the blob into polar coordinates from the center of gravity to quantify the unevenness of the boundary, or a combination of such techniques.

[0052] Finally, the center of gravity coordinates with the maximum reliability are obtained and output (S1005). At this time, the reliability corresponding to the output center of gravity coordinates may be output.

[0053] In the above example, the training data is a circle, but it may be a polygon such as a rectangle. In this case, the reliability can be calculated based on the feature calculated by a known technique for the shape.

[0054] According to this embodiment, position coordinates are obtained by calculating the center of gravity of blobs consisting of the same pixel values, so that position coordinates can be uniquely determined and position coordinates can be obtained with relatively simple processing compared to the method described in embodiment 1. [Variations] A modification of the second embodiment will be described with reference to Fig. 18 to Fig. 20. Fig. 18 is a block diagram showing a configuration example of a position coordinate detection device 1800 according to this modification. The position coordinate detection device 1800 includes a deep learning model 1810 and a position coordinate calculation unit 1820.

[0055] This modification is different from the second embodiment in that it does not include the reliability calculation unit 630 described in the second embodiment. The deep learning model 1810 is the same as the deep learning model 610 in Fig. 6 described in the second embodiment. Therefore, examples of the training image data and the teacher data are the same as the training image data 210 and the teacher data 720 described in Fig. 7.

[0056] Next, the behavior of the position coordinate detection device 1800 when the degree of difficulty in detecting the position coordinates is high will be described with reference to FIG.

[0057] An image 410 in FIG. 19(a) is the same image as the image 410 in FIG. 4(a). An inferred image 1920 in FIG. 19(b) shows an example of an inferred image inferred by providing the image 410 to the deep learning model 1810. Two blobs, blob 1903 and blob 1904, are formed in the inferred image 1920. Here, blob 1904 is a blob corresponding to position coordinates to be detected, and blob 1903 is a blob corresponding to position coordinates other than the position coordinates to be detected. In addition, blob 1903 has a smaller area than blob 1904.

[0058] The position coordinate calculation unit 1820 finds the blob with the largest area from the inference image 1920, finds the centroid coordinates of the found blob, and outputs the centroid coordinates. In this modification, by utilizing the characteristic that the blob for which the centroid coordinates should be found generally has a larger area than other noise blobs, when multiple blobs are detected in the inference image 1920, the blob with the largest area is detected, and it is determined to be the blob that should have been detected without calculating the reliability of each of the multiple blobs.

[0059] FIG. 20 is a flowchart of a position coordinate detection program for performing position coordinate detection according to this modified example using a computer system (not shown).

[0060] First, a deep learning model 1810 is trained using a plurality of pairs of training image data and teacher data (S2001). Examples of the training image data and teacher data are the same as the training image data 210 and teacher data 720 described in the second embodiment with reference to FIG.

[0061] Next, the image for which the position coordinates are to be detected is input to the trained deep learning model to obtain an inferred image (S2002). An example of the image for which the position coordinates are to be detected is shown as image 410 in FIG. 19(a), and an example of the inferred image is shown as inferred image 1920 in FIG. 19(b).

[0062] From the obtained inference image, the blob with the largest area is found, and its center of gravity is calculated (S2003). Finally, the calculated coordinates are output (S2005).

[0063] In the conventional example, when a peak becomes flat or multiple peaks appear in the vicinity, complex processing is required to detect the peak, but this modified example has the advantage that it requires only the simple process of determining the coordinates of the center of gravity of the blob.

[0064] Also, this modification has the advantage that the position coordinates can be obtained without calculating the reliability index. EXAMPLES

[0065] A third embodiment of the present invention will be described with reference to FIGS. 11 is a block diagram showing an example of the configuration of a position coordinate detection device 1100 according to this embodiment. The position coordinate detection device 1100 includes a deep learning model 1110, a position coordinate calculation unit 1120, and a reliability calculation unit 1130.

[0066] The deep learning model 1110 is a trained model trained in advance using multiple pairs of training image data and teacher data. The deep learning model 1110 can also perform semantic segmentation. The deep learning model 1110 randomly drops out internal nodes multiple times and determines pixel labels using the average value. It also outputs an image in which the pixel values ​​are the variations due to multiple dropouts.

[0067] In the second embodiment, a plurality of pairs of training image data 210 and teacher data 720 as described with reference to FIG. 7 are prepared, and a deep learning model 1110 is trained.

[0068] The behavior of the trained deep learning model 1110 will be explained using FIG. 12. An image 1210 in Fig. 12(a) shows an example of a label image (hereinafter, referred to as an inferred image 1210) inferred using an average value of semantic segmentation performed by randomly performing dropout multiple times when the image 410 of the subject described in Fig. 9(a) in the second embodiment is input to the deep learning model 1110. Two blobs, blob 1201 and blob 1202, are formed in the inferred image 1210. Here, it is assumed that the blob 1201 has a higher likelihood than the blob 1202.

[0069] Image 1220 in Fig. 12(b) shows an example of an image inferred using the variation of semantic segmentation performed by randomly performing multiple dropouts when the subject image 410 in Fig. 9(a) is input to the deep learning model 1110 (hereinafter referred to as inferred image 1220). Each pixel in the inferred image 1220 represents the variation of the output value (or the variation of the label) of the deep learning model 1110 for multiple dropouts. In the inferred image 1220, whiter means less variation, and blacker means more variation.

[0070] The position coordinate calculation unit 1120 in the position coordinate detection device 1100 shown in FIG. 11 finds blobs whose area is equal to or greater than a specific threshold value from the inferred image 1210 in FIG. 12(a), finds the centroid coordinates of the found blobs, and outputs the centroid coordinates.

[0071] On the other hand, the reliability calculation unit 1130 in the position coordinate detection device 1100 calculates reliability from the shape of the blob 1203 formed at the center of gravity position output by the position coordinate calculation unit 1120 from the inference image 1220 in Fig. 12(b). Since the blob 1203 has a low circularity, a low reliability is output. This indicates that the position coordinates output by the position coordinate calculation unit 1120 are not appropriate. Instead of the circularity, the amount of variation from a circle with zero variation may be calculated.

[0072] In the above embodiment, only inappropriate position coordinates are output, but multiple position coordinates and their corresponding reliabilities may be output, and appropriate position coordinates may be selected by subsequent processing (not shown). In addition, although the configuration of the position coordinate detection device 1100 is changed, the output of the reliability calculation unit 1130 may be input to the position coordinate calculation unit 1120 to output the position coordinate with the highest reliability.

[0073] FIG. 13 is a flow chart of a position coordinate detection program for detecting the position coordinates described in this embodiment using a computer system (not shown).

[0074] First, a deep learning model is trained using multiple pairs of training image data and teacher data (S1301). This deep learning model performs semantic segmentation by randomly dropping out internal nodes multiple times, and determines pixel labels using the average value. In addition, an image is output in which the pixel value is the variation of semantic segmentation due to multiple dropouts. Examples of training image data and teacher data are shown in training image data 210 in FIG. 7(a) and teacher data 720 in FIG. 7(b), respectively.

[0075] Next, the image for which position coordinates are to be detected is input to the trained deep learning model 1110, and an inferred image in which pixel labels are determined using the average value of semantic segmentation by randomly performing dropout multiple times and an inferred image in which pixel values ​​are the variations in semantic segmentation due to multiple dropouts are obtained (S1302). An example of an image for which position coordinates are to be detected is shown in image 410 in FIG. 9(a). An example of an inferred image in which pixel labels are determined using the average value of semantic segmentation by randomly performing dropout multiple times is shown in inferred image 1210 in FIG. 12(a), and an example of an inferred image in which pixel values ​​are the variations in semantic segmentation due to multiple dropouts is shown in inferred image 1220 in FIG. 12(b).

[0076] Randomly performing dropout multiple times, the labels of pixels are determined using the average value of semantic segmentation from the inferred image, and a maximum of N blobs whose area is equal to or exceeds a specific threshold are obtained from the inferred image, and the centroid position of the blob is obtained (S1303). The specific threshold and N are set in advance.

[0077] A blob corresponding to the calculated center of gravity coordinates is identified from an inference image in which the pixel values ​​are the variations in semantic segmentation due to multiple dropouts, and the reliability is calculated from the shape (S1304). Specific calculation methods may include a method of quantifying the circularity of the blob, a method of calculating a correlation value with a blob used in the training data, a method of converting the blob into polar coordinates from the center of gravity to quantify the unevenness of the boundary, a variation from a circle with zero variation, or any other known technique or combination of such techniques for quantifying differences in shape.

[0078] Finally, the barycentric coordinates with the maximum reliability are obtained and output (S1305). At this time, the reliability corresponding to the output barycentric coordinates may be output.

[0079] In the above example, the training data is a circle, but it may be a polygon such as a rectangle. In this case, the reliability can be calculated based on the feature calculated by a known technique for the shape.

[0080] According to this embodiment, the centroid coordinates and reliability are calculated using the average values ​​and variations of semantic segmentation using multiple dropouts, which makes it possible to obtain position coordinates with higher reliability by improving robustness against noise. EXAMPLES

[0081] Next, a case where the position coordinate detection device 100, 600, 1100 or 1800 described in the first to third embodiments is applied to an actual device will be described.

[0082] 14 is a block diagram showing a configuration example of a sample processing device 1400 according to Example 4. The sample processing device 1400 includes a needle unit 1401, a transport unit 1402, a computer 1403, and an image sensor 1404. The needle unit 1401, the transport unit 1402, the computer 1403, and the image sensor 1404 are connected via a bus 1406. The bus 1406 holds data, control information, and analysis information handled by each processing unit connected to the bus 1406, or mediates the transmission of the data, control information, and analysis information.

[0083] The sample processing device 1400 is connected to a display device 1410 and an input / output device 1411 via a bus 1406. The sample processing device 1400 may be connected to the display device 1410 and the input / output device 1411 by wire or wirelessly. Note that, although an example in which the display device 1410 and the input / output device 1411 are provided outside the sample processing device 1400 is shown in FIG. 14, they may be built into the sample processing device 1400.

[0084] The sample processing device 1400 has the function of transporting a specific sample inside the device and bringing the tip of a needle (not shown) built into the needle portion 1401 close to a specific location on the sample in order to pick up a portion of the sample.

[0085] A processing flow for bringing the tip of a needle (not shown) built into needle portion 1401 into close proximity to a specific location on a sample will be described with reference to FIG.

[0086] First, a target sample (not shown) is transported into the sample processing device 1400 using the transport unit 1402 (S1501). Next, a specific portion of the transported sample is processed by a processing unit (not shown) (S1502). A position coordinate detection program created in advance in the computer 1403 and an image of the sample captured by the image sensor 1404 are used to detect a specific portion of the processed sample (S1503). This position coordinate detection program may be a position coordinate detection program corresponding to the flowchart shown in Figure 13 that has been trained to detect a specific portion of the sample, or may be one based on known technology.

[0087] When a position coordinate detection program corresponding to the flowchart shown in Fig. 13 is used, if the reliability calculated in the step corresponding to S1304 is lower than a predetermined threshold value, an alarm may be issued to notify the operator, or the reliability may be collated with position coordinates calculated using another position coordinate detection program stored in computer 1403, and the operation may be continued if both are within an allowable range. When an alarm is issued to notify the operator, the operator may refer to the image of image sensor 1404 displayed on display device 1610 and the position coordinates of the center of gravity calculated in S1303 of the flowchart shown in Fig. 13 to determine whether to continue or interrupt the operation.

[0088] If the operation is to be continued, the tip of the needle is detected (S1504) using a position coordinate detection program created in advance in the computer 1403 and an image of the needle captured by the image sensor 1404. This position coordinate detection program uses a position coordinate detection program corresponding to the flowchart shown in FIG.

[0089] Here, if the calculated reliability is lower than a predetermined threshold, an alarm may be issued to inform the operator, or the calculated reliability may be compared with the position coordinates calculated using another position coordinate detection program, and if both are within an allowable range, the operation may be continued. If an alarm is issued to inform the operator, the operator may refer to the image of the image sensor and the position coordinates displayed on the display device to determine whether to continue or interrupt the operation. If the operation is to be continued, the needle unit 1401 is controlled to move the needle so that the position coordinates of the detected specific portion of the sample and the position coordinates of the tip of the needle are close to each other (S1505).

[0090] According to this embodiment, the center of gravity coordinates and reliability are calculated using the same method as described in embodiment 3, which improves robustness against noise and enables the needle tip position to be calculated with higher reliability. EXAMPLES

[0091] Next, another example in which the position coordinate detection device 100, 600, 1100 or 1800 described in the first to third embodiments is applied to an actual device will be described.

[0092] 16 is a block diagram showing an example of the configuration of a sample inspection device 1600 according to a fifth embodiment. The sample inspection device 1600 includes a charged particle beam unit 1601, a transport unit 1602, a computer 1603, and an image sensor 1604. The charged particle beam unit 1601, the transport unit 1602, the computer 1603, and the image sensor 1604 are connected via a bus 1606. The bus 1606 holds data, control information, and analysis information handled by each processing unit connected to the bus 1606, or mediates the transmission of the data, control information, and analysis information.

[0093] The sample inspection device 1600 is connected to a display device 1610 and an input / output device 1611 via a bus 1606. The sample inspection device 1600 may be connected to the display device 1610 and the input / output device 1611 by wire or wirelessly. Note that, although an example in which the display device 1610 and the input / output device 1611 are provided outside the sample inspection device 1600 is shown in FIG. 16, they may be built into the sample inspection device 1600.

[0094] The process flow for detecting a specific portion of a sample will be described with reference to FIG.

[0095] First, a target sample (not shown) is transported into the sample inspection device 1600 using the transport unit 1602 (S1701). Next, a specific location of the sample is detected using a position coordinate detection program previously stored in the computer 1603 and an image of the sample captured by the image sensor 1604 or the charged particle beam unit 1601 (S1702). This position coordinate detection program is a position coordinate detection program corresponding to the flowchart described in the third embodiment using FIG. 13, which has been trained to detect a specific location of the sample.

[0096] Here, if the calculated reliability is lower than a predetermined threshold, an alarm may be issued to inform the operator, or the calculated reliability may be collated with position coordinates calculated using another position coordinate detection program stored in computer 1603, and the operation may be continued if both are within an allowable range. If an alarm is issued to inform the operator, the operator may refer to the image detected by image sensor 1604 displayed on display device 1610 and the position coordinates of the center of gravity determined in S1303 of the flowchart shown in Fig. 13 to determine whether to continue or interrupt the operation. If the operation is to be continued, the specific portion of the sample that has been detected is inspected (S1703).

[0097] According to this embodiment, the center of gravity coordinates and reliability are calculated using the same method as described in the third embodiment, so that the robustness against noise is improved and a specific portion of the sample can be determined with a higher reliability. [Explanation of symbols]

[0098] 100, 600, 1100, 1800 Position coordinate detection device 110, 610, 1110, 1810 Deep Learning Models 120,620,1120,1820 Position coordinate calculation section 130,630,1130 Reliability calculation section 201 Subject Image 210 Training image data 220, 720 Teacher data 320, 420, 820, 920, 1210 Inference images 1210 Inference Images 1400 Sample processing equipment 1600 Sample Inspection Equipment

Claims

1. a deep learning model for semantic segmentation trained using a combination of training image data in which a position coordinate to be detected on an image capturing an object is specified, and teacher image data in which a group of pixels having the same pixel value and exhibiting a circular or polygonal shape centered on the position coordinate to be detected is arranged at a position relative to the object at the position coordinate to be detected on the training image data; A position coordinate calculation unit that calculates the desired position coordinates in the image of a new object by inputting an image of the new object to the deep learning model and using inference image data obtained by the inputting the image of the new object to the deep learning model, and includes a process of calculating connected components and a center of gravity of the connected components of the inference image data. a reliability calculation unit that calculates the reliability of the position coordinates calculated by the position coordinate calculation unit using information on a pixel group of the inference image data output from the deep learning model; A position coordinate detection device comprising:

2. In the position coordinate detection device according to claim 1, The reliability calculation unit includes a process of quantifying a difference between global information of the pixel group of the inference image data and a shape of the pixel group of the teacher image data. A position coordinate detection device comprising:

3. In the position coordinate detection device according to claim 1, The reliability calculation unit includes a process of converting global information of the pixel group of the inference image data into a numerical value. A position coordinate detection device comprising:

4. A sample processing device comprising a position coordinate detection device described in any one of claims 1 to 3.

5. A sample inspection device comprising a position coordinate detection device described in any one of claims 1 to 3.

Images