Image processing device, image processing method and program

Abstract

The image processing device includes an image acquisition unit that acquires a target image in which a plurality of fibers are captured; a first region division unit that generates a region division result for each individual by detecting each fiber contained in the target image using a trained region division model for each individual; a second region division unit that generates a region division result for each type by recognizing a region in which a plurality of fibers are captured from the target image using a trained region division model for each type; a region correction unit that corrects the region division result for each individual using the region division result for each type; and a result output unit that outputs the corrected region division result for each individual.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] There is a known technique for performing image segmentation using a machine learning model. In the image segmentation, the results of the segmentation may be corrected in order to improve accuracy.

[0003] For example, Patent Document 1 discloses a metal structure segmentation method using a trained machine learning model. The segmentation method disclosed in Patent Document 1 determines a correction that brings the segmentation result closer to a correct segmentation result based on the result of segmentation performed on a selected image, and applies the correction to the result of segmentation performed on images other than the selected image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-188504 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology has a problem that the accuracy of detecting individual fibers from an image in which multiple fibers are captured is low. For example, when a portion in which multiple fibers are entangled is included, the entangled multiple fibers cannot be accurately identified. Similarly, when a portion in which multiple fibers overlap is included, the overlapping multiple fibers cannot be accurately identified. In addition, when linear stains or the like are present in the background, the fibers may be erroneously detected as fibers.

[0006] One aspect of the present disclosure aims to accurately recognize fibers contained in an image. [Means for solving the problem]

[0007] The present disclosure has the following configuration.

[0008] [1] An image acquisition unit configured to acquire a target image of a plurality of fibers; A first region division unit configured to generate a region division result for each individual in which each of the fibers included in the target image is detected, using a trained region division model for each individual; A second region division unit configured to generate a region division result for each type by recognizing a region in which the plurality of fibers are imaged from the target image by using a trained region division model for each type; an area correction unit configured to correct the area division result for each individual based on the area division result for each type; a result output unit configured to output a correction result of the region segmentation result for each individual; An image processing device comprising:

[0009] [2] The image processing device according to [1] above, The area correction unit is configured to generate the correction result by calculating a logical product of the area division result for each individual and the area division result for each type. Image processing device.

[0010] [3] The image processing device according to [1] above, The area correction unit is configured to generate the correction result by selecting the area segmentation result for each individual or the area segmentation result for each type based on a result of comparing a score of the area segmentation result for each individual with a score of the area segmentation result for each type for each unit of the target image. Image processing device.

[0011] [4] The image processing device according to any one of [1] to [3] above, The individual region segmentation model is a model that performs instance segmentation, The region division model for each type is a model for performing semantic segmentation. Image processing device.

[0012] [5] The image processing device according to [4] above, The individual segmentation model is Mask R-CNN or YOLACT. Image processing device.

[0013] [6] The image processing device according to [4] or [5] above, The size of the bounding box of the individual segmentation model is adjustable. Image processing device.

[0014] [7] The image processing device according to any one of [4] to [6] above, The size of the mask of each individual in the region segmentation model is adjustable. Image processing device.

[0015] [8] The image processing device according to any one of [4] to [7] above, The region segmentation model for each type is DeepLab or U-Net, Image processing device.

[0016] [9] The image processing device according to any one of [1] to [8] above, The area correction unit is configured to correct the area in which the fiber is detected in the area division result for each individual. Image processing device.

[0017]

[10] The image processing device according to [9] above, The area correction unit is configured to enlarge the area divided for each individual by a dilation process or a smoothing process. Image processing device.

[0018]

[11] The computer: acquiring a target image of a plurality of fibers; generating an individual region segmentation result for each of the fibers included in the target image by using a trained individual region segmentation model; A step of generating a region segmentation result for each type by recognizing a region in which the plurality of fibers are imaged from the target image using a trained region segmentation model for each type; A step of correcting the result of the area segmentation for each individual using the result of the area segmentation for each type; A step of outputting a correction result of the region segmentation result for each individual; An image processing method that performs

[0019]

[12] To the computer: acquiring a target image of a plurality of fibers; generating an individual region segmentation result for each of the fibers included in the target image by using a trained individual region segmentation model; A step of generating a region segmentation result for each type by recognizing a region in which the plurality of fibers are imaged from the target image using a trained region segmentation model for each type; A step of correcting the result of the area segmentation for each individual using the result of the area segmentation for each type; A step of outputting a correction result of the region segmentation result for each individual; A program for executing the above. Effect of the Invention

[0020] According to one aspect of the present disclosure, fibers contained in an image can be recognized with high accuracy. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing an example of a target image. [Diagram 2] FIG. 2 is a diagram showing an example of a detection result according to the conventional technology. [Diagram 3] FIG. 3 is a block diagram showing an example of the overall configuration of the object detection system. [Figure 4] FIG. 4 is a block diagram showing an example of a hardware configuration of a computer. [Diagram 5] FIG. 5 is a block diagram illustrating an example of a functional configuration of the object detection system. [Figure 6] FIG. 6 is a flowchart illustrating an example of a processing procedure of the object detection method. [Figure 7] FIG. 7 is a diagram showing an example of a mask score before dilation. [Figure 8] FIG. 8 is a diagram showing an example of a mask score after binarization. [Figure 9] FIG. 9 is a diagram showing an example of a mask score after dilation. [Figure 10] FIG. 10 is a diagram showing an example of the mask score after correction. [Figure 11] FIG. 11 is a diagram illustrating an example of a detection result according to an embodiment. [Figure 12] FIG. 12 is a diagram showing an example of the analysis screen. [Figure 13] FIG. 13 is a diagram showing an example of the image confirmation screen. [Figure 14] FIG. 14 is a diagram showing an example of the first result confirmation screen. [Figure 15] FIG. 15 is a diagram showing an example of the first result confirmation screen. [Figure 16] FIG. 16 is a diagram showing an example of the second result confirmation screen. [Figure 17] FIG. 17 is a diagram showing an example of the second result confirmation screen. [Figure 18] FIG. 18 is a diagram showing an example of the correction result confirmation screen. [Figure 19] FIG. 19 is a diagram showing an example of a target image. [Figure 20] FIG. 20 is a diagram showing an example of a region segmentation result by U-Net. [Figure 21] FIG. 21 is a diagram showing an example of a region segmentation result by DeepLab. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0023] [Embodiment] One embodiment of the present disclosure is an object detection system that detects an object from an image of the object. Hereinafter, an image that is a target of object detection is also referred to as a "target image." The target image in this embodiment is an image that captures a state in which a large number of fibers are dispersed on the surface of an observation sample, and includes a plurality of fibers that are entangled with each other or a plurality of fibers that are overlapped with each other. As an example, the fiber in this embodiment is carbon fiber. However, the fiber to be detected is not limited to carbon fiber, and may be fiber of any material.

[0024] In the task of detecting objects from an image, segmentation using a machine learning model is sometimes performed, including instance segmentation and semantic segmentation based on deep learning.

[0025] Instance segmentation is a task of detecting individual objects contained in an image. In instance segmentation, rectangular regions (bounding boxes) in which objects are captured may be detected individually from an image, or individual objects may be detected and identified pixel by pixel. The object detection result by instance segmentation may include, for each detected object, two-dimensional data (mask score) indicating the region in which the object is captured, a score indicating the likelihood of the object, a mask obtained by binarizing the mask score using a threshold, and information indicating the bounding box. The size of the bounding box may be adjustable. The size of the mask for each individual may be adjustable by setting a score threshold. The region of each individual can be identified to be larger by specifying a larger bounding box size or setting a lower score threshold. The object detection result may also include the reliability of the object detection result.

[0026] A machine learning model that performs instance segmentation is an example of an “individual region segmentation model.” Also, a mask score included in an object detection result by instance segmentation is an example of an “individual region segmentation result.”

[0027] A machine learning model for performing instance segmentation can be Mask R-CNN (Mask Region-Based Convolutional Neural Networks) or YOLACT (You Only Look At CoefficienTs). Details of Mask R-CNN are disclosed in Reference 1.

[0028] [Reference 1] Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick, "Mask R-CNN", Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.

[0029] Semantic segmentation is a task of dividing an object constituting an image into regions that indicate one or more types (classes). In semantic segmentation, a label (class label) indicating a class is predicted for each unit (e.g., pixel) in an image. The result of region division by semantic segmentation can include two-dimensional data (semantic score) including a class label corresponding to each pixel of the image. The result of region division can also include a confidence level for the region division result.

[0030] The machine learning model that performs semantic segmentation is an example of a “type-specific region segmentation model.” Also, the semantic score included in the region segmentation result by semantic segmentation is an example of a “type-specific region segmentation result.”

[0031] U-Net, DeepLab, or the like can be used as a machine learning model for semantic segmentation. Details of U-Net are disclosed in Reference 2. Details of DeepLab (DeepLabv3) are disclosed in Reference 3.

[0032] [Reference 2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation", Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015. [Reference 3] LC Chen, G. Papandreou, F. Schroff, and H. Adam, "Rethinking Atrous Convolution for Semantic Image Segmentation", European Conference on Computer Vision (ECCV), 2017.

[0033] When instance segmentation is performed on an image of a large number of fibers dispersed on a sample surface, the detection accuracy may be low. For example, if the image contains multiple fibers that are entangled with each other, it may not be possible to accurately detect each fiber. Also, for example, if the background of the image contains linear stains or streaks, the stains may be erroneously recognized as fibers.

[0034] Fig. 1 is a diagram showing an example of a target image in this embodiment. As shown in Fig. 1, a target image 900 in this embodiment is an image obtained by capturing a state in which a large number of fibers are dispersed on a metal surface. In the target image 900, a plurality of fibers 901 entangled with each other and a streak 902 that is not a fiber are captured.

[0035] FIG. 2 is a diagram showing an example of a conventional object detection result. FIG. 2 shows the result of detecting fibers from the target image shown in FIG. 1 using Mask R-CNN. In the object detection result shown in FIG. 2, differences in objects are expressed by differences in color. As shown in FIG. 2, in the conventional object detection result 910, an area 911 in which multiple fibers intertwined with each other are captured is detected as multiple separated fibers. Also, an area 912 in which a line that is not a fiber is captured is detected as a fiber.

[0036] The object detection system 1 of this embodiment aims to accurately recognize fibers included in an image. In particular, the object detection system 1 of this embodiment accurately detects multiple fibers that are entangled with each other or multiple fibers that are overlapped with each other. Furthermore, the object detection system 1 of this embodiment reduces erroneous recognition of linear stains or streaks included in the background.

[0037] <Overall configuration of object detection system> The overall configuration of the object detection system in this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the overall configuration of the object detection system in this embodiment.

[0038] 3, the object detection system 1 in this embodiment includes an image capture device 10, an image processing device 20, and a user terminal 30. The image capture device 10, the image processing device 20, and the user terminal 30 are connected to each other so as to be able to communicate data with each other via a communication network N1 such as a LAN (Local Area Network) or the Internet.

[0039] The image acquisition device 10 is an optical device that acquires an image of a target object to be detected. The image acquisition device 10 may be a digital camera that captures still images, or a video camera that captures moving images. The image acquisition device 10 may be an optical microscope, a scanning electron microscope (SEM), a transmission electron microscope (TEM), or the like, depending on the size of the object to be detected. The image acquisition device 10 may also be an information processing device such as a personal computer connected to various cameras, or an inspection device equipped with various cameras.

[0040] The image processing device 20 is an information processing device such as a personal computer, a workstation, or a server that generates an output image indicating the result of detecting an object from a target image acquired by the image acquisition device 10. The image processing device 20 receives the target image from a user terminal 30. The image processing device 20 detects an object from the acquired target image, and transmits an output image indicating the detection result to the user terminal 30.

[0041] The user terminal 30 is an information processing terminal such as a personal computer, a tablet terminal, or a smartphone operated by a user. The user terminal 30 acquires a target image from the image acquisition device 10 in response to a user's operation, and transmits the target image to the image processing device 20. The user terminal 30 receives an output image indicating a detection result from the image processing device 20, and outputs the output image to the user.

[0042] The overall configuration of the object detection system 1 shown in Fig. 3 is an example, and various system configuration examples are possible depending on the application and purpose. For example, the object detection system 1 may include a plurality of image acquisition devices 10, image processing devices 20, and user terminals 30. For example, the image processing device 20 may be realized by a plurality of computers, or may be realized as a cloud computing service. The division of devices such as the image acquisition device 10, image processing device 20, and user terminal 30 shown in Fig. 3 is an example.

[0043] <Hardware configuration of object detection system> The hardware configuration of the object detection system 1 in this embodiment will be described with reference to FIG.

[0044] <Computer hardware configuration> The image acquisition device 10, the image processing device 20, and the user terminal 30 in this embodiment are realized by, for example, a computer. Fig. 4 is a block diagram showing an example of the hardware configuration of a computer 500 in this embodiment.

[0045] 4, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. Each piece of hardware of the computer 500 is connected to each other via a bus line 509. The input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0046] The CPU 501 is a calculation device that reads out programs and data from a storage device such as a ROM 502 or a HDD 504 onto a RAM 503 and executes processes, thereby realizing overall control and functions of the computer 500 .

[0047] The ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 502 functions as a main storage device that stores various programs and data required for the CPU 501 to execute various programs installed in the HDD 504. Specifically, the ROM 502 stores boot programs such as a Basic Input / Output System (BIOS) and an Extensible Firmware Interface (EFI) that are executed when the computer 500 is started, and data such as an Operating System (OS) setting and a network setting.

[0048] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 503 provides a working area in which various programs installed in the HDD 504 are expanded when the CPU 501 executes them.

[0049] The HDD 504 is an example of a non-volatile storage device that stores programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device that uses a flash memory as a storage medium (e.g., an SSD (Solid State Drive) or the like) instead of the HDD 504.

[0050] The input device 505 includes a touch panel, operation keys or buttons, a keyboard or mouse, a microphone for inputting sound data such as voice, etc., which are used by the user to input various signals.

[0051] The display device 506 is composed of a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display for displaying a screen, a speaker for outputting sound data such as voice, and the like.

[0052] The communication I / F 507 is an interface that connects to a communication network and enables the computer 500 to perform data communication.

[0053] The external I / F 508 is an interface with external devices, such as a drive device 510.

[0054] The drive device 510 is a device for setting the recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. The recording medium 511 may also include semiconductor memories that record information electrically, such as a ROM or a flash memory. This allows the computer 500 to read and / or write data from and to the recording medium 511 via the external I / F 508.

[0055] The various programs to be installed in the HDD 504 are installed, for example, by setting the distributed recording medium 511 in a drive device 510 connected to the external I / F 508 and reading out the various programs recorded in the recording medium 511 by the drive device 510. Alternatively, the various programs to be installed in the HDD 504 may be installed by being downloaded from a network different from the communication network via the communication I / F 507.

[0056] <Functional configuration of object detection system> The functional configuration of the object detection system in this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the functional configuration of the object detection system 1 in this embodiment.

[0057] <Functional configuration of the image acquisition device> As shown in FIG. 5, the image acquisition device 10 in this embodiment includes an imaging control unit 101 and an image storage unit 102.

[0058] The imaging control unit 101 is realized by a camera connected to the external I / F 508 shown in Fig. 4. The image storage unit 102 is realized by the HDD 504 shown in Fig. 4.

[0059] The imaging control unit 101 adjusts the imaging conditions (angle of view, imaging magnification, etc.) of the camera so that multiple fibers can be imaged, and captures the target image. The imaging control unit 101 may capture the target image in response to a user's operation, or may capture the target image when a predetermined condition is satisfied. The predetermined condition is, for example, when an inspection target is transported to a predetermined position in the inspection device. The imaging control unit 101 may capture a still image, or may capture a video and extract an image in which multiple fibers are reflected.

[0060] The image storage unit 102 stores a target image captured by the imaging control unit 101. Information about the target image may be stored in association with the target image in the image storage unit 102. The information about the target image may be, for example, the imaging date and time, information indicating the imaging target, imaging conditions, and the like.

[0061] <Functional configuration of image processing device> As shown in FIG. 5, the image processing device 20 in this embodiment includes a model memory unit 200, an image acquisition unit 201, an object detection unit 202 (an example of a first area division unit), an area division unit 203 (an example of a second area division unit), an area correction unit 204, and a result output unit 205.

[0062] The model storage unit 200 is realized by the HDD 504 shown in Fig. 4. The image acquisition unit 201, the object detection unit 202, the area division unit 203, the area correction unit 204, and the result output unit 205 are realized by processing that is executed by the CPU 501 in accordance with a program loaded from the HDD 504 onto the RAM 503 shown in Fig. 4.

[0063] The model storage unit 200 stores a trained object detection model and a trained area segmentation model.

[0064] The object detection model is a machine learning model that performs instance segmentation. The object detection model takes an image as input, and outputs an object detection result that detects each object included in the image. The object detection result includes an area segmentation result for each individual that recognizes the area in which each object is captured. As an example of the object detection model in this embodiment, Mask R-CNN is used. The object detection model is also referred to as an area segmentation model for each individual.

[0065] The object detection model in this embodiment is trained to detect each fiber captured in an input image from the image. The object detection model is trained using training data in which training labels are assigned to images of a sample surface on which multiple fibers are distributed. The training labels include information indicating the range of the area in which each fiber is captured and identification information for identifying the fiber.

[0066] The region segmentation model is a machine learning model that performs semantic segmentation. The region segmentation model receives an image as input, and outputs a region segmentation result for each type that recognizes a region in which a desired object is captured in the image. As an example of the region segmentation model in this embodiment, DeepLabv3 is used. The region segmentation model is also referred to as a region segmentation model for each type.

[0067] The region segmentation model in this embodiment is trained to divide an input image into a region in which any fiber is captured and other regions. The region segmentation model is trained using training data in which training labels are assigned to images of a metal surface on which multiple fibers are distributed. The training labels include information indicating the range of the region in which the fibers are captured.

[0068] The image acquiring unit 201 acquires the target image by receiving the target image from the user terminal 30. The image acquiring unit 201 may acquire the target image from the image acquisition device 10 in response to a request from the user terminal 30.

[0069] The object detection unit 202 uses a trained object detection model read from the model storage unit 200 to detect individual fibers contained in the target image acquired by the image acquisition unit 201, and generates object detection results corresponding to each fiber contained in the target image.

[0070] The area division unit 203 uses the trained area division model read out from the model memory unit 200 to recognize an area in which fibers are captured (hereinafter also referred to as a "fiber area") from the target image acquired by the image acquisition unit 201, and generates an area division result indicating the fiber area.

[0071] Area correction unit 204 corrects an area where fibers are detected (hereinafter also referred to as a "fiber detection area") in the object detection result generated by object detection unit 202. Area correction unit 204 may correct the fiber detection area, for example, by integrating the object detection result generated by object detection unit 202 and the area division result generated by area division unit 203. Area correction unit 204 may correct the fiber detection area by, for example, correcting the fiber detection area through expansion processing or smoothing processing, and then integrating the fiber detection area with the area division result generated by area division unit 203.

[0072] The result output unit 205 transmits an output image indicating the result of the correction by the area correction unit 204 to the user terminal 30 .

[0073] <Functional configuration of user terminal 30> As shown in FIG. 5, the user terminal 30 in this embodiment includes an image transmission unit 301 and a result display unit 302.

[0074] The image transmission unit 301 and the result display unit 302 are realized by processing that is executed by the CPU 501 according to a program loaded onto the RAM 503 from the HDD 504 shown in FIG.

[0075] In response to a user's operation, the image transmission unit 301 acquires a target image from the image acquisition device 10. The image transmission unit 301 transmits the target image acquired from the image acquisition device 10 to the image processing device 20.

[0076] The result display unit 302 receives the output image from the image processing device 20. The result display unit 302 displays the received output image on the display device 506.

[0077] <Processing procedure of object detection system> The processing procedure of the image processing method executed by the object detection system 1 in this embodiment will be described with reference to Fig. 6 to Fig. 11. Fig. 6 is a flowchart showing an example of the processing procedure of the image processing method in this embodiment.

[0078] In step S1, the imaging control unit 101 of the image acquisition device 10 adjusts the angle of view of the camera so that a plurality of fibers are captured, and captures an image of the target. Next, the imaging control unit 101 stores the captured image of the target in the image storage unit 102.

[0079] In step S2, the image transmission unit 301 of the user terminal 30 transmits a request to acquire a target image to the image acquisition device 10 in response to a user operation. In response to the acquisition request received from the user terminal 30, the image acquisition device 10 reads out the target image stored in the image storage unit 102 and transmits it to the user terminal 30. The image transmission unit 301 transmits the target image received from the image acquisition device 10 to the image processing device 20.

[0080] In step S3, the image acquisition unit 201 of the image processing device 20 receives the target image from the user terminal 30. Next, the image acquisition unit 201 sends the received target image to the object detection unit 202 and the area division unit 203.

[0081] In step S4, the object detection unit 202 of the image processing device 20 receives the target image from the image acquisition unit 201. Next, the object detection unit 202 reads out from the model storage unit 200 the trained object detection model.

[0082] Next, the object detection unit 202 inputs the target image to the trained object detection model. The object detection model detects individual fibers included in the target image and outputs an object detection result for the target image. In this way, the object detection unit 202 obtains the object detection result for the target image. Then, the object detection unit 202 sends the object detection result to the area correction unit 204.

[0083] The object detection result of the target image includes a detection result corresponding to each fiber detected from the target image and a reliability of the object detection result. The detection result of each fiber includes a mask score indicating a fiber detection region and a score indicating an object-likelihood. The detection result of each fiber may also include information indicating a mask and a bounding box obtained by binarizing the mask score using a threshold value.

[0084] The object detection unit 202 may discard the detection results corresponding to each fiber whose score (0 to 1) indicating the object-likeness is equal to or less than a predetermined threshold. The threshold may be set arbitrarily, but may be set to 0.3, for example.

[0085] In step S5, the area division unit 203 of the image processing device 20 receives the target image from the image acquisition unit 201. Next, the area division unit 203 reads out from the model storage unit 200 the area division model that has been trained.

[0086] Next, the area division unit 203 inputs the target image to the trained area division model. The area division model recognizes a fiber area from the target image and outputs the area division result of the target image. In this way, the area division unit 203 obtains the area division result of the target image. Then, the area division unit 203 sends the area division result to the area correction unit 204.

[0087] The segmentation result of the target image includes a semantic score indicating the fiber region and a confidence level of the segmentation result.

[0088] In step S6, the area correction unit 204 of the image processing device 20 receives the object detection result from the object detection unit 202. The area correction unit 204 also receives the area division result from the area division unit 203. Next, the area correction unit 204 corrects the object detection result based on the area division result.

[0089] For example, the area correction unit 204 corrects the object detection result by expanding the fiber detection area by dilation processing or smoothing processing. Here, an example of processing for correcting the fiber detection area using dilation processing will be described with reference to Figs. 7 to 10.

[0090] Fig. 7 is a diagram showing an example of a mask score before expansion. As shown in Fig. 7, a fiber detection region in which one fiber is detected is shown in the mask score 931 before expansion. It can be seen that the mask score 931 shown in Fig. 7 has both ends broken and the peripheral portion is not clear.

[0091] First, the area correction unit 204 performs threshold processing on the mask score 931 to binarize it. Fig. 8 is a diagram showing an example of the mask score after binarization. As shown in Fig. 8, the mask score 932 after binarization is binarized so that the fiber detection area is binarized to 1 and the other areas are binarized to 0. It can be seen that the peripheral part of the fiber is clearly defined in the mask score 932 after binarization.

[0092] The mask score after binarization may be output by an object detection model.

[0093] Next, the region correction unit 204 executes an expansion process on the mask score. This expands the fiber detection region. FIG. 9 is a diagram showing an example of the mask score after expansion. As shown in FIG. 9, it can be seen that the fiber detection region is generally expanded in the mask score 933 after expansion compared to the mask score 931 before expansion shown in FIG. 7.

[0094] The above-mentioned expansion process can also be performed by specifying a larger bounding box size or setting a lower score threshold.

[0095] Then, the region correction unit 204 generates a corrected mask score based on the binarized mask score 932 and the dilated mask score 933. Specifically, the region correction unit 204 adopts the binarized mask score 932 inside a region of interest (ROI), and adopts the dilated mask score 933 outside the region of interest. Fig. 10 is a diagram showing an example of a corrected mask score. As shown in Fig. 10, in the corrected mask score 934, both ends and peripheral parts of the fiber are clear, and it can be seen that the outline of the fiber has been detected with high accuracy.

[0096] Returning to FIG. 6, in step S7, the area correction unit 204 of the image processing device 20 integrates the area division result and the corrected object detection result. As a result, the area correction unit 204 obtains the corrected result of the object detection result. Then, the area correction unit 204 sends the corrected result of the object detection result to the result output unit 205.

[0097] For example, the area correction unit 204 may integrate the area division result and the corrected object detection result by calculating the logical product of the object detection result and the area division result. In this case, the area correction unit 204 calculates the logical product of each of the object detection results corresponding to each fiber and the area division result. This generates a detection result corresponding to each fiber (hereinafter also referred to as an "individual detection result").

[0098] Also, for example, the region correction unit 204 may integrate the region segmentation result and the corrected object detection result by selecting the object detection result or the region segmentation result based on the result of comparing the score of the object detection result and the score of the region segmentation result for each pixel of the target image. In this case, the region correction unit 204 compares the mask score of each object detection result corresponding to each fiber with the semantic score of the region segmentation result for each pixel of the target image, and selects the result with the higher score if the scores are different, and selects the result with the higher reliability if the scores are equal. In this way, individual detection results corresponding to each fiber are generated.

[0099] In addition, the area correction unit 204 may reject an individual detection result with a lower mask score if the area of ​​overlap between the individual detection results corresponding to each fiber and the individual detection results corresponding to other fibers is equal to or greater than a predetermined threshold. This is because detection results with a large overlapping area are considered to be detecting the same fiber. The threshold may be set arbitrarily, but may be set to, for example, 0.7 (i.e., reject a detection result if 70% of the area overlaps).

[0100] Then, the region correction unit 204 generates a corrected result by overlaying the individual detection results corresponding to each fiber from top to bottom in descending order of the score indicating the object-likeliness. Finally, the region correction unit 204 adds, as a fiber region, any fiber region shown in the region division result that is not shown as a fiber detection region in the object detection result to the corrected result.

[0101] The region correction unit 204 may calculate the number and distribution of fiber lengths and fiber diameters, as well as statistical values ​​such as the average value, median value, and standard deviation for the fiber regions included in the correction results.

[0102] Fig. 11 is a diagram showing an example of a detection result in this embodiment. Fig. 11 shows the result of detecting fibers by the image processing device 20 from the target image shown in Fig. 1. As shown in Fig. 11, in the detection result 920 in this embodiment, an area 921 in which an image of a plurality of fibers entangled with each other is captured is detected as a plurality of fibers entangled with other fibers. Moreover, an area 922 in which an image of a streak that is not a fiber is captured is not detected as a fiber.

[0103] Returning to FIG. 6, in step S8, the result output unit 205 of the image processing device 20 receives the correction result from the area correction unit 204. Next, the result output unit 205 generates an output image indicating the detection result of the fibers captured in the target image based on the correction result. Then, the result output unit 205 transmits the output image to the user terminal 30.

[0104] The output image may be a mask image in which identification information for identifying the detected fiber is set for each pixel.The output image may be an image in which the area of ​​each detected fiber and identification information for identifying each fiber are shown in the target image.The output image may be an image in which a probability value is shown for each detected fiber.

[0105] In step S9, the result display unit 302 of the user terminal 30 receives the output image from the image processing device 20. Then, the result display unit 302 displays the received output image on the display device 506. The result display unit 302 may display the above-mentioned statistical values ​​calculated for each fiber region at the request of the user.

[0106] <User interface for object detection system> The user interface of the object detection system 1 in this embodiment will be described with reference to FIGS.

[0107] ≪Analysis screen≫ 12 is a diagram showing an example of an analysis screen. The analysis screen is a screen for inputting a target image to be analyzed and executing object detection for the target image.

[0108] As shown in FIG. 12, an analysis screen 1000 has a target selection field 1001 , a target input field 1002 , a reference button 1003 , an analysis execution button 1004 , four confirmation buttons 1011 to 1014 , and an output button 1015 .

[0109] In the initial display, the target selection field 1001, the target input field 1002, and the reference button 1003 are controlled so that they can be pressed or input. On the other hand, in the initial display, the analysis execution button 1004, the confirmation buttons 1011 to 1014, and the output button 1015 are controlled so that they cannot be pressed.

[0110] The target selection field 1001 is an input field for specifying a method for selecting a target image. In the example of Fig. 12, a file and a folder can be selected as options for the analysis target. When a file is selected, it is possible to select one file as the target image. When a folder is selected, all files stored in the specified folder are selected as the target images.

[0111] The target input field 1002 is an input field for specifying the position of the target image. When a file is selected in the target selection field 1001, information indicating the position of the file is input in the target input field 1002. When a folder is selected in the target selection field 1001, information indicating the position of the folder is input in the target input field 1002. The information indicating the position of the file or folder is, for example, a local path or a URL (Uniform Resource Locator) of the file or folder.

[0112] Pressing a browse button 1003 displays a file selection dialog. By selecting a file or folder in the file selection dialog, information indicating the location of the file or folder is entered in a target input field 1002. When information indicating the location of the file or folder is entered in the target input field 1002, an analysis execution button 1004 is controlled to be depressible.

[0113] The analysis execution button 1004 is a button for executing object detection for a target image. The target image is a file input in the target input field 1002 or a file stored in a folder input in the target input field 1002. When object detection for the target image is completed, the confirmation buttons 1011 to 1014 and the output button 1015 are controlled to be pressable.

[0114] Pressing the Confirm button 1011 displays an image confirmation screen for displaying the target image. Pressing the Confirm button 1012 displays a first result confirmation screen for displaying the region segmentation results for each type. Pressing the Confirm button 1013 displays a second result confirmation screen for displaying the region segmentation results for each individual. Pressing the Confirm button 1014 displays a correction result confirmation screen for displaying the correction results. Pressing the Output button 1015 saves electronic data showing the analysis results for the target image.

[0115] <Image confirmation screen> 13 is a diagram showing an example of an image confirmation screen 1100. As shown in FIG.

[0116] The target image is displayed in an image display field 1101. When an end button 1102 labeled "close" is pressed, the image confirmation screen 1100 closes and the analysis screen 1000 is displayed.

[0117] <First result confirmation screen> 14 is a diagram showing a first example of a first result confirmation screen 1200. As shown in FIG.

[0118] Image display field 1201 displays the region segmentation results for each type for the target image. When an end button 1202 labeled "Close" is pressed, first result confirmation screen 1200 ends and analysis screen 1000 is displayed. When a confirmation button 1203 labeled "Confirm Accuracy" is pressed, a score indicating the recognition accuracy of each object is displayed in image display field 1201.

[0119] Fig. 15 is a diagram showing a second example of the first result confirmation screen. A first result confirmation screen 1200 in the second example further includes a threshold setting field 1204 and a reflect button 1205, compared to Fig. 14 showing the first example of the first result confirmation screen described above.

[0120] The threshold setting field 1204 is an input field for setting a threshold for binarizing the semantic score. When a reflect button 1205 is pressed, an image in which the semantic score has been binarized based on the threshold set in the threshold setting field 1204 is displayed in the image display field 1201.

[0121] The threshold value set by operating the threshold value setting field 1204 and the reflect button 1205 is used only to confirm the detection result on the analysis screen 1000. Therefore, the set threshold value is not used continuously in the object detection system 1.

[0122] <<Second result confirmation screen>> 16 is a diagram showing a first example of a second result confirmation screen 1300. As shown in FIG. 16, a second result confirmation screen 1300 has an image display field 1301, an end button 1302, and a confirmation button 1303.

[0123] Image display field 1301 displays the region segmentation results for each individual in the target image. When an end button 1302 labeled "Close" is pressed, second result confirmation screen 1300 ends and analysis screen 1000 is displayed. When a confirmation button 1303 labeled "Confirm Accuracy" is pressed, a score indicating the recognition accuracy of each object is displayed in image display field 1301.

[0124] Fig. 17 is a diagram showing a second example of the second result confirmation screen. A second result confirmation screen 1300 in the second example further includes a first threshold setting field 1304, a second threshold setting field 1305, and a reflect button 1306, compared to Fig. 16 showing the first example of the second result confirmation screen described above.

[0125] The first threshold setting field 1304 is an input field for setting a threshold for a score indicating object-likeness. The second threshold setting field 1305 is an input field for setting a threshold for binarizing the mask score. When the reflect button 1306 is pressed, a detection result with a low score indicating object-likeness is rejected based on the thresholds set in the first threshold setting field 1304 and the second threshold setting field 1305, and an image with the binarized mask score is displayed in the image display field 1301.

[0126] The thresholds set by operating the first threshold setting field 1304, the second threshold setting field 1305, and the reflect button 1306 are used only to confirm the detection result on the analysis screen 1000. Therefore, the set thresholds are not used continuously in the object detection system 1.

[0127] <Correction result confirmation screen> 18 is a diagram showing an example of a correction result confirmation screen. As shown in FIG. 18, a correction result confirmation screen 1400 has an image display field 1401, an end button 1402, a confirmation button 1403, a fiber display button 1404, and an entanglement display button 1405.

[0128] The image display field 1401 displays the correction results of the area segmentation results for each individual. When an end button 1402 labeled "Close" is pressed, the correction result confirmation screen 1400 ends and the analysis screen 1000 is displayed. When a confirmation button 1403 labeled "Confirm accuracy" is pressed, a score indicating the recognition accuracy of each object is displayed in the image display field 1401.

[0129] When the fiber display button 1404 marked with "only fibers" is pressed, the detection result excluding a plurality of fibers that are entangled with each other among the fibers detected in the target image is displayed in the image display column 1401. When the entanglement display button 1405 marked with "fibers + entanglement" is pressed, the detection result including a plurality of fibers that are entangled with each other among the fibers detected in the target image is displayed in the image display column 1401.

[0130] <Comparison between U-Net and DeepLab> As a machine learning model for performing semantic segmentation, for example, U-Net or DeepLab is used. In this embodiment, either U-Net or DeepLab can be used as the region segmentation model. However, if DeepLabv3 is adopted as the region segmentation model, the possibility of improving the accuracy is increased. The reasons are explained below.

[0131] In order to extract the global features of the input image, U-Net performs upsampling multiple times in the encoder and downsampling the same number of times in the decoder. The upsampling in the encoder performs the convolution process that is standardly used in convolutional neural networks.

[0132] On the other hand, DeepLabv3 has a structure that extracts the global features of the input image while suppressing the computational amount by taking a large kernel size through dilated convolution. Note that dilated convolution is a convolution process that reduces the number of parameters by taking a large kernel while setting a part of the kernel to 0.

[0133] As described above, by introducing dilated convolution, DeepLabv3 can incorporate more global features. As a result, it is expected that DeepLabv3 can effectively incorporate the global features of the image more than U-Net, and the recognition accuracy of the region can be improved.

[0134] The comparison results between U-Net and DeepLabv3 will be described with reference to Figs. 19 to 21. Fig. 19 is a diagram showing an example of a target image used in the comparison. Fig. 20 is a diagram showing an example of a region segmentation result by U-Net. Fig. 21 is a diagram showing an example of a region segmentation result by DeepLabv3.

[0135] As shown in Fig. 19, the target image 940 used for comparison is an image capturing a state in which many fibers are dispersed. As shown in Fig. 20, in the region segmentation result 950 by U-Net, there are fibers that cannot be accurately recognized. For example, the target image 940 has a region 941 that includes relatively thick fibers and relatively thin fibers. In the region segmentation result 950 by U-Net, in a region 951 corresponding to the region 941 in the target image 940, the relatively thin fibers are interrupted, and it can be seen that they cannot be accurately recognized.

[0136] 21, in the region segmentation result 960 by DeepLabv3, the recognition accuracy of the fibers captured in the target image 940 is improved compared to U-Net. For example, in the region segmentation result 960 by DeepLabv3, it can be seen that all fibers are accurately recognized in the region 961 corresponding to the region 941 in the target image 940.

[0137] <Effects of the embodiment> The image processing device 20 in this embodiment corrects the area division result by the area division model for each individual with the area division result by the area division model for each type. The area division model for each individual can identify the area of ​​each object included in the image, and the area division model for each type can recognize the contour of the object captured in the image with high accuracy. Therefore, by correcting the area division result for each individual with the area division result for each type, it is possible to detect each fiber whose contour is recognized with high accuracy. Therefore, according to the image processing device 20 in this embodiment, it is possible to recognize the fibers included in the image with high accuracy.

[0138] The image processing device 20 in this embodiment is considered to be particularly effectively used when appropriately evaluating fiber materials under the RoHS Directive and REACH Regulation.

[0139] The image processing device 20 in this embodiment expands the area where fibers are detected in the area segmentation result for each individual. The area segmentation model for each individual tends to have a low detection accuracy at the peripheral part of the detected object, but if the area segmentation result for each individual is expanded to the outside of the peripheral part of the object, it can be complemented with the contour of the object recognized with high accuracy by the area segmentation model for each type. Therefore, the image processing device 20 in this embodiment can accurately recognize the peripheral part of the fiber or the entanglement part of multiple fibers.

[0140] The image processing device 20 in this embodiment calculates the logical product of the area segmentation result for each individual and the area segmentation result for each type. That is, fibers recognized in both the area segmentation result for each individual and the area segmentation result for each type are included in the correction result. Therefore, the image processing device 20 in this embodiment can reduce the possibility of erroneously detecting background stains that should not be detected.

[0141] The image processing device 20 in this embodiment selects the area segmentation result for each individual or the area segmentation result for each type based on the result of comparing the score of the area segmentation result for each individual with the score of the area segmentation result for each type for each pixel of the target image. That is, the result with the higher score between the area segmentation result for each individual and the area segmentation result for each type can be included in the correction result. Therefore, the image processing device 20 in this embodiment can reduce the possibility of erroneously detecting background stains that should not be detected.

[0142] In this embodiment, the image analysis is described for fibers, but it can be performed for particles as well. For example, even if particles overlap, the particles can be accurately identified.

[0143] [supplement] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), conventional circuit module, and other devices designed to execute each function described above.

[0144] Although the embodiments of the present disclosure have been described in detail above, the embodiments disclosed herein are illustrative and not restrictive in all respects. The embodiments can be modified and improved in various forms without departing from the scope and spirit of the appended claims. The matters described in the above embodiments can be configured in other ways as long as they are not inconsistent, and can be combined as long as they are not inconsistent.

[0145] This application claims priority to Japanese Patent Application No. 2023-50607, filed with the Japan Patent Office on March 27, 2023, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0146] 1. Object Detection System 10 Image acquisition device 101 Imaging control unit 102 Image storage unit 20 Image processing device 200 Model memory section 201 Image Acquisition Unit 202 Object detection unit 203 Area division part 204 Area correction section 205 Result output section 30 User terminals 301 Image transmission unit 302 Result display section

Claims

1. an image acquisition unit configured to acquire a target image of a plurality of fibers; A first region division unit configured to generate a region division result for each individual in which each of the fibers included in the target image is detected, using a trained region division model for each individual; A second region division unit configured to generate a region division result for each type by recognizing a region in which the plurality of fibers are imaged from the target image by using a trained region division model for each type; an area correction unit configured to correct the area division result for each individual based on the area division result for each type; a result output unit configured to output a correction result of the region segmentation result for each individual; An image processing device comprising:

2. 2. The image processing device according to claim 1, The area correction unit is configured to generate the correction result by calculating a logical product of the area division result for each individual and the area division result for each type. Image processing device.

3. 2. The image processing device according to claim 1, The area correction unit is configured to generate the correction result by selecting the area segmentation result for each individual or the area segmentation result for each type based on a result of comparing a score of the area segmentation result for each individual with a score of the area segmentation result for each type for each unit of the target image. Image processing device.

4. 4. The image processing device according to claim 1, The individual region segmentation model is a model that performs instance segmentation, The region division model for each type is a model for performing semantic segmentation. Image processing device.

5. 5. The image processing device according to claim 4, The individual segmentation model is Mask R-CNN or YOLACT; Image processing device.

6. 5. The image processing device according to claim 4, The size of the bounding box of the individual segmentation model is adjustable. Image processing device.

7. 5. The image processing device according to claim 4, The size of the mask of each individual in the region segmentation model is adjustable. Image processing device.

8. 5. The image processing device according to claim 4, The area division model for each type is DeepLab or U-Net; Image processing device.

9. 4. The image processing device according to claim 1, The area correction unit is configured to correct the area in which the fiber is detected in the area division result for each individual. Image processing device.

10. 10. The image processing device according to claim 9, The area correction unit is configured to enlarge the area divided for each individual by a dilation process or a smoothing process. Image processing device.

11. The computer acquiring a target image of a plurality of fibers; generating an individual region segmentation result for each of the fibers included in the target image by using a trained individual region segmentation model; A step of generating a region segmentation result for each type by recognizing a region in which the plurality of fibers are imaged from the target image using a trained region segmentation model for each type; A step of correcting the result of the area segmentation for each individual using the result of the area segmentation for each type; A step of outputting a correction result of the region segmentation result for each individual; An image processing method that performs

12. On the computer, acquiring a target image of a plurality of fibers; generating an individual region segmentation result for each of the fibers included in the target image by using a trained individual region segmentation model; A step of generating a region segmentation result for each type by recognizing a region in which the plurality of fibers are imaged from the target image using a trained region segmentation model for each type; A step of correcting the result of the area segmentation for each individual using the result of the area segmentation for each type; A step of outputting a correction result of the region segmentation result for each individual; A program for executing the above.

Images