Image processing device, image processing method, and program

Abstract

This image processing device is provided with: an image acquisition unit that acquires a target image in which a plurality of fibers are captured; a first region division unit that generates a region division result for each individual, in which each fiber included in the target image is detected, using the region division model for each individual that has been learned; a second region division unit that generates a region division result for each type in which a region in which a plurality of fibers are captured is recognized from the target image, using the region division model for each type that has been learned; a region correction unit that corrects the result of the region division for each individual by means of the result of the region division for each type; and a result output unit that outputs a correction result of the region division result for each individual.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, and a program. Background Art

[0002] A technique for performing image segmentation using a machine learning model is known. In image segmentation, the segmentation result may be corrected in order to improve accuracy.

[0003] For example, Patent Document 1 discloses a metal structure segmentation method using a learned machine learning model. The segmentation method disclosed in Patent Document 1 determines a correction close to a correct segmentation result based on the segmentation result of a selected image, and applies the correction to the segmentation result of images other than the selected image.

[0004] <Prior Art Literature>

[0005] <Patent Documents>

[0006] Patent Document 1: Japanese Patent Application Publication No. 2022-188504 Summary of the invention

[0007] <Problems to be Solved by the Invention>

[0008] However, the prior art has a problem of low accuracy in detecting individual fibers from an image obtained by photographing multiple fibers. For example, in the case of a portion where multiple fibers are entangled with each other, the multiple fibers entangled with each other cannot be correctly identified. Similarly, in the case of a portion where multiple fibers overlap with each other, the multiple fibers overlapping with each other cannot be correctly identified. In addition, when there is linear dirt on the background, it may be mistakenly detected as a fiber.

[0009] An object of one embodiment of the present invention is to recognize fibers included in an image with high accuracy.

[0010] <Methods used to solve the problem>

[0011] The present invention has the following structure.

[0012] [1] An image processing device comprising:

[0013] An image acquisition unit configured to acquire an image of a subject in which a plurality of fibers are photographed;

[0014] a first region segmentation unit configured to generate a region segmentation result for each individual fiber detected in the target image using the learned region segmentation model for each individual fiber;

[0015] A second region segmentation unit is configured to generate a region segmentation result for each type by identifying a region where the plurality of fibers are imaged from the target image using the learned region segmentation model for each type;

[0016] an area correction unit configured to correct the area segmentation result for each individual by using the area segmentation result for each type; and

[0017] The result output unit is configured to output a correction result of the region segmentation result for each individual.

[0018] [2] The image processing device according to [1] above, wherein:

[0019] The area correction unit is configured to generate the correction result by calculating a logical product of the area division result for each individual and the area division result for each type.

[0020] [3] The image processing device according to [1] above, wherein:

[0021] The area correction unit is configured to select, for each unit of the object image, the area segmentation result according to each individual or the area segmentation result according to each category based on a result of comparing the score of the area segmentation result according to each individual with the score of the area segmentation result according to each category, thereby generating the correction result.

[0022] [4] An image processing device according to any one of [1] to [3] above, wherein:

[0023] The region segmentation model for each individual is a model for performing instance segmentation.

[0024] The region segmentation model for each category is a model for performing semantic segmentation.

[0025] [5] The image processing device according to [4] above, wherein:

[0026] The region segmentation model for each individual is Mask R-CNN or YOLACT.

[0027] [6] The image processing device according to [4] or [5] above, wherein:

[0028] The region segmentation model for each individual can adjust the size of the bounding box.

[0029] [7] An image processing device according to any one of [4] to [6] above, wherein:

[0030] The region segmentation model for each individual can adjust the size of the mask of each individual.

[0031] [8] An image processing device according to any one of [4] to [7] above, wherein:

[0032] The region segmentation model for each category is DeepLab or U-Net.

[0033] [9] An image processing device according to any one of [1] to [8] above, wherein:

[0034] The region correction unit is configured to correct the region in which the fibers are detected in the region division result for each individual.

[0035]

[10] The image processing device according to [9] above, wherein:

[0036] The region correction unit is configured to enlarge the region divided for each individual by dilation processing or smoothing processing.

[0037]

[11] An image processing method, in which a computer performs the following process:

[0038] A process for obtaining an image of an object having a plurality of fibers;

[0039] A process of generating a result of detecting the individual fibers contained in the target image by using the learned individual fiber segmentation model;

[0040] A process of generating a region segmentation result for each type by identifying a region where the plurality of fibers are photographed from the target image using the learned region segmentation model for each type;

[0041] A process of correcting the region segmentation result for each individual by using the region segmentation result for each category; and

[0042] A process of outputting the correction result of the region segmentation result for each individual.

[0043]

[12] A program for causing a computer to execute the following process:

[0044] A process for obtaining an image of an object having a plurality of fibers;

[0045] A process of generating a result of detecting the individual fibers contained in the target image by using the learned individual fiber segmentation model;

[0046] A process of generating a region segmentation result for each type by identifying a region where the plurality of fibers are photographed from the target image using the learned region segmentation model for each type;

[0047] A process of correcting the region segmentation result for each individual by using the region segmentation result for each category; and

[0048] A process of outputting the correction result of the region segmentation result for each individual.

[0049] <Effects of the Invention>

[0050] According to one embodiment of the present invention, fibers included in an image can be recognized with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A diagram showing an example of a target image.

[0052] Figure 2 This is a diagram showing an example of detection results based on the conventional technology.

[0053] Figure 3 This is a block diagram showing an example of the overall configuration of an object detection system.

[0054] Figure 4 This is a block diagram showing an example of the hardware configuration of a computer.

[0055] Figure 5 This is a block diagram showing an example of the functional structure of the object detection system.

[0056] Figure 6 This is a flowchart showing an example of a processing procedure of the object detection method.

[0057] Figure 7 : is a figure which shows an example of the mask score before dilation.

[0058] Figure 8 : is a figure which shows an example of the mask score after binarization.

[0059] Fig. 9 : is a figure which shows an example of the mask score after dilation.

[0060] Fig.10 : is a diagram showing an example of the mask score after correction.

[0061] Fig.11 This is a diagram showing an example of detection results in one embodiment.

[0062] Fig.12 This is a diagram showing an example of an analysis screen.

[0063] Fig.13 It is a diagram showing an example of the image confirmation screen.

[0064] Fig.14 It is a diagram showing an example of the first result confirmation screen.

[0065] Fig.15 It is a diagram showing an example of the first result confirmation screen.

[0066] Fig.16 It is a diagram showing an example of the second result confirmation screen.

[0067] Fig.17 It is a diagram showing an example of the second result confirmation screen.

[0068] Fig.18 It is a diagram showing an example of a calibration result confirmation screen.

[0069] Fig.19 A diagram showing an example of a target image.

[0070] Fig. 20 This is a diagram showing an example of a region segmentation result based on U-Net.

[0071] Fig.21 This is a diagram showing an example of a region segmentation result using DeepLab. DETAILED DESCRIPTION

[0072] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in this specification and the accompanying drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and repeated descriptions are omitted.

[0073] [Implementation Method]

[0074] One embodiment of the present invention is an object detection system for detecting an object from an image obtained by photographing the object. Hereinafter, the image of the object to be detected is also referred to as an "object image". The object image in this embodiment is an image obtained by photographing a state in which a plurality of fibers are dispersed on the surface of an observation sample and include a plurality of fibers entangled with each other or a plurality of fibers overlapping each other. As an example, the fiber in this embodiment is a carbon fiber. However, the fiber to be detected is not limited to carbon fiber, and can be a fiber of any raw material.

[0075] In the task of detecting objects from images, segmentation using machine learning models is sometimes performed. Segmentation using machine learning models includes instance segmentation based on deep learning and semantic segmentation.

[0076] Instance segmentation is a task for detecting individual objects contained in an image. In instance segmentation, the rectangular area (bounding box) where the object is captured can be detected separately from the image, or each object can be detected and determined in units of pixels. For each detected object, the object detection result based on instance segmentation can include two-dimensional data (mask score) representing the area where the object is captured, a score representing the degree of suspected object, a mask after binarizing the mask score by a threshold, and information representing the bounding box. The size of the bounding box can be adjustable. The size of the mask of each individual can also be adjusted by setting a score threshold. By specifying a larger size of the bounding box or setting a lower score threshold, the area of ​​each individual can be determined to be larger. In addition, the object detection result can include the reliability of the object detection result.

[0077] The machine learning model that performs instance segmentation is an example of a “region segmentation model for each individual.” In addition, the mask score included in the object detection result based on instance segmentation is an example of a “region segmentation result for each individual.”

[0078] As a machine learning model for instance segmentation, Mask R-CNN (Mask Region-Based Convolutional Neural Networks) or YOLACT (You Only Look At CoefficienTs) can be used. The details of Mask R-CNN are disclosed in Reference 1.

[0079] [Reference 1] Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick, "MaskR-CNN", Proceedings of the IEEE International Conference on ComputerVision (ICCV), 2017.

[0080] Semantic segmentation is the task of segmenting the subject constituting an image into regions representing one or more classes. In semantic segmentation, a label representing a class (class label) is predicted for each unit (e.g., pixel) in the image. The region segmentation result based on semantic segmentation can include two-dimensional data (semantic scores) containing class labels corresponding to each pixel of the image. In addition, the region segmentation result can include the reliability of the region segmentation result.

[0081] The machine learning model that performs semantic segmentation is an example of a “region segmentation model for each category.” Furthermore, the semantic score included in the region segmentation result based on semantic segmentation is an example of a “region segmentation result for each category.”

[0082] The machine learning model for semantic segmentation can use U-Net, DeepLab, etc. The details of U-Net are disclosed in Reference 2. The details of DeepLab (DeepLabv3) are disclosed in Reference 3.

[0083] [Reference 2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation", Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.

[0084] [Reference 3]LC Chen, G. Papandreou, F. Schroff, and H. Adam, "Rethinking Atrous Convolution for Semantic Image Segmentation", European Conference on Computer Vision (ECCV), 2017.

[0085] When performing instance segmentation on an image captured in which many fibers are scattered on the surface of a sample, the detection accuracy may sometimes decrease. For example, when an image contains multiple fibers entangled with each other, each fiber may not be accurately detected. In addition, for example, when the background portion of an image contains linear dirt or streaks, the dirt may be mistakenly recognized as fibers.

[0086] Figure 1 FIG. 1 is a diagram showing an example of an object image in the present embodiment. Figure 1 As shown, the target image 900 of this embodiment is an image obtained by photographing a state where a large number of fibers are dispersed on a metal surface. In the target image 900, a large number of fibers 901 entangled with each other and stripes 902 that are not fibers are photographed.

[0087] Figure 2 This is a diagram showing an example of conventional object detection results. Figure 2 It uses Mask R-CNN from Figure 1 The object image shown shows the results of detecting fibers. Figure 2 In the object detection results shown in , different objects are represented by different colors. Figure 2 As shown, in the conventional object detection result 910, the region 911 where a plurality of fibers entangled with each other are photographed is detected as a plurality of separated fibers. In addition, the region 912 where stripes other than fibers are photographed is detected as fibers.

[0088] The object detection system 1 in this embodiment aims to recognize fibers contained in an image with high accuracy. In particular, the object detection system 1 in this embodiment detects multiple fibers entangled with each other or multiple fibers overlapping with each other with high accuracy. In addition, the object detection system 1 in this embodiment reduces the misrecognition of linear dirt or streaks contained in the background.

[0089] <Overall structure of the object detection system>

[0090] Reference Figure 3 The overall configuration of the object detection system in this embodiment will be described. Figure 3 This is a block diagram showing an example of the overall configuration of the object detection system in the present embodiment.

[0091] like Figure 3 As shown, the object detection system 1 in this embodiment includes an image acquisition device 10, an image processing device 20, and a user terminal 30. The image acquisition device 10, the image processing device 20, and the user terminal 30 are connected to each other via a communication network N1 such as a LAN (Local Area Network) or the Internet so as to be data communicable.

[0092] The image acquisition device 10 is an optical device that acquires an image of an object to be detected. The image acquisition device 10 may be a digital camera that takes a still image, or a video camera that takes a moving image. As the image acquisition device 10, an optical microscope, a scanning electron microscope (SEM), a transmission electron microscope (TEM), etc. may be used depending on the size of the object to be detected. In addition, the image acquisition device 10 may be an information processing device such as a personal computer connected to various cameras, or an inspection device equipped with various cameras.

[0093] The image processing device 20 is an information processing device such as a personal computer, a workstation, or a server that generates an output image indicating a result of detecting an object from a target image acquired by the image acquisition device 10. The image processing device 20 receives the target image from the user terminal 30. The image processing device 20 detects an object from the acquired target image, and transmits an output image indicating the detection result to the user terminal 30.

[0094] The user terminal 30 is an information processing terminal such as a personal computer, a tablet terminal, or a smartphone operated by a user. The user terminal 30 acquires the target image from the image acquisition device 10 according to the user's operation, and transmits it to the image processing device 20. The user terminal 30 receives an output image representing the detection result from the image processing device 20, and outputs it to the user.

[0095] in addition, Figure 3 The overall structure of the object detection system 1 shown is an example, and various system configuration examples can be formed according to the application and purpose. For example, one or more of the image acquisition device 10, the image processing device 20, and the user terminal 30 can include multiple devices in the object detection system 1. For example, the image processing device 20 can be implemented by multiple computers or as a cloud computing service. Figure 3 The division of devices such as the image acquisition device 10 , the image processing device 20 , and the user terminal 30 shown is an example.

[0096] <Hardware structure of object detection system>

[0097] Reference Figure 4 The hardware configuration of the object detection system 1 in this embodiment will be described.

[0098] Computer Hardware Structure

[0099] The image acquisition device 10 , the image processing device 20 , and the user terminal 30 in the present embodiment are realized by, for example, a computer. Figure 4 This is a block diagram showing an example of the hardware configuration of the computer 500 in this embodiment.

[0100] like Figure 4 As shown, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. The hardware of the computer 500 is connected to each other via a bus 509. In addition, the input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0101] The CPU 501 is a computing device that realizes the overall control and functions of the computer 500 by reading programs and data from a storage device such as the ROM 502 or the HDD 504 to the RAM 503 and executing the processing.

[0102] ROM502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. ROM502 functions as a main storage device that stores various programs and data required for CPU501 to execute various programs installed in HDD504. Specifically, ROM502 stores boot programs such as BIOS (Basic Input / Output System) and EFI (Extensible Firmware Interface) that are executed when the computer 500 is started, OS (Operating System) settings, network settings, and other data.

[0103] RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are deleted when the power is turned off. RAM 503 is, for example, DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), etc. RAM 503 provides a work area that is expanded when CPU 501 executes various programs installed in HDD 504.

[0104] The HDD 504 is an example of a non-volatile storage device storing programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software for controlling the entire computer 500, and application programs that provide various functions on the OS. In addition, the computer 500 may use a storage device (e.g., SSD: Solid State Drive) that uses a flash memory as a storage medium instead of the HDD 504.

[0105] The input device 505 is a touch panel, operation keys or buttons, a keyboard or a mouse for a user to input various signals, a microphone for inputting sound data such as sound, or the like.

[0106] The display device 506 is constituted by a display such as liquid crystal or organic EL (Electro-Luminescence) for displaying a screen, a speaker for outputting audio data such as sound, and the like.

[0107] The communication I / F 507 is an interface for connecting to a communication network and for the computer 500 to perform data communication.

[0108] The external I / F 508 is an interface with an external device. The external device includes a driver device 510 and the like.

[0109] The drive device 510 is a device for setting the recording medium 511. The recording medium 511 mentioned here includes a medium that records information optically, electrically or magnetically, such as a CD-ROM, a floppy disk, a magneto-optical disk, etc. In addition, the recording medium 511 may also include a semiconductor memory that records information electrically, such as a ROM, a flash memory, etc. Thus, the computer 500 can read and / or write the recording medium 511 via the external I / F 508.

[0110] In addition, various programs installed in HDD504 are installed by, for example, setting the distributed recording medium 511 to the drive device 510 connected to the external I / F508 and reading the various programs recorded on the recording medium 511 through the drive device 510. Alternatively, various programs installed in HDD504 can also be installed by downloading from a network other than the communication network via the communication I / F507.

[0111] <Functional structure of object detection system>

[0112] Reference Figure 5 The functional configuration of the object detection system in this embodiment will be described. Figure 5 1 is a block diagram showing an example of the functional configuration of the object detection system 1 in the present embodiment.

[0113] 《Functional structure of image acquisition device》

[0114] like Figure 5 As shown in FIG. 1 , the image acquisition device 10 in this embodiment includes an imaging control unit 101 and an image storage unit 102 .

[0115] The camera control unit 101 is connected to Figure 4 The image storage unit 102 is implemented by a camera of the external I / F 508 shown in FIG. Figure 4 The HDD 504 shown is implemented.

[0116] The camera control unit 101 adjusts the camera shooting conditions (angle of view, shooting magnification, etc.) so that a plurality of fibers can be photographed to shoot the object image. The camera control unit 101 can shoot the object image according to the user's operation, or can shoot the object image when a predetermined condition is satisfied. The so-called predetermined condition is, for example, when the inspection object is transported to a given position in the inspection device. The camera control unit 101 can shoot a still image or a moving image to extract an image in which a plurality of fibers are reflected.

[0117] The image storage unit 102 stores the target image captured by the image capture control unit 101. The image storage unit 102 may store information related to the target image in association with the target image. The information related to the target image may include, for example, the capture date and time, information indicating the captured object, and capture conditions.

[0118] 《Functional structure of image processing device》

[0119] like Figure 5 As shown, the image processing device 20 in this embodiment includes a model storage unit 200, an image acquisition unit 201, an object detection unit 202 (an example of a first region segmentation unit), a region segmentation unit 203 (an example of a second region segmentation unit), a region correction unit 204 and a result output unit 205.

[0120] The model storage unit 200 is composed of Figure 4 The image acquisition unit 201, the object detection unit 202, the area division unit 203, the area correction unit 204 and the result output unit 205 are implemented by causing the CPU 501 to execute Figure 4 This is achieved by the processing of the program expanded from HDD 504 to RAM 503 as shown.

[0121] The model storage unit 200 stores a learned object detection model and a learned region segmentation model.

[0122] The object detection model is a machine learning model that performs instance segmentation. The object detection model takes an image as input and outputs an object detection result that detects the objects contained in the image individually. The object detection result includes a region segmentation result for each individual, which identifies the area where each object is photographed. As an example, the object detection model in this embodiment uses Mask R-CNN. The object detection model is also called a region segmentation model for each individual.

[0123] The object detection model in this embodiment is learned by detecting fibers captured in an input image individually. The object detection model is learned using teacher data that assigns teacher labels to images captured on the surface of a sample on which a plurality of fibers are dispersed. The teacher labels include information indicating the range of the region in which each fiber is captured and identification information for identifying the fiber.

[0124] The region segmentation model is a machine learning model that performs semantic segmentation. The region segmentation model takes an image as input and outputs a region segmentation result for each category that identifies the area in the image where the desired object is captured. As an example, the region segmentation model in this embodiment uses DeepLabv3. The region segmentation model is also called a region segmentation model for each category.

[0125] The region segmentation model in this embodiment is learned in a manner that divides the input image into a region where any fiber is photographed and other regions. The region segmentation model is learned using teacher data that assigns teacher labels to images obtained by photographing a metal surface on which a plurality of fibers are dispersed. The teacher labels include information indicating the range of the region where the fibers are photographed.

[0126] The image acquisition unit 201 acquires the target image by receiving the target image from the user terminal 30. The image acquisition unit 201 can acquire the target image from the image acquisition device 10 in response to a request from the user terminal 30.

[0127] The object detection unit 202 detects each fiber included in the target image acquired by the image acquisition unit 201 by using the learned object detection model read from the model storage unit 200 , thereby generating an object detection result corresponding to each fiber included in the target image.

[0128] The region segmentation unit 203 uses the learned region segmentation model read from the model storage unit 200 to identify a region where fibers are imaged (hereinafter also referred to as a “fiber region”) from the target image acquired by the image acquisition unit 201 , thereby generating a region segmentation result representing the fiber region.

[0129] The region correction unit 204 corrects the region where the fiber is detected (hereinafter also referred to as "fiber detection region") in the object detection result generated by the object detection unit 202. The region correction unit 204 can correct the fiber detection region by, for example, merging the object detection result generated by the object detection unit 202 with the region segmentation result generated by the region segmentation unit 203. The region correction unit 204 can correct the fiber detection region by, for example, correcting the fiber detection region by dilation processing or smoothing processing and then merging the correction with the region segmentation result generated by the region segmentation unit 203.

[0130] The result output unit 205 transmits an output image indicating the correction result of the area correction unit 204 to the user terminal 30 .

[0131] <<Functional structure of user terminal 30>>

[0132] like Figure 5 As shown, the user terminal 30 in this embodiment includes an image transmission unit 301 and a result display unit 302 .

[0133] The image transmission unit 301 and the result display unit 302 execute the following operations by the CPU 501: Figure 4 The program shown is implemented by expanding it onto the RAM 503 from the HDD 504 .

[0134] The image transmission unit 301 acquires the target image from the image acquisition device 10 in accordance with the user's operation. The image transmission unit 301 transmits the target image acquired from the image acquisition device 10 to the image processing device 20.

[0135] The result display unit 302 receives the output image from the image processing device 20. The result display unit 302 displays the received output image on the display device 506.

[0136] <Object detection system processing>

[0137] Reference Figures 6 to 11 The processing procedure of the image processing method executed by the object detection system 1 in this embodiment will be described. Figure 6 This is a flowchart showing an example of the processing procedure of the image processing method in this embodiment.

[0138] In step S1 , the imaging control unit 101 of the image acquisition device 10 adjusts the angle of view of the camera so as to capture a plurality of fibers, and captures a target image. Next, the imaging control unit 101 stores the captured target image in the image storage unit 102 .

[0139] In step S2, the image transmission unit 301 of the user terminal 30 transmits a request for obtaining the target image to the image acquisition device 10 in response to the user's operation. In response to the acquisition request received from the user terminal 30, the image acquisition device 10 reads the target image stored in the image storage unit 102 and transmits it to the user terminal 30. The image transmission unit 301 transmits the target image received from the image acquisition device 10 to the image processing device 20.

[0140] In step S3 , the image acquisition unit 201 of the image processing device 20 receives the target image from the user terminal 30 . Next, the image acquisition unit 201 sends the received target image to the object detection unit 202 and the region division unit 203 .

[0141] In step S4 , the object detection unit 202 of the image processing device 20 receives the target image from the image acquisition unit 201 . Next, the object detection unit 202 reads the learned object detection model from the model storage unit 200 .

[0142] Next, the object detection unit 202 inputs the object image into the learned object detection model. The object detection model detects each fiber included in the object image and outputs the object detection result of the object image. Thus, the object detection unit 202 obtains the object detection result of the object image. Then, the object detection unit 202 sends the object detection result to the region correction unit 204.

[0143] The object detection result of the object image includes the detection result corresponding to each fiber detected from the object image and the reliability of the object detection result. The detection result of each fiber includes a mask score indicating the fiber detection area and a score indicating the degree of suspected object. In addition, the detection result of each fiber may also include a mask obtained by binarizing the mask score through a threshold and information indicating a bounding box.

[0144] The object detection unit 202 may also discard detection results corresponding to each fiber whose score (0 or more and 1 or less) indicating the degree of suspected object is less than a given threshold. The threshold may be arbitrarily determined, and may be set to 0.3, for example.

[0145] In step S5 , the region segmentation unit 203 of the image processing device 20 receives the target image from the image acquisition unit 201 . Next, the region segmentation unit 203 reads out the learned region segmentation model from the model storage unit 200 .

[0146] Next, the region segmentation unit 203 inputs the target image into the learned region segmentation model. The region segmentation model identifies the fiber region from the target image and outputs the region segmentation result of the target image. Thus, the region segmentation unit 203 obtains the region segmentation result of the target image. Then, the region segmentation unit 203 sends the region segmentation result to the region correction unit 204.

[0147] The region segmentation result of the target image includes a semantic score representing the fiber region and the reliability of the region segmentation result.

[0148] In step S6, the region correction unit 204 of the image processing device 20 receives the object detection result from the object detection unit 202. Furthermore, the region correction unit 204 receives the region segmentation result from the region segmentation unit 203. Next, the region correction unit 204 corrects the object detection result based on the region segmentation result.

[0149] For example, the area correction unit 204 corrects the object detection result by enlarging the fiber detection area through dilation processing or smoothing processing. Figures 7 to 10 An example of a process of correcting the fiber detection area using the expansion process will be described.

[0150] Figure 7 is a diagram showing an example of the mask score before dilation. Figure 7 As shown in FIG. 1 , the mask score 931 before dilation shows the fiber detection area where one fiber is detected. Figure 7 In the mask score 931 shown, both ends are interrupted and the peripheral portion is unclear.

[0151] First, the area correction unit 204 performs threshold processing on the mask score 931 to perform binarization. Figure 8is a diagram showing an example of a binarized mask score. Figure 8 As shown, the mask score 932 after binarization is binarized such that the fiber detection region takes 1 and the other regions take 0. It can be seen that in the mask score 932 after binarization, the peripheral portion of the fiber becomes clear.

[0152] In addition, the binarized mask score can also be output by the object detection model.

[0153] Next, the region correction unit 204 performs dilation processing on the mask score, thereby enlarging the fiber detection region. Fig. 9 is a diagram showing an example of the mask score after dilation. Fig. 9 As shown, it can be seen that the mask score after expansion is 933 and Figure 7 Compared to the mask score 931 before expansion shown, the fiber detection area is enlarged as a whole.

[0154] The above-mentioned expansion processing may also be performed by specifying a larger size of the bounding box or setting a lower score threshold.

[0155] Then, the region correction unit 204 generates a corrected mask score based on the binarized mask score 932 and the expanded mask score 933. Specifically, the region correction unit 204 uses the binarized mask score 932 inside the region of interest (ROI) and uses the expanded mask score 933 outside the region of interest. Fig.10 is a diagram showing an example of a corrected mask score. Fig.10 As shown, it can be seen that in the corrected mask score 934, both ends and the surrounding part of the fiber become clear, and the contour of the fiber can be detected with high accuracy.

[0156] Back to Figure 6 In step S7, the region correction unit 204 of the image processing device 20 merges the region segmentation result with the corrected object detection result. Thus, the region correction unit 204 obtains the corrected result of the object detection result. Then, the region correction unit 204 sends the corrected result of the object detection result to the result output unit 205.

[0157] For example, the region correction unit 204 may also combine the region segmentation result with the corrected object detection result by calculating the logical product of the object detection result and the region segmentation result. In this case, the region correction unit 204 calculates the logical product of each object detection result corresponding to each fiber and the region segmentation result. Thus, the detection result corresponding to each fiber (hereinafter also referred to as "individual detection result") can be generated.

[0158] In addition, for example, the region correction unit 204 may select an object detection result or a region segmentation result based on the result of comparing the score of the object detection result with the score of the region segmentation result for each pixel of the target image, thereby merging the region segmentation result with the corrected object detection result. In this case, the region correction unit 204 compares the mask score of each object detection result corresponding to each fiber with the semantic score of the region segmentation result for each pixel of the target image, and selects the result with the higher score if the scores are different, and selects the result with the higher reliability if the scores are equal. In this way, a separate detection result corresponding to each fiber can be generated.

[0159] In addition, the region correction unit 204 may also discard the individual detection result with a lower mask score when the area overlapped with the individual detection result corresponding to other fibers among the individual detection results corresponding to each fiber is greater than a predetermined threshold. This is because the detection result with a large overlapped area can be considered to have detected the same fiber. The threshold can be determined arbitrarily, for example, it can be set to 0.7 (i.e., the detection result is discarded when 70% of the area overlaps).

[0160] Then, the region correction unit 204 generates a correction result by overlapping the individual detection results corresponding to the fibers in descending order of the scores indicating the degree of suspected objects. Finally, the region correction unit 204 adds the regions of the fiber regions indicated by the region segmentation results that are not indicated as fiber detection regions by the object detection results as fiber regions to the correction result.

[0161] Furthermore, the region correction unit 204 may calculate the number and distribution of fiber lengths and fiber diameters, as well as statistical values ​​such as the average value, median value, and standard deviation for the fiber region included in the correction result.

[0162] Fig.11 This is a diagram showing an example of the detection result in this embodiment. Fig.11 The image processing device 20 is Figure 1 The result of detecting fibers in the object image shown in the figure. Fig.11 As shown, in the detection result 920 in this embodiment, the region 921 where a plurality of fibers entangled with each other are photographed is detected as a plurality of fibers entangled with other fibers. In addition, the region 922 where stripes other than fibers are photographed is not detected as fibers.

[0163] Back to Figure 6In step S8, the result output unit 205 of the image processing device 20 receives the correction result from the area correction unit 204. Next, the result output unit 205 generates an output image representing the detection result of the fiber captured in the target image based on the correction result. Then, the result output unit 205 sends the output image to the user terminal 30.

[0164] The output image may be a mask image in which identification information for identifying the detected fibers is set for each pixel. The output image may also be an image showing the region of each detected fiber and the identification information for identifying each fiber in the object image. The output image may also be an image showing a probability value for each detected fiber.

[0165] In step S9, the result display unit 302 of the user terminal 30 receives the output image from the image processing device 20. Then, the result display unit 302 displays the received output image on the display device 506. The result display unit 302 may also display the above-mentioned statistical values ​​calculated for each fiber region in response to a user's request.

[0166] <Object Detection System User Interface>

[0167] Reference Figures 12 to 18 The user interface of the object detection system 1 in the present embodiment will be described.

[0168] Analysis screen

[0169] Fig.12 2 is a diagram showing an example of an analysis screen. The analysis screen is a screen for inputting a target image to be analyzed and performing object detection on the target image.

[0170] like Fig.12 As shown, the analysis screen 1000 includes an object selection field 1001 , an object input field 1002 , a reference button 1003 , an analysis execution button 1004 , four confirmation buttons 1011 to 1014 , and an output button 1015 .

[0171] In the initial display, the object selection field 1001, the object input field 1002, and the reference button 1003 are controlled to be pressable or inputtable. On the other hand, in the initial display, the analysis execution button 1004, the confirmation buttons 1011 to 1014, and the output button 1015 are controlled to be non-pressable.

[0172] The object selection field 1001 is an input field for designating a method for selecting an object image. Fig.12In the example of , files and folders can be selected as the options for the analysis object. When a file is selected, one file can be selected as the target image. When a folder is selected, all files stored in the specified folder are selected as the target images.

[0173] The object input field 1002 is an input field for specifying the location of the object image. When a file is selected in the object selection field 1001, information indicating the location of the file is input in the object input field 1002. When a folder is selected in the object selection field 1001, information indicating the location of the folder is input in the object input field 1002. The information indicating the location of the file or folder is, for example, a local path or URL (Uniform Resource Locator) of the file or folder.

[0174] When the reference button 1003 is pressed, a file selection dialog box is displayed. By selecting a file or folder in the file selection dialog box, information indicating the location of the file or folder is input into the target input field 1002. When the information indicating the location of the file or folder is input into the target input field 1002, the analysis execution button 1004 is controlled to be pressable.

[0175] The analysis execution button 1004 is a button for executing object detection for the target image. The target image is a file input in the target input field 1002 or a file stored in a folder input in the target input field 1002. When the object detection for the target image is completed, the confirmation buttons 1011 to 1014 and the output button 1015 are controlled to be pressable.

[0176] When the confirmation button 1011 is pressed, an image confirmation screen for displaying the target image is displayed. When the confirmation button 1012 is pressed, a first result confirmation screen for displaying the region segmentation result for each category is displayed. When the confirmation button 1013 is pressed, a second result confirmation screen for displaying the region segmentation result for each individual is displayed. When the confirmation button 1014 is pressed, a correction result confirmation screen for displaying the correction result is displayed. When the output button 1015 is pressed, electronic data representing the analysis result related to the target image is saved.

[0177] 《Image confirmation screen》

[0178] Fig.13 FIG. 2 is a diagram showing an example of an image confirmation screen. Fig.13 As shown, the image verification screen 1100 includes an image display field 1101 and an end button 1102 .

[0179] The target image is displayed in the image display field 1101. When an end button 1102 labeled "Close" is pressed, the image confirmation screen 1100 ends and the analysis screen 1000 is displayed.

[0180] 《First result confirmation screen》

[0181] Fig.14 is a diagram showing a first example of a first result confirmation screen. Fig.14 As shown, the first result confirmation screen 1200 includes an image display field 1201 , an end button 1202 , and a confirmation button 1203 .

[0182] The image display column 1201 displays the region segmentation result for each type related to the target image. When the end button 1202 marked with "Close" is pressed, the first result confirmation screen 1200 is terminated and the analysis screen 1000 is displayed. When the confirmation button 1203 marked with "Accuracy Confirmation" is pressed, the score indicating the recognition accuracy of each object is displayed in the image display column 1201.

[0183] Fig.15 1200 is a diagram showing a second example of the first result confirmation screen. The first result confirmation screen 1200 in the second example is different from the first example of the first result confirmation screen shown above. Fig.14 In comparison, there is also a threshold setting column 1204 and a reflection button 1205 .

[0184] The threshold setting field 1204 is an input field for setting a threshold for binarizing the semantic score. When the reflection button 1205 is pressed, an image obtained by binarizing the semantic score according to the threshold set in the threshold setting field 1204 is displayed in the image display field 1201 .

[0185] The threshold value set by operating the threshold value setting field 1204 and the reflection button 1205 is used only to confirm the detection result on the inspection and analysis screen 1000. Therefore, the set threshold value is not continuously used in the object detection system 1.

[0186] 《Second result confirmation screen》

[0187] Fig.16 is a diagram showing a first example of the second result confirmation screen. Fig.16 As shown, the second result confirmation screen 1300 includes an image display field 1301 , an end button 1302 , and a confirmation button 1303 .

[0188] The image display column 1301 displays the region segmentation result for each individual related to the target image. When the end button 1302 marked with a "close" is pressed, the second result confirmation screen 1300 is terminated and the analysis screen 1000 is displayed. When the confirmation button 1303 marked with a "confirmation of accuracy" is pressed, the score indicating the recognition accuracy of each object is displayed in the image display column 1301.

[0189] Fig.17 2 is a diagram showing a second example of the second result confirmation screen. Fig.16 In contrast, the second result confirmation screen 1300 in the second example further includes a first threshold setting field 1304 , a second threshold setting field 1305 , and a reflection button 1306 .

[0190] The first threshold setting column 1304 is an input column for setting a threshold for a score indicating the degree of a suspected object. The second threshold setting column 1305 is an input column for setting a threshold for binarizing the mask score. When the reflection button 1306 is pressed, the detection results with low scores indicating the degree of a suspected object are discarded according to the thresholds set in the first threshold setting column 1304 and the second threshold setting column 1305, and the image after binarizing the mask score is displayed in the image display column 1301.

[0191] The threshold value set by operating first threshold value setting field 1304, second threshold value setting field 1305, and reflection button 1306 is used only to confirm the detection result on analysis screen 1000. Therefore, the set threshold value is not continuously used in object detection system 1.

[0192] 《Calibration result confirmation screen》

[0193] Fig.18 FIG. 2 is a diagram showing an example of a calibration result confirmation screen. Fig.18 As shown, the calibration result confirmation screen 1400 includes an image display field 1401 , an end button 1402 , a confirmation button 1403 , a fiber display button 1404 , and a winding display button 1405 .

[0194] The image display column 1401 is used to display the correction result of the region segmentation result for each individual. When the end button 1402 marked with a record "Close" is pressed, the correction result confirmation screen 1400 is ended and the analysis screen 1000 is displayed. When the confirmation button 1403 marked with a record "Accuracy Confirmation" is pressed, the score indicating the recognition accuracy of each object is displayed in the image display column 1401.

[0195] When the fiber display button 1404 labeled "fiber only" is pressed, the detection result excluding the plurality of fibers intertwined with each other among the fibers detected in the target image is displayed in the image display field 1401. When the entanglement display button 1405 labeled "fiber+entanglement" is pressed, the detection result including the plurality of fibers intertwined with each other among the fibers detected in the target image is displayed in the image display field 1401.

[0196] <Comparison between U-Net and DeepLab>

[0197] As a machine learning model for semantic segmentation, for example, U-Net or DeepLab is used. In this embodiment, U-Net or DeepLab can be used as a region segmentation model. However, when DeepLabv3 is used as a region segmentation model, the possibility of further improving the accuracy increases. The reason is explained below.

[0198] In order to extract the global features of the input image, U-Net performs multiple upsampling in the encoder and the same number of downsampling in the decoder. During the upsampling in the encoder, the convolution process used in the standard convolutional neural network is performed.

[0199] On the other hand, DeepLabv3 uses dilated convolution to obtain a larger kernel size, extract global features of the input image, and suppress the amount of calculation. In addition, the so-called dilated convolution refers to a convolution process that reduces the number of parameters by making the kernel of the convolution larger and setting part of the kernel to 0.

[0200] As mentioned above, DeepLabv3 can introduce more global features by introducing dilated convolution. As a result, compared with U-Net, DeepLabv3 can effectively introduce global features of the image, and it is expected that the recognition accuracy of the region will be improved.

[0201] Reference Figures 19 to 21 To illustrate the comparison results between U-Net and DeepLabv3. Fig.19 FIG. 1 is a diagram showing an example of a target image used for comparison. Fig. 20 This is a diagram showing an example of a region segmentation result based on U-Net. Fig.21 This is a diagram showing an example of a region segmentation result based on DeepLabv3.

[0202] like Fig.19 As shown, the object image 940 used in the comparison is an image taken in a state where many fibers are dispersed. Fig. 20As shown, there are fibers that are not correctly recognized in the U-Net-based region segmentation result 950. For example, in the object image 940, there is a region 941 that includes relatively thick fibers and relatively thin fibers. In the U-Net-based region segmentation result 950, in the region 951 corresponding to the region 941 of the object image 940, the relatively thin fibers are interrupted, and it can be seen that they are not correctly recognized.

[0203] On the other hand, Fig.21 As shown, in the region segmentation result 960 based on DeepLabv3, the recognition accuracy of the fibers captured in the object image 940 is improved compared with U-Net. For example, in the region segmentation result 960 based on DeepLabv3, it can be seen that in the region 961 corresponding to the region 941 of the object image 940, any fiber can be recognized with high accuracy.

[0204] <Effects of implementation>

[0205] The image processing device 20 of this embodiment corrects the region segmentation result based on the region segmentation model for each individual by using the region segmentation result based on the region segmentation model for each type. The region segmentation model for each individual can identify the region of each object included in the image, and the region segmentation model for each type can identify the outline of the object captured in the image with high accuracy. Therefore, if the region segmentation result for each individual is corrected by using the region segmentation result for each type, each fiber whose outline is identified with high accuracy can be detected. Therefore, according to the image processing device 20 of this embodiment, the fiber included in the image can be identified with high accuracy.

[0206] The image processing device 20 in this embodiment can be used particularly effectively when appropriately evaluating fiber materials in accordance with the RoHS Directive or the REACH Regulation.

[0207] The image processing device 20 in this embodiment enlarges the area where the fiber is detected in the area segmentation result for each individual. Although the area segmentation model for each individual has a tendency to reduce the detection accuracy at the peripheral part of the detected object, if the area segmentation result for each individual is enlarged to the outside of the peripheral part of the object, it can be supplemented by the contour of the object recognized with high accuracy by the area segmentation model for each type. Therefore, according to the image processing device 20 in this embodiment, the peripheral part of the fiber or the entanglement part of multiple fibers can be recognized with high accuracy.

[0208] The image processing device 20 in this embodiment calculates the logical product of the region segmentation result for each individual and the region segmentation result for each type. That is, the fibers identified in both the region segmentation result for each individual and the region segmentation result for each type are included in the correction result. Therefore, according to the image processing device 20 in this embodiment, the possibility of erroneously detecting dirt or the like in the background that should not be detected can be reduced.

[0209] The image processing device 20 in this embodiment selects the region segmentation result for each individual or the region segmentation result for each category based on the result of comparing the score of the region segmentation result for each individual with the score of the region segmentation result for each category for each pixel of the target image. That is, the result with the higher score between the region segmentation result for each individual and the region segmentation result for each category can be included in the correction result. Therefore, according to the image processing device 20 in this embodiment, the possibility of erroneously detecting dirt or the like in the background that should not be detected can be reduced.

[0210] In the present embodiment, the image analysis is described with respect to fibers, but the same can be performed with respect to particles. For example, even when particles overlap, the particles can be correctly identified.

[0211] [Replenish]

[0212] Each function of the above-described embodiments can be implemented by one or more processing circuits. Here, the "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module, etc., which is designed to perform each function described above.

[0213] The embodiments of the present invention are described in detail above, but the embodiments disclosed this time are illustrative and non-restrictive in all aspects. The embodiments can be modified and improved in various ways without departing from the scope and gist of the attached claims. The matters recorded in the above multiple embodiments may also adopt other structures within the scope of non-contradiction, and may also be combined within the scope of non-contradiction.

[0214] This application claims the priority of Japanese Patent Application No. 2023-50607 filed with the Japan Patent Office on March 27, 2023, the entire contents of which are incorporated herein by reference.

[0215] Explanation of symbols

[0216] 1: Object Detection System

[0217] 10: Image acquisition device

[0218] 101: Camera control unit

[0219] 102: Image storage unit

[0220] 20: Image processing device

[0221] 200: Model storage unit

[0222] 201: Image acquisition unit

[0223] 202: Object Detection Department

[0224] 203: Area segmentation

[0225] 204: Regional Correction Department

[0226] 205: Result output unit

[0227] 30: User terminal

[0228] 301: Image sending unit

[0229] 302: Result display unit.

Claims

1. An image processing device comprising: An image acquisition unit configured to acquire an image of a subject in which a plurality of fibers are photographed; a first region segmentation unit configured to generate a region segmentation result for each individual fiber detected in the target image using the learned region segmentation model for each individual fiber; a second region segmentation unit configured to generate a region segmentation result for each type by identifying a region where the plurality of fibers are imaged from the target image using the learned region segmentation model for each type; an area correction unit configured to correct the area segmentation result for each individual by using the area segmentation result for each type; and The result output unit is configured to output a correction result of the region segmentation result for each individual.

2. The image processing device according to claim 1, wherein: The area correction unit is configured to generate the correction result by calculating a logical product of the area division result for each individual and the area division result for each type.

3. The image processing device according to claim 1, wherein: The area correction unit is configured to select, for each unit of the object image, the area segmentation result according to each individual or the area segmentation result according to each category based on a result of comparing the score of the area segmentation result according to each individual with the score of the area segmentation result according to each category, thereby generating the correction result.

4. The image processing device according to any one of claims 1 to 3, wherein: The region segmentation model for each individual is a model for performing instance segmentation. The region segmentation model for each category is a model for performing semantic segmentation.

5. The image processing device according to claim 4, wherein: The region segmentation model for each individual is Mask R-CNN or YOLACT.

6. The image processing device according to claim 4 or 5, wherein: The region segmentation model for each individual can adjust the size of the bounding box.

7. The image processing device according to any one of claims 4 to 6, wherein: The region segmentation model for each individual can adjust the size of the mask of each individual.

8. The image processing device according to any one of claims 4 to 7, wherein: The region segmentation model for each category is DeepLab or U-Net.

9. The image processing device according to any one of claims 1 to 8, wherein: The region correction unit is configured to correct the region in which the fibers are detected in the region division result for each individual.

10. The image processing device according to claim 9, wherein: The region correction unit is configured to enlarge the region divided for each individual by dilation processing or smoothing processing.

11. An image processing method, in which a computer performs the following process: A process for obtaining an image of an object having a plurality of fibers; A process of generating a result of detecting the individual fibers contained in the target image by using the learned individual fiber segmentation model; A process of generating a region segmentation result for each type by identifying a region where the plurality of fibers are photographed from the target image using the learned region segmentation model for each type; A process of correcting the region segmentation result for each individual by using the region segmentation result for each category; as well as A process of outputting the correction result of the region segmentation result for each individual.

12. A program for causing a computer to execute the following process: A process for obtaining an image of an object having a plurality of fibers; A process of generating a result of detecting the individual fibers contained in the target image by using the learned individual fiber segmentation model; A process of generating a region segmentation result for each type by identifying a region where the plurality of fibers are photographed from the target image using the learned region segmentation model for each type; A process of correcting the region segmentation result for each individual by using the region segmentation result for each category; A process of outputting the correction result of the region segmentation result for each individual.

Images