Apparatus, method and system for generating a model for identifying an object of interest from an image

Abstract

Disclosed is an apparatus that generates a model for recognizing an object of interest from an image. The apparatus includes an input section that receives an input image, an object segmentation section that separates a region of an object of interest and a background region from the input image received by the input section, and a learning section that learns a model used by the object segmentation section. The learning section performs training of a new model using training data and an output of an existing model for the training data, adds the trained new model to the existing model, and repeatedly performs the training of the new model and the addition of the trained new model to the existing model.

Description

[0001] This application claims priority to Japanese Patent Application No. 2020-132371, filed on August 4, 2020, which is incorporated herein by reference. Technical Field

[0002] This invention relates to the identification of objects of interest within an image. Background Technology

[0003] In recent years, schemes for detecting unknown objects in images or other visual media have been proposed. For example, Patent Document 1 discloses a technique that clusters different object types in an image based on saliency scores and multiple statistical models to discover new objects.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: US Patent No. 9224071 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] A key element of quantitative analysis of objects in an image is the accurate identification of the desired object of interest. Objects within an image are sometimes known objects of interest, sometimes unknown objects of interest, and sometimes objects outside the scope of interest. For example, an image may include a large background area and many unwanted objects that visually resemble the desired object. To obtain the desired portion of the image for quantitative analysis, the desired foreground needs to be separated from the image, requiring classification for detailed quantitative analysis. However, while Patent Document 1 can identify unknown objects of interest, it cannot classify them. Identifying the types of objects of interest is crucial for detailed quantitative analysis.

[0009] Methods for solving problems

[0010] One aspect of the present invention is an apparatus for generating a model for identifying an object of interest from an image, comprising: an input unit for receiving an input image; an object segmentation unit for receiving the input image from the input unit and separating a region of interest from a background region; and a learning unit for learning a model used by the object segmentation unit. The learning unit trains a new model using training data and the output of an existing model for the training data, appends the trained new model to the existing model, and repeatedly performs the training of the new model and the appending of the trained new model to the existing model.

[0011] Invention Effects

[0012] One aspect of the present invention is capable of generating a model for identifying objects of interest from images. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating the functional structure of the quantitative analysis device for objects of interest in an image according to Embodiment 1.

[0014] Figure 2 This is a schematic diagram illustrating an example of the hardware structure of the quantitative analysis device of Embodiment 1.

[0015] Figure 3 This schematically illustrates an example of the processing performed by the object segmentation section.

[0016] Figure 4 This schematically illustrates an example of the processing performed by the object classification department.

[0017] Figure 5 An example illustrating the processing results of the display unit.

[0018] Figure 6 A flowchart illustrating an example of a method performed by the threshold processing unit of Embodiment 1.

[0019] Figure 7 This represents an example of the processing performed by the threshold processing unit.

[0020] Figure 8 This is an example of the method executed by the stop point determination unit in Embodiment 1.

[0021] Figure 9A This represents the basic collaboration between the input section, the object segmentation section, and the learning section.

[0022] Figure 9B and Figure 9A The process together represents the threshold processing department's handling of discarding unwanted garbage areas.

[0023] Figure 9C and Figure 9B The process together indicates that the stop point determination unit automatically determines the appropriate number of repetitions.

[0024] Figure 10 A flowchart illustrating the technology of Embodiment 1 that can be used as an object classification unit.

[0025] Figure 11 A flowchart illustrating the actions of the quantitative analysis processing unit.

[0026] Figure 12 This illustrates an example of how the quantitative analysis device of Embodiment 1 provides quantitative information to the user through the coordinated operation of the object segmentation unit, object classification unit, quantitative calculation unit, and display unit.

[0027] Figure 13 The functional structure of the anomaly detection system in Example 2 is illustrated schematically.

[0028] Figure 14 The hardware structure of the anomaly detection system in Example 2 is illustrated schematically.

[0029] Figure 15 The diagram illustrates a structural example of an image inspection system including a quantitative analysis system, as described in Example 3.

[0030] Figure 16 Other system architecture examples are shown in Embodiment 3. Detailed Implementation

[0031] Hereinafter, embodiments of the present invention will be described in detail using the accompanying drawings. For convenience, the description will be divided into multiple parts or embodiments as needed, but unless specifically stated otherwise, they are not unrelated to each other, but rather are variations, details, supplementary descriptions, etc., in which one is a part or all of the other. Furthermore, when referring to the quantity of elements (including number, value, quantity, range, etc.), the references are not limited to that specific quantity, except as specifically stated or explicitly limited in principle to a particular quantity; the quantity may be more than or less than that specific quantity.

[0032] This device and system can be a physical computer system (one or more physical computers) or a device or system built on a computing resource cluster (multiple computing resources) such as cloud computing. The computer system or computing resource cluster includes one or more interface devices (e.g., including communication devices and input / output devices), one or more storage devices (e.g., including memory (main memory) and auxiliary storage devices), and one or more computing devices.

[0033] When a function is implemented by executing a program containing command code through a computing device, the determined processing is performed appropriately using storage devices and / or interface devices, so the function can also be at least a part of the computing device. The processing described with the function as the subject can also be processing performed by the computing device or a system having that computing device. The program can be installed from a program source. The program source can be, for example, a program distribution computer or a computer-readable storage medium (e.g., a computer-readable non-transitory storage medium). The description of each function is an example; multiple functions can be combined into one function, or one function can be divided into multiple functions.

[0034] [Abstract]

[0035] First, a general overview of this embodiment will be provided. In this invention, the image, as the object, is composed of objects of interest. The user needs to examine the objects of interest within the image, or examine the characteristics of the entire image or a portion of the image related to the objects of interest. However, not all objects in the image are necessarily known. Furthermore, the image is not limited to being composed only of the desired objects; sometimes it includes objects of no interest or objects that do not need to be examined. In addition, unknown or known objects mentioned in this specification can have arbitrary shape, color, or texture attributes. Objects mentioned in this specification may lack any natural color and may not be observable by the naked eye.

[0036] Detection of known objects can be easily performed by humans. However, when multiple objects of interest exist, and their features are unknown, automated processing is required to handle thousands to millions of images in a short time. Therefore, to automate this processing, it is necessary to detect desired objects in the image and discard unwanted objects and portions within the image. From this perspective, an embodiment described below in this specification can detect desired foreground objects and discard unwanted objects of no interest.

[0037] The learning unit assists the object segmentation unit in learning new groups of unknown objects. The threshold processing unit helps the learning unit discard unwanted objects from the learning data. The stopping point determination unit determines the number of steps (repetitions) required for the learning unit to learn new unknown objects. The object classification unit categorizes objects of interest into their respective known categories and unknown groups. Then, the quantitative calculation unit performs accurate quantitative analysis of the objects, and finally, the display unit provides the user with the necessary quantitative information.

[0038] The input unit is the functional unit in the quantitative analysis apparatus that receives input images. The input images are sent to the object segmentation unit, becoming the target of further processing by the quantitative analysis apparatus. Images containing both objects of interest and unwanted objects are first sent to the object segmentation unit.

[0039] The object segmentation unit is a functional unit that can distinguish desired objects from unwanted objects according to pre-defined rules. The segmentation rules are derived from the learning unit. The learning unit learns to separate (segment) objects that serve as foreground and other objects that serve as background. The thresholding unit helps the learning unit discard unwanted and uninterested objects detected as background. The learning unit learns segmentation by repeatedly using the data.

[0040] The stopping point determination unit determines the optimal number of steps required by the learning unit. The rules for segmentation can be coded in the object segmentation unit. For example, the object segmentation unit can have any structure to identify the desired foreground objects. The object segmentation unit can, for example, be constructed from a neural network, using image-label pairs when labels are available, and training (learning) with self-generated labels when actual images and labels are unavailable, thus accurately detecting foreground objects. Alternatively, the object segmentation unit can use other methods to generate foreground regions in the same way.

[0041] Example 1

[0042] Figure 1 This is a block diagram illustrating the functional structure of the quantitative analysis device 1 for objects of interest in an image according to Embodiment 1. Figure 1 As shown, the quantitative analysis device 1 includes an input unit 10, an object segmentation unit 11, a threshold processing unit 13, a learning unit 12, a stop point determination unit 14, an object classification unit 15, a quantitative calculation unit 16, a display unit 17, a storage unit 90, and a control unit 91. Furthermore, Figure 1 The functional parts of the quantitative analysis device 1 shown can be implemented by various hardware, software or combinations thereof, for example by one or more information processing devices such as personal computers or servers.

[0043] The input unit 10 acquires an input image that is the object of processing by the quantitative analysis device 1 from an external source. This data is obtained by any image capturing device (not shown) and provided to the input unit 10 as an input image. Furthermore, as an input image, an image captured by a camera and obtained via an information communication network or a recording medium (not shown) can be used. The input image can be a still image, or it can be generated by extracting frames from a moving image. For example, still image data formats can be JPEG, JPEG2000, PNG, or BMP. On the other hand, moving image data formats can be MotionJPEG, MPEG, H.246, HD / SDI, etc.

[0044] The object segmentation unit 11 segments the image into a foreground region and a background region containing the object, thereby separating the desired foreground region from the unwanted region.

[0045] The thresholding unit 13 helps the learning unit 12 discard unwanted or uninterested objects that are detected as background. Images sometimes contain objects of interest, and sometimes they also contain unwanted objects. The thresholding unit 13 prevents the learning unit 12 from learning unwanted objects as foreground objects.

[0046] The learning unit 12 learns (trains) a new model for the object segmentation unit 11. The learning unit 12 learns new types of unknown objects along with known objects, enabling the object segmentation unit 11 to separate objects of interest within an image from other objects. The learning unit 12 receives an input image and repeatedly learns about unknown objects of interest within the image.

[0047] The stop point determination unit 14 determines the optimal number of repetitions required for the learning unit 12 to learn about unknown objects of interest in the image. The object classification unit 15 classifies the foreground region of the original image into groups of known and unknown objects based on the foreground region segmented by the object segmentation unit 11, for quantitative analysis. The quantitative calculation unit 16 calculates the required quantitative information related to the objects identified as foreground objects. The quantitative information can be of various types, such as quantity, ratio (but not limited to these).

[0048] The storage unit 90 stores the inputs, outputs, and intermediate results of each component within the quantitative analysis device 1. The control unit 91 is connected to each of the aforementioned components of the quantitative analysis device 1 and controls the operation of these components. In other examples, each component may operate automatically as needed without being controlled by the control unit 91.

[0049] [Hardware Structure of Quantitative Analysis Device]

[0050] Figure 2 This is a schematic diagram illustrating an example of the hardware structure of the quantitative analysis device 1 in Embodiment 1. For example... Figure 2 As shown, in one example, the quantitative analysis device 1 includes a CPU 201, a memory 202, a storage device 206, an output device 203, an input device 204, a communication device 205, and a bus 207 that connects the components to each other. The number of these components may also be two or more.

[0051] CPU201 is a processor that reads various programs from memory 202 as needed and executes them accordingly. Figure 1 The functional components shown, including the input unit 10, object segmentation unit 11, threshold processing unit 13, learning unit 12, stop point determination unit 14, object classification unit 15, quantitative calculation unit 16, display unit 17, and control unit 91, can be implemented by the CPU 201 that performs the processing.

[0052] Storage device 206 stores various data used in quantitative analysis device 1 and can install reference data. Figure 1 The storage unit 90 is described above. For example, the storage device 206 stores the input image, object, learned model repeatedly learned by the learning unit, and classified images by category, as described above.

[0053] The output device 203 outputs object segmentation images (object separation images), classified object images, and quantitative information related to the objects, and is composed of a display device, a printer, a speaker, etc. For example, when using a display device as the output device 203, the segmentation image generated by the object segmentation unit 11, the classified object image obtained by the object classification unit 15, and the quantitative information related to the objects obtained by the quantitative calculation unit 16 are displayed on the screen of the display device.

[0054] The input device 204 is a device for user input instructions, etc., and consists of a keyboard, mouse, microphone, etc. For example, the input image that is the object of processing by the quantitative analysis device 1 is determined by the input device 204 together with the user's instructions.

[0055] The communication device 205 is a device for communicating with other devices. For example, it can receive input images transmitted from the imaging device, and it can also send and receive various data with a server connected via a network. In other examples, the communication device 205 can be connected to the quantitative analysis device 1 externally, instead of installing the communication device 205 inside the quantitative analysis device 1.

[0056] [The structure and actions of each component]

[0057] Next, the operation of each of the object segmentation unit 11, learning unit 12, threshold processing unit 13, stop point determination unit 14, object classification unit 15, quantitative calculation unit 16, and display unit 17 will be described in detail below.

[0058] Figure 3 This schematically illustrates an example of the processing performed by the object segmentation unit 11. Figure 3 In the example, image C is the input image, having an image background R1002, a foreground object R1003, and other unwanted objects R1004. Figure 3 In the diagram, the symbols R1003 and R1004 indicate each of the foreground objects and the unwanted objects.

[0059] Image CA represents an example of an input image after being transformed (segmented) by the object segmentation unit 11 into a foreground region (object) R2001 and a background region R2002. One of the foreground objects is indicated by the symbol R2001. The segmented (separated) image CA obtained by the object segmentation unit 11 is stored in the storage unit 90.

[0060] Figure 4 This schematically illustrates an example of the processing performed by the object classification unit 15. Figure 4In the image, image CA represents an example of an image separated (segmented) by object segmentation unit 11 into a background region R2002 and a foreground object R2001. Image CB represents an example of an input image after object classification.

[0061] In the classified images, symbols R4001, R4002, and R4003 indicate objects of different categories after classification by object classification department 15. Figure 4 In this process, symbols R4001, R4002, and R4003 are used to indicate an object belonging to a different category. The classified image CB obtained by the object classification unit 15 is stored in the storage unit 90.

[0062] Figure 5 This is an example illustrating the processing result of the display unit 17. The display unit 17 receives information from the object classification unit 15 and outputs an image, and provides prompts to the user in a predetermined image format DS1. For example, a pattern representing the category is overlaid on the actual object image OV1, and is presented along with quantitative information QN1 such as quantity and ratio.

[0063] [Object Segmentation Section 11]

[0064] The input image received by the input unit 10 may include unwanted objects and a general image background along with the object of interest. The object segmentation unit 11 transforms the input image into the desired background and foreground regions. The object segmentation unit 11 receives the image C, which is the output of the input unit 10, and performs processing to generate the segmented image CA. For example, a general segmentation method can be performed using a convolutional neural network.

[0065] As an example of the object segmentation unit 11, a segmented image with a foreground region R2001 and a background region R2002, as shown in image CA, can be generated by using a convolutional neural network to perform conventional convolution processing or the well-known fully convolutional operation. However, the same segmented image can be generated using any technique.

[0066] [Study Department 12]

[0067] Learning Unit 12 is activated when a user needs to learn from new data (training data). The new data can contain objects of both unknown and known types.

[0068] The task of the learning unit 12 is to generate a new model for the object segmentation unit 11 by discovering new unknown objects of interest in the given data. To achieve this task, the learning unit 12 receives an input image and labels of objects of known categories from the input unit 10.

[0069] One example of the learning unit 12 is that it can use a convolutional neural network to perform conventional convolution processing or well-known full convolution processing to generate a separated image with a foreground region R2001 and a background region R2002, as shown in image CA.

[0070] However, conventional convolutional neural network processing is insufficient for discovering all unknown objects of interest. One possible solution is for the learning unit 12 to use known category information to repeatedly learn from the input data to gradually discover all regions of unknown objects of interest. Alternatively, any method can be used to discover all regions of unknown objects of interest.

[0071] In each iteration, the learning unit 12 also generates candidate regions for unknown objects of interest. These candidate regions gradually increase as the learning unit 12 repeatedly learns from the given data. The learning unit 12 uses information from known objects to detect the similarity of patterns in other unknown objects.

[0072] In each iteration, more unknown objects of interest are learned and detected, and the pattern quantity of known objects cumulatively increases. Candidate regions are accumulated in each iteration cycle and can be used as new learned labels in subsequent iterations. However, all newly generated candidate regions are not necessarily actual areas of interest and sometimes contain unwanted objects. To remove these, the thresholding processing unit 13, described below, can be used.

[0073] [Threshold processing unit 13]

[0074] The threshold processing unit 13 can be started simultaneously with the learning unit 12, for example, when the user needs to learn from new data. The new data may contain objects of both known and unknown object types. When new data becomes the processing object of the object segmentation unit 11, it is not always possible to accurately discard unwanted objects. Therefore, the threshold processing unit 13 is used to avoid incorrect learning in the learning unit 12 due to unwanted objects becoming objects of interest.

[0075] In order for the learning unit 12 to learn regions that accurately detect new unknown objects of interest, it is important that the learning unit 12 can accurately distinguish between unknown objects of interest and unwanted objects in the image data. However, during repeated training in the learning unit 12, there are no teachers (labels) available for unknown objects, so it is possible to detect unwanted objects. To prevent this, a thresholding method is used to discard unwanted objects in the learning unit 12.

[0076] Figure 6 A flowchart illustrating an example of a method executed by the threshold processing unit 13 of Embodiment 1.

[0077] Step T1

[0078] The object segmentation unit 11 calculates a segmentation score for each pixel in the image. Based on the segmentation score, the pixels are classified as foreground or background. In step T1, the thresholding unit 13 receives the output of the object segmentation unit 11 and determines the segmentation score for each pixel of a known object.

[0079] Step T2

[0080] The threshold processing unit 13 calculates a histogram of the segmentation scores of all pixels of the known object.

[0081] Step T3

[0082] The threshold processing unit 13 determines the appropriate segmentation score for the pixels of the known object based on the generated histogram.

[0083] Step T4

[0084] Regarding unwanted objects and objects of interest, although not all, there are several features shared by known categories, so unwanted objects can sometimes be detected as objects of interest. Therefore, the segmentation score of unwanted pixels detected as foreground becomes a value smaller than the appropriate value mentioned above for foreground pixels of true objects of interest. Using this information, during the training of the segmentation model in learning unit 12, pixels with segmentation scores smaller than the appropriate value for foreground pixels of true objects of interest can be ignored.

[0085] Figure 7 This illustrates an example of the processing performed by the threshold processing unit 13. Figure 7 In the image, image CA represents an example of the output of object segmentation unit 11, containing background region R2002, multiple foreground objects R2001, and multiple other unwanted objects R2003. Image CT represents an example of the output of thresholding unit 13, where unwanted objects (regions) R2003 have been removed. Image CT can be used as an accurate label for model training in learning unit 12. Both image CA and image CT are stored in storage unit 90.

[0086] [Stop Point Determination Unit 14]

[0087] The stop point determination unit 14 can be activated when the learning unit 12 starts. For example, it can be activated when the user needs to learn new data through the device. The stop point determination unit 14 determines the appropriate number of repetition steps required for the learning unit 12 to properly learn all new unknown objects of interest in the data.

[0088] In order for the learning unit 12 to correctly learn new unknown object of interest regions, it is important that the learning unit 12 repeatedly learns the training data received from the input unit 10, cumulatively learning and discovering new unknown objects of interest. However, in order to determine the stopping point, it is important to determine an appropriate number of repetitions. To determine an appropriate stopping point, a stopping point determination unit 14 is used. The stopping point determination unit 14 can use information about the detection accuracy of the background region and the detection accuracy of the foreground (object of interest) region to determine the stopping point of the learning unit 12. The detection accuracy of the background or foreground can be represented, for example, by the proportion of known background pixels or known foreground pixels that are determined as background or foreground.

[0089] Figure 8 This illustrates an example of the method executed by the stop point determination unit 14 in Embodiment 1. The accuracy mentioned in the following steps may also be the accuracy of a single model output or a moving average of the accuracies of multiple model outputs.

[0090] Step S0

[0091] The stop point determination unit 14 calculates the accuracy of the learning unit 12 for the evaluation data. The evaluation data can include known objects in the foreground region, known background regions, and unknown regions.

[0092] Step S1

[0093] If the accuracy of the inference AFt of the new model related to the foreground region of the known object in the evaluation data increases, such that AFt > AFt-1, the stop point determination unit 14 proceeds to step S2; otherwise, it proceeds to step S4.

[0094] Step S2

[0095] If the accuracy of the new model related to the known background region ABt of the evaluation data increases, such that ABt > ABt-1, the stop point determination unit 14 proceeds to step S5; otherwise, it proceeds to step S3.

[0096] Step S3

[0097] When the accuracy of the new model related to the known background region of the evaluation data decreases for the background region, resulting in ABt < ABt-1, the stop point determination unit 14 checks whether the background-related accuracy is within the allowable level Ba. If ABt > Ba (within the allowable level), the stop point determination unit 14 proceeds to step S5; otherwise, it proceeds to step S4. This improves the detection accuracy of the foreground region and allows for obtaining the desired level of background detection accuracy. The allowable level is, for example, preset by the user.

[0098] Step S4

[0099] End the repetition.

[0100] Step S5

[0101] Repeat this process once more.

[0102] Figures 9A to 9C These are three examples illustrating the collaborative processing flow of the different functional units described above in order to repeatedly detect areas of unknown and known objects. The quantitative analysis device 1 can execute these processes separately, eliminating the need for functional units required to execute each process. Figure 9A This indicates the basic collaboration between the input unit 10, the object segmentation unit 11, and the learning unit 12. Figure 9B Is Figure 9A An example of discarding unwanted garbage areas by the threshold processing unit 13 has been added.

[0103] Figure 9C Is Figure 9B An example of automatically determining the appropriate number of iterations by the stop point determination unit 14 is added. Throughout the process, the aim is to use only the information of known objects as the initial segmentation label, and to add the newly generated segmentation labels through learning to the existing segmentation labels so that they can be used for model training in the next iteration.

[0104] Reference Figure 9A This describes an example of the coordinated operation of the input unit 10, the object segmentation unit 11, and the learning unit 12 when learning a new, unknown object, as illustrated in Embodiment 1. This is one of the basic structures of an embodiment described in this specification.

[0105] Step UD1

[0106] The object segmentation unit 11 receives new data from the input unit 10 and outputs segmentation results as provisional foreground and background segmentation labels. The provisional labels are based on the output of past models and can change according to newly discovered foreground interest regions by the model generated in subsequent iterations.

[0107] Step UD2

[0108] Learning Unit 12 uses the aforementioned segmentation labels to train a new segmentation model M.

[0109] Step UD3

[0110] The object segmentation unit 11 receives a new model M from the learning unit 12 and adds the new model M to the existing model. In this way, the model held by the object segmentation unit 11 accumulates with repeated iterations, which can increase the number of models that can be used in the segmentation of step UD1.

[0111] Step UD4

[0112] If further iterations are required based on user requests or a predetermined total number of iterations, the process returns to step UD1.

[0113] refer to Figure 9B This example illustrates the coordinated operation of the input unit 10, object segmentation unit 11, learning unit 12, and threshold processing unit 13 when learning a new, unknown object, as described in Embodiment 1. This structure is relative to… Figure 9A The basic structure shown also includes the processing of the threshold processing unit 13.

[0114] Step UDT1

[0115] The object segmentation unit 11 receives new data from the input unit 10 and outputs the segmentation result based on the output of one or more existing models (e.g., all existing models).

[0116] Step UDT2

[0117] The thresholding unit 13 receives the output from the object segmentation unit 11 and performs thresholding processing on the pixels of the segmentation labels according to a preset strategy, as described above. The thresholding unit 13 generates provisional foreground and background segmentation labels. The provisional labels are based on the output of the existing model of the object segmentation unit 11 and can change in subsequent iterations based on the foreground interest region newly discovered by the next model.

[0118] Step UDT3

[0119] The learning unit 12 receives the output of the threshold processing unit 13 and uses the segmentation labels to train a new segmentation model M.

[0120] Step UDT4

[0121] The object segmentation unit 11 takes the new model M obtained from the learning unit 12 and adds it to the existing model. In this way, the model of the object segmentation unit 11 is accumulated. In step UDT1, the object segmentation unit 11 outputs the segmentation result using the accumulated model.

[0122] Step UDT5

[0123] If further iterations are needed, the process returns to step UDT1; otherwise, it ends.

[0124] Figure 9C This example illustrates the coordinated operation of the object segmentation unit 11, learning unit 12, threshold processing unit 13, and stop point determination unit 14 when learning new, unknown objects, as described in Embodiment 1. This structure is relative to… Figure 9B The basic structure shown also includes the processing of the stop point determination unit 14.

[0125] Step UDTS1

[0126] The object segmentation unit 11 receives new data from the input unit 10 and outputs the segmentation result based on the output of one or more existing models (e.g., all existing models).

[0127] Step UDTS2

[0128] The thresholding unit 13 receives the output from the object segmentation unit 11 and performs thresholding processing on the pixels of the segmentation labels according to a preset strategy, as described above. The thresholding unit 13 generates provisional foreground and background segmentation labels. The provisional labels are based on the output of the existing model of the object segmentation unit 11 and can change in subsequent iterations based on the foreground interest region newly discovered by the next model.

[0129] Step UDTS3

[0130] The learning unit 12 receives the output of the threshold processing unit 13 and uses the known segmentation labels to train a new segmentation model M.

[0131] Step UDTS4

[0132] The stopping point determination unit 14 evaluates the new model M.

[0133] Step UDTS5

[0134] For reference Figure 8 As explained, the stop point determination unit 14 determines whether a repetition needs to be performed next. If the output of the stop point determination unit 14 indicates that a repetition is required, the process proceeds to step UDTS1. Otherwise, it proceeds to step UDTS6.

[0135] Step UDTS6

[0136] The object segmentation unit 11 will add the new model M obtained from the learning unit 12 to the existing model.

[0137] [Object Classification Section 15]

[0138] The object classification unit 15 includes a classifier (e.g., a convolutional neural network that uses convolutional output values ​​to classify object types) that classifies the separated foreground regions into known object categories and unknown object groups.

[0139] For example, when using a convolutional neural network as a classifier, features are calculated using a classification model trained beforehand to classify known object categories. The classification model consists of a feature extraction unit and a recognition unit. As an example, the classification model may consist of a multi-layer convolutional neural network, with the final layer serving as the recognition unit and the remaining layers as the feature extraction unit.

[0140] During training, the feature extraction unit and the recognition unit are trained so that the output of the recognition unit can classify known object categories. The output of the recognition unit represents the types (categories) of known objects, but it cannot classify groups of unknown objects. Therefore, clustering is performed using the output of the feature extraction unit to classify both known object categories and groups of unknown objects.

[0141] As a clustering method, one example is as follows: Images of two objects are input to the feature extraction unit, and the resulting feature vectors are compared. If the similarity, such as cosine similarity, is above a certain threshold, the objects are classified as belonging to the same category. This threshold can be predetermined by the user or automatically calculated by the device based on the known similarity between object categories.

[0142] The above is an example. As an indicator of the similarity of feature vectors, Euclidean distance or Manhattan distance can be used. As a clustering method, unsupervised learning methods such as self-organizing maps and the EM algorithm can be used. Furthermore, as features used in clustering, machine learning methods other than convolutional neural networks can be used, or handcrafted features such as HOG and SIFT can be used. Alternatively, the results of clustering as described above can be used as training data, utilizing results obtained through classifier-oriented machine learning.

[0143] Figure 10 Referring to Embodiment 1, a flowchart example of a technique that can be used as the object classification unit 15 is shown. The object segmentation unit 11 receives input data from the input unit 10 and outputs the image CA after segmentation processing using one or more models, such as all models, to the object classification unit 15.

[0144] Step CON1

[0145] The foreground region of image CA is segmented into multiple small pieces that can be input into the classifier of object classification unit 15. The size of the small pieces depends on the amount of memory allowed by the device.

[0146] Step CON2

[0147] The object classification unit 15 selects a new piece from the multiple pieces generated in step CON1.

[0148] Step CON3

[0149] The object classification unit 15 inputs the small piece selected in step CON2 into the classifier. The classifier classifies the foreground region of the original image from the input unit 10 into a known object type or an unknown object type based on the foreground region from the object segmentation unit 11. Regarding unknown objects, they can be classified into various object types based on the classifier output value (e.g., convolution output value) from the convolutional neural network.

[0150] Step CON4

[0151] The object classification section 15 checks whether all small pieces have been classified. If "yes", it proceeds to step CON5; if "no", it returns to step CON2.

[0152] Step CON5

[0153] The object classification unit 15 arranges the classified small pieces and stores them as images CB in the storage device 206.

[0154] [Quantitative Calculation Section 16]

[0155] The quantitative calculation unit 16 calculates the quantitative information of the classified foreground object R2001 in the image CB. For example, as shown in Embodiments 2 and 3, the quantitative information can also be used as special information for other tasks.

[0156] Figure 11 This is a flowchart illustrating the operation of a quantitative processing example performed by the quantitative calculation unit 16. In this example, the quantitative information calculated is the total number of objects obtained by counting the detected objects, and the ratio of objects of interest in the input image CB.

[0157] Step Q1

[0158] The quantitative calculation unit 16 receives the output of the object classification unit 15, i.e., image CB, and selects one of the combined foreground regions.

[0159] Step Q2

[0160] The quantitative calculation unit 16 determines the category of the region based on the image CB, which is the classification result of the object classification unit 15, and increments the counter of that specific known category or unknown group by 1 unit.

[0161] Step Q3

[0162] The quantitative calculation unit 16 determines whether all foreground regions in image CB have been counted. If yes, proceed to step Q4; otherwise, return to step Q1.

[0163] Step Q4

[0164] The quantitative calculation unit 16 summarizes the total values ​​for each category.

[0165] Step Q5

[0166] The quantitative calculation unit 16 calculates the ratio of each category.

[0167] The quantitative information from steps Q4 and Q5 is output and stored in storage device 206. The quantitative information can also be displayed using a display device within output device 203. In the example above, the quantitative information is the number of objects and their ratio. Using this information, samples can be analyzed using various methods.

[0168] [Display Unit 17]

[0169] Display unit 17 displays the output of quantitative calculation unit 16 to the user. The displayed information may include, for example, the ratio of known objects to unknown object groups. The output of display unit 17 can be displayed overlaid with the input image, and the overlaid information may indicate, for example, objects belonging to known categories and unknown groups.

[0170] Figure 12 This section describes an example of the associated operation of the object segmentation unit 11, object classification unit 15, quantitative calculation unit 16, and display unit 17 of Embodiment 1 when the quantitative analysis device 1 of Embodiment 1 needs to provide quantitative information to the user, for example, during the estimation stage.

[0171] Step IN1

[0172] The object segmentation unit 11 receives new data from the input unit 10 and outputs the segmentation result based on the output of one or more existing models (e.g., all existing models).

[0173] Step IN2

[0174] The object classification unit 15 receives the output of the object segmentation unit 11 and classifies the objects of interest into one or more known object types and one or more unknown object groups.

[0175] Step IN3

[0176] The quantitative calculation unit 16 receives the output from the object classification unit 15 and generates the required quantitative information.

[0177] Step IN4

[0178] The display unit 17 receives quantitative information from the quantitative calculation unit 16 and displays the quantitative information to the user.

[0179] As described above, according to the embodiments of this specification, the input image from the input unit 10 is converted into an image representing the foreground object region and the background region.

[0180] Furthermore, the foreground area is classified using the object classification unit 15, which can accurately determine the precise category of each known object and one or more unknown object groups.

[0181] Quantitative information obtained from the classified image, such as the number of objects and the object ratio, can be obtained from the quantitative calculation unit 16. Furthermore, the results of the quantitative and qualitative information can be presented to the user along with the input image via the display unit 17. The user can use the provided information to study the characteristics of the detected objects and their co-occurrence patterns, and can determine the phenomenon related to the detected objects based on these results.

[0182] The following is a summary of Embodiment 1 described above in this specification. The quantitative analysis device 1 includes an input unit 10 and an object segmentation unit 11. The input unit 10 receives an image from an imaging device, for example, and outputs it to the object segmentation unit 11. The object segmentation unit 11 is capable of efficiently and accurately representing the desired background region and the foreground region containing the object of interest from the input image.

[0183] The quantitative analysis device 1 also includes a learning unit 12, which is capable of learning new groups of unknown objects from the data. The output from the learning unit 12 is a new segmentation model that can be used in the object segmentation unit 11. The learning unit 12 can cumulatively reuse the data to detect regions of unknown objects. The object segmentation unit 11 can use the model from the learning unit 12 to detect new unknown objects.

[0184] The quantitative analysis device 1 also includes a threshold processing unit 13. The threshold processing unit 13 can prevent the learning unit 12 from detecting unwanted objects as objects in the region of interest. By using the threshold processing unit 13, the learning unit 12 can generate a high-precision model, and the object segmentation unit 11 can accurately separate (segment) known objects, unknown object groups, and background regions based on the model.

[0185] The quantitative analysis device 1 also includes a stop point determination unit 14, which is capable of determining the optimal number of repetitions in the learning unit 12. The learning unit 12 learns new unknown objects through repetition. Therefore, determining the optimal number of repetitions is important for the optimal operation of the learning unit 12. The stop point determination unit 14 uses the data from the learning unit 12 and determines the stop point of the repetitions of the learning unit 12 based on a predetermined quality strategy.

[0186] The quantitative analysis device 1 also includes an object classification unit 15, capable of efficiently and accurately classifying objects into known categories and unknown object groups. As an example, a general convolutional neural network can be optimized for this task. Convolutional neural networks are image feature capture models that do not require manual training and can automatically learn features from images, enabling them to group unknown objects based on predetermined classification scores.

[0187] The quantitative analysis device 1 also includes a quantitative calculation unit 16, which is capable of efficiently and accurately calculating quantitative information for the foreground object. As an example, as described above, it calculates the quantity of each category of the object and the ratio of the objects.

[0188] The quantitative analysis device 1 also includes a display unit 17, which can display the output of the quantitative calculation unit 16 to the user. For example, the displayed information may include the ratio of known objects to unknown object groups. The output of the display unit 17 can be overlaid on the input image, and this information may, for example, represent objects belonging to known categories and unknown groups.

[0189] According to the structure of this embodiment, known and unknown objects of interest can be accurately identified within an image, and quantitative information for these objects can be accurately determined. Furthermore, for example, the input unit 10, object segmentation unit 11, learning unit 12, threshold processing unit 13, and stop point determination unit 14 can be installed in a device or system different from the quantitative analysis device, or the threshold processing unit 13 and stop point determination unit 14 can be omitted depending on the design of the device or system.

[0190] Example 2

[0191] Sometimes it is necessary to detect anomalous objects in a given image and explicitly inform the user of these anomalous objects. This situation may occur, for example, when it is assumed that most objects within the image are known objects. Therefore, the anomaly detection system of Embodiment 2 will be described below.

[0192] The anomaly detection system A1 identifies objects of interest in the unknown group classified by the object classification unit 15 as anomalous objects. The anomaly detection system A1 then displays an overlay of the input image via the display unit 17, thereby providing the user with information about the detected anomalous objects.

[0193] Figure 13 The functional structure of the anomaly detection system A1 in Embodiment 2 is illustrated schematically. The anomaly detection system A1 has the following functions: Figure 1 The quantitative analysis device 1 shown is structured without the quantitative calculation unit 16. Furthermore, in the anomaly detection system A1, the display unit 17 can provide the user with information about detected anomalies by overlaying an emphasized display area onto the input image.

[0194] Figure 14 The diagram illustrates an example of the hardware structure of the anomaly detection system A1 in Embodiment 2. For example... Figure 14 As shown, the anomaly detection system A1 can have the same hardware structure as the quantitative analysis device 1. In embodiment 2, the CPU 201 does not function as the quantitative calculation unit 16.

[0195] Example 3

[0196] Figure 15The diagram schematically illustrates a structural example of an image inspection system W1 including a quantitative analysis device 1 according to Embodiment 3. The quantitative analysis device 1 sometimes needs to provide quantitative information within the image inspection system W1, providing quantitative information for a series of images captured at different time frames.

[0197] The image inspection system W1 may include an imaging device W11, which may capture a series of image frames of an object in formats such as motion JPEG, MPEG, H.264, or HD / SDI. For example, an RGB camera or any type of camera or imaging device may be used.

[0198] The image inspection system W1 may include the quantitative analysis device 1 of Embodiment 1, which generates quantitative information for objects in a series of images. The image inspection system W1 may include an unknown sample analysis device W12. Using the unknown sample analysis device W12, the characteristics of the object can be analyzed based on the quantitative information of the unknown sample. The image inspection system W1 may include a display device W13. The display device W13 can display information about the analysis results obtained by the unknown sample analysis device W12 based on the quantitative information of the sample obtained from the quantitative analysis device 1.

[0199] The image inspection system W1 can capture a series of images in the imaging device W11, and can obtain quantitative information of the input images for different time frames via the quantitative analysis device 1. The unknown sample analysis device W12 can display the analysis results of changes in the quantitative information of the sample within a predetermined period.

[0200] The image inspection system W1 can acquire a series of image pairs simultaneously or separately. An image pair is, for example, a pair of images captured at time t1 and at time t2, or an image of a sample to which an external intervention method was applied and an image of a sample to which the intervention method was not applied. Using the quantitative analysis device 1, the image inspection system W1 calculates the quantitative information of the input image pairs. Using the unknown sample analysis device W12, for example, it is possible to analyze the changes in quantitative information between samples to which the external intervention method was applied and samples to which the external intervention method was not applied.

[0201] The analysis results from the unknown sample analysis device W12 can be displayed to the user via the display device W13. By studying the output of the display device W13, the user can understand the effect of intervention methods on the sample within a single time frame or a predetermined period. For example, for a certain material, when the unknown object discovered is magnetic particles with different physical properties, its pattern and diameter may sometimes be clearly normal, or when intervention methods such as heat or pressure are applied, it may be clearly the behavior of magnetic particles within a cluster.

[0202] Figure 16In another example of Embodiment 3, the quantitative analysis device 1 is installed within the server W3. The server W3 acquires an input image from the imaging device W11 of the image inspection system W2 via the communication device W22 of the image inspection system W2. The server W3 stores the acquired image in the storage device W31. The communication device W32 transmits the image received from the storage device W31 to the quantitative analysis device 1 of Embodiments 1 and 3.

[0203] According to Embodiments 1 and 3, the quantitative analysis device 1 in server W3 calculates quantitative information about objects in an image and sends it to image inspection system W2 via communication devices W32 and W22.

[0204] Information received by the communication device W22 of the image inspection system W2 is input to the unknown sample analysis device W12. The unknown sample analysis device W12 analyzes the characteristics based on the quantitative information of the object. For example, when the input image is of magnetic particles, quantitative analysis of the magnetic particle density can sometimes clarify the characteristics of the magnetic particles in the sample. The analysis results are input to the display device W13 to provide information to the user.

[0205] The display device W13 can display information related to the objects in the image and the analysis results of the unknown sample analysis device W12. Users can draw various conclusions by referring to the displayed results. For example, they can check the quality of the sample. The display device can be either a desktop or portable type.

[0206] Furthermore, the present invention is not limited to the above embodiments and includes various modifications. For example, the above embodiments are described in detail for ease of understanding of the present invention and are not limited to having all the described structures. In addition, a part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and it is also possible to add the structure of another embodiment to the structure of a certain embodiment. Furthermore, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0207] Furthermore, some or all of the aforementioned structures, functions, and processing units can be implemented in hardware, for example, through integrated circuit design. Alternatively, the aforementioned structures and functions can be implemented in software by having a computing device interpret and execute programs to implement each function. The programs, tables, files, and other information implementing each function can be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards or SD cards.

[0208] Furthermore, the control lines and information lines shown are those deemed necessary for the description, but do not necessarily represent all control lines and information lines on the product. In fact, it can be assumed that almost all structures are interconnected.

Claims

1. An apparatus for generating a segmentation model for identifying an object of interest from an image, characterized in that, The device includes: The input section receives the input image; An object segmentation unit receives the input image from the input unit, calculates a segmentation score for each pixel of the input image using an existing segmentation model, and separates the region of interest and the background region based on the segmentation score; The learning unit learns the segmentation model used by the object segmentation unit. Threshold processing unit; as well as The stopping point determination unit determines whether to proceed with the training of a new segmentation model in the learning unit. The learning unit uses training data and the output of the existing segmentation model based on the training data to train a new segmentation model, and appends the trained new segmentation model to the existing segmentation model. This process of training the new segmentation model and appending it to the existing segmentation model is repeated. The threshold processing unit determines whether an unknown object is an unwanted object by comparing the segmentation score of pixels in the unknown object detected by the existing segmentation model with a threshold, and outputs the label of the known object of interest. The stopping point determination unit determines whether to execute the training of the next segmentation model based on the accuracy of the estimation of the segmentation model trained in this session and the accuracy of the estimation of the segmentation model trained in the previous session.

2. A quantitative analysis device for calculating quantitative information of an object of interest in an image, characterized in that, The quantitative analysis device includes: The apparatus according to claim 1; The object classification department classifies the objects of interest separated from the object segmentation into known categories and unknown groups; A quantitative calculation unit that calculates quantitative information for the known category and the unknown group; as well as The display unit displays the quantitative information calculated by the quantitative calculation unit to the user. The display unit displays a result on the input image by overlaying at least one of the quantitative information calculated by the quantitative calculation unit and the classification results of the known category and the unknown group of the object of interest calculated by the object classification unit.

3. The quantitative analysis device according to claim 2, characterized in that, The existing segmentation model is used to separate the object of interest from the input image data. The separated objects of interest are classified into known categories and unknown groups. The objects in the unknown group are treated as exception objects, and the output image is highlighted in the input image.

4. An image inspection system, characterized in that, Include: The quantitative analysis device according to claim 2; A camera device that captures a series of images; An unknown sample analysis device that analyzes the characteristics of an unknown sample based on the output of the quantitative analysis device; as well as A display device that displays the analysis results output by the unknown sample analysis device. The quantitative analysis device, for each image in the series of images, uses the existing segmentation model to separate the object of interest from the input image data, classifies the separated object of interest into known categories and unknown groups, and calculates quantitative information for the known categories and the unknown groups. The unknown sample analysis device outputs the analysis results of the changes in the quantitative information in the series of images.

5. An image inspection system, characterized in that, Include: The quantitative analysis device according to claim 2; The imaging device acquires a series of images of a first sample and a series of images of a second sample to which an external intervention method has been applied; An unknown sample analysis device that analyzes the characteristics of an unknown sample based on the output of the quantitative analysis device; as well as A display device that displays the analysis results output by the unknown sample analysis device. The quantitative analysis device, using the existing segmentation model, separates the object of interest from the input image data for each image in a series of images of the first sample and the second sample. The separated object of interest is then classified into known categories and unknown groups, and quantitative information is calculated for the known categories and the unknown groups. The unknown sample analysis device outputs the differences in quantitative information in a series of images of the first sample and the second sample as the analysis result.

6. A method for generating a segmentation model for identifying objects of interest from an image, characterized in that, The method includes: Input steps, receive input image; The object segmentation step involves receiving the input image, calculating a segmentation score for each pixel of the input image using an existing segmentation model, and separating the region of interest from the background region based on the segmentation score. as well as The learning steps involve learning the segmentation model used in the object segmentation steps. In the learning step, a new segmentation model is trained using training data and the output of an existing segmentation model based on the training data. The trained new segmentation model is then appended to the existing segmentation model. This process of training the new segmentation model and appending it to the existing segmentation model is repeated. The method further includes a thresholding step, in which the segmentation scores of pixels in the unknown object of interest detected by the existing segmentation model are compared with a threshold to determine whether the unknown object of interest is an unwanted object, and the label of the known object of interest is output. In the learning step, the new segmentation model is learned using the training data and the labels of the objects of interest calculated in the thresholding step. The method further includes a stopping point determination step to determine whether to execute the training of the next new segmentation model in the learning step. In the stopping point determination step, based on the accuracy of the estimation of the segmentation model trained this time and the accuracy of the estimation of the segmentation model trained last time, it is determined whether to execute the training of the next segmentation model.

7. A quantitative analysis method for calculating quantitative information of an object of interest in an image, characterized in that, The quantitative analysis method includes: Perform the steps of the method described in claim 6; The object classification step classifies the objects of interest separated by the object segmentation step into known categories and unknown groups; The quantitative calculation step involves calculating quantitative information for the known category and the unknown group. as well as The display steps prompt the user with the quantitative information calculated in the quantitative calculation steps. In the display step, a result is displayed on the input image by overlaying at least one of the quantitative information calculated in the quantitative calculation step and the classification results of the known category and the unknown group of the object of interest calculated in the object classification step.

Images