Image processing device, method and image processing system

Abstract

To provide an apparatus, method, and system for identifying a target object in an image accurately at a high speed.SOLUTION: An image processing apparatus includes: an image input unit 201 which receives an input image; an image generation unit 202 which generates an image for feature quantity extraction from the input image; a feature quantity extraction unit 203 which extracts feature quantities from the generation image generated by the image generation unit; an identifying unit 205 which identifies an object in the image with the feature quantities output by the feature quantity extraction unit; an output unit 206 which outputs an identification result output by the identifying unit; and a feature quantity map generation unit 204 which instructs the image generation unit to generate a new generation image as necessary, based on the feature quantities output by the feature quantity extraction unit, and generates a feature quantity map indicating a feature quantity extraction condition for the new generation image, to be output to the feature quantity extraction unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, method and system for identifying an object in an image. [Background technology]

[0002] In recent years, area segmentation and classification using machine learning such as deep learning are often used to analyze images captured by cameras or microscopes. While classification using machine learning can achieve high classification accuracy, it has the problem that the processing cost is generally higher than classification using manually designed features. When real-time performance or execution on a device with low computing power is required, it is necessary to reduce the amount of calculation. For example, Patent Document 1 states that "area segmentation is performed on a low-resolution image, and the area near the boundary in the result of the area segmentation is used as the processing target area, and area segmentation is performed on a high-resolution image." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-176281 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 is a method for improving the accuracy of boundaries of region division results for low-resolution images by using features extracted from high-resolution images, so it is necessary to divide each region into the correct class at the low-resolution image stage. However, in order to correctly divide an input image into regions or identify it into a specific class, a high-resolution image is often required. For example, in cell or bacteria analysis using a microscopic image, it is difficult to correctly estimate the state of each cell or bacteria using only a low-resolution image. Also, in semiconductor inspection using a microscopic image, there is a possibility that an abnormality that cannot be determined using only a low-resolution image may be included. Even in inspection using a visible light image, for example, the class of the entire region may be determined by the presence or absence of characters or patterns that exist locally. In these cases, it is difficult to divide the region into the correct class using only a low-resolution image, and it is only possible to identify the class using the features of the high-resolution image.

[0005] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to provide an image processing device, method and system that performs area division and classification processing with high accuracy and high speed by quickly extracting the minimum amount of features required for classification. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the image processing device disclosed in the present invention is an image processing device that identifies objects in an image, and includes an image input unit that accepts an input image, an image generation unit that generates an image for feature extraction from the input image, a feature extraction unit that extracts features from a generated image generated by the image generation unit, an identification unit that identifies objects in the image using the features output by the feature extraction unit, an output unit that outputs the identification result output by the identification unit, and a feature map generation unit that instructs the image generation unit to generate a new generated image as necessary based on the features output by the feature extraction unit, and generates a feature map indicating feature extraction conditions for the new generated image and outputs it to the feature extraction unit. Moreover, the image processing method disclosed in the present invention is an image processing method for identifying an object in an image, and includes an image input step of accepting an input image, an image generation step of generating an image for feature extraction from the input image, a feature extraction step of extracting features from a generated image generated by the image generation step, an identification step of detecting or identifying an object in the image using the features calculated in the feature extraction step, and an output step of outputting the identification result calculated in the identification step, and further includes a feature map generation step of causing the image generation step to generate a new generated image as necessary based on the features output by the feature extraction step, and generating a feature map indicating feature extraction conditions for the new generated image and reflecting it in processing by the feature extraction step. Effect of the Invention

[0007] According to the present invention, it is possible to provide an apparatus, method and system for identifying a target object in an image with high accuracy and high speed. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to a first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a functional block diagram of an image processing apparatus according to a first embodiment. [Diagram 3] FIG. 13 is a diagram illustrating an example of a feature amount extraction map. [Figure 4] FIG. 11 is a diagram illustrating an example of pixel-unit discrimination processing. [Diagram 5] FIG. 4 is a diagram illustrating an example of a processing flow of an image processing method according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a hardware configuration of an image processing system according to a third embodiment. [Figure 7] 1A and 1B are diagrams illustrating an example of input and output of an image processing device in determining an object placement. [Figure 8A] FIG. 13 is a diagram showing an example of display of an object placement verification result. [Figure 8B]FIG. 13 is a diagram showing an example of display of an object placement verification result. [Figure 8C] FIG. 13 is a diagram showing an example of display of an object placement verification result. [Figure 9] FIG. 13 is a diagram illustrating an example of a hardware configuration of an image processing system according to a fourth embodiment. [Figure 10A] FIG. 11 is a diagram showing an example of an input image to the image processing device 100 according to the third embodiment. [Figure 10B] FIG. 11 is a diagram showing an example of an input image to the image processing device 100 according to the third embodiment. [Figure 11] FIG. 13 is a diagram showing an example of display of an object placement verification result using a certainty factor. [Figure 12] FIG. 11 is a diagram illustrating an example of a hardware configuration of an image processing system according to a second embodiment. [Figure 13A] FIG. 13 is a diagram showing an example of a region division result according to the fourth embodiment. [Figure 13B] FIG. 13 is a diagram showing an example of a region division result according to the fourth embodiment. [Figure 14] FIG. 13 is a diagram showing an example of a correspondence table between cup types, raised-bottom containers, and container types. [Figure 15] FIG. 13 is a diagram showing an example of a method for calculating the length of a raised container. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of an image processing device, method, and system according to the present invention will be described with reference to the accompanying drawings. In the following description and the accompanying drawings, components having the same functional configuration are designated by the same reference numerals, and duplicated explanations will be omitted. EXAMPLES

[0010] <Hardware configuration of image processing device> The hardware configuration of the image processing device according to the first embodiment will be described with reference to Fig. 1. The image processing device 100 includes an interface unit 110, a calculation unit 111, a memory 112, and a bus 113. The interface unit 110, the calculation unit 111, and the memory 112 transmit and receive information via the bus 113.

[0011] Each unit of the image processing device 100 will be described. The interface unit 110 is a communication device that transmits and receives signals to and from devices external to the image processing device 100. Devices that communicate with the interface unit 110 include an imaging device 120 such as a camera or a microscope, and a display device 121 such as a monitor or a printer.

[0012] The calculation unit 111 is a device that executes various processes within the image processing device 100, and is, for example, a CPU (Central Processing Unit) or an FPGA (Field-Programmable Gate Array). Functions executed by the calculation unit 111 will be described later with reference to FIG. The memory 112 is a device that stores the programs executed by the calculation unit 111, parameters, coefficients, processing results, etc., and is an HDD, SSD, RAM, ROM, flash memory, or the like.

[0013] <Functional configuration of image processing device> 2 is an example of a functional block diagram according to a first embodiment of the image processing device 100. Each of these functional units may be realized by software that runs on the calculation unit 111, or may be realized by dedicated hardware.

[0014] The image processing device 100 includes, as functional units, an image input unit 201, an image generation unit 202, a feature amount extraction unit 203, a feature amount map generation unit 204, a classification unit 205, and an output unit 206. Each functional unit will be described below.

[0015] The image input unit 201 receives an input image, such as a visible light image or a microscope image, input from the interface unit 110 .

[0016] The image generating unit 202 generates an image for feature extraction using the input image. When the image generating unit 202 receives a generated image signal from a feature map generating unit 204 (described later), the image generating unit 202 determines an image to be generated based on the generated image signal. When no generated image signal is input, an image is extracted with a predetermined setting.

[0017] The feature extraction unit 203 extracts features from the image generated by the image generation unit 202. When the feature extraction unit 203 receives a feature extraction map from a feature map generation unit 204 (described later), the feature extraction unit 203 extracts features only for areas and feature types in the feature extraction map that have values ​​equal to or greater than a preset threshold value. When there is no input of a feature extraction map, the feature extraction unit 203 extracts all types of features from the entire image, or extracts only predetermined areas and feature types.

[0018] A feature amount map generating unit 204 generates a feature amount extraction map by using the feature amount output by the feature amount extraction unit 203. The feature amount extraction map specifies the image to be generated next by the image generating unit 202, and the region and type of feature amount from which the feature amount extraction unit 203 will next extract features.

[0019] The identification unit 205 performs identification processing using all or part of the features output by the feature extraction unit 203. The identification processing includes image-based identification processing for classifying the input image into predetermined classes, pixel-based identification processing (area division), processing for detecting the position of an object by a rectangle, etc. This identification processing makes it possible to detect the presence of an object and to distinguish the type of object.

[0020] The output unit 206 outputs the classification result output by the classification unit 205 to the outside of the device. When the classification result is on an image basis, the class information and likelihood information for each image may be output in the form of numerical data, or the information may be output in a visually understandable form such as a character string. When the classification result is on a pixel basis, the class and likelihood information for each pixel may be output in the form of numerical data, or the information may be output as an image in which the class is represented by a unique color. When the classification result is rectangle information, the class, likelihood, and rectangle information for each rectangle may be output in the form of numerical data, or the information may be output as an image in which the rectangle and class information are represented on the image by a frame and color, or the like.

[0021] The above functions do not need to be configured according to the functional units in Fig. 2, but only need to be able to perform processing to realize the operation of each functional block. Fig. 5 shows an example of a processing flow diagram according to the first embodiment. Each step corresponds to each element in the functional block diagram shown in Fig. 2.

[0022] In the image input step 501 , an input image such as a visible light image or a microscope image input from the interface unit 110 is accepted.

[0023] In the image generating step 502, an image for feature extraction is generated using the input image. When the image generating step 502 receives a generated image signal generated in the feature map generating step 504 described later, the image to be generated is determined based on the generated image signal. When there is no input of a generated image signal, the image is generated with a preset setting.

[0024] In the feature extraction step 503, features are extracted from the image generated in the image generation step 502. When the feature extraction step 503 receives a feature extraction map from the feature map generation step 504 described later, the feature extraction step 503 extracts features only for areas and feature types in the feature extraction map that have values ​​equal to or greater than a preset threshold value. When there is no input of a feature extraction map, all types of features are extracted from the entire image, or only predetermined areas and feature types are extracted.

[0025] In the feature amount map generation step 504, a feature amount extraction map is generated using the feature amounts calculated in the feature amount extraction step 503. The feature amount extraction map specifies the image to be generated next in the image generation step 502 and the type and area of ​​the feature amount to be extracted next in the feature amount extraction step 503.

[0026] In the identification step 505, identification processing is performed using all or a part of the features output in the feature extraction step 503. The identification processing includes image-based identification processing for classifying the input image into predetermined classes, pixel-based identification processing (area division), detection processing for identifying the position of an object by a rectangle, and the like.

[0027] In the output step 506, the classification result calculated in the classification step 505 is output to the outside of the device. When the classification result is on an image basis, the class information and likelihood information for each image may be output in the form of numerical data, or the information may be output in a visually understandable form such as a character string. When the classification result is on a pixel basis, the class and likelihood information for each pixel may be output in the form of numerical data, or the information may be output as an image in which the class is represented by a unique color. When the classification result is rectangle information, the class, likelihood, and rectangle information for each rectangle may be output in the form of numerical data, or the information may be output as an image in which the rectangle and class information are represented on the image by a frame and color, or the like.

[0028] Hereinafter, detailed operations will be described with each functional unit as the subject, but each step corresponding to each functional unit may also be interpreted as the subject.

[0029] <Configuration and operation of each part> Among the functional units, the operations of the image generating unit 202, the feature extracting unit 203, the feature map generating unit 204, and the identifying unit 205 will be described in detail below.

[0030] The image generating unit 202 generates an image for feature extraction using an input image. The image generating unit operates differently depending on whether or not it receives a generated image signal from a feature map generating unit 204 described later. When it receives a generated image signal, it generates an image based on the generated image signal. The generated image signal specifies, for example, the reduction ratio and image size of the image. The specified image may be an image having a specific frequency component using a low-pass filter, a high-pass filter, a band-pass filter, or the like, an image to which image processing such as correction, emphasis, and noise removal has been applied, or an image to which masking processing using a feature extraction map described later has been applied. Alternatively, an image obtained by combining these image processes may be used as a candidate. The types of candidate images are determined in advance.

[0031] Furthermore, when there is no input of a generated image signal, the image generating unit 202 sets an image with a predetermined initial setting. For example, an image is generated by reducing the input image with a predetermined reduction ratio or image size. The initial setting of the generated image may represent one of the above candidate images.

[0032] The feature extraction unit 203 extracts features from the generated image output by the image generation unit 202. The features to be extracted may be handcrafted features such as HOG or Haar-Like features, or may be features automatically obtained by machine learning such as deep learning or random forest. In addition, the features to be extracted or the parameters of the features may be changed based on the generated image signal.

[0033] When the feature extraction unit 203 receives a feature extraction map from the feature map generation unit 204 described below, it limits the area from which features are to be extracted and the type of feature based on the feature extraction map. The feature extraction map is an array corresponding to the type of feature and the area, and when a value in the feature extraction map is higher than a preset threshold, feature extraction is performed on the corresponding feature and area.

[0034] An example of a feature extraction map is shown in Fig. 3. Image 300 is an image to be subjected to feature extraction input from image generating unit 202, and feature extraction maps 301, 302, and 303 are feature extraction maps for feature amounts A, B, and C, respectively. White areas in feature extraction maps 301, 302, and 303 represent areas where the values ​​in the feature extraction map are equal to or greater than a preset threshold, and black represents areas below the threshold. Target object 304 is an object to be detected, and a bottle is shown here as an example. Target object 304 is assumed to be composed of the bottle body, cap, and label. Areas 305 and 306 are areas in the feature extraction map that have values ​​equal to or greater than the threshold.

[0035] For example, in feature extraction map 301, only region 305 is above the threshold, so feature A is extracted only from the region corresponding to region 305 in the generated image. That is, feature A is applied only to the region around the bottle, and not to the background or floor regions. In addition, in feature extraction map 302, only region 306 is above the threshold, so feature B is extracted only from the region corresponding to region 306 in the generated image. That is, feature B is applied only to the label region, and not to other regions. In addition, in feature extraction map 303, all regions are below the threshold, so feature extraction processing is not executed. This feature extraction map is determined based on the features extracted from the previous generated image, so the regions that are enabled change depending on the features.

[0036] Also, a feature extraction map common to all features may be specified. In other words, it is a map showing only the area for feature extraction. For example, when feature extraction map 301 in Fig. 3 is input as a feature extraction map common to all features, feature extraction for area 305 is performed, and feature extraction for other areas is not performed.

[0037] Also, the feature extraction map may not specify the area to be extracted, but may specify only the feature to be extracted. For example, the feature extraction map may have three arrays corresponding to feature amounts A, B, and C, and only feature amounts that are equal to or greater than a threshold value as a result of the feature map calculation may be extracted, and other feature amounts may not be extracted.

[0038] In this way, the feature extraction unit 203 reduces the feature extraction processing by limiting the area from which features are extracted and the type of features based on the feature extraction map, thereby achieving high-speed classification processing in the classification unit 205.

[0039] The feature map generation unit 204 generates a generated image signal that specifies the type of image to be generated by the image generation unit 202, and a feature extraction map that is used by the feature extraction unit 203 to limit the area from which features are to be extracted and the type of feature.

[0040] A method for generating a generated image signal will be described. The generated image signal is determined by the image generating unit 202 to generate the next image based on the feature amount output by the feature amount extraction unit 203. For example, the feature amount extraction unit 203 calculates the frequency component amount contained in the generated image as the feature amount, and determines the image to be generated based on the distribution of the frequency components. The determination may also be performed by machine learning or the like. For example, N candidates for the next image to be generated are prepared, the feature amount output by the feature amount extraction unit 203 is input, and a deep learning network with N output units is designed. A selector that selects the next generated image can be generated by learning to select a generated image that extracts a feature amount effective for identification based on the input feature amount.

[0041] Next, a method for generating a feature extraction map will be described. For example, there is a method for performing provisional classification using features extracted up to now, and generating a feature extraction map based on the certainty of classification. That is, new feature extraction is not performed for areas that have been classified with high certainty using only features extracted up to now, and new feature extraction is performed for areas with low certainty to improve the certainty. Also, machine learning may be used to estimate from which features or areas features effective for classification can be extracted. The method is the same as for the generated image signal described above, and the extent to which features extracted from candidate feature types or areas are effective for classification is estimated.

[0042] The feature map generating unit 204 outputs an extraction end signal to the classification unit 205, indicating the end of the feature extraction process. The end condition of the feature extraction process is determined in advance by the user, and is determined based on, for example, the certainty of the provisional classification, the number of images generated, the number of features extracted, the processing time, etc. The end condition based on the certainty may be, for example, when the certainty for the entire region is above a certain level in pixel-based classification, or when the average of the certainty is above a certain level. In image-based classification, the end condition may be when the certainty of the classification output for one image is above a certain level, or when the classification score is above a certain level. It is also possible to set upper limits for the number of generated images, the number of extracted features, the processing time, etc., and end the process when any one or all of the upper limits are reached. Alternatively, the end may be determined comprehensively based on the number of generated images, the number of extracted features, the processing time, and the certainty of the classification.

[0043] The classification unit 205 receives an extraction end signal output from the feature map generation unit 204, performs final classification using all or part of the features extracted by the feature extraction unit 203, and outputs the classification result.

[0044] FIG. 4 shows an example of pixel-by-pixel discrimination processing. FIG. 4 shows an example in which a plurality of reduced images are used as generated images. Generated images 400, 401, and 402 are a 1 / 4 reduced image, a 1 / 2 reduced image, and a non-reduced image, respectively. Feature amounts 410, 411, and 412 are feature amounts extracted by the feature amount extraction unit 203 from the generated images 400, 401, and 402, respectively. Original image unit feature amounts 420 and 421 are feature amounts obtained by enlarging the feature amounts 410 and 411 by 4 times and 2 times, respectively. Generated image signals 430 and 431 are generated image signals output by the feature amount map generation unit 204. Feature amount extraction maps 440 and 441 are feature amount extraction maps output by the feature amount map generation unit 204. A combined feature amount 450 is a feature amount obtained by combining the feature amount 412 and the original image unit feature amounts 420 and 421. Classification result 460 is an example of a pixel-by-pixel classification result obtained by inputting combined feature 450 to classification unit 205. Here, generated images 400, 401, 402 have dimensions of vertical direction y, horizontal direction x, and channel ch. Also, features 410, 411, 412, original image unit features 420, 421, feature extraction maps 440, 441, and combined feature 450 have dimensions of vertical direction y, horizontal direction x, and feature type ch. Also, classification result 460 has dimensions of vertical direction y, horizontal direction x, and class ch.

[0045] In the example of FIG. 4, first, the feature extraction unit 203 accepts the generated image 400 and outputs the feature 410. The generated image 400 is an image that the image generation unit 202 generates from the input image when there is no input of a generated image signal, and in the example of FIG. 4, a 1 / 4 reduced image is generated. Next, the feature map generation unit 204 accepts the feature 410 and outputs the generated image signal 430 and the feature extraction map 440. As described above, the generated image signal 430 is a signal that the feature map generation unit 204 outputs to the image generation unit 202 to determine the next image to be generated. In the example of FIG. 4, an image reduction ratio is selected as the generated image signal. The circles in the generated image signals 430 and 431 represent selectable image reduction ratios, and the black circles represent the image reduction ratios selected by the feature map generation unit 204. It is assumed that a reduction ratio of 1 / 2 is selected in the generated image signal 430, and the image generating unit 202 outputs the generated image 401 (1 / 2 reduced image) based on the generated image signal 430. Furthermore, a feature extraction map 440 indicates an area from which the feature extraction unit 203 should extract features from the generated image 401. In the example of FIG. 4, features are extracted only from the areas indicated in white in the feature extraction map 440. The feature extraction unit 203 receives the generated image 401 and the feature extraction map 440, and outputs a feature 411. The feature 411 is extracted only from the white areas of the feature extraction map 440, and is not extracted from other areas. A predetermined value, for example 0, is stored in the areas in the feature 411 that are not subject to feature extraction.

[0046] Similarly, feature amount map generation unit 204 receives feature amount 411, and outputs a generated image signal 431 and a feature amount extraction map 441. Image generation unit 202 outputs a generated image 402 (non-reduced image) based on the generated image signal 431. Furthermore, feature amount extraction map 441 indicates an area from which feature amount should be extracted from generated image 402 by feature amount extraction unit 203. Feature amount extraction unit 203 receives generated image 402 and feature amount extraction map 441, and outputs feature amount 412. Feature amount 412 is extracted only from the area indicated in white in feature amount extraction map 441, and is not extracted from other areas.

[0047] Since the features 410 and 411 are reduced by the same magnification as the generated image from which the features were extracted, they cannot be assigned as features to each pixel as they are. Therefore, the features 410 and 411 are converted into a form that can be assigned to each pixel of the original image, such as original image unit features 420 and 421. For example, in the example of FIG. 4, the features 410 and 411 are enlarged four times and two times. After that, by combining all the features as in the combined feature 450, a comprehensive feature corresponding to each pixel can be obtained. Finally, the classification unit 205 receives the combined feature 450 as an input and outputs the classification result 460. The classification result 460 indicates class information for each pixel.

[0048] The above-mentioned method of generating the original image unit features 420 and 421 is merely an example, and other methods may be used as long as each feature can be assigned to each pixel in the original image. For example, a method may be used in which the feature 410 is enlarged twice, the feature 410 is combined with the feature 411, a new feature is generated based on the combined feature, and then the new feature is further enlarged twice, thereby assigning the feature to the original image in stages.

[0049] In the case of image-based classification, the step of assigning features to pixels can be omitted and classification can be performed using all the obtained features, or classification can be performed using features that have been further thinned out, for example by taking the maximum value for each region.

[0050] The output unit 206 outputs the classification result output by the classification unit 205 to the outside of the device. The classification result array or numerical value may be output as is, or in the case of region division, an image in which each pixel is colored according to the classification result may be output. In addition, in the case of image-based classification, the classification result may be output in a form that is easy for the user to understand, such as a character string.

[0051] As a result, it is possible to provide an apparatus and method for detecting or identifying a target object in an image with high accuracy and high speed. EXAMPLES

[0052] The second embodiment is an image processing system that detects an object at high speed based on the region segmentation result or the classification result output by the image processing device described in the first embodiment and a detection-related designation signal that designates settings and information related to detection.

[0053] 12 shows a hardware configuration diagram according to the second embodiment. The image processing system 1200 according to the second embodiment includes an imaging device 1201, an image processing device 100, an object detection device 1202, a display device 1203, and a storage device 1204.

[0054] The image capturing device 1201 is a device for capturing an image of a target object, such as a camera. The image processing device 100 is the image processing device described in the first embodiment, and calculates area segmentation or classification results of a specific object area from an image captured by the imaging device 1201, or both. The object detection device 1202 outputs object detection information based on the area segmentation result or classification result output by the image processing device 100 and a detection target designation signal, which is a signal that designates the detection target, among the detection-related designation signals output by the memory device 1204. A display device 1203 displays the object detection result by the object detection device 1202 to the user.

[0055] The storage device 1204 holds a detection-related designation signal preset by a user. The detection-related designation signal is a signal for designating various settings and information related to detection. The detection-related designation signal in the second embodiment includes a detection target designation signal that designates a detection target in the object detection device 1202. The storage device 1204 outputs the detection target designation signal to the object detection device 1202. Although not specifically shown in FIG. 12, the image processing system 1200 may include a user interface device such as a keyboard, and may be configured so that the user can rewrite the detection-related designation signal recorded in the storage device 1204 during system operation.

[0056] <Operation of each device> The operations of the imaging device 1201, the display device 1203, and the storage device 1204 are as described above. The operations of the image processing device 100 are as described in the first embodiment. Therefore, only the object detection device 1202 will be described here.

[0057] A case where the image processing device 100 outputs the region segmentation result will be described. Fig. 7 shows an example of an image captured by the imaging device 1201 and the region segmentation result output by the image processing device 100. Image 701 is an example of an image of a plurality of medicine bottles, and region segmentation result 702 is an example of the result of region segmentation of image 701 by the image processing device 100. Bottles 703, 704, and 705 are bottles of medicines A, B, and C, respectively, and bottle regions 706, 707, and 708 are bottle regions of medicines A, B, and C obtained by region segmentation. Bottle regions 706, 707, and 708 are each segmented into different classes, and differences in texture represent differences in class.

[0058] The region segmentation result 702 is useful for the user to visually confirm the situation. However, if the user requires information on an object-by-object basis (such as whether or not a particular type of bottle is present, how many bottles are present in the image, and in what order the bottles are arranged), a different process is required. Therefore, the object detection device 1202 receives a detection target designation signal from the storage device 1204, extracts object information based on the detection target designation signal, and outputs it as the detection result.

[0059] The object detection device 1202 can output the presence or absence, number, coordinate information, and location information of objects belonging to each class, or information determined by a combination of these, as the detection result, and which information is to be output is specified by the detection target specification signal. The method of extracting each piece of information will be explained below.

[0060] First, a method for the object detection device 1202 to detect the presence or absence of an object belonging to each class will be described. The presence or absence of an object belonging to each class can be detected by extracting a combined region in the region segmentation result and determining whether or not there is a combined region belonging to each class. At this time, since there is a possibility that the region segmentation result includes an erroneous identification region, the combined region may be extracted after excluding the erroneous identification region based on the state of the combined region. As an example of a method for excluding an erroneous identification region, there is a method in which a threshold value for the size of the combined region is set in advance, and if the number of pixels in the combined region is below the threshold, the combined region is determined to be an erroneous identification region and is excluded. This is an exclusion method that utilizes the fact that the size of the combined region of the erroneous identification region tends to be smaller than the size of the combined region of the correct answer region. In addition, the combined region may be excluded by using a noise removal process such as a geometric transformation of the image (e.g., shrinkage / expansion process) or a median filter, or by a method of performing threshold processing on a quantitative value calculated based on the certainty of the identification result in each pixel or the combined region.

[0061] Furthermore, the object detection device 1202 can calculate the number of objects belonging to each class by extracting connected regions belonging to each class in the region segmentation result using the above-mentioned technique and counting the number of connected regions for each class.

[0062] In addition, the object detection device 1202 can extract the combined area of ​​each class using the above-mentioned technique, calculate the representative point of the combined area (e.g., the upper end, lower end, left end, right end, center of gravity, etc.), and output the coordinate position of the representative point as the detection result.

[0063] Furthermore, the object detection device 1202 can output the position of each combined area in the area division result as a detection result. For example, the coordinate information of the representative point is extracted from each combined area in the area division result by the above-mentioned method, and is recorded in association with the class information of each combined area. After that, by sorting based on the horizontal (or vertical) coordinate information and extracting the class information from the sorting result, it is possible to obtain the horizontal (or vertical) arrangement order of the objects in the image as the arrangement information. Alternatively, the image may be divided into a grid, and the presence or absence of objects of each class in the grid may be determined based on the occupied area of ​​the combined area in each grid, thereby obtaining grid-shaped object arrangement information.

[0064] Furthermore, the overall state of one or more objects may be detected by combining the presence / absence, number, coordinate information, and location information of objects belonging to each class described above. As a result, it is possible to provide an image processing system that quickly detects a target object within an image. EXAMPLES

[0065] The third embodiment is an image processing system that uses the image processing system described in the second embodiment to determine whether or not the arrangement of target objects is in a predetermined positional relationship. 6 shows a hardware configuration diagram according to the embodiment 3. The image processing system 600 according to the embodiment 2 includes an imaging device 1201, an image processing device 100, an object detection device 1202, an object placement verification device 602, a display device 1203, and a storage device 1204.

[0066] The imaging device 1201, image processing device 100, object detection device 1202, and display device 1203 are similar to those described in the second embodiment, and therefore their explanation will be omitted. As described in the second embodiment, the object detection device 1202 receives a detection target designation signal that designates a detection target from the storage device 1204. In this embodiment, the detection target designation signal designates the arrangement information of each combined region in the region segmentation result. The object detection device 1202 extracts the object arrangement information by the method described in the second embodiment.

[0067] <Operation of each device> The operation of the storage device 1204 will now be described. In this embodiment, the detection-related designation signals stored in the storage device 1204 include two signals: a detection target designation signal that designates a detection target, and a normal placement state designation signal that indicates the correct placement state of a target object. The storage device 1204 outputs the detection target designation signal to the object detection device 1202, and outputs a normal placement state designation signal to the object placement verification device 602. It is assumed that the detection target designation signal and the normal placement state designation signal are set in advance by the user.

[0068] The operation of the object placement verification device 602 will now be described. The object placement verification device 602 receives the object placement information output by the object detection device 1202 and the correct placement state designation signal output by the recording device 1204, and verifies whether the object in the image captured by the imaging device 1201 is correctly placed based on whether the two match.

[0069] An example of bottle placement verification will be described with reference to FIG. 7. Here, an example of determining the horizontal placement order of bottles will be described. First, the types of medicines in a correct placement state are registered in order from the left as a correct placement state designation signal (for example, "ABC"). Using the method described in the second embodiment, the object detection device 1202 acquires horizontal placement information of each object region from the region segmentation result 702. As a result, in the example of the region segmentation result 702, the label placement information is calculated as "ABC", and by comparing this with the correct bottle placement registered by the user, it can be determined that all bottles are correctly placed.

[0070] The object placement verification device 602 creates an image for presenting to the user the result of determining whether the placement is correct or incorrect. Figures 8A, 8B, and 8C show examples of the display of object placement verification results. Window 801 is the result display window presented to the user, overall verification result 802 represents the overall placement verification result, and individual verification results 803, 804, and 805 represent the placement verification results of each target object. In the example of Figure 8, overall verification result 802 displays "OK" if all target objects are correctly placed, and displays "Error" if not.

[0071] In the example of FIG. 8A, since each bottle is correctly positioned, the individual verification results 803, 804, 805 and the overall verification result 802 are all displayed as "OK." In the example of FIG. 8B, the placement of the second and third bottles from the left is incorrect, so the individual verification results 804 and 805 are displayed as "Error," and the overall verification result 802 is also displayed as "Error." The example in FIG. 8C is an example in which the bottle on the far right was not detected correctly because it was different from the expected bottle. The arrangement of the bottles on the left and in the middle is correct, so individual verification results 803 and 804 are displayed as "OK". However, the calculation result of the arrangement information is "AB", which does not match the correct registered bottle arrangement of "ABC". Therefore, overall verification result 802 is displayed as "Error". Also, instead of displaying "Error", it is also possible to display a message indicating that the number of target objects that should have been detected was not detected, or to display the number of target objects that were detected.

[0072] In addition, in Figs. 8A, 8B, and 8C, the correctness of the placement of each object is indicated by the display of "OK" and "Error", but for example, the type of the detected target object may also be displayed. The correctness of the placement of the area and the type of the target object may be displayed not only by characters but also by the color of the detection frame or area, the type of line, an icon, etc. Any display may be used as long as it can convey the placement information of the target object to the user. In addition, although the above description was given as an example of placement from the left, it is also possible to determine the top, bottom, left, right, nested structure, etc. if the relative placement of the target object can be defined. In addition, it is not necessary to display both the overall verification result and the individual verification result, and either one may be displayed as necessary.

[0073] The object placement verification device 602 may also determine the individual verification result for each bottle by using the confidence level of the region division result output by the image processing device 100. The confidence level is a value that indicates how confident the image processing device 100 is in dividing each pixel into regions. For example, when obtaining identification scores for the background, bottle A, bottle B, and bottle C for each pixel by region division, there is a method of using the maximum value of the identification scores as the confidence level. In this case, it is desirable to normalize the range of each identification score from 0.0 to 1.0 and the sum of all classes of identification scores to 1.0. For example, the object placement verification device 602 may determine that a bottle region that has not been detected with sufficient confidence is an insufficient confidence level or a false detection, and display it in the individual verification result. Also, the confidence level may be presented to the user by superimposing it on the object placement verification result screen.

[0074] FIG. 11 shows an example of object placement verification using certainty. A result display window 1101 is a result display window presented to a user, a region division result 1102 is a region division result by the image processing device 100, and a certainty superimposition result 1113 is an example in which the certainty for each pixel of the region division result output by the image processing device 100 is superimposed on the result display window 1101. The result display window 1101 includes a bottle A 1103, a bottle B 1104, and a bottle C 1105, and displays individual verification results 1106, 1107, and 1108 of each bottle and a comprehensive verification result 1109. Here, it is assumed that the bottle C 1105 does not have a label with a medicine name written on it, and the medicine name cannot be seen. The region division result 1102 shows a region division result A 1110, a region division result B 1111, and a region division result C 1112 as region division results for the bottle A 1103, the bottle B 1104, and the bottle C 1105. It is assumed that the bottle A1103, the bottle B1104, and the bottle C1105 are correctly identified as their types. The confidence superimposition result 1113 displays the confidence A1113, the confidence B1114, and the confidence C1115 as the confidences of the region division result A1110, the region division result B1111, and the region division result C1112 in a form of being superimposed on the result display window 1101. In the confidence superimposition result 1113, the confidence for each pixel is displayed as a brightness value, and the closer to white, the higher the confidence, and the closer to black, the lower the confidence. As described in the first embodiment, the image processing device 100 extracts features from multiple generated images based on the feature extraction map, so that this feature extraction method is reflected in the confidence. Specifically, for example, the confidence superimposition result 1113 in FIG. 11 shows an example of feature extraction from an image reduced by multiple magnifications, and the resolution of the confidence varies locally. For example, the character regions on the label and the outline of the bottle are extracted with high resolution, so high-resolution confidence levels are obtained, but the resolution of the confidence level decreases as you move away from the label or bottle outline.

[0075] As described above, the image processing device 100 correctly identifies the bottle types of the bottle A1103, the bottle B1104, and the bottle C1105, but there is a difference in the certainty. Here, as an example of a method for determining an individual judgment result based on the certainty, a method in which a threshold for the certainty is set in advance and an individual judgment result is determined based on whether the maximum certainty in each combined region is equal to or greater than the threshold will be described. The certainty A1113 and the certainty B1114 for the bottle A1103 and the bottle B1104, whose labels are visible, are obtained with a certainty exceeding the threshold, especially around the labels, while the certainty C1115 for the bottle C1105, whose label is not visible, is not obtained with a certainty equal to or greater than the threshold. In this case, it is determined that the certainty of the bottle A1103 and the bottle B1104 is sufficient, and the bottle type obtained by the region division is adopted as the detection result as it is. Since the bottle type and arrangement of bottle A 1103 and bottle B 1104 are correct, "OK" is displayed as individual verification results 1106, 1107. On the other hand, bottle C 1105 could not be segmented with a confidence level equal to or higher than the threshold, so "Low Conf" indicating insufficient confidence is displayed as individual verification result 1108. Furthermore, since bottle C 1105 could not be detected correctly, "Error" is displayed as overall verification result 1109. Note that the content displayed when it is determined that the confidence level is insufficient is just an example, and it would also be possible to display, for example, "Error" indicating incorrect arrangement.

[0076] By verifying the placement only when the confidence level is high as described above, it is possible to increase the reliability of the object placement verification result and further reduce the risk of misplacement of the bottle. In addition, by presenting the confidence level to the user by superimposing it on the object placement verification result screen or the input image, the user can obtain information on which area the image processing device 100 used to identify the bottle type, or on the lack of features for identification.

[0077] As a result, it is possible to provide an image processing system that determines whether or not a target object in an image is in a predetermined positional relationship. EXAMPLES

[0078] Example 4 is an image processing system for automatic blood analysis, including a function for quickly and accurately determining the type of container storing a sample to be analyzed using the image processing system described in Example 2. The blood analysis includes biochemical analysis and immunological analysis. By automatically determining the type of container inputted to the image processing system, it becomes possible to apply a specimen state determination function according to the container type (for example, detection of the presence or absence of air bubbles or foreign objects that affect the analysis, etc.), thereby improving the reliability of the blood analysis.

[0079] 9 shows a hardware configuration diagram according to the fourth embodiment. The image processing system 900 according to the fourth embodiment includes an imaging device 1201, an image processing device 100, an object detection device 1202, a specimen state determination device 902, a sample collection device 903, an analysis device 904, a display device 1203, and a storage device 1204. The imaging device 1201 and the display device 1203 are the same as those described in the second embodiment, and therefore the description will be omitted. However, the imaging device 1201 takes an image of a container storing a sample (hereinafter, a specimen) from the side. The image processing device 100 performs area division to identify the container type, and the object detection device 1202 determines and outputs the container type based on the area division result. Details will be described later.

[0080] <Operation of each device> The operations of the image processing device 100, the object detection device 1202, the specimen state determination device 902, the sample collection device 903, and the analysis device 904 will be described in detail below.

[0081] The operation of the storage device 1204 will be described. In this embodiment, the detection-related designation signals stored in the storage device 1204 include two: a detection target designation signal that designates a detection target, and a specimen and / or container state determination target designation signal that designates a target for state determination in the specimen state determination device 902. The storage device 1204 outputs a detection target designation signal to the object detection device 1202, and outputs a specimen and / or container state determination target designation signal to the specimen state determination device 902. The detection target designation signal designates container type information that is determined based on the presence or absence of an object belonging to each class and its coordinate position. The specimen and / or container state determination target designation signal designates the state of the specimen and / or container that the user wants to determine, such as the presence or absence of air bubbles or foreign matter present in the sample, the inclination of the container, etc. It is assumed that the detection target designation signal and the specimen state and / or container state determination target designation signal are set in advance by the user.

[0082] The operation of the image processing device 100 and the object detection device 1202 according to the fourth embodiment will be described. An example of an input image to the image processing device 100 according to the fourth embodiment is shown in Figs. 10A and 10B. The input image 1000 in Figs. 10A and 10B shows an example of capturing an image of a state in which a cup A 1003 and a cup B 1004 are placed on raised-bottom containers 1001 and 1002. In addition, a sample 1005 is stored in the cup A 1003 and the cup B 1004. The raised-bottom containers 1001 and 1002 and the cup A 1003 and the cup B 1004 are separate and may be combined in any way. Here, an example of determining the container type based on a combination of the type of cup and the height of the raised-bottom container will be described.

[0083] In order to identify the type of cup, it is necessary to extract features from the periphery of cup A1003 and cup B1004. In addition, in order to identify the height of a raised-bottom container, it is necessary to detect the positions of the top and bottom ends of the cup. To detect the position of the cup with high accuracy, it is necessary to input a high-resolution image, but when extracting features from the entire image, the length of processing time becomes an issue. On the other hand, by using the image processing device 100 described in the first embodiment, for example, a feature extraction map is generated from a reduced image, and features required for cup type identification and cup position detection are extracted only from the periphery of cup A1003 and cup B1004, so that it is possible to increase the speed.

[0084] Here, an example will be described in which an input image is segmented into regions based on the type of cup using the image processing device 100. Figures 13A and 13B show examples of region segmentation results when the input images shown in Figures 10A and 10B are used. Region segmentation result 1300 includes cup region A 1303 and cup region B 1304, and each type of cup is expressed as a different texture.

[0085] The region segmentation results by the image processing device 100 are input to the object detection device 1202. In addition, the object detection device 1202 receives a detection target designation signal from the storage device 1204. As described above, the detection target designation signal designates information on the type of container, which is determined based on the presence or absence of an object belonging to each class (i.e., the type of cup present in the image and its presence or absence) and its coordinate position.

[0086] First, the object detection device 1202 extracts a combined region from the region segmentation result 1300 by the method described in the second embodiment, and identifies the type of cup in the image. Then, the object detection device 1202 estimates the length of the raised container from the position of the extracted combined region. The length L of the raised container in pixels is estimated by the following formula: L=d2-l+d1 Here, d2 is the vertical coordinate position of the bottom end of the raised container, 1 is the vertical coordinate position of the bottom end of the cup region estimated by the image processing device 100, and d1 is the distance from the bottom end of the cup to the attachment point with the raised container. FIG. 15 illustrates the relationship of each parameter in the example of FIG. 10A. The distance d1 from the bottom end of the cup to the attachment point with the raised container is measured in advance for each type of container and recorded in the storage device 1204, and is read from the storage device 1204 according to the type of cup identified by the object detection device 1202. The vertical coordinate position d2 of the bottom end of the raised container is assumed to be the same position for all raised containers, and is measured in advance and recorded in the storage device 1204. However, if the variation of the bottom end of the raised container is large, the image processing device 100 may estimate the position of the bottom end of the raised container separately from the cup region.

[0087] Next, the type of container is determined from the identified type of cup and the length of the raised container. For example, a correspondence table 1400 of cup types and lengths of raised containers as shown in FIG. 14 is stored in advance in the storage device 1204. The horizontal axis of the correspondence table 1400 indicates the type of cup, and the vertical axis indicates the height of the raised container. Using the correspondence table 1400, it is possible to automatically determine the type of container from the identified type of cup and length of the raised container. Note that, with regard to the length of the raised container, an element of the candidates in the correspondence table 1400 whose value is closest to the estimated value obtained by the above formula is selected.

[0088] The specimen status determination device 902 is a device that determines the status of a sample and / or a container, and uses any existing technology to determine the presence or absence of air bubbles in a sample, the tilt of a container, etc. In this case, by switching the determination algorithm based on the container type output by the object detection device 1202, it is possible to determine the specimen status with higher accuracy.

[0089] The sample collecting device 903 controls the sample collecting process based on the sample state determination result output by the sample state determining device 902. The analysis device 904 performs blood analysis by mixing the sample obtained by the sample collection device 903 with chemicals. The output device 905 is a device for displaying the blood analysis results, and is a monitor, printer, or the like.

[0090] In this manner, the image processing device 100 and the object detection device 1202 enable the type of sample container to be identified quickly and with high accuracy.

[0091] The sample collecting device 903 controls the sample collection based on the sample state determined by the sample state determining device 902. For example, if the sample contains air bubbles that may affect the analysis, the sample is not collected and the user is notified of the reason for non-collection, or the sample is sent to a different sample transport route.

[0092] As a result, by quickly and accurately determining the container type, the sample condition can be determined with high accuracy, thereby improving the reliability of blood analysis.

[0093] <Modification> When calculating a generated image signal in the feature map generation unit 204 of the first embodiment, the generated image signal may be calculated so as to generate multiple images according to the score of the generated image signal. In this case, the image generation unit 202 that receives the generated image signal generates multiple images based on the generated image signal and outputs them to the feature extraction unit 203. At this time, the feature map generation unit 204 generates a feature extraction map for each generated image. The feature extraction unit 203 uses the group of generated images received from the image generation unit 202 and the group of feature extraction maps received from the feature map generation unit 204 to extract features from each pair (combination of a generated image and a feature extraction map).

[0094] The type of feature selected by the feature extraction unit 203 in the first embodiment may be not only a difference in the nature of the feature, such as a difference in the weighting coefficient of a feature extraction filter, but also a parameter related to the application method, such as the stride or dilate in the convolution process.

[0095] The method of displaying the certainty in the third embodiment may be, for example, a method in which a certainty display button is arranged on the object placement verification result screen, and the user switches between ON / OFF of the certainty superimposition, or between the original image and the certainty. In the second embodiment, the certainty is expressed as a luminance value, but it may be displayed as a heat map, contour lines, or color-coded for each class. Only pixels having a certain degree of certainty or higher may be superimposed. If there is a bottle determined to have an insufficient certainty, the bottle or the state of the photographing tends to be abnormal, so a message may be displayed to encourage the user to review the bottle or the state of the photographing according to the certainty.

[0096] The method of displaying each individual judgment result in the third embodiment may display not only the final judged bottle type but also the confidence level for multiple bottle types. The confidence level to be displayed may be, for example, the maximum, average, or sum of the confidence levels in the combined region, and may be combined with other image processing such as threshold processing or normalization processing.

[0097] In the image processing system described in Example 4, a method of determining the type of container using the image processing system described in Example 2 has been described, but in addition to the type of container, the container condition, such as the presence or absence of a lid or label, may also be determined.

[0098] As described above, the image processing device for identifying an object in an image disclosed in Example 1 includes an image input unit 201 that accepts an input image, an image generation unit 202 that generates an image for feature extraction from the input image, a feature extraction unit 203 that extracts features from a generated image generated by the image generation unit, a classification unit 205 that identifies an object in an image using the features output by the feature extraction unit, an output unit 206 that outputs the classification result output by the classification unit, and a feature map generation unit 204 that instructs the image generation unit to generate a new generated image as necessary based on the features output by the feature extraction unit 203, and generates a feature map indicating feature extraction conditions for the new generated image and outputs it to the feature extraction unit 203.

[0099] Furthermore, the image generation unit 202 reduces the input image to generate an initial generated image, and if the feature map generation unit 204 determines that the features already extracted are insufficient for the classification, it instructs the generation of a new generated image with a lower reduction ratio. In this way, by starting processing with a generated image with a high reduction ratio and using a generated image with a lower reduction ratio as necessary, it is possible to identify target objects in the image with high accuracy and high speed. Furthermore, the feature amount map generating unit 204 identifies an area where the already extracted feature amounts are insufficient for the discrimination, and generates a feature amount map that specifies the area as the range for feature amount extraction. In this way, by excluding areas where sufficient features for identification have been obtained and limiting the range of feature extraction to perform feature extraction again, it is possible to identify target objects in an image with high accuracy and speed.

[0100] Furthermore, the feature map generating unit 204 determines the generating conditions for a new generated image and the type of feature to be extracted from the new generated image based on the extraction results of the existing feature. Therefore, the type of generated image, the reduction ratio, and the type of feature to be extracted can be appropriately changed, and the target object in the image can be identified with high accuracy and high speed.

[0101] Furthermore, when a predetermined condition is met, the feature map generation unit 204 outputs an extraction end signal indicating the end of the feature extraction process to the classification unit 205, and when the classification unit 205 receives the extraction end signal, it identifies objects in the image by comprehensively using the features related to the one or more generated images that have been generated up to that point. This makes it possible to eliminate unnecessary processing load, generate necessary features efficiently, and identify target objects in images with high accuracy and speed.

[0102] Furthermore, the feature amount map generating unit 204 executes a tentative classification using some or all of the features that have been extracted by the feature amount extracting unit 203, and generates a feature amount extraction map based on the degree of certainty of the classification. Therefore, the necessity for extracting new features can be determined with high accuracy.

[0103] In addition, the feature extraction map output by the feature map generating unit 204 can also be output to the outside via the output unit. By outputting the feature extraction map externally, it is possible to provide information about the type of recognition processing that is being performed.

[0104] Moreover, the image processing system disclosed in Example 2 for quickly detecting an object in an image includes an imaging device 1201 for imaging a target object to obtain the input image, the image processing device 100 disclosed in Example 1, an object detection device 1202 for detecting a target object based on the identification result output by the image processing device 100 and detection target information output from a memory device 1204, a memory device 1204 for outputting detection target information specifying the object to be detected by the object detection device 1202, and a display device 1203 for presenting the object detection result output by the object detection device 1202 to a user. According to the configuration shown in the second embodiment, it is possible to quickly detect a detection target object in an image. In addition, in the image processing system disclosed in the third embodiment, it is possible to use the object identification results and feature extraction map to display information suggesting the possibility that "the object position has been swapped" or "the label orientation is incorrect."

[0105] The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment is described in detail to easily explain the present invention, and is not necessarily limited to the embodiment having all the configurations described. In addition, each component may be replaced or added, not limited to the deletion of each component. [Explanation of symbols]

[0106] 100: image processing device, 110: interface unit, 111: calculation unit, 112: memory, 113: bus, 201: image input unit, 202: image generation unit, 203: feature extraction unit, 204: feature map generation unit, 205: classification unit, 206: output unit, 500: image processing method, 501: image input step, 502: image generation step, 503: feature extraction step, 504: feature map generation step, 505: classification step, 506: output step, 1200: image processing system, 1201: imaging device, 1202: object detection device, 1203: display device, 1204: storage device, 1400: correspondence table

Claims

1. 1. An image processing device for identifying an object in an image, comprising: an image input unit that accepts an input image; an image generation unit that generates an image for extracting features from the input image; a feature extraction unit that extracts features from the generated image generated by the image generation unit; a classification unit that classifies an object in an image by using the feature amount output by the feature amount extraction unit; an output unit that outputs the classification result output by the classification unit; a feature amount map generating unit that instructs the image generating unit to generate a new generated image as necessary based on the feature amount output by the feature amount extracting unit, generates a feature amount map that specifies an extraction area of ​​the feature amount for the new generated image and a type of the feature amount to be extracted for each area, and outputs the feature amount map to the feature amount extracting unit; An image processing device comprising:

2. 2. The image processing device according to claim 1, The image generation unit reduces the input image to generate an initial generated image; The image processing device according to claim 1, wherein the feature map generating unit instructs generation of a new generated image with a lower reduction ratio when it determines that the already extracted features are insufficient for the classification.

3. 2. The image processing device according to claim 1, The image processing device is characterized in that the feature map generation unit identifies an area where already extracted features are insufficient for the classification, and generates a feature map that specifies the area as a range for feature extraction.

4. 2. The image processing device according to claim 1, The image processing apparatus is characterized in that the feature map generation unit determines a generation condition for a new generated image and a type of feature to be extracted from the new generated image based on an extraction result of an existing feature.

5. 2. The image processing device according to claim 1, the feature map generating unit outputs an extraction end signal indicating an end of the feature extraction process to the identifying unit when a predetermined condition is satisfied; When the identification unit receives the extraction end signal, the identification unit identifies an object in the image by comprehensively using feature amounts related to one or more generated images that have been generated up to that point.

13. An image processing device comprising:

6. 2. The image processing device according to claim 1, The image processing device is characterized in that the feature map generation unit performs a tentative classification using some or all of the features that have been extracted by the feature extraction unit, and generates a feature extraction map based on the certainty of the classification.

7. 2. The image processing device according to claim 1, The image processing device further comprises: an output unit that outputs a feature extraction map output by the feature map generating unit to an outside of the image processing device;

8. An image processing system for quickly detecting an object in an image, comprising: an imaging device for capturing an image of a target object to obtain the input image; an image processing device according to claim 1 , which outputs a region segmentation result or a classification result based on the input image output by the imaging device; a storage device that outputs a detection target designation signal for designating a detection target as a detection related designation signal which is a signal for designating settings and information related to detection; an object detection device that detects an object based on the area division result or the classification result output by the image processing device and the detection target designation signal output by the storage device; a display device that presents to a user the object detection result output by the object detection device; An image processing system comprising:

9. 9. The image processing system according to claim 8, The image processing device outputs a region segmentation result based on the input image, the storage device outputs, as detection-related designation signals, a detection target designation signal for detecting the arrangement information of a plurality of objects present in the input image, and a normal arrangement state designation signal for the target object, which is set in advance by a user; the object detection device receives the region division result and the detection target designation signal, and detects arrangement information of a plurality of objects present in the image; An object placement verification device that verifies whether the placement of the target object is correct based on the object placement information output by the object detection device and the correct placement state designation signal output by the storage device. Further equipped with The image processing system according to claim 1, wherein the display device presents to a user an object placement verification result output by the object placement verification device.

10. 9. The image processing system according to claim 8, the imaging device images a container storing a sample to obtain the input image; The image processing device outputs a region segmentation result or a classification result based on the input image, the storage device outputs, as detection-related designation signals, a detection target designation signal for detecting the type of container present in the input image, and a sample state determination target designation signal and / or a container state determination target designation signal for determining the state of the sample and / or the container; the object detection device determines a type of container based on the region division result or the identification result and the detection target designation signal; a specimen status determination device that determines a status of the sample and / or the container based on the type of the container output by the object detection device and the sample status determination target designation signal and / or the container status determination target designation signal output by the storage device; a sample collecting device that controls the collection of the sample based on the state of the sample and / or the container determined by the specimen state determination device; an analyzer for performing a blood analysis by mixing the sample collected by the sample collecting device with a reagent; Further equipped with The image processing system is characterized in that the display device presents a blood analysis result obtained by the analysis device to a user.

11. 1. A method of image processing for identifying an object in an image, comprising: an image input step of accepting an input image; an image generating step of generating an image for extracting features from the input image; a feature extraction step of extracting a feature from the generated image generated by the image generation step; a classification step of detecting or classifying an object in an image by using the feature amount calculated in the feature amount extraction step; an output step of outputting the classification result calculated in the classification step, an image processing method comprising: causing the image generation step to generate a new generated image as necessary based on the features output by the feature extraction step; and generating a feature map generation step that generates a feature extraction area for the new generated image in the feature extraction step and a feature map that specifies, for each area, the type of feature to be extracted.

12. 12. The image processing method according to claim 11, The image generating step includes reducing the input image to generate an initial generated image; The image processing method is characterized in that the feature map generation step instructs the generation of a new generated image with a lower reduction ratio when it is determined that already extracted features are insufficient for the classification.

13. 12. The image processing method according to claim 11, The image processing method is characterized in that the feature map generating step identifies areas where already extracted features are insufficient for the classification, and generates a feature map that specifies those areas as a range for feature extraction.

14. 12. The image processing method according to claim 11, The image processing method is characterized in that the feature map generation step performs a tentative classification using some or all of the features extracted in the feature extraction step up to that point, and generates a feature extraction map based on the certainty of the classification.

15. 12. The image processing method according to claim 11, The image processing method according to claim 1, further comprising the step of: outputting, to an outside, the feature extraction map outputted in the feature map generating step.

Images