Image processing device, image processing method, and image processing system

Abstract

To provide technology that, when detecting a specific object from an image, performs detection without separation into multiple segments, even when occlusion due to an obstacle divides the object.SOLUTION: An image processing device according to the present invention comprises: an input unit that receives an input image; a feature amount extraction unit that extracts a feature amount from the input image; an object region detection unit that detects an object region in the image by using the feature amount; an object edge point estimation unit that estimates the coordinate of an edge point of the object in the object region; an object crosslink unit that calculates a trajectory for complementing on the input image between the edge points in two object regions; a same object determination unit that determines from a luminance value transition on the trajectory whether the two object regions are included in the same object; an object region correction unit that corrects the object region outputted by the object region detection unit on the basis of the determination result outputted by the same object determination unit; and an output unit that outputs the corrected object region.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing technique for detecting a specific object in an image. [Background technology]

[0002] In recent years, detection of specific objects is often performed in the analysis of images captured by cameras, etc. In object detection, there is a technology described in, for example, JP 2021-056899 (Patent Document 1) to prevent a decrease in detection accuracy due to obstruction of the object by an obstacle. This publication states that "based on the position of a moving object area detected from an image and a predetermined area in the image, a corrected image is generated in which the color of at least a part of the predetermined area is corrected based on the color of the moving object area." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-056899 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, among the small areas obtained by dividing a predetermined area set as a region where occlusion may occur, only the parts adjacent to the detection target object are corrected. However, if the occlusion is widespread, there will be small occlusion areas that are not adjacent to the target object. Furthermore, if the object is divided by occlusion, the division of the object will not be eliminated even in the corrected image, and the object may be detected as being divided into multiple pieces.

[0005] Therefore, an object of the present invention is to provide a technique for alleviating the problem that an object divided by occlusion is detected as multiple separate pieces. [Means for solving the problem]

[0006] In order to solve the above problem, one of the representative image processing devices, methods, and systems of the present invention analyzes the change in brightness values ​​between two detected object regions, and if it is determined that both objects are a single object separated by an obstacle, it merges the two object regions and corrects the detection result of the object.

[0007] Another aspect of the present invention is an image processing device that detects a specific object in an image, including an input unit that accepts an input image, a feature extraction unit that extracts features from the input image, an object region detection unit that detects an object region in the image using the features, an object endpoint estimation unit that estimates coordinates of an object's endpoints in the object region, an object bridging unit that calculates a trajectory that complements the endpoints of two object regions on the input image, an identical object determination unit that determines whether the two object regions are included in the same object based on the transition of brightness values ​​on the trajectory, an object region correction unit that corrects the object region output by the object region detection unit based on the determination result output by the identical object determination unit, and an output unit that outputs the corrected object region.

[0008] Another aspect of the present invention is an image processing method in which an apparatus detects a specific object in an image, the apparatus detecting an object region from an input image, the apparatus extracting a brightness value transition between two detected object regions, the apparatus determining whether the two object regions contain the same object from the brightness value transition, and the apparatus modifying the object region in accordance with a result of the determination. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to provide an apparatus, method, and system that can detect an object that is fragmented due to occlusion by an obstacle in an image as a single object without splitting it into multiple objects. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a first embodiment; [Diagram 3] 1 is a flowchart illustrating an example of a processing flow of an image processing method according to a first embodiment. [Figure 4A] FIG. 1 is a diagram for explaining a process for bridging object regions according to the first embodiment; [Figure 4B] FIG. 1 is a diagram for explaining a process for bridging object regions according to the first embodiment; [Figure 4C] FIG. 1 is a diagram for explaining a process for bridging object regions according to the first embodiment; [Figure 4D] FIG. 1 is a diagram for explaining a process for bridging object regions according to the first embodiment; [Diagram 5] FIG. 1 is a diagram for explaining a process for extracting luminance values ​​between object regions according to the first embodiment; [Figure 6] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a second embodiment. [Figure 7] 11 is a flowchart illustrating an example of a processing flow of an image processing method according to a second embodiment. [Figure 8A] FIG. 11 is a diagram for explaining a process for selecting a combination of object regions according to the second embodiment. [Figure 8B] FIG. 11 is a diagram for explaining a process for selecting a combination of object regions according to the second embodiment. [Figure 8C] FIG. 11 is a diagram for explaining a process for selecting a combination of object regions according to the second embodiment. [Figure 8D] FIG. 11 is a diagram for explaining a process for selecting a combination of object regions according to the second embodiment. [Figure 9] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a third embodiment. [Figure 10] 11 is a flowchart illustrating an example of a processing flow of an image processing method according to a third embodiment. [Figure 11] FIG. 13 is a diagram illustrating an example of a hardware configuration of an image processing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment will be described with reference to the attached drawings. In one embodiment of the present specification, the transition of the luminance value between two detected object regions is analyzed, and if it is determined that the two objects are a single object separated by an obstacle, the two object regions are integrated and the detection result of the object is corrected. In this way, even if an object is separated by occlusion due to an obstacle on the image, it can be detected as a single object. EXAMPLES

[0012] The first embodiment provides an image processing device and method for detecting an area of ​​a specific object in an image, and further determining whether or not the target object is divided due to occlusion by an obstacle by referring to the transition of brightness values ​​between the areas, and correcting the detected object area.

[0013] More specifically, when an object region is detected as being divided into multiple regions due to occlusion by an obstacle on an image, a line connecting the divided object regions is calculated, and the presence or absence of division due to occlusion is determined from the brightness value transition on the line. This makes it possible to integrate object regions originating from the same object, and to suppress overdetection in object detection. It is possible to detect various types of object, including, for example, rod-shaped bacilli, nanorods (nanoscale and rod-shaped metal or semiconductor materials), human bodies, etc. <Hardware configuration of image processing device>

[0014] 1 is a diagram illustrating an example of a hardware configuration of an image processing device in Example 1. The image processing device 1 includes an interface device 10, a calculation device 11, a memory 12, and a bus 13. The interface device 10, the calculation device 11, and the memory 12 transmit and receive information via the bus 13.

[0015] We will now explain each part of the image processing device 1. The interface device 10 is a communication device that transmits and receives signals to and from devices external to the image processing device 1. Devices that communicate with the interface device 10 include an imaging device 20 such as a camera or a microscope, and a display device 21 such as a monitor or a printer.

[0016] The arithmetic device 11 is a device that executes various processes within the image processing device 1, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), etc. The functions executed by the arithmetic device 11 will be described later with reference to FIG.

[0017] The memory 12 is a device that stores the programs executed by the arithmetic unit 11, the networks used in the processing and their weights, the processing results, etc., and is, for example, a hard disk drive (HDD) or a solid state drive (SSD). <Functional configuration of image processing device>

[0018] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing apparatus according to the first embodiment.

[0019] The image processing device 1 includes an input unit 100, a feature extraction unit 101, an object region detection unit 102, an object edge estimation unit 103, an object bridge unit 104, a bridge luminance value extraction unit 105, a same object determination unit 106, an object region correction unit 107, and an output unit 108. Each functional unit may be realized as a program that runs on the calculation device 11, or may be realized as a module having dedicated hardware.

[0020] The input unit 100 accepts an image for detecting a target object, which is input from the interface device 10. The feature extraction unit 101 calculates features for the image input to the input unit 100. The object region detection unit 102 calculates the region of the target object in the image as a rectangle circumscribing the object, based on the features output by the feature extraction unit 101. Also, based on the features, it calculates the object identification result indicated by each detected region, and the likelihood indicating the accuracy of the identification result.

[0021] Object end point estimation unit 103 calculates the coordinate values ​​of two end points of each object for all object regions output by object region detection unit 102. For a combination of two object regions selected from the object regions output by object region detection unit 102, object bridging unit 104 calculates a line connecting the object end points of both regions output by object end point estimation unit 103 as a set of coordinate values ​​in the input image.

[0022] The bridge luminance value extraction unit 105 extracts luminance values ​​on the line output by the object cross-linking unit 104 from the image input to the input unit 100. The same object determination unit 106 receives as input the luminance values ​​on the line connecting the end points output by the bridge luminance value extraction unit 105 for each combination of object regions cross-linked by the object cross-linking unit 104, and determines whether or not the object regions originate from the same object separated by occlusion.

[0023] The object region correction unit 107 corrects the target object region output by the object region detection unit 102 based on the determination result output by the same object determination unit 106. The output unit 108 outputs the detection result of the target object region corrected by the object region correction unit 107 to the outside of the device. <Configuration and operation of each part>

[0024] Below, we will explain in detail the operations of the functional units, namely, feature extraction unit 101, object region detection unit 102, object end point estimation unit 103, object bridge unit 104, bridge portion luminance value extraction unit 105, same object determination unit 106, object region correction unit 107, and output unit 108.

[0025] The feature extraction unit 101 calculates features for an image input to the input unit 100. Deep learning such as a Convolutional Neural Network (CNN) is used to calculate the features.

[0026] The object region detection unit 102 calculates the region of the target object in the image, the object identification result indicated by each detected region, and the likelihood indicating the accuracy of the identification result, based on the feature amount output by the feature amount extraction unit 101. Note that deep learning such as CNN is used to calculate the object region, the identification result, and the likelihood.

[0027] For learning, many learning data sets of input images and correct object regions and classification results are prepared. Images are input to feature extraction unit 101, and the output from object region detection unit 102 is compared with the correct answer, and parameters of feature extraction unit 101 and object region detection unit 102 are updated based on the comparison result. Even when there is only one type of object to be detected, it is possible to calculate the classification result that the detected object region indicates the target object, and also calculate the likelihood that it is the target object.

[0028] The object end point estimation unit 103 calculates the coordinate values ​​of two end points of each object for all object regions output by the object region detection unit 102. Deep learning such as CNN is used to estimate the end points. For learning, many learning data sets indicating the coordinate values ​​of the detected object regions and correct end points are prepared. An image of the object region is input to the object end point estimation unit 103, the output from the object end point estimation unit 103 is compared with the correct answer, and the parameters of the object end point estimation unit 103 are updated based on the comparison result. The calculation of the end points may be performed based on a preset condition. For example, the intersection point between the longest line among the straight lines passing through the center of gravity of the object and the contour of the object may be set as the end point.

[0029] For each combination of object regions with two identically identified results selected from the object regions output by the object region detection unit 102, the object bridging unit 104 calculates a line (trajectory) connecting the object end points of both regions output by the object end point estimation unit 103 as a set of coordinate values ​​in the input image. The bridging line may be a broken line connecting adjacent end points of both objects with a straight line, or may be a curved line that complements the end points of both objects using a method such as spline interpolation in case the shape of the target object is curved.

[0030] Bridge luminance value extraction section 105 extracts luminance values ​​of coordinates on the line output by object bridge section 104 from the image input to input section 100 for each combination of two selected object regions output by object region detection section 102. The extracted luminance values ​​are converted into a one-dimensional vector (array of luminance values) and output to same object determination section 106. The luminance values ​​to be extracted may be pixel values ​​of the input image directly, or pixel values ​​of an image obtained by applying a smoothing process to the input image to remove noise. In a color image, pixel luminance may be calculated from R, G, and B luminance values ​​using a predetermined formula.

[0031] The same object determination unit 106 receives the luminance value on the bridged line output by the bridge luminance value extraction unit 105 and determines whether or not the two object regions are the same object divided by an obstacle, for each combination of two object regions selected from the object region detection unit 102. The determination may be made by comparing the luminance value transition on the line with a preset pattern of luminance value transition, or by using machine learning such as SVM (Support Vector Machine) or deep learning such as CNN. In the determination, a likelihood indicating the likelihood that both bridged object regions originate from the same divided object is calculated. If the calculated likelihood is equal to or greater than a threshold value preset by the user, the two object regions are determined to originate from the same object.

[0032] The model of the same object determination unit 106 is trained by preparing a one-dimensional vector indicating the luminance value transition on the line and many training data sets indicating the correct answer of whether the objects are the same or different. The likelihood of the correct answer may be, for example, 1 if the objects are the same, and 0 if the objects are different. The one-dimensional vector indicating the luminance value on the line is input to the model, the output likelihood is compared with the correct answer, and the model parameters are updated based on the comparison result. As described above, the same object determination unit 106 may use the model to calculate the likelihood of the objects being the same, and compare the value with a preset threshold value to make a determination.

[0033] As described above, the same object determination unit 106 may compare the luminance value transition on the line with a preset pattern of luminance value transition and determine whether or not the two object regions are the same object according to the degree of similarity. As will be described later with reference to Fig. 5, the luminance value transition of an obstacle and the same object divided by the obstacle may have a specific shape (pattern).

[0034] Specifically, when an obstacle casts a shadow on a target object behind, a valley-shaped brightness drop may occur in the brightness transition as shown in Fig. 5. When an obstacle in front casts a shadow on an object behind, the brightness transition shows a steep drop in brightness at the point where the obstacle in front approaches the shadow area, and the shadow gradually fades and the brightness increases as the obstacle in front moves away. Therefore, when the brightness transition shows this pattern of brightness drop, it can be determined that an object that can become an obstacle is present in front.

[0035] One or more patterns of luminance value transitions showing the same object divided by an obstacle may be prepared in advance, and if the similarity with any of the patterns exceeds a threshold, the two object regions may be determined to be parts of the same object divided by an obstacle. Alternatively, one or more patterns of luminance value transitions showing two different objects may be further prepared, and the state shown by the pattern with the greatest similarity may be determined to be the current state of the target.

[0036] The object region correction unit 107 corrects the object region for each combination of two selected object regions from the object region detection unit 102 based on the determination result output by the same object determination unit 106. When the same object determination unit 106 determines that the two bridged object regions originate from the same object, both object regions are deleted, and a rectangle circumscribing both object regions is created as a new object region. The likelihood of the new object region may be the likelihood of either of the two object regions before correction calculated by the object region detection unit 102, or a new likelihood such as the average value of the likelihoods of both object regions may be calculated and used.

[0037] Output unit 108 outputs to the outside of the device the target object region output by object region detection unit 102 or the detection result of the target object region corrected by object region correction unit 107, and the identification result and likelihood of the detected object region. As for the output format, the coordinate values ​​indicating the target object region, the identification result and its likelihood may be output as numerical data, or they may be drawn as rectangles or characters on the input image and output as an image.

[0038] 3 is a flowchart showing an example of a processing procedure of the image processing method according to the embodiment 1. In the following, each functional unit of the image processing device 1 will be described as an operating subject, but it may be interpreted as the operating device 11 being the operating subject, and the operating device 11 executing each functional unit as a program.

[0039] The input unit 100 receives an image for detecting a target object, and inputs the image to the feature extraction unit 101 (step 200).

[0040] The feature extraction unit 101 obtains information related to the feature extractor stored in the memory 12, and creates a feature extractor. The information related to the feature extractor is, for example, the structural formula of the network and the weighting coefficients of each layer in the network. Furthermore, the feature extractor is used to calculate the feature amount for an input image. (Step 201)

[0041] The object region detection unit 102 calculates the region of the target object in the input image, the object identification result for each region, and the likelihood indicating the accuracy of the identification result from the feature amount extracted by the feature amount extraction unit 101. The detected object region is output to the object edge estimation unit 103 and the object region correction unit. (Step 202)

[0042] FIG. 4A is a diagram showing an example of how occlusion by an obstacle 301 may occur in the input image received in step 200, dividing the object 300 to be detected. FIG. 4B is a diagram showing an example of object 300 that may be output by step 202 and is detected as being divided into regions 302 and 303 due to the division.

[0043] The object edge estimation unit 103 calculates the coordinates of two edge points of the object in each object region detected by the object region detection unit 102 (step 203).

[0044] Fig. 4C is a diagram showing an example of the end point coordinate estimation results of each object output in step 203 for object region 302 and object region 303 in Fig. 4B. Since two end point coordinates are estimated from each object region, end point coordinates 304 and 305 are estimated from object region 302, and end point coordinates 306 and 307 are estimated from object region 303.

[0045] The object bridge unit 104 calculates a line connecting the total of four end points calculated by the object end point estimation unit 103 for a combination of two selected object regions from among those detected by the object region detection unit 102, as a set of coordinate values ​​in the input image (step 204).

[0046] 4D is a diagram showing an example of a line 308 that connects object endpoints 304 and 305 in object region 302 with object endpoints 306 and 307 in object region 303, which is output in step 204. Here, line 308 can be regarded as a trajectory that passes through the center of object region 302 and object region 303 on the assumption that they originate from the same object, and by determining the presence or absence of occlusion on this trajectory, it is possible to determine whether object region 302 and object region 303 are the same object.

[0047] The bridge luminance value extraction unit 105 extracts the luminance values ​​on the line calculated by the object bridge 104 in the input image, and creates a one-dimensional vector (step 205).

[0048] 5 is a diagram showing an example of a luminance value transition 309 on a line 308 in an input image that can be extracted in step 205. From the luminance value transition 309, it can be inferred, for example, that an obstacle with a high luminance value exists from the convex part in the center of the luminance value transition, and that the concave parts on both sides of the convex part have a sudden drop in luminance value and a gradual rise in luminance value from the center to the ends, indicating the presence of a shadow of the obstacle, i.e., that the obstacle is in front of the object 300 in the input image.

[0049] Even if the luminance value of the obstacle in front is low, it can be estimated from the presence of a concave portion where the luminance value gradually increases from the center to the edge. If there are multiple concave and convex shapes, it can be estimated that there are multiple obstacles in front. Even if there are concave portions on both sides of a convex portion, if the concave portions are caused by a sudden drop in luminance value and a sudden rise in luminance value, it can be estimated that two separate objects are casting shadows on the background, and the state can be considered not to be a division of the same object.

[0050] The same object determination unit 106 receives the vector calculated by the bridge luminance value extraction unit 105 and determines whether the combination of object regions is derived from the same object separated by occlusion (step 206).

[0051] When the same object determination unit 106 determines that the combination of object regions selected from the object region detection unit 102 originates from the same object, the object region correction unit 107 integrates the two object regions output from the object region detection unit 102 (step 207).

[0052] The above steps 204, 205, 206, and 207 are repeated for all combinations of two selected object regions.

[0053] The output unit 108 outputs the object region detected by the object region detection unit 102 and corrected by the object region correction unit 107 in accordance with the determination output by the same object determination unit 106 (step 208).

[0054] As a result, it is possible to provide an image processing device and method that detects a target object from an image and prevents an object that is divided due to occlusion from being detected as multiple separate objects. EXAMPLES

[0055] The second embodiment provides an image processing apparatus and method for reducing the number of repetitions of the processes of steps 204, 205, 206, and 207 in the first embodiment and improving throughput.

[0056] Fig. 6 is a block diagram showing the functional configuration of an image processing device in Example 2. The image processing device 1 in Example 2 includes many components similar to those of the image processing device 1 in Example 1 (see Fig. 1), but includes an object candidate selection unit 109 as a new component. In the following, the points that overlap with Example 1 will be omitted, and different components will be mainly described. <Configuration and operation of each part>

[0057] The operations of input unit 100, feature amount extraction unit 101, object region detection unit 102, object edge estimation unit 103, and output unit 108 are the same as those in the embodiment 1. Therefore, the following will explain object candidate selection unit 109, object bridge unit 104, bridge portion luminance value extraction unit 105, same object determination unit 106, and object region correction unit 107.

[0058] The object candidate selection unit 109 selects one or more combinations of regions that may be divided by occlusion from among the object regions detected by the object region detection unit 102. Specifically, the object candidate selection unit 109 selects the combinations based on the positional relationship of the object endpoint coordinates in each region output by the object endpoint estimation unit 103. As the positional relationship of the object endpoint coordinates, for example, the distance between the closest endpoints of the two regions may be calculated and used, or the angle between adjacent lines that connect the closest endpoints of both regions may be calculated and used.

[0059] The object bridging unit 104 calculates a line connecting the object endpoints of both the regions output by the object endpoint estimation unit 103 as a set of coordinate values ​​in the input image for the combination of object regions selected by the object candidate selection unit 109. The other operations are the same as those in the first embodiment.

[0060] The bridge luminance value extraction unit 105 extracts luminance values ​​of coordinates on the line output by the object bridge unit 104 from the image input to the input unit 100 for the combination of object regions selected by the object candidate selection unit 109. Other operations are the same as those in the first embodiment.

[0061] The same object determination unit 106 receives the luminance value on the bridge line output by the bridge luminance value extraction unit 105 for the combination of object regions selected by the object candidate selection unit 109, and determines whether or not they are the same object separated by an obstacle. Other operations are the same as those in the first embodiment.

[0062] The object region correction section 107 corrects the object region output by the object region detection section 102 based on the determination result output by the same object determination section 106 for the combination of object regions selected by the object candidate selection section 109 .

[0063] Fig. 7 is a flow chart showing an example of a processing procedure of an image processing method according to the second embodiment. In the following, each functional unit of the image processing device 1 will be described as the subject of operation, but it may be interpreted as the computing device 11 being the subject of operation and executing each functional unit as a program. The image processing method in the second embodiment includes many steps similar to those of the image processing method in the first embodiment (see Fig. 3), but also includes step 209 as a new step. In the following, the processing procedure shown in Fig. 7 will be described with the portions that overlap with those in the first embodiment omitted, and different steps will be described.

[0064] The processes of steps 200, 201, 202, and 208 in the second embodiment are similar to the image processing method in the first embodiment.

[0065] The object edge point estimation unit 103 calculates the coordinates of two edge points of the object in each object region detected by the object region detection unit 102, and outputs the calculated coordinates to the object candidate selection unit 109. (Step 203)

[0066] The object candidate selection unit 109 calculates the positional relationship of the endpoint coordinates of each object calculated by the object endpoint estimation unit 103 for each pair of two selected object regions from the object regions detected by the object region detection unit 102. The positional relationship of the object endpoint coordinates is the distance between the closest endpoint coordinates between the two object regions, the angle between adjacent straight lines when the closest endpoints of both regions are connected by straight lines, etc. (Step 209)

[0067] Furthermore, the object candidate selection unit 109 selects candidates for combinations of two object regions that may originate from the same object based on the positional relationship of the endpoint coordinates. For example, if the closest endpoint coordinates between two object regions are more than a certain distance apart, it may be determined that they cannot originate from the same object because they are far apart in distance, or if the straight lines intersect at an angle less than a certain angle, it may be determined that they cannot originate from the same object because the target object does not bend excessively. (Step 210)

[0068] 8A to 8D are diagrams showing examples of the positional relationship of object endpoint coordinates calculated in step 209. Fig. 8A shows an example in which distance 402 between adjacent endpoint coordinates of both objects is short, and Fig. 8B shows an example in which distance 405 is long. For example, in step 210, if a threshold value of the distance between object endpoints preset by the user according to the target object is longer than distance 402 and shorter than distance 405, the combination of region 400 and region 401 is selected as being likely to originate from the same object, and the combination of region 403 and region 404 is not selected as being unlikely to originate from the same object, thereby narrowing down the combinations of object regions to be output to the subsequent steps.

[0069] 8C shows an example in which angle 408 between adjacent straight lines connecting adjacent end points of both objects is large, and Fig. 8D shows an example in which the angle 411 is small. In step 210, for example, if a threshold value for the angle between the straight lines connecting object end points, which is preset by the user according to the target object, is smaller than angle 408 and larger than angle 411, the combination of region 406 and region 407 is selected as being likely to originate from the same object, and the combination of region 409 and region 410 is not selected as being unlikely to originate from the same object, thereby narrowing down the combinations of object regions to be output to the subsequent steps.

[0070] For each combination of two object regions selected by the object candidate selection unit 109 as possibly originating from the same object, the object bridging unit 104, bridge portion brightness value extraction unit 105, same object determination unit 106 and object region correction unit 107 bridge the objects (step 204), obtain the brightness value of the bridged portion (step 205), determine that both object regions originate from the same object (step 206), and correct the object region (step 207), in the same manner as the processing in Example 1.

[0071] The above steps 204, 205, 206, and 207 are repeated for all combinations of two object regions output by the object candidate selection unit.

[0072] As a result, it is possible to provide an image processing apparatus and method that can reduce the number of times that the steps in Example 1, such as bridging between object regions (step 204), extracting the brightness value of the bridging portion (step 205), determining that both object regions originate from the same object (step 206), and correcting the object regions (step 207), are performed, thereby improving throughput. EXAMPLES

[0073] The third embodiment provides an image processing apparatus and a method thereof for improving the accuracy of determining object regions originating from the same object in the first embodiment.

[0074] 9 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment of the present invention. The image processing device 1 according to the third embodiment includes many components similar to those of the image processing device 1 according to the first embodiment (see FIG. 1), but includes one or both of an object shape extraction unit 110 and an object luminance value extraction unit 111 as new components. In the following, the points that overlap with the first embodiment will be omitted, and different configurations will be described. <Configuration and operation of each part>

[0075] The operations of input unit 100, feature amount extraction unit 101, object region detection unit 102, object edge estimation unit 103, object bridge unit 104, bridge portion luminance value extraction unit 105, object region correction unit 107, and output unit 108 are the same as those in embodiment 1. Therefore, the following will explain object shape extraction unit 110, object luminance value extraction unit 111, and same object determination unit 106.

[0076] The object shape extraction unit 110 extracts features derived from the shape of an object in each object region detected by the object region detection unit 102. Specifically, features derived from the shape of an object include the average and transition of the object width, and the degree of unevenness of the object contour. The degree of unevenness indicates the roughness of the contour, and may be expressed, for example, as the ratio of the length of the contour line to the average contour line. The shape features may be calculated by extracting the detailed shape of the object by image processing, or may be calculated by estimating the features derived from the shape of the object by using an image of the object region as input by deep learning or the like.

[0077] The object luminance value extraction unit 111 extracts features derived from the luminance value of an object in each object region detected by the object region detection unit 102. Specifically, features derived from the luminance value of an object include an average luminance value in each object region and a transition of luminance values ​​between object edges. The luminance value features of the object may be calculated from the object region and the object edge points, or may be calculated using a detailed object shape obtained by image processing, deep learning, or the like.

[0078] The same object determination unit 106 determines whether or not the combination of object regions selected by the object candidate selection unit 109 is the same object separated by an obstacle. In the determination, the presence or absence of occlusion between the object regions is determined based on the luminance value on the bridge line output by the bridge luminance value extraction unit 105.

[0079] Furthermore, the unit 106 determines that both object regions originate from the same object by taking into account the similarity and continuity of features in both objects from either or both of the object shape features output by the object shape extraction unit 110 and the object luminance value features output by the object luminance value extraction unit 111. If the similarity of features is higher than a threshold value or if the continuity of features is higher than a threshold value, it determines that the two object regions represent parts of the same object. The similarity can be calculated, for example, by difference or cosine similarity. The continuity may be calculated, for example, from the difference in features at adjacent end points. As described above, the unit 106 determines that both objects are separate objects from the similarity and continuity of the features of both objects, even when an occlusion by an obstacle exists between the separate objects.

[0080] Fig. 10 is a flow chart showing an example of a processing procedure of an image processing method according to the third embodiment. In the following, each functional unit of the image processing device 1 will be described as the subject of operation, but it may be interpreted as the computing device 11 being the subject of operation and executing each functional unit as a program. The image processing method in the third embodiment includes many steps similar to those of the image processing method in the first embodiment (see Fig. 3), but also includes either step 210 or step 211, or both, as new steps. In the following, the processing procedure shown in Fig. 7 will be described with the portions overlapping with those in the first embodiment omitted, and different steps will be described.

[0081] The processes of steps 200, 201, 202, 203, 204, 205, 207, and 208 in the third embodiment are similar to the image processing method in the first embodiment.

[0082] The object shape extraction unit 110 extracts a feature amount derived from the shape of the object for each object region output by the object region detection unit 102 (step 211).

[0083] The object luminance value extraction unit 111 extracts a feature amount derived from the luminance value of the object for each object region output by the object region detection unit 102 (step 212).

[0084] The same object determination unit 106 receives the vector calculated by the bridge luminance value extraction unit 105, the feature value derived from the object shape output by the object shape extraction unit 110, and the feature value derived from the luminance value of the object output by the object luminance value extraction unit 111, and determines whether the combination of the object regions is derived from the same object separated by occlusion (step 206).

[0085] As a result, it is possible to provide an image processing apparatus and method that improves the accuracy of determining object regions originating from the same object divided by occlusion in the first embodiment. EXAMPLES

[0086] The fourth embodiment is an image processing system that uses the image processing device according to the first embodiment to detect a target object and correct a detected object region that is divided by occlusion.

[0087] FIG. 11 is a diagram showing a hardware configuration of an image processing system according to a fourth embodiment of the present invention.

[0088] The image processing system 1000 includes an imaging device 1001 , an image processing device 1 , a storage device 1002 , and a display device 1003 .

[0089] The imaging device 1001 is a device for capturing an image for detecting a target object, such as a camera or a microscope.

[0090] The image processing device 1 is the image processing device described in the first embodiment, and calculates the area of ​​the target object from the image captured by the imaging device 1001, and the classification result and its likelihood for each detected area, or a combination of these.

[0091] The storage device 1002 holds information related to object detection that is preset by a user. Specifically, the information related to object detection includes information on a detection target object such as an object name, a threshold value to be set for the likelihood output by the image processing device 1, and the like.

[0092] As a result, it is possible to provide an image processing system that can detect a target object in an image as a single object even if the object is divided due to occlusion.

[0093] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0094] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0095] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0096] 1: image processing device, 10: interface unit, 12: memory, 13: bus, 100: input unit, 101: feature extraction unit, 102: object region detection unit, 103: object edge estimation unit, 104: object bridge unit, 105: bridge brightness value extraction unit, 106: same object determination unit, 107: object region correction unit, 108: output unit, 109: object candidate selection unit, 110: object shape extraction unit, 111: object brightness value extraction unit, 1000: image processing system, 1001: imaging device, 1002: storage device, 1003: display device

Claims

1. An image processing device for detecting a specific object in an image, an input unit that accepts an input image; a feature extraction unit that extracts features from the input image; an object region detection unit that detects an object region in the input image using the feature amount; an object edge point estimation unit that estimates coordinates of edge points of an object in the object region; an object bridging unit that calculates a trajectory that complements the endpoints of two object regions on the input image; an identical object determination unit that determines whether the two object regions are included in the same object based on the transition of brightness values ​​on the trajectory; an object region correction unit that corrects the object region output by the object region detection unit based on the determination result output by the identical object determination unit; an output unit that outputs the corrected object region.

2. 2. The image processing device according to claim 1, The object bridging unit interpolates between endpoints of the two object regions by spline interpolation, and calculates a curved trajectory bridging the two object regions.

3. 2. The image processing device according to claim 1, an object candidate selection unit that selects a combination of two object regions to be output to the object bridging unit from the object regions output by the object region detection unit based on the positional relationship of the coordinates of the endpoints;

4. 2. The image processing device according to claim 1, an object shape extraction unit that extracts feature quantities of an object shape for each object region output by the object region detection unit, and uses the extracted feature quantities of the object shape as input to the same object determination unit.

5. 2. The image processing device according to claim 1, an object luminance value extraction unit that extracts a feature of the luminance value of an object for each object region output by the object region detection unit, and uses the extracted feature of the luminance value of the object as input to the same object determination unit.

6. 2. The image processing device according to claim 1, the same object determination unit determines whether the two object regions are included in the same object using a model; the input of the model includes a brightness value progression on the orbit; the model calculates a likelihood indicating the likelihood that the two object regions are included in the same object; The image processing device, wherein the model has been trained using a training dataset including a combination of brightness value transitions on the trajectory and likelihoods.

7. 2. The image processing device according to claim 1, the object edge point estimation unit estimates coordinates of edge points of an object in the object region using a model; inputs of the model include images of the detected object regions; The model outputs the coordinates of the end points of the input object region; The image processing device, wherein the model has been trained using a training dataset including a combination of an image of an object region and coordinates of endpoints of the object region.

8. An image processing method in which an apparatus detects a specific object in an image, comprising: The device detects an object region from an input image; The device extracts a luminance value transition between two detected object regions; the device determines whether the two object regions are included in the same object based on the brightness value transition; the device modifying the object region in response to a result of the determination; The extraction of the luminance value transition is For each of the detected object regions, coordinates of the object's endpoints are determined; For each combination of two of the object regions, interpolate between the endpoints to calculate a trajectory; extracting the brightness value transition on the trajectory from the input image.

9. The image processing method according to claim 8, The image processing method includes extracting the brightness value transition by narrowing down a combination of two object regions for which the trajectory is calculated based on a positional relationship between the endpoints of the object regions.

10. An image processing system for detecting a specific object in an image, comprising: an imaging device for imaging the sample; The image processing device according to claim 1 ; a storage device for storing information used in the image processing device; a display device that displays the image processing result output by the image processing device.

11. The image processing system according to claim 10, The image processing system is configured such that the image processing device acquires information stored in the storage device and creates a model to be used for processing.

12. The image processing system according to claim 10, when an image is input from the imaging device, the image processing device transmits a detection result of a target object in the input image to a display device; The display device outputs the detection result of the target object.

Images