Image processing device, image processing method, and image processing system
The image processing device corrects object detection by analyzing brightness value transitions between regions, integrating them if necessary, to ensure accurate detection of objects as a single entity despite occlusion, thereby enhancing detection accuracy.
Patent Information
- Application Number
- PCT/JP2024/015752
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-04-22
- Publication Date
- 2024-12-05
AI Technical Summary
Existing image processing techniques fail to accurately detect objects as a single entity when occlusion occurs, leading to division into multiple parts, which affects detection accuracy.
An image processing device and method that analyzes the transition of brightness values between detected object regions to determine if they belong to the same object, integrating regions if necessary to correct the detection and prevent over-detection.
Enables the detection of objects as a single entity even when divided by occlusion, improving detection accuracy and preventing false segmentation.
Smart Images

Figure JP2024015752_05122024_PF_FP_ABST
Abstract
Description
Image processing device, image processing method, and image processing system Incorporation by Reference
[0001] This application claims priority from Japanese Patent Application No. 2023-090096, filed on May 31, 2023, the contents of which are incorporated herein by reference.
[0002] The present invention relates to an image processing technique for detecting a specific object in an image.
[0003] In recent years, the detection of specific objects has become common in the analysis of images captured by cameras and the like. In object detection, a technique for preventing a decrease in detection accuracy due to obstructions of the object by obstacles is described, for example, in Japanese Patent Application Laid-Open No. 2021-056899. This publication states that "based on the position of a moving object area detected from an image and a predetermined area in the image, a corrected image is generated in which the color of at least a portion of the predetermined area is corrected based on the color of the moving object area."
[0004] In JP 2021-056899 A, of the small areas divided from a predetermined area set as a potential area for occlusion, only the portions adjacent to the detection target object are corrected. However, if the occlusion is widespread, there will be small occlusion areas that are not adjacent to the target object. Furthermore, if the object is divided by occlusion, the division of the object will not be resolved even in the corrected image, and the object may be detected as multiple pieces.
[0005] Therefore, an object of the present invention is to provide a technique for alleviating the problem of an object divided by occlusion being detected as multiple separate pieces.
[0006] To solve the above problem, one of the representative image processing devices, methods, and systems of the present invention analyzes the transition in brightness values between two detected object regions, and if it is determined that both objects are a single object separated by an obstacle, it merges the two object regions and corrects the detection result of the object.
[0007] Another aspect of the present invention is an image processing device that detects a specific object in an image, including an input unit that accepts an input image, a feature extraction unit that extracts features from the input image, an object region detection unit that detects an object region in the image using the features, an object endpoint estimation unit that estimates coordinates of endpoints of an object in the object region, an object bridging unit that calculates a trajectory that complements the endpoints of two object regions on the input image, an identical object determination unit that determines whether the two object regions are included in the same object based on the transition of brightness values on the trajectory, an object region correction unit that corrects the object region output by the object region detection unit based on the determination result output by the identical object determination unit, and an output unit that outputs the corrected object region.
[0008] Another aspect of the present invention is an image processing method in which an apparatus detects a specific object in an image, the method including: detecting an object region from an input image; extracting a luminance value transition between two detected object regions; determining whether the two object regions contain the same object based on the luminance value transition; and modifying the object region in accordance with the result of the determination.
[0009] According to one aspect of the present invention, it is possible to provide an apparatus, method, and system that can detect an object that is divided into multiple parts on an image due to occlusion by an obstacle as a single object without splitting it into multiple parts.
[0010] FIG. 1 is a diagram showing an example of a hardware configuration of an image processing device according to Example 1; FIG. 2 is a block diagram showing an example of a functional configuration of an image processing device according to Example 1; FIG. 3 is a flowchart explaining an example of a processing flow of an image processing method according to Example 1; FIG. 4 is a diagram explaining a process of bridging between object regions according to Example 1; FIG. 5 is a diagram explaining a process of bridging between object regions according to Example 1; FIG. 6 is a diagram explaining a process of extracting luminance values between object regions according to Example 1;
[0011] The following describes an embodiment with reference to the accompanying drawings. In one embodiment of this specification, the transition of brightness values between two detected object regions is analyzed, and if it is determined that the two objects are a single object separated by an obstacle, the two object regions are integrated and the detection result of the object is corrected. This allows an object separated by an occlusion caused by an obstacle in the image to be detected as a single object.
[0012] Example 1 provides an image processing device and method that detects an area of a specific object from an image, and further refers to the transition in brightness values between the areas to determine whether the target object is divided by occlusion caused by an obstacle, and corrects the detected object area.
[0013] More specifically, when an object region is detected as being divided into multiple regions due to occlusion by an obstacle in an image, a line connecting the divided object regions is calculated, and the presence or absence of division due to occlusion is determined from the brightness value transition on that line. This makes it possible to integrate object regions originating from the same object, thereby preventing overdetection in object detection. Various types of object can be detected, including, for example, rod-shaped bacilli, nanorods (nanoscale and rod-shaped metal or semiconductor materials), and the human body. <Hardware configuration of image processing device>
[0014] 1 is a diagram illustrating an example of the hardware configuration of an image processing device according to Example 1. The image processing device 1 includes an interface device 10, a calculation device 11, a memory 12, and a bus 13. The interface device 10, the calculation device 11, and the memory 12 transmit and receive information via the bus 13.
[0015] We will now explain each part of the image processing device 1. The interface device 10 is a communication device that transmits and receives signals to and from devices external to the image processing device 1. Devices that communicate with the interface device 10 include an imaging device 20 such as a camera or microscope, and a display device 21 such as a monitor or printer.
[0016] The arithmetic unit 11 is a device that executes various processes within the image processing device 1, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The functions executed by the arithmetic unit 11 will be described later with reference to FIG. 2.
[0017] The memory 12 is a device that stores the programs executed by the arithmetic unit 11, the networks used in the processing and their weights, the processing results, etc., and is, for example, a hard disk drive (HDD) or a solid state drive (SSD).
[0018] FIG. 2 is a block diagram illustrating an example of the functional configuration of the image processing apparatus according to the first embodiment.
[0019] The image processing device 1 includes an input unit 100, a feature extraction unit 101, an object region detection unit 102, an object edge estimation unit 103, an object bridging unit 104, a bridge luminance value extraction unit 105, a same object determination unit 106, an object region correction unit 107, and an output unit 108. Each functional unit may be realized as a program that runs on the calculation device 11, or may be realized as a module having dedicated hardware.
[0020] The input unit 100 receives an image for detecting a target object, which is input from the interface device 10. The feature extraction unit 101 calculates features for the image input to the input unit 100. The object region detection unit 102 calculates the region of the target object in the image as a rectangle circumscribing the object, based on the features output by the feature extraction unit 101. Furthermore, based on the features, it calculates an identification result of the object indicated by each detected region, and a likelihood indicating the certainty of the identification result.
[0021] The object edge point estimation unit 103 calculates the coordinate values of two edge points of each object for all object regions output by the object region detection unit 102. For a combination of two object regions selected from the object regions output by the object region detection unit 102, the object bridging unit 104 calculates a line connecting the object edge points of both regions output by the object edge point estimation unit 103 as a set of coordinate values in the input image.
[0022] The bridge luminance value extraction unit 105 extracts luminance values on the line output by the object bridging unit 104 from the image input to the input unit 100. The same object determination unit 106 receives as input the luminance values on the line connecting the endpoints output by the bridge luminance value extraction unit 105 for each combination of object regions bridged by the object bridging unit 104, and determines whether the object regions originate from the same object separated by occlusion.
[0023] The object region correction unit 107 corrects the target object region output by the object region detection unit 102 based on the determination result output by the same object determination unit 106. The output unit 108 outputs the detection result of the target object region corrected by the object region correction unit 107 to the outside of the device. <Configuration and operation of each unit>
[0024] Below, we will explain in detail the operations of the functional units, namely, the feature extraction unit 101, the object region detection unit 102, the object edge estimation unit 103, the object bridging unit 104, the bridge region brightness value extraction unit 105, the same object determination unit 106, the object region correction unit 107, and the output unit 108.
[0025] The feature extraction unit 101 calculates feature amounts for an image input to the input unit 100. Deep learning such as a convolutional neural network (CNN) is used to calculate the feature amounts.
[0026] The object region detection unit 102 calculates the region of the target object in the image, the object identification result indicated by each detected region, and the likelihood indicating the accuracy of the identification result, based on the feature amounts output by the feature extraction unit 101. Note that deep learning such as CNN is used to calculate the object region, the identification result, and the likelihood.
[0027] For learning, many learning data sets of input images, correct object regions, and classification results are prepared. Images are input to feature extraction unit 101, and the output from object region detection unit 102 is compared with the correct answer, and parameters of feature extraction unit 101 and object region detection unit 102 are updated based on the comparison result. Even when there is only one type of object to be detected, it is possible to calculate the classification result that the detected object region indicates the target object, and also calculate the likelihood that it is the target object.
[0028] The object edge point estimation unit 103 calculates the coordinate values of two edge points of each object for all object regions output by the object region detection unit 102. Deep learning such as CNN is used to estimate the above edge points. For learning, many training data sets indicating the coordinate values of the detected object regions and the correct edge points are prepared. An image of the object region is input to the object edge point estimation unit 103, and the output from the object edge point estimation unit 103 is compared with the correct answer, and the parameters of the object edge point estimation unit 103 are updated based on the comparison result. Note that the calculation of the edge points may be performed based on preset conditions. For example, the intersection of the longest line passing through the center of gravity of the object and the object's contour may be determined as the edge point.
[0029] For each combination of object regions with two identical identification results selected from the object regions output by the object region detection unit 102, the object bridging unit 104 calculates a line (trajectory) connecting the object endpoints of both regions output by the object endpoint estimation unit 103 as a set of coordinate values in the input image. The bridging line may be a polygonal line connecting adjacent endpoints of both objects with a straight line, or may be a curved line interpolating between the endpoints of both objects using a method such as spline interpolation in case the shape of the target object is curved.
[0030] The bridge luminance value extraction unit 105 extracts luminance values of coordinates on the line output by the object bridge unit 104 from the image input to the input unit 100 for each combination of two selected object regions output by the object region detection unit 102. The extracted luminance values are converted into a one-dimensional vector (array of luminance values) and output to the same object determination unit 106. The extracted luminance values may be directly extracted from the pixel values of the input image, or may be extracted from an image obtained by smoothing the input image to remove noise. In a color image, the luminance of a pixel may be calculated from the luminance values of R, G, and B using a predetermined formula.
[0031] The same object determination unit 106 receives the luminance values on the bridge line output by the bridge luminance value extraction unit 105 and determines, for each combination of two selected object regions output by the object region detection unit 102, whether the two regions are the same object separated by an obstacle. The determination may be made by comparing the luminance value transition on the line with a preset luminance value transition pattern, or by using machine learning such as an SVM (Support Vector Machine) or deep learning such as a CNN. In the determination, a likelihood indicating the likelihood that both bridged object regions originate from the same separated object is calculated. If the calculated likelihood is equal to or greater than a threshold value preset by the user, the two object regions are determined to originate from the same object.
[0032] The model training of the same object determination unit 106 involves preparing a one-dimensional vector indicating the transition of brightness values on a line and many training data sets indicating the correct answer of whether the objects are the same or different. The likelihood of the correct answer may be, for example, 1 if the objects are the same, or 0 if the objects are different. The one-dimensional vector indicating the brightness values on the line is input to the model, the output likelihood is compared with the correct answer, and the model parameters are updated based on the comparison result. As described above, the same object determination unit 106 may use the model to calculate the likelihood of the objects being the same, and compare this value with a preset threshold to make a determination.
[0033] As described above, the same object determination unit 106 may compare the luminance value transition on the line with a preset luminance value transition pattern and determine whether the two object regions are the same object based on the degree of similarity. As will be described later with reference to Figure 5, the luminance value transition of an obstacle and the same object separated by the obstacle may have a specific shape (pattern).
[0034] Specifically, when an obstacle casts a shadow on a target object behind, a valley-shaped brightness drop may occur in the brightness transition as shown in Figure 5. When an obstacle in front casts a shadow on an object behind, the brightness transition shows a sharp drop in brightness at the point where the front obstacle approaches the shadow area, and the shadow gradually fades and the brightness increases as the distance from the front obstacle increases. Therefore, when the brightness transition shows this pattern of brightness drop, it can be determined that an object that may act as an obstacle is present in front.
[0035] One or more patterns of luminance value transitions showing the same object divided by an obstacle may be prepared in advance, and if the similarity with any of the patterns exceeds a threshold, the two object regions may be determined to be parts of the same object divided by an obstacle. Alternatively, one or more patterns of luminance value transitions showing two different objects may be further prepared, and the state shown by the pattern with the greatest similarity may be determined to be the current state of the target.
[0036] For each combination of two selected object regions output from object region detection unit 102, object region correction unit 107 corrects the object region based on the determination result output from same object determination unit 106. If same object determination unit 106 determines that two bridged object regions originate from the same object, both object regions are deleted, and a rectangle circumscribing both object regions is created as a new object region. The likelihood of the new object region may be the likelihood of either of the two object regions before correction calculated by object region detection unit 102, or a new likelihood such as the average value of the likelihoods of both object regions may be calculated and used.
[0037] Output unit 108 outputs to the outside of the device the target object region output by object region detection unit 102 or the detection result of the target object region corrected by object region correction unit 107, as well as the identification result and likelihood of the detected object region. The output format may be such that the coordinate values indicating the target object region, the identification result, and its likelihood are output as numerical data, or such that they are drawn as rectangles or characters on the input image and output as an image.
[0038] 3 is a flowchart showing an example of a processing procedure of the image processing method according to Example 1. In the following, each functional unit of the image processing device 1 will be described as the subject of operations, but it may also be interpreted as the computing device 11 being the subject of operations, and the computing device 11 executing each functional unit as a program.
[0039] The input unit 100 receives an image from which a target object is to be detected, and inputs the image to the feature extraction unit 101 (step 200).
[0040] The feature extraction unit 101 acquires information related to the feature extractor stored in the memory 12 and creates a feature extractor. The information related to the feature extractor includes, for example, the structural formula of the network and the weighting coefficients of each layer in the network. Furthermore, the feature extractor is used to calculate the feature of the input image (step 201).
[0041] The object region detection unit 102 calculates the region of the target object in the input image, the object identification result for each region, and the likelihood indicating the accuracy of the identification result, based on the feature amounts extracted by the feature extraction unit 101. The detected object region is output to the object edge point estimation unit 103 and the object region correction unit (step 202).
[0042] FIG. 4A is a diagram showing an example of how an occlusion by an obstacle 301 may occur on the input image received in step 200, dividing the object 300 to be detected. FIG. 4B is a diagram showing an example of an object 300 that may be output by step 202, where the object 300 is detected as being divided into regions 302 and 303 due to the division.
[0043] The object edge point estimation unit 103 calculates the coordinates of two edge points of the object in each object region detected by the object region detection unit 102 (step 203).
[0044] 4C is a diagram showing an example of the results of estimating the endpoint coordinates of each object output in step 203 for object region 302 and object region 303 in FIG. 4B. Two endpoint coordinates are estimated from each object region, so endpoint coordinates 304 and 305 are estimated from object region 302, and endpoint coordinates 306 and 307 are estimated from object region 303.
[0045] The object bridging unit 104 calculates, for a combination of two selected object regions from those detected by the object region detection unit 102, a line connecting the four endpoints calculated by the object endpoint estimation unit 103 as a set of coordinate values in the input image (step 204).
[0046] 4D is a diagram showing an example of a line 308 that connects object endpoints 304 and 305 in object region 302 with object endpoints 306 and 307 in object region 303, as output in step 204. Here, line 308 can be considered as a trajectory that passes through the center of object region 302 and object region 303, assuming that they originate from the same object. By determining whether or not there is an occlusion on this trajectory, it is possible to determine whether object region 302 and object region 303 are the same object.
[0047] The bridge luminance value extraction unit 105 extracts the luminance values on the line calculated by the object bridge 104 in the input image, and creates a one-dimensional vector (step 205).
[0048] 5 is a diagram showing an example of a luminance value transition 309 on a line 308 in the input image that can be extracted in step 205. From the luminance value transition 309, it is possible to infer, for example, that an obstacle with a high luminance value exists from the convex part in the center of the luminance value transition, and that the shadow of the obstacle exists from the concave parts on both sides of the convex part, which show a sudden drop in luminance value and a gradual rise in luminance value from the center to the ends, i.e., that the obstacle is in front of the object 300 in the input image.
[0049] Even if the brightness value of the obstacle in front is low, it can be inferred from the presence of a concave portion where the brightness value gradually increases from the center to the edge. Furthermore, if there are multiple such concave and convex shapes, it can be inferred that there are also multiple obstacles in front. Furthermore, even if there are concave portions on both sides of a convex portion, if the concave portions are caused by a sudden drop in brightness value and a sudden rise in brightness value, it can be inferred that two separate objects are casting shadows on the background, and the state can be considered not to be a division of the same object.
[0050] The same object determination unit 106 receives the vector calculated by the bridge luminance value extraction unit 105 as an input and determines whether the combination of object regions is derived from the same object separated by occlusion (step 206).
[0051] When the same object determination unit 106 determines that a combination of two selected object regions from the object region detected by the object region detection unit 102 originates from the same object, the object region correction unit 107 integrates the two object regions output by the object region detection unit 102 (step 207).
[0052] The above steps 204, 205, 206, and 207 are repeated for all combinations of two selected object regions.
[0053] The output unit 108 outputs the object region detected by the object region detection unit 102 and corrected by the object region correction unit 107 in accordance with the determination output by the identical object determination unit 106 (step 208).
[0054] As described above, it is possible to provide an image processing device and method that detects a target object from an image and prevents an object that is divided by occlusion from being detected as multiple separate parts.
[0055] The second embodiment provides an image processing apparatus and method that reduces the number of repetitions of the processes of steps 204, 205, 206, and 207 in the first embodiment, thereby improving throughput.
[0056] 6 is a block diagram showing the functional configuration of an image processing device in Example 2. The image processing device 1 in Example 2 includes many components similar to those of the image processing device 1 in Example 1 (see FIG. 1), but includes an object candidate selection unit 109 as a new component. In the following, points that overlap with Example 1 will be omitted, and different components will be mainly described. <Configuration and operation of each unit>
[0057] The operations of the input unit 100, feature amount extraction unit 101, object region detection unit 102, object edge point estimation unit 103, and output unit 108 are the same as those in Example 1. Therefore, the following will explain the object candidate selection unit 109, object bridge unit 104, bridge portion luminance value extraction unit 105, same object determination unit 106, and object region correction unit 107.
[0058] The object candidate selection unit 109 selects one or more combinations of regions that may be separated by occlusion from the object regions detected by the object region detection unit 102. Specifically, the object candidate selection unit 109 selects the combinations based on the positional relationship between the object endpoint coordinates in each region output by the object endpoint estimation unit 103. As the positional relationship between the object endpoint coordinates, for example, the distance between the closest endpoints of the two regions may be calculated and used, or the angle between adjacent lines that connect the closest endpoints of both regions may be calculated and used.
[0059] For the combination of object regions selected by the object candidate selection unit 109, the object bridging unit 104 calculates a line connecting the object endpoints of both regions output by the object endpoint estimation unit 103 as a set of coordinate values in the input image. Other operations are the same as those in the first embodiment.
[0060] The bridge luminance value extraction unit 105 extracts luminance values of coordinates on the line output by the object bridge unit 104 from the image input to the input unit 100 for the combination of object regions selected by the object candidate selection unit 109. Other operations are the same as those in the first embodiment.
[0061] The same object determination unit 106 receives the luminance values on the bridged line output by the bridge luminance value extraction unit 105 and determines whether or not the combinations of object regions selected by the object candidate selection unit 109 are the same object separated by an obstacle. Other operations are the same as those in the first embodiment.
[0062] The object region correction unit 107 corrects the object region output by the object region detection unit 102 based on the determination result output by the same object determination unit 106 for the combination of object regions selected by the object candidate selection unit 109 .
[0063] FIG. 7 is a flowchart showing an example of the processing procedure of an image processing method according to Example 2. In the following, each functional unit of the image processing device 1 will be described as the main operating unit, but it may also be interpreted as the arithmetic device 11 performing the operations and executing each functional unit as a program. The image processing method according to Example 2 includes many steps similar to those of the image processing method according to Example 1 (see FIG. 3), but also includes a new step, step 209. In the following, the processing procedure shown in FIG. 7 will be omitted for the parts that overlap with Example 1, and only the different steps will be described.
[0064] The processes of steps 200, 201, 202, and 208 in the second embodiment are the same as the image processing method in the first embodiment.
[0065] The object edge point estimation unit 103 calculates the coordinates of two edge points of the object in each object region detected by the object region detection unit 102, and outputs the coordinates to the object candidate selection unit 109 (step 203).
[0066] The object candidate selection unit 109 calculates the positional relationship of the endpoint coordinates of each object calculated by the object endpoint estimation unit 103 for each pair of two selected object regions from the object regions detected by the object region detection unit 102. The positional relationship of the object endpoint coordinates refers to the distance between the closest endpoint coordinates between the two object regions, the angle formed by adjacent lines when connecting the closest endpoints of both regions, etc. (Step 209).
[0067] Furthermore, the object candidate selection unit 109 selects candidates for combinations of two object regions that may originate from the same object based on the positional relationship of the endpoint coordinates. For example, if the closest endpoint coordinates between two object regions are separated by more than a certain distance, it may be determined that they cannot originate from the same object because of the distance, or if the lines intersect at an angle less than a certain angle, it may be determined that the target object does not curve excessively and therefore cannot originate from the same object (step 210).
[0068] 8A to 8D are diagrams showing examples of the positional relationship of object endpoint coordinates calculated in step 209. Fig. 8A shows an example in which distance 402 between adjacent endpoint coordinates of both objects is short, while Fig. 8B shows an example in which distance 405 is long. In step 210, for example, if a threshold value for the distance between object endpoints preset by the user in accordance with the target object is longer than distance 402 and shorter than distance 405, the combination of region 400 and region 401 is selected as having a high probability of originating from the same object, and the combination of region 403 and region 404 is not selected as having a low probability of originating from the same object, thereby narrowing down the combinations of object regions to be output to subsequent steps.
[0069] 8C shows an example in which angle 408 formed by adjacent straight lines when connecting adjacent endpoints of both objects with straight lines is large, and Fig. 8D shows an example in which angle 411 is small. For example, in step 210, if a threshold value for the angle formed by the straight lines connecting the object endpoints, which is set in advance by the user in accordance with the target object, is smaller than angle 408 and larger than angle 411, the combination of region 406 and region 407 is selected as having a high possibility of being derived from the same object, and the combination of region 409 and region 410 is not selected as having a low possibility of being derived from the same object, thereby narrowing down the combinations of object regions to be output to subsequent steps.
[0070] For each combination of two object regions selected by the object candidate selection unit 109 as possibly originating from the same object, the object bridging unit 104, bridge portion brightness value extraction unit 105, same object determination unit 106, and object region correction unit 107 perform the same processing as in Example 1, namely, bridging between the objects (step 204), obtaining the brightness value of the bridge portion (step 205), determining that both object regions originate from the same object (step 206), and correcting the object region (step 207).
[0071] The above steps 204, 205, 206, and 207 are repeated for all combinations of two object regions output by the object candidate selection unit.
[0072] As a result of the above, it is possible to provide an image processing device and method that improves throughput by reducing the number of times that the steps of bridging between object regions (step 204), extracting the brightness value of the bridging portion (step 205), determining that both object regions originate from the same object (step 206), and correcting the object regions (step 207) in Example 1 are performed.
[0073] The third embodiment provides an image processing apparatus and method for improving the accuracy of determining object regions originating from the same object in the first embodiment.
[0074] 9 is a block diagram showing an example of the functional configuration of an image processing device according to a second embodiment of the present invention. The image processing device 1 according to the third embodiment includes many components similar to those of the image processing device 1 according to the first embodiment (see FIG. 1), but includes one or both of an object shape extraction unit 110 and an object luminance value extraction unit 111 as new components. In the following, the same points as those in the first embodiment will be omitted, and only the different components will be described. <Configuration and operation of each unit>
[0075] The operations of input unit 100, feature amount extraction unit 101, object region detection unit 102, object edge estimation unit 103, object bridging unit 104, bridge portion luminance value extraction unit 105, object region correction unit 107, and output unit 108 are the same as those in Example 1. Therefore, the object shape extraction unit 110, object luminance value extraction unit 111, and same object determination unit 106 will be described below.
[0076] The object shape extraction unit 110 extracts features derived from the shape of an object from each object region detected by the object region detection unit 102. Specifically, features derived from the shape of an object include the average and transition of the object width, and the degree of unevenness of the object contour. The degree of unevenness indicates contour roughness and may be expressed, for example, as the ratio between the length of the contour line and the average contour line. The shape features may be calculated by extracting the detailed shape of the object through image processing, or may be calculated by estimating the features derived from the shape of the object using an image of the object region as input through deep learning or the like.
[0077] The object luminance value extraction unit 111 extracts features derived from the luminance values of the object in each object region detected by the object region detection unit 102. Specifically, features derived from the luminance values of the object include the average luminance value in each object region and the transition of the luminance values between object edges. The luminance value features of the object may be calculated from the object region and the object edge points, or may be calculated using a detailed object shape obtained by image processing, deep learning, or the like.
[0078] The same object determination unit 106 determines whether or not the combination of object regions selected by the object candidate selection unit 109 is the same object separated by an obstacle. In this determination, the presence or absence of occlusion between the object regions is determined based on the brightness values on the bridge line output by the bridge brightness value extraction unit 105.
[0079] Furthermore, the unit 106 determines whether the two object regions originate from the same object by taking into account the similarity and continuity of the features of both objects from either or both of the object shape features output by the object shape extraction unit 110 and the object luminance value features output by the object luminance value extraction unit 111. If the similarity of the features is higher than a threshold or if the continuity of the features is higher than a threshold, the unit 106 determines that the two object regions represent parts of the same object. Similarity can be calculated, for example, by difference or cosine similarity. Continuity may be calculated, for example, from the difference in features at adjacent endpoints. As described above, the same object determination unit 106 determines that the two objects are separate objects based on the similarity and continuity of the features of both objects, even when an occlusion by an obstacle exists between the separate objects.
[0080] FIG. 10 is a flowchart showing an example of the processing procedure of an image processing method according to Example 3. In the following, each functional unit of the image processing device 1 will be described as the main operating unit, but it may also be interpreted as the arithmetic device 11 performing the operations and executing each functional unit as a program. The image processing method according to Example 3 includes many steps similar to those of the image processing method according to Example 1 (see FIG. 3), but also includes a new step, either step 210 or step 211, or both. In the following, the processing procedure shown in FIG. 7 will be omitted for the parts that overlap with Example 1, and only the different steps will be described.
[0081] The processes of steps 200, 201, 202, 203, 204, 205, 207, and 208 in the third embodiment are the same as the image processing method in the first embodiment.
[0082] The object shape extraction unit 110 extracts feature quantities derived from the shape of the object for each object region output by the object region detection unit 102 (step 211).
[0083] The object luminance value extraction unit 111 extracts a feature quantity derived from the luminance value of the object for each object region output by the object region detection unit 102 (step 212).
[0084] The same object determination unit 106 receives as input the vector calculated by the bridge luminance value extraction unit 105, the feature values derived from the object shape output by the object shape extraction unit 110, and the feature values derived from the luminance values of the object output by the object luminance value extraction unit 111, and determines whether the combination of object regions is derived from the same object separated by occlusion (step 206).
[0085] As described above, it is possible to provide an image processing apparatus and method that improves the accuracy of determining object regions derived from the same object divided by occlusion in the first embodiment.
[0086] The fourth embodiment is an image processing system that uses the image processing device described in the first embodiment to detect a target object and correct the detected object region that is divided by occlusion.
[0087] FIG. 11 is a diagram showing the hardware configuration of an image processing system according to a fourth embodiment of the present invention.
[0088] The image processing system 1000 includes an image capturing device 1001 , an image processing device 1 , a storage device 1002 , and a display device 1003 .
[0089] The image capturing device 1001 is a device for capturing an image for detecting a target object, such as a camera or a microscope.
[0090] The image processing device 1 is the image processing device described in Example 1, and calculates one or more of the target object area from the image captured by the imaging device 1001, the identification result for each detected area, and its likelihood.
[0091] The storage device 1002 stores information related to object detection that is preset by the user. Specifically, the information related to object detection includes information about the object to be detected, such as the object name, and a threshold value to be set for the likelihood output by the image processing device 1.
[0092] As a result, it is possible to provide an image processing system that can detect a target object in an image as a single object even if the object is divided due to occlusion.
[0093] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.
[0094] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.
[0095] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.
Claims
1. An image processing device for detecting a specific object in an image, comprising: an input unit for accepting an input image; a feature extraction unit for extracting features from the input image; an object region detection unit for detecting an object region in the input image using the features; an object endpoint estimation unit for estimating coordinates of an object endpoint in the object region; an object bridging unit for calculating a trajectory that complements the endpoints of two object regions on the input image; an identical object determination unit for determining whether the two object regions are included in the same object based on the transition of brightness values on the trajectory; an object region correction unit for correcting the object region output by the object region detection unit based on the determination result output by the identical object determination unit; and an output unit for outputting the corrected object region.
2. An image processing device according to claim 1, wherein the object bridging section uses spline interpolation to complement the area between the end points of the two object regions and calculates a curved trajectory bridging the two object regions.
3. An image processing device as described in claim 1, comprising an object candidate selection unit which selects a combination of two object regions to be output to the object bridging unit from the object regions output by the object region detection unit based on the positional relationship of the coordinates of the end points.
4. An image processing device as described in claim 1, comprising an object shape extraction unit that extracts object shape features for each object region output by the object region detection unit, and uses the extracted object shape features as input to the same object determination unit.
5. An image processing device as claimed in claim 1, comprising an object luminance value extraction unit which extracts object luminance value features for each object region output by said object region detection unit, and which uses the extracted object luminance value features as input to said same object determination unit.
6. An image processing device as described in claim 1, wherein the same object determination unit uses a model to determine whether the two object regions are included in the same object, the input of the model includes a brightness value transition on the trajectory, the model calculates a likelihood indicating the likelihood that the two object regions are included in the same object, and the model has been trained using a training data set including a combination of the brightness value transition on the trajectory and the likelihood.
7. An image processing device as described in claim 1, wherein the object endpoint estimation unit estimates coordinates of endpoints of an object in the object region using a model, an input of the model includes an image of the detected object region, the model outputs coordinates of the input endpoints of the object region, and the model has been trained using a training data set including combinations of images of the object region and the coordinates of the endpoints of the object region.
8. An image processing method in which an apparatus detects a specific object in an image, the image processing method including: detecting an object region from an input image; extracting a brightness value transition between two detected object regions; determining whether the two object regions contain the same object based on the brightness value transition; and modifying the object region in accordance with the result of the determination.
9. An image processing method according to claim 8, wherein the extraction of the luminance value transition includes: determining the coordinates of the object endpoints for each of the detected object regions; calculating a trajectory by interpolating between the endpoints for each combination of two of the object regions; and extracting the luminance value transition on the trajectory from the input image.
10. An image processing method according to claim 9, wherein the extraction of the luminance value transition narrows down the combination of two object regions for which the trajectory is calculated based on the positional relationship of the end points of the object regions.
11. An image processing method as claimed in claim 8, further comprising extracting object shape features and / or brightness value features for each of the detected object regions, and determining whether the two object regions are included in the same object using the shape features and / or brightness value features in addition to the brightness value transition.
12. An image processing system for detecting a specific object in an image, comprising: an imaging device for imaging a sample; an image processing device according to claim 1; a storage device for holding information used in said image processing device; and a display device for displaying the image processing results output by said image processing device.
13. An image processing system according to claim 12, wherein the image processing device acquires information stored in the storage device and creates a model to be used in processing.
14. An image processing system as described in claim 12, wherein, when an image is input from the imaging device, the image processing device transmits a detection result of a target object in the input image to a display device, and the display device outputs the detection result of the target object.
Citation Information
Patent Citations
Image processor, image processing method, and program
JP2021056899A
Vanin-1 inhibitor
JP2023090096A
Object detection device
JP2011128756A
Image processing apparatus, image processing method, and program
JP2014229210A
Object detection device and object detection method
WO2014073322A1
