Image processing apparatus, image processing method, and image processing system

Abstract

An image processing apparatus includes: an input unit that receives an input image; a feature amount extraction unit that extracts a feature amount from the input image; an object region detection unit that detects an object region in the image using the feature amount; an object end point estimation unit that estimates coordinates of end points of an object in the object region; an object bridging unit that calculates a trajectory that complements between the endpoints in the two object regions on the input image; a same object determination unit that determines whether or not the two object regions are included in the same object on the basis of the brightness value transition on the track; an object region correction unit that corrects the object region output by the object region detection unit on the basis of the determination result output by the same object determination unit; and an output unit that outputs the corrected object region.

Description

[0001] Incorporation by reference

[0002] This application claims the benefit of Japanese patent application No. 2023-090096, filed on May 31, 2023, the contents of which are hereby incorporated by reference into this application. Technical Field

[0003] The present invention relates to an image processing technique for detecting a specific object in an image. Background Art

[0004] In recent years, the analysis of images captured by cameras and other devices has often focused on the detection of specific objects. To prevent the reduction in detection accuracy caused by obstructions in object detection, a technique described in Japanese Patent Application Laid-Open No. 2021-056899 is available. This publication describes a method for generating a corrected image by correcting the color of at least a portion of a moving object region detected in an image based on the color of the moving object region, based on the position of the moving object region detected in the image and a predetermined region in the image. Summary of the Invention

[0005] Problems to be solved by the invention

[0006] Japanese Patent Application Laid-Open No. 2021-056899 corrects only the portion of the image adjacent to the target object within the small regions created by dividing a predetermined area designated as potentially occluded. However, if occlusion occurs over a large area, there may be small occluded regions that are not adjacent to the target object. Furthermore, if the object is segmented due to occlusion, it is conceivable that the segmentation of the object will not be eliminated in the corrected image, and the object may be detected as multiple separate parts.

[0007] Therefore, an object of the present invention is to provide a technique for suppressing the problem of detecting an object that has been segmented due to occlusion as a plurality of objects.

[0008] Means for solving problems

[0009] In order to solve the above-mentioned problems, one of the representative image processing devices, methods and systems of the present invention analyzes the change in brightness values ​​between two detected object areas, and when it is determined that the two objects are a single object separated by an obstacle, integrates the two object areas and corrects the detection result of the object.

[0010] Another embodiment of the present invention is an image processing device for detecting a specific object in an image, comprising: an input unit that receives an input image; a feature extraction unit that extracts feature values ​​from the input image; an object region detection unit that uses the feature values ​​to detect an object region within the image; an object endpoint estimation unit that estimates the coordinates of the endpoints of the object in the object region; an object bridging unit that calculates a track that complements the endpoints in two object regions on the input image; a same object determination unit that determines whether the two object regions are included in the same object based on a shift in brightness values ​​on the track; an object region correction unit that corrects the object region output by the object region detection unit based on a determination result output by the same object determination unit; and an output unit that outputs the corrected object region.

[0011] Another embodiment of the present invention is an image processing method, in which a device detects a specific object in an image, and the image processing method includes: a device detects an object area from an input image; the device extracts a brightness value shift between two detected object areas; the device determines whether the two object areas are included in the same object based on the brightness value shift; and the device corrects the object area based on the result of the determination.

[0012] Effects of the Invention

[0013] According to one embodiment of the present invention, it is possible to provide an apparatus, method, and system that can detect an object as a single object without being divided into multiple objects, even if the object is blocked and divided by an obstacle in an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram showing an example of the hardware configuration of the image processing apparatus according to the first embodiment.

[0015] Figure 2 This is a block diagram showing an example of the functional configuration of the image processing device according to the first embodiment.

[0016] Figure 3 This is a flowchart illustrating an example of the processing flow of the image processing method according to the first embodiment.

[0017] Figure 4A This is a diagram illustrating the bridging process between object regions in the first embodiment.

[0018] Figure 4B This is a diagram illustrating the bridging process between object regions in the first embodiment.

[0019] Figure 4C This is a diagram illustrating the bridging process between object regions in the first embodiment.

[0020] Figure 4DThis is a diagram illustrating the bridging process between object regions in the first embodiment.

[0021] Figure 5 This is a diagram illustrating the process of extracting brightness values ​​between object regions in Example 1.

[0022] Figure 6 This is a block diagram showing an example of the functional structure of the image processing device of the second embodiment.

[0023] Figure 7 This is a flowchart illustrating an example of the processing flow of the image processing method of the second embodiment.

[0024] Figure 8A This is a diagram illustrating the process of selecting a combination of object regions in the second embodiment.

[0025] Figure 8B This is a diagram illustrating the process of selecting a combination of object regions in the second embodiment.

[0026] Figure 8C This is a diagram illustrating the process of selecting a combination of object regions in the second embodiment.

[0027] Figure 8D This is a diagram illustrating the process of selecting a combination of object regions in the second embodiment.

[0028] Figure 9 This is a block diagram showing an example of the functional structure of the image processing device of the third embodiment.

[0029] Figure 10 This is a flowchart showing an example of the processing flow of the image processing method of the third embodiment.

[0030] Figure 11 This is a diagram showing an example of the hardware configuration of the image processing system of the fourth embodiment. DETAILED DESCRIPTION

[0031] The following describes an embodiment with reference to the accompanying drawings. In one embodiment of the present disclosure, the transition between the brightness values ​​of two detected object regions is analyzed. If the two objects are determined to be a single object separated by an obstacle, the two object regions are integrated and the object detection result is corrected. This allows objects to be detected as a single object even if they are separated by occlusion caused by an obstacle.

[0032] Example 1

[0033] Embodiment 1 provides an image processing device and method thereof, which detects a specific object region from an image, and further refers to the brightness value transition between regions to determine the segmentation of the target object when the target object is segmented by occlusion caused by an obstacle, and corrects the detected object region.

[0034] More specifically, if an object is detected as being divided into multiple regions due to occlusion by an obstacle, a line connecting the divided regions is calculated, and the brightness shift along this line is used to determine whether the object is divided due to occlusion. This allows regions belonging to the same object to be integrated, preventing overdetection during object detection. A variety of objects can be detected, such as rod-shaped bacteria, nanorods (nanoscale and rod-shaped metal or semiconductor materials), and the human body.

[0035] <Hardware Structure of Image Processing Device>

[0036] Figure 1 1 is a diagram showing an example of the hardware configuration of the image processing apparatus according to Example 1. The image processing apparatus 1 includes an interface device 10, a computing device 11, a memory 12, and a bus 13. The interface device 10, the computing device 11, and the memory 12 transmit and receive information via the bus 13.

[0037] The components of the image processing device 1 are described below. The interface device 10 is a communication device that transmits and receives signals to and from devices outside the image processing device 1. Devices that communicate with the interface device 10 include an imaging device 20 such as a camera or microscope, and a display device 21 such as a monitor or printer.

[0038] The computing device 11 is a device that performs various processes in the image processing device 1, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. Figure 2 This will be described later.

[0039] The memory 12 is a device that stores the program executed by the computing device 11, the network used in the processing and its weights, processing results, etc., and is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0040] <Functional Structure of Image Processing Device>

[0041] Figure 2 This is a block diagram showing an example of the functional configuration of the image processing device according to the first embodiment.

[0042] The image processing device 1 includes an input unit 100, a feature extraction unit 101, an object region detection unit 102, an object endpoint estimation unit 103, an object bridging unit 104, a bridge brightness value extraction unit 105, an identical object determination unit 106, an object region correction unit 107, and an output unit 108. Each functional unit may be implemented as a program running on the computing device 11 or as a module with dedicated hardware.

[0043] The input unit 100 receives an image of a detection target object from the interface device 10. The feature extraction unit 101 calculates feature quantities for the image input to the input unit 100. The object region detection unit 102 calculates the region of the target object within the image as a rectangle circumscribing the object based on the feature quantities output by the feature extraction unit 101. Furthermore, based on the feature quantities, the unit calculates the recognition result of the object represented by each detected region and the likelihood indicating the probability of the recognition result.

[0044] The object endpoint estimation unit 103 calculates the coordinate values ​​of the two endpoints of each object for all object regions output by the object region detection unit 102. The object bridging unit 104 calculates, for a combination of two object regions selected from the object regions output by the object region detection unit 102, a line connecting the object endpoints of the two regions output by the object endpoint estimation unit 103 as a set of coordinate values ​​in the input image.

[0045] The bridge brightness value extraction unit 105 extracts the brightness values ​​on the line output by the object bridging unit 104 from the image input to the input unit 100. The same object determination unit 106 uses the brightness values ​​on the line connecting the endpoints output by the bridge brightness value extraction unit 105 as input for each combination of object regions bridged by the object bridging unit 104, and determines whether the combination is an object region derived from the same object that was divided by occlusion.

[0046] The object region correction unit 107 corrects the target object region output by the object region detection unit 102 based on the determination result output by the identical object determination unit 106. The output unit 108 outputs the target object region detection result corrected by the object region correction unit 107 to the outside of the device.

[0047] <Structure and Movement of Each Part>

[0048] The following describes in detail the operations of the feature extraction unit 101, object region detection unit 102, object endpoint estimation unit 103, object bridging unit 104, bridge brightness value extraction unit 105, identical object determination unit 106, object region correction unit 107, and output unit 108 in the functional units.

[0049] The feature extraction unit 101 calculates a feature of the image input to the input unit 100. The feature is calculated using deep learning such as CNN (Convolutional Neural Network).

[0050] Based on the features output by the feature extraction unit 101, the object region detection unit 102 calculates the target object region within the image, the recognition result of the object represented by each detected region, and the likelihood indicating the probability of the recognition result. Deep learning, such as CNN, is used to calculate the object region, recognition result, and likelihood.

[0051] For learning, multiple training data sets containing input images, correct object regions, and recognition results are prepared. The image is input to feature extraction unit 101, and the output from object region detection unit 102 is compared with the correct answer. The parameters of feature extraction unit 101 and object region detection unit 102 are updated based on the comparison results. If only one object is detected, the recognition result may be calculated based on the fact that the detected object region represents the target object, and the likelihood of the target object being the target object is calculated.

[0052] The object endpoint estimation unit 103 calculates the coordinate values ​​of the two endpoints of each object for all object regions output by the object region detection unit 102. Deep learning such as CNN is used for the estimation of the above endpoints. For learning, multiple learning data sets representing the coordinate values ​​of the detected object regions and the correct endpoints are prepared. The image of the object region is input to the object endpoint estimation unit 103, and the output from the object endpoint estimation unit 103 is compared with the correct answer. The parameters of the object endpoint estimation unit 103 are updated based on the comparison result. In addition, the calculation of the endpoint can also be performed based on pre-set conditions. For example, the intersection of the longest line among the straight lines passing through the center of gravity of the object and the outline of the object can also be used as the endpoint.

[0053] For each combination of two object regions that have the same recognition result selected from the object regions output by the object region detection unit 102, the object bridging unit 104 calculates a line (orbit) connecting the object endpoints of the two regions output by the object endpoint estimation unit 103 as a set of coordinate values ​​in the input image. The bridging line can be a broken line connecting the adjacent endpoints of the two objects with a straight line, or a curved line that fills the gap between the endpoints of the two objects using a method such as spline interpolation to accommodate curved shapes of the target objects.

[0054] The bridge brightness value extraction unit 105 extracts the brightness values ​​of the coordinates on the line output by the object bridge unit 104 from the image input to the input unit 100 for each combination of two regions in the selected object region output by the object region detection unit 102. The extracted brightness values ​​are output as a one-dimensional vector (brightness value array) to the same object determination unit 106. The extracted brightness values ​​can be directly extracted from the pixel values ​​of the input image or extracted from an image after smoothing the input image to remove noise. In a color image, the brightness of a pixel can be calculated using a predetermined formula based on the brightness values ​​of R, G, and B pixels.

[0055] For each combination of two object regions output by the selected object region detection unit 102, the same object determination unit 106 uses the brightness values ​​along the bridge line output by the bridge brightness value extraction unit 105 as input to determine whether the two regions are the same object separated by an obstacle. This determination can be made by comparing the brightness value transitions along the line with a pre-set pattern of brightness value transitions, or by using machine learning methods such as SVMs (Support Vector Machines) or deep learning methods such as CNNs. This determination calculates a likelihood, representing the probability that the two bridged object regions originate from the same segmented object. If the calculated likelihood is above a threshold set by the user, the two object regions are determined to originate from the same object.

[0056] The same object determination unit 106 learns the model by preparing a one-dimensional vector representing the brightness value transitions on a line and multiple training data sets indicating correct answers for the same object or different objects. The correct likelihood can be, for example, 1 for the same object and 0 for different objects. The one-dimensional vector representing the brightness values ​​on the line is input into the model, and the output likelihood is compared with the correct answer. The model parameters are updated based on the comparison results. As described above, the same object determination unit 106 uses the model to calculate the likelihood of the same object and compares this value with a pre-set threshold to make a decision.

[0057] As described above, the same object determination unit 106 may also compare the brightness value transition on the line with a preset brightness value transition pattern and determine whether the two object regions are the same object based on the similarity. Figure 5 As will be described later, the transition of brightness values ​​of an obstacle and the same object divided by the obstacle can have a specific shape (pattern).

[0058] Specifically, the shadow of the obstacle is cast on the object behind it, which may cause the brightness value to shift. Figure 5The valley-like area of ​​decreased brightness shown in the image above. When a front obstacle casts a shadow on an object behind it, the brightness transition shows a sharp drop in brightness as the shadowed area approaches the obstacle. As the area moves away from the obstacle, the shadow gradually becomes lighter and the brightness increases. Therefore, if this pattern of decreasing brightness is present in the brightness transition, it can be determined that an obstacle is present.

[0059] Alternatively, one or more patterns representing the transition of brightness values ​​for the same object separated by an obstacle may be prepared in advance. If the similarity with any of these patterns exceeds a threshold, the two object regions may be determined to be parts of the same object separated by the obstacle. Alternatively, one or more patterns representing the transition of brightness values ​​for two different objects may be prepared, and the state represented by the pattern with the greatest similarity may be determined as the current state of the object.

[0060] The object region correction unit 107 corrects the object region for each combination of two selected object regions output by the object region detection unit 102 based on the determination result output by the same object determination unit 106. If the same object determination unit 106 determines that the two bridged object regions originate from the same object, both object regions are deleted and a rectangle circumscribing the two object regions is created as a new object region. The likelihood of the new object region can be the likelihood of either of the two object regions before correction, as calculated by the object region detection unit 102, or a new likelihood, such as the average of the likelihoods of the two object regions, can be calculated and used.

[0061] The output unit 108 outputs the target object region output by the object region detection unit 102, or the detection result of the target object region corrected by the object region correction unit 107, or the recognition result and likelihood of the detected object region to the outside of the device. The output format may be numerical data representing the coordinate values ​​of the target object region, the recognition result, and its likelihood, or an image drawn as a rectangle or text on the input image.

[0062] Figure 3 This is a flowchart showing an example of the processing procedure of the image processing method of Example 1. Hereinafter, each functional unit of the image processing device 1 is described as the main operating unit, but the computing device 11 may be used as the main operating unit instead, and the computing device 11 may execute each functional unit as a program.

[0063] The input unit 100 receives an image of the detection target object and inputs it to the feature extraction unit 101. (Step 200)

[0064] The feature extraction unit 101 stores information related to the feature extractor stored in the memory 12 and creates the feature extractor. Information related to the feature extractor includes, for example, the structural formula of the network and the weight coefficients of each layer in the network. Furthermore, the feature extractor is used to calculate the feature value of the input image. (Step 201)

[0065] The object region detection unit 102 calculates the likelihood of the region representing the object in the input image, the object recognition result for each region, and the probability of the recognition result based on the feature extracted by the feature extraction unit 101. The detected object region is output to the object endpoint estimation unit 103 and the object region correction unit. (Step 202)

[0066] Figure 4A 3 is a diagram showing an example of a situation in which an object 300 to be detected is segmented by occlusion caused by an obstacle 301 that may occur on the input image received in step 200. Figure 4B 3 is a diagram showing an example of an object 300 that can be output in step 202 and is detected by being divided into a region 302 and a region 303 by the above-mentioned segmentation.

[0067] The object endpoint estimation unit 103 calculates the coordinates of the two endpoints of the object in each object region detected by the object region detection unit 102. (Step 203)

[0068] Figure 4C It means targeting Figure 4B 203, and an example of the result of estimating the endpoint coordinates of each object output in step 203. Two endpoint coordinates are estimated from each object region. Therefore, endpoint coordinates 304 and 305 are estimated from object region 302, and endpoint coordinates 306 and 307 are estimated from object region 303.

[0069] The object bridging unit 104 calculates a line connecting the four endpoints calculated by the object endpoint estimating unit 103 as a set of coordinate values ​​in the input image for each of the two selected combinations of the object regions detected by the object region detecting unit 102 (step 204).

[0070] Figure 4D This diagram shows an example of line 308, output from step 204, connecting object endpoints 304 and 305 in object region 302 with object endpoints 306 and 307 in object region 303. Line 308 can be considered a trajectory passing through the center of the object, assuming that object regions 302 and 303 originate from the same object. By determining the presence or absence of occlusion along this trajectory, it is possible to determine whether object regions 302 and 303 are the same object.

[0071] The bridge brightness value extraction unit 105 extracts the brightness value on the line calculated by the object bridge unit 104 in the input image and generates a one-dimensional vector. (Step 205)

[0072] Figure 5 This diagram shows an example of brightness value transition 309 on line 308 in the input image that can be extracted in step 205. Based on brightness value transition 309, it can be estimated that, for example, an obstacle with high brightness exists at the convex portion in the center of the brightness value transition, and that the shadow of the obstacle exists because the brightness value of the obstacle decreases sharply and increases gradually from the center to the ends at the concave portions on both sides of the convex portion. In other words, the obstacle is located in front of object 300 in the input image.

[0073] Even if the brightness of the obstacle in front is low, it can be inferred from the presence of a concave portion whose brightness gradually increases from the center to the end. Furthermore, if there are multiple concave and convex shapes as described above, it can be inferred that there are multiple obstacles in front. Furthermore, even if there are concave portions on both sides of a convex portion, if the concave portions are caused by a sharp decrease in brightness and a sharp increase in brightness, it can be inferred that two other objects are casting shadows on the background, and the situation can be considered not to be a division of the same object.

[0074] The same object determination unit 106 takes the vector calculated by the bridge brightness value extraction unit 105 as input and determines whether the combination of the above object regions originates from the same object that is divided by occlusion. (Step 206)

[0075] When the object region detection unit 102 selects a combination of two object regions detected by the object region detection unit 102 and the same object determination unit 106 determines that the combination of object regions is from the same object, the object region correction unit 107 integrates the two object regions output by the object region detection unit 102 (step 207).

[0076] The above steps 204, 205, 206, and 207 are repeated for all combinations of two selected object regions.

[0077] The output unit 108 outputs the object region corrected by the object region correction unit 107 based on the determination output by the object region detection unit 102 and the object determination unit 106. (Step 208)

[0078] As described above, it is possible to provide an image processing device and method for detecting a target object from an image, thereby suppressing the detection of an object that has been divided due to occlusion from being divided into multiple parts.

[0079] Example 2

[0080] Embodiment 2 provides an image processing device and method thereof, which reduces the number of repetitions of the processing of step 204, step 205, step 206, and step 207 in embodiment 1 and improves throughput.

[0081] Figure 6 This is a block diagram showing the functional structure of the image processing device of the second embodiment. The image processing device 1 of the second embodiment mostly includes the same components as the image processing device 1 of the first embodiment (see Figure 1 ) has the same structure as that of the first embodiment, but includes an object candidate selection unit 109 as a new structure. In the following, the points repeated with the first embodiment are omitted, and the different structures are mainly described.

[0082] <Structure and Movement of Each Part>

[0083] The operations of the input unit 100, feature extraction unit 101, object region detection unit 102, object endpoint estimation unit 103, and output unit 108 are the same as those in Example 1. Therefore, the object candidate selection unit 109, object bridging unit 104, bridge brightness value extraction unit 105, identical object determination unit 106, and object region correction unit 107 will be described below.

[0084] The object candidate selection unit 109 selects one or more combinations of regions that may be occluded and divided from the object regions detected by the object region detection unit 102. Specifically, the object endpoint coordinates in each region output by the object endpoint estimation unit 103 are used to select these combinations based on their positional relationship. For example, the positional relationship between the object endpoint coordinates can be calculated by calculating the distance between the closest endpoints of two regions, or by connecting the closest endpoints of two regions with a straight line and calculating the angle between the adjacent straight lines.

[0085] The object bridging unit 104 calculates a line connecting the object endpoints of the two regions output by the object endpoint estimation unit 103 as a set of coordinate values ​​in the input image for each combination of object regions selected by the object candidate selection unit 109. The remaining operations are the same as those in the first embodiment.

[0086] The bridge luminance value extraction unit 105 extracts luminance values ​​of coordinates on the line output by the object bridging unit 104 from the image input to the input unit 100 for the combination of object regions selected by the object candidate selection unit 109. Other operations are the same as those in the first embodiment.

[0087] The same object determination unit 106 uses the brightness values ​​on the bridge lines output by the bridge brightness value extraction unit 105 as input to determine whether the combination of object regions selected by the object candidate selection unit 109 is the same object divided by the obstacle.

[0088] The object region correction unit 107 corrects the object region output by the object region detection unit 102 based on the determination result output by the identical object determination unit 106 for the combination of object regions selected by the object candidate selection unit 109 .

[0089] Figure 7 This is a flowchart showing an example of the processing sequence of the image processing method of Example 2. In the following, each functional unit of the image processing device 1 is described as the action subject, but it can also be replaced by the operation device 11 as the action subject, and the operation device 11 executes each functional unit as a program. In addition, the image processing method of Example 2 mostly includes the image processing method of Example 1 (see Figure 3 ) The same steps as above, but including step 209 as a new step. Figure 7 The processing steps shown are omitted, and the parts that are repeated in Example 1 are explained.

[0090] The processing of step 200, step 201, step 202, and step 208 in embodiment 2 is the same as the image processing method in embodiment 1.

[0091] The object endpoint estimation unit 103 calculates the coordinates of the two endpoints of the object in each object region detected by the object region detection unit 102 and outputs them to the object candidate selection unit 109. (Step 203)

[0092] The object candidate selection unit 109 calculates the positional relationship of the endpoint coordinates of each object calculated by the object endpoint estimation unit 103 for each pair of two selected object regions detected by the object region detection unit 102. The positional relationship of the object endpoint coordinates refers to the distance between the closest endpoint coordinates of the two object regions, the angle formed by adjacent straight lines connecting the closest endpoints of the two regions, and the like. (Step 209)

[0093] The object candidate selection unit 109 then selects a candidate combination of two object regions that are likely to be from the same object based on the positional relationship of the endpoint coordinates. For example, if the closest endpoint coordinates between the two object regions are a certain distance apart, it can be determined that the two object regions cannot be from the same object due to the distance. Alternatively, if the straight lines intersect at an angle below a certain angle, it can be determined that the two object regions cannot be from the same object because the object cannot be extremely curved.

[0094] (Step 210)

[0095] Figures 8A to 8D 209 is a diagram showing an example of the positional relationship of the object endpoint coordinates calculated in step 209. Figure 8AIn this example, the distance 402 between the endpoint coordinates of two objects approaching each other is short. Figure 8B This shows an example where distance 405 is long. In step 210, for example, if the threshold value of the distance between object endpoints, which is preset by the user based on the target object, is longer than distance 402 and shorter than distance 405, the combination of regions 400 and 401 is selected as having a high probability of originating from the same object, while the combination of regions 403 and 404 is not selected as having a low probability of originating from the same object, thereby filtering the combinations of object regions to be output to subsequent steps.

[0096] in addition, Figure 8C This shows an example where the angle 408 formed by the adjacent straight lines when the adjacent end points of two objects are connected by straight lines is large. Figure 8D This shows an example where angle 411 is small. For example, in step 210, if the threshold value for the angle between straight lines connecting the endpoints of the object, which is pre-set by the user based on the target object, is less than angle 408 and greater than angle 411, the combination of regions 406 and 407 is selected as having a high probability of originating from the same object, and the combination of regions 409 and 410 is not selected as having a low probability of originating from the same object, thereby filtering the combinations of object regions to be output to subsequent steps.

[0097] The object bridging unit 104, the bridge brightness value extraction unit 105, the same object determination unit 106, and the object area correction unit 107 perform bridging between objects (step 204), acquisition of the brightness value of the bridging part (step 205), determination that the two object areas are from the same object (step 206), and correction of the object area (step 207) in the same manner as in Example 1 for each combination of two object areas selected by the object candidate selection unit 109 as possibly coming from the same object.

[0098] Regarding the above-mentioned steps 204 , 205 , 206 , and 207 , these steps are repeated for all combinations of two object regions output by the object candidate selection unit.

[0099] In summary, an image processing device and method can be provided that can reduce the number of bridging between object areas in Example 1 (step 204), extraction of the brightness value of the bridging part (step 205), determination that two object areas are from the same object (step 206), and correction of the object area (step 207), thereby improving throughput.

[0100] Example 3

[0101] Embodiment 3 provides the image processing device and method for improving the accuracy of determining object regions from the same object in embodiment 1.

[0102] Figure 91 is a block diagram showing an example of the functional configuration of an image processing apparatus in accordance with a second embodiment of the present invention. The image processing apparatus 1 in accordance with the third embodiment includes many features similar to those of the image processing apparatus 1 in accordance with the first embodiment (see Figure 1 ) has the same structure as that of the first embodiment, but includes as a new structure either or both of the object shape extraction unit 110 and the object brightness value extraction unit 111. Hereinafter, the points repeated with the first embodiment will be omitted, and the different structures will be described.

[0103] <Structure and Movement of Each Part>

[0104] The operations of the input unit 100, feature value extraction unit 101, object region detection unit 102, object endpoint estimation unit 103, object bridging unit 104, bridge brightness value extraction unit 105, object region correction unit 107, and output unit 108 are the same as those in Example 1. Therefore, the object shape extraction unit 110, object brightness value extraction unit 111, and identical object determination unit 106 will be described below.

[0105] The object shape extraction unit 110 extracts features derived from the object's shape from each object region detected by the object region detection unit 102. Specifically, these features include the average and shift of the object's width, the degree of convexity and concavity of the object's contour, and so on. The degree of convexity and concavity indicates contour roughness and can be represented, for example, by the ratio of the contour length to the contour average. These shape features can be calculated by extracting the detailed shape of the object through image processing, or by inferring these features derived from the object's shape using deep learning or other methods, taking an image of the object region as input.

[0106] The object brightness value extraction unit 111 extracts features derived from the brightness values ​​of the objects in each object region detected by the object region detection unit 102. Specifically, features derived from the brightness values ​​of the objects include, for example, the average brightness value in each object region and the shift in brightness values ​​between object ends. These object brightness value features can be calculated based on the object regions and object endpoints, or using detailed object shapes obtained through image processing, deep learning, or the like.

[0107] The same object determination unit 106 determines whether the object regions selected by the object candidate selection unit 109 are the same object divided by an obstacle. In this determination, the presence of occlusion between the object regions is determined based on the brightness values ​​of the bridge lines output by the bridge brightness value extraction unit 105.

[0108] Furthermore, based on either or both of the shape features of the object output by the object shape extraction unit 110 and the brightness value features of the object output by the object brightness value extraction unit 111, the two object regions are determined to be from the same object, taking into account the similarity and continuity of the features in the two objects. When the similarity of the features is higher than a threshold value, or when the continuity of the features is higher than a threshold value, the two object regions are determined to represent parts of the same object. Similarity can be calculated, for example, by difference or cosine similarity. Continuity can also be calculated, for example, based on the difference in features at adjacent endpoints. As described above, the same object determination unit 106 determines that two objects are different objects based on the similarity and continuity of the features of the two objects, even when occlusion caused by an obstacle exists in a manner that covers different objects.

[0109] Figure 10 This is a flowchart showing an example of the processing sequence of the image processing method of Example 3. In the following, each functional unit of the image processing device 1 is described as the action subject, but it can also be replaced by the operation device 11 as the action subject, and the operation device 11 executes each functional unit as a program. In addition, the image processing method of Example 3 mostly includes the image processing method of Example 1 (see Figure 3 ) The same steps as above, but as new steps, including any one or both of step 210 and step 211. Figure 7 The processing sequence shown omits the parts that are repeated in Example 1 and explains the different steps.

[0110] The processing of step 200, step 201, step 202, step 203, step 204, step 205, step 207, and step 208 in embodiment 3 is the same as the image processing method in embodiment 1.

[0111] The object shape extraction unit 110 extracts a feature quantity derived from the shape of each object region output by the object region detection unit 102. (Step 211)

[0112] The object brightness value extraction unit 111 extracts a feature quantity derived from the brightness value of each object region output by the object region detection unit 102. (Step 212)

[0113] The same object determination unit 106 receives as input the vector calculated by the bridge brightness value extraction unit 105, the feature quantity derived from the object shape output by the object shape extraction unit 110, and the feature quantity derived from the object brightness value output by the object brightness value extraction unit 111, and determines whether the combination of object regions is derived from the same object that was divided by occlusion. (Step 206)

[0114] In summary, an image processing apparatus and method according to the first embodiment can be provided, which improve the accuracy of determining object regions originating from the same object that is divided by occlusion.

[0115] Example 4

[0116] The fourth embodiment is an image processing system that uses the image processing apparatus described in the first embodiment to detect a target object and correct the object region detected by occlusion and segmentation.

[0117] Figure 11 This is a diagram showing the hardware configuration of an image processing system in accordance with a fourth embodiment of the present invention.

[0118] The image processing system 1000 includes an imaging device 1001 , an image processing device 1 , a storage device 1002 , and a display device 1003 .

[0119] The imaging device 1001 is a device for capturing an image of a detection target object, and is, for example, a camera, a microscope, or the like.

[0120] The image processing device 1 is the image processing device described in the first embodiment, and calculates one or more of the region of the target object, the recognition result for each detected region, and its likelihood from the image captured by the imaging device 1001 .

[0121] The storage device 1002 stores information related to object detection that is preset by the user. Specifically, the information related to object detection includes information about the detection target object, such as the object name, and a threshold value set for the likelihood output by the image processing device 1.

[0122] In summary, it is possible to provide an image processing system that can detect a target object in an image as a single object even if the object is divided by occlusion.

[0123] Furthermore, the present invention is not limited to the above-described embodiments and encompasses various variations. For example, the above-described embodiments are examples described in detail to facilitate understanding of the present invention and are not necessarily limited to having all of the described structures. Furthermore, a portion of the structure of one embodiment can be replaced with a structure of another embodiment, and a structure of another embodiment can be added to a structure of one embodiment. Furthermore, with respect to a portion of the structure of each embodiment, other structures can be added, deleted, or substituted.

[0124] Furthermore, the aforementioned structures, functions, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them using integrated circuits. Furthermore, the aforementioned structures, functions, etc. may be implemented in software by having a processor interpret and execute programs that implement the respective functions. Information such as programs, tables, and files that implement the respective functions may be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card.

[0125] In addition, the control lines and information lines are those considered necessary for explanation and do not necessarily represent all the control lines and information lines on the product. In fact, it can be assumed that almost all the components are connected to each other.

Claims

1. An image processing device for detecting a specific object in an image, characterized in that: The image processing device comprises: an input unit that receives an input image; a feature quantity extraction unit that extracts a feature quantity from the input image; an object region detecting unit that detects an object region in the input image using the feature amount; an object endpoint estimating unit for estimating the coordinates of the endpoint of the object in the object area; an object bridging unit for calculating a trajectory for complementing the endpoints of two object regions on the input image; a same object determination unit configured to determine whether the two object regions are included in the same object based on the brightness value transition on the track; an object region correction unit that corrects the object region output by the object region detection unit based on the determination result output by the same object determination unit; and An output unit outputs the corrected object region.

2. The image processing device according to claim 1, wherein The object bridging unit complements the endpoints of the two object regions by spline interpolation, and calculates a curved track for bridging the two object regions.

3. The image processing device according to claim 1, wherein The image processing device includes an object candidate selection unit that selects a combination of two object regions output to the object bridging unit from among the object regions output by the object region detection unit based on a positional relationship between coordinates of the end points.

4. The image processing device according to claim 1, wherein The image processing device includes an object shape extraction unit that extracts a feature value of an object shape for each object region output by the object region detection unit, and uses the extracted feature value of the object shape as an input to the same-object determination unit.

5. The image processing device according to claim 1, wherein The image processing device includes an object brightness value extraction unit that extracts a feature value of a brightness value of an object for each object region output by the object region detection unit, and uses the extracted feature value of the brightness value of the object as an input to the same-object determination unit.

6. The image processing device according to claim 1, wherein The same object determination unit uses a model to determine whether the two object regions are included in the same object. The input of the model includes the brightness value transitions on the track, The model calculates a likelihood representing the probability that the two object regions are contained in the same object. The model is learned using a learning dataset that includes a combination of brightness value transitions and likelihoods on a track.

7. The image processing device according to claim 1, wherein The object endpoint estimating unit estimates the coordinates of the endpoint of the object in the object area using a model. The input of the model includes an image of the detected object area, The model outputs the coordinates of the endpoints of the input object area, The model is learned using a learning dataset comprising a combination of images of object regions and coordinates of endpoints of the object regions.

8. An image processing method, wherein a device detects a specific object in an image, characterized in that: The method comprises: The apparatus detects object regions from an input image; The device extracts the brightness value shift between the two detected object areas; The device determines whether the two object regions are included in the same object based on the brightness value transition; and The device corrects the object area according to the result of the determination.

9. The image processing method according to claim 8, characterized in that: The extraction of the brightness value shift includes: For each detected object area, calculating the coordinates of the endpoints of the object; For each combination of two object regions, interpolating between the endpoints to calculate a trajectory; and The brightness value transition on the track is extracted from the input image.

10. The image processing method according to claim 9, wherein: In the extraction of the brightness value transition, a combination of two object regions for which the trajectory is calculated is narrowed down based on the positional relationship between the end points of the object region.

11. The image processing method according to claim 8, wherein: The image processing method further includes: extracting a feature value of the shape and / or a feature value of the brightness value of the object for each detected object area, Whether the two object regions are included in the same object is determined by using not only the brightness value transition but also the shape feature and / or brightness value feature.

12. An image processing system for detecting a specific object in an image, characterized in that: The image processing system comprises: a photographing device for photographing the sample; The image processing device according to claim 1; a storage device that holds information used by the image processing device; and A display device displays the image processing result output by the image processing device.

13. The image processing system according to claim 12, wherein: The image processing device acquires the information stored in the storage device and creates a model for processing.

14. The image processing system according to claim 12, wherein: When an image is input from the imaging device, the image processing device transmits a detection result of a target object in the input image to a display device. The display device outputs a detection result of the target object.

Images