Image processing method, image processing apparatus, and computer program product

Abstract

An image processing method for extracting a target object from an image includes: obtaining, by using an image segmentation model, a first segmentation mask of the target object in a target region covering the image and a segmentation probability map representing a segmentation probability of each pixel in the target region belonging to the target object; detecting, by using a contour detection model, at least one image contour in the target region; determining, for the detected at least one image contour, whether a pixel inside each image contour belongs to the target object based on the segmentation probability map to obtain a second segmentation mask representing pixels belonging to the target object; and extracting the target object based on the first segmentation mask and the second segmentation mask.

Description

Technical Field

[0001] This disclosure relates to the field of computer image processing, and more specifically to image processing methods, image processing apparatuses, and computer program products for extracting target objects from images. Background Technology

[0002] In computer image processing, there are image segmentation techniques that separate and extract target objects such as text, objects, and people from the image background. Existing image segmentation techniques, such as those using deep learning-based image segmentation models, determine the probability that each pixel in the image belongs to a target object in order to extract the target object. However, existing image segmentation techniques sometimes fail to accurately distinguish target objects from the image background, especially when background residue may remain near the edges of the segmented target object. Summary of the Invention

[0003] In view of the above problems, this disclosure provides an image processing method, image processing apparatus and computer program product for extracting target objects from images, which can extract target objects from images more accurately.

[0004] According to one aspect of this disclosure, an image processing method for extracting a target object from an image is provided, comprising: using an image segmentation model to obtain a first segmentation mask of the target object in a target region covering the image, and a segmentation probability map representing the segmentation probability that each pixel in the target region belongs to the target object; using a contour detection model to detect at least one image contour in the target region; for the detected at least one image contour, determining whether a pixel inside each image contour belongs to the target object based on the segmentation probability map, so as to obtain a second segmentation mask representing pixels belonging to the target object; and extracting the target object based on the first segmentation mask and the second segmentation mask.

[0005] According to one embodiment of this disclosure, determining whether a pixel inside each image contour belongs to a target object based on a segmentation probability map includes: determining the inward trend of the segmentation probability of pixels in the inward direction from each image contour and the outward trend of the segmentation probability of pixels in the outward direction from each image contour; and determining whether a pixel inside each image contour belongs to a target object based on the inward and outward trends of each image contour.

[0006] According to one embodiment of this disclosure, determining the inward and outward variation trends of each image contour includes: for each image contour, reducing the size by a first step length representing the degree of reduction in the inward direction by a first number, resulting in a first number of reduced contours equal to the first number, wherein the first number and the first quantity are greater than 1; and expanding the size by a second step length representing the degree of expansion in the outward direction by a second number, resulting in a second number of expanded contours equal to the second number, wherein the second number and the second quantity are greater than 1; and determining the inward variation trend of each image contour based on each image contour and its first number of reduced contours, and determining the outward variation trend of each image contour based on each image contour and its second number of expanded contours.

[0007] According to one embodiment of this disclosure, determining an inward variation trend of each image contour based on each image contour and a first number of reduced contours, and determining an outward variation trend of each image contour based on each image contour and a second number of expanded contours, includes: for each image contour, calculating the average value of the segmentation probability of all pixels in each region enclosed by every two adjacent contour lines in the contour lines including the image contour and its first number of reduced contours and the second number of expanded contours; and determining an inward variation trend based on the variation trend of multiple average values ​​in the inward direction, and determining an outward variation trend based on the variation trend of multiple average values ​​in the outward direction.

[0008] According to another embodiment of this disclosure, determining an inward variation trend of each image contour based on each image contour and a first number of reduced contours, and determining an outward variation trend of each image contour based on each image contour and a second number of expanded contours, includes: for each image contour, calculating the average value of the segmentation probability of all pixels on the image contour; for the first number of reduced contours, calculating the average value of the segmentation probability of all pixels on each reduced contour; for the second number of expanded contours, calculating the average value of the segmentation probability of all pixels on each expanded contour; and determining an inward variation trend based on the variation trend of multiple average values ​​in the inward direction, and determining an outward variation trend based on the variation trend of multiple average values ​​in the outward direction.

[0009] According to one embodiment of this disclosure, determining whether a pixel inside each image contour belongs to a target object is based on predefined rules. The predefined rules are as follows: for each image contour, if the inward trend is increasing, the pixel inside the image contour is determined to belong to the target object; if the inward trend is decreasing, the pixel inside the image contour is determined not to belong to the target object; and if the inward trend is unchanged, if the outward trend is increasing, the pixel inside the image contour is determined not to belong to the target object, and if the outward trend is decreasing, the pixel inside the image contour is determined to belong to the target object.

[0010] According to one embodiment of this disclosure, when each image contour is reduced in the inward direction, if any reduced contour of the image contour overlaps with other image contours detected by the contour detection model, the first number and / or the first step length are reduced so that the reduced contour does not overlap with other image contours; and when each image contour is expanded in the outward direction, if any expanded contour of the image contour overlaps with other image contours detected by the contour detection model, the second number and / or the second step length are reduced so that the expanded contour does not overlap with other image contours.

[0011] According to one embodiment of this disclosure, the target object is extracted based on the result of the AND operation between the first segmentation mask and the second segmentation mask.

[0012] According to one embodiment of this disclosure, if the number of at least one image contour is greater than 2 and there are any two nested image contours, then the pixels inside each image contour are determined to belong to the target object in the order from the outer contour to the inner contour.

[0013] According to another aspect of this disclosure, an image processing apparatus for extracting a target object from an image is provided, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor performs the image processing method described above.

[0014] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, performs the image processing method described above.

[0015] Based on the image processing method, image processing apparatus, and computer program product provided in this disclosure, it is possible to extract target objects from images more accurately. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an image processing method 100 according to an embodiment of the present disclosure.

[0017] Figure 2 This is an example diagram illustrating an image processing method 100 according to an embodiment of the present disclosure.

[0018] Figure 3 Yes Figure 2 An example image showing a magnified portion of the segmentation probability map.

[0019] Figure 4 This is an example diagram illustrating predefined rules for determining whether pixels inside an image outline belong to a target object.

[0020] Figure 5This is a schematic block diagram illustrating an image processing apparatus 500 according to an embodiment of the present disclosure. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include other steps and / or omit certain steps.

[0023] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0024] It should be understood that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules, units, models, data, etc., and are not used to limit the order of the functions performed by these devices, modules, units, models, data, or their interdependencies.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0026] To address the issue of residual image background near the edges of target objects, this disclosure proposes to further utilize contour detection technology on top of the image segmentation model to detect contours in the target region of the image. Based on the segmentation probability map generated by the image segmentation model, it is determined whether the pixels inside each contour belong to the target object, thereby improving the accuracy of extracting target objects from the image.

[0027] The following is for reference Figure 1 and Figure 2 The image processing method 100 according to an embodiment of the present disclosure will be described in detail. Figure 1 This is a flowchart illustrating an image processing method 100 according to an embodiment of the present disclosure. Figure 2This is an example diagram illustrating an image processing method 100 according to an embodiment of the present disclosure.

[0028] Image processing method 100, for example, is an image processing method that interactively receives a target region specified by a user in an image and automatically extracts a target object from the target region. The user can select (e.g., click, draw a box, etc.) a target region in the original image, including the target object to be extracted, using an input device such as a mouse or touchscreen. Image processing method 100 extracts the target object from the user-selected target region and displays the extracted target object on a display device such as a monitor or touchscreen. Figure 2 In the scenario shown, suppose the user wants to extract the text "100 million yen" from the original image and selects the target region indicated by the black solid line box. The target region is then input into the image segmentation model 210 and the contour detection model 220, respectively.

[0029] like Figure 1 As shown, the image processing method 100 may include steps S110 to S140.

[0030] In step S110 of image processing method 100, image segmentation model 210 is used to obtain a first segmentation mask of the target object in the target region covering the image, and a segmentation probability map representing the segmentation probability that each pixel in the target region belongs to the target object.

[0031] Here, the image segmentation model 210 can calculate the probability [0,1] (hereinafter referred to as segmentation probability) of each pixel in the target region belonging to the target object. For example, pixels with a segmentation probability of 0.5 or higher than a pre-set threshold are determined to belong to the target object, and pixels with a segmentation probability lower than the pre-set threshold are determined to not belong to the target object, so as to obtain a first segmentation mask covering the target object in the target region of the image. Here, the first segmentation mask is a binary image with the same size (number of pixels) as the target region, and the pixel value of each pixel is "0" or "1". In the first segmentation mask, a pixel with a value of "0" indicates that the segmentation probability of the pixel at the corresponding position in the target region is lower than the pre-set threshold (not belonging to the target object), and a pixel with a value of "1" indicates that the segmentation probability of the pixel at the corresponding position in the target region is higher than the pre-set threshold (belonging to the target object). Figure 2 In the first segmentation mask shown, white represents pixels with a value of "1" and black represents pixels with a value of "0".

[0032] The segmentation probability map is a graph composed of the segmentation probabilities of each pixel in the target region, calculated by the image segmentation model 210. The size (number of pixels) of the segmentation probability map is the same as the target region, and the value of each pixel in the segmentation probability map represents the segmentation probability of the corresponding pixel in the target region. Figure 2In the shown segmentation probability map, the darker the pixel color (the larger the grayscale value), the greater the segmentation probability of the pixel at the corresponding position in the target area, and vice versa, the smaller the segmentation probability of the pixel at the corresponding position in the target area. Here, the segmentation probability map is not limited to representing the magnitude of the segmentation probability by grayscale values, and different colors can also be used to represent the magnitude of the segmentation probability.

[0033] The image segmentation model 210 can achieve calculating the segmentation probability for each pixel in the target area and generating a segmentation mask. Various existing neural network models can be used to implement the above processing, such as the "Faster-R-CNN" model, the "Mask-R-CNN" model, etc.

[0034] In step S120 of the image processing method 100, at least one image contour in the target area is detected by using the contour detection model 220. The image contour here includes the contour of the target object. Figure 2 In the shown "image contour", each closed white line represents an image contour detected by the contour detection model 220. For example, for the character "1", the contour detection model 220 detects an image contour similar to an inverted "L" shape. For the character "円", the contour detection model 220 detects three image contours, namely two image contours in the shape of "□" (refer to the image contours C and D in Figure 3 ), and one image contour surrounding them (refer to the image contour E in Figure 3 ). The contour detection model 220 can be implemented by using a variety of existing methods. Traditional edge detection operators such as the Canny operator can be used, or neural network-based models such as "DiffusionEdge" and "DeepEdge" can also be used.

[0035] In step S130 of the image processing method 100, for the at least one detected image contour, it is determined whether the pixels inside each image contour belong to the target object based on the segmentation probability map, so as to obtain a second segmentation mask representing the pixels belonging to the target object.

[0036] In other words, in the image processing method 100 of the present disclosure, the second segmentation mask is obtained based on the segmentation probability map formed by the segmentation probabilities calculated by the image segmentation model 210 and the image contours detected by the contour detection model 220, and is used to extract the target object from the target region together with the above-mentioned first segmentation mask. Similar to the above-mentioned first segmentation mask, the second segmentation mask is also a binary image having the same size (number of pixels) as the target region, and the pixel value of each pixel is "0" or "1". For example, a pixel with a value of "0" in the second segmentation mask indicates that the pixel at the corresponding position in the target region is determined not to belong to the target object, and a pixel with a value of "1" indicates that the pixel at the corresponding position in the target region is determined to belong to the target object. In Figure 2 the second segmentation mask shown, pixels with a value of "1" are represented by white, and pixels with a value of "0" are represented by black. The specific method for determining whether the pixels inside each image contour belong to the target object based on the segmentation probability map will be described later.

[0037] After that, in step S140 of the image processing method 100, the target object is extracted based on the first segmentation mask and the second segmentation mask.

[0038] That is, in the processing of step S140, the processing results of both the image segmentation model 110 and the contour detection model 220 are combined, and the first segmentation mask and the second segmentation mask are merged (for example, an AND operation) to obtain a merged mask, and each pixel in the target region is filtered, and only the pixels corresponding to the positions with a value of "1" in the merged mask are retained, so that an image extraction result can be obtained. In Figure 2 the image extraction result shown, in order to easily display the extracted white text, a background represented by a lighter gray is added to the image extraction result. Actually, the background region of the lighter gray can be transparent.

[0039] As Figure 2 shown, the image processing method 100 according to the embodiment of the present disclosure can more accurately determine whether the pixels inside multiple small contours in the text "億円" belong to the target object, so that the pixels inside the small contours can be accurately included or not included in the image extraction result. Therefore, the target object can be extracted more accurately.

[0040] The following describes the specific method for determining whether the pixels inside each image contour belong to the target object based on the segmentation probability map. In the following description, one of the multiple image contours detected by the contour detection model 220 is used as an illustrative object, and the specific method for determining whether the pixels inside the image contour belong to the target object is described. The same processing is performed for other contours detected by the contour detection model 220. That is, the processing described below can be performed separately for all the contours detected by the contour detection model 220.

[0041] Here, if the number of image contours detected by the contour detection model 220 is greater than 2 and any two image contours are nested (that is, among the image contours detected by the contour detection model​​​​​​​​​​​​​​​​​​​​​​​​​​​exist Figure 3 In the example on the right, the direction from the image outline C inwards is the direction from the rectangle enclosed by the white solid line toward the inside of the rectangle; conversely, the direction from the image outline outwards is the direction from the rectangle enclosed by the white solid line toward the outside of the rectangle.

[0045] By using the segmentation probability map, it is easy to grasp the inward trend of the segmentation probability of pixels in the direction inward from each image contour, and the outward trend of the segmentation probability of pixels in the direction outward from each image contour. Furthermore, by determining whether the pixels inside each image contour belong to the target object based on the inward and outward trends of each image contour, the problem of designing universally applicable rules can be solved. At the same time, it can avoid designing a fixed threshold to crudely filter the segmentation probability of pixels, thereby avoiding the impact of improper threshold design on the extraction of target objects, and further improving the versatility of the image processing method 200.

[0046] More specifically, in one embodiment, determining the inward and outward variation trends of each image contour includes: for each image contour, reducing the size by a first step length representing the degree of reduction in the inward direction by a first number, resulting in a first number of reduced contours equal to the first number, wherein the first number and the first quantity are greater than 1; and expanding the size by a second step length representing the degree of expansion in the outward direction by a second number, resulting in a second number of expanded contours equal to the second number, wherein the second number and the second quantity are greater than 1; and determining the inward variation trend of each image contour based on each image contour and its first number of reduced contours, and determining the outward variation trend of each image contour based on each image contour and its second number of expanded contours.

[0047] Still using Figure 3 The example is used to illustrate this. Figure 3 The diagram exemplarily illustrates two reduced contours I1 and I2 obtained by scaling down the image contour C inwards twice (i.e., the first reduction is 2) with a 5% reduction ratio each time (i.e., the first step length is 5%), centered on the centroid (not shown) of the image contour C. It also illustrates two expanded contours O1 and O2 obtained by scaling down the image contour C outwards twice (i.e., the second expansion is 2) with a 5% expansion ratio each time (i.e., the second step length is 5%). Figure 3 In the diagram, the reduced outlines I1 and I2, and the expanded outlines O1 and O2 are represented by white dashed boxes. In the following description, the image outline C, as well as its reduced outlines I1 and I2, and expanded outlines O1 and O2, are sometimes collectively referred to as the "outline".

[0048] The image contour C can be enlarged and reduced using a variety of existing methods.

[0049] For example, for Figure 3 The rectangular (polygonal) image contour C shown can be proportionally reduced or expanded around its centroid as follows: Let the centroid coordinates of image contour C be (x0, y0), and the coordinates of a vertex of image contour C be (x, y). Let the first step length be λ1 and the second step length be λ2. Then, the coordinates (x1, y1) of the corresponding vertex after reduction can be expressed as (x1 = (1-λ1)*(x-x0)+x0, y1 = (1-λ1)*(y-y0)+y0). The coordinates (x2, y2) of the corresponding vertex after expansion can be expressed as (x2 = (1+λ2)*(x-x0)+x0, y2 = (1+λ2)*(y-y0)+y0). The same calculation is performed on each vertex of image contour C to obtain the coordinates of each corresponding vertex after reduction or expansion. Connecting the coordinates of each vertex after reduction or expansion yields the reduced or expanded contour.

[0050] Alternatively, each vertex of the image contour C can be moved inward / outward by a certain number of pixels along the centroid direction (in this case, the first step length / second step length is in pixels), and the vertices after being moved inward / outward can be connected to obtain the reduced / enlarged contour of the image contour C.

[0051] Alternatively, for an image contour C of an arc, the pixels can be moved inward / outward by a pixel distance of a first step / second step (in this case, the first step / second step is in pixels) along the direction perpendicular to the tangent at each pixel of the image contour C, and the pixels after being moved inward / outward can be connected to obtain a shrinking contour / enlarging contour.

[0052] Alternatively, a morphological method similar to image erosion / dilation can be used. A rectangular operator of appropriate size is defined based on the first / second step length, and operations are performed on each pixel of the image contour C to obtain a reduced / enlarged contour of image contour C. In this case, the first / second step length is in pixels, and by changing the size of the rectangular operator multiple times, multiple reductions / enlargements with different first / second step lengths can be achieved.

[0053] The methods for shrinking / enlarging image contours are not limited to those described above. Furthermore, existing libraries such as OpenCV and the Python-based pyclipper library already include functions for shrinking / enlarging image contours. Therefore, those skilled in the art can employ any suitable method to achieve the shrinking / enlarging of image contours.

[0054] Furthermore, in the above example, the first number (first quantity) and the second number (second quantity) are both 2, but this disclosure is not limited to this, and any other appropriate number / quantity can be used. Also, the first number (first quantity) and the second number (second quantity) can be equal or unequal. Additionally, as mentioned above, the first step length and the second step length can be in units of proportion or pixels, or other appropriate units can be used depending on the actual method of enlarging / shrinking the image contour. Furthermore, the first step length and the second step length can be equal or unequal. Furthermore, in the above example, the shrinking contours I1, I2 and the enlarging contours O1, O2 are obtained by shrinking or enlarging around the centroid of the image contour C. However, the image processing method 100 of this disclosure describes each shrinking and enlarging operation around different centers, and each enlargement or shrinking operation can not be proportional. For example, multiple enlarging contours can be obtained by enlarging the circumcircle / circumcircle rectangle of the image contour by multiple different proportions, and multiple shrinking contours can be obtained by shrinking the incircle / incircle rectangle of the image contour by multiple different proportions. As long as the image contour C does not overlap with its various reduced and expanded contours, each reduced contour is surrounded by the image contour, and each expanded contour surrounds the image contour.

[0055] In addition, in one embodiment of this disclosure, when each image contour C is reduced in the inward direction, if any reduced contour of the image contour overlaps (partially or completely overlaps) with other image contours detected by the contour detection model 220, the first number and / or the first step length are reduced so that the reduced contour does not overlap with other image contours; and when each image contour C is expanded in the outward direction, if any expanded contour of the image contour overlaps with other image contours detected by the contour detection model 220, the second number and / or the second step length are reduced so that the expanded contour does not overlap with other image contours.

[0056] Here, "overlap" can refer to any reduced contour of an image contour having at least one intersection point with other image contours detected by contour detection model 220. For example, when expanding image contour C according to a specific second step size and / or second number, the leftmost edge of a certain expanded contour of the rectangular image contour C may intersect with image contour D (reference). Figure 3The rightmost edge overlaps. At this point, it is ensured that no shrinking or expanding contour of image contour C overlaps with other image contours. For example, imagine shrinking (or expanding) image contour C 5 times (i.e., the first number / second number is 5) with a first step length (or second step length) of 10 pixels. In this case, the 4th shrinking (or expanding) contour overlaps with other image contours. In this situation, the first step length (second step length) of 10 pixels can be reduced to, for example, 5 pixels, and the image contour C can be shrunk (or expanded) 5 times again (i.e., the first number / second number remains the same), so that the shrunk (or expanded) contour does not overlap with other image contours. Alternatively, when an overlapping shrunk (or expanded) contour occurs, the shrinking (or expanding) of image contour C should be stopped after the previous shrinking (or expanding) that occurs, which in this example is the 3rd shrinking (or expanding). In other words, the first number / second number can be reduced to 3, while keeping the first step length (or second step length) unchanged at 10 pixels. Alternatively, both the first step length (or second step length) and the first number (or second number) can be reduced simultaneously, ensuring that the shrinking (or expanding) contour does not overlap with other image contours. This processing allows for a more accurate determination of the inward and outward variation trends of each image contour.

[0057] In one implementation, determining the inward variation trend of each image contour based on each image contour and its first number of reduced contours, and determining the outward variation trend of each image contour based on each image contour and its second number of expanded contours, includes: for each image contour, calculating the average value of the segmentation probability of all pixels in each region enclosed by every two adjacent contour lines, including the image contour and its first number of reduced contours and its second number of expanded contours; and determining the inward variation trend based on the variation trend of multiple average values ​​in the inward direction, and determining the outward variation trend based on the variation trend of multiple average values ​​in the outward direction.

[0058] Back Figure 3For example, in this implementation, after obtaining a first number of reduced contours I1, I2 and a second number of expanded contours O1, O2 of the image contour C, the inward trend of the image contour C can be determined as follows: calculate the average segmentation probability of all pixels in the region enclosed by reduced contour I1 and image contour C, for example, 0.6, and calculate the average segmentation probability of all pixels in the region enclosed by reduced contour I2 and reduced contour I1, for example, 0.5. At this time, from image contour C inward, the average segmentation probability decreases from 0.6 to 0.5, that is, the inward trend of the image contour C shows a decreasing trend. Similarly, the outward trend of the image contour C can be determined as follows: calculate the average segmentation probability of all pixels in the region enclosed by expanded contour O1 and image contour C, for example, 0.55, and calculate the average segmentation probability of all pixels in the region enclosed by expanded contour O2 and expanded contour O1, for example, 0.6. At this point, starting from image contour C and moving outwards, the average segmentation probability of pixels increases from 0.55 to 0.6, meaning that the outward trend of image contour C shows an increasing trend.

[0059] Based on this implementation method, the inward and outward variation trends of the image contour can be calculated based on the pixels within the area enclosed by two adjacent contour lines, resulting in more accurate inward and outward variation trends.

[0060] Alternatively, in another implementation, instead of calculating the average segmentation probability of pixels within the region enclosed by two adjacent contour lines, the inward and outward variation trends of the image contours can be determined by calculating only the average segmentation probability of pixels on the contour lines. That is, in another implementation, for each image contour, the average segmentation probability of all pixels on that image contour is calculated; for a first number of reduced contours, the average segmentation probability of all pixels on each reduced contour is calculated; for a second number of expanded contours, the average segmentation probability of all pixels on each expanded contour is calculated; and the inward variation trend is determined based on the variation trend of multiple average values ​​in the inward direction, and the outward variation trend is determined based on the variation trend of multiple average values ​​in the outward direction.

[0061] Specifically, in Figure 3In the example, for image contour C, its inward trend can be determined as follows: calculate the average segmentation probability of all pixels on image contour C, for example, 0.55; calculate the average segmentation probability of all pixels on shrinking contour I1, for example, 0.5; calculate the average segmentation probability of all pixels on shrinking contour I2, for example, 0.45. At this point, starting from image contour C and moving inward, the average segmentation probability of pixels decreases in the order of 0.55, 0.5, and 0.45, meaning the inward trend of image contour C shows a decreasing trend. Similarly, for image contour C, its outward trend can be determined as follows: calculate the average segmentation probability of all pixels on image contour C, for example, 0.55; calculate the average segmentation probability of all pixels on expanding contour O1, for example, 0.57; calculate the average segmentation probability of all pixels on expanding contour O2, for example, 0.6. At this point, starting from image contour C and moving outwards, the average segmentation probability of pixels increases by 0.55, 0.57, and 0.6 respectively, indicating that the outward trend of image contour C shows an increasing trend.

[0062] Based on this implementation method, it is not necessary to calculate the average segmentation probability of all pixels within the region enclosed by adjacent contour lines. It is possible to calculate the inward and outward variation trend of the image contour based on fewer pixels, thereby reducing the amount of computation and saving processing overhead.

[0063] In the two implementation methods described above, the average value of the multiple segmentation probabilities calculated based on the image contour C and its reduced contours I1, I2 and / or enlarged contours O1, O2 may not monotonically increase or decrease, but may fluctuate between increase and decrease.

[0064] In such cases, a curve composed of the average values ​​of multiple segmentation probabilities can be fitted, and the inward and outward trends can be determined based on the fitting results. For example, the least squares method can be used for fitting. When the slope of the fitted line is greater than 0, the inward and / or outward trend is determined to be increasing; when the slope of the fitted line is less than 0, the inward and / or outward trend is determined to be decreasing; and when the slope of the fitted line is equal to 0, the inward and / or outward trend is determined to be unchanged.

[0065] Alternatively, the inward and outward trends of the image contour can be determined simply by using the rate of change between the two endpoints of the curve formed by the average of multiple segmentation probabilities—that is, the average of the segmentation probabilities corresponding to the start and end points of the curve. When the rate of change is greater than 0, the inward and / or outward trend can be determined to be increasing; when the rate of change is less than 0, the inward and / or outward trend can be determined to be decreasing; and when the rate of change is equal to 0, the inward and / or outward trend can be determined to be unchanged.

[0066] For example, in the second implementation example above, assume the average segmentation probabilities of all pixels on image contour C, reduced contour I1, and reduced contour I2 are 0.5, 0.6, and 0.4, respectively. Then, the rate of change between the starting point (i.e., the average segmentation probability corresponding to image contour C) 0.5 and the ending point (i.e., the average segmentation probability corresponding to reduced contour I2) 0.4 of the curve formed by the average segmentation probabilities in the inward direction can be calculated as: (0.4-0.5) / 0.5 = -0.2. Therefore, it can be determined that the inward trend of image contour C is decreasing.

[0067] Furthermore, for cases where the average value of the multiple segmentation probabilities calculated based on the image contour C and its reduced contours I1, I2 and / or enlarged contours O1, O2 does not change significantly—in other words, when the slope or rate of change of the aforementioned "fitted line" approaches 0—multiple intervals can be pre-defined for the slope or rate of change of the "fitted line" to determine the inward and / or outward trends of change. For example, when the slope or rate of change of the "fitted line" in the inward direction is within the first interval [-0.1, 0.1] (i.e., approaching 0), the inward trend of change of the image contour C is determined to be unchanged; when the slope or rate of change of the "fitted line" in the inward direction is within the second interval (-∞, -0.1), the inward trend of change of the image contour C is determined to be decreasing; and when the slope or rate of change of the "fitted line" in the inward direction is within the third interval (0.1, +∞), the inward trend of change of the image contour C is determined to be increasing. The outward trend can be determined similarly. This allows for a more accurate determination of the inward and outward variation trends of the image contour C.

[0068] The above describes in detail the method of determining the inward and outward variation trends by shrinking / enlarging the image contour C, but the method for determining the inward and outward variation trends is not limited to this. For example, the inward and outward variation trends of the image contour can also be determined by the following method: For an image contour detected by the contour detection model 220, a straight line is taken that runs through the interior and exterior of the image contour. Based on the segmentation probability of the pixels corresponding to at least one intersection point of the straight line with the image contour, and the segmentation probability of the pixels corresponding to a predetermined number of points on the straight line at predetermined distances in the outward direction from the intersection point, the outward variation trend of the image contour is determined; and based on the segmentation probability of the pixels corresponding to the intersection point, and the segmentation probability of the pixels corresponding to a predetermined number of points on the straight line at predetermined distances in the inward direction from the intersection point, the inward variation trend of the image contour is determined. More specifically, for example, suppose that the segmentation probability of the pixel corresponding to the intersection point P of a straight line running through the interior and exterior of an image contour is 0.5. If, along this straight line, points are taken every 2 pixels (i.e., a preset distance of 2 pixels) from the intersection point P in the outward direction of the image contour, and the segmentation probabilities of the pixels corresponding to the two points (i.e., a preset number of 2) are 0.6 and 0.7 respectively, then the outward trend of the image contour can be determined to be increasing. Furthermore, along this straight line, points are taken every 2 pixels from the intersection point P in the inward direction of the image contour, and the segmentation probabilities of the two points corresponding to the two points are 0.4 and 0.3 respectively, then the inward trend of the image contour can be determined to be decreasing. Here, similar to the "first / second step size" and "first / second number" described in the method for shrinking / expanding the image contour, if a point obtained overlaps with other image contours detected by the contour detection model 220 during the process of taking points in the inward and / or outward directions of the image contour, the preset distance and / or preset number can be reduced. In addition, if the segmentation probability of the pixels corresponding to a preset number of points is not monotonically increasing or decreasing, the same method as the "slope of the fitted line" or "rate of change" in the method of shrinking / enlarging the image contour can be introduced to determine the inward and outward trends of the image contour. Specific examples are omitted here.

[0069] Those skilled in the art should understand that the various methods and values ​​for determining the inward and outward trends described above are merely exemplary, and other suitable methods or values ​​can also be used to determine the inward and outward trends of the image contour. After determining the inward and outward trends of the image contour C, it is determined whether the pixels inside the image contour C belong to the target object based on the inward and outward trends of the image contour C. According to one embodiment of this disclosure, determining whether the pixels inside the image contour C belong to the target object can be done according to predefined rules. The predefined rules can be: for each image contour, when the inward trend is increasing, the pixels inside the image contour are determined to belong to the target object; when the inward trend is decreasing, the pixels inside the image contour are determined not to belong to the target object; and when the inward trend is unchanged, if the outward trend is increasing, the pixels inside the image contour are determined not to belong to the target object, and if the outward trend is decreasing, the pixels inside the image contour are determined to belong to the target object.

[0070] Figure 4 This is an example diagram illustrating predefined rules for determining whether pixels within an image's outline belong to a target object. In the above... Figure 3 In the example, for the image contour C, its inward trend is decreasing, and its outward trend is increasing. Therefore, according to... Figure 4 The predefined rules shown determine that pixels inside the image contour C do not belong to the target object.

[0071] Here, when both the inward and outward trends of an image contour remain unchanged, it can be considered difficult to determine whether pixels inside the image contour belong to the target object solely based on these trends. Therefore, other methods are used for determination. For example, if an image contour has an outer contour, and both its inward and outward trends remain unchanged, whether pixels inside the contour belong to the target object can depend on the determination result of the outer contour. Alternatively, the average segmentation probability of all pixels inside the image contour can be compared with a preset threshold. If the average is greater than the threshold, pixels inside the contour are determined to belong to the target object; otherwise, they are determined not to. Alternatively, whether pixels inside the contour belong to the target object can also depend on the determination result of the image segmentation model 210 on these pixels (the first segmentation mask). Furthermore, as mentioned above, "unchanged inward / outward trends" does not necessarily mean that the rate of change (or the average rate of change) of the average segmentation probability is strictly zero, but rather that it falls within a preset range close to zero.

[0072] It is important to note that Figure 4The predefined rule shown is only an example, and those skilled in the art can use any other appropriate predefined rule.

[0073] Furthermore, determining whether pixels inside image contour C belong to a target object is not limited to using predefined rules; other methods can also be employed. For example, a machine learning (or neural network) model can be pre-trained to determine whether pixels inside image contour C belong to a target object based on the inward and outward variation trends of image contour C. Those skilled in the art can use any other appropriate method to determine whether pixels inside image contour C belong to a target object.

[0074] The above describes the specific method for determining whether pixels inside each image contour belong to the target object based on the segmentation probability map.

[0075] Furthermore, for processing that may be involved in image processing method 100 but is not specifically described above, it can be assumed that it can be implemented by existing methods.

[0076] For example, in the above description, it is assumed that all image contours detected by the contour detection model 220 are closed contours. However, for cases where the image contours detected by the contour detection model 220 are not closed, the endpoints of the image contours can be connected to close them, and then the subsequent processing of the image processing method 100 can be performed.

[0077] For example, to determine whether a pixel is inside or outside a contour within a target region, the existing ray casting method can be used. This involves drawing a ray in any direction from each pixel in the target region and counting the intersections of the ray with the contour. If the number of intersections is odd, the pixel is considered outside the contour; if the number is even, the pixel is considered inside. Alternatively, algorithms such as Scan-Line Filling or Seed Filling can also be used to determine whether a pixel is inside a contour.

[0078] For example, regarding how to determine whether an image contour detected by contour detection model 220 is an inner contour or an outer contour, existing contour detection models provide methods for detecting inner or outer contours. For instance, when using the OpenCV library to implement contour detection, the detected image contours can be represented by a tree structure, where the root node of the tree structure represents the outer contour and the leaf nodes represent the inner contour.

[0079] For example, merging the first and second segmentation masks is not limited to AND operations. Different weights can be assigned to each pixel in both masks, and a weighted average can be calculated for pixel values ​​at the same location in both masks. If the weighted average exceeds a certain threshold, the corresponding pixel value in the merged mask is set to "1," otherwise it is set to "0." More specifically, if the inward and outward trends of an image contour remain constant, it can be considered difficult to determine whether pixels inside the contour belong to the target object. In this case, the weights of the corresponding positions inside the contour in the first segmentation mask can be increased, while the weights of the corresponding positions in the second segmentation mask can be decreased, making the image extraction result more dependent on the first mask. Conversely, if the inward trend of an image contour shows a significant increase or decrease, the weights of the corresponding positions inside the contour in the second segmentation mask can be increased, making the image extraction result more dependent on the second mask.

[0080] The image processing method 100 according to embodiments of the present disclosure has been described in detail above. The above description is illustrative, and those skilled in the art can implement the corresponding processing using any suitable existing method.

[0081] According to another aspect of this disclosure, an image processing apparatus 500 is also provided. Figure 5 A schematic block diagram of an image processing apparatus 500 according to an embodiment of the present disclosure is shown.

[0082] like Figure 5 As shown, the image processing apparatus 500 of this embodiment includes a processor 510 and a memory 520. The memory 520 stores a computer program.

[0083] Processor 510 is any processing device, such as a microprocessor, that operates according to a program stored in memory 520. Memory 520 is any volatile or non-volatile storage element, such as a hard disk, solid-state drive, ROM, or RAM. The program executed by processor 510, etc., is stored in memory 520. Figure 5 The image processing apparatus 500 shown can be used to implement the image processing method according to embodiments of the present disclosure.

[0084] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program that, when executed by a processor, can implement the image processing method according to embodiments of this disclosure.

[0085] The hardware computing device or its components described in this disclosure can be implemented by various suitable hardware means, including but not limited to FPGA, ASIC, SoC, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The devices, apparatuses, methods, and systems involved in this disclosure are not limited to any particular hardware architecture or configuration. Components in the disclosed devices, apparatuses, and systems can be discrete or integrated, and can be combined in different ways and / or replaced or supplemented by other components. It should be understood that the teachings of this disclosure can be implemented in various forms, including hardware, software, firmware, dedicated processors, or combinations thereof.

[0086] The block diagrams of apparatuses, devices, methods, systems, etc., involved in this disclosure are merely exemplary and are not intended to require or imply that connections, arrangements, or configurations must be made in the manner shown in the block diagrams. As those skilled in the art will recognize, these circuits, devices, apparatuses, and systems can be connected, arranged, and configured in any manner that achieves the desired purpose.

[0087] In the foregoing description, the present invention has been described based on embodiments. These embodiments are merely illustrative, and those skilled in the art should understand that the combination of constituent elements and processes of these embodiments can be modified in various ways, and such modifications are also within the scope of the present invention.

Claims

1. An image processing method for extracting a target object from an image, comprising: Using an image segmentation model, a first segmentation mask covering the target region of the image and a segmentation probability map representing the segmentation probability that each pixel in the target region belongs to the target object are obtained; Using a contour detection model, detect at least one image contour in the target region; For at least one detected image contour, determine whether the pixels inside each image contour belong to the target object based on the segmentation probability map, so as to obtain a second segmentation mask representing the pixels belonging to the target object; as well as The target object is extracted based on the first segmentation mask and the second segmentation mask.

2. The image processing method as described in claim 1, wherein, Determining whether a pixel within each image contour belongs to the target object based on the segmentation probability map includes: Determine the inward variation trend of the segmentation probability of pixels in the inward direction from each image contour, and the outward variation trend of the segmentation probability of pixels in the outward direction from each image contour; and Based on the inward and outward variation trends of each image contour, it is determined whether the pixels inside each image contour belong to the target object.

3. The image processing method as described in claim 2, wherein, Determining the inward and outward variation trends of each image contour includes: For each image contour, The first step length, representing the degree of reduction at each step, is reduced inward for the first number of times, resulting in a first number of reduced contours equal to the first number of reductions, wherein the first number and the first number are greater than 1. The expansion is increased a second time in the outward direction by a second step size representing the degree of expansion each time, resulting in an expansion profile of a second number equal to the second number, wherein the second number and the second quantity are greater than 1; and The inward variation trend of each image contour is determined based on each image contour and its first number of reduced contours, and the outward variation trend of each image contour is determined based on each image contour and its second number of expanded contours.

4. The image processing method as described in claim 3, wherein, Determining the inward variation trend of each image contour based on each image contour and its first number of reduced contours, and determining the outward variation trend of each image contour based on each image contour and its second number of expanded contours, includes: For each image contour For the region enclosed by every two adjacent contour lines, including the image contour and its first number of reduced contours and second number of expanded contours, calculate the average segmentation probability of all pixels within each region; and The inward trend is determined based on the changing trend of multiple average values ​​in the inward direction, and the outward trend is determined based on the changing trend of multiple average values ​​in the outward direction.

5. The image processing method as described in claim 3, wherein, Determining the inward variation trend of each image contour based on each image contour and its first number of reduced contours, and determining the outward variation trend of each image contour based on each image contour and its second number of expanded contours, includes: For each image contour Calculate the average segmentation probability of all pixels on the image contour; For the first number of reduced contours, calculate the average segmentation probability of all pixels on each reduced contour; For the second number of enlarged contours, calculate the average segmentation probability of all pixels on each enlarged contour; and The inward trend is determined based on the changing trend of multiple average values ​​in the inward direction, and the outward trend is determined based on the changing trend of multiple average values ​​in the outward direction.

6. The image processing method according to any one of claims 2-5, wherein, Whether a pixel within each image contour belongs to the target object is determined according to a predefined rule, which is as follows: For each image contour When the inward change trend is increasing, the pixels inside the image outline are determined to belong to the target object; When the inward trend is decreasing, it is determined that the pixels inside the image contour do not belong to the target object; as well as When the inward trend remains unchanged If the outward change trend is increasing, then the pixels inside the image contour are determined not to belong to the target object. If the outward change trend is decreasing, then the pixels inside the image outline are determined to belong to the target object.

7. The image processing method according to any one of claims 3-5, wherein, When shrinking each image contour along the inward direction, if any shrunk contour of that image contour overlaps with other image contours detected by the contour detection model, then the first number of iterations and / or the first step length are reduced so that the shrunk contour does not overlap with the other image contours; and When expanding each image contour along the outward direction, if any expanded contour of the image contour overlaps with other image contours detected by the contour detection model, the second number and / or the second step size are reduced so that the expanded contour does not overlap with the other image contours.

8. The image processing method according to any one of claims 1-5, wherein, The target object is extracted based on the result of the AND operation between the first segmentation mask and the second segmentation mask.

9. An image processing apparatus for extracting a target object from an image, comprising: processor; as well as Memory, which stores computer programs When the computer program is executed by the processor, the processor performs the image processing method as described in any one of claims 1-8.

10. A computer program product comprising a computer program that, when executed by a processor, performs the image processing method as described in any one of claims 1-8.

Images