Region-of-interest mask extraction method and device, equipment and storage medium

Through color space conversion, Schmidt orthogonal decomposition and edge detection, the problem of mask extraction in areas of interest under complex background and similar color interference is solved, and a robust, accurate and efficient mask image generation is achieved. Improved recall and interference coverage.

CN118887237APending Publication Date: 2024-11-01SANLI VIDEO FREQUENCY SCI & TECH SHENZHEN
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410932822.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-15
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

In the case of complex background interference and similar color interference, it is difficult to generate mask maps of regions of interest robustly, accurately and efficiently.

Method used

RGB and HSV images were obtained through color space conversion, color threshold segmentation and Schmidt orthogonal decomposition were performed, the minimum channel difference was calculated, edge detection and filtering were performed, and finally the binarized images were fused to generate a mask image.

Benefits of technology

It realizes the generation of mask maps of the regions of interest robustly, accurately and efficiently under complex backgrounds and similar color interference, which improves the recall rate and covers the neighborhood interference area.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887237A_ABST
    Figure CN118887237A_ABST
Patent Text Reader

Abstract

The invention discloses a region-of-interest mask extraction method, device and equipment and a storage medium, and the method comprises the steps: carrying out the color space conversion of a to-be-extracted image, obtaining two different color gamut images, carrying out the color threshold segmentation according to the color of a region of interest, and obtaining two binary images; determining two RGB (Red, Green, Blue) base vectors with relatively large angle difference with the unit color vector of the color of the region of interest, performing Schmidt orthogonalization decomposition to obtain a unit orthogonal basis, and performing coordinate transformation on the RGB color gamut image to obtain a color representation image; calculating a difference image according to the color representation image; performing edge detection on the difference image to obtain a third binary image; and performing closing operation, opening operation, geometric feature filtering and gating fusion on the three binarized images in sequence to obtain a binarized fused image, and performing morphological filtering to generate a mask image corresponding to the region of interest. According to the invention, the mask image of the region of interest can be accurately and efficiently generated.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This case is a divisional application based on the invention patent with application date of May 15, 2024, application number 202410598438.8, and name “Region of Interest Mask Extraction Method, Device, Equipment and Storage Medium” as the parent case. Technical Field

[0002] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for extracting a mask of a region of interest. Background Art

[0003] Limited by the shooting scene and image quality, people's demand for image restoration and image editing is increasing. With the rapid development of AIGC, some AI image restoration methods and editing tools based on deep learning have emerged. Compared with professional tools such as PS, this type of AI tool shows good ease of use and amazing restoration effects.

[0004] Traditional image restoration tools, such as the stamp tool in Photoshop, fill and repair local areas in a structural propagation manner based on the principle of image self-similarity.

[0005] Generally, AI repair tools consist of a front-end page and a back-end model inference framework. Users upload images to the front-end page, use a brush to draw the target area to be repaired or removed, and generate a mask corresponding to the original image position. The back-end inference model receives both the original image and the mask as input, and outputs the repaired image end-to-end.

[0006] To improve the repair effect, the most advanced AI repair tool introduces the large-scale language model LLM and the image segmentation model SAM, allowing users to roughly prompt the repair content and expected goals through text input to guide the deep learning model to complete the repair. The SAM segmentation model generates the segmented area of ​​the target object according to the user's click position, and then assists in generating the mask layer.

[0007] In addition to the auxiliary mask generation method based on the image segmentation model mentioned above, the prior art also has a solution based on traditional image processing, which is essentially the detection of geometric shapes. Geometric shape detection is used to extract ROI (Range of Interesting). In industrial application scenarios, detecting labels with special color geometric patterns can provide accurate positioning of the target object ROI for subsequent production processes, such as: AprilTag, QR code, etc.; for example, by arranging in advance or subsequently drawing special color geometric shapes, ROI can be automatically extracted and located from the captured video or image, which is convenient for subsequent applications. In actual applications, document correction, ID shooting, and file scanning also require the extraction of rectangular frame ROI.

[0008] For detecting ROI areas with fixed colors, the classic method is based on color threshold segmentation. The color threshold range is set by predefined colors, the input image is binary segmented in the specified color domain, and then morphological filtering is used to filter noise and fill pores. However, this method does not consider the geometric features of the target graphics and is only applicable to scenes with monotonous backgrounds and no obvious color interference. The advantage is that the operation speed is fast and intuitive, but the actual application requires frequent parameter adjustments and poor robustness.

[0009] AprilTag extracts possible tag regions based on the four-element detection algorithm. The specific steps are as follows: (1) Perform spatial filtering on the image and then adaptively segment the image using the threshold to cope with light-changing scenes; (2) Find all connected domains and their corresponding contour edges; (3) Segment each contour to find the convex quadrilateral contour with the minimum residual; (4) Propose the obviously abnormal quadrilaterals to obtain the detection results. This method fully considers the geometric shape of the target, but the adaptive segmentation does not fully consider the color attributes of the target, making it difficult to cope with complex interference backgrounds; and this method is only suitable for non-overlapping independent quadrilaterals, and cannot detect the situation where multiple rectangles of different sizes overlap; due to the use of multiple exhaustive search strategies, the algorithm's computational efficiency decreases. Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide a method, device, equipment and storage medium for extracting a mask of a region of interest, which can robustly, accurately and efficiently generate a mask map of the region of interest under the conditions of complex background interference and similar color interference.

[0011] In a first aspect, the present invention provides a method for extracting a mask of a region of interest, comprising:

[0012] Performing color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and performing color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted, to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut;

[0013] According to the unit color vector of the color of the region of interest, two basis vectors with a large angle difference with the unit color vector are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the color of the region of interest;

[0014] According to the color representation image, the minimum channel difference is calculated to obtain a difference image;

[0015] Performing edge detection on the differential image to obtain a third binary image;

[0016] performing filtering processing on the first binarized image, the second binarized image and the third binarized image respectively;

[0017] The first binarized image, the second binarized image and the third binarized image after filtering are fused to obtain a mask image corresponding to the region of interest.

[0018] In a second aspect, the present invention further provides a device for extracting a mask of a region of interest, comprising:

[0019] A threshold segmentation module is used to perform color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and perform color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut;

[0020] A coordinate conversion module is used to determine two basis vectors with a large angle difference from the unit color vector of the color of the region of interest from the basis vectors of the RGB color domain, perform Schmidt orthogonal decomposition, obtain a unit orthogonal basis, and perform coordinate conversion on the RGB color domain image according to the unit orthogonal basis to obtain a color representation image based on the color of the region of interest;

[0021] A difference module, used for calculating the minimum channel difference according to the color representation image to obtain a difference image;

[0022] An edge detection module, used for performing edge detection on the differential image to obtain a third binary image;

[0023] A filtering module, used for performing filtering processing on the first binarized image, the second binarized image and the third binarized image respectively;

[0024] The fusion module is used to fuse the first binary image, the second binary image and the third binary image after filtering to obtain a mask image corresponding to the region of interest.

[0025] In a third aspect, the present invention further provides an electronic device, the electronic device comprising:

[0026] one or more processors;

[0027] A storage device for storing one or more programs;

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the region of interest mask extraction method provided in the first aspect.

[0029] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for extracting a mask of a region of interest as provided in the first aspect is implemented.

[0030] The beneficial effects of the present invention are as follows: by performing color threshold segmentation in different color domains based on the absolute color information of the region of interest; by performing binarization processing based on the relative color gradient of the region of interest, the binarization segmentation of any color region of interest can be realized; by performing closing and opening operations in morphological filtering on three binary images in sequence, internal pores and noise are eliminated and virtual areas are connected; by performing geometric feature filtering on the three binary images after morphological filtering, the wrong segmentation area is eliminated according to the geometric properties of the region of interest; by fusing the three binary images, the advantages of multiple segmentation and extraction methods are complemented to improve the recall rate; by performing morphological filtering on the binary fused image, the interference area existing in the neighborhood of the region of interest is covered.

[0031] The present invention can generate a mask image of the region of interest robustly, accurately and efficiently under the interference of complex backgrounds and similar colors according to the geometric features, color differences and spatial gradients of the region of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for extracting a mask of a region of interest provided by the present invention;

[0033] Figure 2 This is a flow chart of a method for extracting a mask of a region of interest according to a first embodiment of the present invention;

[0034] Figure 3 A schematic flow chart of a method for extracting a mask of a region of interest according to a first embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of an image to be extracted according to the first embodiment of the present invention;

[0036] Figure 5 It is a schematic diagram of a first binary image corresponding to the RGB color gamut of the first embodiment of the present invention;

[0037] Figure 6 Schematic diagram of a second binarized image corresponding to the HSV color gamut according to the first embodiment of the present invention;

[0038] Figure 7 is a schematic diagram of a differential image according to the first embodiment of the present invention;

[0039] Figure 8 is a schematic diagram of a third binarized image according to the first embodiment of the present invention;

[0040] Fig. 9 Schematic diagram of three binary images after geometric feature filtering according to the first embodiment of the present invention;

[0041] Fig.10 It is a schematic diagram of a binary fused image and a mask image of a region of interest according to the first embodiment of the present invention;

[0042] Fig.11 is a schematic diagram of a mask image of a region of interest after being negated according to the first embodiment of the present invention;

[0043] Fig.12 A schematic structural diagram of a mask extraction device for a region of interest provided by the present invention;

[0044] Fig.13 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0046] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, sub-computer program, etc.

[0047] In addition, the terms "first", "second", etc. can be used in this article to describe various directions, actions, steps or elements, but these directions, actions, steps or elements are not limited by these terms. These terms are only used to distinguish a first direction, action, step or element from another direction, action, step or element. For example, without departing from the scope of the present application, the first information can be referred to as the second information, and similarly, the second information can be referred to as the first information. Both the first information and the second information are information, but they are not the same information. The terms "first", "second", etc. cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can expressly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0048] like Figure 1 As shown, a method for extracting a mask of a region of interest includes:

[0049] Performing color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and performing color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted, to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut;

[0050] According to the unit color vector of the color of the region of interest, two basis vectors with a large angle difference with the unit color vector are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the color of the region of interest;

[0051] According to the color representation image, the minimum channel difference is calculated to obtain a difference image;

[0052] Performing edge detection on the differential image to obtain a third binary image;

[0053] performing filtering processing on the first binarized image, the second binarized image and the third binarized image respectively;

[0054] The first binarized image, the second binarized image and the third binarized image after filtering are fused to obtain a mask image corresponding to the region of interest.

[0055] From the above description, it can be seen that the mask image of the region of interest can be generated robustly, accurately and efficiently under the conditions of complex background interference and similar color interference.

[0056] In an optional embodiment, according to the color of the region of interest to be extracted, the RGB color gamut image and the HSV color gamut image are respectively subjected to color threshold segmentation to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut, specifically:

[0057] Determine a first segmentation threshold according to the color of the region of interest to be extracted, and perform binarization processing on the RGB color gamut image according to the first segmentation threshold to obtain a first binarized image corresponding to the RGB color gamut;

[0058] A second segmentation threshold is determined according to the color and saturation of the region of interest to be extracted, and the HSV color gamut image is binarized according to the second segmentation threshold to obtain a second binarized image corresponding to the HSV color gamut.

[0059] It can be seen from the above description that based on the absolute color information of the region of interest, color threshold segmentation is performed in different color domains.

[0060] In an optional embodiment, according to the unit color vector of the color of the region of interest, two basis vectors with a large angle difference with the unit color vector are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the color of the region of interest, specifically:

[0061] The color vector mean of the color of the region of interest is obtained by sampling, and normalized to obtain a unit color vector;

[0062] Calculate the cosine similarity between each basis vector of the RGB color domain and the unit color vector respectively, and replace the basis vector corresponding to the minimum cosine similarity with the unit color vector to obtain an orthogonalized basis vector mark;

[0063] According to the unit color vector, the other two basis vectors in the basis vector mark are subjected to Schmidt orthogonal decomposition to obtain a unit orthogonal basis;

[0064] According to the coordinate conversion formula, the RGB color gamut image is converted to obtain a color representation image based on the color of the region of interest. The coordinate conversion formula is [x] c =C -1 [x] rgb , where [x] c represents the coordinates of the color representation image, C represents the unit orthogonal basis, [x] rgb Represents the coordinates of an RGB color space image.

[0065] From the above description, it can be seen that the binarization processing is performed based on the relative color gradient of the region of interest to achieve the binary segmentation of the region of interest of any color.

[0066] In an optional embodiment, the difference image I diff =min(I c 1 -I c 2 , I c 1 -I c 3 ), where I c 1 ,I c 2 and I c 3 Respectively represent the color representation image I c Channel 1, Channel 2 and Channel 3 of the .

[0067] In an optional embodiment, before performing edge detection on the differential image to obtain the third binarized image, the method further includes:

[0068] The differential image is denoised by using an edge-preserving filtering algorithm.

[0069] From the above description, it can be seen that by denoising the single-channel difference image, the noise of non-main gradients can be removed and the false detection of edges can be reduced.

[0070] In an optional embodiment, the filtering process is performed on the first binarized image, the second binarized image and the third binarized image respectively, comprising:

[0071] performing closing operation and opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively;

[0072] According to the geometric properties of the region of interest, geometric feature filtering is performed on the first binarized image, the second binarized image and the third binarized image after morphological filtering respectively.

[0073] From the above description, it can be seen that the three binary images are subjected to closing and opening operations in morphological filtering in turn to eliminate internal gaps and noise and connect virtual areas; the three binary images after morphological filtering are subjected to geometric feature filtering respectively to eliminate the wrong segmentation areas according to the geometric properties of the region of interest.

[0074] In an optional embodiment, the geometric feature filtering is performed on the first binarized image, the second binarized image and the third binarized image after morphological filtering respectively according to the geometric attributes of the region of interest, specifically:

[0075] Obtaining external closed contours in a binary image, and respectively calculating the contour convex hull area and the internal solidity rate of each external closed contour, wherein the binary image includes a first binary image, a second binary image, and a third binary image;

[0076] If the convex hull area of ​​an external closed contour is less than or equal to a preset area threshold, the area where the external closed contour is located is set as an uninterested area;

[0077] If the convex hull area of ​​an external closed contour is greater than a preset area threshold and the internal solid ratio of the external closed contour is greater than a preset solid ratio threshold, polygon fitting is performed on the external closed contour, and the number of vertices of the polygon obtained by fitting and the average of the absolute values ​​of the cosines of its internal angles are counted;

[0078] If the number of vertices of the polygon is different from the number of vertices of the region of interest and the mean of the absolute values ​​of the cosine is greater than a preset mean threshold, the region where the external closed contour is located is set as a region of no interest.

[0079] From the above description, it can be seen that by performing geometric feature filtering, the wrong segmentation area can be eliminated according to the geometric properties of the region of interest.

[0080] In an optional embodiment, the first binarized image, the second binarized image and the third binarized image after filtering are fused to obtain a mask image corresponding to the region of interest, specifically:

[0081] Performing gated fusion on the first binarized image, the second binarized image, and the third binarized image after filtering to obtain a binarized fused image;

[0082] The binary fused image is expanded to generate a mask image corresponding to the region of interest.

[0083] From the above description, it can be seen that by fusing the three binary images, the advantages of various segmentation and extraction methods can be complemented to improve the recall rate; by dilating the binary fused image with morphological filtering, the interference area in the neighborhood of the area of ​​interest can be covered.

[0084] In an optional embodiment, the first binarized image, the second binarized image and the third binarized image after filtering are fused to obtain a mask image corresponding to the region of interest, specifically:

[0085] Performing dilation processing on the first binarized image, the second binarized image and the third binarized image after filtering processing respectively;

[0086] The first binary image, the second binary image and the third binary image after the dilation process are gated and fused to generate a mask image corresponding to the region of interest.

[0087] It can be seen from the above description that the expansion processing and the fusion processing can be performed in any order.

[0088] The present invention also provides a device for extracting a mask of a region of interest, comprising:

[0089] A threshold segmentation module is used to perform color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and perform color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut;

[0090] A coordinate conversion module is used to determine two basis vectors with a large angle difference from the unit color vector of the color of the region of interest from the basis vectors of the RGB color domain, perform Schmidt orthogonal decomposition, obtain a unit orthogonal basis, and perform coordinate conversion on the RGB color domain image according to the unit orthogonal basis to obtain a color representation image based on the color of the region of interest;

[0091] A difference module, used for calculating the minimum channel difference according to the color representation image to obtain a difference image;

[0092] An edge detection module, used for performing edge detection on the differential image to obtain a third binary image;

[0093] A filtering module, used for performing filtering processing on the first binarized image, the second binarized image and the third binarized image respectively;

[0094] The fusion module is used to fuse the first binary image, the second binary image and the third binary image after filtering to obtain a mask image corresponding to the region of interest.

[0095] The present invention further provides an electronic device, comprising:

[0096] one or more processors;

[0097] A storage device for storing one or more programs;

[0098] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned region of interest mask extraction method.

[0099] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for extracting a mask of a region of interest as described above is implemented.

[0100] Embodiment 1

[0101] Please refer to Figure 2-11 The first embodiment of the present invention is: a method for extracting a mask of a region of interest (ROI), which can be applied to scenes such as industrial visual positioning, document scanning and correction, image restoration, and video surveillance. In this embodiment, an example of applying to image restoration and using a ROI as a color target frame is used for explanation.

[0102] like Figure 2 and Figure 3 As shown, the following steps are included:

[0103] S101: Perform color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and perform color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut. Figure 4 As shown, the region of interest is the detection frame of the excavator in the figure, and the first binary image and the second binary image are respectively as follows Figure 5-6 shown.

[0104] Specifically, the image I to be extracted is first converted into the HSV color space and the RGB color space, respectively. Then, according to the known target frame color prior, color threshold segmentation is performed in different color spaces to obtain the first binary image S rgb and the second binarized image S hsv .

[0105] This step is to perform binarization based on the absolute color information of the target frame. When the color of the target frame is known, the traditional color segmentation can efficiently extract the segmentation area of ​​a simple sample (without obvious color interference). To this end, the segmentation threshold can be set directly in the RGB color domain according to the known color of the target frame to perform color segmentation and obtain the first binary image S rgb .

[0106] For samples with similar color interference, the color segmentation threshold can be set according to the saturation in the HSV color domain, because the color of the target frame is always the maximum saturation when drawn. Therefore, the image to be repaired I is converted to the HSV color domain, and the segmentation threshold is set according to the prior information (i.e. the color and saturation of the target frame) to perform color segmentation, and the second binary image S is obtained. hsv .

[0107] In this embodiment, for color segmentation of the RGB color gamut, the segmentation threshold may include an interval upper threshold and an interval lower threshold, that is, a valid interval is determined according to the color of the target frame, such as [a, b]. Since the pixel value is generally represented by a three-dimensional vector, such as (x[0], x[1], x[2]), the segmentation threshold is also three-dimensional, and the valid pixels need to be within the valid interval in each dimension, that is, a[0]≤x[0]≤b[0]&a[1]≤x[1]≤b[1]&a[2]≤x[2]≤b[2]. The pixel value within the valid interval is determined as a valid pixel value, and the pixel value not within the valid interval is determined as an invalid pixel value.

[0108] For example, if the color of the target frame is red, that is, the pixel value is (255, 0, 0), then the effective intervals of R, G, and B three-dimensional separation can be set to [180, 255], [0, 50], and [0, 50], respectively.

[0109] The color segmentation of HSV color gamut is the same as that of RGB color gamut.

[0110] S102: According to the unit color vector of the color of the region of interest, two basis vectors with a large angle difference with the unit color vector are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the color of the region of interest.

[0111] This step is to perform binarization based on the target relative color gradient.

[0112] In practical applications, due to image quality degradation and complex and diverse shooting scenes, the data set to be processed often contains both simple samples and difficult samples (with artifacts, color casts, interference, etc.). Using absolute color thresholds for segmentation cannot take into account both precision and recall, and requires targeted parameter adjustment, which does not meet the original design intention of efficient and automated. To this end, consider introducing relative color information for binary segmentation and filtering based on the target box geometry prior to take into account both segmentation accuracy and recall. Color gradient difference requires that the segmented color is exactly a basis for orthogonal color representation, while the RGB color gamut is only suitable for the segmentation of red, green and blue colors.

[0113] To achieve binary segmentation of any color target frame, first, the three-dimensional representation of the RGB color domain is transformed. Specifically, the actual sample is sampled to obtain the color vector mean of the actual target frame, and then normalized to obtain the unit color vector [V] rgb ∈R 3 .

[0114] Then, for the basis vector of the RGB color space, that is, the 3x3 unit matrix, calculate its basis vector and unit color vector [V] respectively. rgb The cosine similarity of , and the unit color vector [V] rgb Replace the basis vector corresponding to the minimum cosine similarity to obtain the orthogonalized basis vector symbol C'=[V, e1, e2], where e1 and e2 represent the unit color vector [V] rgb The two basis vectors with the largest cosine similarity.

[0115] Next, the unit color vector [V] in C' is labeled with the basis vector rgb As the initial value, perform Schmidt orthogonal decomposition on the other two basis vectors to obtain the unit orthogonal basis C.

[0116] Finally, according to the coordinate transformation formula [x] c =C -1 [x] rgb , for RGB color gamut image Irgb Perform coordinate transformation to obtain the color representation image I based on the target frame prior color V c . Where [x] c Represents color representation image I c The coordinates of [x] rgb Represents RGB color gamut image I rgb The coordinates of .

[0117] Furthermore, if the target frame color itself is one of red, green, and blue, there is no need to perform basis transformation.

[0118] S103: Calculate the minimum channel difference according to the color representation image to obtain a difference image, such as Figure 7 shown.

[0119] Specifically, the single-channel difference image I diff =min(I c 1 -I c 2 , I c 1 -I c 3 ), where I c 1 ,I c 2 and I c 3 Respectively represent the color representation image I c The 1st, 2nd, and 3rd channels of , the 1st channel corresponds to the basis vector of the target box color vector, and the 2nd and 3rd channels correspond to the other two orthogonal basis vectors.

[0120] S104: De-noising the difference image by using an edge-preserving filtering algorithm, and performing edge detection on the de-noised difference image to obtain a third binary image, such as Figure 8 shown.

[0121] Specifically, in order to avoid the potential noise of the degraded image affecting the edge extraction, the visible image noise situation uses an edge-preserving filtering algorithm (this embodiment uses a bilateral filtering algorithm) to process the difference image, remove the noise of non-main gradients, and reduce false edge detection. diff The significant gradient in the target box is taken as the edge of the target box, and the denoised difference image I diff Perform edge detection (Canny edge detection algorithm is used in this embodiment) to obtain the third binary image S diff .

[0122] S105: performing closing operation and opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively.

[0123] Specifically, for the first binary image S rgb , the second binary image S hsv And the third binary image S diff , and perform closing and opening operations in morphological filtering in sequence to eliminate internal gaps and noise and connect virtual areas.

[0124] S106: According to the geometric attributes of the region of interest, geometric feature filtering is performed on the first binary image, the second binary image, and the third binary image after morphological filtering, and the first binary image, the second binary image, and the third binary image after geometric feature filtering are respectively as follows: Fig. 9 As shown in (a)(b)(c).

[0125] Specifically, it is known that the target box is hollow, and a solid segmentation area greater than the threshold can be achieved according to the solid ratio. First, the external closed contours in the binary image are extracted, and each external closed contour is traversed to calculate the contour convex hull area A and the internal solid ratio r%, where the calculation formula of the internal solid ratio is r% = the area of ​​the area to be repaired with non-zero pixel values ​​in A / A×100%.

[0126] After calculating the contour convex hull area A and the internal solid rate r%, the noise area can be removed according to the geometric prior of the target frame. Specifically, for contours whose contour convex hull area A is less than or equal to the preset area threshold (625 in this embodiment), the area where it is located is set as a non-repairing area to remove tiny areas that cannot be the target frame; for contours whose contour convex hull area A is greater than the preset area threshold and whose internal solid rate r% is less than or equal to the preset solid rate threshold (75% in this embodiment), they are retained; for contours whose contour convex hull area A is greater than the preset area threshold and whose internal solid rate r% is greater than the preset solid rate threshold, further checks are required to avoid erroneously removing moments that are completely filled due to morphological operations. The specific operation is as follows: first, perform contour polygon fitting, and then count the number of vertices of the polygon obtained by fitting and the mean of the absolute value of the cosine of each internal angle. According to the geometric prior of the target frame (generally a rectangle, that is, the number of vertices is 4), the contours with 4 vertices and the mean of the absolute value of the cosine less than or equal to the preset mean threshold (0.1 in this embodiment) are retained; for the contours with not 4 vertices and the mean of the absolute value of the cosine greater than the preset mean threshold, the area where they are located is set as a non-repairing area to eliminate the contour area that obviously does not conform to the shape of the target frame and retain possible overlapping figures of multiple rectangular frames.

[0127] The three binary images after morphological filtering are filtered by geometric features to eliminate the wrong segmentation areas according to the geometric properties of the region of interest.

[0128] S107: performing gated fusion on the first binary image, the second binary image and the third binary image after geometric feature filtering to obtain a binary fused image, such as Fig.10 As shown in (a) in .

[0129] Specifically, the first binary image S' after geometric feature filtering rgb , the second binary image S' hsv And the third binary image S' diff Gated fusion is performed to achieve complementary advantages of multiple segmentation and extraction methods. The redundant detection mechanism can improve the recall rate. The binary fusion image S f =S' rgb ∩S' hsv ∩S' diff If one of the binary images cannot provide effective information, the gating mechanism can be used to close the calculation path to improve the calculation efficiency.

[0130] S108: Performing a morphological filtering dilation process on the binary fused image to generate a mask image corresponding to the region of interest.

[0131] Specifically, in order to avoid the potential artifacts and noise in the neighborhood of the region of interest affecting the restoration results (the compression artifacts generated by JPEG generally appear in the 8×8 neighborhood range at the edge of the target frame), the binary fused image S f Morphological filtering (dilation) is performed to cover the interference area and generate a mask image M of the area of ​​interest, such as Fig.10 As shown in (b) in the figure. Further, according to the actual application needs, the binary pixel values ​​in the mask image can be inverted, that is, the pixel values ​​of the original pixel values ​​1 are changed to 0, and the pixel values ​​of the original pixel values ​​0 are changed to 1. The mask image obtained after inversion is as follows: Fig.11 shown.

[0132] Furthermore, in another optional embodiment, the three binary images after geometric feature filtering may be firstly dilated by morphological filtering, and then the three binary images after dilation processing may be fused.

[0133] This embodiment improves the traditional color threshold segmentation method and uses the color difference, spatial gradient, and geometric features of the region of interest to obtain an accurate, consistent, and robust region of interest mask image using a gated post-fusion strategy.

[0134] Embodiment 2

[0135] Please refer to Fig.12The second embodiment of the present invention is: a device for extracting a mask of a region of interest, which can execute the method for extracting a mask of a region of interest provided by the first embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software / or hardware, specifically including:

[0136] The threshold segmentation module 201 is used to perform color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and perform color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut;

[0137] A coordinate conversion module 202 is used to determine two basis vectors with a large angle difference from the unit color vector of the color of the region of interest from the basis vectors of the RGB color domain, perform Schmidt orthogonal decomposition, obtain a unit orthogonal basis, and perform coordinate conversion on the RGB color domain image according to the unit orthogonal basis to obtain a color representation image based on the color of the region of interest;

[0138] A difference module 203 is used to calculate the minimum channel difference according to the color representation image to obtain a difference image;

[0139] An edge detection module 204 is used to perform edge detection on the differential image to obtain a third binary image;

[0140] A filtering module 205, configured to perform filtering processing on the first binarized image, the second binarized image and the third binarized image respectively;

[0141] The fusion module 206 is used to fuse the first binarized image, the second binarized image and the third binarized image after filtering to obtain a mask image corresponding to the region of interest.

[0142] In an optional implementation, the threshold segmentation module 201 includes:

[0143] A first segmentation unit is used to determine a first segmentation threshold according to the color of the region of interest to be extracted, and to perform binarization processing on the RGB color gamut image according to the first segmentation threshold to obtain a first binarized image corresponding to the RGB color gamut;

[0144] The second segmentation unit is used to determine a second segmentation threshold according to the color and saturation of the region of interest to be extracted, and to binarize the HSV color gamut image according to the second segmentation threshold to obtain a second binarized image corresponding to the HSV color gamut.

[0145] In an optional implementation, the coordinate conversion module 202 includes:

[0146] A normalization unit, used to obtain the color vector mean of the color of the region of interest by sampling, and perform normalization processing to obtain a unit color vector;

[0147] A calculation unit, used for respectively calculating the cosine similarity between each basis vector of the RGB color domain and the unit color vector, and replacing the basis vector corresponding to the minimum cosine similarity with the unit color vector to obtain an orthogonalized basis vector mark;

[0148] A decomposition unit, used for performing Schmidt orthogonal decomposition on the other two basis vectors in the basis vector mark according to the unit color vector to obtain a unit orthogonal basis;

[0149] A conversion unit is used to perform coordinate conversion on the RGB color gamut image according to a coordinate conversion formula to obtain a color representation image based on the color of the region of interest, wherein the coordinate conversion formula is [x] c =C -1 [x] rgb , where [x] c represents the coordinates of the color representation image, C represents the unit orthogonal basis, [x] rgb Represents the coordinates of an RGB color space image.

[0150] In an optional embodiment, the device further comprises:

[0151] The denoising module is used to denoise the difference image by using an edge-preserving filtering algorithm.

[0152] In an optional implementation, the filtering module 205 includes:

[0153] A first filtering unit, used for sequentially performing a closing operation and an opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively;

[0154] The filtering unit is used to perform geometric feature filtering on the first binary image, the second binary image and the third binary image after morphological filtering according to the geometric attributes of the region of interest.

[0155] In an optional embodiment, the filtering unit is specifically used for:

[0156] Obtaining external closed contours in a binary image, and respectively calculating the contour convex hull area and the internal solidity rate of each external closed contour, wherein the binary image includes a first binary image, a second binary image, and a third binary image;

[0157] If the convex hull area of ​​an external closed contour is less than or equal to a preset area threshold, the area where the external closed contour is located is set as an uninterested area;

[0158] If the convex hull area of ​​an external closed contour is greater than a preset area threshold and the internal solid ratio of the external closed contour is greater than a preset solid ratio threshold, polygon fitting is performed on the external closed contour, and the number of vertices of the polygon obtained by fitting and the average of the absolute values ​​of the cosines of its internal angles are counted;

[0159] If the number of vertices of the polygon is different from the number of vertices of the region of interest and the mean of the absolute values ​​of the cosine is greater than a preset mean threshold, the region where the external closed contour is located is set as a region of no interest.

[0160] In an optional implementation, the fusion module 206 includes:

[0161] A first fusion unit is used to perform gated fusion on the first binarized image, the second binarized image and the third binarized image after filtering to obtain a binarized fused image;

[0162] The first expansion unit is used to perform expansion processing on the binary fused image to generate a mask image corresponding to the region of interest.

[0163] In another optional implementation, the fusion module 206 includes:

[0164] A second expansion unit is used to expand the first binary image, the second binary image and the third binary image after filtering respectively;

[0165] The second fusion unit is used to perform gated fusion on the first binary image, the second binary image and the third binary image after the dilation process to generate a mask image corresponding to the region of interest.

[0166] Embodiment 3

[0167] Please refer to Fig.13 , Embodiment 3 of the present invention is: an electronic device, the electronic device comprising:

[0168] One or more processors 301;

[0169] Storage device 302, used to store one or more programs;

[0170] When the one or more programs are executed by the one or more processors 301, the one or more processors 301 implement the various processes in the embodiment of the region of interest mask extraction method as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0171] Embodiment 4

[0172] Embodiment 4 of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes in the embodiment of the method for extracting the mask of the region of interest as described above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0173] In summary, the present invention provides a method, device, equipment and storage medium for extracting a mask of a region of interest. Based on the absolute color information of the region of interest, color threshold segmentation is performed in different color domains respectively; by performing binary segmentation based on the relative color gradient of the region of interest, binary segmentation of any color region of interest can be achieved; by denoising a single-channel differential image, noise of non-main gradients can be removed to reduce falsely detected edges; by performing closing and opening operations in morphological filtering on three binary images in sequence, internal gaps and noise are eliminated and virtual areas are connected; by performing geometric feature filtering on the three binary images after morphological filtering, erroneous segmentation areas are eliminated according to the geometric properties of the region of interest; by fusing the three binary images, the advantages of multiple segmentation and extraction methods are complemented to improve the recall rate; by performing morphological filtering on the binary fused image, interference areas existing in the neighborhood of the region of interest are covered.

[0174] The present invention can generate a mask image of the region of interest robustly, accurately and efficiently under the interference of complex backgrounds and similar colors according to the geometric features, color differences and spatial gradients of the region of interest.

[0175] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0176] It is worth noting that in the embodiment of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0177] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for extracting a mask of a region of interest, characterized in that: include: Performing color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and performing color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted, to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut; According to the unit color vector of the color of the region of interest, two basis vectors with a large angle difference with the unit color vector are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the color of the region of interest; According to the color representation image, the minimum channel difference is calculated to obtain a difference image; Performing edge detection on the differential image to obtain a third binary image; performing filtering processing on the first binarized image, the second binarized image and the third binarized image respectively; The first binary image, the second binary image and the third binary image after filtering are fused to obtain a mask image corresponding to the region of interest; The filtering process is performed on the first binarized image, the second binarized image and the third binarized image respectively, comprising: performing closing operation and opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively; According to the geometric properties of the region of interest, geometric feature filtering is performed on the first binarized image, the second binarized image and the third binarized image after morphological filtering respectively.

2. The method for extracting a region of interest mask according to claim 1, characterized in that: According to the color of the region of interest to be extracted, the RGB color gamut image and the HSV color gamut image are respectively segmented by color threshold to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut, specifically: Determine a first segmentation threshold according to the color of the region of interest to be extracted, and perform binarization processing on the RGB color gamut image according to the first segmentation threshold to obtain a first binarized image corresponding to the RGB color gamut; A second segmentation threshold is determined according to the color and saturation of the region of interest to be extracted, and the HSV color gamut image is binarized according to the second segmentation threshold to obtain a second binarized image corresponding to the HSV color gamut.

3. The method for extracting a region of interest mask according to claim 1, characterized in that: According to the unit color vector of the color of the region of interest, two basis vectors with a large angle difference are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the color of the region of interest, specifically: The color vector mean of the color of the region of interest is obtained by sampling, and normalized to obtain a unit color vector; Calculate the cosine similarity between each basis vector of the RGB color domain and the unit color vector respectively, and replace the basis vector corresponding to the minimum cosine similarity with the unit color vector to obtain an orthogonalized basis vector mark; According to the unit color vector, the other two basis vectors in the basis vector mark are subjected to Schmidt orthogonal decomposition to obtain a unit orthogonal basis; According to the coordinate conversion formula, the RGB color gamut image is converted to obtain a color representation image based on the color of the region of interest. The coordinate conversion formula is [x] c =C -1 [x] rgb , where [x] c represents the coordinates of the color representation image, C represents the unit orthogonal basis, [x] rgb Represents the coordinates of an RGB color space image.

4. The method for extracting a mask of a region of interest according to claim 1, characterized in that: The difference image I diff =min(I c 1 -I c 2 , I c 1 -I c 3 ), where I c 1 ,I c 2 and I c 3 Respectively represent the color representation image I c Channel 1, Channel 2 and Channel 3 of the .

5. The method for extracting a mask of a region of interest according to claim 1, characterized in that: According to the geometric attributes of the region of interest, the first binary image, the second binary image and the third binary image after morphological filtering are respectively subjected to geometric feature filtering, specifically: Obtaining external closed contours in a binary image, and respectively calculating the contour convex hull area and the internal solidity rate of each external closed contour, wherein the binary image includes a first binary image, a second binary image, and a third binary image; If the convex hull area of ​​an external closed contour is less than or equal to a preset area threshold, the area where the external closed contour is located is set as an uninterested area; If the convex hull area of ​​an external closed contour is greater than a preset area threshold and the internal solid ratio of the external closed contour is greater than a preset solid ratio threshold, polygon fitting is performed on the external closed contour, and the number of vertices of the polygon obtained by fitting and the average of the absolute values ​​of the cosines of its internal angles are counted; If the number of vertices of the polygon is different from the number of vertices of the region of interest and the mean of the absolute values ​​of the cosine is greater than a preset mean threshold, the region where the external closed contour is located is set as a region of no interest.

6. The method for extracting a mask of a region of interest according to claim 1, characterized in that: The first binary image, the second binary image and the third binary image after filtering are fused to obtain a mask image corresponding to the region of interest, specifically: Performing gated fusion on the first binarized image, the second binarized image, and the third binarized image after filtering to obtain a binarized fused image; The binary fused image is expanded to generate a mask image corresponding to the region of interest.

7. The method for extracting a region of interest mask according to claim 1, characterized in that: The first binary image, the second binary image and the third binary image after filtering are fused to obtain a mask image corresponding to the region of interest, specifically: Performing dilation processing on the first binarized image, the second binarized image and the third binarized image after filtering processing respectively; The first binary image, the second binary image and the third binary image after the dilation process are gated and fused to generate a mask image corresponding to the region of interest.

8. A device for extracting a mask of a region of interest, characterized in that: include: A threshold segmentation module is used to perform color space conversion on the image to be extracted to obtain an RGB color gamut image and an HSV color gamut image, and perform color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the region of interest to be extracted to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut; A coordinate conversion module is used to determine two basis vectors with a large angle difference from the unit color vector of the color of the region of interest from the basis vectors of the RGB color domain, perform Schmidt orthogonal decomposition, obtain a unit orthogonal basis, and perform coordinate conversion on the RGB color domain image according to the unit orthogonal basis to obtain a color representation image based on the color of the region of interest; A difference module, used for calculating the minimum channel difference according to the color representation image to obtain a difference image; An edge detection module, used for performing edge detection on the differential image to obtain a third binary image; A filtering module, used for performing filtering processing on the first binarized image, the second binarized image and the third binarized image respectively; A fusion module is used to fuse the first binary image, the second binary image and the third binary image after filtering to obtain a mask image corresponding to the region of interest; The filtering module comprises: A first filtering unit, used for sequentially performing a closing operation and an opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively; The filtering unit is used to perform geometric feature filtering on the first binary image, the second binary image and the third binary image after morphological filtering according to the geometric attributes of the region of interest.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the region of interest mask extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for extracting a mask of a region of interest as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Visual and intelligent detection method for surface quality of FPC (Flexible Printed Circuit)

    CN105445277A

  • A detection and identification method and device for grazing sheep to feed on typical plant species

    CN109190571A

  • Material falling detection method, device and equipment and storage medium

    CN110047063A

  • Thyroid ultrasound image interference interface automatic identification and removal method based on threshold segmentation

    CN112634240A

  • Device and method for color correction, optical instrument, and color correction program

    JP2005165683A