Unsupervised image salient bounding box detection method based on two-stage clustering
Through the two-stage clustering method, combined with multi-level feature extraction and self-supervised pre-training model, the training and application difficulties of unsupervised significant bounding box detection technology under the lack of labeled data is solved, which improves detection accuracy and robustness and is more adaptable.
Patent Information
- Application Number
- CN202411890234.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-15
AI Technical Summary
The existing unsupervised significant bounding box detection technology is difficult to effectively train and apply without labeling data. The traditional clustering method has poor detection effect and low robustness, and has weak adaptability to different types of images.
Using the two-stage clustering method, multi-level features are first extracted through ResNet-50, which is self-supervised and pre-trained by MoCo v2, and initial segmentation and screening are combined with spectral clustering and custom boundary priors; then point cloud and edge detection technology are introduced for contour extraction and hierarchical clustering for regional merger, and a significant bounding box is finally generated.
It improves the detection accuracy and robustness of significant objects, enhances the model's understanding of image content, reduces its dependence on labeled data, is more adaptable, and can work effectively in variable application scenarios.
Smart Images
Figure CN119832263A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of machine learning, and in particular relates to an unsupervised image salient bounding box detection method based on double-stage clustering. Background Art
[0002] Salient bounding box detection plays an important role in many fields such as autonomous driving, intelligent monitoring, and medical image analysis. Currently, salient bounding box detection mainly relies on supervised learning methods, which face challenges in many practical applications because they require a large amount of labeled data to train the model (Hussain M.Yolov1 to v8:Unveiling each variant a comprehensive review of yolo[J].IEEE Access, 2024, 12:42816-42833.). Although unsupervised salient detection methods have begun to be explored, most of them are based on deep learning models and rely on the support of large-scale data. The difficulty lies in the lack of necessary labeled data for unsupervised methods themselves (Shin G, Albanie S, Xie W.Unsupervised salient object detection with spectral cluster voting[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:3971-3980.).
[0003] Existing unsupervised salient bounding box detection technologies mainly face the following problems: First, existing methods focus on converting unsupervised methods into supervised methods, but this does not solve the core challenge of unsupervised methods, that is, how to effectively train and apply them without labeled data. Second, traditional clustering methods perform poorly in detection results and have low robustness. In addition, due to the dependence of existing unsupervised methods on labeled data, their adaptability to different types of images is weak and they cannot effectively meet the changing application requirements.
[0004] Difficulty in solving the above technical problems: The shapes, colors and textures of objects in complex scenes often show great diversity, and traditional clustering algorithms are difficult to cope with such high complexity. In addition, in the absence of labeled data, the lack of effective supervision signals makes it impossible for the algorithm to effectively learn the true features of the object, affecting the accuracy and reliability of the detection results. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides an unsupervised image salient bounding box detection method based on two-stage clustering.
[0006] The present invention implements an unsupervised image salient bounding box detection method based on two-stage clustering, the method comprising:
[0007] In the first step, ResNet-50 pre-trained with MoCo v2 self-supervision is used as the feature extractor to extract the features of the 1st, 4th and 5th convolutional blocks from the image and generate the corresponding feature maps. Then, the feature maps are interpolated to the same size as the image using the nearest neighbor interpolation method. The adjusted feature maps are recorded as F1, F4 and F5 respectively, where h and w represent the row height and column width of the image, respectively. i Indicates the number of channels of the feature map, i∈{1,4,5};
[0008] In the second step, F1, F4 and F5 are segmented based on spectral clustering, and a set of binary mask sets M1, M4 and M5 are generated respectively to highlight the salient areas;
[0009] The third step is to combine the custom boundary prior and area prior to screen M1, M4 and M5 respectively, and retain the binary masks that meet the conditions to form a new binary mask set M1′, M4′ and M5′;
[0010] The fourth step is to introduce point cloud and edge detection technology to extract contours from the binary masks in M1′, M4′ and M5′, and calculate the number of extracted contours;
[0011] The fifth step is to merge the binary masks in M1′, M4′ and M5′ based on the hierarchical clustering method and the number of contours;
[0012] In the sixth step, based on the information extracted and merged in the previous steps, the final bounding box is generated by region merging.
[0013] The advantages and positive effects of the present invention are as follows: the present invention can refine processing in a multi-level feature space through multi-level feature extraction, thereby enhancing the model's ability to understand image content; the design of two-stage clustering enables more flexible consideration of multiple features when generating bounding boxes, and the selected mask has high representativeness and robustness, thereby improving the detection accuracy of significant objects. In addition, point cloud computing and hierarchical clustering are used to further optimize the screening and generation of bounding boxes, ensuring that the output results can accurately cover significant objects, thereby improving the accuracy and robustness of detection.
[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a flowchart of an unsupervised image salient bounding box detection method based on two-stage clustering provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0016] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0017] In order to solve the problem that the unsupervised bounding box detection method is highly dependent on the deep learning model, so that it requires a large amount of labeled data and inevitably increases the computational complexity, the present invention provides an unsupervised image salient bounding box detection method based on two-stage clustering, which can be used for salient bounding box detection in an unsupervised state. By combining the advantages of clustering algorithms, multi-level and multi-stage clustering analysis is performed to improve the detection accuracy and robustness. By overcoming the dependence on a large amount of labeled data, the present invention can significantly reduce the cost of data preparation and annotation, thereby achieving effective object detection in data-scarce scenarios and improving the training efficiency of the model.
[0018] The application principle of the present invention is described in detail below in conjunction with the accompanying drawings.
[0019] like Figure 1 As shown, the unsupervised image salient bounding box detection method based on two-stage clustering provided by the embodiment of the present invention includes the following steps:
[0020] S101 uses ResNet-50 pre-trained with MoCo v2 self-supervision as a feature extractor to extract the features of the 1st, 4th, and 5th convolutional blocks from the image and generate the corresponding feature maps. Then the feature maps are interpolated to the same size as the image using the nearest neighbor interpolation method. The adjusted feature maps are recorded as F1, F4, and F5, respectively, where h and w represent the row height and column width of the image, respectively. i Indicates the number of channels of the feature map, i∈{1,4,5};
[0021] S102, segmenting F1, F4 and F5 based on spectral clustering, and generating a set of binary mask sets M1, M4 and M5 for highlighting salient areas respectively;
[0022] S103, combining the custom boundary prior and the area prior to screen M1, M4 and M5 respectively, and retain the binary masks that meet the conditions to form a new binary mask set M1′, M4′ and M5′;
[0023] S104, introducing point cloud and edge detection technology to extract contours from the binary masks in M1′, M4′ and M5′, and calculating the number of extracted contours;
[0024] S105, based on the hierarchical clustering method and the number of contours, the binary masks in M1′, M4′ and M5′ are respectively merged;
[0025] S106, based on the information extracted and merged in the previous steps, a final bounding box is generated by region merging.
[0026] An unsupervised image salient bounding box detection method based on two-stage clustering provided by an embodiment of the present invention specifically comprises the following steps:
[0027] Step 1: Use ResNet-50 pre-trained with MoCo v2 self-supervision as the feature extractor to extract the features of the 1st, 4th and 5th convolutional blocks from the image and generate the corresponding feature maps; then use the nearest neighbor interpolation method to interpolate the feature maps to the same size as the image. The adjusted feature maps are recorded as F1, F4 and F5 respectively, where h and w represent the row height and column width of the image, respectively. i Represents the number of channels of the feature map, i∈{1,4,5}.
[0028] Step 2: segment F1, F4 and F5 based on spectral clustering, and generate a set of binary mask sets M1, M4 and M5 for highlighting salient areas respectively;
[0029] The specific steps include:
[0030] (1) Feature map Each eigenvector f in i j ∈R ci can be regarded as vertex The vertices of the undirected graph are connected by a non-negative weight w. jk The weight connection matrix can be formalized as:
[0031]
[0032] The vertex f i j The degree of ∈V is defined as
[0033] (2) Degree matrix D i Defined as The degree matrix and Laplace matrix are calculated through the adjacency matrix, where the Laplace matrix is defined as:
[0034] Li =D i -W i , (2)
[0035] (3) Perform eigenvalue decomposition on the constructed Laplace matrix to solve the generalized eigenvalue problem:
[0036] L i u=λD i u,(3)
[0037] where λ and u∈R N They represent the eigenvalue and its corresponding eigenvector respectively.
[0038] (4) Take the k eigenvectors with the smallest eigenvalues, and then use the k-means algorithm to cluster these eigenvectors. According to the clustering results, the saliency binary mask of the image is obtained.
[0039] Step 3: Combine the custom boundary prior and area prior to filter M1, M4 and M5 respectively, and retain the binary masks that meet the conditions to form new sets M1′, M4′ and M5′;
[0040] The specific steps include:
[0041] (1) The custom boundary prior is set as follows: if the sum of the pixel values on the boundary is greater than half of the total boundary length, the binary mask is discarded, that is, if the binary mask m a ∈M1′∪M4′∪M5′, we have:
[0042]
[0043] where (i,j)∈boundary is m a Coordinates of boundary points, m a [i,j] represents m a The value at coordinate (i, j), H(m a ) and W(m a ) represent m a The row height and column width of
[0044] (2) The custom area prior is set as follows: if the total number of non-zero pixels in the binary mask is greater than 95% of the total number of pixels, the binary mask is discarded; at the same time, if the total number of non-zero pixels in the binary mask is less than 5% of the total number of pixels, the binary mask is also discarded. That is, if m a ∈M1′∪M4′∪M5′, we have:
[0045]
[0046] Step 4: Introduce point cloud and edge detection technology to extract contours from the binary masks in M1′, M4′ and M5′, and calculate the number of extracted contours;
[0047] The specific steps include:
[0048] (1) Extract binary mask m a ∈M1′∪M4′∪M5′, the coordinates of all non-zero elements, denoted as set p a :
[0049] p a ={(i,j)|m a [i,j]≠0}, (6)
[0050] (2) The coordinate points of all non-zero elements of the binary masks in M1′, M4′ and M5′ are respectively formed into sets and That is, point cloud data, where |·| represents the number of elements;
[0051] (3) Each point cloud data p a ∈P1∪P4∪P5 are mapped to a two-dimensional image data format, denoted as a two-dimensional matrix I a :
[0052]
[0053] The two-dimensional matrix sets corresponding to the point cloud data P1, P4 and P5 are denoted as X1, X4 and X5 respectively, where
[0054]
[0055] (4) Apply Canny edge detection technology For each I a ∈X1∪X4∪X5 to obtain its edge image set and
[0056] (5) Use the contour detection function to extract each edge image e a ∈E1∪E4∪E5 and calculate the number of contours.
[0057] Step 5: Based on the hierarchical clustering method and the number of contours, the binary masks in M1′, M4′ and M5′ are merged respectively;
[0058] The specific steps include:
[0059] (1) Let the binary mask m a The contour set of ∈M1′∪M4′∪M5′ is Each contour has its corresponding coordinate set; if Then we need to perform binary mask m a The coordinates corresponding to the contours are hierarchically clustered into k clusters, namely Otherwise, the cluster set is
[0060] (2) Traverse each cluster or And extract its coordinate set
[0061]
[0062] If the collection If it is not empty, then calculate its bounding box coordinates, that is, the upper left corner (x min ,y min ) and the lower right corner (x max ,y max )’s coordinates:
[0063]
[0064] (3) For each minimum enclosing rectangle, calculate the sum of its intersection and union ratios with all other minimum enclosing rectangles, and select the minimum enclosing rectangle with the largest sum of intersection and union ratios as m a The bounding box of
[0065] (4) Repeat steps (1), (2), and (3) to obtain the bounding boxes of all binary masks and generate the bounding box sets corresponding to M1′, M4′, and M5′. and
[0066] (5) If Then calculate Each element in The sum of the intersection and union ratios of all other elements in the same frame is calculated, and the element with the largest intersection and union ratio is retained as the final detection result of the bounding box at this stage, denoted as Otherwise, record
[0067] Step 6: Based on the information extracted and merged in the previous steps, the final bounding box is generated by region merging; specifically, it includes:
[0068] set up and The corresponding coordinates of the upper left corner and lower right corner are and Where i∈{1,4,5}, then the final bounding box for and The union of can be written as:
[0069]
[0070] in, represents the foreground area, and the rest of the image is the background area; the final bounding box Upper left corner and lower right corner The coordinates are:
[0071]
[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An unsupervised image salient bounding box detection method based on two-stage clustering, characterized in that: The unsupervised image salient bounding box detection method based on two-stage clustering includes: In the first step, ResNet-50 pre-trained with MoCo v2 self-supervision is used as the feature extractor to extract the features of the 1st, 4th and 5th convolutional blocks from the image and generate the corresponding feature maps. Then, the feature maps are interpolated to the same size as the image using the nearest neighbor interpolation method. The adjusted feature maps are recorded as F1, F4 and F5 respectively, where h and w represent the row height and column width of the image, respectively. i Indicates the number of channels of the feature map, i∈{1,4,5}; In the second step, F1, F4 and F5 are segmented based on spectral clustering, and a set of binary mask sets M1, M4 and M5 are generated respectively to highlight the salient areas; The third step is to combine the custom boundary prior and area prior to screen M1, M4 and M5 respectively, and retain the binary masks that meet the conditions to form a new binary mask set M1′, M4′ and M5′; The fourth step is to introduce point cloud and edge detection technology to extract contours from the binary masks in M1′, M4′ and M5′, and calculate the number of extracted contours; The fifth step is to merge the binary masks in M1′, M4′ and M5′ based on the hierarchical clustering method and the number of contours; In the sixth step, based on the information extracted and merged in the previous steps, the final bounding box is generated by region merging.
2. The unsupervised image salient bounding box detection method based on two-stage clustering as claimed in claim 1, characterized in that: The third step combines the custom boundary prior and the area prior to screen M1, M4 and M5 respectively, and retains the binary masks that meet the conditions to form a new binary mask set M1′, M4′ and M5′ including: (1) The custom boundary prior is set as follows: if the sum of the pixel values on the boundary is greater than half of the total boundary length, the binary mask is discarded, that is, if the binary mask m a ∈M1′∪M4′∪M5′, we have: where [i,j]∈boundary is m a Coordinates of boundary points, m a [i,j] represents m a The value at coordinate (i, j), H(m a ) and W(m a ) represent m a The row height and column width of (2) The custom area prior is set as follows: if the total number of non-zero pixels in the binary mask is greater than λ1 times the total number of pixels, the binary mask is discarded; at the same time, if the total number of non-zero pixels in the binary mask is less than λ2 times the total number of pixels, the binary mask is also discarded. That is, if the binary mask m a ∈M1′∪M4′∪M5′, we have: Where 0<λ1,λ2<1.
3. The unsupervised image salient bounding box detection method based on two-stage clustering as claimed in claim 1, characterized in that: The fourth step introduces point cloud and edge detection technology to extract contours from the binary masks in M1′, M4′ and M5′, and calculates the number of extracted contours including: (1) Extract binary mask m a ∈M1′∪M4′∪M5′, the coordinates of all non-zero elements, denoted as set p a : p a ={(i,j)|m a [i,j]≠0}, (3) (2) The coordinate points of all non-zero elements of the binary masks in M1′, M4′ and M5′ are respectively formed into sets and That is, point cloud data, where |·| represents the number of elements; (3) Each point cloud data p a ∈P1∪P4∪P5 are mapped to a two-dimensional image data format, denoted as a two-dimensional matrix I a : The two-dimensional matrix sets corresponding to the point cloud data P1, P4 and P5 are denoted as X1, X4 and X5 respectively, where (4) Apply Canny edge detection technology For each I a ∈X1∪X4∪X5 to obtain the edge image set and (5) Use the contour detection function to extract each edge image e a ∈E1∪E4∪E5 and calculate the number of contours.
4. The unsupervised image salient bounding box detection method based on two-stage clustering as claimed in claim 1, characterized in that: The fifth step is to merge the regions of the binary masks in M1′, M4′ and M5′ respectively based on the hierarchical clustering method and the number of contours, including: (1) Let the binary mask m a The contour set of ∈M1′∪M4′∪M5′ is Each contour has its corresponding coordinate set; if Then we need to perform binary mask m a The coordinates corresponding to the contours are hierarchically clustered into k clusters, namely Otherwise, the cluster set is (2) Traverse each cluster or And extract its coordinate set If the collection If it is not empty, then calculate its bounding box coordinates, that is, the upper left corner (x min ,y min ) and the lower right corner (x max ,y max )’s coordinates: (3) For each minimum enclosing rectangle, calculate the sum of its intersection and union ratios with all other minimum enclosing rectangles, and select the minimum enclosing rectangle with the largest sum of intersection and union ratios as m a The bounding box of (4) Repeat steps (1), (2), and (3) to obtain the bounding boxes of all binary masks and generate the bounding box sets corresponding to M1′, M4′, and M5′. and (5) If Then calculate Each element in The sum of the intersection and union ratios of all other elements in the same frame is calculated, and the element with the largest intersection and union ratio is retained as the final detection result of the bounding box at this stage, denoted as Otherwise, record 5. The unsupervised image salient bounding box detection method based on two-stage clustering as claimed in claim 1, characterized in that: The sixth step generates a final bounding box by merging regions based on the information extracted and merged in the previous step, including: and The corresponding coordinates of the upper left corner and lower right corner are and Where i∈{1,4,5}, then the final bounding box for and The union of can be written as: in, represents the foreground area, and the rest of the image is the background area; the final bounding box Upper left corner and lower right corner The coordinates are: