Method and device for classifying and counting objects in image, medium and equipment

Abstract

The invention discloses an object classification and counting method and device in an image, a medium and equipment, and the method comprises the steps: inputting a plurality of training images into an image segmentation model, obtaining a plurality of training segmentation images, employing an unsupervised category division method, carrying out the training of a model based on each training segmentation image, and obtaining an object classification and counting model; segmenting the to-be-identified image based on the image segmentation model to obtain a plurality of first segmented images, and inputting the plurality of first segmented images into an object classification counting model to obtain a first classification result; adjusting the resolution of the to-be-recognized image, and segmenting the image with the adjusted resolution based on the image segmentation model; performing feature fusion on the plurality of first segmented images and the plurality of segmented images with adjusted resolution, and inputting a fused feature map into an object classification counting model to obtain a second classification result; and based on the second classification result, calibrating the first classification result to obtain a final classification result. According to the invention, the precision of object classification counting is improved.

Description

Technical Field

[0001] This invention relates to the field of object counting technology, and in particular to a method, apparatus, medium and device for classifying and counting objects in an image. Background Technology

[0002] Object counting in images is an important research area in computer vision, and it is widely used in various scenarios such as crowd monitoring, smart agriculture, traffic management, biomedical analysis, and ecological surveys. Common methods for object counting in images include specific-class counting and multi-class counting, with multi-class counting having a wider range of applications.

[0003] Currently, commonly used multi-class counting methods in images require labeling the training images with categories during model training. Then, features are extracted from the labeled images, and the model is trained based on the standard class training images. However, the model cannot accurately identify unlabeled categories, resulting in poor generalization ability. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, medium and device for classifying and counting objects in an image, the main purpose of which is to solve the problem that current methods for classifying and counting multi-category objects in images can accurately identify and count objects with labeled categories, but cannot identify and count objects without labels.

[0005] According to one aspect of this application, a method for classifying and counting objects in an image is provided, the method comprising: Multiple training images are acquired, and each training image is segmented based on a preset image segmentation model to obtain multiple training segmentation maps. An unsupervised class division method is used to train an initial object classification and counting model based on each training segmentation map to obtain a trained object classification and counting model. A target image with a preset first resolution is obtained, and the target image is segmented based on the image segmentation model to obtain multiple first segmented images. The multiple first segmented images are then input into a trained object classification and counting model to obtain a first classification result. The image to be identified is adjusted according to a preset second resolution and a preset third resolution to obtain a second image and a third image. The second image and the third image are then input into the image segmentation model to obtain multiple second segmented images and multiple third segmented images. Multiple first segmented images, multiple second segmented images, and multiple third segmented images are fused to obtain a fused feature map. The fused feature map is then input into a trained object classification and counting model to obtain a second classification result. Based on the counts of each category in the first classification result and the counts of each category in the second classification result, the deviation rate corresponding to each category is calculated. Based on the deviation rate, the counts of each category in the first classification result are calibrated, and the calibrated first classification result is taken as the final classification result.

[0006] Optionally, the step of employing an unsupervised class partitioning method to train the initial object classification and counting model based on each of the training segmentation maps to obtain the trained object classification and counting model includes: The HDBSCAN algorithm is used to cluster multiple training segmentation maps under different clustering parameters to obtain multiple candidate clustering results. For each candidate clustering result, the contour coefficient of each training segmentation map is calculated, and the candidate clustering result with the largest contour coefficient is taken as the optimal clustering result of the training segmentation map, thus obtaining the pseudo-class corresponding to the training segmentation map. Based on each training segmentation map, a ground truth density map is generated for each pseudo-class. Based on each training segmentation map, the pseudo-class corresponding to each training segmentation map, and the ground truth density map corresponding to each pseudo-class, the initial object classification and counting model is trained to obtain the trained object classification and counting model.

[0007] Optionally, the initial object classification and counting model is trained, including: The object classification and counting model extracts visual features and semantic features for each training segmentation map, and then fuses the visual features and semantic features corresponding to each training segmentation map to obtain the fused features corresponding to each training segmentation map. The object classification and counting model performs segmentation optimization and density prediction on each training segmentation map, and outputs multiple optimized segmentation maps and multiple predicted density maps; Based on multiple optimized segmentation maps, multiple predicted density maps, and the fusion features corresponding to each training segmentation map, a cost matrix is ​​constructed. Based on the cost matrix, the Hungarian algorithm is used to match the optimized segmentation maps and the predicted density maps. Loss is calculated based on the optimized segmentation map and predicted density map of the matching, and the parameters in the object classification and counting model are adjusted according to the loss value; The loss value is calculated using the following loss function:

[0008] in, The loss calculated by optimizing the intersection-union ratio (IoU) of the segmentation map and the training segmentation map for an object classification and counting model. Mean squared error loss for the predicted density map and the ground truth density map of an object classification counting model. For weights.

[0009] Optionally, generating a ground truth density map for each pseudo-class based on each training segmentation map includes: Obtain the bounding box of each training segmentation map, determine the center coordinates of each bounding box, and perform Gaussian distribution processing on the density of the training segmentation map based on the center coordinates of each bounding box to obtain the density Gaussian distribution map corresponding to each training segmentation map. For each pseudo-class, the density Gaussian distribution maps corresponding to each training segmentation map of the pseudo-class are superimposed to obtain the initial density map corresponding to each pseudo-class. Based on the density values ​​of the cue point grid in the initial density map corresponding to each pseudo-category, high sampling density region, medium sampling density region and low sampling density region are determined in the initial density map; The number of cue point grids in the high sampling density region is reduced, the number of cue point grids in the low sampling density region is increased, and the number of cue point grids in the medium sampling density region remains unchanged, thus obtaining the ground truth density map corresponding to each pseudo-category.

[0010] Optionally, the step of fusing features from the plurality of first segmented images, the plurality of second segmented images, and the plurality of third segmented images to obtain a fused feature map includes: Each first segmented image, each second segmented image, and each third segmented image are processed by Gaussian distribution of density to obtain multiple first density maps, multiple second density maps, and multiple third density maps. Each first segmentation image, each second segmentation image, each third segmentation image, each first density map, each second density map, and each third density map are scale-aligned and adjusted to obtain multiple first adjusted segmentation images, multiple second adjusted segmentation images, multiple third adjusted segmentation images, multiple first adjusted density maps, multiple second adjusted density maps, and multiple third adjusted density maps. Based on the cross-attention mechanism, each first adjusted segmentation image and each first adjusted density map, a first weight matrix is ​​calculated; based on the cross-attention mechanism, each second adjusted segmentation image and each second adjusted density map, a second weight matrix is ​​calculated; based on the cross-attention mechanism, each third adjusted segmentation image and each third adjusted density map, a third weight matrix is ​​calculated. A first feature image is calculated based on each first adjusted segmentation image and a first weight matrix; a second feature image is calculated based on each second adjusted segmentation image and a second weight matrix; and a third feature image is calculated based on each third adjusted segmentation image and a third weight matrix. The first feature image, the second feature image, and the third feature image are weighted and summed to obtain a fused feature map.

[0011] Optionally, the step of calculating the deviation rate for each category based on the counts of each category in the first classification result and the counts of each category in the second classification result, and calibrating the counts of each category in the first classification result based on the deviation rate, includes: Calculate the area of ​​each first segmented image, and use the first segmented image with an area smaller than a first preset area as the image to be calibrated and counted; The category of the image to be calibrated is taken as the category of the count to be calibrated. The first number of the count to be calibrated is obtained from the first classification result, and the second number of the count to be calibrated is obtained from the second classification result. Based on the first quantity and the second quantity, the deviation rate corresponding to the category of the count to be calibrated is calculated. When the deviation rate is greater than a preset deviation rate threshold, the first quantity and the second quantity are weighted and summed to obtain the final count result of the category of the count to be calibrated. When the deviation rate is less than or equal to the preset deviation rate threshold, the first quantity is used as the final count result of the category of the count to be calibrated.

[0012] Optionally, each of the training segmentation maps includes a cue point grid, and the segmentation of each training image based on a preset image segmentation model yields multiple training segmentation maps, including: Each training image is segmented based on the preset image segmentation model to obtain an initial segmented training image; Calculate the area and aspect ratio of each initial segmentation training image, and delete the initial segmentation training images with an area smaller than the second preset area and the initial segmentation training images with an aspect ratio greater than the preset threshold to obtain multiple first intermediate segmentation training images. Calculate the intersection-union ratio (CIU) of every two first intermediate segmentation training images, and delete the first intermediate segmentation training image with the smaller area among the two first intermediate segmentation training images whose CIU is greater than the preset CIU threshold, to obtain multiple second intermediate segmentation training images. Calculate the area of ​​each second intermediate segmentation training image, and increase the number of cue point grids in the second intermediate segmentation training images whose area is smaller than the first preset area to obtain multiple training segmentation images.

[0013] According to another aspect of this application, an object classification and counting device in an image is provided, comprising: The training module is used to acquire multiple training images, segment each training image based on a preset image segmentation model to obtain multiple training segmentation maps, and use an unsupervised class partitioning method to train an initial object classification and counting model based on each training segmentation map to obtain a trained object classification and counting model. The first classification module is used to acquire an image to be identified with a preset first resolution, segment the image to be identified based on a preset image segmentation model to obtain multiple first segmented images, and input the multiple first segmented images into a trained object classification and counting model to obtain a first classification result. The multi-scale segmentation module is used to adjust the resolution of the image to be identified according to a preset second resolution and a preset third resolution to obtain a second image and a third image. The second image and the third image are then input into a preset image segmentation model to obtain multiple second segmented images and multiple third segmented images. The second classification module is used to fuse features from multiple first segmented images, multiple second segmented images, and multiple third segmented images to obtain a fused feature map. The fused feature map is then input into the trained object classification and counting model to obtain the second classification result. The calibration module is used to calculate the deviation rate corresponding to each category based on the count of each category in the first classification result and the count of each category in the second classification result, calibrate the count of each category in the first classification result based on the deviation rate, and take the calibrated first classification result as the final classification result.

[0014] Optionally, the training module is further configured to: The HDBSCAN algorithm is used to cluster multiple training segmentation maps under different clustering parameters to obtain multiple candidate clustering results. For each candidate clustering result, the contour coefficient of each training segmentation map is calculated, and the candidate clustering result with the largest contour coefficient is taken as the optimal clustering result of the training segmentation map, thus obtaining the pseudo-class corresponding to the training segmentation map. Based on each training segmentation map, a ground truth density map is generated for each pseudo-class. Based on each training segmentation map, the pseudo-class corresponding to each training segmentation map, and the ground truth density map corresponding to each pseudo-class, the initial object classification and counting model is trained to obtain the trained object classification and counting model.

[0015] Optionally, the training module is further configured to: The object classification and counting model extracts visual features and semantic features for each training segmentation map, and then fuses the visual features and semantic features corresponding to each training segmentation map to obtain the fused features corresponding to each training segmentation map. The object classification and counting model performs segmentation optimization and density prediction on each training segmentation map, and outputs multiple optimized segmentation maps and multiple predicted density maps; Based on multiple optimized segmentation maps, multiple predicted density maps, and the fusion features corresponding to each training segmentation map, a cost matrix is ​​constructed. Based on the cost matrix, the Hungarian algorithm is used to match the optimized segmentation maps and the predicted density maps. Loss is calculated based on the optimized segmentation map and predicted density map of the matching, and the parameters in the object classification and counting model are adjusted according to the loss value; The loss value is calculated using the following loss function:

[0016] in, The loss calculated by optimizing the intersection-union ratio (IoU) of the segmentation map and the training segmentation map for an object classification and counting model. Mean squared error loss for the predicted density map and the ground truth density map of an object classification counting model. For weights.

[0017] Optionally, the training module is further configured to: Obtain the bounding box of each training segmentation map, determine the center coordinates of each bounding box, and perform Gaussian distribution processing on the density of the training segmentation map based on the center coordinates of each bounding box to obtain the density Gaussian distribution map corresponding to each training segmentation map. For each pseudo-class, the density Gaussian distribution maps corresponding to each training segmentation map of the pseudo-class are superimposed to obtain the initial density map corresponding to each pseudo-class. Based on the density values ​​of the cue point grid in the initial density map corresponding to each pseudo-category, high sampling density region, medium sampling density region and low sampling density region are determined in the initial density map; The number of cue point grids in the high sampling density region is reduced, the number of cue point grids in the low sampling density region is increased, and the number of cue point grids in the medium sampling density region remains unchanged, thus obtaining the ground truth density map corresponding to each pseudo-category.

[0018] Optionally, the second classification module is further configured to: Each first segmented image, each second segmented image, and each third segmented image are processed by Gaussian distribution of density to obtain multiple first density maps, multiple second density maps, and multiple third density maps. Each first segmentation image, each second segmentation image, each third segmentation image, each first density map, each second density map, and each third density map are scale-aligned and adjusted to obtain multiple first adjusted segmentation images, multiple second adjusted segmentation images, multiple third adjusted segmentation images, multiple first adjusted density maps, multiple second adjusted density maps, and multiple third adjusted density maps. Based on the cross-attention mechanism, each first adjusted segmentation image and each first adjusted density map, a first weight matrix is ​​calculated; based on the cross-attention mechanism, each second adjusted segmentation image and each second adjusted density map, a second weight matrix is ​​calculated; based on the cross-attention mechanism, each third adjusted segmentation image and each third adjusted density map, a third weight matrix is ​​calculated. A first feature image is calculated based on each first adjusted segmentation image and a first weight matrix; a second feature image is calculated based on each second adjusted segmentation image and a second weight matrix; and a third feature image is calculated based on each third adjusted segmentation image and a third weight matrix. The first feature image, the second feature image, and the third feature image are weighted and summed to obtain a fused feature map.

[0019] Optionally, the calibration module is further configured to: Calculate the area of ​​each first segmented image, and use the first segmented image with an area smaller than a first preset area as the image to be calibrated and counted; The category of the image to be calibrated is taken as the category of the count to be calibrated. The first number of the count to be calibrated is obtained from the first classification result, and the second number of the count to be calibrated is obtained from the second classification result. Based on the first quantity and the second quantity, the deviation rate corresponding to the category of the count to be calibrated is calculated. When the deviation rate is greater than a preset deviation rate threshold, the first quantity and the second quantity are weighted and summed to obtain the final count result of the category of the count to be calibrated. When the deviation rate is less than or equal to the preset deviation rate threshold, the first quantity is used as the final count result of the category of the count to be calibrated.

[0020] Optionally, the training module is further configured to: Each training image is segmented based on the preset image segmentation model to obtain an initial segmented training image; Calculate the area and aspect ratio of each initial segmentation training image, and delete the initial segmentation training images with an area smaller than the second preset area and the initial segmentation training images with an aspect ratio greater than the preset threshold to obtain multiple first intermediate segmentation training images. Calculate the intersection-union ratio (CIU) of every two first intermediate segmentation training images, and delete the first intermediate segmentation training image with the smaller area among the two first intermediate segmentation training images whose CIU is greater than the preset CIU threshold, to obtain multiple second intermediate segmentation training images. Calculate the area of ​​each second intermediate segmentation training image, and increase the number of cue point grids in the second intermediate segmentation training images whose area is smaller than the first preset area to obtain multiple training segmentation images.

[0021] According to another aspect of this application, a storage medium is provided that stores at least one executable instruction, which causes a processor to perform operations corresponding to the object classification and counting method in the above-described image.

[0022] According to another aspect of this application, a computer device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the object classification and counting method in the above image.

[0023] By employing the above-described technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages: This application provides a method, apparatus, device, and medium for object classification and counting in images. It employs an unsupervised class partitioning method, training an initial object classification and counting model based on a training segmentation map. The image to be identified at a first scale is segmented, and the resulting first segmented image is input into the object classification and counting model to obtain a first classification result. The image to be identified is then resolution-converted to obtain a second image at a second scale and a third image at a third scale. These second and third images are then segmented to obtain multiple second and third segmented images. Feature fusion is performed on the segmented images at the first, second, and third scales, and the fused feature map is then input into the object classification and counting model to obtain a second classification result. Multi-scale feature fusion highlights the features of small target regions; therefore, the second classification result is used to calibrate the first classification result, improving the accuracy of small target counting. The unsupervised class partitioning method for training the object classification and counting model eliminates the need for labeled data and predefined categories, solving the category dependency problem of existing methods, which prevents the identification and counting of unlabeled categories, thus further improving the accuracy of object classification and counting.

[0024] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart of an object classification and counting method in an image provided by an embodiment of this application is shown; Figure 2This illustration shows another flowchart of an object classification and counting method in an image provided by an embodiment of this application; Figure 3 This paper shows another flowchart of an object classification and counting method in an image provided by an embodiment of the present application; Figure 4 This illustration shows a block diagram of an object classification and counting device in an image provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown.

[0026] in, Figure 4 In Chinese: 402 - Training module; 404 - First classification module; 406 - Multi-scale segmentation module; 408 - Second classification module; 410 - Calibration module; Figure 5 In Chinese: 502 - Processor; 504 - Communication interface; 506 - Memory; 508 - Communication bus; 510 - Program. Detailed Implementation

[0027] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present invention can be combined with each other.

[0028] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the specific embodiments, structures, features, and effects according to the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "an embodiment" or "an embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0029] To address the problem that current methods for classifying and counting multi-category objects in images can accurately identify and count labeled objects, but cannot identify and count unlabeled objects, this application provides a method for classifying and counting objects in images, such as... Figure 1 As shown, the method includes: 102: Acquire multiple training images, segment each training image based on a preset image segmentation model to obtain multiple training segmentation maps, and use an unsupervised class partitioning method to train the initial object classification and counting model based on each training segmentation map to obtain the trained object classification and counting model; 104: Obtain the image to be recognized with a preset first resolution, segment the image to be recognized based on the image segmentation model to obtain multiple first segmented images, input the multiple first segmented images into the trained object classification and counting model to obtain the first classification result; 106: Adjust the resolution of the image to be recognized according to the preset second resolution and the preset third resolution respectively to obtain the second image and the third image. Input the second image and the third image into the image segmentation model respectively to obtain multiple second segmentation images and multiple third segmentation images; 108: Fuse features from multiple first segmentation images, multiple second segmentation images, and multiple third segmentation images to obtain a fused feature map. Input the fused feature map into the trained object classification and counting model to obtain the second classification result. 110: Based on the counts of each category in the first classification result and the counts of each category in the second classification result, calculate the deviation rate corresponding to each category. Based on the deviation rate, calibrate the counts of each category in the first classification result, and take the calibrated first classification result as the final classification result.

[0030] Specifically, traditional image object classification and counting methods rely on manual annotation of objects in the image. For small targets, manual annotation may sometimes miss some, thus affecting the final classification and counting results. This application uses an unsupervised class partitioning method to determine the classification of each target in the image.

[0031] First, each training image is segmented using a pre-defined image segmentation model. The training images are images with a first resolution and cropped to a preset size (672×672). The image segmentation model used is the SAM model (Segment Anything Model). After the SAM model segments the training image into multiple training segmentation maps, an unsupervised class partitioning method is used to cluster each training segmentation map, obtaining pseudo-classes for each segmentation map. In supervised learning, the training images are labeled with tags indicating the object's category; in unsupervised learning, the training images are not labeled, and pseudo-labels (pseudo-classes) are obtained through the unsupervised class partitioning method. Based on the training segmentation maps and their corresponding pseudo-classes, an object classification and counting model is trained, resulting in the trained object classification and counting model.

[0032] The image to be identified is an image with a first resolution and cropped to a preset size. This image is input into a preset image segmentation model, which outputs multiple first segmented images. These first segmented images are then input into a trained object classification and counting model to obtain the first classification result. When the object classification and counting model performs classification and counting on an image with a single resolution, small target regions may be missed. Therefore, the first classification result is calibrated for small targets.

[0033] The image to be identified is converted from a first resolution image to a second resolution image and a third resolution image. The second resolution image is for small targets, and the third resolution image is for large targets. The second image is input into the image segmentation model to obtain multiple second segmented images, and the third image is input into the image segmentation model to obtain multiple third segmented images. The first, second, and third segmented images represent different scales, so feature fusion is performed on the first, second, and third segmented images. An attention mechanism is used during feature fusion, so the fused feature map can reflect global features and local features of small targets. The fused feature map is input into the trained object classification and counting model to obtain the second classification result. For the category of small targets, the deviation rate between the count of that category in the first classification result and the count of that category in the second classification result is calculated. Based on the magnitude of the deviation rate, it is determined whether the count of small targets needs to be calibrated. If calibration is required, the count of small targets in the first classification result is calibrated, while the counts of other categories in the first classification result remain unchanged. The calibrated first classification result is used as the final classification result.

[0034] This application provides a method for object classification and counting in images. Compared with existing technologies, it adopts an unsupervised class partitioning method, trains an initial object classification and counting model based on a training segmentation map, segments the image to be identified at a first scale, and then inputs the obtained first segmented image into the object classification and counting model to obtain a first classification result. The resolution of the image to be identified is converted to obtain a second image at a second scale and a third image at a third scale. The second and third images are segmented separately to obtain multiple second and third segmented images. The segmented images at the first, second, and third scales are fused, and the fused feature map is then input into the object classification and counting model to obtain a second classification result. Multi-scale feature fusion can highlight the features of small target regions. Therefore, the second classification result is used to calibrate the first classification result, improving the accuracy of small target counting. The object classification and counting model is trained using an unsupervised class partitioning method, which does not require labeled data or predefined categories, solving the problem of category dependence in existing methods, which makes it impossible to identify and count unlabeled categories, and further improving the accuracy of object classification and counting.

[0035] In one embodiment of the present invention, such as Figure 2 As shown, an unsupervised class partitioning method is used to train the initial object classification and counting model based on each training segmentation map, resulting in a trained object classification and counting model, including: 202: The HDBSCAN algorithm is used to cluster multiple training segmentation maps under different clustering parameters to obtain multiple candidate clustering results; 204: For each candidate clustering result, calculate the silhouette coefficient of each training segmentation map, and take the candidate clustering result with the largest silhouette coefficient as the optimal clustering result of the training segmentation map to obtain the pseudo-class corresponding to the training segmentation map; 206: Based on each training segmentation map, generate a ground truth density map corresponding to each pseudo-class. Based on each training segmentation map, the pseudo-class corresponding to each training segmentation map, and the ground truth density map corresponding to each pseudo-class, train the initial object classification and counting model to obtain the trained object classification and counting model.

[0036] Specifically, the unsupervised classification method uses the HDBSCAN algorithm (Hierarchical Density-Based Spatial Clustering of Applications with Noise), which automatically adapts to the category distribution without requiring manual setting of the number of categories. To obtain accurate classification results, a silhouette coefficient filtering mechanism is introduced. The training segmentation map is clustered under different clustering parameters, and the silhouette coefficient of each training segmentation map under different clustering parameters is calculated. The clustering result with the largest silhouette coefficient is selected as the optimal clustering result, which is the pseudo-category corresponding to the training segmentation map.

[0037] The core clustering parameters of the HDBSCAN algorithm include: min_samples (minimum number of samples): The minimum number of samples required to form a "core point" (default 5, adjustable range 2~20), which affects the density requirements of the cluster; min_cluster_size (minimum cluster size): minimum cluster size (default 5, adjustable range 2~30). Clusters smaller than this size will be considered noise. cluster_selection_epsilon (cluster selection distance threshold): The maximum distance threshold for cluster merging (default None, adjustable range 0.1~1.0). Clusters exceeding this distance will not be merged. Metric: Distance metric (default Euclidean distance, optional cosine distance, Manhattan distance, etc.), adaptable to distance calculation needs with different characteristics.

[0038] The contour coefficients of each training segmentation map are calculated using the following method:

[0039] in, For the first The average distance between a training segmentation map and all other training segmentation maps in the same cluster (intra-cluster similarity, the smaller the better). For the first The minimum average distance between a training segmentation map and all other training segmentation maps in all clusters (inter-cluster separation, the larger the better). They are and The maximum value in.

[0040] During the training phase, the central feature vector of each cluster is calculated to construct a "prototype cluster library" that stores information such as cluster central features, average area, and typical semantics. For the clustering results of newly input training segmentation images, the cosine similarity with the prototype cluster is calculated. If the similarity is >0.85, the prototype cluster library is reused; otherwise, it is determined as a new category and the prototype cluster library is updated.

[0041] Based on multiple training segmentation maps corresponding to each pseudo-class, a ground truth density map corresponding to that pseudo-class is generated. The ground truth density map is used as the true value of the predicted density map of the object classification and counting model. The pseudo-class corresponding to each training segmentation map is used as the true value of the classification result of the object classification and counting model. The object classification and counting model is trained based on the training segmentation map, its corresponding pseudo-class, and the predicted density map corresponding to the pseudo-class, resulting in the trained object classification and counting model.

[0042] In this embodiment, each training segmentation image includes multiple cue point grids. Each training image is segmented based on a preset image segmentation model to obtain multiple training segmentation images. This includes: segmenting each training image based on the preset image segmentation model to obtain initial segmentation training images; calculating the area and aspect ratio of each initial segmentation training image, and deleting initial segmentation training images with an area smaller than a second preset area and initial segmentation training images with an aspect ratio greater than a preset threshold to obtain multiple first intermediate segmentation training images; calculating the intersection-union ratio (IUGR) of every two first intermediate segmentation training images, and deleting the smaller first intermediate segmentation training image from the two first intermediate segmentation training images with an IUGR greater than a preset IUGR threshold to obtain multiple second intermediate segmentation training images; calculating the area of ​​each second intermediate segmentation training image, and increasing the number of cue point grids in the second intermediate segmentation training images with areas smaller than the first preset area to obtain multiple training segmentation images.

[0043] Specifically, a pre-trained image segmentation model is called to segment each training image, obtaining initial segmentation training images, such as the SAM model. The area and aspect ratio of each initial segmentation training image are calculated, and deletions are made based on the area and aspect ratio. For example, small segmentation results with an area < 672 × 672 (normalized image area) × 0.1% (approximately 45 pixels) are deleted, as well as distorted images with an aspect ratio > 10 or < 0.1, resulting in multiple first intermediate segmentation training images. For the retained first intermediate segmentation training images, the intersection-union ratio (IUGR) of any two first intermediate segmentation training images is calculated, and retention is determined based on the IUGR. For example, if there are two first intermediate segmentation training images with an IUGR > 0.7, the first intermediate segmentation training image with the larger area is retained, and the first intermediate segmentation training image with the smaller area is deleted, resulting in second intermediate segmentation training images.

[0044] A dynamic density grid cue strategy is employed. The area of ​​each second intermediate segmentation training image is calculated. For second intermediate segmentation training images with areas smaller than a first preset area, the number of sampling points is increased. For example, for small target regions (area < 1% of the total image area), the number of sampling points is doubled (step size 8-32 pixels). For large target regions, the default number of sampling points is maintained (step size 16-64 pixels). This ensures that small targets are not missed and large targets are not redundant, generating initial instance segmentation and obtaining multiple training segmentation images. The cue point grid serves as point prompts. When the segmentation training images are input to the object classification and counting model, the model can perform segmentation based on these cue point grids.

[0045] In this embodiment, generating a ground truth density map for each pseudo-class based on each training segmentation map includes: obtaining the bounding box of each training segmentation map, determining the center coordinates of each bounding box, and performing Gaussian distribution processing on the training segmentation map based on the center coordinates of each bounding box to obtain a Gaussian density distribution map for each training segmentation map; for each pseudo-class, superimposing the Gaussian density distribution maps for each training segmentation map corresponding to the pseudo-class to obtain an initial density map for each pseudo-class; determining high-sampling density regions, medium-sampling density regions, and low-sampling density regions in the initial density map based on the density values ​​of the cue point grids in the initial density map for each pseudo-class; reducing the number of cue point grids in the high-sampling density region, increasing the number of cue point grids in the low-sampling density region, and keeping the number of cue point grids in the medium-sampling density region unchanged to obtain a ground truth density map for each pseudo-class.

[0046] Specifically, the generation of a true density map essentially involves "converting discrete target locations into a continuous density distribution," and its core relies on a density regressor. The specific process is as follows: For each pseudo-class, collect the bounding boxes of each training segmentation map corresponding to that pseudo-class, and extract the center coordinates of each bounding box.

[0047] The core principle of the density regressor is that it assumes the density distribution of each target follows a two-dimensional Gaussian function, with the highest density value at the target center and gradually decreasing towards the edges; the standard deviation of the Gaussian kernel is positively correlated with the target area, ensuring a reasonable density distribution for targets of different sizes. Therefore, based on the center coordinates of each bounding box, a Gaussian density distribution is applied to each training segmentation map to obtain a density Gaussian distribution map corresponding to each training segmentation map. For each pseudo-class, the density Gaussian distribution maps corresponding to each training segmentation map for that pseudo-class are superimposed to obtain the initial density map corresponding to that pseudo-class.

[0048] Each training segmentation map contains multiple sampling points. When the training segmentation map is processed with a Gaussian density distribution, the sampling points are also processed with a Gaussian density distribution. The density value of the cue point grid in the initial density map corresponding to each pseudo-class is calculated. Based on the density value of the cue point grid, high sampling density area, medium sampling density area and low sampling density area are determined in the initial density map. The high sampling density area is the target dense area, the medium sampling density area is the target sparse area, and the low sampling density area is the background.

[0049] Reduce the number of prompt point grids in high sampling density areas (halve the step size) to increase the prompt signal density; maintain the default number of prompt point grids in medium sampling density areas (keep the step size unchanged); increase the number of prompt point grids in low sampling density areas (double the step size) to weaken the prompt signal and avoid background missegmentation.

[0050] In this embodiment, training the initial object classification and counting model includes: the object classification and counting model extracts visual and semantic features for each training segmentation map, and fuses the visual and semantic features corresponding to each training segmentation map to obtain the fused features corresponding to each training segmentation map; the object classification and counting model optimizes segmentation and predicts density for each training segmentation map, and outputs multiple optimized segmentation maps and multiple predicted density maps; based on the multiple optimized segmentation maps, multiple predicted density maps, and the fused features corresponding to each training segmentation map, a cost matrix is ​​constructed; based on the cost matrix, the Hungarian algorithm is used to match the optimized segmentation maps and predicted density maps; loss is calculated based on the matched optimized segmentation maps and predicted density maps, and the parameters in the object classification and counting model are adjusted according to the loss value.

[0051] The loss value is calculated using the following loss function:

[0052] in, The loss calculated by optimizing the intersection-union ratio (IoU) of the segmentation map and the training segmentation map for an object classification and counting model. Mean squared error loss for the predicted density map and the ground truth density map of an object classification counting model. For weights.

[0053] Specifically, when training the object classification and counting model, visual features and semantic features are extracted for each training segmentation map. Visual features can reflect the appearance and shape information of the target, while semantic features can reflect the semantic attributes of the target. The two types of feature vectors are concatenated to obtain the fused features.

[0054] The object classification and counting model yields optimized segmentation maps and predicted density maps. These object groups have the same number of objects, which may lead to mismatch issues. To improve the accuracy of the model, it is necessary to optimize the matching of these object groups. This application uses the Hungarian algorithm for matching.

[0055] First, construct the unsupervised cost matrix, with matrix elements defined as follows: ,in: For the first The predicted density map of the first category and the first category The intersection-union ratio (IUU) of the optimized segmentation graphs for each category reflects the degree of spatial overlap. For the first Predicted density plots for each category, For the first Optimized segmentation maps for each category; For feature similarity, For the first The fusion features corresponding to the first segmentation map of each category For the first The fusion features corresponding to the first segmentation map of each category are obtained by calculating cosine similarity. .

[0056] Calculate the feature distance variance for each pseudo-class. Matching threshold The larger the variance (the greater the intra-cluster differences), the higher the threshold should be to avoid false matches.

[0057] Based on the cost matrix and matching threshold, the optimized segmentation map and prediction density map are optimally matched to achieve a one-to-one correspondence between pseudo-classes and prediction counts.

[0058] The Hungarian algorithm avoids dependence on manual thresholds, reduces the root mean square error of the counting results by 18%-22%, and significantly improves stability.

[0059] In one embodiment of the present invention, such as Figure 3 As shown, multiple first segmented images, multiple second segmented images, and multiple third segmented images are fused to obtain a fused feature map, including: 302: Perform Gaussian distribution processing on each first segmented image, each second segmented image, and each third segmented image to obtain multiple first density maps, multiple second density maps, and multiple third density maps; 304: Each first segmentation image, each second segmentation image, each third segmentation image, each first density map, each second density map, and each third density map are scale aligned and adjusted to obtain multiple first adjusted segmentation images, multiple second adjusted segmentation images, multiple third adjusted segmentation images, multiple first adjusted density maps, multiple second adjusted density maps, and multiple third adjusted density maps; 306: Based on the cross-attention mechanism, each first adjusted segmentation image and each first adjusted density map, a first weight matrix is ​​calculated; based on the cross-attention mechanism, each second adjusted segmentation image and each second adjusted density map, a second weight matrix is ​​calculated; based on the cross-attention mechanism, each third adjusted segmentation image and each third adjusted density map, a third weight matrix is ​​calculated. 308: A first feature image is calculated based on each first adjusted segmentation image and a first weight matrix; a second feature image is calculated based on each second adjusted segmentation image and a second weight matrix; and a third feature image is calculated based on each third adjusted segmentation image and a third weight matrix. 310: The first feature image, the second feature image, and the third feature image are weighted and summed to obtain the fused feature map.

[0060] Specifically, the image to be identified is a cropped image, usually cropped to 1× (672×672, medium resolution). The resolution of the image to be identified is adjusted to generate a 0.5× (336×336, low resolution) image and a 2× (1344×1344, high resolution) image, namely the second and third images. These are then added to the original 1× (672×672, original resolution) image, resulting in three scales. The low resolution focuses on capturing the overall information of large targets, while the high resolution focuses on the details of small targets. However, large targets are easier to capture and have higher counting accuracy, so only the counting of small targets needs to be calibrated.

[0061] The second and third images are input into the image segmentation model to obtain multiple second segmented images and multiple third segmented images. Gaussian distribution processing is applied to the density of each first segmented image, each second segmented image, and each third segmented image to obtain multiple first density maps, multiple second density maps, and multiple third density maps. Scale alignment adjustment is performed to obtain multiple scale-aligned first adjusted segmented images, multiple second adjusted segmented images, multiple third adjusted segmented images, multiple first adjusted density maps, multiple second adjusted density maps, and multiple third adjusted density maps.

[0062] A cross-attention mechanism is used to achieve refined fusion. The specific method is as follows: The adjusted segmentation image and adjusted density map at each scale are input into the cross-attention module to calculate the attention weight matrix corresponding to each scale. For each scale, the adjusted segmentation image at that scale is weighted and summed according to its corresponding attention weight matrix to obtain the feature image corresponding to that scale. The feature images at each scale are weighted and summed to obtain the fused feature map. This fused feature map can highlight the features of small targets and suppress background interference, thereby obtaining a more accurate small target count.

[0063] In one embodiment of the invention, based on the counts of each category in the first classification result and the counts of each category in the second classification result, a deviation rate corresponding to each category is calculated. Based on the deviation rate, the counts of each category in the first classification result are calibrated, including: Calculate the area of ​​each first segmented image, and use the first segmented image with an area smaller than a first preset area as the image to be calibrated and counted; The category of the image to be calibrated is taken as the category of the count to be calibrated. The first number of the count to be calibrated is obtained from the first classification result, and the second number of the count to be calibrated is obtained from the second classification result. Based on the first quantity and the second quantity, the deviation rate corresponding to the category of the count to be calibrated is calculated. When the deviation rate is greater than the preset deviation rate threshold, the first quantity and the second quantity are weighted and summed to obtain the final count result of the category of the count to be calibrated. When the deviation rate is less than or equal to the preset deviation rate threshold, the first quantity is used as the final count result of the category of the count to be calibrated.

[0064] Specifically, small target regions have a low pixel ratio, making it easy for object classification and counting models to miss detections (e.g., small targets are occluded by the background or overlap with other targets). In addition, based on a single-scale (based on the first resolution) segmentation map, the features of small targets are easily submerged by large targets or the background. When classifying, object classification and counting models may misclassify small targets of different categories as the same category or miss small targets of the same category. Therefore, it is necessary to calibrate the counting of small targets.

[0065] The second classification result incorporates multi-scale features, especially high-resolution small target details and small target density maps, making the classification and counting of small targets more accurate. Therefore, the small target count in the first classification result is calibrated based on the small target count in the second classification result.

[0066] First, the category containing the small target in the first classification result is taken as the category to be calibrated. The first number of items contained in the category to be calibrated is obtained from the first classification result, and the second number of items contained in the category to be calibrated is obtained from the second classification result. The deviation rate corresponding to the category to be calibrated is calculated using the following formula:

[0067] in, As the first quantity, For the second quantity, Let the deviation rate be... If the deviation exceeds 30%, then weighted fusion calibration will be used. The final count result of the category to be calibrated is obtained. If the deviation rate is less than or equal to 0.3, no calibration is required, and the first count is used as the final count result of the category to be calibrated. The categories and counts of other objects in the first classification result remain unchanged. The accuracy of small target counting is improved through calibration.

[0068] Taking unsupervised multi-category inventory management of retail goods in a smart supermarket as an example, with the server as the execution entity, traditional supermarket inventory management relies on manual counting of each item, which is inefficient (it takes 8-12 hours to inventory a 1000㎡ supermarket) and prone to errors (manual missed detection / false judgment rate of about 5%-8%). Furthermore, new products require re-entry of category information after being put on the shelves, making rapid adaptation impossible. Existing automated inventory management solutions require predefined product categories and rely on a large amount of labeled data, resulting in high costs for adapting to new products. Small-packaged goods (such as chewing gum and stationery) are easily missed due to their indistinct features. The unsupervised multi-category counting method proposed in this application can perfectly solve the above problems, achieving highly efficient inventory management with "zero labeling, automatic classification, and accurate counting".

[0069] Supermarket staff use mobile inventory devices equipped with high-definition cameras (such as tablets with stands) to capture images of the shelf area (original resolution 1080×1080) to obtain initial images. These initial images are then transmitted to a server, which crops the initial 672×672 images, removing irrelevant background elements (such as shelf edges and the floor) to ensure the processing area focuses on the merchandise, resulting in training images. A pre-defined image segmentation model is then used to segment the training images, producing multiple initial segmentation training images. A dynamic cue point grid strategy is then employed to adjust these initial segmentation training images. For example, for small packages (such as chewing gum boxes, with an area less than 1% of the total area of ​​the 672×672 image), the cue point grid step size is reduced from 16 pixels to 8 pixels to ensure no small targets are missed. For large packages (such as laundry detergent bottles), the step size remains at 16 pixels to avoid redundant calculations. Then, debris with an area less than 45 pixels, deformed objects with an aspect ratio greater than 10 (such as packaging wrinkles), and duplicate segmentation regions with a merging ratio greater than 0.7 (such as overlapping parts of stacked merchandise) are removed, resulting in multiple training segmentation images.

[0070] The HDBSCAN algorithm is used to perform unsupervised clustering on the training segmentation map under various parameters: visual features (such as shape and color) and semantic features (such as packaging texture) of the products are extracted, the contour coefficients after classification under each parameter are calculated, the optimal clustering parameters are selected by contour coefficients, pseudo-categories are obtained, and the products on the shelf are automatically divided into pseudo-categories such as beverages, snacks, stationery, and daily necessities. New products (such as newly launched limited-edition snacks) do not need to be manually labeled and are automatically assigned to the new pseudo-categories.

[0071] The server generates an independent ground truth density map for each pseudo-category: taking the beverage category as an example, a Gaussian density map is generated based on the center coordinates of the bounding box of each beverage bottle. The density value is high in dense areas of the bottle body (such as where beverages are stacked) and low in the background area. The distribution of cue points is adjusted according to the density map: the cue point grid is densified in the dense beverage area (high response area) and the cue points in the background area (low response area) are simplified to optimize the segmentation boundary and avoid segmentation ambiguity caused by product stacking.

[0072] Based on each training segmentation map, the pseudo-class corresponding to each training segmentation map, and the ground truth density map corresponding to each pseudo-class, the initial object classification and counting model is trained to obtain the trained object classification and counting model.

[0073] The system acquires images of goods to be identified in a supermarket, segments these images using an image segmentation model to obtain multiple first segmented images, and inputs these first segmented images into a trained object classification and counting model to obtain the first classification result.

[0074] The images to be recognized are scaled, for example, generating images at three scales: 0.5×(336×336), 1×(672×672), and 2×(1344×1344). Multi-scale features are fused through a cross-attention mechanism to obtain a fused feature map. The fused feature map is then input into the trained object classification and counting model to obtain the second classification result. The high-resolution image highlights the details of small packaged stationery, while the low-resolution image captures the overall information of large packaged daily necessities, thus strengthening the feature expression of small targets.

[0075] The server performs an initial count for each pseudo-category: for large targets such as beverages and daily necessities, it directly outputs the first classification result; for small targets such as stationery and chewing gum, it calibrates the first count based on the deviation rate between the first count in the first classification result and the second count in the second classification result. Calculate the deviation rate: If the first count of a certain small target category ( =28) and the second count ( =35) Deviation rate =25% < 30%, output directly =28; if the deviation rate of another category =32% > 30%, then pass =0.7×28+0.3×35=29.1, rounded down to 29 to ensure accurate counting.

[0076] The final inventory results are as follows: a category count report is generated, which includes 32 items in beverages, 45 items in snacks, 29 items in stationery, 18 items in daily necessities, and 7 items in new products. The report also includes images of typical products in each category (e.g., cola bottles for beverages and potato chip bags for snacks) for staff to check.

[0077] This application has the following advantages over traditional classification and counting methods: 1. Efficiency Improvement: Inventory time for a 1000㎡ supermarket is reduced to 1-2 hours, improving efficiency by more than 8 times; 2. Zero labeling cost: New product listings do not require manual entry of category information; the system automatically identifies and categorizes products, reducing adaptation costs by 90%. 3. Accurate counting: The missed detection rate for small target items is <1%, and the overall counting error is <2%, which is better than manual and existing automated solutions; 4. High generalization: It can adapt to different supermarket shelf layouts and product packaging styles without the need for fine-tuning the model for specific scenarios.

[0078] Furthermore, as a response to the above Figure 1 The implementation of the method shown in this invention provides an object classification and counting device in an image, such as... Figure 4 As shown, the device includes: Training module 402 is used to acquire multiple training images, segment each training image based on a preset image segmentation model to obtain multiple training segmentation maps, and use an unsupervised class division method to train an initial object classification and counting model based on each training segmentation map to obtain a trained object classification and counting model. The first classification module 404 is used to acquire an image to be identified with a preset first resolution, segment the image to be identified based on a preset image segmentation model to obtain multiple first segmented images, and input the multiple first segmented images into the trained object classification and counting model to obtain the first classification result. The multi-scale segmentation module 406 is used to adjust the resolution of the image to be identified according to a preset second resolution and a preset third resolution to obtain a second image and a third image, and input the second image and the third image into a preset image segmentation model to obtain multiple second segmented images and multiple third segmented images; The second classification module 408 is used to fuse features of multiple first segmentation images, multiple second segmentation images and multiple third segmentation images to obtain a fused feature map, and input the fused feature map into the trained object classification and counting model to obtain the second classification result. The calibration module 410 is used to calculate the deviation rate corresponding to each category based on the count of each category in the first classification result and the count of each category in the second classification result, and to calibrate the count of each category in the first classification result based on the deviation rate, and to take the calibrated first classification result as the final classification result.

[0079] This application provides an object classification and counting device for images. Compared with existing technologies, it adopts an unsupervised class partitioning method, trains an initial object classification and counting model based on a training segmentation map, segments the image to be identified at a first scale, and then inputs the obtained first segmented image into the object classification and counting model to obtain a first classification result. The resolution of the image to be identified is converted to obtain a second image at a second scale and a third image at a third scale. The second and third images are segmented separately to obtain multiple second and third segmented images. The segmented images at the first, second, and third scales are fused, and the fused feature map is then input into the object classification and counting model to obtain a second classification result. Multi-scale feature fusion can highlight the features of small target regions. Therefore, the second classification result is used to calibrate the first classification result, improving the accuracy of small target counting. The unsupervised class partitioning method is used to train the object classification and counting model, which does not require labeled data or predefined categories, solving the problem of category dependence in existing methods, which makes it impossible to identify and count unlabeled categories, and further improving the accuracy of object classification and counting.

[0080] In one embodiment, the training module is also used for: The HDBSCAN algorithm is used to cluster multiple training segmentation maps under different clustering parameters to obtain multiple candidate clustering results. For each candidate clustering result, the silhouette coefficient of each training segmentation map is calculated. The candidate clustering result with the largest silhouette coefficient is taken as the optimal clustering result of the training segmentation map, and the pseudo-class corresponding to the training segmentation map is obtained. Based on each training segmentation map, a ground truth density map is generated for each pseudo-class. Based on each training segmentation map, the pseudo-class corresponding to each training segmentation map, and the ground truth density map corresponding to each pseudo-class, the initial object classification and counting model is trained to obtain the trained object classification and counting model.

[0081] In one embodiment, the training module is also used for: The object classification and counting model extracts visual features and semantic features for each training segmentation map, and then fuses the visual features and semantic features corresponding to each training segmentation map to obtain the fused features corresponding to each training segmentation map. The object classification and counting model performs segmentation optimization and density prediction on each training segmentation map, and outputs multiple optimized segmentation maps and multiple predicted density maps; Based on multiple optimized segmentation maps, multiple predicted density maps, and the fusion features corresponding to each training segmentation map, a cost matrix is ​​constructed. Based on the cost matrix, the Hungarian algorithm is used to match the optimized segmentation maps and the predicted density maps. Loss is calculated based on the optimized segmentation map and predicted density map of the matching, and the parameters in the object classification and counting model are adjusted according to the loss value; The loss value is calculated using the following loss function:

[0082] in, The loss calculated by optimizing the intersection-union ratio (IoU) of the segmentation map and the training segmentation map for an object classification and counting model. Mean squared error loss for the predicted density map and the ground truth density map of an object classification counting model. For weights.

[0083] In one embodiment, the training module is also used for: Obtain the bounding box of each training segmentation map, determine the center coordinates of each bounding box, and perform Gaussian distribution processing on the density of each training segmentation map based on the center coordinates of each bounding box to obtain the density Gaussian distribution map corresponding to each training segmentation map. For each pseudo-class, the density Gaussian distribution maps corresponding to each training segmentation map of the pseudo-class are superimposed to obtain the initial density map corresponding to each pseudo-class. Based on the density values ​​of the cue point grid in the initial density map corresponding to each pseudo-category, determine the high sampling density region, medium sampling density region, and low sampling density region in the initial density map; The number of cue point grids in the high sampling density region is reduced, the number of cue point grids in the low sampling density region is increased, and the number of cue point grids in the medium sampling density region remains unchanged, thus obtaining the ground truth density map corresponding to each pseudo-class.

[0084] In one embodiment, the second classification module is further used for: Each first segmented image, each second segmented image, and each third segmented image are processed by Gaussian distribution of density to obtain multiple first density maps, multiple second density maps, and multiple third density maps. Each first segmentation image, each second segmentation image, each third segmentation image, each first density map, each second density map, and each third density map are scale-aligned and adjusted to obtain multiple first adjusted segmentation images, multiple second adjusted segmentation images, multiple third adjusted segmentation images, multiple first adjusted density maps, multiple second adjusted density maps, and multiple third adjusted density maps. Based on the cross-attention mechanism, each first adjusted segmentation image and each first adjusted density map, a first weight matrix is ​​calculated; based on the cross-attention mechanism, each second adjusted segmentation image and each second adjusted density map, a second weight matrix is ​​calculated; based on the cross-attention mechanism, each third adjusted segmentation image and each third adjusted density map, a third weight matrix is ​​calculated. A first feature image is calculated based on each first adjusted segmentation image and a first weight matrix; a second feature image is calculated based on each second adjusted segmentation image and a second weight matrix; and a third feature image is calculated based on each third adjusted segmentation image and a third weight matrix. The first feature image, the second feature image, and the third feature image are weighted and summed to obtain the fused feature map.

[0085] In one embodiment, the calibration module is also used for: Calculate the area of ​​each first segmented image, and use the first segmented image with an area smaller than a first preset area as the image to be calibrated and counted; The category of the image to be calibrated is taken as the category of the count to be calibrated. The first number of the count to be calibrated is obtained from the first classification result, and the second number of the count to be calibrated is obtained from the second classification result. Based on the first quantity and the second quantity, the deviation rate corresponding to the category of the count to be calibrated is calculated. When the deviation rate is greater than the preset deviation rate threshold, the first quantity and the second quantity are weighted and summed to obtain the final count result of the category of the count to be calibrated. When the deviation rate is less than or equal to the preset deviation rate threshold, the first quantity is used as the final count result of the category of the count to be calibrated.

[0086] In one embodiment, the training module is also used for: Each training image is segmented based on a preset image segmentation model to obtain an initial segmented training image; Calculate the area and aspect ratio of each initial segmentation training image, and delete the initial segmentation training images with an area smaller than the second preset area and the initial segmentation training images with an aspect ratio greater than the preset threshold to obtain multiple first intermediate segmentation training images. Calculate the intersection-union ratio (CIU) of every two first intermediate segmentation training images, and delete the first intermediate segmentation training image with the smaller area among the two first intermediate segmentation training images whose CIU is greater than the preset CIU threshold, to obtain multiple second intermediate segmentation training images. Calculate the area of ​​each second intermediate segmentation training image, and increase the number of sampling points in the second intermediate segmentation training images whose area is smaller than the first preset area to obtain multiple training segmentation images.

[0087] According to one embodiment of the present invention, a storage medium is provided, the storage medium storing at least one executable instruction, the computer-executable instruction being able to execute the object classification and counting method in the image in any of the above method embodiments.

[0088] Figure 5 The diagram illustrates a structural schematic of a computer device according to an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0089] like Figure 5 As shown, the computer device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0090] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0091] Communication interface 504 is used to communicate with other network elements such as clients or other servers.

[0092] The processor 502 is used to execute program 510, specifically to execute the relevant steps in the above-described embodiment of the object classification and counting method in the image.

[0093] Specifically, program 510 may include program code that includes computer operation instructions.

[0094] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0095] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0096] Specifically, program 510 can be used to cause processor 502 to perform the following operations: Multiple training images are acquired, and each training image is segmented based on a preset image segmentation model to obtain multiple training segmentation maps. An unsupervised class partitioning method is used to train an initial object classification and counting model based on each training segmentation map to obtain a trained object classification and counting model. Obtain an image to be identified with a preset first resolution, segment the image to be identified based on an image segmentation model to obtain multiple first segmented images, and input the multiple first segmented images into a trained object classification and counting model to obtain the first classification result; The image to be identified is adjusted according to the preset second resolution and the preset third resolution to obtain the second image and the third image. The second image and the third image are then input into the image segmentation model to obtain multiple second segmentation images and multiple third segmentation images. Multiple first segmentation images, multiple second segmentation images, and multiple third segmentation images are fused to obtain a fused feature map. The fused feature map is then input into the trained object classification and counting model to obtain the second classification result. Based on the counts of each category in the first classification result and the counts of each category in the second classification result, the deviation rate corresponding to each category is calculated. Based on the deviation rate, the counts of each category in the first classification result are calibrated, and the calibrated first classification result is taken as the final classification result.

[0097] It will be apparent to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. In one embodiment, they can be implemented using device-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0098] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A method for classifying and counting objects in an image, characterized in that, include: Multiple training images are acquired, and each training image is segmented based on a preset image segmentation model to obtain multiple training segmentation maps. An unsupervised class division method is used to train an initial object classification and counting model based on each training segmentation map to obtain a trained object classification and counting model. A target image with a preset first resolution is obtained, and the target image is segmented based on the image segmentation model to obtain multiple first segmented images. The multiple first segmented images are then input into a trained object classification and counting model to obtain a first classification result. The image to be identified is adjusted according to a preset second resolution and a preset third resolution to obtain a second image and a third image. The second image and the third image are then input into the image segmentation model to obtain multiple second segmented images and multiple third segmented images. Multiple first segmented images, multiple second segmented images, and multiple third segmented images are fused to obtain a fused feature map. The fused feature map is then input into a trained object classification and counting model to obtain a second classification result. Based on the counts of each category in the first classification result and the counts of each category in the second classification result, the deviation rate corresponding to each category is calculated. Based on the deviation rate, the counts of each category in the first classification result are calibrated, and the calibrated first classification result is taken as the final classification result.

2. The method of claim 1, wherein, The method employs an unsupervised class partitioning approach, training the initial object classification and counting model based on each training segmentation map to obtain the trained object classification and counting model, including: The HDBSCAN algorithm is used to cluster multiple training segmentation maps under different clustering parameters to obtain multiple candidate clustering results. For each candidate clustering result, the contour coefficient of each training segmentation map is calculated, and the candidate clustering result with the largest contour coefficient is taken as the optimal clustering result of the training segmentation map, thus obtaining the pseudo-class corresponding to the training segmentation map. Based on each training segmentation map, a ground truth density map is generated for each pseudo-class. Based on each training segmentation map, the pseudo-class corresponding to each training segmentation map, and the ground truth density map corresponding to each pseudo-class, the initial object classification and counting model is trained to obtain the trained object classification and counting model.

3. The method of claim 2, wherein, Training the initial object classification and counting model includes: The object classification and counting model extracts visual features and semantic features for each training segmentation map, and then fuses the visual features and semantic features corresponding to each training segmentation map to obtain the fused features corresponding to each training segmentation map. The object classification and counting model performs segmentation optimization and density prediction on each training segmentation map, and outputs multiple optimized segmentation maps and multiple predicted density maps; Based on multiple optimized segmentation maps, multiple predicted density maps, and the fusion features corresponding to each training segmentation map, a cost matrix is ​​constructed. Based on the cost matrix, the Hungarian algorithm is used to match the optimized segmentation maps and the predicted density maps. Loss is calculated based on the optimized segmentation map and predicted density map of the matching, and the parameters in the object classification and counting model are adjusted according to the loss value; The loss value is calculated using the following loss function: wherein, a loss of intersection over union of the segmentation map optimized for the object classification counting model and the training segmentation map, a mean squared error loss of the predicted density map of the object classification counting model and the ground truth density map, is a weight.

4. The method of claim 2, wherein, The step of generating a ground truth density map for each pseudo-class based on each training segmentation map includes: Obtain the bounding box of each training segmentation map, determine the center coordinates of each bounding box, and perform Gaussian distribution processing on the density of the training segmentation map based on the center coordinates of each bounding box to obtain the density Gaussian distribution map corresponding to each training segmentation map. For each pseudo-class, the density Gaussian distribution maps corresponding to each training segmentation map of the pseudo-class are superimposed to obtain the initial density map corresponding to each pseudo-class. Based on the density values ​​of the cue point grid in the initial density map corresponding to each pseudo-category, high sampling density region, medium sampling density region and low sampling density region are determined in the initial density map; The number of cue point grids in the high sampling density region is reduced, the number of cue point grids in the low sampling density region is increased, and the number of cue point grids in the medium sampling density region remains unchanged, thus obtaining the ground truth density map corresponding to each pseudo-category.

5. The method of claim 1, wherein, The step of fusing features from the plurality of first segmented images, the plurality of second segmented images, and the plurality of third segmented images to obtain a fused feature map includes: Each first segmented image, each second segmented image, and each third segmented image are processed by Gaussian distribution of density to obtain multiple first density maps, multiple second density maps, and multiple third density maps. Each first segmentation image, each second segmentation image, each third segmentation image, each first density map, each second density map, and each third density map are scale-aligned and adjusted to obtain multiple first adjusted segmentation images, multiple second adjusted segmentation images, multiple third adjusted segmentation images, multiple first adjusted density maps, multiple second adjusted density maps, and multiple third adjusted density maps. Based on the cross-attention mechanism, each first adjusted segmentation image and each first adjusted density map, a first weight matrix is ​​calculated; based on the cross-attention mechanism, each second adjusted segmentation image and each second adjusted density map, a second weight matrix is ​​calculated; based on the cross-attention mechanism, each third adjusted segmentation image and each third adjusted density map, a third weight matrix is ​​calculated. A first feature image is calculated based on each first adjusted segmentation image and a first weight matrix; a second feature image is calculated based on each second adjusted segmentation image and a second weight matrix; and a third feature image is calculated based on each third adjusted segmentation image and a third weight matrix. The first feature image, the second feature image, and the third feature image are weighted and summed to obtain a fused feature map.

6. The method of claim 1, wherein, The step of calculating the deviation rate for each category based on the counts of each category in the first classification result and the counts of each category in the second classification result, and calibrating the counts of each category in the first classification result based on the deviation rate, includes: Calculate the area of ​​each first segmented image, and use the first segmented image with an area smaller than a first preset area as the image to be calibrated and counted; The category of the image to be calibrated is taken as the category of the count to be calibrated. The first number of the count to be calibrated is obtained from the first classification result, and the second number of the count to be calibrated is obtained from the second classification result. Based on the first quantity and the second quantity, the deviation rate corresponding to the category of the count to be calibrated is calculated. When the deviation rate is greater than a preset deviation rate threshold, the first quantity and the second quantity are weighted and summed to obtain the final count result of the category of the count to be calibrated. When the deviation rate is less than or equal to the preset deviation rate threshold, the first quantity is used as the final count result of the category of the count to be calibrated.

7. The method of claim 1, wherein, Each of the training segmentation maps includes a cue point grid. The training image is segmented based on a preset image segmentation model to obtain multiple training segmentation maps, including: Each training image is segmented based on the preset image segmentation model to obtain an initial segmented training image; Calculate the area and aspect ratio of each initial segmentation training image, and delete the initial segmentation training images with an area smaller than the second preset area and the initial segmentation training images with an aspect ratio greater than the preset threshold to obtain multiple first intermediate segmentation training images. Calculate the intersection-union ratio (CIU) of every two first intermediate segmentation training images, and delete the first intermediate segmentation training image with the smaller area among the two first intermediate segmentation training images whose CIU is greater than the preset CIU threshold, to obtain multiple second intermediate segmentation training images. Calculate the area of ​​each second intermediate segmentation training image, and increase the number of cue point grids in the second intermediate segmentation training images whose area is smaller than the first preset area to obtain multiple training segmentation images.

8. A device for classifying and counting objects in an image, characterized in that, include: The training module is used to acquire multiple training images, segment each training image based on a preset image segmentation model to obtain multiple training segmentation maps, and use an unsupervised class partitioning method to train an initial object classification and counting model based on each training segmentation map to obtain a trained object classification and counting model. The first classification module is used to acquire an image to be identified with a preset first resolution, segment the image to be identified based on a preset image segmentation model to obtain multiple first segmented images, and input the multiple first segmented images into a trained object classification and counting model to obtain a first classification result. The multi-scale segmentation module is used to adjust the resolution of the image to be identified according to a preset second resolution and a preset third resolution to obtain a second image and a third image. The second image and the third image are then input into a preset image segmentation model to obtain multiple second segmented images and multiple third segmented images. The second classification module is used to fuse features from multiple first segmented images, multiple second segmented images, and multiple third segmented images to obtain a fused feature map. The fused feature map is then input into the trained object classification and counting model to obtain the second classification result. The calibration module is used to calculate the deviation rate corresponding to each category based on the count of each category in the first classification result and the count of each category in the second classification result, calibrate the count of each category in the first classification result based on the deviation rate, and take the calibrated first classification result as the final classification result.

9. A storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the object classification and counting method in an image as claimed in any one of claims 1-7.

10. A computer device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform operations corresponding to the object classification and counting method in the image as described in any one of claims 1-7.

Images