Target counting method and system based on image shape embedding

Abstract

The invention discloses a target counting method and system based on image shape embedding, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-counted image and an example sample image; preprocessing the to-be-counted image and the example sample image; performing image segmentation processing on the preprocessed to-be-counted image and the preprocessed example sample image to obtain a first image block sequence corresponding to the preprocessed to-be-counted image and a second image block sequence corresponding to the preprocessed example sample image; the first image block sequence and the second image block sequence are input into a counting model for target counting, a target counting result is obtained, the counting model adopts a visual converter based on a class-independent counting framework, and the visual converter comprises a self-similarity calculation module, a shape information embedding module and a density map regression module. According to the method, the counting task irrelevant to the category is realized, and the accuracy and generalization ability of the counting task are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a target counting method and system based on image shape embedding. Background Art

[0002] Object counting in images is a critical task in computer vision, which has wide applications in multiple fields; however, even human vision has difficulty providing accurate estimates when the scene contains densely arranged objects.

[0003] Traditional object counting methods usually use supervised learning object counting methods in specific fields, but this method requires a lot of manual annotation work and is often only effective for the specific data sets used when they are trained. When the scene changes, retraining is required, which is a very time-consuming and laborious process and lacks cross-scene versatility and application flexibility. To solve this problem, tichu proposed a few-sample object counting method based on weak supervision, which introduced a similarity comparison module to identify and count target objects through a small number of query image examples provided by the user. However, the information richness of the similarity graph obtained by this method is not as good as that of the feature graph, and it is difficult to distinguish the boundaries of densely arranged objects by relying solely on the similarity graph for regression analysis.

[0004] In order to address the above challenges, class-independent counting methods have been proposed in recent years. When training the model, only a small number of image examples containing the target object are required to accurately estimate the number of times the target appears in the image. This model is able to generalize and recognize unlabeled targets in the same category, even if these targets differ in shape, size, or color from the examples seen during training. The advantage of this method is that it reduces the dependence on the amount of training data and can be more easily generalized to other scenarios, while also significantly reducing the additional work required to apply the model to different scenarios. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a target counting method and system based on image shape embedding, which realizes category-independent counting tasks and significantly improves the accuracy and generalization ability of counting tasks.

[0006] In order to solve the above technical problems, the present invention provides a target counting method based on image shape embedding, the method comprising:

[0007] Obtain an image to be counted and an example sample image;

[0008] Preprocessing the image to be counted and the example sample image to obtain the preprocessed image to be counted and the example sample image;

[0009] Performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image;

[0010] The first image block sequence and the second image block sequence are input into a counting model to perform target counting to obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework, and the visual transformer includes a self-similarity calculation module, a shape information embedding module and a density map regression module.

[0011] Optionally, performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image includes:

[0012] Performing size unification processing on the preprocessed image to be counted and the example sample image to obtain the image to be counted and the example sample image after size unification processing;

[0013] Performing image block division on the image to be counted and the example sample image after the size uniformization processing, to obtain a plurality of first image blocks corresponding to the image to be counted after the size uniformization processing and a plurality of second image blocks corresponding to the example sample image after the size uniformization processing;

[0014] Linear mapping is performed on a plurality of first image blocks and a plurality of second image blocks respectively to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image.

[0015] Optionally, the self-similarity calculation module includes a multi-head self-attention module, which generates a high-dimensional feature representation with self-similarity information based on the first image block sequence and the second image block sequence.

[0016] Optionally, the expression of the high-dimensional feature representation is:

[0017]

[0018] Among them, D m is a high-dimensional feature representation, Vit is a visual transformer, and CrossAtten is a joint-recross mechanism. is the query vector generated by the first image block sequence, is the key vector generated by the second image block sequence, is the value vector generated by the second image block sequence, D x is the first image block sequence, D y is the second image block sequence.

[0019] Optionally, the shape information embedding module calculates the width and height of each annotation box, and generates a two-dimensional matrix based on the width and height of the annotation box;

[0020] Mapping the two-dimensional matrix into a shape feature having the same dimension as the high-dimensional feature representation based on a linear mapping;

[0021] The shape feature is enhanced by embedding shape information to obtain the shape feature after embedding shape information enhancement.

[0022] Optionally, the expression of the shape feature is:

[0023] D s =f shape )linear([w i ,h i ])),

[0024] Among them, D s is the shape feature, f shape is the shape extractor, linear is the linear mapping layer, w i is the width of the annotation box, h i is the height of the annotation box, [w i ,h i ] is a two-dimensional matrix.

[0025] Optionally, the expression of the shape feature after the shape information is embedded in the enhancement is:

[0026]

[0027] Among them, D a is the shape feature after shape information embedding enhancement, Vit is the visual transformer, CrossAtten is the joint-recross mechanism, is the query input generated by the high-dimensional feature representation, For the key input generated by the shape feature, Enter for the value generated by the shape feature.

[0028] Optionally, the density map regression module includes a density regression layer and a plurality of upsampling layers;

[0029] The upsampling layer performs convolution and bilinear interpolation processing on the shape features after the shape information is embedded and enhanced to obtain an upsampling feature map;

[0030] The density map regression layer converts the upsampled feature map into a single-channel density heat map, calculates the cumulative value of all pixel values ​​in the single-channel density heat map, and determines the target counting result based on the cumulative value.

[0031] Optionally, the expression of the single-channel density heat map is:

[0032] T=UpSample(Conv(D a )) k k∈{1,2,3,4},

[0033] Among them, T is a single channel density heat map, UpSample is the upsampling stage, Conv is the convolution stage, and D a is the shape feature after shape information embedding enhancement, k is the index of upsampling stage and convolution stage;

[0034] The expression of the accumulated value is:

[0035] TCount=∑T,

[0036] Among them, TCount is the accumulated value, and T is the single-channel density heat map.

[0037] In addition, the present invention also provides a target counting system based on image shape embedding, the system comprising:

[0038] Image acquisition module: used to acquire the image to be counted and the sample image;

[0039] Image preprocessing module: used for preprocessing the image to be counted and the example sample image to obtain the preprocessed image to be counted and the example sample image;

[0040] Image segmentation module: used for performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image;

[0041] Target counting module: used to input the first image block sequence and the second image block sequence into the counting model to perform target counting and obtain the target counting result. The counting model adopts a visual transformer based on a class-independent counting framework, and the visual transformer includes a self-similarity calculation module, a shape information embedding module and a density map regression module.

[0042] In an embodiment of the present invention, a first image block sequence corresponding to an image to be counted and a second image block sequence corresponding to an example sample image are input into a counting model to perform target counting and obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework. The visual transformer includes a self-similarity calculation module, a shape information embedding module, and a density map regression module. The self-similarity calculation module enhances the model's ability to capture the internal structure of the image and strengthens the similarity features between the image to be counted and the example sample image. The shape information embedding module enhances the model's sensitivity to the shape and boundary information of the target object, thereby improving the recognition ability in scenes with dense objects or partial occlusion, so that the model can not only effectively handle the object categories encountered during training, but also better generalize to new and unseen categories. The density map regression module improves the accuracy of the counting task. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a flow chart of a target counting method based on image shape embedding in an embodiment of the present invention;

[0045] Figure 2 is a flow chart of a target counting method based on image shape embedding in another embodiment of the present invention;

[0046] Figure 3 It is a schematic diagram of the structural composition of the target counting system based on image shape embedding in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Embodiment 1

[0049] See also Figure 1 , Figure 1 : is a flow chart of a target counting method based on image shape embedding in an embodiment of the present invention, the method comprising:

[0050] S11: Obtain the image to be counted and the example sample image;

[0051] In the specific implementation process of the present invention, the example sample image includes a labeling frame, and the number of the labeling frame can be set to one to three. At the same time, the example sample images do not need to be too many, and only three are sufficient to assist in completing the image counting process.

[0052] S12: preprocessing the image to be counted and the example sample image to obtain the preprocessed image to be counted and the example sample image;

[0053] In the specific implementation process of the present invention, the image to be counted and the example sample image are grayed, the pixel values ​​of the three primary colors in the image to be counted and the example sample image are calculated, the average value is calculated according to the pixel values ​​of the three primary colors, the sensitivity of the naked eye to the three primary colors is combined with the average value to generate a weighted average value, and the image to be counted and the example sample image are processed according to the weighted average value to obtain the grayed image to be counted and the grayed example sample image. The grayed image to be counted and the grayed example sample image are denoised, and the grayed image to be counted and the grayed example sample image are filtered according to the set filter value, the average gray value of adjacent pixels in the grayed image to be counted and the grayed example sample image is calculated, and the grayed image to be counted and the grayed example sample image are denoised according to the average gray value to obtain the grayed image to be counted and the grayed example sample image after denoising. Image enhancement processing is performed on the grayscale image to be counted and the grayscale example sample image after denoising, and the graphic histogram distribution of the grayscale image to be counted and the grayscale example sample image after denoising is changed to make the details of the image clearer and improve the readability of the image. After the image enhancement processing is completed, the preprocessing is completed to obtain the preprocessed image to be counted and the example sample image.

[0054] S13: performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image;

[0055] In a specific implementation of the present invention, the image segmentation processing is performed on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image, including: performing size unification processing on the preprocessed image to be counted and the example sample image to obtain the size-unified image to be counted and the example sample image; performing image block division on the size-unified image to be counted and the example sample image to obtain a plurality of first image blocks corresponding to the size-unified image to be counted and a plurality of second image blocks corresponding to the size-unified example sample image; and linear mapping is performed on the plurality of first image blocks and the plurality of second image blocks respectively to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image.

[0056] Specifically, the preprocessed image to be counted and the example sample image are processed to be uniform in size, that is, the preprocessed image to be counted and the example sample image are adjusted to a uniform preset fixed size, and the image to be counted and the example sample image after the uniform size processing are obtained. The image to be counted and the example sample image after the uniform size processing are divided into image blocks, and the image to be counted and the example sample image after the uniform size processing are divided into a number of 16*16 pixel image blocks, and a number of first image blocks corresponding to the image to be counted after the uniform size processing and a number of second image blocks corresponding to the example sample image after the uniform size processing are obtained. Linear mapping is performed on the first image blocks and the second image blocks respectively, that is, each image block is converted into a one-dimensional vector through linear mapping, and then they are combined into corresponding sequences, and a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image are obtained. The image is divided into a number of image blocks and formed into a sequence in order to meet the input requirements of the visual transformer.

[0057] S14: Inputting the first image block sequence and the second image block sequence into a counting model to perform target counting and obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework, and the visual transformer includes a self-similarity calculation module, a shape information embedding module and a density map regression module.

[0058] In a specific implementation of the present invention, the self-similarity calculation module includes a multi-head self-attention module, which generates a high-dimensional feature representation with self-similarity information based on the first image block sequence and the second image block sequence.

[0059] Furthermore, the expression of the high-dimensional feature representation is:

[0060]

[0061] Among them, D m is a high-dimensional feature representation, Vit is a visual transformer, and CrossAtten is a joint-recross mechanism. is the query vector generated by the first image block sequence, is the key vector generated by the second image block sequence, is the value vector generated by the second image block sequence, D x is the first image block sequence, D y is the second image block sequence.

[0062] Furthermore, the shape information embedding module calculates the width and height of each annotation box, and generates a two-dimensional matrix based on the width and height of the annotation box; maps the two-dimensional matrix to a shape feature with the same dimension as the high-dimensional feature representation based on a linear mapping; and performs shape information embedding enhancement on the shape feature to obtain the shape feature after shape information embedding enhancement.

[0063] Furthermore, the expression of the shape feature is:

[0064] D s =f shape (linear([w i ,h i ])),

[0065] Among them, D s is the shape feature, f shape is the shape extractor, linear is the linear mapping layer, w i is the width of the annotation box, h i is the height of the annotation box, [w i ,h i ] is a two-dimensional matrix.

[0066] Furthermore, the expression of the shape feature after the shape information is embedded in the enhancement is:

[0067]

[0068] Among them, D a is the shape feature after shape information embedding enhancement, Vit is the visual transformer, CrossAtten is the joint-recross mechanism, is the query input generated by the high-dimensional feature representation, For the key input generated by the shape feature, Enter for the value generated by the shape feature.

[0069] Furthermore, the density map regression module includes a density regression layer and several upsampling layers; the upsampling layer performs convolution and bilinear interpolation processing on the shape features after shape information embedding enhancement to obtain an upsampling feature map; the density map regression layer converts the upsampling feature map into a single-channel density heat map, calculates the cumulative value of all pixel values ​​in the single-channel density heat map, and determines the target counting result based on the cumulative value.

[0070] Furthermore, the expression of the single-channel density heat map is:

[0071] T=UpSample(Conv(D a )) k k∈{1,2,3,4},

[0072] Among them, T is a single channel density heat map, UpSample is the upsampling stage, Conv is the convolution stage, and D a is the shape feature after shape information embedding enhancement, k is the index of upsampling stage and convolution stage;

[0073] The expression of the accumulated value is:

[0074] TCount=∑T,

[0075] Among them, TCount is the accumulated value, and T is the single-channel density heat map.

[0076] Specifically, the first image block sequence and the second image block sequence are input into the counting model for target counting to obtain the target counting result. The counting model adopts a visual transformer based on a class-independent counting framework, and the visual transformer includes a self-similarity calculation module, a shape information embedding module and a density map regression module. The self-similarity calculation module includes a multi-head self-attention module, which has a 12-layer encoder stack to fully extract and fuse the context information of the image. The multi-head self-attention module generates a high-dimensional feature representation with self-similarity information based on the first image block sequence and the second image block sequence. The second image block sequence generated by the example sample image is input into the self-similarity calculation module as a key vector and a value vector, and the first image block sequence generated by the image to be counted is input into the self-similarity calculation module as a query vector to form a high-dimensional feature map, that is, a high-dimensional feature representation. The self-similarity calculation module not only enhances the model's ability to capture the internal structure of the image, but also strengthens the similarity characteristics between the image to be counted and the example sample image through cross-comparison of the self-attention mechanism. After multi-layer self-attention calculation, the obtained high-dimensional feature representation contains rich self-similarity information. By cross-processing the image to be counted and the example sample image, the model's sensitivity to the object shape and boundary information can be enhanced, while better capturing the correlation between the image and the sample.

[0077] The expression of the high-dimensional feature representation is:

[0078]

[0079] Among them, D m is a high-dimensional feature representation, Vit is a visual transformer, and CrossAtten is a joint-recross mechanism. is the query vector generated by the first image block sequence, is the key vector generated by the second image block sequence, is the value vector generated by the second image block sequence, D x is the first image block sequence, D y is the second image block sequence.

[0080] The shape information embedding module calculates the width and height of each annotation box. Each annotation box contains position coordinate information. The width and height of each annotation box can be calculated according to the position coordinate system, and a two-dimensional matrix is ​​generated based on the width and height of the annotation box. Based on linear mapping, the two-dimensional matrix is ​​mapped to a shape feature with the same dimension as the high-dimensional feature representation. A three-layer linear mapping is used for processing. The dimension change first changes from 2 dimensions to 64 dimensions. In the last layer of mapping, the space size is adjusted to the product of the embedding dimension of the number of annotation boxes. The shape information embedding is enhanced for the shape feature, the shape feature is used as the key input and the value input, and the high-dimensional feature representation is used as the query input. The shape linear embedding module can consider the global features and the local shape information at the same time when calculating the attention weight, and obtain the shape feature after the shape information embedding enhancement. The shape linear embedding module extracts shape features by analyzing the coordinates of the annotation box and fuses these features into the feature representation of the model. This fusion strategy enhances the model's sensitivity to the shape and boundary information of the target object, thereby improving the recognition ability in scenes with dense objects or partial occlusion. At the same time, the shape information embedding module introduces shape information, which enables the model to not only effectively handle the object categories encountered during training, but also better generalize to new, unseen categories. This is because shape features have strong distinguishability between different object categories. By learning these features, the model can adapt to a wider range of counting scenarios, improving its robustness and generalization ability in practical applications.

[0081] The expression of the shape feature is:

[0082] D s =f shape (linear([w i ,h i ])),

[0083] Among them, D sis the shape feature, f shape is the shape extractor, linear is the linear mapping layer, w i is the width of the annotation box, h i is the height of the annotation box, [w i ,h i ] is a two-dimensional matrix, and the shape extractor is used to extract shape features from the width and height of the annotation box.

[0084] The expression of the shape feature after the shape information is embedded in the enhancement is:

[0085]

[0086] Among them, D a is the shape feature after shape information embedding enhancement, Vit is the visual transformer, CrossAtten is the joint-recross mechanism, is the query input generated by the high-dimensional feature representation, For the key input generated by the shape feature, Enter for the value generated by the shape feature.

[0087] The density map regression module includes a density regression layer and several upsampling layers. The density map regression module is used to convert the shape features after shape information embedding enhancement into feature representations that match the resolution of the original input image. The upsampling layer performs convolution and bilinear interpolation processing on the shape features after shape information embedding enhancement. In order to gradually improve the resolution of the shape features after shape information embedding enhancement until it reaches the same scale as the original input image, a staged upsampling strategy is adopted, and four upsampling layers are set. Each upsampling layer contains a series of convolution processing and bilinear interpolation processing to achieve a doubling of spatial resolution. The purpose of the convolution operation is to extract and refine the key information in the feature map, and the purpose of the bilinear interpolation processing is to expand the spatial dimension of the feature map on the basis of maintaining feature coherence, thereby obtaining an upsampled feature map. The density map regression layer converts the upsampled feature map into a single-channel density heat map. The density regression layer includes a 1*1 convolution kernel, and calculates the cumulative value of all pixel values ​​in the single-channel density heat map. Each pixel value in the single-channel density heat map represents the probability density of the object. By accumulating these values, the total number of objects in the image can be accurately estimated, and efficient visual object counting can be achieved, that is, the target counting result is determined based on the cumulative value.

[0088] The expression of the single channel density heat map is:

[0089] T=UpSample(Conv(D a )) k k∈{1,2,3,4},

[0090] Among them, T is a single channel density heat map, UpSample is the upsampling stage, Conv is the convolution stage, and D a is the shape feature after shape information embedding enhancement, k is the index of upsampling stage and convolution stage;

[0091] The expression of the accumulated value is:

[0092] TCount=∑T,

[0093] Among them, TCount is the accumulated value, and T is the single-channel density heat map.

[0094] In order to verify the performance advantages of the counting model in the present invention, the FSC*147 dataset is used to evaluate the performance of different models, and the performance of each model in the few-sample object counting task is examined, especially its effectiveness and generalization ability, which is measured by the mean absolute error (MAE) and root mean square error (RMSE) of the counting results obtained by each model in the counting task.

[0095]

[0096] p i is the predicted count for the i-th image, g i It represents the true count of the i-th image, and n represents the total number of images involved in the evaluation. By comparing the predicted counts and the true counts of all images, the overall error index can be calculated.

[0097] The performance comparison table of each model is as follows:

[0098]

[0099] It can be seen from the above table that in the performance comparison of various models, the present invention adopts the visual transformer based on the class-independent counting framework as the counting model, which shows a more outstanding performance advantage.

[0100] In an embodiment of the present invention, a first image block sequence corresponding to an image to be counted and a second image block sequence corresponding to an example sample image are input into a counting model to perform target counting and obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework. The visual transformer includes a self-similarity calculation module, a shape information embedding module, and a density map regression module. The self-similarity calculation module enhances the model's ability to capture the internal structure of the image and strengthens the similarity features between the image to be counted and the example sample image. The shape information embedding module enhances the model's sensitivity to the shape and boundary information of the target object, thereby improving the recognition ability in scenes with dense objects or partial occlusion, so that the model can not only effectively handle the object categories encountered during training, but also better generalize to new and unseen categories. The density map regression module improves the accuracy of the counting task.

[0101] Embodiment 2

[0102] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a target counting method based on image shape embedding in another embodiment of the present invention, wherein the method comprises:

[0103] S201: Acquire an image to be counted and an example sample image;

[0104] In the specific implementation process of the present invention, the example sample image includes a labeling frame, and the number of the labeling frame can be set to one to three. At the same time, the example sample images do not need to be too many, and only three are sufficient to assist in completing the image counting process.

[0105] S202: preprocessing the image to be counted and the example sample image to obtain the preprocessed image to be counted and the example sample image;

[0106] In the specific implementation process of the present invention, the image to be counted and the example sample image are grayed, the pixel values ​​of the three primary colors in the image to be counted and the example sample image are calculated, the average value is calculated according to the pixel values ​​of the three primary colors, the sensitivity of the naked eye to the three primary colors is combined with the average value to generate a weighted average value, and the image to be counted and the example sample image are processed according to the weighted average value to obtain the grayed image to be counted and the grayed example sample image. The grayed image to be counted and the grayed example sample image are denoised, and the grayed image to be counted and the grayed example sample image are filtered according to the set filter value, the average gray value of adjacent pixels in the grayed image to be counted and the grayed example sample image is calculated, and the grayed image to be counted and the grayed example sample image are denoised according to the average gray value to obtain the grayed image to be counted and the grayed example sample image after denoising. Image enhancement processing is performed on the grayscale image to be counted and the grayscale example sample image after denoising, and the graphic histogram distribution of the grayscale image to be counted and the grayscale example sample image after denoising is changed to make the details of the image clearer and improve the readability of the image. After the image enhancement processing is completed, the preprocessing is completed to obtain the preprocessed image to be counted and the example sample image.

[0107] S203: performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image;

[0108] In the specific implementation process of the present invention, the preprocessed image to be counted and the example sample image are subjected to size unification processing, that is, the preprocessed image to be counted and the example sample image are adjusted to a unified preset fixed size, and the image to be counted and the example sample image after the size unification processing are obtained. The image to be counted and the example sample image after the size unification processing are divided into image blocks, and the image to be counted and the example sample image after the size unification processing are divided into a number of 16*16 pixel image blocks, and a number of first image blocks corresponding to the image to be counted after the size unification processing and a number of second image blocks corresponding to the example sample image after the size unification processing are obtained. Linear mapping is performed on the first image blocks and the second image blocks respectively, that is, each image block is converted into a one-dimensional vector by linear mapping, and then they are combined into corresponding sequences, and a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image are obtained. The image is divided into a number of image blocks and formed into a sequence in order to meet the input requirements of the visual transformer.

[0109] S204: inputting the first image block sequence and the second image block sequence into a counting model, wherein a self-similarity calculation module in the counting model generates a high-dimensional feature representation based on the first image block sequence and the second image block sequence;

[0110] In the specific implementation process of the present invention, the self-similarity calculation module includes a multi-head self-attention module, which has a 12-layer encoder stack to fully extract and fuse the context information of the image. The multi-head self-attention module generates a high-dimensional feature representation with self-similarity information based on the first image block sequence and the second image block sequence, and inputs the second image block sequence generated by the example sample image as a key vector and a value vector into the self-similarity calculation module, and inputs the first image block sequence generated by the image to be counted as a query vector into the self-similarity calculation module to form a high-dimensional feature map, that is, a high-dimensional feature representation. The self-similarity calculation module not only enhances the model's ability to capture the internal structure of the image, but also strengthens the similarity characteristics between the image to be counted and the example sample image through cross-comparison of the self-attention mechanism. After multi-layer self-attention calculation, the obtained high-dimensional feature representation contains rich self-similarity information. Through the cross-processing of the image to be counted and the example sample image, the model can be more sensitive to the shape and boundary information of the object, while more accurately capturing the correlation between the image and the sample.

[0111] The expression of the high-dimensional feature representation is:

[0112]

[0113] Among them, D m is a high-dimensional feature representation, Vit is a visual transformer, and CrossAtten is a joint-recross mechanism. is the query vector generated by the first image block sequence, is the key vector generated by the second image block sequence, is the value vector generated by the second image block sequence, D x is the first image block sequence, D y is the second image block sequence.

[0114] S205: The shape information embedding module in the counting model generates shape features based on the high-dimensional feature representation, and performs shape information embedding enhancement on the shape features to obtain shape features after shape information embedding enhancement;

[0115] In the specific implementation process of the present invention, the shape information embedding module calculates the width and height of each annotation box, each annotation box contains position coordinate information, and the width and height of each annotation box can be calculated according to the position coordinate system, and a two-dimensional matrix is ​​generated based on the width and height of the annotation box. Based on linear mapping, the two-dimensional matrix is ​​mapped to a shape feature with the same dimension as the high-dimensional feature representation, and a three-layer linear mapping is used for processing. The dimension change first changes from 2 dimensions to 64 dimensions. In the last layer of mapping, the space size is adjusted to the product of the embedding dimension of the number of annotation boxes. The shape feature is embedded with shape information, and the shape feature is used as a key input and a value input, and the high-dimensional feature representation is used as a query input. The shape linear embedding module can consider global features and local shape information at the same time when calculating the attention weight, and obtain the shape feature after shape information embedding enhancement. The shape linear embedding module extracts shape features by analyzing the coordinates of the annotation box and fuses these features into the feature representation of the model. This fusion strategy enhances the model's sensitivity to the shape and boundary information of the target object, thereby improving the recognition ability in scenes with dense objects or partial occlusion. At the same time, the shape information embedding module introduces shape information, which enables the model to not only effectively handle the object categories encountered during training, but also better generalize to new, unseen categories. This is because shape features have strong distinguishability between different object categories. By learning these features, the model can adapt to a wider range of counting scenarios, improving its robustness and generalization ability in practical applications.

[0116] The expression of the shape feature is:

[0117] D s =f shape (linear([w i ,h i ])),

[0118] Among them, D s is the shape feature, f shape is the shape extractor, linear is the linear mapping layer, w i is the width of the annotation box, h i is the height of the annotation box, [w i ,h i ] is a two-dimensional matrix, and the shape extractor is used to extract shape features from the width and height of the annotation box.

[0119] The expression of the shape feature after the shape information is embedded in the enhancement is:

[0120]

[0121] Among them, D ais the shape feature after shape information embedding enhancement, Vit is the visual transformer, CrossAtten is the joint-recross mechanism, is the query input generated by the high-dimensional feature representation, For the key input generated by the shape feature, Enter for the value generated by the shape feature.

[0122] S206: The density map regression module in the counting model performs target counting based on the shape features after shape information embedding enhancement to obtain a target counting result.

[0123] In the specific implementation process of the present invention, the density map regression module includes a density regression layer and several upsampling layers. The density map regression module is to convert the shape features after the shape information is embedded and enhanced into feature representations that match the resolution of the original input image. The upsampling layer performs convolution and bilinear interpolation processing on the shape features after the shape information is embedded and enhanced. In order to gradually improve the resolution of the shape features after the shape information is embedded and enhanced until it reaches the same scale as the original input image, a staged upsampling strategy is adopted, and four upsampling layers are set. Each upsampling layer includes a series of convolution processing and bilinear interpolation processing to achieve a doubling of the spatial resolution. The purpose of the convolution operation is to extract and refine the key information in the feature map, and the purpose of the bilinear interpolation processing is to expand the spatial dimension of the feature map on the basis of maintaining feature coherence, thereby obtaining an upsampled feature map. The density map regression layer converts the upsampled feature map into a single-channel density heat map. The density regression layer includes a 1*1 convolution kernel, and calculates the cumulative value of all pixel values ​​in the single-channel density heat map. Each pixel value in the single-channel density heat map represents the probability density of the object. By accumulating these values, the total number of objects in the image can be accurately estimated, and efficient visual object counting can be achieved, that is, the target counting result is determined based on the cumulative value.

[0124] The expression of the single channel density heat map is:

[0125] T=UpSample(Conv(D a )) k k∈{1,2,3,4},

[0126] Among them, T is a single channel density heat map, UpSample is the upsampling stage, Conv is the convolution stage, and D a is the shape feature after shape information embedding enhancement, k is the index of upsampling stage and convolution stage;

[0127] The expression of the accumulated value is:

[0128] TCount=∑T,

[0129] Among them, TCount is the accumulated value, and T is the single-channel density heat map.

[0130] In an embodiment of the present invention, a first image block sequence corresponding to an image to be counted and a second image block sequence corresponding to an example sample image are input into a counting model to perform target counting and obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework. The visual transformer includes a self-similarity calculation module, a shape information embedding module, and a density map regression module. The self-similarity calculation module enhances the model's ability to capture the internal structure of the image and strengthens the similarity features between the image to be counted and the example sample image. The shape information embedding module enhances the model's sensitivity to the shape and boundary information of the target object, thereby improving the recognition ability in scenes with dense objects or partial occlusion, so that the model can not only effectively handle the object categories encountered during training, but also better generalize to new and unseen categories. The density map regression module improves the accuracy of the counting task.

[0131] Embodiment 3

[0132] See also Figure 3 , Figure 3 : is a schematic diagram of the structure of a target counting system based on image shape embedding in an embodiment of the present invention, the system comprises:

[0133] Image acquisition module 31: used to acquire the image to be counted and the example sample image;

[0134] Image preprocessing module 32: used for preprocessing the image to be counted and the example sample image to obtain the preprocessed image to be counted and the example sample image;

[0135] Image segmentation module 33: used to perform image segmentation processing on the preprocessed image to be counted and the example sample image, to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image;

[0136] Target counting module 34: used to input the first image block sequence and the second image block sequence into a counting model to perform target counting and obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework, and the visual transformer includes a self-similarity calculation module, a shape information embedding module and a density map regression module.

[0137] In the specific implementation process of the present invention, the specific implementation methods of the system items can be referred to the above embodiments, which will not be repeated here.

[0138] In an embodiment of the present invention, a first image block sequence corresponding to an image to be counted and a second image block sequence corresponding to an example sample image are input into a counting model to perform target counting and obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework. The visual transformer includes a self-similarity calculation module, a shape information embedding module, and a density map regression module. The self-similarity calculation module enhances the model's ability to capture the internal structure of the image and strengthens the similarity features between the image to be counted and the example sample image. The shape information embedding module enhances the model's sensitivity to the shape and boundary information of the target object, thereby improving the recognition ability in scenes with dense objects or partial occlusion, so that the model can not only effectively handle the object categories encountered during training, but also better generalize to new and unseen categories. The density map regression module improves the accuracy of the counting task.

[0139] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0140] In addition, the above describes in detail a target counting method and system based on image shape embedding provided by an embodiment of the present invention. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A target counting method based on image shape embedding, characterized in that: The method comprises: Obtain an image to be counted and an example sample image; Preprocessing the image to be counted and the example sample image to obtain the preprocessed image to be counted and the example sample image; Performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image; The first image block sequence and the second image block sequence are input into a counting model to perform target counting to obtain a target counting result. The counting model adopts a visual transformer based on a class-independent counting framework, and the visual transformer includes a self-similarity calculation module, a shape information embedding module and a density map regression module.

2. The target counting method based on image shape embedding according to claim 1 is characterized in that: The step of performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image includes: Performing size unification processing on the preprocessed image to be counted and the example sample image to obtain the image to be counted and the example sample image after size unification processing; Performing image block division on the image to be counted and the example sample image after the size uniformization processing, to obtain a plurality of first image blocks corresponding to the image to be counted after the size uniformization processing and a plurality of second image blocks corresponding to the example sample image after the size uniformization processing; Linear mapping is performed on a plurality of first image blocks and a plurality of second image blocks respectively to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image.

3. The target counting method based on image shape embedding according to claim 1, characterized in that: The self-similarity calculation module includes a multi-head self-attention module, which generates a high-dimensional feature representation with self-similarity information based on the first image block sequence and the second image block sequence.

4. The target counting method based on image shape embedding according to claim 3 is characterized in that: The expression of the high-dimensional feature representation is: Among them, D m is a high-dimensional feature representation, Vit is a visual transformer, and CrossAtten is a joint-recross mechanism. is the query vector generated by the first image block sequence, is the key vector generated by the second image block sequence, is the value vector generated by the second image block sequence, D x is the first image block sequence, D y is the second image block sequence.

5. The target counting method based on image shape embedding according to claim 1, characterized in that: The shape information embedding module calculates the width and height of each annotation box, and generates a two-dimensional matrix based on the width and height of the annotation box; Mapping the two-dimensional matrix into a shape feature having the same dimension as the high-dimensional feature representation based on a linear mapping; The shape feature is enhanced by embedding shape information to obtain the shape feature after embedding shape information enhancement.

6. The target counting method based on image shape embedding according to claim 5, characterized in that: The expression of the shape feature is: D s =f shape (linear([w i ,h i ])), Among them, D s is the shape feature, f shape is the shape extractor, linear is the linear mapping layer, w i is the width of the annotation box, h i is the height of the annotation box, [w i ,h i ] is a two-dimensional matrix.

7. The object counting method based on image shape embedding according to claim 6 is characterized in that: The expression of the shape feature after the shape information is embedded in the enhancement is: Among them, D a is the shape feature after shape information embedding enhancement, Vit is the visual transformer, CrossAtten is the joint-recross mechanism, is the query input generated by the high-dimensional feature representation, For the key input generated by the shape feature, Enter for the value generated by the shape feature.

8. The target counting method based on image shape embedding according to claim 1, characterized in that: The density map regression module includes a density regression layer and a plurality of upsampling layers; The upsampling layer performs convolution and bilinear interpolation processing on the shape features after the shape information is embedded and enhanced to obtain an upsampling feature map; The density map regression layer converts the upsampled feature map into a single-channel density heat map, calculates the cumulative value of all pixel values ​​in the single-channel density heat map, and determines the target counting result based on the cumulative value.

9. The object counting method based on image shape embedding according to claim 8, characterized in that: The expression of the single channel density heat map is: T=UpSample(Conv(D a )) k k∈{1,2,3,4}, Among them, T is a single channel density heat map, UpSample is the upsampling stage, Conv is the convolution stage, and D a is the shape feature after shape information embedding enhancement, k is the index of upsampling stage and convolution stage; The expression of the accumulated value is: TCount=∑T, Among them, TCount is the accumulated value, and T is the single-channel density heat map.

10. An object counting system based on image shape embedding, characterized in that: The system comprises: Image acquisition module: used to acquire the image to be counted and the sample image; Image preprocessing module: used for preprocessing the image to be counted and the example sample image to obtain the preprocessed image to be counted and the example sample image; Image segmentation module: used for performing image segmentation processing on the preprocessed image to be counted and the example sample image to obtain a first image block sequence corresponding to the preprocessed image to be counted and a second image block sequence corresponding to the preprocessed example sample image; Target counting module: used to input the first image block sequence and the second image block sequence into the counting model to perform target counting and obtain the target counting result. The counting model adopts a visual transformer based on a class-independent counting framework, and the visual transformer includes a self-similarity calculation module, a shape information embedding module and a density map regression module.

Images