Self-supervised target counting fine-tuning method and system based on domain adaptation

Abstract

The application discloses a kind of based on field self-adapting self-supervised target counting fine-tuning method and system, comprising: for given image is split, each independent subblock will first be independently through model network, and produce multiple sets of embedding vector visual feature word coding, as the independent fine-grained feature representation of different regions in image.Then calculate the cosine similarity between type text embedding vector and visual category information vector, by establishing the uncertainty estimation of different regions of image realizes the deconstruction and classification discussion of foreground region and background region.And foreground subblock and background subblock are respectively self-supervised and two-dimensional segmentation.The application realizes self-supervised field self-adapting from source domain to target domain, optimizes the problem that most target counting is not strong when migrating to specific type target counting task, positioning correct type counting target ability, prediction density graph accuracy is poor.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to a self-supervised target counting fine-tuning method and system based on domain adaptation. BACKGROUND

[0002] With the rapid economic development and the continuous improvement of people's living standards, crowd gathering activities have become more frequent, bringing great pressure to social public security management and public transportation scheduling. Statistics of the distribution and density of targets in the scene has become a visual subject with important scientific research value. Target counting technology based on deep learning has been widely used in many fields, especially target counting models based on Transformer backbone networks. In practical scenarios, target counting tasks are also widely used in many production fields such as agriculture, industry, and biological science, such as estimating the total number of crops in the field, counting the total number of industrial products on the factory production line, and counting the presence of wild animals in the wild. With the widespread expansion of application scenarios, target counting tasks have encountered more challenges, including but not limited to unfamiliar counting object categories, complex background interference, and uneven target distribution. These problems limit the counting performance of the algorithm and are also difficult points that need to be solved urgently. SUMMARY

[0003] The purpose of the present application is to solve the problem that the target counting task in the prior art is limited by the counting performance of the algorithm when facing unfamiliar counting object categories, complex background interference, and uneven target distribution, and to provide a self-supervised target counting fine-tuning method and system based on domain adaptation.

[0004] To achieve the above purpose, the technical scheme is adopted as follows:

[0005] The self-supervised target counting fine-tuning method based on domain adaptation provided by the present application comprises:

[0006] The given image is divided to obtain a plurality of subblocks with an intersection-over-union ratio of zero, and then the same number of embedding vector visual feature word encodings are obtained; the word encoding vector is used as the representation form of the visual feature;

[0007] A plurality of groups of type text embedding features are preset, and the cosine similarity between each group of type text embedding vectors and the global category visual vector of the subblock in the visual feature is calculated based on the global category visual vector of the subblock in the visual feature, to estimate the uncertainty of the classification to which the different blocks belong;

[0008] In the uncertainty estimation process, the image subblocks are divided into foreground subblocks and background subblocks; the foreground subblocks are input into a density map regression network after a scale transformation operation to obtain a plurality of groups of outputs;

[0009] Regularized self-supervisory loss is performed based on the consistency of multiple outputs. At the same time, inconsistent results based on model outputs are used as self-supervised training signals for gradient descent of the weight network to achieve backpropagation and model fine-tuning.

[0010] Based on binary segmentation, the point-level density noise containing background semantic information in the background sub-block is suppressed to complete the target count adjustment.

[0011] Optionally, obtaining the same number of embedded vector visual feature word codes is specifically: passing the obtained sub-blocks through a CLIP ViT-based visual feature preprocessor to generate the same number of embedded vector visual feature word codes as independent feature representations of different regions in the image.

[0012] Optionally, the global visual category vector of the sub-block based on the visual features is specifically: each set of visual features includes a global visual embedding vector, and the global visual embedding vector includes the global visual category vector of the corresponding sub-block;

[0013] The cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks is calculated as follows: the cosine similarity between the global visual embedding vectors of U different blocks and the V type text embeddings is calculated, and the similarity information is normalized using the softmax function to obtain a U×V similarity matrix.

[0014] (1)

[0015] in, I i and T j Represent the global visual embedding vector and the type text embedding vector respectively, t represents the temperature coefficient; inner product I i T j Indicates the i image category information vector and the j The cosine similarity of the texts; softmax The function outputs different types of classification confidence, which is used to estimate the uncertainty of the types of different blocks.

[0016] Set the counting object and compare and match the global visual embedding vector of each slice image with the text embedding of the counting object by cosine similarity, as shown in the following formula:

[0017] (2)

[0018] Where: D is a set of data sets, and is any ordered pair (I i , T j ) establishes the association between dataset images and type texts, where I i and T j Respectively represent i image category information vector and the j The inner product of text embedding vectors; For all a collection of permutations; For the i The ordinal number of the text embedding with the highest similarity to the image category vector.

[0019] Optionally, the foreground sub-block is scaled and then input into a density map regression network to obtain multiple sets of outputs, specifically: based on the original image corresponding to the image sub-block; the original image is input into the model network to generate a set of embedded vector visual feature word codes, and finally output a point-level density map; the image sub-blocks are each input into the model network to generate 16 sets of embedded vector visual feature word codes, and finally output 16 point-level density maps; the 16 point-level density maps corresponding to the image sub-blocks are arranged in corresponding positions and will correspond one-to-one to each sub-area of ​​the point-level density map of the original image.

[0020] Optionally, the regularized self-supervisory loss based on multi-output consistency includes: using output results of different scales as pseudo-labels of self-supervisory signals; designing adaptive smoothing mask weights, imposing additional constraints on teacher labels generated by the network, and correcting the weights of output pseudo-labels with different confidence levels to reduce the negative impact of invalid pseudo-labels in the self-supervisory loss based on consistency regularization.

[0021] Optionally, the design of the adaptive smoothing mask weight is specifically as follows:

[0022] Adaptive smoothing mask weights impose additional constraints on the teacher labels produced by the multi-output density map generation network. The process is expressed as follows:

[0023] (3)

[0024] in: w i For the i mask weights; p i For the i Large-scale output prediction counts; c For the i Small-scale output prediction counts; m is the high confidence ratio parameter;s is a numerical value from the average degree of dispersion.

[0025] Optionally, the consistency regularization self-supervised loss based on multiple outputs is specifically:

[0026] (4)

[0027] The calculation method of the loss is to traverse C cutting blocks belonging to the counting target type, calculate the square error of the pixel points at the same corresponding position one by one, and obtain the mean square error on the matrix level. After multiplying the loss corresponding to each cutting block by the corresponding adaptive smoothing mask weight wi, the weighted sum result of each sub-loss is calculated.

[0028] Optionally, the inconsistent result based on the model output is used as a self-supervised training signal for gradient descent of the weight network, and backward propagation and model fine-tuning are realized. Specifically, in the network backward propagation training link, the pseudo-label density map f p is calculated, which is treated as a true value not containing gradient, and a consistency loss is established with the predicted density distribution map f c . Finally, the calculation graph of f c is used for backward propagation to fine-tune the weights of the entire model network.

[0029] Optionally, the background sub-block containing background semantic information is suppressed based on binary segmentation of point-level density noise. Specifically, each image sub-block is input into the model network and outputs multiple groups of point-level density maps. The mask weight is established for the blocks divided into different types in the uncertainty estimation, as shown in the following formula:

[0030] (5)

[0031] In the formula: IBM for ideal binary mask; is the serial number of the text embedding with the highest similarity to the i-th image category vector; i

[0032] For a group of preset text embedding vectors, the element with index 1 represents the target counting type, and the element not equal to 1 represents other irrelevant types. The background suppression global loss based on binary segmentation is calculated as follows:

[0033] (6)

[0034] In the formula: U is the number of cutting blocks; B is the length and width of the cutting block subgraph.​f c to predict the density map; Loss T for background suppression loss.

[0035] The application also provides a self-supervised target counting fine-tuning system based on domain adaptation, comprising:

[0036] A division module divides a given image to obtain a plurality of sub-blocks with an intersection-over-union ratio of zero, and then obtains the same number of embedding vector visual feature word encodings; the word encoding vector is used as the representation form of the visual feature;

[0037] A calculation module predefines a plurality of groups of type text embedding features, calculates the cosine similarity between each group of type text embedding vectors and the global category visual vector of the sub-blocks in the visual feature based on the global category visual vector of the sub-blocks in the visual feature, and estimates the uncertainty of the classification to which the different blocks belong;

[0038] An input module divides the image sub-blocks into foreground sub-blocks and background sub-blocks in the process of uncertainty estimation; the foreground sub-blocks are input into a density map regression network after a scale transformation operation, and a plurality of outputs are obtained;

[0039] A self-supervised loss module performs regular self-supervised loss based on the consistency of the plurality of outputs, and simultaneously uses the inconsistent results of the model output as a self-supervised training signal for gradient descent of the weight network, so as to realize back propagation and model fine-tuning;

[0040] A suppression module suppresses the point-level density noise containing background semantic information in the background sub-blocks based on binary segmentation, and completes target counting adjustment.

[0041] Compared with the prior art, the application has the following beneficial effects:

[0042] The application splits a given image, each independent sub-block will first be independently passed through a model network, and a plurality of embedding vector visual feature word encodings are generated, which are used as independent fine-grained feature representations of different regions in the image. Then, the cosine similarity between the type text embedding vector and the visual category information vector is calculated, and the deconstruction and classification discussion of the foreground region and the background region are realized by establishing the uncertainty estimation of different regions of the image. The foreground sub-blocks and the background sub-blocks are respectively self-supervised and two-dimensionally segmented. The application realizes self-supervised domain adaptation from a source domain to a target domain, and under the premise of not increasing the data labeling cost, uses text as a cross-modal guide to optimize the problem that most target counting pre-training models have weak ability to locate correct type counting targets and poor prediction accuracy of the density map when migrating to a specific type of target counting task. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0044] Figure 1 The flowchart of the self-supervised target counting fine-tuning method based on domain adaptation of the present application;

[0045] Figure 2 The structural diagram of the self-supervised target counting fine-tuning system based on domain adaptation of the present application;

[0046] Figure 3 The overall structure diagram of the model of the present application;

[0047] Figure 4 The block zero-shot classification and density map regression prediction diagram of the present application;

[0048] Figure 5 The network structure diagram of the block uncertainty estimation task of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0050] Therefore, the detailed description of the embodiments of the present application provided in the drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art on the basis of the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0051] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0052] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, or the orientation or position relationship when the product of the present application is usually placed, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0053] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0054] In the description of the embodiments of the present application, it should be noted that unless otherwise explicitly specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood in a broad sense. For example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0055] The present application will be described in further detail below with reference to the accompanying drawings:

[0056] Referring to Figure 1 The present application discloses a self-supervised target counting fine-tuning method based on domain adaptation, comprising:

[0057] S101, the given image is divided to obtain a plurality of subblocks with an intersection-over-union ratio of zero, and then the same number of embedding vector visual feature word encodings are obtained; the word encoding vector is used as the representation form of the visual feature;

[0058] The obtained subblocks are processed by a CLIP ViT-based visual feature preprocessor to generate the same number of embedding vector visual feature word encodings as the independent feature representation of different regions in the image.

[0059] S102, a plurality of groups of type text embedding features are preset, and the cosine similarity between each group of type text embedding vector and the global category visual vector of the subblock in the visual feature is calculated based on the global category visual vector of the subblock in the visual feature, to estimate the uncertainty of the classification to which the different blocks belong;

[0060] A global visual category vector of a sub-block based on the visual features, specifically: each set of visual features includes a global visual embedding vector, and the global visual embedding vector includes the global visual category vector of the sub-block to which it belongs;

[0061] The cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks is calculated as follows: the cosine similarity between the global visual embedding vectors of U different blocks and the V type text embeddings is calculated, and the similarity information is normalized using the softmax function to obtain a U×V similarity matrix.

[0062] (1)

[0063] in, I i and T j Represent the global visual embedding vector and the type text embedding vector respectively, t represents the temperature coefficient; inner product I i T j Indicates the i image category information vector and the j The cosine similarity of the texts; softmax The function outputs different types of classification confidence, which is used to estimate the uncertainty of the types of different blocks.

[0064] Set the counting object and compare and match the global visual embedding vector of each slice image with the text embedding of the counting object by cosine similarity, as shown in the following formula:

[0065] (2)

[0066] Where: D is the dataset set, and the association between the dataset image and type text is established for any ordered pair (Ii, Tj), where Ii and Tj represent the inner product of the i-th image category information vector and the j-th text embedding vector respectively; For all a collection of permutations; For the i The ordinal number of the text embedding with the highest similarity to the image category vector.

[0067] S103, dividing the image sub-blocks into foreground sub-blocks and background sub-blocks during the uncertainty estimation process; performing a scale transformation operation on the foreground sub-blocks and inputting them into a density map regression network to obtain multiple sets of outputs;

[0068] Based on the original image corresponding to the image sub-block; input the original image into the model network to generate a set of embedding vector visual feature word encoding, and finally output a point-level density map; the image sub-block is input into the model network to generate 16 sets of embedding vector visual feature word encoding, and finally output 16 point-level density maps; the 16 point-level density maps corresponding to the image sub-block are arranged according to the corresponding positions, which will be respectively corresponding to each sub-region of the point-level density map of the original image.

[0069] S104, based on the consistency of multiple outputs, regular self-supervised loss is carried out, and based on the inconsistent results of model output as self-supervised training signal of gradient descent of weight network, reverse propagation and model fine-tuning are realized;

[0070] Based on the consistency of multiple outputs, regular self-supervised loss is carried out, including: taking the output results of different scales as the pseudo-labels of self-supervised signal; designing adaptive smoothing mask weight, applying additional constraint to the teacher label generated by the network, and modifying the weight of the output pseudo-label with different confidence to reduce the negative influence of invalid pseudo-label in the self-supervised loss based on consistency regularization.

[0071] The adaptive smoothing mask weight is designed, specifically:

[0072] The adaptive smoothing mask weight applies additional constraint to the teacher label generated by the density map generation network based on multiple outputs, which is expressed by the following formula:

[0073] (3)

[0074] Where: wi is the i th mask weight; pi is the i th large-scale output prediction count; ci is the i th small-scale output prediction count; m is a high confidence proportion parameter; and sigma is the numerical value of the average value dispersion degree.

[0075] Based on the consistency of multiple outputs, the regular self-supervised loss is specifically:

[0076] (4)

[0077] The calculation method of the loss is to traverse C cutting blocks belonging to the counting target type, calculate the square error of the pixel points at the same corresponding position one by one, and obtain the mean square error on the matrix level by averaging; multiply the loss corresponding to each cutting block by the corresponding adaptive smoothing mask weight wi, and calculate the weighted sum of each sub-loss.

[0078] The inconsistent result based on the model output is used as a self-supervised training signal of the weight network gradient descent to realize back propagation and model fine tuning, specifically: in the network back propagation training link, the calculation graph of the pseudo label density map fp is stripped to be treated as a true value without gradient, and consistency loss is established with the predicted density distribution map fc; finally, the weight of the entire model network is fine tuned through the back propagation of the calculation graph of fc.

[0079] S105, based on the binary segmentation, the point-level density noise containing background semantic information in the background sub-block is suppressed, and the target counting adjustment is completed.

[0080] Each image sub-block is input to the model network and outputs multiple groups of point-level density maps; in the uncertainty estimation, different types of blocks are divided to establish corresponding mask weights, as shown in the following formula:

[0081] (5)

[0082] In the formula: IBM is an ideal binary mask; is the serial number of the text embedding with the highest similarity to the i-th image category vector;

[0083] For a set of preset text embedding vectors, the element with index 1 represents the target counting type, and the element not equal to 1 represents other irrelevant types; the background suppression global loss based on binary segmentation is calculated as follows:

[0084] (6)

[0085] In the formula: U is the number of cut blocks; B is the length and width of the cut block subgraph; fc is the predicted density map; LossT is the background suppression loss.

[0086] Referring to Figure 2 The application discloses a self-supervised target counting fine tuning system based on domain adaptation, comprising:

[0087] A division module divides a given image to obtain a plurality of sub-blocks with an intersection-over-union ratio of zero, and then obtains the same number of embedding vector visual feature word encodings; the word encoding vector is used as a visual feature representation form;

[0088] A calculation module presets a plurality of type text embedding features, calculates the cosine similarity between each group of type text embedding vectors and the global category visual vector of different sub-blocks based on the global visual category vector of the sub-blocks in the visual features, and estimates the uncertainty of the classification to which different blocks belong;

[0089] An input module, wherein the input module divides the image sub-blocks into foreground sub-blocks and background sub-blocks during the uncertainty estimation process; performs a scale transformation operation on the foreground sub-blocks and then inputs them into a density map regression network to obtain multiple sets of outputs;

[0090] A self-supervised loss module, which performs regularized self-supervised loss based on the consistency of multiple outputs and uses inconsistent results of model outputs as self-supervised training signals for gradient descent of the weight network to achieve backpropagation and model fine-tuning;

[0091] The suppression module suppresses the point-level density noise containing background semantic information in the background sub-block based on binary segmentation to complete the target count adjustment

[0092] Example:

[0093] The present invention discloses a domain-adaptive self-supervised target counting fine-tuning method, comprising:

[0094] Step 1: For a given dataset, preprocess the image and input it into the backbone network to generate multiple sets of feature outputs

[0095] Given a dataset with dimensions of Input image , according to its actual aspect ratio, the image is cropped to obtain a square image with equal length and width as large as possible through random displacement of the sliding window, and the image dimension becomes or . Then, bilinear interpolation is used to convert the dimension to , and then divide it into 16 sub-blocks according to 4×4 equal divisions, and the dimension of each sub-block is .

[0096] For further information, see Figure 3 , each sub-block passes through the visual feature preprocessor based on CLIP ViT to generate 16 sets of embedding vector visual feature word encodings with a feature dimension of 197×512, which serve as independent feature representations of different regions in the image.

[0097] Step 2: Calculate the cosine similarity between the text embedding vectors of each group type and the global category visual vectors of different sub-blocks to estimate the uncertainty of the classification of different blocks

[0098] See also Figure 3 , the multiple sets of feature embedding vectors generated by the image sub-blocks have a dimension of 197×512 for visual feature word encoding. In addition to the block-level embedding vector, they all contain a global category embedding vector with a dimension of 1×512, which contains the global category information of the block to which it belongs, and will be used to estimate the uncertainty of the classification to which different blocks belong.

[0099] Furthermore, for the preset multiple types of text embeddings, it is necessary to calculate the cosine similarity between the global visual embedding vectors of U different blocks and the pairwise cosine similarities of V types of text embeddings, and use the softmax function to normalize these similarity information to obtain a U×V similarity matrix.

[0100] (7)

[0101] Where Ii and Tj represent the global visual embedding vector and the type text embedding vector, respectively, and τ represents the temperature coefficient. The inner product IiTj represents the cosine similarity between the i-th image category information vector and the j-th text. Finally, a softmax function is used to output different types of classification confidence.

[0102] Furthermore, taking the crowd counting task as an example, since the counting object is "human heads", each slice image is used to embed the text "A photo of heads" for cosine similarity comparison and matching, as shown in the following formula.

[0103] (8)

[0104] Where: D is a set of data, with any ordered pair ( I i , T j ) establishes the association between dataset images and type texts, where I i and T j Respectively represent i image category information vector and the j The inner product of text embedding vectors; For all a collection of permutations; is the sequence number of the text embedding with the highest similarity to the i-th image category vector.

[0105] Step 3: Design of point-level density map self-supervised loss function

[0106] During the training phase, data augmentation was applied. This involves altering and augmenting the training set images using methods such as Gaussian noise, Gaussian blur, and color dithering. Furthermore, horizontal flipping and rotation were used to simulate the perspectives of images from different directions and positions in real-world scenes. The original images from the augmented training set and the ground-truth density map served as input for supervised training of the model network.

[0107] See also Figure 3 , for a series of dimensions image sub-blocks, the size of the region where the corresponding original image is located is The original image is input into the model network to generate a set of embedding vector visual feature word encodings with a feature dimension of 197x512, and finally output a point-level density map with a dimension of The image sub-blocks are each input into the model network to generate 16 sets of embedding vector visual feature word encodings with a feature dimension of 197x512, and finally output 16 point-level density maps with a dimension of The 16 point-level density maps corresponding to the image sub-blocks are arranged in corresponding positions to respectively correspond to each sub-region of the point-level density map of the original image.

[0108] Further, referring to Figure 4 For image sub-blocks classified as foreground in uncertainty estimation, as regions containing counting targets in the image, they correspond to point-level density maps with non-zero values. The consistency regularization loss based on multiple outputs proposed in the present application uses different scales of output results as pseudo-labels of self-supervised signals, that is, the prediction output result of the model at one scale is used as the pseudo-label of the prediction output result at another scale. The pseudo-label of the output pseudo-label is used as a signal to supervise another output result, and an additional regularization term is proposed for model training, so that the gradient descent training direction of the weight network tends to approach the pseudo-label. The essential optimization goal is to minimize the difference between multiple output results of the counting model, and to enhance the robustness of the model through self-supervised training.

[0109] Further, in order to solve the problem of deviation of the model training direction caused by the confidence of the pseudo-label teacher output, in the present application, the adaptive smoothing mask weight is used to impose additional constraints on the teacher label generated by the density map generation network based on multiple outputs. This process can be expressed by the following formula:

[0110] (9)

[0111] In the formula: w i is the mask weight of the i th p i is the prediction count of the i th c is the prediction count of the i th m is the high confidence proportion parameter; s is the numerical value from the average value dispersion degree.

[0112] Further, in the mask weight map w , the self-supervised training weight corresponding to the i th cropped block of the counting target type is denoted asw i . p i denotes the target count calculated from the cut-out block after scale scaling, c i denotes the target count of the corresponding sub-block when predicting the original image, and the ratio of the two reflects the inconsistency of the model under the multi-scale output condition, which serves as a self-supervised signal to guide the domain adaptive task. Specifically, when the ratio of the prediction quantity result after scale scaling and the quantity of the corresponding region of the overall prediction output is closer to the high confidence ratio parameter m , it indicates that the pseudo label confidence of the cut-out block is higher, and the corresponding self-supervised training weight is higher. In addition, when the ratio of the multi-output result deviates from the high confidence ratio parameter m by more than the discrete value s , the self-supervised training weight of the cut-out block w i will be zero, indicating that the multi-scale prediction result of the block is not enough to be used as a self-supervised signal.

[0113] Further, in the design of the self-supervised loss function, an adaptive smoothing mask is used to give different weights to different blocks, so that in the model fine-tuning process, the recall rate of the multi-output pseudo label under different conditions is limited, and more attention is paid to effective self-supervised signals, reducing the negative impact of invalid pseudo labels in the self-supervised loss based on consistency regularization. The block-level consistency regularization loss function based on multi-output is as follows:

[0114]

[0115] The calculation method of the loss is to traverse C cut-out blocks belonging to the count target type, calculate the square error of the same corresponding pixel point one by one, and then average to get the mean square error on the matrix level. Then multiply the loss corresponding to each cut-out block by the corresponding adaptive smoothing mask weight w i , and then calculate the weighted sum of each sub-loss. In this way, the multi-output consistency regularization based on multi-output will be established between the multiple output point level density maps of different blocks, and the essence of the optimization goal is to minimize the difference between the multiple output results of the counting model, and to enhance the robustness of the model.

[0116] Further, in the network backpropagation training link, the computation graph of the pseudo label density map f p needs to be stripped, which is treated as a true value without gradient, and a consistency loss is established between it and the predicted density distribution map f c . Finally, through f cThe computing graph is back-propagated to fine-tune the weights of the entire model network.

[0117] Further, referring to Figure 4 For the image sub-blocks classified as background in the uncertainty estimation, which as the region in the image does not contain the counting target, corresponding to the point-level density map with the value of zero. For all image sub-blocks, input them into the model network, and output multiple groups of point-level density maps with the dimension of. To achieve background suppression, the corresponding mask weight is established for the sub-blocks classified as different types in the uncertainty estimation, as shown in the following formula:

[0118]

[0119] In the formula, IBM is the ideal binary mask; is the serial number of the text embedding with the highest similarity to the i image category vector.

[0120] Further, for a set of preset serial numbers of text embedding vectors, the element with index 1 represents the target counting type, and the element not equal to 1 represents other irrelevant types. Thus, the background suppression global loss based on binary segmentation is calculated as follows:

[0121]

[0122] In the formula, U is the number of cropped blocks; B is the length and width of the cropped block subgraph; f c is the predicted density map; Loss T is the background suppression loss.

[0123] Further, the background suppression loss is calculated using the mask weight map IBM is the binary mask intersection of all cropped blocks and is given the corresponding weight. The classification result is the cropped area of the irrelevant type, which represents the background area in the graph, and the target density true value graph should be all zero. Therefore, the predicted density map f c is taken as the value of all pixels, and the square sum represents the point-level density map loss corresponding to the background area of the block. Thus, the background suppression global loss based on binary segmentation is finally obtained.

[0124] Further, the self-supervised domain adaptation model of the present application adopts an unlabeled target domain crowd counting dataset as an adaptation target. For the pre-trained target counting model, the cross-domain task from the source domain model to the target domain dataset is realized through a consistency regularization task based on multiple outputs and a background suppression task using binary segmentation. The calculation method of the domain adaptation loss of the two-part self-supervised task is as follows.

[0125]

[0126] In the formula: Loss M is the consistency based on multiple outputs; Loss T is the background suppression global loss using binary segmentation. l is the consistency offset factor; Loss S is the comprehensive self-supervised loss based on uncertainty estimation.

[0127] Further, the present method obtains a comprehensive loss by comprehensively training two-part training tasks, and is used for model self-supervised training to realize domain adaptation from the source domain to the target domain. Without increasing the data labeling cost, the text is used as a cross-modal guide to optimize the problem that most target counting pre-trained models have weak ability to locate the correct type of counting target and poor prediction accuracy of the density map when migrating to a specific type of target counting task, and improve the performance of the model in the specified setting type counting task.

[0128] Step 4: use in the downstream target counting task

[0129] The self-supervised target counting fine-tuning method based on domain adaptation provided by the present application provides a general model fine-tuning method for any type of target counting task downstream, which can be applied to any target counting downstream task without any labeled training set, and the pre-trained model is fine-tuned for secondary training. In the experimental process, the Vision Transformer backbone network based on CLIP is used as the model network for visual feature extraction, and the generation of the prediction density map is realized based on the single-stage regression task.

[0130] Further, the present application can be applied to any target type downstream target counting task, including but not limited to crowd counting task, flying bird counting task, vehicle counting task, etc. When performing inference tasks in a certain type of downstream target counting task, the present application needs to use the English noun of the counting type as the natural language prompt to guide the target counting model to accurately focus on the object of interest in the input image. For example: in the crowd counting task, “people” is used as the prompt; in the flying bird counting task, “birds” is used as the prompt, and so on.

[0131] Further, referring to Figure 5 In the self-supervised domain adaptation of the present application, when estimating the uncertainty of the classification of different blocks, multiple sets of type text embedding features need to be preset, and the similarity between each set of type text embedding vector and the global class visual vector of different sub-blocks is calculated. For example: in the crowd counting task, “heads”, “trees”, “buildings”, “background” and the like can be used as prompt keywords for generating preset text embedding, and the division block with the global class visual vector closest to “heads” is considered as the foreground type, and vice versa.

[0132] The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A self-supervised object counting fine-tuning method based on domain adaptation, characterized by: include: Divide the given image into several sub-blocks with zero intersection-over-union ratio, and then obtain the same number of embedded vector visual feature word encodings; Use word encoding vectors as the representation of visual features; Preset multiple sets of text embedding features. Based on the global visual category vectors of the sub-blocks in the visual features, calculate the cosine similarity between each set of text embedding vectors and the global category visual vectors of different sub-blocks to estimate the uncertainty of the classification of different blocks. In the uncertainty estimation process, the image sub-blocks are divided into foreground sub-blocks and background sub-blocks; the foreground sub-blocks are scaled and then input into the density map regression network to obtain multiple sets of outputs; Regularized self-supervisory loss is performed based on the consistency of multiple outputs. At the same time, inconsistent results based on model outputs are used as self-supervised training signals for gradient descent of the weight network to achieve backpropagation and model fine-tuning. Based on binary segmentation, the point-level density noise containing background semantic information in the background sub-block is suppressed to complete the target count adjustment; The regularized self-supervisory loss based on multi-output consistency includes: using output results of different scales as pseudo labels of self-supervisory signals; designing adaptive smoothing mask weights to impose additional constraints on teacher labels generated by the network, and correcting the weights of output pseudo labels with different confidence levels to reduce the negative impact of invalid pseudo labels in the self-supervisory loss based on consistency regularization; The design of adaptive smoothing mask weights is specifically as follows: Adaptive smoothing mask weights impose additional constraints on the teacher labels produced by the multi-output density map generation network. The process is expressed as follows: （3） in: w i For the i mask weights; p i For the i Large-scale output prediction counts; ci For the i Small-scale output prediction counts; m is the high confidence ratio parameter; σ The degree of dispersion of the values ​​from the mean.

2. The domain-adaptive self-supervised object counting fine-tuning method according to claim 1, characterized in that: The obtaining of the same number of embedded vector visual feature word codes is specifically as follows: the obtained sub-blocks are passed through a CLIPViT-based visual feature preprocessor to generate the same number of embedded vector visual feature word codes as independent feature representations of different regions in the image.

3. The domain-adaptive self-supervised object counting fine-tuning method according to claim 2, characterized in that: The global visual category vector of the sub-block based on the visual features is specifically: each set of visual features includes a global visual embedding vector, and the global visual embedding vector includes the global visual category vector of the sub-block to which it belongs; The cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks is calculated as follows: the cosine similarity between the global visual embedding vectors of U different blocks and the V type text embeddings is calculated, and the similarity information is normalized using the softmax function to obtain a U×V similarity matrix. （1） in, I i and T j Represent the global visual embedding vector and the type text embedding vector respectively, τ represents the temperature coefficient; inner product I i T j Indicates the i image category information vector and the j The cosine similarity of the texts; softmax The function outputs different types of classification confidence, which is used to estimate the uncertainty of the types of different blocks. Set the counting object and compare and match the global visual embedding vector of each slice image with the text embedding of the counting object by cosine similarity, as shown in the following formula: （2） Where: D is a set of data sets, and is any ordered pair ( I i , T j ) establishes the association between dataset images and type texts, where I i and T j Represent the global visual embedding vector and the type text embedding vector respectively; For all a collection of permutations; For the i The ordinal number of the text embedding with the highest similarity to the image category vector.

4. The domain-adaptive self-supervised object counting fine-tuning method according to claim 3, characterized in that: The foreground sub-block is scaled and then input into the density map regression network to obtain multiple sets of outputs, specifically: based on the original image corresponding to the image sub-block; the original image is input into the model network to generate a set of embedded vector visual feature word codes, and finally output a point-level density map; the image sub-blocks are each input into the model network to generate 16 sets of embedded vector visual feature word codes, and finally output 16 point-level density maps; the 16 point-level density maps corresponding to the image sub-blocks are arranged in corresponding positions and correspond one-to-one to each sub-region of the point-level density map of the original image.

5. The domain-adaptive self-supervised object counting fine-tuning method according to claim 1, characterized in that: The multi-output consistency regularized self-supervisory loss is specifically: （4） The loss is calculated by traversing C cropped blocks of the counting target type, calculating the square of the error of the pixels at the same corresponding position one by one, and averaging them to obtain the mean square error at the matrix level; after multiplying the loss corresponding to each cropped block by the corresponding adaptive smoothing mask weight wi, the weighted sum of each sub-loss is calculated; Represents the predicted density map; B represents the length and width of the cropped block sub-image; Represents the pseudo label density map.

6. The domain-adaptive self-supervised object counting fine-tuning method according to claim 5, characterized in that: The inconsistent results of the model output are used as self-supervised training signals for the weight network gradient descent to achieve back propagation and model fine-tuning. Specifically, in the network back propagation training link, the pseudo-label density map is stripped f p The computational graph of , which is treated as a true value without gradient, and compared with the predicted density distribution map f c Establish consistency loss between them; finally, through f c The computation graph of is back-propagated to fine-tune the weights of the entire model network.

7. The domain-adaptive self-supervised object counting fine-tuning method according to claim 6, characterized in that: The binary segmentation-based method suppresses the point-level density noise containing background semantic information in the background sub-block. Specifically, each image sub-block is input into the model network and multiple sets of point-level density maps are output. In the uncertainty estimation, the blocks are divided into different types and corresponding mask weights are established, as shown in the following formula: （5） Where: IBM Ideal binary masking; For the i The ordinal number of the text embedding with the highest similarity between the image category vectors; For a set of preset text embedding vector serial numbers, the element with index 1 represents the target count type, and the element with index not 1 represents other irrelevant types; the global background suppression loss based on binary segmentation is calculated as follows: （6） Where: U is the number of cropping blocks; B The length and width of the cropped block sub-image; f c is the predicted density map; Loss T is the background suppression loss.

8. A domain-adaptive self-supervised target counting fine-tuning system implementing the method of claim 1, characterized in that: include: A partitioning module, which partitions a given image into a number of sub-blocks with an intersection-over-union ratio of zero, thereby obtaining the same number of embedded vector visual feature word codes; Use word encoding vectors as the representation of visual features; a calculation module, wherein the calculation module presets multiple groups of type text embedding features, calculates the cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks based on the global visual category vectors of the sub-blocks in the visual features, and estimates the uncertainty of the classification to which different blocks belong; An input module, wherein the input module divides the image sub-blocks into foreground sub-blocks and background sub-blocks during the uncertainty estimation process; performs a scale transformation operation on the foreground sub-blocks and then inputs them into a density map regression network to obtain multiple sets of outputs; A self-supervised loss module, which performs regularized self-supervised loss based on the consistency of multiple outputs and uses inconsistent results of model outputs as self-supervised training signals for gradient descent of the weight network to achieve backpropagation and model fine-tuning; The suppression module suppresses point-level density noise containing background semantic information in the background sub-block based on binary segmentation to complete target count adjustment.

Images