Self-supervised target counting fine tuning method and system based on domain self-adaption

Abstract

The invention discloses a self-supervision target counting fine tuning method and system based on domain self-adaption, and the method comprises the steps: carrying out the splitting of a given image, enabling each independent sub-block to pass through a model network independently, generating a plurality of groups of embedded vector visual feature word codes, and independent fine-grained feature representation of different areas in the image is realized. And then calculating cosine similarity between a type text embedding vector and a visual category information vector, and realizing deconstruction and classified discussion of a foreground region and a background region by establishing uncertainty estimation of different regions of the image. And the foreground sub-blocks and the background sub-blocks are subjected to self-supervision and two-dimensional segmentation respectively. According to the method, self-supervision field self-adaption from the source domain to the target domain is achieved, and the problems that when most target counting is migrated to a specific type of target counting task, the capacity of positioning a correct type of counting target is not high, and the prediction density map accuracy is not good are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and relates to a self-supervised target counting fine-tuning method and system based on domain adaptation. Background Art

[0002] With the rapid economic development and the continuous improvement of people's living standards, crowd gathering activities have become more frequent, which has brought great pressure to social security management and public transportation scheduling. The distribution and density of targets in statistical scenes have become a visual topic with important scientific research value. Target counting technology based on deep learning has been widely used in many fields, especially the target counting model based on the Transformer backbone network. In actual scenarios, target counting tasks are also widely used in many production fields such as agriculture, industry, and biological sciences, such as estimating the total number of crops in the field, counting the total number of industrial products on the factory production line, and counting the appearance of wild animals in the wild. With the wide expansion of application scenarios, target counting tasks have ushered in more challenges, including but not limited to unfamiliar types of counting objects, complex background interference, and uneven target distribution. These problems limit the counting performance of the algorithm and are also the difficulties that need to be solved urgently. Summary of the invention

[0003] The purpose of the present invention is to solve the problem in the prior art that the counting performance of the algorithm is limited when facing unfamiliar types of counting objects, complex background interference, and uneven target distribution, and to provide a self-supervised target counting fine-tuning method and system based on domain adaptation.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: The present invention provides a self-supervised target counting fine-tuning method based on domain adaptation, comprising: Divide a given image into several sub-blocks with an intersection-over-union ratio of zero, and then obtain the same number of embedded vector visual feature word encodings; use the word encoding vector as the representation of the visual feature; Preset multiple groups of text embedding features, calculate the cosine similarity between each group of text embedding vectors and the global visual category vectors of different sub-blocks based on the global visual category vectors of the sub-blocks in the visual features, and estimate the uncertainty of the classification of different blocks; In the process of uncertainty estimation, the image sub-blocks are divided into foreground sub-blocks and background sub-blocks; the foreground sub-blocks are scaled and then input into the density map regression network to obtain multiple groups of outputs; Regularized self-supervisory loss is performed based on the consistency of multiple outputs, and the inconsistent results of the model output are used as self-supervisory training signals for gradient descent of the weight network to achieve back propagation and model fine-tuning; Based on binary segmentation, the point-level density noise containing background semantic information in the background sub-block is suppressed to complete the target count adjustment.

[0005] Optionally, the obtaining of the same number of embedded vector visual feature word codes is specifically: passing the obtained sub-blocks through a CLIP ViT-based visual feature preprocessor to generate the same number of embedded vector visual feature word codes as independent feature representations of different regions in the image.

[0006] Optionally, the global visual category vector of the sub-block based on the visual feature is specifically: each group of visual features includes a global visual embedding vector, and the global visual embedding vector includes the global visual category vector of the sub-block to which it belongs; The cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks is calculated, specifically: the cosine similarity between the global visual embedding vectors of U different blocks and the V type text embeddings is calculated, and the similarity information is normalized using a softmax function to obtain a U×V similarity matrix. (1) in, I i and T j Represent the global visual embedding vector and the type text embedding vector respectively, τ represents the temperature coefficient; inner product I i T j Indicates i image category information vector and the j The cosine similarity of the texts; softmax The function outputs different types of classification confidence, which is used to estimate the uncertainty of the types to which different blocks belong; Set the counting object and compare and match the global visual embedding vector of each slice image with the text embedding of the counting object by cosine similarity, as shown in the following formula: (2) Where: D is a set of data sets, and is any ordered pair ( I i , T j ) establishes the association between dataset images and type text, where I i and T j Respectively represent i image category information vector and thej The inner product of the text embedding vectors; For all A collection of permutations; For the i The ordinal number of the text embedding with the highest similarity to the image category vector.

[0007] Optionally, the foreground sub-block is scaled and then input into a density map regression network to obtain multiple groups of outputs, specifically: based on the original image corresponding to the image sub-block; the original image is input into the model network to generate a group of embedded vector visual feature word codes, and finally output a point-level density map; the image sub-blocks are each input into the model network to generate 16 groups of embedded vector visual feature word codes, and finally output 16 point-level density maps; the 16 point-level density maps corresponding to the image sub-blocks are arranged in corresponding positions and will correspond one-to-one to each sub-area of ​​the point-level density map of the original image.

[0008] Optionally, the regularized self-supervisory loss based on multi-output consistency includes: using output results of different scales as pseudo-labels of self-supervisory signals; designing adaptive smoothing mask weights, imposing additional constraints on teacher labels generated by the network, and correcting the weights of output pseudo-labels with different confidence levels to reduce the negative impact of invalid pseudo-labels in the self-supervisory loss based on consistency regularization.

[0009] Optionally, the design of the adaptive smoothing mask weight is specifically: Adaptive smoothing mask weights impose additional constraints on the teacher labels produced by the multi-output density map generation network, which is expressed as follows: (3) in: w i For the i mask weights; p i For the i A large-scale output prediction count; ci For the i Small-scale output prediction counts; m is the high confidence ratio parameter; σ is the degree of dispersion of the values ​​from the mean.

[0010] Optionally, the multi-output consistency regularized self-supervision loss is specifically: (4) The loss is calculated by traversing C cropped blocks belonging to the counting target type, calculating the square of the error of the pixel points at the same corresponding position one by one, and averaging them to obtain the mean square error at the matrix level; after multiplying the loss corresponding to each cropped block by the corresponding adaptive smoothing mask weight wi, the weighted sum of each sub-loss is calculated.

[0011] Optionally, the inconsistent results based on the model output are used as self-supervised training signals for weight network gradient descent to achieve back propagation and model fine-tuning, specifically: in the network back propagation training link, the pseudo-label density map is stripped f p The computational graph is treated as a true value without gradient and compared with the predicted density distribution map f c Establish a consistent loss between them; finally, through f c The computation graph of is back-propagated to fine-tune the weights of the entire model network.

[0012] Optionally, the point-level density noise containing background semantic information in the background sub-block is suppressed based on binary segmentation, specifically: each image sub-block is input into the model network and multiple groups of point-level density maps are output; and corresponding mask weights are established for blocks divided into different types in uncertainty estimation, as shown in the following formula: (5) Where: IBM Ideal binary masking; For the i The ordinal number of the text embedding with the highest similarity among the image category vectors; For a set of preset text embedding vector serial numbers, the element with index 1 represents the target count type, and the element with index not 1 represents other irrelevant types; the global background suppression loss based on binary segmentation is calculated as follows: (6) Where: U is the number of cut blocks; B The length and width of the cropped block sub-image; f c is the predicted density map; Loss T is the background suppression loss.

[0013] The present invention also provides a domain-adaptive self-supervised target counting fine-tuning system, comprising: A partitioning module, wherein the partitioning module partitions a given image to obtain a plurality of sub-blocks with an intersection-over-union ratio of zero, and then obtains the same number of embedded vector visual feature word codes; and uses the word code vector as a representation of the visual feature; A calculation module, wherein the calculation module presets multiple groups of type text embedding features, calculates the cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks based on the global visual category vectors of the sub-blocks in the visual features, and estimates the uncertainty of the classification to which different blocks belong; An input module, wherein the input module divides the image sub-blocks into foreground sub-blocks and background sub-blocks during uncertainty estimation; performs a scale transformation operation on the foreground sub-blocks and then inputs them into a density map regression network to obtain multiple groups of outputs; A self-supervised loss module, which performs regular self-supervised loss based on the consistency of multiple outputs, and uses inconsistent results of model outputs as self-supervised training signals for gradient descent of the weight network to achieve back propagation and model fine-tuning; The suppression module suppresses the point-level density noise containing background semantic information in the background sub-block based on binary segmentation to complete the target count adjustment.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention splits a given image, and each independent sub-block will first be independently passed through the model network, and multiple sets of embedded vector visual feature word encodings will be generated as independent fine-grained feature representations of different regions in the image. Then the cosine similarity between the type text embedding vector and the visual category information vector is calculated, and the deconstruction and classification discussion of the foreground area and the background area are achieved by establishing uncertainty estimates for different areas of the image. The foreground sub-block and the background sub-block are self-supervised and two-dimensionally segmented respectively. The present invention realizes self-supervised domain adaptation from the source domain to the target domain, and uses text as a cross-modal guide without increasing the cost of data annotation, thereby optimizing the problem that most target counting pre-trained models are not capable of locating the correct type of counting targets and have poor accuracy in predicting density maps when migrating to specific types of target counting tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 It is a flowchart of the self-supervised target counting fine-tuning method based on domain adaptation of the present invention; Figure 2 It is a structural schematic diagram of the self-supervised target counting fine-tuning system based on domain adaptation of the present invention; Figure 3 It is the overall structure diagram of the model of the present invention; Figure 4 It is a schematic diagram of block zero-shot classification and density map regression prediction of the present invention; Figure 5 A schematic diagram of the network structure of the block uncertainty estimation task of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0018] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0020] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0021] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0022] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1 The present invention discloses a self-supervised target counting fine-tuning method based on domain adaptation, comprising: S101, dividing a given image to obtain a number of sub-blocks with an intersection-over-union ratio of zero, and then obtaining the same number of embedded vector visual feature word codes; using the word code vector as a representation of the visual feature; The obtained sub-blocks are passed through a visual feature preprocessor based on CLIP ViT to generate the same number of embedded vector visual feature word encodings as independent feature representations of different regions in the image.

[0024] S102, presetting multiple groups of type text embedding features, and calculating the cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks based on the global visual category vectors of the sub-blocks in the visual features, and estimating the uncertainty of the classification to which different blocks belong; A global visual category vector of a sub-block based on the visual features, specifically: each group of visual features includes a global visual embedding vector, and the global visual embedding vector includes a global visual category vector of the sub-block to which it belongs; The cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks is calculated, specifically: the cosine similarity between the global visual embedding vectors of U different blocks and the V type text embeddings is calculated, and the similarity information is normalized using a softmax function to obtain a U×V similarity matrix. (1) in, I i and T j Represent the global visual embedding vector and the type text embedding vector respectively, τ represents the temperature coefficient; inner product I i T j Indicates iimage category information vector and the j The cosine similarity of the texts; softmax The function outputs different types of classification confidence, which is used to estimate the uncertainty of the types to which different blocks belong; Set the counting object and compare and match the global visual embedding vector of each slice image with the text embedding of the counting object by cosine similarity, as shown in the following formula: (2) Where: D is the dataset set, and the association between the dataset image and the type text is established for any ordered pair (Ii, Tj), where Ii and Tj represent the inner product of the i-th image category information vector and the j-th text embedding vector respectively; For all A collection of permutations; For the i The ordinal number of the text embedding with the highest similarity to the image category vector.

[0025] S103, in the process of uncertainty estimation, dividing the image sub-blocks into foreground sub-blocks and background sub-blocks; performing a scale transformation operation on the foreground sub-blocks and inputting them into a density map regression network to obtain multiple groups of outputs; Based on the original image corresponding to the image sub-block; the original image is input into the model network to generate a set of embedded vector visual feature word encodings, and finally output a point-level density map; the image sub-blocks are respectively input into the model network to generate 16 sets of embedded vector visual feature word encodings, and finally output 16 point-level density maps; the 16 point-level density maps corresponding to the image sub-blocks are arranged in corresponding positions and will correspond one by one to each sub-area of ​​the point-level density map of the original image.

[0026] S104, regularizing self-supervisory loss based on the consistency of multiple outputs, and using inconsistent results of model output as self-supervisory training signals for gradient descent of weight network to achieve back propagation and model fine-tuning; A regularized self-supervised loss is performed based on the consistency of multiple outputs, including: using output results of different scales as pseudo-labels of self-supervised signals; designing adaptive smoothing mask weights, imposing additional constraints on the teacher labels generated by the network, and correcting the weights of output pseudo-labels with different confidence levels to reduce the negative impact of invalid pseudo-labels in the self-supervised loss based on consistency regularization.

[0027] Design adaptive smoothing mask weights, specifically: Adaptive smoothing mask weights impose additional constraints on the teacher labels produced by the multi-output density map generation network, which is expressed as follows: (3) Where: wi is the i-th mask weight; pi is the i-th large-scale output prediction count; ci is the i-th small-scale output prediction count; m is the high confidence ratio parameter; σ is the degree of dispersion of the value from the mean value.

[0028] The consistency regularized self-supervision loss based on multiple outputs is: (4) The loss is calculated by traversing C cropped blocks belonging to the counting target type, calculating the square of the error of the pixel points at the same corresponding position one by one, and averaging them to obtain the mean square error at the matrix level; after multiplying the loss corresponding to each cropped block by the corresponding adaptive smoothing mask weight wi, the weighted sum of each sub-loss is calculated.

[0029] The inconsistent results based on the model output are used as self-supervised training signals for the gradient descent of the weight network to achieve back propagation and model fine-tuning. Specifically, in the network back propagation training phase, the calculation graph of the pseudo-label density map fp is stripped off, and it is treated as a true value without gradient, and a consistency loss is established between it and the predicted density distribution map fc; finally, back propagation is performed through the calculation graph of fc to fine-tune the weights of the entire model network.

[0030] S105, based on binary segmentation, suppressing the point-level density noise containing background semantic information in the background sub-block to complete the target count adjustment.

[0031] Each image sub-block is input into the model network and multiple sets of point-level density maps are output; in uncertainty estimation, the blocks are divided into different types and the corresponding mask weights are established, as shown in the following formula: (5) Where: IBM is the ideal binary mask; is the sequence number of the text embedding with the highest similarity to the i-th image category vector; For a set of preset text embedding vector serial numbers, the element with index 1 represents the target count type, and the element with index not 1 represents other irrelevant types; the global background suppression loss based on binary segmentation is calculated as follows: (6) Where: U is the number of cropped blocks; B is the length and width of the cropped block sub-image; fc is the predicted density map; LossT is the background suppression loss.

[0032] See also Figure 2 The present invention discloses a self-supervised target counting fine-tuning system based on domain adaptation, comprising: A partitioning module, wherein the partitioning module partitions a given image to obtain a plurality of sub-blocks with an intersection-over-union ratio of zero, and then obtains the same number of embedded vector visual feature word codes; and uses the word code vector as a representation of the visual feature; A calculation module, wherein the calculation module presets multiple groups of type text embedding features, calculates the cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks based on the global visual category vectors of the sub-blocks in the visual features, and estimates the uncertainty of the classification to which different blocks belong; An input module, wherein the input module divides the image sub-blocks into foreground sub-blocks and background sub-blocks during uncertainty estimation; performs a scale transformation operation on the foreground sub-blocks and then inputs them into a density map regression network to obtain multiple groups of outputs; A self-supervised loss module, which performs regular self-supervised loss based on the consistency of multiple outputs, and uses inconsistent results of model outputs as self-supervised training signals for gradient descent of the weight network to achieve back propagation and model fine-tuning; The suppression module suppresses the point-level density noise containing background semantic information in the background sub-block based on binary segmentation to complete the target count adjustment Example: The present invention discloses a self-supervised target counting fine-tuning method based on domain adaptation, comprising: Step 1: For a given data set, preprocess the image and input it into the backbone network to generate multiple sets of feature outputs Given a dataset with dimensions The input image , according to its actual aspect ratio, the image is cropped to obtain a square image with equal length and width as large as possible through random displacement of the sliding window, and the image dimension becomes or . After that, the dimension is converted to , and then divide it into 16 sub-blocks according to 4×4 equal parts, and the dimension of each sub-block is .

[0033] For further information, see Figure 3 ,Each sub-block passes through the visual feature preprocessor based on CLIP ViT to generate 16 sets of embedding vector visual feature word encodings with a feature dimension of 197×512, which serve as independent feature representations of different regions in the image.

[0034] Step 2: Calculate the cosine similarity between the text embedding vectors of each group type and the global category visual vectors of different sub-blocks to estimate the uncertainty of the classification of different blocks See also Figure 3, the multiple sets of feature embedding vectors with a dimension of 197×512 generated by the image sub-blocks are visual feature word encodings. In addition to the block-level embedding vector, they all contain a global category embedding vector with a dimension of 1×512, which contains the global category information of the block to which it belongs, and will be used to estimate the uncertainty of the classification to which different blocks belong.

[0035] Furthermore, for the preset multiple types of text embeddings, it is necessary to calculate the cosine similarities between the global visual embedding vectors of U different blocks and the V types of text embeddings, and use the softmax function to normalize these similarity information to obtain the U×V similarity matrix.

[0036] (7) Among them, Ii and Tj represent the global visual embedding vector and the type text embedding vector respectively, and τ represents the temperature coefficient. The inner product IiTj represents the cosine similarity between the i-th image category information vector and the j-th text. Finally, the softmax function is used to output different types of classification confidence.

[0037] Furthermore, taking the crowd counting task as an example, since the counting object is "human head", each slice image is used to embed the text of "A photo of heads" for cosine similarity comparison and matching, as shown in the following formula.

[0038] (8) Where: D is a set of data, using any ordered pair ( I i , T j ) establishes the association between dataset images and type text, where I i and T j Respectively represent i image category information vector and the j The inner product of text embedding vectors; For all A collection of permutations; is the sequence number of the text embedding with the highest similarity to the i-th image category vector.

[0039] Step 3: Design of point-level density map self-supervised loss function Data enhancement was applied during the training phase, i.e., Gaussian noise, Gaussian blur, color jitter, etc. were used to change and expand the images in the training set. In addition, horizontal flipping and rotation were used to simulate the perspectives of images in different directions and positions in real scenes. The original images of the expanded training set and the true values ​​of the density map were used as input for supervised training of the model network.

[0040] See also Figure 3 , for a series of dimensions The size of the image sub-block corresponding to the original image area is The original image is input into the model network, generating a set of embedding vectors with a feature dimension of 197×512, visual feature word encoding, and finally outputting a dimension of The image sub-blocks are input into the model network to generate 16 sets of embedding vectors with a feature dimension of 197×512, visual feature word encoding, and finally output 16 images with a dimension of The 16 point-level density maps corresponding to the image sub-blocks are arranged in corresponding positions and correspond one by one to each sub-region of the point-level density map of the original image.

[0041] For further information, see Figure 4 , for the image sub-block that is divided into foreground in uncertainty estimation, it is the area in the image containing the counting target, corresponding to the point-level density map with a non-zero value. The multi-output-based consistency regularization loss proposed in the present invention uses the output results of different scales as pseudo-labels of the self-supervisory signal, that is, the predicted output result of the model at one scale is used as the pseudo-label of the predicted output result at another scale. The pseudo-label of the output pseudo-label is used as a signal to supervise another output result, and an additional regularization term is proposed for model training, so that the gradient descent training direction of the weight network tends to be close to the pseudo-label. The actual optimization goal is to minimize the difference between multiple output results of the counting model, and enhance the robustness of the model through self-supervised training.

[0042] Furthermore, in order to solve the problem of model training direction deviation that may be caused by the confidence of the pseudo-label teacher output, the present invention adaptively smooths the mask weights and imposes additional constraints on the teacher labels generated by the multi-output density map generation network. The process can be expressed as follows: (9) Where: w i For the i mask weights; p i For the i A large-scale output prediction count; ci For thei Small-scale output prediction counts; m is the high confidence ratio parameter; σ is the degree of dispersion of the values ​​from the mean.

[0043] Furthermore, in the mask weight map w In i The self-supervised training weights corresponding to the cropped blocks belonging to the counting target type are recorded as w i . p i Represents the target count calculated for the scaled cropped area. c i It represents the target count of the corresponding sub-block when predicting the original image. The ratio of the two reflects the inconsistency of the model under multi-scale output conditions, and serves as a self-supervisory signal to guide the domain adaptation task. Specifically, when the ratio of the predicted number result after scaling to the number of corresponding areas of the overall predicted output is closer to the high confidence ratio parameter m , indicating that the higher the confidence of the pseudo-label of the cropped block, the higher the corresponding self-supervised training weight. In addition, the ratio of the multi-output results and the high confidence ratio parameter m The deviation is greater than the discrete value σ When , the self-supervised training weight of the cut block w i It is recorded as zero, indicating that the multi-scale prediction results of this block are not sufficient to be used as a self-supervisory signal.

[0044] Furthermore, in the design of the self-supervised loss function, an adaptive smoothing mask is used to assign different weights to different blocks, so that during the model fine-tuning process, the recall rate of multi-output pseudo-labels in different situations is limited, and more attention is paid to effective self-supervised signals, reducing the negative impact of invalid pseudo-labels in the self-supervised loss based on consistency regularization. The block-level consistency regularization loss function based on multi-output is as follows:

[0045] The loss is calculated by traversing C cropped blocks of the counting target type, calculating the square of the error of the pixels at the same corresponding position one by one, and then averaging to get the mean square error at the matrix level. Then multiply the loss corresponding to each cropped block by the corresponding adaptive smoothing mask weight w i After that, the weighted sum of each sub-loss is calculated. In this way, a multi-output consistency regularization will be established between multiple output point-level density maps of different blocks. The essence of its optimization goal is to minimize the difference between multiple output results of the counting model and enhance the robustness of the model.

[0046] Furthermore, in the network back-propagation training phase, it is necessary to remove the pseudo-label density map f p The computational graph is treated as a true value without gradient and compared with the predicted density distribution map f c Finally, through f c The computation graph of is back-propagated to fine-tune the weights of the entire model network.

[0047] For further information, see Figure 4 , for the image sub-blocks that are classified as background in uncertainty estimation, as the area in the image that does not contain counting targets, it corresponds to a point-level density map with a value of zero. For all image sub-blocks, they are input into the model network, and multiple sets of point-level density maps with dimensions of will be output. In order to achieve background suppression, the corresponding mask weights are established for the blocks that are divided into different types in uncertainty estimation, as shown in the following formula:

[0048] In the formula, IBM It is an ideal binary mask; For the i The ordinal number of the text embedding with the highest similarity to the image category vector.

[0049] Furthermore, for a set of preset text embedding vector numbers, the element with index 1 represents the target count type, and the element with index not 1 represents other irrelevant types. The global background suppression loss based on binary segmentation is calculated as follows:

[0050] In the formula, U is the number of cut blocks; B The length and width of the cropped block sub-image; f c is the predicted density map; Loss T is the background suppression loss.

[0051] Furthermore, the background suppression loss is calculated using the mask weight map IBM Take the binary mask intersection for all cropped blocks and assign corresponding weights. The classification result is an irrelevant type of cropped area, which represents the background area in the image, corresponding to the target density truth map which should be all zero. Therefore, the predicted density map f c The square sum of all pixel values ​​of represents the point-level density map loss corresponding to the block as the background area. Thus, the global loss of background suppression based on binary segmentation is finally obtained.

[0052] Furthermore, the self-supervised domain adaptation model of the present invention uses an unlabeled target domain crowd counting dataset as the adaptation target. For the pre-trained target counting model, a cross-domain task from the source domain model to the target domain dataset is achieved through a multi-output consistency regularization task and a background suppression task using binary segmentation. The calculation method of the domain adaptation loss of the two parts of the self-supervised task is as follows.

[0053]

[0054] Where: Loss M For consistency based on multiple outputs; Loss T A global loss for background suppression using binary segmentation. λ is the consistency offset factor; Loss S is a comprehensive self-supervised loss based on uncertainty estimation.

[0055] Furthermore, this method combines the two parts of the training task to obtain a comprehensive loss, which is used for model self-supervision training to achieve domain adaptation from the source domain to the target domain. Without increasing the cost of data annotation, using text as a cross-modal guide optimizes the problem that most target counting pre-trained models have poor ability to locate the correct type of counting targets and poor accuracy in predicting density maps when migrating to specific types of target counting tasks, thereby improving the performance of the model in the specified setting type counting tasks.

[0056] Step 4: Use in downstream object counting tasks The present invention provides a domain-adaptive self-supervised target counting fine-tuning method, which provides a general model fine-tuning method for any type of downstream target counting tasks. The self-supervised method can be applied to any downstream target counting task without any labeled training set, and the pre-trained model can be trained and fine-tuned for a second time. In the experiment, the present invention uses the CLIP-based Vision Transformer backbone network as the model network for visual feature extraction, and realizes the generation of prediction density maps based on a single-stage regression task.

[0057] Furthermore, the present invention can be applied to downstream target counting tasks of any target type, including but not limited to crowd counting tasks, bird counting tasks, vehicle counting tasks, etc. When performing reasoning tasks in a specific type of downstream target counting task, the present invention needs to use English nouns of the counting type as natural language prompts to guide the target counting model to accurately focus on the objects of interest in the input image. For example: in the crowd counting task, "people" needs to be used as a prompt; in the bird counting task, "birds" needs to be used as a prompt, and so on.

[0058] For further information, see Figure 5 In the self-supervised domain adaptation of the present invention, when estimating the uncertainty of the classification to which different blocks belong, it is necessary to preset multiple groups of type text embedding features and calculate the similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks. For example: in the crowd counting task, "heads", "trees", "buildings", "background" and so on can be used as prompt keywords for generating preset text embeddings, and the divided blocks whose similarity matches the global category visual vector closest to "heads" are regarded as foreground types, otherwise they are background types.

[0059] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A self-supervised target counting fine-tuning method based on domain adaptation, characterized in that: include: Divide the given image into several sub-blocks with zero intersection-over-union ratio, and then obtain the same number of embedded vector visual feature word encodings; Using word encoding vectors as the representation of visual features; Preset multiple groups of text embedding features, calculate the cosine similarity between each group of text embedding vectors and the global visual category vectors of different sub-blocks based on the global visual category vectors of the sub-blocks in the visual features, and estimate the uncertainty of the classification of different blocks; In the process of uncertainty estimation, the image sub-blocks are divided into foreground sub-blocks and background sub-blocks; the foreground sub-blocks are scaled and then input into the density map regression network to obtain multiple groups of outputs; Regularized self-supervisory loss is performed based on the consistency of multiple outputs, and the inconsistent results of the model output are used as self-supervisory training signals for gradient descent of the weight network to achieve back propagation and model fine-tuning; Based on binary segmentation, the point-level density noise containing background semantic information in the background sub-block is suppressed to complete the target count adjustment.

2. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 1, characterized in that: The obtaining of the same number of embedded vector visual feature word codes is specifically as follows: the obtained sub-blocks are passed through a CLIPViT-based visual feature preprocessor to generate the same number of embedded vector visual feature word codes as independent feature representations of different regions in the image.

3. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 2, characterized in that: The global visual category vector of the sub-block based on the visual feature is specifically: each group of visual features includes a global visual embedding vector, and the global visual embedding vector includes the global visual category vector of the sub-block to which it belongs; The cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks is calculated, specifically: the cosine similarity between the global visual embedding vectors of U different blocks and the V type text embeddings is calculated, and the similarity information is normalized using a softmax function to obtain a U×V similarity matrix. （1） in, I i and T j Represent the global visual embedding vector and the type text embedding vector respectively, τ represents the temperature coefficient; inner product I i T j Indicates i image category information vector and the j The cosine similarity of the texts; softmax The function outputs different types of classification confidence, which is used to estimate the uncertainty of the types to which different blocks belong; Set the counting object and compare and match the global visual embedding vector of each slice image with the text embedding of the counting object by cosine similarity, as shown in the following formula: （2） Where: D is a set of data sets, and is any ordered pair ( I i , T j ) establishes the association between dataset images and type text, where I i and T j Respectively represent i image category information vector and the j The inner product of the text embedding vectors; For all A collection of permutations; For the i The ordinal number of the text embedding with the highest similarity to the image category vector.

4. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 3, characterized in that: The scale transformation operation is performed on the foreground sub-block and then input into the density map regression network to obtain multiple groups of outputs, specifically: based on the original image corresponding to the image sub-block; the original image is input into the model network to generate a group of embedded vector visual feature word codes, and finally output a point-level density map; the image sub-blocks are respectively input into the model network to generate 16 groups of embedded vector visual feature word codes, and finally output 16 point-level density maps; the 16 point-level density maps corresponding to the image sub-blocks are arranged in corresponding positions and will correspond one by one to each sub-region of the point-level density map of the original image.

5. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 4, characterized in that: The regularized self-supervisory loss based on multi-output consistency includes: using output results of different scales as pseudo labels of self-supervisory signals; designing adaptive smoothing mask weights, imposing additional constraints on teacher labels generated by the network, and correcting the weights of output pseudo labels with different confidence levels to reduce the negative impact of invalid pseudo labels in the self-supervisory loss based on consistency regularization.

6. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 5, characterized in that: The design of adaptive smoothing mask weights is specifically: Adaptive smoothing mask weights impose additional constraints on the teacher labels produced by the multi-output density map generation network, which is expressed as follows: （3） in: w i For the i mask weights; p i For the i A large-scale output prediction count; ci For the i Small-scale output prediction counts; m is the high confidence ratio parameter; σ is the degree of dispersion of the values ​​from the mean.

7. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 6, characterized in that: The multi-output-based consistency regularized self-supervision loss is specifically: （4） The loss is calculated by traversing C cropped blocks belonging to the counting target type, calculating the square of the error of the pixel points at the same corresponding position one by one, and averaging them to obtain the mean square error at the matrix level; after multiplying the loss corresponding to each cropped block by the corresponding adaptive smoothing mask weight wi, the weighted sum of each sub-loss is calculated.

8. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 7, characterized in that: The inconsistent results based on the model output are used as self-supervised training signals for weight network gradient descent to achieve back propagation and model fine-tuning. Specifically, in the network back propagation training phase, the pseudo-label density map is stripped f p The computational graph is treated as a true value without gradient and compared with the predicted density distribution map f c Establish a consistent loss between them; finally, through f c The computation graph of is back-propagated to fine-tune the weights of the entire model network.

9. The method for fine-tuning self-supervised target counting based on domain adaptation according to claim 8, characterized in that: The point-level density noise containing background semantic information in the background sub-block is suppressed based on binary segmentation, specifically: each image sub-block is input into the model network and multiple groups of point-level density maps are output; the blocks divided into different types in uncertainty estimation are established with corresponding mask weights, as shown in the following formula: （5） Where: IBM Ideal binary masking; For the i The ordinal number of the text embedding with the highest similarity among the image category vectors; For a set of preset text embedding vector serial numbers, the element with index 1 represents the target count type, and the element with index not 1 represents other irrelevant types; the global background suppression loss based on binary segmentation is calculated as follows: （6） Where: U is the number of cut blocks; B The length and width of the cropped block sub-image; f c is the predicted density map; Loss T is the background suppression loss.

10. A self-supervised object counting fine-tuning system based on domain adaptation, characterized in that: include: A partitioning module, wherein the partitioning module partitions a given image to obtain a plurality of sub-blocks with an intersection-over-union ratio of zero, thereby obtaining the same number of embedded vector visual feature word codes; Using word encoding vectors as the representation of visual features; A calculation module, wherein the calculation module presets multiple groups of type text embedding features, calculates the cosine similarity between each group of type text embedding vectors and the global category visual vectors of different sub-blocks based on the global visual category vectors of the sub-blocks in the visual features, and estimates the uncertainty of the classification to which different blocks belong; An input module, wherein the input module divides the image sub-blocks into foreground sub-blocks and background sub-blocks during uncertainty estimation; performs a scale transformation operation on the foreground sub-blocks and then inputs them into a density map regression network to obtain multiple groups of outputs; A self-supervised loss module, which performs regular self-supervised loss based on the consistency of multiple outputs, and uses inconsistent results of model outputs as self-supervised training signals for gradient descent of the weight network to achieve back propagation and model fine-tuning; The suppression module suppresses the point-level density noise containing background semantic information in the background sub-block based on binary segmentation to complete the target count adjustment.

Images