A method for training an unsupervised object detection model based on metric learning

Abstract

This invention discloses an unsupervised object detection model training method based on metric learning, comprising the following steps: S1, extracting objects from images in the training dataset using an open-source pre-trained model to obtain pseudo-labels; S2, the object detection model to be trained includes an image encoding backbone model, a feature pyramid network, and a detection head; inputting images from the training dataset into the object detection model to be trained to obtain the overall image i and the region images of each object; determining the pseudo-labels of all objects obtained by the object detection model to be trained, and then extracting the feature vectors of the objects obtained by the object detection model to be trained; S3, training the object detection model to be trained based on metric learning. This scheme is applicable to the training of most common object detection models.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and in particular to a method for training an unsupervised object detection model based on metric learning. Background Technology

[0002] In recent years, object detection has gradually become an important research topic in the field of computer vision and has been widely applied in areas such as the industrial internet, security, medical auxiliary diagnosis, and remote sensing image analysis. Object detection primarily involves identifying the location and category of objects of interest from visual images or videos. Deep learning has made significant progress in object detection due to its ability to deeply extract image features. However, current mainstream object detection models still require a large amount of manually labeled data for training. The high cost of labeling and the uncertainty of the objects of interest in object detection applications make training these models expensive and hinder their widespread application. Furthermore, in recent years, large-scale pre-trained models based on multimodal data have enabled deep learning to utilize multimodal data and textual information. However, current pre-trained models still have limitations in their ability to represent object features and extract detailed information about objects, restricting their effectiveness in practical applications.

[0003] To address these two challenges, how to improve the ability of object detection models to express object features and detailed information without using manually labeled data, and thus improve the effectiveness of object detection, is a problem that urgently needs to be solved. Summary of the Invention

[0004] This invention primarily addresses the technical problem of existing technologies that make it difficult to train object detection models and improve object detection performance without manually labeled data. It provides an unsupervised object detection model training method based on metric learning, which can complete the training process of the object detection model without manually labeled data.

[0005] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a method for training an unsupervised object detection model based on metric learning, comprising the following steps:

[0006] S1. Extract targets from the images in the training dataset using an open-source pre-trained model to obtain pseudo-labels. The training dataset consists of image-text pairs, which contain images and text labels describing the images. The pseudo-labels include the coordinate information of the target and the descriptive text of the target.

[0007] S2. The target detection model to be trained includes an image encoding backbone model, a feature pyramid network, and a detection head. After random initialization or loading of pre-trained model detection points (loading pre-trained model detection points also indicates that the target detection model has obtained basic parameters and can perform the target detection process), the images from the training dataset are input into the target detection model to be trained. The results include the overall image i and the region images of each target. Target j is one of the targets obtained by the target detection model to be trained. Target j is compared with the targets obtained by the open-source pre-trained model. The pseudo-labels of targets j' with an overlap greater than 0.5 are used as the pseudo-labels of target j, and the same method is used to determine the pseudo-labels of targets j. The training object detection model obtains pseudo-labels for all targets, and then extracts the feature vectors of the targets obtained by the training object detection model. The detection head is used to detect targets from the overall image i. Generally, the overall image i is first input into the image coding backbone model to obtain the overall multi-layer feature map of image i, and then the feature pyramid network is used to extract the overall feature vector from the overall multi-layer feature map. Finally, the detection head detects targets from the overall image i based on the overall feature vector. The overlap is the intersection of the two pseudo-labels divided by the union of the two pseudo-labels. If there are multiple pseudo-labels with an overlap greater than 0.5, the pseudo-label with the largest overlap is selected as the pseudo-label of target j. If there is no pseudo-label with an overlap greater than 0.5, the target is discarded and not added to the training.

[0008] S3. Train the target detection model based on the metric model. For each target j, the loss function is:

[0009] L=λ metric ·L metric +λ L1 ·L L1 +λ giou ·L giou +λ cntrst ·L cntrst

[0010] Among them, L metric The loss is learned as a measure of the similarity of the target bounding boxes to the triplet, λ. metric For L metric The corresponding weight; L L1 For the L1 loss of the target box position, λ L1 For L L1 The corresponding weight; L giou For the location of the target bounding box, GIOU loss, λ giou For L giou The corresponding weight; L cntrst For image-text contrast loss, λ cntrst For L cntrst The corresponding weights.

[0011] This approach is applicable to training any conventional object detection model, as long as the model comprises three parts: an image encoding backbone model, a feature pyramid network, and a detection head. The open-source pre-trained model can be a standard, pre-trained open-source object detection model, provided it can output both object detection results and descriptive text.

[0012] Preferably, in step S2, the specific process of extracting the feature vector of target j is as follows:

[0013] S201. Input the overall image i into the image coding backbone model, and extract the features output by each output layer to obtain the overall multi-layer feature map of image i.

[0014] S202. Based on the position of the region image of target j in image i, extract the target multi-layer feature map from the overall multi-layer feature map;

[0015] S203. Use a feature pyramid network to extract and pool the multi-layer feature map of the target to obtain the feature vector of target j.

[0016] Preferably, the learning loss for the triplet similarity metric is:

[0017] L metric =max(S(a,p)-S(a,n)+margin,0)

[0018] In the formula, max represents the maximum value, and S is the cosine distance between the two target feature vectors within the parentheses. The specific formula is as follows:

[0019]

[0020] In the formula, T represents transpose, and the double vertical lines represent calculating the vector length, i.e., norm2; the definition of S(a,n) is the same as that of S(a,p), except that p is replaced by n; a is the feature vector of the baseline target sample, p is the feature vector of the positive target sample, and n is the feature vector of the negative target sample; margin is the preset interval parameter; during training, each target j generated by the target detection model to be trained and matched with the pseudo-label is a baseline target sample; for each baseline target sample j, the target with the farthest cosine distance from the target feature vector of target j and containing the same pseudo-label description text is selected as the positive target sample; the target with the closest cosine distance from the target feature vector of target j and containing different pseudo-label description text is selected as the negative target sample.

[0021] By learning the loss through the metric of the target triples, the network module of the target detection model can extract detailed features of the target in different text descriptions, thereby improving the model's ability to recognize targets with fine granularity.

[0022] As a preferred option, the L1 loss for the target box position is as follows:

[0023]

[0024] In the formula, ti represents the bounding box coordinates of target j obtained by the target detection model to be trained, which is a four-point vector {tx, ty, tw, th}, representing the position of the coordinates on the x-axis and y-axis, and the width w and height h of the target, respectively. ti* represents the bounding box coordinates of target j' obtained by the open-source pre-trained model, which is a four-point vector {tx*, ty*, tw*, th*}.

[0025] As a preferred option, the GIOU loss for the target box position is as follows:

[0026]

[0027] In the formula, A represents the bounding box of target j detected by the target detection model to be trained, B is the bounding box of target j' corresponding to target j detected by the open-source pre-trained model, C represents the area of ​​the minimum bounding rectangle of the two bounding boxes A and B, and IOU is the overlap between A and B. A, B, and C are all areas.

[0028] As a preferred option, the image-text comparison loss is specifically as follows:

[0029]

[0030] In the formula, N represents the total number of image-text pairs used for training. The training samples include the overall image and its text labels (derived from the data). v_m is the overall multi-layer feature map obtained after the m-th image-text pair sample is processed by the target detection model to be trained (obtained in step S201). l_m is the text representation obtained after the text label of the m-th image-text pair sample is processed by the text encoder. The text encoder uses Bert or Roberta, but it does not need to be the text encoder of the S1 model. The text encoder participates in training but is not updated. The calculation formula for p(v_m, l_m) is as follows:

[0031]

[0032] In the formula, S is the similarity between the two objects in parentheses, τ is the temperature hyperparameter, and Nri is the set of text representations obtained by the text encoder after the text labels of the other image-text pairs in the same batch of training samples, except for the m-th image-text pair sample. That is, the text label k does not match the image i, but matches other images k in the same batch (training sample set).

[0033] The open-source pre-trained model includes an image encoder and a text encoder, which are aligned in semantic space during pre-training. The image encoder generates the coordinate information of target j in image i and extracts target features for each target. The text encoder extracts features for each word / phrase of the text corresponding to the image. Utilizing the multimodal alignment capability of the open-source pre-trained model, the target features of the image and the features of each word / phrase of the text are multiplied. The result of this multiplication is used as the alignment score between the image target and the text word / phrase. For each target j, the text word / phrase with the highest alignment score is selected as the descriptive text for that target.

[0034] The substantial effect of this invention is that it enables unsupervised target detection training using multimodal image and text data and large-scale pre-training, without the need for manually labeled data. First, the target detection model is initialized using a large-scale pre-trained model, generating pseudo-label information for targets and their labels. Then, using this pseudo-label information, metric learning is performed on the region representation of image targets; specifically, a triplet similarity metric model is employed to distinguish detailed information between targets, thereby fully extracting fine-grained information about image targets and improving target detection performance. Attached Figure Description

[0035] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0036] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0037] Example: A method for training an unsupervised object detection model based on metric learning, such as... Figure 1 As shown, it includes the following steps:

[0038] S1. Extract targets from the images in the training dataset using an open-source pre-trained model to obtain pseudo-labels. The training dataset consists of image-text pairs, which contain images and text labels describing the images. The pseudo-labels include the coordinate information of the target and the descriptive text of the target.

[0039] S2. The target detection model to be trained includes an image coding backbone model, a feature pyramid network, and a detection head. After random initialization or loading of pre-trained model detection points, the images from the training dataset are input into the target detection model. The results include the overall image i and the region images of each target. Target j is one of the targets obtained by the target detection model to be trained. Target j is compared with the targets obtained by the open-source pre-trained model, and the pseudo-labels of targets j' with an overlap greater than 0.5 are used as the pseudo-labels of target j. The pseudo-labels of all targets obtained by the target detection model to be trained are determined in the same way. Then, the feature vector of the target obtained by the target detection model to be trained is extracted; the detection head is used to detect the target from the overall image i; generally, the overall image i is first input into the image coding backbone model to obtain the overall multi-layer feature map of image i, and then the feature pyramid network is used to extract the overall feature vector from the overall multi-layer feature map. Finally, the detection head detects the target from the overall image i based on the overall feature vector; the overlap is the intersection of the two divided by the union of the two; if there are multiple pseudo-labels with an overlap greater than 0.5, the pseudo-label with the largest overlap is selected as the pseudo-label of target j; if there is no pseudo-label with an overlap greater than 0.5, the target is discarded and not added to the training;

[0040] S3. Train the target detection model based on the metric model. For each target j, the loss function is:

[0041] L=λ metric ·L metric +λ L1 ·L L1 +λ giou ·L giou +λ cntrst ·L cntrst

[0042] Among them, L metric The loss is learned as a measure of the similarity of the target bounding boxes to the triplet, λ. metric For L metric The corresponding weight; L L1 For the L1 loss of the target box position, λ L1 For L L1 The corresponding weight; L giou For the location of the target bounding box, GIOU loss, λ giou For L giou The corresponding weight; L cntrst For image-text contrast loss, λ cntrst For L cntrst The corresponding weights.

[0043] This approach is applicable to training any conventional object detection model, as long as the model comprises three parts: an image encoding backbone model, a feature pyramid network, and a detection head. The image encoding backbone model can be any image-based neural network model, such as ResNet, ConvNext, or Vision Transformer. The object detection result containing object j includes image i and an image of the region containing object j. Image i is typically a natural image with three color channels; the image of the region containing object j can be understood as an image cropped from image i, specifically a region enclosed by several coordinates. The open-source pre-trained model can be a conventional, already trained open-source object detection model, as long as it can output object detection results and descriptive text.

[0044] In step S2, the specific process of extracting the feature vector of target j is as follows:

[0045] S201. Input the overall image i into the image coding backbone model, and extract the features output by each output layer to obtain the overall multi-layer feature map of image i.

[0046] S202. Based on the position of the region image of target j in image i, extract the target multi-layer feature map from the overall multi-layer feature map;

[0047] S203. Use a feature pyramid network to extract and pool the multi-layer feature map of the target to obtain the feature vector of target j.

[0048] Extraction pooling involves using the output layers {C2,C3,C4,C5} of conv2, conv3, conv4, and conv5 in the image encoding backbone model as features of the FPN. For image target i, pooling operations are performed at the positions of target {C2,C3,C4,C5}, such as ROIAlign, to obtain the feature vector of target j.

[0049] The specific steps are as follows:

[0050] 1) Suppose the input is an 800*800 image, containing a 640*640 target j. After feature extraction by the backbone network, the feature map is scaled with a stride of 32, resulting in a 25*25 full-image feature map. For target j, the size of the target feature map is (640 / 32=)20*20;

[0051] 2) Assuming the feature pooling within the bounding box needs to be 7*7 in size, the 20*20 target feature map mapped on the feature map will be divided into 49 small regions of equal size. The size of each small region is 20 / 7 = 2.86, that is, 2.86*2.86.

[0052] 3) Assuming the number of sampling points is 4, this means that for each small region of 2.86*2.86, it is divided into four equal parts. The center point of each part is taken, and the pixel value at the center point is calculated using bilinear interpolation. This will yield the pixel values ​​of four points. Finally, the maximum value of the four pixel values ​​is taken as the pixel value of this small region. This process is repeated to obtain 49 pixel values, which become a 7x7 feature map.

[0053] For target j, the feature maps obtained by pooling {C2, C3, C4, C5} are concatenated to obtain the feature map of target j. This feature map is then transformed by a non-linear transformation (fully connected layer of a neural network) to map the target feature representation to the space of the contrastive loss. For target j in image i, the target is extracted into a feature vector v_ij by an image region encoder.

[0054] The learning loss for the triplet similarity metric is:

[0055] L metric =max(S(a,p)-S(a,n)+margin,0)

[0056] In the formula, max represents the maximum value, and S is the cosine distance between the two target feature vectors within the parentheses. The specific formula is as follows:

[0057]

[0058] In the formula, T represents transpose, and the double vertical lines represent calculating the vector length, i.e., norm2; the definition of S(a,n) is the same as that of S(a,p), except that p is replaced by n; a is the feature vector of the baseline target sample, p is the feature vector of the positive target sample, and n is the feature vector of the negative target sample; margin is a preset interval parameter; during training, each target j generated by the target detection model to be trained and matched with the pseudo-label is a baseline target sample; for each baseline target sample j, the target with the farthest cosine distance from the target feature vector of target j and containing the same pseudo-label description text is selected as the positive target sample; the target with the closest cosine distance from the target feature vector of target j and containing different pseudo-label description text is selected as the negative target sample.

[0059] The L1 loss for the target box position is as follows:

[0060]

[0061] In the formula, ti represents the bounding box coordinates of target j obtained by the target detection model to be trained, which is a four-point vector {tx, ty, tw, th}, representing the position of the coordinates on the x-axis and y-axis, and the width w and height h of the target, respectively. ti* represents the bounding box coordinates of target j' obtained by the open-source pre-trained model, which is a four-point vector {tx*, ty*, tw*, th*}.

[0062] As a preferred option, the GIOU loss for the target box position is as follows:

[0063]

[0064] In the formula, A represents the bounding box of target j detected by the target detection model to be trained, B is the bounding box of target j' corresponding to target j detected by the open source pre-trained model, C represents the minimum bounding rectangle area of ​​the two bounding boxes A and B, and IOU is the overlap between A and B.

[0065] The specific loss in image-text comparison is as follows:

[0066]

[0067] In the formula, N represents the total number of image-text pairs used for training. The training samples include the overall image and its text labels (derived from the data). v_m is the image representation of the m-th image-text pair after being extracted by the target detection model to be trained. l_m is the text representation of the text label of the m-th image-text pair after being processed by a text encoder. The text encoder can be Bert or Roberta; it does not need to be the text encoder of the S1 model. The text encoder participates in training but is not updated. The formula for calculating p(v_m, l_m) is as follows:

[0068]

[0069] In the formula, S is the similarity between the two objects in parentheses, τ is the temperature hyperparameter, and Nri is the set of text representations obtained by the text encoder after the text labels of the other image-text pairs in the same batch of training samples, except for the m-th image-text pair sample. That is, the text label k does not match the image i, but matches other images k in the same batch (training sample set).

[0070] The open-source pre-trained model includes an image encoder and a text encoder, which are aligned in semantic space during pre-training. The image encoder generates the coordinate information of target j in image i and extracts target features for each target. The text encoder extracts features for each word / phrase of the text corresponding to the image. Utilizing the multimodal alignment capability of the open-source pre-trained model, the target features of the image and the features of each word / phrase of the text are multiplied. The result of this multiplication is used as the alignment score between the image target and the text word / phrase. For each target j, the text word / phrase with the highest alignment score is selected as the descriptive text for that target.

[0071] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0072] Although this paper uses terms such as pseudo-labels, image coding backbone networks, and detection heads extensively, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A method for training an unsupervised object detection model based on metric learning, characterized in that, Includes the following steps: S1. Extract targets from the images in the training dataset using an open-source pre-trained model to obtain pseudo-labels. The training dataset consists of image-text pairs, which contain images and text labels describing the images. The pseudo-labels include the coordinate information of the target and the descriptive text of the target. S2. The target detection model to be trained includes an image encoding backbone model, a feature pyramid network, and a detection head. After random initialization or loading of pre-trained model detection points, the target detection model to be trained inputs images from the training dataset into the target detection model to be trained. The results obtained include the overall image i and the region images of each target. Target j is one of the targets obtained by the target detection model to be trained. Target j is compared with each target obtained by the open-source pre-trained model. The pseudo-label of target j' with an overlap greater than 0.5 is used as the pseudo-label of target j. The pseudo-labels of all targets obtained by the target detection model to be trained are determined in the same way. Then, the feature vectors of the targets obtained by the target detection model to be trained are extracted. S3. Train the target detection model based on the metric model. For each target j, the loss function is: Among them, L metric The loss is learned as a measure of the similarity of the target bounding boxes to the triplet, λ. metric For L metric The corresponding weight; L L1 For the L1 loss of the target box position, λ L1 For L L1 The corresponding weight; L giou For the location of the target bounding box, GIOU loss, λ giou For L giou The corresponding weight; L cntrst For image-text contrast loss, λ cntrst For L cntrst The corresponding weights; The learning loss for the triplet similarity metric is: L metric =max(S(a,p)-S(a,n)+margin,0) In the formula, max represents the maximum value, and S is the cosine distance between the two target feature vectors within the parentheses. The specific formula is as follows: In the formula, T represents transpose, double vertical lines represent the vector length, i.e., norm2; S(a,n) is defined the same as S(a,p), except that p is replaced by n; a is the feature vector of the baseline target sample, p is the feature vector of the positive target sample, and n is the feature vector of the negative target sample; margin is the preset interval parameter. The L1 loss for the target box position is as follows: In the formula, ti represents the bounding box coordinates of target j obtained by the target detection model to be trained. It is a four-point vector {tx, ty, tw, th}, which represents the position of the coordinates on the x-axis and y-axis and the width w and height h of the target, respectively; ti* represents the bounding box coordinates of target j' obtained by the open-source pre-trained model. It is a four-point vector {tx*, ty*, tw*, th*}.

2. The method for training an unsupervised object detection model based on metric learning according to claim 1, characterized in that, In step S2, the specific process of extracting the feature vector of target j is as follows: S201. Input the overall image i into the image coding backbone model, and extract the features output by each output layer to obtain the overall multi-layer feature map of image i. S202. Based on the position of the region image of target j in image i, extract the target multi-layer feature map from the overall multi-layer feature map; S203. Use a feature pyramid network to extract and pool the multi-layer feature map of the target to obtain the feature vector of target j.

3. A method for training an unsupervised object detection model based on metric learning according to claim 1 or 2, characterized in that, During training, each target j generated by the target detection model to be trained and matched with a pseudo-label is a baseline target sample. For each baseline target sample j, the target with the farthest cosine distance from the target feature vector of target j and containing the same pseudo-label description text is selected as a positive target sample. The target with the closest cosine distance from the target feature vector of target j and containing different pseudo-label description text is selected as a negative target sample.

4. The method for training an unsupervised object detection model based on metric learning according to claim 1, characterized in that, The GIOU loss for the target bounding box location is as follows: In the formula, A represents the bounding box of target j detected by the target detection model to be trained, B is the bounding box of target j' corresponding to target j detected by the open source pre-trained model, C represents the minimum bounding rectangle area of ​​the two bounding boxes A and B, and IOU is the overlap between A and B.

5. The method for training an unsupervised object detection model based on metric learning according to claim 4, characterized in that, The specific loss in image-text comparison is as follows: In the formula, N is the total number of image-text pairs used for training. The training samples include the overall image and the text labels of the overall image. v_m is the overall multi-layer feature map image representation obtained after the image of the m-th image-text pair sample is extracted by the target detection model to be trained. l_m is the text representation obtained after the text label of the m-th image-text pair sample is processed by the text encoder. The text encoder uses Bert or Roberta. The text encoder participates in training but is not updated. The calculation formula of p(v_m,l_m) is as follows: In the formula, S is the similarity between the two objects in parentheses, τ is the temperature hyperparameter, and Nri is the set of text representations obtained by the text encoder after the text labels of the other image-text pairs in the same batch of training samples, except for the m-th image-text pair sample. That is, the text label k does not match the image i, but matches other images k in the same training sample set.

Images