Model training method, target detection method, device, equipment and program product
By constructing a semi-supervised data set and using teacher models to generate target pseudo-labels, the problem of high data annotation cost in AI algorithm development is solved, and the effect of reducing costs and improving model accuracy is achieved.
Patent Information
- Application Number
- CN202411490078.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, the cost of developing AI algorithms is high, mainly because the data labeling process is time-consuming and costly.
By constructing a semi-supervised dataset, the teacher model is used to predict unlabeled images, generate target pseudo-labels, and update the supervised dataset for training the student model, reducing the time and cost of data annotation.
It reduces the cost of algorithm development, improves the accuracy of the target detection model, and reduces the time and capital cost of data annotation.
Smart Images
Figure CN119445283A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a model training method, target detection method, device, equipment and program product. Background Art
[0002] The rapid development of artificial intelligence (AI) relies on three core elements: data, algorithms, and computing power. This view has been highly recognized by the industry. The accuracy of the algorithm is highly dependent on the quality of the data set. The construction of a high-quality data set is inseparable from data labeling, that is, relying on manual search for the ground truth, and its cost and demand are rising steadily. Data labeling not only consumes money, but is also the most time-consuming part of training models. From data collection to final identification, it is likely to take a month, which seriously affects the development progress. Network intelligence with the concept of "AI+network" is rapidly advancing. The development of AI algorithms in related technologies is mostly supervised learning, which requires a lot of labeling work. The construction of industry data sets requires a lot of time and manpower costs, resulting in high algorithm development costs. Summary of the invention
[0003] The embodiments of the present application provide a model training method, a target detection method, an apparatus, a device and a program product to solve the problem of high cost of algorithm development in related technologies.
[0004] To solve the above technical problems, this application is implemented as follows:
[0005] In a first aspect, an embodiment of the present application provides a model training method, comprising:
[0006] Constructing a semi-supervised dataset, wherein the semi-supervised dataset includes a supervised dataset and an unsupervised dataset, wherein the supervised dataset includes a plurality of annotated images and labels corresponding to the annotated images, and the unsupervised dataset includes a plurality of unannotated images;
[0007] Based on the supervision data set and the teacher model, the annotation quality of the prompt word set is evaluated to obtain a target prompt word, wherein the target prompt word is the prompt word with the highest annotation quality in the prompt word set;
[0008] Predicting the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label, where the target pseudo label is a prediction result of the teacher model;
[0009] The supervised dataset is updated based on the target pseudo-label and the corresponding unlabeled image, and the student model is trained based on the updated supervised dataset to obtain a target detection model.
[0010] Optionally, updating the supervised dataset based on the target pseudo-label and the corresponding unlabeled image, and training the student model based on the updated supervised dataset to obtain the target detection model includes:
[0011] The target pseudo-label and the corresponding unlabeled image are added to the supervised data set, and the step of predicting the unlabeled image based on the target prompt word and the teacher model to obtain the target pseudo-label is returned until the number of images in the supervised data set is greater than a preset value;
[0012] The student model is trained based on the updated supervised dataset to obtain the target detection model.
[0013] Optionally, before training the student model based on the updated supervision data set to obtain the target detection model, the method further includes:
[0014] A label smoothing algorithm is used to optimize the target pseudo labels in the updated supervised dataset.
[0015] Optionally, constructing a semi-supervised dataset includes:
[0016] Acquire N images, each of the N images including a detection object;
[0017] Randomly select S images from the N images for data annotation to obtain a supervised data set;
[0018] A semi-supervised dataset is constructed based on a supervised dataset and an unsupervised dataset, wherein the unsupervised dataset includes images in the N images except the S images, wherein N and S are both integers greater than 0.
[0019] Optionally, the step of performing a labeling quality assessment on the prompt word set based on the supervision data set and the teacher model to obtain a target prompt word includes:
[0020] Constructing a plurality of prompt words corresponding to the detection object to obtain a prompt word set, wherein the prompt words are used to describe the detection object;
[0021] Inputting the prompt word set and the annotated images of the supervision data set into the teacher model to obtain the prediction results of each prompt word for each annotated image;
[0022] Calculate the loss value between the prediction result corresponding to each prompt word and the label of the supervised data set to obtain multiple loss values;
[0023] The target prompt word is determined according to the multiple loss values, and the loss value corresponding to the target prompt word is the smallest loss value among the multiple loss values.
[0024] Optionally, the prediction result includes at least one of a classification category, a target positioning box and a confidence level, and the loss value includes at least one of a target positioning box loss value and a classification loss value.
[0025] Optionally, predicting the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label includes:
[0026] Performing image denoising on the unlabeled image to obtain K-1 noisy images, where K is an integer greater than 1;
[0027] Predict the unlabeled image and the K-1 noisy images based on the target prompt word and the teacher model to obtain K pseudo labels;
[0028] Calculating a consistency loss value based on the K pseudo labels, where the consistency loss value is used to indicate the consistency of the labels of the noisy image and the unlabeled image;
[0029] When the consistency loss value is less than or equal to a preset threshold, the pseudo label corresponding to the unlabeled image is determined to be a target pseudo label.
[0030] Optionally, the student model includes an attention mechanism module, and the attention mechanism module is used to perform weighted processing on image features.
[0031] In a second aspect, an embodiment of the present application provides a target detection method, including:
[0032] The input image is detected based on the target detection model to obtain a detection result, wherein the target detection model is a model generated based on the model training method described in the first aspect above.
[0033] In a third aspect, an embodiment of the present application provides a model training device, comprising:
[0034] A construction module is used to construct a semi-supervised dataset, wherein the semi-supervised dataset includes a supervised dataset and an unsupervised dataset, wherein the supervised dataset includes a plurality of annotated images and labels corresponding to the annotated images, and the unsupervised dataset includes a plurality of unannotated images;
[0035] An evaluation module, used to evaluate the annotation quality of the prompt word set based on the supervision data set and the teacher model to obtain a target prompt word, wherein the target prompt word is the prompt word with the highest annotation quality in the prompt word set;
[0036] A prediction module, configured to predict the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label, wherein the target pseudo label is a prediction result of the teacher model;
[0037] The training module is used to update the supervised data set based on the target pseudo-label and the corresponding unlabeled image, and train the student model based on the updated supervised data set to obtain the target detection model.
[0038] Optionally, the training module includes:
[0039] A prediction submodule, configured to add the target pseudo-label and the corresponding unlabeled image to the supervision data set, and return to execute the step of predicting the unlabeled image based on the target prompt word and the teacher model to obtain the target pseudo-label, until the number of images in the supervision data set is greater than a preset value;
[0040] The training submodule is used to train the student model based on the updated supervision dataset to obtain the target detection model.
[0041] Optionally, the training module further includes:
[0042] The optimization submodule is used to optimize the target pseudo labels in the updated supervision data set by using a label smoothing algorithm.
[0043] Optionally, the building blocks include:
[0044] An acquisition submodule, used for acquiring N images, each of the N images including a detection object;
[0045] A labeling submodule is used to randomly select S images from the N images for data labeling to obtain a supervised data set;
[0046] The first construction submodule is used to construct a semi-supervised dataset based on a supervised dataset and an unsupervised dataset, wherein the unsupervised dataset includes images in the N images except the S images, wherein N and S are both integers greater than 0.
[0047] Optionally, the evaluation module includes:
[0048] The second construction submodule is used to construct a plurality of prompt words corresponding to the detection object to obtain a prompt word set, wherein the prompt words are used to describe the detection object;
[0049] An input submodule, used for inputting the prompt word set and the annotated images of the supervision data set into the teacher model to obtain the prediction results of each prompt word for each annotated image;
[0050] A calculation submodule, used to calculate the loss value between the prediction result corresponding to each prompt word and the label of the supervision data set to obtain multiple loss values;
[0051] A determination submodule is used to determine the target prompt word according to the multiple loss values, and the loss value corresponding to the target prompt word is the minimum loss value among the multiple loss values.
[0052] Optionally, the prediction result includes at least one of a classification category, a target positioning box and a confidence level, and the loss value includes at least one of a target positioning box loss value and a classification loss value.
[0053] Optionally, the prediction submodule includes:
[0054] A processing unit, configured to perform image noise processing on the unlabeled image to obtain K-1 noisy images, where K is an integer greater than 1;
[0055] A prediction unit, configured to predict the unlabeled image and the K-1 noisy images based on the target prompt word and the teacher model to obtain K pseudo labels;
[0056] A calculation unit, configured to calculate a consistency loss value based on the K pseudo labels, wherein the consistency loss value is used to indicate the consistency of the labels of the noisy image and the unlabeled image;
[0057] A determining unit is used to determine that the pseudo label corresponding to the unlabeled image is a target pseudo label when the consistency loss value is less than or equal to a preset threshold.
[0058] Optionally, the student model includes an attention mechanism module, and the attention mechanism module is used to perform weighted processing on image features.
[0059] In a fourth aspect, an embodiment of the present application provides a target detection device, including:
[0060] A detection module is used to detect an input image based on a target detection model to obtain a detection result, wherein the target detection model is a model generated based on the model training method described in the first aspect above.
[0061] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the model training method described in the first aspect are implemented; or, when the program is executed by the processor, the steps of the target detection method described in the second aspect are implemented.
[0062] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the model training method as described in the first aspect above are implemented; or, when the computer program is executed by a processor, the steps of the target detection method as described in the second aspect above are implemented.
[0063] In the seventh aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the model training method as described in the first aspect above; or, when executed by a processor, implement the steps of the target detection method as described in the second aspect above.
[0064] In the embodiment of the present application, the above-mentioned model training method can save the time and financial costs of data annotation through semi-supervised learning, and by evaluating the annotation quality of the prompt word set, it can improve the generation quality of the target pseudo-label, and then improve the accuracy of the target detection model, so that while ensuring the quality of the target detection model, the cost of algorithm development can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0066] Figure 1 This is one of the flow charts of a model training method provided in an embodiment of the present application;
[0067] Figure 2 is a prediction flow chart of a teacher model provided in an embodiment of the present application;
[0068] Figure 3 is a network structure diagram of a student model provided in an embodiment of the present application;
[0069] Figure 4 This is the second flowchart of a model training method provided in an embodiment of the present application;
[0070] Figure 5 is a flow chart of a target detection method provided in an embodiment of the present application;
[0071] Figure 6 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;
[0072] Figure 7It is a structural schematic diagram of a target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0074] For ease of understanding, some contents involved in the embodiments of the present application are described below:
[0075] Most of the existing AI algorithm development is supervised learning, which requires a lot of labeling work. Labeling work consumes a lot of manpower, material resources, and time costs, and is an important factor limiting the rapid development and application of AI. Therefore, many companies and research institutions have begun to use semi-supervised learning methods, hoping to get rid of human dependence and achieve automation.
[0076] The related technology provides a weakly supervised learning driven large and small model co-evolution method and terminal, which introduces the existing large model to perform preliminary detection on the current task to obtain the initial detection result; obtain the category of the initial detection result confirmed by a judgment, which is a high-quality result or a low-quality result. The high-quality result is the result in the correct answer set, and the low-quality result is the result outside the correct answer set; determine whether the category is a high-quality result. If so, build a supervision model based on the high-quality result, and replace the judgment to perform subsequent automatic detection of the same type of tasks, otherwise enter step S4; obtain and construct a supervision model based on the secondary annotation of the low-quality result, and replace the judgment to perform subsequent automatic detection of the same type of tasks. The whole process of this scheme only requires the introduction of two manual interventions, and the subsequent task detections are all automated, without the need to introduce manual intervention again, thereby effectively reducing manual participation while ensuring the detection quality. However, this scheme ignores the limitations of the large model prompt word library to the industry field, and the general prompt words cannot assist the large model in identifying communication equipment.
[0077] The related technology also provides a weakly supervised 3D scene graph generation method, device, equipment and medium, which obtains training data including 3D point cloud, visual image and triple label set text; aligns the 3D point cloud with the video frame corresponding to the object; performs feature extraction respectively to obtain point cloud features, image features and text features; assigns node pseudo labels to objects according to the first similarity matrix between image features and text features; determines relationship pseudo labels according to the second similarity matrix between point cloud features and text features; performs feature transfer between node features and edge feature nodes through graph neural network, and calculates predicted labels; constructs cross entropy loss function for classification training; completes the generation of objects and relationship categories in 3D scenes according to the trained object classifier, and obtains 3D scene graphs, which can reduce the annotation work of the full supervision process and promote the application of 3D scene graphs. This solution integrates three modalities: point cloud, image and text. However, the cost of point cloud image acquisition is large, the scale of communication equipment is large, and it cannot cover the whole amount. In addition, there is no screening of detection results, which has certain hidden dangers.
[0078] The related technology also provides a semi-supervised method for scene recognition of remote sensing images of nature reserves, including S1 constructing a data set D, including a supervised data set and an unsupervised data set; selecting Resnet 50 to extract features of the input remote sensing image and perform classification prediction, and output its predicted category; designing supervised data loss, unsupervised data loss and data affinity loss; constructing a comprehensive loss function L based on supervised data loss, unsupervised data loss and data affinity loss, training Resnet 50 to obtain a scene recognition model; obtaining the remote sensing image to be identified, inputting it into the scene recognition model, and outputting its predicted category. This scheme assigns pseudo labels and learning weights to unsupervised data in neural network learning by constraining clustering and affinity relationships between data, and realizes semi-supervised scene recognition of remote sensing images, so that the scene recognition model does not need to rely on large-scale labeled data during training, saving annotation time. However, this scheme adopts a clustering algorithm to assign pseudo labels, which is not applicable to complex scenes in the communication field, and does not screen the detection results, which has certain hidden dangers.
[0079] In the embodiments of the present application, a model training method, a target detection method, an apparatus, a device and a program product are proposed to solve the problem of high cost of algorithm development in related technologies.
[0080] See also Figure 1 , Figure 1 This is one of the flow charts of a model training method provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0081] Step 101: construct a semi-supervised dataset, wherein the semi-supervised dataset includes a supervised dataset and an unsupervised dataset, wherein the supervised dataset includes a plurality of annotated images and labels corresponding to the annotated images, and the unsupervised dataset includes a plurality of unlabeled images.
[0082] Specifically, the above-mentioned supervised data set can be a training data set for indicating the labeled classification target value, that is, it includes the labeled image and the corresponding label that have completed the data labeling. The above-mentioned labeled image can be the target object in the image that has been labeled. For example, assuming that the detection object is a spectrometer, the position of the spectrometer in the labeled image is located and labeled, and the corresponding label is the spectrometer. It can be understood that the above-mentioned unlabeled image is the image of the unlabeled detection object.
[0083] Step 102: Based on the supervision data set and the teacher model, the annotation quality of the prompt word set is evaluated to obtain a target prompt word, where the target prompt word is the prompt word with the highest annotation quality in the prompt word set.
[0084] Specifically, the teacher model can be an open-vocabulary object detection model, which introduces a "prompt first, then detect" strategy, uses pre-calculated prompts, including titles or categories, and uses a contrastive language-image pre-training (CLIP) pre-training model as a text encoder. After extracting the corresponding text embeddings, it is matched with the corresponding image to achieve target detection, thereby simplifying the detection process.
[0085] The above-mentioned prompt word set can be a customized prompt word set obtained by describing the detection object, and can include prompt words corresponding to various feature descriptions of the detection object. For example, assuming that the detection object is a splitter, the splitter is a device in the communication field, which is a field not covered by commonly used open source data sets. The prompt word set can include fiber optic communication terms, appearance descriptions, etc.
[0086] It is understandable that the above-mentioned annotation quality assessment can be to perform annotation quality assessment on each prompt word in the prompt word set, obtain the annotation quality corresponding to each prompt word, and select the prompt word with the highest annotation quality as the target prompt word. The above-mentioned annotation quality assessment can be to evaluate the annotation quality of the prompt word by comparing the difference between the prediction result of the annotated image and the label of the annotated image.
[0087] Step 103: predict the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label, where the target pseudo label is a prediction result of the teacher model.
[0088] Step 104: update the supervised dataset based on the target pseudo-label and the corresponding unlabeled image, and train the student model based on the updated supervised dataset to obtain a target detection model.
[0089] Specifically, the updating of the supervised dataset is to add the target pseudo-label and the corresponding unlabeled image to the supervised dataset to obtain an updated supervised dataset. The structure and parameters of the student model can refer to the teacher model, and the architecture of the teacher model can be directly simplified, or an existing small architecture can be selected.
[0090] In the embodiment of the present application, the above-mentioned model training method can save the time and financial costs of data annotation through semi-supervised learning, and by evaluating the annotation quality of the prompt word set, it can improve the generation quality of the target pseudo-label, and then improve the accuracy of the target detection model, so that while ensuring the quality of the target detection model, the cost of algorithm development can be reduced.
[0091] Optionally, updating the supervised dataset based on the target pseudo-label and the corresponding unlabeled image, and training the student model based on the updated supervised dataset to obtain the target detection model includes:
[0092] The target pseudo-label and the corresponding unlabeled image are added to the supervised data set, and the step of predicting the unlabeled image based on the target prompt word and the teacher model to obtain the target pseudo-label is returned until the number of images in the supervised data set is greater than a preset value;
[0093] The student model is trained based on the updated supervised dataset to obtain the target detection model.
[0094] Specifically, the preset value may be an expected value of a supervised learning data set used to indicate the student model. The training of the student model based on the updated supervised data set may be to divide the updated supervised data set into a training set and a validation set to train the student model. For example, the training set is used to train the model, and the model weights are continuously adjusted through multiple iterations. After each training cycle or several cycles, the validation set is used to evaluate the model performance.
[0095] In this embodiment, the above-mentioned model training method can use the target pseudo-label and the corresponding unlabeled image to update the supervised data set until the number of images in the supervised data set meets the preset value, and train the student model based on the updated supervised data set to obtain the target detection model, thereby reducing the data labeling cost of the supervised data set, thereby further reducing the cost of supervised learning.
[0096] Optionally, before training the student model based on the updated supervision data set to obtain the target detection model, the method further includes:
[0097] A label smoothing algorithm is used to optimize the target pseudo labels in the updated supervised dataset.
[0098] Specifically, the formula corresponding to the above label smoothing algorithm is as follows:
[0099] f(x′)=(1-∈)*f(x)+∈ / C
[0100] ∈=min(PML j )
[0101] Among them, f(x) is the target pseudo label, ∈ is the label smoothing factor, and C is the number of categories.
[0102] In this embodiment, the above-mentioned model training method can utilize a label smoothing algorithm to optimize the target pseudo labels in the updated supervision data set, thereby reducing the interference of some erroneous labels in the supervision data set on the student model learning, thereby further improving the accuracy of the target detection model.
[0103] Optionally, constructing a semi-supervised dataset includes:
[0104] Acquire N images, each of the N images including a detection object;
[0105] Randomly select S images from the N images for data annotation to obtain a supervised data set;
[0106] A semi-supervised dataset is constructed based on a supervised dataset and an unsupervised dataset, wherein the unsupervised dataset includes images in the N images except the S images, wherein N and S are both integers greater than 0.
[0107] Specifically, the above-mentioned detection object is used to indicate the detection target corresponding to the target detection model, and the above-mentioned acquisition of N images can be obtained by various acquisition means for the detection target, including but not limited to shooting, crawling, adversarial generation and the like. Exemplarily, assuming that the detection object is a spectrometer, N images can be collected as a semi-supervised data set D, N ≥ E (E is the expected value of the student model supervised learning data set). The goal of the algorithm is image recognition of the spectrometer. The spectrometer photos are collected by front-line employees taking photos of the spectrometer on site. The data set covers different scenes, different regions, different time periods, and different types of spectrometers such as corridors, roofs, utility poles, and weak current wells.
[0108] The above-mentioned randomly selecting S images from the N images for data annotation to obtain the supervised data set can specifically be randomly selecting S images as the supervised data set D s , D s All images in the article are labeled with GroudTruth, and examples are labeled with Contains the target category label Category Lable{c i}, target positioning box Bounding Box {b i The remaining U unlabeled images constitute the unsupervised dataset D u , where S< <U<N。
[0109] The data annotation of the S images can be completed manually. For example, data annotation can include two methods: one is to manually annotate the location of the target in each image through an annotation tool, and the other is to manually review and correct it after annotating with an automatic annotation tool. It should be noted that the annotation requires that the positioning box must tightly surround each target, there should be no extra gaps between the targets, and no target should be missing a label. The same target under multiple angles and multiple scenes needs to be annotated. Whether all instances in the image are annotated is positively correlated with the performance of the student algorithm.
[0110] In this embodiment, the above-mentioned model training method can obtain N images, each of which includes a detection object, randomly select S images from the N images for data annotation, and obtain a supervised data set, and then construct a semi-supervised data set based on the supervised data set and the unsupervised data set, so that the model can be trained by combining the labeled data and the unlabeled data, thereby reducing the cost of data annotation.
[0111] Optionally, the step of performing a labeling quality assessment on the prompt word set based on the supervision data set and the teacher model to obtain a target prompt word includes:
[0112] Constructing a plurality of prompt words corresponding to the detection object to obtain a prompt word set, wherein the prompt words are used to describe the detection object;
[0113] Inputting the prompt word set and the annotated images of the supervision data set into the teacher model to obtain the prediction results of each prompt word for each annotated image;
[0114] Calculate the loss value between the prediction result corresponding to each prompt word and the label of the supervised data set to obtain multiple loss values;
[0115] The target prompt word is determined according to the multiple loss values, and the loss value corresponding to the target prompt word is the smallest loss value among the multiple loss values.
[0116] Specifically, the plurality of prompt words corresponding to the detection object can be manually set as the task label of the detection object, and the corresponding description text is constructed. These description texts describe the image content represented by the task label, forming a prompt word set. For example, assuming that the detection object is a spectrometer, the corresponding description text may be "OpticalDivider" or "Fiber Optical Splitter". It is understandable that the design of the prompt word set helps the model to more accurately understand the features in the image and match them, guides the model to focus on the spectrometer, and improves the relevance and accuracy of the prediction results.
[0117] The above-mentioned prompt word set and the annotated images of the supervision data set are input into the teacher model to obtain the prediction results of each prompt word for each annotated image. Specifically, the prompt word set T and the supervision data set D are input into the teacher model to obtain the prediction results of each prompt word for each annotated image. s The image is input to the teacher model and the prediction result is output, prompting word t j The prediction result for image i is P ji , which can include classification category, confidence and target positioning box.
[0118] The above loss value can be used to indicate the difference between the prediction result and the label of the supervised dataset. Specifically, the prompt word evaluation model can be used to evaluate the label quality of each prompt word in the prompt word set T, and the prediction result of the teacher model and the supervised dataset D can be calculated. s The prompt mean loss (PML) is used as the evaluation standard, which mainly includes two aspects: target positioning box loss (bbox_loss) and classification loss (cls_loss). The target prompt word is t = argmin (PML j ).
[0119] In this embodiment, the above-mentioned model training method can obtain the prediction results of each prompt word for each annotated image by inputting the prompt word set and the annotated images of the supervisory data set into the teacher model, and calculate the loss value between the prediction result corresponding to each prompt word and the label of the supervisory data set to obtain multiple loss values, and then determine the prompt word with the highest annotation quality, so as to improve the relevance and accuracy of the prediction results.
[0120] Optionally, the prediction result includes at least one of a classification category, a target positioning box and a confidence level, and the loss value includes at least one of a target positioning box loss value and a classification loss value.
[0121] Specifically, the above classification category can be used to indicate the result predicted by the teacher model, and the above target positioning box can be used to indicate the position of the target accurately positioned in the image. The positioning box is usually represented by four values, such as the coordinates of the upper left corner and the lower right corner or the coordinates of the center point plus the width and height. The above confidence can be a quantitative score of the degree of credibility of the teacher model for the prediction result, usually between 0 and 1. The above target positioning box loss value can be used to measure the position difference between the bounding box predicted by the model and the real bounding box, and the above classification loss value can be used to evaluate whether the model predicts the target category accurately, that is, to measure the difference between the classification label predicted by the model and the actual label.
[0122] Exemplarily, the expression for calculating the loss function corresponding to the loss value is as follows:
[0123]
[0124]
[0125] in, is the classification loss value, is the target positioning box loss value, S is the number of images in the supervised data set, P is the predicted category probability, y is the actual category, i.e., Ground truth, IOU is the intersection-over-union ratio of the predicted box and the real box, ρ is the distance between the predicted box and the center of the real box, c is the diagonal length of the minimum enclosing rectangular box of the predicted box and the real box, α is the balance coefficient, and v is the distance of the aspect ratio of the predicted box and the real box.
[0126] In this implementation, the above model training method can comprehensively consider the target positioning box loss value and the classification loss value, thereby better evaluating the quality of the prompt word.
[0127] Optionally, predicting the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label includes:
[0128] Performing image denoising on the unlabeled image to obtain K-1 noisy images, where K is an integer greater than 1;
[0129] Predict the unlabeled image and the K-1 noisy images based on the target prompt word and the teacher model to obtain K pseudo labels;
[0130] Calculating a consistency loss value based on the K pseudo labels, where the consistency loss value is used to indicate the consistency of the labels of the noisy image and the unlabeled image;
[0131] When the consistency loss value is less than or equal to a preset threshold, the pseudo label corresponding to the unlabeled image is determined to be a target pseudo label.
[0132] Specifically, the teacher model can be an open vocabulary-oriented target detection model, and the prompt word can be used to assist the detector in identifying the unsupervised dataset D u , and output pseudo labels. Figure 2 is a prediction flow chart of a teacher model provided in an embodiment of the present application, such as Figure 2 As shown in the figure, the above teacher model is mainly divided into three modules: text encoder (TextEncoder); image encoder (Image Encoder); vision-language encoder (Vision-Language Encoder).
[0133] First, input the prompt word t and use the n-gram algorithm to get the noun phrase, and then use the pre-trained Text Encoder to get the text embedding set of the detection class W = TextEncoder(t)∈R C×D , where C is the number of phrases and D is the dimension of embedding; then the input image is passed through the Image Encoder to obtain the multi-scale image feature X l ∈R H×W×D (l∈{3,4,5}), where H is the feature map height, W is the feature map width, and D is the feature map dimension; Vision-LanguageEncoder performs multi-modal fusion output of multi-scale image features and text features; finally, the decoupling head outputs the predicted bounding box and the object embedding matched with the region-text.
[0134] It can be understood that pseudo-labeling is an effective semi-supervised learning method for deep neural networks. It is a prediction of unsupervised data. The class with the largest prediction probability is selected as GroundTruth, so it is called pseudo-labeling. In this application, the prediction result of the teacher model is selected as the pseudo-label.
[0135] Specifically, the process of obtaining the target pseudo-label can be based on the consistency regularization model, evaluating the quality of the pseudo-labels generated by the teacher model, screening the correct pseudo-labels, and expanding the supervised dataset D. s It is understandable that due to the extreme lack of spectrometer information in the teacher model, the quality of its pseudo-labels cannot be guaranteed, and further evaluation and screening of pseudo-label samples is required. The central idea of the evaluation model is that the teacher model should keep the same prediction results for each image before and after random noise is added to the image.
[0136] For example, the model is evaluated on an unsupervised dataset D u For each image in , the mean consistency loss (Mean Consistency Loss, MCL) between the pseudo label of the noisy image and the benchmark value (i.e., the pseudo label of the non-noisy image) can be calculated as the evaluation criterion according to the following formula, where K represents the number of pseudo label cycle generation times. represents random noise. If MCL is lower than the predetermined threshold τ, the data (x u ,f(x u )) Expand to labeled dataset Right now Traverse D u , update the supervised dataset Until S ≥ E (E is the expected value of the data set of the student model):
[0137]
[0138] Among them, f(x u ) is the pseudo label corresponding to the unnoised image, is the pseudo label corresponding to the noisy image, IOU is the intersection-over-union ratio of the predicted box and the real box, ρ is the distance between the predicted box and the center of the real box, c is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box, α is the balance coefficient, and v is the distance of the aspect ratio of the predicted box and the real box.
[0139] In this embodiment, the above-mentioned model training method can obtain K-1 noisy images by performing image denoising on the unlabeled image, predict the unlabeled image and the K-1 noisy images based on the target prompt word and the teacher model to obtain K pseudo-labels, calculate the consistency loss value based on the K pseudo-labels, and when the consistency loss value is less than or equal to a preset threshold, determine that the pseudo-label corresponding to the unlabeled image is the target pseudo-label, and then screen out more reliable and informative pseudo-label samples, thereby further improving the accuracy of the target detection model.
[0140] Optionally, the student model includes an attention mechanism module, and the attention mechanism module is used to perform weighted processing on image features.
[0141] For example, Figure 3 is a network structure diagram of a student model provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the student model refers to the image encoder of the teacher model and adopts a one-stage target detection algorithm, including three parts: backbone, neck, and head. There are a large number of partially occluded and complex background targets in the scene image. The feature pyramid (FPN) and path aggregation network (PANet) of the neck part make it difficult for the dense port and pigtail edge features to be effectively expressed in the high-level feature map. In this implementation, the C2f module at the connection between the neck and Detect can be combined with the convolutional block attention module (CBAM) to enhance the detection ability of partially occluded and blurred boundary targets.
[0142] In this embodiment, the above-mentioned model training method can enhance the target detection model's ability to detect targets with partial occlusion and blurred boundaries by adding an attention mechanism module to the student model, thereby improving the robustness of the target detection model.
[0143] For example, Figure 4 This is a second flow chart of a model training method provided in an embodiment of the present application. Figure 4 As shown, the specific steps include:
[0144] The first step is to construct a semi-supervised dataset D, including the supervised dataset D s and the unsupervised dataset D u ;
[0145] The second step is prompt fine-tuning, based on the characteristics of the detected object, through the supervised dataset D s The model continuously optimizes prompt and determines the prompt word t with the highest annotation quality;
[0146] The third step is pseudo label generation. The prompt word t in the second step is combined with the unsupervised dataset D u , input into the teacher model and obtain the prediction result P;
[0147] The fourth step is to evaluate the quality of pseudo labels. Based on the consistency regularization model, the quality of pseudo labels generated by the teacher model is evaluated, the correct pseudo labels are screened, and the supervised dataset D is expanded.s ;
[0148] The fifth step is to train the student model and convert D s The dataset is divided into a training set and a validation set. The label smoothing strategy is used to perform fully supervised learning of the student model to obtain a more robust target detection model.
[0149] See also Figure 5 , Figure 5 is a flow chart of a target detection method provided in an embodiment of the present application, such as Figure 5 As shown, the method comprises the following steps:
[0150] Step 501: Detect the input image based on the target detection model to obtain a detection result, wherein the target detection model is a model generated based on the above-mentioned model training method.
[0151] Specifically, the input image may be an image of an object to be detected.
[0152] In an embodiment of the present application, since the target detection method includes detecting an input image based on a target detection model to obtain a detection result, wherein the target detection model is a model generated based on the above-mentioned model training method, the target detection method can achieve all the beneficial effects of the above-mentioned model training method. In order to avoid repetition, it will not be described here.
[0153] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a model training device provided in an embodiment of the present application, such as Figure 6 As shown, the model training device 600 includes:
[0154] A construction module 601 is used to construct a semi-supervised dataset, wherein the semi-supervised dataset includes a supervised dataset and an unsupervised dataset, wherein the supervised dataset includes a plurality of annotated images and labels corresponding to the annotated images, and the unsupervised dataset includes a plurality of unannotated images;
[0155] An evaluation module 602 is used to evaluate the annotation quality of the prompt word set based on the supervision data set and the teacher model to obtain a target prompt word, where the target prompt word is the prompt word with the highest annotation quality in the prompt word set;
[0156] A prediction module 603 is used to predict the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label, where the target pseudo label is a prediction result of the teacher model;
[0157] The training module 604 is used to update the supervision data set based on the target pseudo-label and the corresponding unlabeled image, and train the student model based on the updated supervision data set to obtain a target detection model.
[0158] Optionally, the training module includes:
[0159] A prediction submodule, configured to add the target pseudo-label and the corresponding unlabeled image to the supervision data set, and return to execute the step of predicting the unlabeled image based on the target prompt word and the teacher model to obtain the target pseudo-label, until the number of images in the supervision data set is greater than a preset value;
[0160] The training submodule is used to train the student model based on the updated supervision dataset to obtain the target detection model.
[0161] Optionally, the training module further includes:
[0162] The optimization submodule is used to optimize the target pseudo labels in the updated supervision data set by using a label smoothing algorithm.
[0163] Optionally, the building blocks include:
[0164] An acquisition submodule, used for acquiring N images, each of the N images including a detection object;
[0165] A labeling submodule is used to randomly select S images from the N images for data labeling to obtain a supervised data set;
[0166] The first construction submodule is used to construct a semi-supervised dataset based on a supervised dataset and an unsupervised dataset, wherein the unsupervised dataset includes images in the N images except the S images, wherein N and S are both integers greater than 0.
[0167] Optionally, the evaluation module includes:
[0168] The second construction submodule is used to construct a plurality of prompt words corresponding to the detection object to obtain a prompt word set, wherein the prompt words are used to describe the detection object;
[0169] An input submodule, used for inputting the prompt word set and the annotated images of the supervision data set into the teacher model to obtain the prediction results of each prompt word for each annotated image;
[0170] A calculation submodule, used to calculate the loss value between the prediction result corresponding to each prompt word and the label of the supervision data set to obtain multiple loss values;
[0171] A determination submodule is used to determine the target prompt word according to the multiple loss values, and the loss value corresponding to the target prompt word is the minimum loss value among the multiple loss values.
[0172] Optionally, the prediction result includes at least one of a classification category, a target positioning box and a confidence level, and the loss value includes at least one of a target positioning box loss value and a classification loss value.
[0173] Optionally, the prediction submodule includes:
[0174] A processing unit, configured to perform image noise processing on the unlabeled image to obtain K-1 noisy images, where K is an integer greater than 1;
[0175] A prediction unit, configured to predict the unlabeled image and the K-1 noisy images based on the target prompt word and the teacher model to obtain K pseudo labels;
[0176] A calculation unit, configured to calculate a consistency loss value based on the K pseudo labels, wherein the consistency loss value is used to indicate the consistency of the labels of the noisy image and the unlabeled image;
[0177] A determining unit is used to determine that the pseudo label corresponding to the unlabeled image is a target pseudo label when the consistency loss value is less than or equal to a preset threshold.
[0178] Optionally, the student model includes an attention mechanism module, and the attention mechanism module is used to perform weighted processing on image features.
[0179] It should be noted that the model training device provided in the embodiment of the present application is a device capable of executing the above-mentioned model training method, and all implementation methods in the above-mentioned model training method embodiment are applicable to the device and can achieve the same or similar beneficial effects. To avoid repeated description, this embodiment will not be repeated.
[0180] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a target detection device provided in an embodiment of the present application, such as Figure 7 As shown, the target detection device 700 includes:
[0181] The detection module 701 is used to detect the input image based on the target detection model to obtain the detection result, wherein the target detection model is a model generated based on the model training method described in the first aspect above.
[0182] It should be noted that the target detection device provided in the embodiment of the present application is a device capable of executing the above target detection method, and all implementation methods in the above target detection method embodiment are applicable to the device and can achieve the same or similar beneficial effects. To avoid repeated description, this embodiment will not be repeated.
[0183] An embodiment of the present application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, each process of the above-mentioned model training method or target detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0184] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned model training method or target detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium, such as read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0185] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned model training method or target detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.
[0186] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0188] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A model training method, characterized in that: include: Constructing a semi-supervised dataset, wherein the semi-supervised dataset includes a supervised dataset and an unsupervised dataset, wherein the supervised dataset includes a plurality of annotated images and labels corresponding to the annotated images, and the unsupervised dataset includes a plurality of unannotated images; Based on the supervision data set and the teacher model, the annotation quality of the prompt word set is evaluated to obtain a target prompt word, wherein the target prompt word is the prompt word with the highest annotation quality in the prompt word set; Predicting the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label, where the target pseudo label is a prediction result of the teacher model; The supervised dataset is updated based on the target pseudo-label and the corresponding unlabeled image, and the student model is trained based on the updated supervised dataset to obtain a target detection model.
2. The method according to claim 1, characterized in that: The method of updating the supervised data set based on the target pseudo-label and the corresponding unlabeled image, and training the student model based on the updated supervised data set to obtain the target detection model includes: The target pseudo-label and the corresponding unlabeled image are added to the supervised data set, and the step of predicting the unlabeled image based on the target prompt word and the teacher model to obtain the target pseudo-label is returned until the number of images in the supervised data set is greater than a preset value; The student model is trained based on the updated supervised dataset to obtain the target detection model.
3. The method according to claim 2, characterized in that Before training the student model based on the updated supervision data set to obtain the target detection model, the method further includes: A label smoothing algorithm is used to optimize the target pseudo labels in the updated supervised dataset.
4. The method according to claim 1, characterized in that: The construction of a semi-supervised dataset comprises: Acquire N images, each of the N images including a detection object; Randomly select S images from the N images for data annotation to obtain a supervised data set; A semi-supervised dataset is constructed based on a supervised dataset and an unsupervised dataset, wherein the unsupervised dataset includes images in the N images except the S images, wherein N and S are both integers greater than 0.
5. The method according to any one of claims 1 to 4, characterized in that The step of evaluating the annotation quality of the prompt word set based on the supervision data set and the teacher model to obtain the target prompt word includes: Constructing a plurality of prompt words corresponding to the detection object to obtain a prompt word set, wherein the prompt words are used to describe the detection object; Inputting the prompt word set and the annotated images of the supervision data set into the teacher model to obtain the prediction results of each prompt word for each annotated image; Calculate the loss value between the prediction result corresponding to each prompt word and the label of the supervised data set to obtain multiple loss values; The target prompt word is determined according to the multiple loss values, and the loss value corresponding to the target prompt word is the smallest loss value among the multiple loss values.
6. The method according to claim 5, characterized in that The prediction result includes at least one of a classification category, a target positioning box, and a confidence level, and the loss value includes at least one of a target positioning box loss value and a classification loss value.
7. The method according to any one of claims 1 to 4, characterized in that The step of predicting the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label includes: Performing image denoising on the unlabeled image to obtain K-1 noisy images, where K is an integer greater than 1; Predict the unlabeled image and the K-1 noisy images based on the target prompt word and the teacher model to obtain K pseudo labels; Calculating a consistency loss value based on the K pseudo labels, where the consistency loss value is used to indicate the consistency of the labels of the noisy image and the unlabeled image; When the consistency loss value is less than or equal to a preset threshold, the pseudo label corresponding to the unlabeled image is determined to be a target pseudo label.
8. The method according to any one of claims 1 to 4, characterized in that The student model includes an attention mechanism module, which is used to perform weighted processing on image features.
9. A target detection method, characterized in that: include: The input image is detected based on a target detection model to obtain a detection result, wherein the target detection model is a model generated based on the model training method described in any one of claims 1 to 8.
10. A model training device, characterized in that: include: A construction module is used to construct a semi-supervised dataset, wherein the semi-supervised dataset includes a supervised dataset and an unsupervised dataset, wherein the supervised dataset includes a plurality of annotated images and labels corresponding to the annotated images, and the unsupervised dataset includes a plurality of unannotated images; An evaluation module, used to evaluate the annotation quality of the prompt word set based on the supervision data set and the teacher model to obtain a target prompt word, wherein the target prompt word is the prompt word with the highest annotation quality in the prompt word set; A prediction module, configured to predict the unlabeled image based on the target prompt word and the teacher model to obtain a target pseudo label, wherein the target pseudo label is a prediction result of the teacher model; The training module is used to update the supervised data set based on the target pseudo-label and the corresponding unlabeled image, and train the student model based on the updated supervised data set to obtain the target detection model.
11. A target detection device, characterized in that: include: A detection module is used to detect an input image based on a target detection model to obtain a detection result, wherein the target detection model is a model generated by the model training method described in any one of claims 1 to 8.
12. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the model training method as described in any one of claims 1 to 8 are implemented; or, when the program is executed by the processor, the steps of the target detection method as described in claim 9 are implemented.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the model training method as described in any one of claims 1 to 8; or, when executed by a processor, implements the steps of the target detection method as described in claim 9.
14. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the steps of the model training method as described in any one of claims 1 to 8; or, when executed by a processor, the computer instructions implement the steps of the target detection method as described in claim 9.