Weak supervision target detection method and system based on iteration pseudo-full supervision training

Abstract

The invention relates to the technical field of target detection of deep learning, in particular to a weak supervision target detection method and system based on iterative pseudo-full supervision training. The core of the refining strategy is as follows: any two pseudo-truth value instances with the same category, higher category score and intersection-to-union ratio are subjected to weighted fusion according to respective category score instead of deleting the pseudo-truth value instances with lower category score in the existing method; compared with the pseudo-truth value instance before fusion, the pseudo-truth value instance after weighted fusion can cover the target body to a greater extent, so that the detection effect can be effectively improved. And on the other hand, the refined pseudo-truth value is updated by using a pseudo-truth value updating strategy provided by the invention, then the updated pseudo-truth value is used for training the YOLO model, and finally, the problem of target missing detection can be solved by continuously iteratively training the YOLO model, so that the recall rate of model detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection of deep learning, and particularly relates to a weakly supervised target detection method and system based on iterative pseudo-full supervision training. BACKGROUND

[0002] Target detection is one of the most important tasks in the field of image processing, and is a key technology in the field of computer vision and image processing, which is committed to identifying and accurately positioning high-value targets in images or videos. At present, it has been widely used in military reconnaissance, security monitoring, intelligent transportation and automatic driving, agricultural and resource monitoring and other fields, and has important civil and military value.

[0003] The existing target detection method is mainly full supervision target detection. However, the training of full supervision target detection model depends on a large number of instance-level hand-labeled samples, which requires high time and labor cost. Weakly supervised learning refers to the case where the label information is incomplete during the training process. For example, in target detection, if only image-level labels (for example, whether the image contains a cat) are available, without specific bounding boxes to indicate the location of the target, it can be considered as weakly supervised learning. That is, the weakly supervised target detection method only needs image-level labeled samples, which significantly reduces the time and labor cost of labeling, so it has received widespread attention in recent years.

[0004] Existing weakly supervised object detection models can be divided into two categories: models based on traditional multiple instance learning and pseudo fully supervised object detection models that have emerged in recent years. Among them, a landmark work in models based on traditional multiple instance learning is OICR [P. Tang, X. Wang, X. Bai, and W. Liu, “Multiple instance detection network with online instance classifier refinement,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 2843-2851.], which first generates target candidate boxes using the selective search algorithm [J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders, “Selective Search for Object Recognition,” Int. J. Comput. Vis., vol. 104, pp. 154-171, 2013.], then generates pseudo labels for each target candidate box using multiple instance learning for training the instance classifier refinement branch, and finally uses the instance classifier refinement branch to infer the class of the target candidate box. Pseudo fully supervised object detection models use the results generated by weakly supervised object detection models as pseudo ground truth to guide the training of fully supervised object detection models. Due to the advantages of fully supervised object detection models in detection accuracy, inference speed, and technical iteration speed, pseudo fully supervised object detection models show more excellent performance and thus receive more attention.

[0005] Despite the significant progress made by pseudo fully supervised object detection methods, there are still two problems to be solved:

[0006] 1) Tend to focus on the most prominent part of the target rather than the whole target. The pseudo ground truth used to train the fully supervised object detection model is usually generated using a method similar to SOS-WSOD [L. Sui, C.-L. Zhang, and J. Wu, "Salvage of supervision in weakly supervised object detection and segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10394-10408, 2023.]: first, remove the instances whose class scores output by the weakly supervised object detection model are lower than a certain threshold; then, apply a non-maximum suppression operation to the remaining instances to remove redundant instances. The above scheme ensures that the generated pseudo ground truth instances have high class scores, however, the detection results with high class scores given by existing weakly supervised object detection models usually focus on locating the most prominent part of the target, and other less prominent parts are often ignored, so the fully supervised object detection model trained by the above pseudo ground truth instances also tends to focus on the most prominent part of the target.

[0007] 2) Extremely prone to target missing problem. As mentioned above, the current pseudo ground truth instances are generated by traditional weakly supervised object detection models, and the weakly supervised object detection model driven by image-level labels usually only focuses on the most prominent target in each class, because the image-level label of each class can be determined by a single target in the class, in this case, if the image contains multiple targets of the same class, the pseudo ground truth instances generated by the existing method are difficult to cover all targets, and the fully supervised object detection model trained by such pseudo ground truth instances will be extremely prone to target missing problem. SUMMARY

[0008] The purpose of the application is to solve the technical problems in the background art, the application provides a weakly supervised object detection method based on iterative pseudo fully supervised training, and the application also discloses a weakly supervised object detection system based on iterative pseudo fully supervised training.

[0009] Technical scheme: In a first aspect, the application provides a weakly supervised object detection method based on iterative pseudo fully supervised training, the method comprising the following steps:

[0010] S1 Obtain a certain number of sample images to form sample data, and divide the sample data into a training set and a test set according to a certain proportion;

[0011] S2 inputs the training set into a multi-instance self-training model to obtain a corresponding detection result, taking the detection result as an initial pseudo ground truth of the sample image, wherein the initial pseudo ground truth includes a pseudo ground truth instance set P and a corresponding instance category score set S;

[0012] S3 performs filtering operations on the pseudo ground truth instance set P and the corresponding instance category score set S respectively to obtain a filtered pseudo ground truth instance set PF and a corresponding category score set SF, wherein the filtering operations are used to remove pseudo ground truth instances with a category score lower than a set threshold value;

[0013] S4 performs a category score guided weighted synthesis operation on the obtained filtered pseudo ground truth instance set PF and the corresponding category score set SF to obtain a refined pseudo ground truth instance set PR and a corresponding category score set SR, wherein the category score guided weighted synthesis operation is used to: for the filtered pseudo ground truth instance set PF, compare the intersection over union of any two instances with the same category with a set threshold value, and determine whether to directly add the related instance to the set PR or to add it to the set PR after synthesis according to the comparison result with the threshold value, and the corresponding category score is also added to the set SR;

[0014] S5 trains the YOLO model using the refined pseudo ground truth instance set PR to obtain a corresponding detection result, i.e., a detected instance set PY and a corresponding category score set SY;

[0015] S6 inputs PR, SR, PY and SY into a pseudo ground truth updating strategy to obtain an updated pseudo ground truth instance set PIR and a corresponding category score set SIR, wherein the pseudo ground truth updating strategy includes the filtering operation and an intersection over union guided weighted synthesis operation, and the set PY and the corresponding category score set SY are denoted as the set PFY and the set SFY respectively after the filtering operation, and the intersection over union guided weighted synthesis operation is used to: for any one instance in the set PFY, find the same category instance in the set PR with the maximum intersection over union, compare the maximum intersection over union with a set threshold value, and determine whether to directly add the related instance in the set PFY to the set PIR or to add it to the set PIR after synthesis with the same category instance in the set PR with the maximum intersection over union, and the corresponding category score is also added to the set SIR;

[0016] S7 re-train the YOLO model using the updated pseudo ground truth instance set PIR, obtain the corresponding detection results, empty the elements in set PY and set SY, and assign the elements in the instance set and the corresponding class score set in the detection results to set PY and set SY respectively; empty the elements in set PR and set SR, and assign the elements in set PIR and set SIR to set PR and set SR respectively;

[0017] S8 iterate the training of steps S6 and S7 for several times to obtain the trained IPS-YOLO model;

[0018] S9 perform inference prediction on the IPS-YOLO model using a test set to obtain the corresponding prediction results.

[0019] Further, comprising:

[0020] The class score guided weighted synthesis operation comprises:

[0021] For any two instances with the same class in set PF, if the intersection over union of the two is less than or equal to a set low threshold, both are included in set PR; if the intersection over union of the two is greater than the set low threshold and less than a set high threshold, the instance with lower class score is eliminated and the other is included in set PR; if the intersection over union of the two is greater than or equal to the high threshold, the two are first weighted synthesized, the synthesized instance is included in set PR, and the instances participating in the synthesis are no longer retained; the corresponding class score is also included in set SR.

[0022] Further, comprising:

[0023] The intersection over union guided weighted synthesis operation comprises:

[0024] For any one instance in set PFY, find the instance with the same class in set PR that has the maximum intersection over union with it, if the maximum intersection over union is less than or equal to a set low threshold, the instance in set PFY is included in set PIR; if the maximum intersection over union is greater than the set low threshold and less than a set high threshold, the instance in set PFY is eliminated; if the maximum intersection over union is greater than or equal to the set high threshold, the two are second weighted synthesized according to the maximum intersection over union, the synthesized result is included in set PIR; all instances in set PR that do not participate in the second weighted synthesis are included in set PIR; the corresponding class score is also included in set SIR.

[0025] Further, comprising:

[0026] The step S3 is filtering the pseudo ground truth instance set P and the corresponding instance category score set S respectively to obtain the filtered pseudo ground truth instance set PF and the corresponding category score set SF, including:

[0027] S31 represents the relevant parameters of input and output, the input is:

[0028] P={P1,...,P c ,...,P C} and S={S1,...,S c ,...,S C},

[0029] The output is:

[0030] PF={PF1,...,PF c ,...,PF C} and SF={SF1,...,SF c ,...,SF C};

[0031] Wherein, P c represents the set of the cth initial pseudo ground truth instance, S c represents the category score of P c , PF c represents the instance set after filtering operation on P c , SF c represents the category score of PF c , C is the total number of categories of initial pseudo ground truth instances, and initially

[0032] S32 calculates the corresponding instance number idm that makes S c,i maximum, wherein S c,i represents the category score of P c,i , P c,i represents the ith instance in P c , and compares S c,idm with the category score threshold T CCS , if S c,idm <T CCS , the corresponding P c,idm is added to the output set PF; otherwise, step S33 is executed.

[0033] S33 traverses all instances in the set P c , and repeats the following operation: compare the corresponding S c,i with the category score threshold T CCS , if S c,i ≥T CCS, the corresponding PF c,i is added to the output set PF, and the corresponding SF c,i is added to the output set SF.

[0034] S34 iterates steps S32 and S33 until all C classes of pseudo true instances are traversed.

[0035] Further, comprising:

[0036] The class score guided weighted synthesis operation specifically implements steps including:

[0037] S41 represents the relevant parameters of input and output, the input is:

[0038] PF = {PF1,..., PF c ,..., PF C}, SF = {SF1,..., SF c ,..., SF C}, T L , T H , wherein T L , T H represent low and high intersection-over-union thresholds respectively; PF c represents the pseudo true instance set after filtering belonging to the c-th class, SF c represents the class score of PF c , and C is the total number of classes of pseudo true instances;

[0039] The output is:

[0040] PR = {PR1,..., PR c ,..., PR C} and SR = {SR1,..., SR c ,..., SR C};

[0041] Wherein, PR c represents the instance set after PF c weighted synthesis guided by the class score, SR c represents the class score of PR c , and initially

[0042] S42 calculates the corresponding instance number idn that makes SF c,i maximum, wherein SF c,i represents the class score of PF c,i , and PF c,i represents the i-th instance in PF c .

[0043] S43 traverse set PF c All instances in , and repeat the following operation: If the intersection over union (IoU) c,idn ,PF c,i )≤T L , then PR c and PF c,i Take the union, SR c and SF c,i Take the union; otherwise, if the intersection over IoU (PF c,idn ,PF c,i )≥T H , then i is included in the index collector IB, where the index collector IB is initially an empty set; PF c,i and SF c,i From PF c and SF c Delete in;

[0044] S44 calculates the number of elements in the index collector IB, marked as M, and obtains the instance coordinate Cor after the first weighted synthesis, and obtains And add it to PR c Among them, represents the weighted composite instance with coordinates Cor belonging to the cth class; and then we get Corresponding category scores And add it to SR c middle;

[0045] S45 loops through steps S42-S44 until all pseudo-truth instances of the C categories are traversed.

[0046] Further, including:

[0047] The instance coordinate Cor after the first weighted synthesis is expressed as:

[0048]

[0049] Among them, ω j It is Cor_PF c,IB{j} The weight of Cor_PF c,IB{j} PF c,IB{j} Coordinates of PF c,IB{j} Representative PF c The first instance of IB{j} in SF c,IB{j} Representative PF c,IB{j} the corresponding category scores;

[0050] described Corresponding category scores Expressed as:

[0051]

[0052] Further, comprising:

[0053] The specific implementation steps of the weighted synthesis operation guided by the intersection-over-union include:

[0054] S61 represents the relevant parameters of input and output, the input is:

[0055] PR={PR1,...,PR c ,...,PR C},

[0056] SR={SR1,...,SR c ,...,SR C},

[0057] PFY={PFY1,...,PYF c ,...,PYF C};

[0058] SFY={SFY1,...,SYF c ,...,SYF C},

[0059] T L ,T H , wherein T L , T H represent low intersection-over-union and high intersection-over-union thresholds respectively; PFY c represents a set of instances in the set PFY that belong to the c-th class, SFY c represents the class score of the set PFY c , PFY is the instance set of the set PY after the filtering operation, SFY is the class score set corresponding to the set PFY, PR c represents the instance set of PF c after weighted synthesis guided by the class score, SR c represents the class score of PR c , PF c represents the instance set of P c after performing the filtering operation;

[0060] The output is:

[0061] PIR={PIR1,...,PIR c ,...,PIR C} and SIR={SIR1,...,SIR c ,...,SIR C}.

[0062] Among them, PIR c It represents PR c Updated set, SIR c It stands for PIR c The category score set of

[0063] S62 calculation makes PFY c,i With PR c,j The instance number idk corresponding to the maximum intersection-union ratio between c,i With PR c,idk The intersection-over-union ratio between them is recorded as MIoU; among them, PR c,j Represents PR c The j∈[1,|PR c |] instances, PFY c,i It stands for PFY c The i∈[1,|PFY c |] instances;

[0064] S63 If ​​MIoU≤T L , then the corresponding PFY c,i Join PIR c In the c,i Join SIR c Among them, SFY c,i It stands for PFY c,i Category score; otherwise: if MIoU ≥ T H , then PR c,idk and PFY c,i Perform the second weighted synthesis operation to obtain the corresponding coordinates CWS, and then combine the weighted synthesised instances with the coordinates CWS belonging to the cth class Join PIR c Calculation Corresponding category scores and will Join SIR c PR c,idk and SR c,idk From PR c and SR c Remove the SR c,idk Represents PR c,idk Category score of

[0065] S64 will PFY c,i and SFY c,i From PFY c SFY c Remove;

[0066] S65 loop steps S62-S64 until all instances of PYF are traversed c ;

[0067] The result of merging PIR c and PR c is assigned to PIR c , and the result of merging SIR c and SR c is assigned to SIR c ;

[0068] S66 loop steps S62-S65 until all instances of C categories of pseudo ground truth are traversed.

[0069] Further comprising:

[0070] In the step S63, the operation of the second weighted synthesis is performed on PR c,idk and PFY c,i to obtain the corresponding coordinate CWS, denoted as:

[0071]

[0072] wherein Cor PR c,idk represents the coordinates of PR c,idk , and Cor PFY c,i represents the coordinates of PFY c,i .

[0073] Further comprising:

[0074] The corresponding category score is denoted as:

[0075]

[0076] In another aspect, the present application also provides a weakly supervised object detection system based on iterative pseudo full supervision training, which comprises:

[0077] A data set acquisition module is configured to acquire a certain number of sample images to form sample data, and cut the sample data into a training set and a test set according to a certain proportion;

[0078] A model construction and training module:

[0079] A benchmark model training unit is configured to input the training set into a multi-instance self-training model to obtain a corresponding detection result, and use the detection result as an initial pseudo ground truth of the sample image, wherein the initial pseudo ground truth comprises a pseudo ground truth instance set P and a corresponding instance category score set S; ​

[0080] a filtering unit configured to perform filtering operations on the pseudo ground truth instance set P and the corresponding instance category score set S respectively, to obtain a filtered pseudo ground truth instance set PF and a corresponding category score set SF, the filtering operations being configured to remove instances with category scores lower than a set threshold value;

[0081] a first weighted synthesis unit configured to perform category score guided weighted synthesis operations on the obtained filtered pseudo ground truth instance set PF and the corresponding category score set SF, to obtain a refined pseudo ground truth instance set PR and a corresponding category score set SR, the category score guided weighted synthesis operations being configured to: for the filtered pseudo ground truth instance set PF, compare the intersection over union of any two instances with the same category with a set threshold value, and determine, according to the comparison result with the threshold value, whether to directly add the relevant instance to the set PR or to add the relevant instance to the set PR after synthesis, and the corresponding category score is also added to the set SR;

[0082] a YOLO model training unit configured to train the YOLO model using the refined pseudo ground truth instance PR to obtain corresponding detection results, i.e., detected instances PY and corresponding category scores SY;

[0083] an update strategy unit configured to input PR, SR, PY and SY into a pseudo ground truth update strategy to obtain an updated pseudo ground truth instance set PIR and a corresponding category score set SIR, the pseudo ground truth update strategy including the filtering operations and the intersection over union guided weighted synthesis operations, the set PY and the corresponding category score set SY are denoted as sets PFY and SFY respectively after the filtering operations, and the intersection over union guided weighted synthesis operations are configured to: for any one instance in the set PFY, find the same category instance in the set PR with the maximum intersection over union, compare the maximum intersection over union with a set threshold value, and determine, according to the comparison result, whether to directly add the relevant instance in the set PFY to the set PIR or to add the relevant instance to the set PIR after synthesis with the same category instance in the set PR with the maximum intersection over union, and the corresponding category score is also added to the set SIR;

[0084] a YOLO model secondary training unit configured to train the YOLO model again using the updated pseudo ground truth instance set PIR to obtain corresponding detection results, to empty the elements in the set PY and the set SY, and to assign the elements in the instance set and the corresponding category score set in the detection results to the set PY and the set SY respectively; to empty the elements in the set PR and the set SR, and to assign the elements in the set PIR and the set SIR to the set PR and the set SR respectively;

[0085] An iteration unit is configured to iterate training for several times to obtain the IPS-YOLO model after training.

[0086] A prediction module is configured to perform inference prediction on the IPS-YOLO model by using a test set to obtain a corresponding prediction result.

[0087] Beneficial effects: Compared with the prior art, the present application has the following advantages:

[0088] On one hand, the present application proposes a refining strategy for the initial pseudo ground truth generated by the traditional weakly supervised object detection model to solve the problem 1) proposed in the background technology, that is, to focus on the most salient part of the target rather than the entire target body. The core innovation of the refining strategy includes a class score guided weighted synthesis operation, which is based on a set threshold. If the intersection over union of any two instances with the same class in the set PF is less than or equal to the set low threshold, both of them are included in the set PR, that is, if the distance between the two detection instances is far, both of them are considered as the pseudo ground truth instance for subsequent consideration. If the intersection over union of any two instances with the same class in the set PF is greater than the set low threshold and less than the set high threshold, the instance with the lower class score is eliminated, and the other is included in the set PR, that is, the instance with the higher class score is considered as the pseudo ground truth instance for subsequent consideration. If the intersection over union of any two instances with the same class in the set PF is greater than or equal to the high threshold, the first weighted synthesis is performed on the two instances, and the synthesized instance is included in the set PR, that is, if the two instances have a high degree of overlap, the two instances are weighted and synthesized. Therefore, the present application weightedly fuses any two pseudo ground truth instances with the same class, high class score and intersection over union according to their respective class scores, rather than deleting the pseudo ground truth instance with a relatively low class score as in the prior art. The pseudo ground truth instance after weighted fusion can cover the target body to a greater extent than the pseudo ground truth instance before fusion, thereby effectively improving the detection effect, which is reflected in the improvement of the average precision mAP and the average localization precision CorLoc of object detection.

[0089] In another aspect, the present application also proposes an iterative pseudo-full supervision training scheme of a YOLO model to solve the problem 2) proposed in the background art, i.e., the problem of easy target missing detection. The core innovation of the training scheme is: using the refined pseudo-ground truth to perform full supervision training on the YOLO model, inputting the sample image into the trained YOLO model to obtain the corresponding detection result, updating the refined pseudo-ground truth by using the detection result and combining the pseudo-ground truth updating strategy proposed in the present application, and performing full supervision training on the YOLO model by using the updated pseudo-ground truth. As described in problem 2) in the background art, the existing pseudo-ground truth often only focuses on the most prominent target in each class, which is the core reason for target missing detection. However, the YOLO model trained by the existing pseudo-ground truth can learn the key features of various targets, so the YOLO model trained by the existing pseudo-ground truth has a high probability of detecting other targets in addition to the most prominent target in each class. Therefore, updating the pseudo-ground truth by using the detection result of the YOLO model can naturally make the updated pseudo-ground truth cover more targets. Finally, the YOLO model trained by the iteratively updated pseudo-ground truth can solve the problem of target missing detection, thereby improving the recall rate of target detection, and reflecting in the data as the improvement of the average precision mAP of target detection, because the average precision mAP is a comprehensive index representing the precision and recall. BRIEF DESCRIPTION OF DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0091] Figure 1 A flowchart of a weakly supervised target detection method based on iterative pseudo-full supervision training is provided for the embodiments of the present application.

[0092] Figure 2 A general block diagram of the weakly supervised target detection method based on iterative pseudo-full supervision training is provided for the embodiments of the present application.

[0093] Figure 3 A flowchart of a filtering operation method is provided for the embodiments of the present application.

[0094] Figure 4 A structure schematic diagram of a weakly supervised target detection system based on iterative pseudo-full supervision training is provided for the embodiments of the present application.

[0095] Figure 5 A visual comparison diagram of the detection results of the method and other mainstream weakly supervised target detection methods is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0096] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0097] As shown in the formula (I), the present application provides a weakly supervised object detection method based on iterative pseudo-full supervision training, which comprises the following steps: Figure 1

[0098] S1, a certain number of sample images are obtained to form sample data, and the sample data is divided into a training set and a test set according to a certain proportion;

[0099] S2, the training set is input into a multi-instance self-training model to obtain a corresponding detection result, and the detection result is used as an initial pseudo ground truth of the sample image, wherein the initial pseudo ground truth comprises a pseudo ground truth instance set P and a corresponding instance class score set S;

[0100] S3, filtering operations are respectively performed on the pseudo ground truth instance set P and the corresponding instance class score set S to obtain a filtered pseudo ground truth instance set PF and a corresponding class score set SF, and the filtering operation is used to remove pseudo ground truth instances with a class score lower than a set threshold;

[0101] S4, a class score guided weighted synthesis operation is performed on the obtained filtered pseudo ground truth instance set PF and the corresponding class score set SF to obtain a refined pseudo ground truth instance set PR and a corresponding class score set SR, and the class score guided weighted synthesis operation is used to: for the filtered pseudo ground truth instance set PF, the intersection over union of any two instances with the same class is compared with a set threshold, and according to the comparison result with the threshold, it is determined whether the related instance is directly added to the set PR or is synthesized and then added to the set PR, and the corresponding class score is also added to the set SR;

[0102] Preferably, in the present embodiment, one specific implementation of the class score guided weighted synthesis operation comprises:

[0103] ​For any two instances with the same category in the set PF, if the intersection-over-union of the two is less than or equal to a set low threshold, both are included in the set PR; if the intersection-over-union of the two is greater than the set low threshold and less than a set high threshold, the instance with the lower category score is eliminated and the other is included in the set PR; if the intersection-over-union of the two is greater than or equal to the high threshold, the two are first weighted combined, and the combined instance is included in the set PR, and the instance participating in the combination is no longer retained; the corresponding category score is also included in the set SR.

[0104] S5 training the YOLO model using the refined pseudo ground truth instance set PR to obtain corresponding detection results, i.e., a detected instance set PY and a corresponding category score set SY;

[0105] S6 inputting PR, SR, PY and SY into a pseudo ground truth updating strategy to obtain an updated pseudo ground truth instance set PIR and a corresponding category score set SIR, the pseudo ground truth updating strategy including the filtering operation and the intersection-over-union guided weighted combination operation, the set PY and the corresponding category score set SY after the filtering operation are denoted as a set PFY and a set SFY respectively, and the intersection-over-union guided weighted combination operation is used for: for any one instance in the set PFY, finding a same-category instance in the set PR with the largest intersection-over-union with the instance, comparing the largest intersection-over-union with a set threshold, and determining, according to a comparison result, whether the related instance in the set PFY is directly added to the set PIR or is combined with the same-category instance in the set PR with the largest intersection-over-union and then added to the set PIR, and the corresponding category score is also added to the set SIR.

[0106] Preferably, in the embodiment, a specific implementation of the intersection-over-union guided weighted combination operation includes:

[0107] For any one instance in the set PFY, finding a same-category instance in the set PR with the largest intersection-over-union with the instance, if the largest intersection-over-union is less than or equal to a set low threshold, the instance in the set PFY is included in the set PIR; if the largest intersection-over-union is greater than the set low threshold and less than a set high threshold, the instance in the set PFY is eliminated; if the largest intersection-over-union is greater than or equal to the set high threshold, the two are second weighted combined according to the largest intersection-over-union, and the combined result is included in the set PIR; all instances in the set PR that do not participate in the second weighted combination are all included in the set PIR; and the corresponding category score is also included in the set SIR.

[0108] S7 re-train the YOLO model with the updated pseudo ground truth instance set PIR, and obtain the corresponding detection results, empty the elements in set PY and set SY, and assign the elements in the instance set and the corresponding class score set in the detection results to set PY and set SY respectively; empty the elements in set PR and set SR, and assign the elements in set PIR and set SIR to set PR and set SR respectively.

[0109] S8 iterate the training of steps S6 and S7 for several times to obtain the trained IPS-YOLO model.

[0110] In the embodiment, steps S2-S8 are mainly to construct a weakly supervised object detection model based on iterative pseudo full supervision training, which is referred to as IPS-YOLO model hereinafter, and to train the model.

[0111] The IPS-YOLO model takes MIST [Z. Ren, Z. Yu, X. Yang, M.-Y. Liu, Y. J. Lee, A. G. Schwing, and J. Kautz, “Instance-aware, context-focused, and memory-efficient weakly supervised object detection,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 10598-10607.] as the benchmark model, inputs the sample image into the benchmark model to obtain the corresponding detection result, and takes the detection result as the initial pseudo ground truth of the sample image. The initial pseudo ground truth includes an initial pseudo ground truth instance set and a corresponding instance class score set, which are denoted as P and S respectively.

[0112] In the embodiment, an initial pseudo ground truth refining strategy is first proposed to improve the quality of the initial pseudo ground truth obtained in step S2. The initial pseudo ground truth refining strategy includes a filtering operation and a class score guided weighted synthesis operation. The purpose of the filtering operation is to remove instances with a class score lower than a set threshold T CCS The class score guided weighted synthesis operation is as follows: for two instances with the same class, if the intersection over union of the two instances is less than or equal to a set low threshold T L , both of them are retained; if the intersection over union of the two instances is greater than T L and less than a set high threshold T H , the instance with a lower class score is removed; if the intersection over union of the two instances is greater than or equal to T HIf the two are not the same, then the two are combined according to the respective category scores. That is, first, the instances with too low category scores are filtered out, and then the remaining instances of the same category are combined according to the category scores. The main purpose of this operation is to make the model not only focus on the most prominent part of the target of the category, but also pay attention to other parts of the target that have not been considered, so as to make the target coverage higher.

[0113] In this embodiment, the intersection over union IOU is a abbreviation of English intersection over union, which means the ratio of the intersection to the union of any two rectangular frames. Assuming that A and B are any two rectangular frames, the intersection of A and B is A∩B, and the union of A and B is A∪B, and the calculation formula of IoU is: IoU=(A∩B) / (A∪B).

[0114] The details of the filtering operation and the category score guided weighted combination operation are shown in Algorithm 1 and Algorithm 2 respectively. First, P and S are input into Algorithm 1 to obtain the filtered pseudo ground truth instance set and the corresponding category score set, which are marked as PF and SF respectively. Then, PF and SF are input into Algorithm 2 to obtain the refined pseudo ground truth instance set and the corresponding category score set, which are marked as PR and SR respectively.

[0115] Algorithm 1: Pseudo code of filtering operation

[0116]

[0117]

[0118] Algorithm 2: Pseudo code of category score guided weighted combination operation

[0119]

[0120]

[0121] Specifically, in this embodiment, as shown in Figure 3 , the pseudo ground truth instance set P and the corresponding instance category score set S are respectively subjected to filtering operation to obtain the filtered pseudo ground truth instance PF and the corresponding category score SF, which include:

[0122] S31 represents the relevant parameters of input and output, and the input is:

[0123] P={P1,...,P c ,...,P C} and S={S1,...,S c ,...,S C},

[0124] The output is:

[0125] PF = {PF1,..., PF c ,...,PF C} and SF = {SF1,..., SF c ,...,SF C};

[0126] wherein P c represents a set of initial pseudo-true instances of the c-th category, S c represents a category score of P c , PF c represents a set of instances after performing a filtering operation on P c , SF c represents a category score of PF c , C is a total number of categories of initial pseudo-true instances, and initially

[0127] S32 calculates a corresponding instance number idm that makes S c,i max, wherein S c,i represents a category score of P c,i , P c,i represents an i-th instance in P c , and S c,idm is compared with a category score threshold T CCS , if S c,idm <T CCS , the corresponding P c,idm is added to the output set PF; otherwise, step S33 is performed.

[0128] S33 traverses all instances in the set P c , and repeats the following operation: S c,i is compared with a category score threshold T CCS , if S c,i ≥T CCS , the corresponding PF c,i is added to the output set PF, and the corresponding SF c,i is added to the output set SF.

[0129] S34 iteratively performs steps S32 and S33 until all pseudo-true instances of C categories are traversed.

[0130] In this embodiment, the obtained filtered pseudo-true instances PF and the corresponding category scores SF are guided to perform a weighted synthesis operation, so as to obtain refined pseudo-true instances PR and the corresponding category scores SR, which specifically include the following steps:

[0131] S41 represents the relevant parameters of input and output, the input is:

[0132] PF={PF1,...,PF c ,...,PF C}、SF={SF1,...,SF c ,...,SF C}、T L 、T H , wherein T L , T H represent low and high intersection-over-union thresholds respectively; PF c represents the set of pseudo ground truth instances belonging to the c-th category after filtering, SF c represents the category score of PF c , and C is the total number of categories of pseudo ground truth instances;

[0133] The output is:

[0134] PR={PR1,...,PR c ,...,PR C} and SR={SR1,...,SR c ,...,SR C};

[0135] wherein PR c represents the set of instances after weighted synthesis guided by the category score of PF c , SR c represents the category score of PR c , and initially

[0136] S42 calculates the corresponding instance number idm that maximizes SF c,i , wherein SF c,i represents the category score of PF c,i , and PF c,i represents the i-th instance in PF c .

[0137] S43 traverses all instances in the set PF c , and repeats the following operations: if the intersection-over-union IoU(PF c,idm , PF c,i )≤T L , then take the union of PR c and PF c,i , and take the union of SR c and SF c,i ; otherwise, if the intersection-over-union IoU(PF c,idm , PF c,i )≥T HIf i is included in the index collector IB, where IB is initially an empty set; PF c,i and SF c,i are deleted from PF c and SF c respectively.

[0138] S44 calculates the number of elements in the index collector IB, denoted as M, and obtains the first weighted synthesized instance coordinates Cor, and obtains the corresponding category score and adds it to PR c , where represents the weighted synthesized instance with coordinates Cor belonging to the c-th category; and further obtains the corresponding category score and adds it to SR c .

[0139] S45 repeats steps S42-S44 until all pseudo-true value instances of the C categories are traversed.

[0140] Preferably, the first weighted synthesized instance coordinates Cor in the embodiment are represented as:

[0141]

[0142] where ω j is the weight of Cor_PF c,IB{j} , Cor_PF c,IB{j} represents the coordinates of PF c,IB{j} , PF c,IB{j} represents the IB{j}th instance in PF c , SF c,IB{j} represents the corresponding category score of PF c,IB{j} .

[0143] the corresponding category score is represented as:

[0144]

[0145] From step S5, the embodiment begins to focus on solving the problem of missing detection of different categories of targets, and uses the refined pseudo-true value instance set PR to train the YOLO model to obtain the corresponding detection result, i.e., the detected instance set PY and the corresponding category score set SY.

[0146] An example of a fully supervised object detection model used in the embodiments of the present application is the YOLO (You Only Look Once) model, which is a popular object detection model, and the embodiments of the present application do not limit the specific version of the YOLO model. For a fully supervised object detection model, the training data of the model needs to contain both input images and corresponding correct labels, so that the model can learn by comparing the predicted results and the true labels.

[0147] Characteristics of fully supervised object detection model training: label requirement: each training sample needs an explicit label, which is necessary for object detection, because the model needs to know the class of each object and the accurate position of the object, and the position of the object is usually represented by a rectangular box; direct optimization: the model directly optimizes the difference between the predicted results and the true labels, usually through a loss function.

[0148] The working principle of the YOLO model mainly includes: input: the image is input to the network. Prediction: the network outputs a series of bounding boxes, each bounding box contains the class probability and position information of the object. Loss calculation: the loss is calculated by calculating the difference between the predicted bounding box and the true bounding box, which includes regression loss and classification loss. Optimization: adjust the network weights through the backpropagation algorithm to minimize the loss function.

[0149] Based on the above basic content, the present application proposes an iterative pseudo-full supervision training of the YOLO model, which can continuously improve the quality of the refined pseudo-true value obtained in step S4 during the iterative training process, so as to continuously improve the training quality of the YOLO model.

[0150] Specifically, in the present embodiment, the process of iterative pseudo-full supervision training is as follows: first, use the PR obtained in algorithm 2 to train the YOLO model, and the training method is described in the following literature:

[0151] [G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” Software, version 8.0.0, 2023. [Online]. Available: https: / / github.com / ultralytics / ultralytics].

[0152] Next, the sample image is imported into the trained YOLO model to obtain the corresponding detection result, i.e., the detected instance and the corresponding class score, denoted as PY and SY, respectively. Finally, PR, SR, PY and SY are input into the pseudo ground truth updating strategy to obtain the updated pseudo ground truth instance and the corresponding class score, denoted as PIR and SIR, respectively. The above steps are repeated NIR times, i.e., PR = PIR and SR = SIR. The YOLO model after iterative training is denoted as IPS-YOLO model.

[0153] As shown in Figure 2 The pseudo ground truth updating strategy includes filtering operation and weighted synthesis operation guided by intersection over union. Details of the filtering operation are shown in Algorithm 1, and details of the weighted synthesis operation guided by intersection over union are shown in Algorithm 3.

[0154] Therefore, the execution process of the pseudo ground truth updating strategy is as follows.

[0155] First, PY and SY are substituted into Algorithm 1 to obtain the filtered instance and the corresponding class score, denoted as PFY and SFY, respectively. In this embodiment, since PR has been subjected to the above refining operation, it does not need to be filtered here, and the weighted synthesis operation guided by intersection over union can directly achieve better results.

[0156] Then, PR, SR, PFY and SFY are substituted into Algorithm 3 to obtain set PIR and set SIR.

[0157] Algorithm 3: Weighted synthesis operation guided by intersection over union pseudo code

[0158]

[0159]

[0160] According to the above pseudo code, algorithm 3 is to fuse the results output by YOLO with the pseudo ground truth used for training, so as to obtain a pseudo ground truth set that can cover more targets. The algorithm is for processing different categories of targets. For any instance in set PFY, find the same category instance in set PR with the maximum intersection-over-union, if the maximum intersection-over-union is less than or equal to the set low intersection-over-union threshold, then the instance in set PFY is included in set PIR, and the corresponding category score is also included in set SIR; that is, if the coincidence degree of the two instances is not high, then they are respectively added to the set as new pseudo ground truth instances. Since set PFY is obtained from the YOLO model, and set PR is obtained according to the refining operation of the present application, it is more reliable, therefore, if the maximum intersection-over-union is greater than the set low intersection-over-union threshold and less than the set high intersection-over-union threshold, then the instance in set PFY is selected to be eliminated in the embodiment, that is, if the coincidence degree of the two instances is high, only one target can be considered, then the corresponding element in set PFY is deleted, and only the element in set PR that is more reliable is retained as the subsequent pseudo ground truth instance; but if the maximum intersection-over-union is greater than or equal to the set high intersection-over-union threshold, then the two are secondarily weighted and synthesized according to the maximum intersection-over-union, and the synthesized result is included in set PIR, and the corresponding category score is also included in set SIR; that is, if the coincidence degree of the two instances is very high, then the two instances are synthesized to obtain a new pseudo ground truth instance. In the present embodiment, all instances in set PR that do not participate in the second weighted synthesis are included in set PIR, and the corresponding category score is also included in set SIR. That is, the above algorithm 3 can solve the problem of target missed detection when multiple targets of the same category are included.

[0161] Therefore, in the present embodiment, the updating strategy of the input pseudo ground truth PR, SR, PY and SY is used to obtain the updated pseudo ground truth instance PIR and the corresponding category score SIR, which specifically includes:

[0162] S61 represents the relevant parameters of input and output, and the input is:

[0163] PR = {PR1,...,PR c ,...,PR C},

[0164] SR = {SR1,...,SR c ,...,SR C},

[0165] PFY = {PFY1,...,PYF c ,...,PYF C};

[0166] SFY = {SFY1,...,SYF c ,...,SYFC},

[0167] T L ,T H , where T L 、T H Represent low and high IoU thresholds respectively; PFY c represents the set of instances belonging to class c in PFY, SFY c Represents the set PFY c PFY is the set of instances of PY after the filtering operation, SFY is the set of class scores corresponding to the set PFY, PR c It represents PF c The instance set after weighted synthesis guided by category scores, SR c Represents PR c Category score, PF c Represents the P c The instance set after the filtering operation is performed;

[0168] The output is:

[0169] PIR={PIR1,...,PIR c ,...,PIR C} and SIR={SIR1,...,SIR c ,...,SIR C};

[0170] Among them, PIR c It represents PR c Updated set, SIR c It stands for PIR c The category score set of

[0171] S62 calculation makes PFY c,i With PR c,j The instance number idk corresponding to the maximum intersection-union ratio between c,i With PR c,idm The intersection-over-union ratio between them is recorded as MIoU; among them, PR c,j Represents PR c The jth instance in PFY c,i It stands for PFY c The i-th instance in ;

[0172] S63 If ​​MIoU≤T L , then the corresponding PFY c,i Join PIR c In thec,i Merged into SIR c Among them, SFY c,i It stands for PFY c,i Category score; otherwise: if MIoU ≥ T H , then PR c,idm and PFY c,i Perform the second weighted synthesis operation to obtain the corresponding coordinates CWS, and then combine the weighted synthesised instances with the coordinates CWS belonging to the cth class Join PIR c Calculation Corresponding category scores and will Join SIR c PR c,idm and SR c,idm From PR c and SR c Remove the SR c,idm Represents PR c,idm Category score of

[0173] S64 will PFY c,i and SFY c,i From PFY c SFY c Remove;

[0174] S65 loops through steps S62-S64 until PYF is traversed. c All instances of ;

[0175] PIR c With PR c The combined result is assigned to PIR c , SIR c With SR c The combined result is assigned to SIR c .

[0176] S66 loops through steps S62-S65 until all pseudo-truth value instances of C categories are traversed.

[0177] In this embodiment, in step S63, PR c,idm and PFY c,i The second weighted synthesis operation is performed to obtain the corresponding coordinate CWS, which is expressed as:

[0178]

[0179] Among them, Cor_PR c,idm Represents PR c,idm Coordinates of Cor_PFY c,irepresenting the coordinates of PFY c,i .

[0180] corresponding class score is represented as:

[0181]

[0182] S9 will test the IPS-YOLO model for the test set, and get the corresponding detection result.

[0183] As shown in Figure 2 , the input image is imported into the YOLO model after iterative training, that is, the IPS-YOLO model, and the corresponding detection result can be inferred. For specific inference strategies, see YOLOv8 [G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” Software, version 8.0.0, 2023. [Online]. Available: https: / / github.com / ultralytics / ultralytics].

[0184] In order to verify the effectiveness of the present application, the following detailed experimental description of the scheme is given in this embodiment:

[0185] Step one: this embodiment constructs a weakly supervised object detection model based on iterative pseudo-full supervision training, hereinafter referred to as IPS-YOLO model. The IPS-YOLO model takes MIST [Z. Ren, Z. Yu, X. Yang, M.-Y. Liu, Y. J. Lee, A. G. Schwing, and J. Kautz, “Instance-aware, context-focused, and memory-efficient weakly supervised object detection,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 10598-10607.] as the benchmark model, and inputs the sample image into the benchmark model to obtain the corresponding detection result. The detection result is used as the initial pseudo ground truth of the sample image. The initial pseudo ground truth includes an initial pseudo ground truth instance set and a corresponding instance class score set, which are marked as P and S, respectively.

[0186] Step two: the refining strategy of initial pseudo ground truth. The present invention proposes a refining strategy of initial pseudo ground truth to improve the quality of the initial pseudo ground truth obtained in step one. The refining strategy of initial pseudo ground truth includes filtering operation and class score guided weighted synthesis operation. The purpose of filtering operation is to remove instances with class score lower than a set threshold T CCS = 0.3. The purpose of class score guided weighted synthesis operation is: for two instances with the same class, if their intersection over union is less than or equal to a set low intersection over union threshold T L = 0.4, both are retained; if their intersection over union is greater than T L and less than a set high intersection over union threshold T H = 0.8, the instance with lower class score is removed; if their intersection over union is greater than or equal to T H , both are first weighted synthesized according to their respective class scores. The details of filtering operation and class score guided weighted synthesis operation are shown in Algorithm 1 and Algorithm 2 respectively. First, P and S are input into Algorithm 1 to obtain filtered pseudo ground truth instances and corresponding class scores, denoted as PF and SF respectively. Then, PF and SF are input into Algorithm 2 to obtain refined pseudo ground truth instances and corresponding class scores, denoted as PR and SR respectively.

[0187] Step three: iterative pseudo fully supervised training of YOLO model. The present invention proposes an iterative pseudo fully supervised training of YOLO model, which can continuously improve the quality of refined pseudo ground truth obtained in step two during the iterative training process, so as to continuously improve the training quality of YOLO model.

[0188] The process of iterative pseudo fully supervised training is as follows: first, use PR obtained in Algorithm 2 to train YOLO model, and the training method is referred to in the literature [G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” Software, version 8.0.0, 2023. [Online]. Available: https: / / github.com / ultralytics / ultralytics].

[0189] Next, the sample image is imported into the trained YOLO model to obtain the corresponding detection results, i.e. the detected instances and corresponding class scores, denoted as PY and SY respectively.

[0190] Finally, the updating strategy of PR, SR, PY and SY into pseudo ground truth is obtained, and the updated pseudo ground truth instance and the corresponding category score are denoted as PIR and SIR, respectively. The above steps are repeated NIR=2 times, i.e. PR=PIR and SR=SIR. The YOLO model after iterative training is the IPS-YOLO model.

[0191] The updating strategy of the pseudo ground truth includes filtering operation and weighted synthesis operation guided by intersection over union. The details of the filtering operation are shown in Algorithm 1, and the details of the weighted synthesis operation guided by intersection over union are shown in Algorithm 3. In this experiment, the purpose of the filtering operation is to remove instances with a category score lower than a set threshold T CCS =0.3. Therefore, PY and SY are first substituted into Algorithm 1 to obtain the filtered instance and the corresponding category score, which are denoted as PFY and SFY, respectively. Since PR has been subjected to the refining operation described above, it does not need to be filtered here, and the weighted synthesis operation guided by intersection over union can achieve better results.

[0192] Then, PR, SR, PFY and SFY are substituted into Algorithm 3 for the weighted synthesis operation guided by intersection over union to obtain set PIR and set SIR.

[0193] In the weighted synthesis operation guided by intersection over union in this experiment, if the maximum intersection over union is less than or equal to a set low intersection over union threshold T L =0.4, the instances in set PFY are included in set PIR, and the corresponding category scores are also included in set SIR. If the maximum intersection over union is greater than T L and less than a set high intersection over union threshold T H =0.8, this embodiment chooses to eliminate the instances in set PFY. If the maximum intersection over union is greater than or equal to T H , the two are subjected to a second weighted synthesis according to the maximum intersection over union, and the synthesized result is included in set PIR, and the corresponding category score is also included in set SIR.

[0194] Step 4: Inference of the IPS-YOLO model. The input image is imported into the IPS-YOLO model after iterative training, i.e. the IPS-YOLO model, to obtain the corresponding detection result. The specific inference strategy is shown in YOLOv8 [G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” Software, version 8.0.0, 2023. [Online]. Available: https: / / github.com / ultralytics / ultralytics].

[0195] The hardware configuration for implementing the present application is: CPU: 2x E5-2650V4@2.20GHz, 512GB memory, 4x NVIDIA RTX TITAN 24GB graphics card; software configuration: CUDA12.6, Ubuntu22.04, python3.8, pytorch1.13.1.

[0196] In order to better prove the performance of the present application, a comparative experiment is selected on the public data set NWPU VHR-10 [K. Li, G. Cheng, S. Bu, and X. You, “Rotation-insensitive and context augmented object detection in remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 4, pp. 2337-2348, 2018.]. In the experiment of the present embodiment, the specific version of the YOLO model used is YOLOv8m, and details of YOLOv8m can be found in [G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” Software, version 8.0.0, 2023. [Online]. Available: https: / / github.com / ultralytics / ultralytics].

[0197] As shown in Table 1, the present application is compared with 14 kinds of similar methods, and the 14 kinds of similar methods are

[0198] (1) WSDDN [H. Bilen, A. Vedaldi, Weakly supervised deep detection networks, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2846-2854],

[0199] (2) OICR [P. Tang, X. Wang, X. Bai, W. Liu, Multiple instance detection network with online instance classifier refinement, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 3059-2953067],

[0200] (3) PCL [P. Tang, X. Wang, S. Bai, W. Shen, X. Bai, W. Liu, A. L. Yuille, PCL: proposal cluster learning for weakly supervised object detection, IEEE Trans. Pattern Anal. Mach. Intell. 42(1) (2020) 176-191],

[0201] (4) MIST [Z. Ren, Z. Yu, X. Yang, M.-Y. Liu, Y. J. Lee, A. G. Schwing, and J. Kautz, “Instance-aware, context-focused, and memory-efficient weakly supervised object detection,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 10598-10607.],

[0202] (5) PCIR [X. Feng, J. Han, X. Yao, and G. Cheng, “Progressive contextual instance refinement for weakly supervised object detection in remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 11, pp. 8002-8012, Apr. 2020.],

[0203] (6) MIG [B. Wang, Y. Zhao, and X. Li, “Multiple instance graph learning for weakly supervised remote sensing object detection,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1-12, 2021.],

[0204] (7) TCA [X. Feng, J. Han, X. Yao, and G. Cheng, “Tcanet: Triple context aware network for weakly supervised object detection in remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 8, pp. 6946-6955, 2021.],

[0205] (8) SAE [X. Feng, X. Yao, G. Cheng, J. Han, and J. Han, “Saenet: Self-supervised adversarial and equivariant network for weakly supervised object detection in remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-11, 2022.],

[0206] (9) SPG [G. Cheng, X. Xie, W. Chen, X. Feng, X. Yao, and J. Han, “Self-guided proposal generation for weakly supervised object detection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-11, 2022.],

[0207] (10) MHQ-PSL [X. Qian, Y. Huo, G. Cheng, C. Gao, X. Yao, and W. Wang, “Mining high-quality pseudo-instance soft labels for weakly supervised object detection in remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, no. 5607615, pp. 1-15, 2023.],

[0208] (11) SGPLM-IR [X. Qian, C. Li, W. Wang, X. Yao, and G. Cheng, “Semantic segmentation guided pseudo label mining and instance re-detection for weakly supervised object detection in remote sensing images,” International Journal of Applied Earth Observation and Geoinformation, vol. 119, no. 103301, pp. 1-11, 2023.],

[0209] (12) PIIFR [S. Zheng, Z. Wu, Y. Xu, and Z. Wei, “Weakly supervised object detection for remote sensing images via progressive image-level and instance-level feature refinement,” Remote Sensing, vol. 16, no. 7, 2024. [Online]. Available: https: / / www.mdpi.com / 2072-4292 / 16 / 7 / 1203],

[0210] (13) HIMWR [P. Xing, M. Huang, C. Wang, and Y. Cao, “High-quality instance mining and weight re-assigning for weakly supervised object detection in remote sensing images,” Electronics, vol. 13, no. 23, 2024. [Online]. Available: https: / / www.mdpi.com / 2079-9292 / 13 / 23 / 4753],

[0211] (14) AE-IS [X. Xie, G. Cheng, X. Feng, X. Yao, X. Qian, and J. Han, “Attention erasing and instance sampling for weakly supervised object detection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-10, 2024.].

[0212] Table 1 Comparative experimental results of the method of the present application and 14 kinds of similar methods on the NWPU VHR-10 dataset

[0213] method mAP CorLoc WSDDN 35.12 35.24 OICR 34.52 40.01 PCL 39.41 45.06 MIST 51.52 70.30 PCIR 54.97 71.87 MIG 55.95 70.16 TCA 58.82 72.76 SAE 60.72 73.46 SPG 62.80 73.41 MHQ-PSL 63.80 74.30 SGPLM-IR 65.20 75.40 PIIFR 65.20 75.20 HIMWR 66.24 76.89 AE-IS 72.00 76.10 The present invention 76.20 85.68

[0214] The comparative experimental results of the method of the present application and 14 kinds of similar methods on the NWPU VHR-10 dataset are shown in Table 1. According to the experimental results, the mAP and Corloc of the present application on the NWPU VHR-10 dataset reached 76.20% and 85.68%, respectively, which are better than 14 kinds of similar methods. mAP and Corloc represent the average precision and average positioning accuracy of target detection, respectively.

[0215] The visual comparison of the detection results of the present application and the mainstream weakly supervised target detection method is shown in Figure 5 , where GT represents the ground truth of manual annotation. As shown in Figure 5 (a), other weakly supervised target detection methods usually focus on the most salient part of the port, while the IPS-YOLO of the present application can completely locate the port. Figure 5 Similar situations also occur in Figure 5 (d), which shows that the present application can solve problem 1) in the background art: the tendency to focus on the most salient part of the target rather than the entire target body. As shown in Figure 5 (c), similar situations also occur, which shows that the present application can solve problem 2) in the background art: the problem of target omission is prone to occur.

[0216] On the other hand, as shown in Figure 4 , the present application also provides a weakly supervised target detection system based on iterative pseudo-full supervision training, which comprises:

[0217] A dataset acquisition module for acquiring a certain number of sample images to form sample data, and cutting the sample data into a training set and a test set according to a certain proportion.

[0218] a model construction and training module,

[0219] a benchmark model training unit configured to input the training set into a multi-instance self-training model to obtain a corresponding detection result, and use the detection result as an initial pseudo ground truth of the sample image, wherein the initial pseudo ground truth includes a pseudo ground truth instance set P and a corresponding instance category score set S.

[0220] a filtering unit configured to perform filtering operations on the pseudo ground truth instance set P and the corresponding instance category score set S respectively to obtain filtered pseudo ground truth instances PF and corresponding category scores SF, wherein the filtering operations are configured to remove instances with category scores lower than a set threshold.

[0221] a first weighted synthesis unit configured to perform category score guided weighted synthesis operations on the obtained filtered pseudo ground truth instance set PF and the corresponding category score set SF to obtain refined pseudo ground truth instances PR and a corresponding category score set SR, wherein the category score guided weighted synthesis operations are configured to: for the filtered pseudo ground truth instance set PF, compare an intersection over union of any two instances with the same category with a set threshold, and determine, according to a comparison result with the threshold, whether to directly add the related instances to the set PR or to add the related instances to the set PR after synthesis, and the corresponding category scores are also added to the set SR.

[0222] a YOLO model training unit configured to train a YOLO model using the refined pseudo ground truth instances PR to obtain a corresponding detection result, i.e., detected instances PY and corresponding category scores SY.

[0223] an update strategy unit configured to input PR, SR, PY and SY into a pseudo ground truth update strategy to obtain updated pseudo ground truth instances PIR and a corresponding category score set SIR, wherein the pseudo ground truth update strategy includes the filtering operations and intersection over union guided weighted synthesis operations, and the set PY and the corresponding category score set SY are denoted as sets PFY and SFY respectively after the filtering operations, and the intersection over union guided weighted synthesis operations are configured to: for any instance in the set PFY, find a same-category instance in the set PR with a maximum intersection over union, compare the maximum intersection over union with a set threshold, and determine, according to a comparison result, whether to directly add the related instance in the set PFY to the set PIR or to add the related instance to the set PIR after synthesis with the same-category instance in the set PR with the maximum intersection over union, and the corresponding category scores are also added to the set SIR.

[0224] The YOLO model secondary training unit is configured to retrain the YOLO model by using the updated pseudo ground truth instance set PIR, obtain a corresponding detection result, empty the elements in the set PY and the set SY, and assign the elements in the instance set and the corresponding class score set in the detection result to the set PY and the set SY, respectively; empty the elements in the set PR and the set SR, and assign the elements in the set PIR and the set SIR to the set PR and the set SR, respectively.

[0225] The iteration unit is configured to iteratively train for several times to obtain the trained IPS-YOLO model.

[0226] The prediction module is configured to perform inference prediction on the IPS-YOLO model by using a test set to obtain a corresponding prediction result.

[0227] The other technical features of the weakly supervised object detection system based on the iterative pseudo full-supervised training according to the application are similar to those of the weakly supervised object detection method based on the iterative pseudo full-supervised training, and thus are not described herein.

[0228] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0229] Although the preferred embodiments of the application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the application.

[0230] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the application without departing from the spirit and scope of the embodiments of the application. Thus, if these modifications and variations of the embodiments of the application fall within the scope of the claims of the application and their equivalent technologies, the application also intends to include these modifications and variations.

Claims

1. A weakly supervised target detection method based on iterative pseudo-fully supervised training, characterized in that: The method comprises the following steps: S1 obtains a certain number of sample images to form sample data, and divides the sample data into a training set and a test set according to a certain ratio; S2 inputs the training set into a multi-instance self-training model to obtain a corresponding detection result, and uses the detection result as an initial false-true value of the sample image, wherein the initial false-true value includes a false-true value instance set P and a corresponding instance category score set S; S3 performs filtering operations on the pseudo-truth instance set P and its corresponding instance category score set S, respectively, to obtain a filtered pseudo-truth instance set PF and a corresponding category score set SF, wherein the filtering operation is used to remove pseudo-truth instances whose category scores are lower than a set threshold; S4 performs a category score-guided weighted synthesis operation on the obtained filtered pseudo-truth instance set PF and the corresponding category score set SF, thereby obtaining a refined pseudo-truth instance set PR and a corresponding category score set SR, wherein the category score-guided weighted synthesis operation is used to: for the filtered pseudo-truth instance set PF, compare the intersection-over-union ratio of any two instances of the same category with a set threshold, and determine, based on the comparison result with the threshold, whether to directly add the relevant instances to the set PR or to synthesize them before adding them to the set PR, and the corresponding category scores are also simultaneously added to the set SR; S5 uses the refined pseudo-truth instance set PR to train the YOLO model to obtain corresponding detection results, namely: a detected instance set PY and a corresponding category score set SY; S6 inputs PR, SR, PY and SY into the pseudo-truth value update strategy to obtain the updated pseudo-truth value instance set PIR and the corresponding category score set SIR, wherein the pseudo-truth value update strategy includes the filtering operation and the weighted synthesis operation guided by the intersection-and-union ratio. The set PY and the corresponding category score set SY are respectively recorded as sets PFY and SFY after the filtering operation. The weighted synthesis operation guided by the intersection-and-union ratio is used to: for any instance in the set PFY, find the same category instance with the maximum intersection-and-union ratio in the set PR, compare the maximum intersection-and-union ratio with the set threshold, and determine whether to directly add the relevant instance in the set PFY to the set PIR or to synthesize it with the same category instance with the maximum intersection-and-union ratio in the set PR and then add it to the set PIR based on the comparison result. The corresponding category score is also synchronously added to the set SIR. S7 uses the updated pseudo-truth instance set PIR to train the YOLO model again to obtain corresponding detection results, clears the elements in the set PY and the set SY, and assigns the elements in the instance set and the corresponding class score set in the detection result to the set PY and the set SY respectively; clears the elements in the set PR and the set SR, and assigns the elements in the set PIR and the set SIR to the set PR and the set SR respectively; S8 iterates steps S6 and S7 several times to obtain a trained IPS-YOLO model; S9 uses the test set to infer and predict the IPS-YOLO model to obtain corresponding prediction results.

2. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 1, characterized in that The class score guided weighted synthesis operation includes: For any two instances of the same category in set PF, if their intersection-over-union ratio is less than or equal to the set low threshold, both are included in set PR; if their intersection-over-union ratio is greater than the set low threshold and less than the set high threshold, the instance with the lower category score is eliminated and the other is included in set PR; if their intersection-over-union ratio is greater than or equal to the high threshold, the two are first weightedly combined, and the combined instance is included in set PR, while the instances participating in the combination are no longer retained; the corresponding category scores are also simultaneously included in set SR.

3. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 1, characterized in that The weighted synthesis operation guided by the intersection-over-union ratio includes: For any instance in set PFY, search for the instance of the same category in set PR with the maximum intersection-over-union ratio. If the maximum intersection-over-union ratio is less than or equal to the set low threshold, the instance in set PFY is included in set PIR; if the maximum intersection-over-union ratio is greater than the set low threshold and less than the set high threshold, the instance in set PFY is eliminated; if the maximum intersection-over-union ratio is greater than or equal to the set high threshold, the two are subjected to a second weighted synthesis based on the maximum intersection-over-union ratio, and the synthesized result is included in set PIR; all instances in set PR that do not participate in the second weighted synthesis are included in set PIR; the corresponding category scores are also simultaneously included in set SIR.

4. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 1, characterized in that In step S3, filtering operations are performed on the pseudo-truth instance set P and its corresponding instance category score set S to obtain a filtered pseudo-truth instance set PF and a corresponding category score set SF, including: S31 represents the relevant parameters of input and output, and the input is: P={P1,...,P c ,...,P C } and S={S1,...,S c ,...,S C }, The output is: PF = {PF1,..., PF c ,..., PF C} and SF = {SF1,..., SF c ,..., SF C}; Among them, P c represents the set of initial pseudo-truth instances of type c, S c Represents P c Category score, PF c Represents the P c The instance set after the filtering operation is performed, SF c It represents PF c The category score of C is the total number of categories of the initial pseudo-truth instances, and initially S32 calculation makes S c,i The largest corresponding instance number idm, where S c,i Represents P c,i The category score, P c,i It represents P c The i-th instance in S c,idm and the category score threshold T CCS For comparison, if S c,idm <T CCS , then the corresponding P c,idm Add to the output set PF; otherwise, execute step S33; S33 traverses the set P c All instances of , and repeat the following steps: c,i and the category score threshold T CCS Compare, if S c,i ≥T CCS , then the corresponding PF c,i Add to the output set PF, and the corresponding SF c,i Add to the output set SF; S34 iterates steps S32 and S33 in a loop until all pseudo-truth value instances of C categories are traversed.

5. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 2, characterized in that The specific implementation steps of the category score-guided weighted synthesis operation include: S41 represents the relevant parameters of input and output, and the input is: PF={PF1,...,PF c ,...,PF C }、SF={SF1,...,SF c ,...,SF C }、T L 、T H , where T L 、T H Represent low and high IoU thresholds respectively; PF c represents the set of pseudo-truth instances belonging to category c after filtering, SF c It represents PF c The category score of , C is the total number of categories of false true value instances; The output is: PR = {PR1,..., PR c ,..., PR C} and SR = {SR1,..., SR c ,..., SR C}; Among them, PR c It represents PF c The instance set after weighted synthesis guided by category scores, SR c Represents PR c The category score of S42 calculation makes SF c,i The largest corresponding instance number idn, where SF c,i It represents PF c,i Category score, PF c,i It stands for PF c The i-th instance in ; S43 traverse set PF c All instances in , and repeat the following operation: If the intersection over union (IoU) c,idn ,PF c,i )≤T L , then PR c and PF c,i Take the union, SR c and SF c,i Take the union; otherwise, if the intersection over IoU (PF c,idn ,PF c,i )≥T H , then i is included in the index collector IB, where the index collector IB is initially an empty set; PF c,i and SF c,i From PF c and SF c Delete in; S44 calculates the number of elements in the index collector IB, marked as M, and obtains the instance coordinate Cor after the first weighted synthesis, and obtains And add it to PR c Among them, represents the weighted composite instance with coordinates Cor belonging to the cth class; and then we get Corresponding category scores And add it to SR c middle; S45 loops through steps S42-S44 until all pseudo-truth instances of the C categories are traversed.

6. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 5, characterized in that The instance coordinate Cor after the first weighted synthesis is expressed as: Among them, ω j It is Cor_PF c,IB{j} The weight of Cor_PF c,IB{j} PF c,IB{j} Coordinates of PF c,IB{j} Representative PF c The first instance of IB{j} in SF c,IB{j} Representative PF c,IB{j} the corresponding category scores; described Corresponding category scores Expressed as:

7. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 3, characterized in that The specific implementation steps of the weighted synthesis operation guided by the intersection-over-union ratio include: S61 represents the relevant parameters of input and output, and the input is: PR={PR1,...,PR c ,...,PR C , SR={SR1,...,SR c ,...,SR C }, PFY={PFY1,...,PYF c ,...,PYF C }; SFY={SFY1,...,SYF c ,...,SYF C }, T L ,T H , where T L 、T H Represent low and high IoU thresholds respectively; PFY c represents the set of instances belonging to the cth class in the set PFY, SFY c Represents the set PFY c PFY is the set of instances of the set PY after the filtering operation, SFY is the set of class scores corresponding to the set PFY, PR c It represents PF c The instance set after weighted synthesis guided by category scores, SR c Represents PR c Category score, PF c Represents the P c The instance set after the filtering operation is performed; The output is: PIR = {PIR1,..., PIR c ,..., PIR C} and SIR = {SIR1,..., SIR c ,..., SIR C}; Among them, PIR c It represents PR c Updated set, SIR c It stands for PIR c The category score set of S62 calculation makes PFY c,i With PR c,j The instance number idk corresponding to the maximum intersection-union ratio between c,i With PR c,idk The intersection-over-union ratio between them is recorded as MIoU; among them, PR c,j Represents PR c The j∈[1,|PR c |] instances, PFY c,i It stands for PFY c The i∈[1,|PFY c |] instances; S63 If ​​MIoU≤T L , then the corresponding PFY c,i Join PIR c In the c,i Join SIR c Among them, SFY c,i It stands for PFY c,i Category score; otherwise: if MIoU ≥ T H , then PR c,idk and PFY c,i The second weighted synthesis operation is performed to obtain the corresponding coordinates CWS, and the weighted synthesis instance PW belonging to the cth class with the coordinates CWS is c CWS Join PIR c Calculate PW c CWS Corresponding category score SW c CWS , and SW c CWS Join SIR c PR c,idk and SR c,idk From PR c and SR c Remove the SR c,idk Represents PR c,idk Category score of S64 will PFY c,i and SFY c,i From PFY c SFY c Remove; S65 loops through steps S62-S64 until PYF is traversed. c All instances of ; PIR c With PR c The combined result is assigned to PIR c , SIR c With SR c The combined result is assigned to SIR c ; S66 loops through steps S62-S65 until all pseudo-truth value instances of C categories are traversed.

8. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 7, characterized in that In step S63, PR c,idk and PFY c,i The second weighted synthesis operation is performed to obtain the corresponding coordinate CWS, which is expressed as: Among them, Cor_PR c,idk Represents PR c,idk Coordinates of Cor_PFY c,i It stands for PFY c,i 's coordinates.

9. The weakly supervised target detection method based on iterative pseudo fully supervised training according to claim 8, characterized in that The PW c CWS Corresponding category score SW c CWS Expressed as:

10. A weakly supervised target detection system based on iterative pseudo-fully supervised training, characterized in that: The system includes: The data set acquisition module is used to acquire a certain number of sample images to form sample data, and divide the sample data into a training set and a test set according to a certain ratio; Model building and training modules: a benchmark model training unit, configured to input the training set into a multi-instance self-training model to obtain a corresponding detection result, and use the detection result as an initial false-true value of the sample image, wherein the initial false-true value includes a set P of false-true value instances and a set S of corresponding instance category scores; a filtering unit configured to perform filtering operations on the pseudo-truth instance set P and its corresponding instance category score set S, respectively, to obtain filtered pseudo-truth instances PF and corresponding category scores SF, wherein the filtering operation is used to remove instances whose category scores are lower than a set threshold; a first weighted synthesis unit, configured to perform a category score-guided weighted synthesis operation on the obtained filtered pseudo truth instance set PF and the corresponding category score set SF, thereby obtaining a refined pseudo truth instance set PR and a corresponding category score set SR, wherein the category score-guided weighted synthesis operation is configured to: for the filtered pseudo truth instance set PF, compare the intersection-over-union ratio of any two instances of the same category with a set threshold, and determine, based on the comparison result with the threshold, whether to directly add the relevant instances to the set PR or to synthesize them before adding them to the set PR, and simultaneously add the corresponding category scores to the set SR; A YOLO model training unit is used to train the YOLO model using the refined pseudo-truth instances PR to obtain corresponding detection results, namely: a detected instance set PY and a corresponding category score SY; an update strategy unit, configured to input PR, SR, PY, and SY into an update strategy for pseudo-truth values, and obtain an updated pseudo-truth value instance set PIR and a corresponding category score set SIR, wherein the pseudo-truth value update strategy includes the filtering operation and an intersection-and-union (IoU)-guided weighted synthesis operation. The set PY and the corresponding category score set SY are respectively recorded as sets PFY and SFY after the filtering operation. The IoU-guided weighted synthesis operation is configured to: for any instance in the set PFY, search for an instance of the same category in the set PR with a maximum IoU, compare the maximum IoU with a set threshold, and determine, based on the comparison result, whether to directly add the relevant instance in the set PFY to the set PIR or to synthesize it with the instance of the same category in the set PR with the maximum IoU and then add it to the set PIR. The corresponding category score is also synchronously added to the set SIR. A YOLO model secondary training unit is configured to train the YOLO model again using the updated pseudo-truth instance set PIR to obtain corresponding detection results, clear the elements in the set PY and the set SY, and assign the elements in the instance set and the corresponding class score set in the detection result to the set PY and the set SY respectively; clear the elements in the set PR and the set SR, and assign the elements in the set PIR and the set SIR to the set PR and the set SR respectively; Iteration unit, used for iterative training several times to obtain the trained IPS-YOLO model; The prediction module is used to perform inference prediction on the IPS-YOLO model using the test set to obtain corresponding prediction results.

Images