Noisy target frame correction method based on uncertainty perception and complementary information

Abstract

The present invention discloses a noise target frame correction method based on uncertainty perception and complementary information. It mainly solves the problem of low accuracy of existing noise target frame correction technology. Its implementation scheme is: acquiring image tensor data; extracting image pyramid features; generating an initial candidate frame set based on the noise target frame; constructing a candidate frame optimization module based on uncertainty perception and a target frame correction module based on complementary information, and constructing a noise target frame correction network based on uncertainty perception and complementary information based on these two modules; inputting the image pyramid features and the initial candidate frame set into the target frame correction network, and correcting the noise target frame once through the candidate frame optimization module and the target frame correction module in turn to obtain a corrected target frame; repeating the correction process of the noise target frame to obtain the final corrected target frame, and completing the correction of the noise target frame. The present invention significantly improves the correction effect of the noise target frame and can be used to obtain a high-quality labeled data set.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and further designs a noise target frame correction method that can be used to obtain a high-quality labeled data set. Background Art

[0002] With its widespread application in various fields, object detection technology has experienced rapid development. As a key element of artificial intelligence, data plays a crucial role in the rapid development of object detection technology. However, manually constructing datasets is a complex and time-consuming task. Furthermore, datasets obtained through a combination of automated annotation and crawler technologies often contain problems such as noisy object boxes. Furthermore, even manually annotated datasets inevitably contain noisy object boxes. Noisy object boxes can cause the network to learn incorrect knowledge, affecting model accuracy and negatively impacting network performance.

[0003] With the continuous advancement of technology, more and more fields and industries have begun to realize the importance of data and have designed relevant algorithms to correct noisy target boxes in datasets, improving the robustness of target detection networks to noisy target boxes. Wu et al. designed the SSD-Det method to correct noisy target boxes during network training and use the corrected target boxes to supervise network training, reducing the impact of noisy target boxes on network performance. Although this method has achieved good results in correcting noisy target boxes, because it uses multi-instance learning scores to screen candidate boxes, the corrected target boxes obtained have the following two problems:

[0004] (1) Target frame offset problem: When there are multiple targets in an image, the corrected target frame corresponding to one target may overlap another target. This is because multi-instance learning only considers target category information. When the target categories are the same, the selected candidate frames may offset the corresponding target and overlap another target, resulting in the corrected target frame having a target frame offset problem.

[0005] (2) The problem of only covering the local area of ​​the target, that is, the corrected target box is not accurate and only contains the local area of ​​the target in the image. This is because multi-instance learning tends to "focus" on the salient information of the target, so the candidate box selected based on the multi-instance learning score may only cover the local area of ​​the target.

[0006] Patent application number CN114549863A discloses a light field salient object detection method based on pixel-level noisy label supervision. This method guides and assists the detection of salient objects by mining the rich scene information in high-dimensional light field data. This method fully utilizes the salient information available in the noisy labels and reduces the confidence level of the noisy information, effectively reducing the pre-experimental cost of salient object detection. However, this method fails to account for noisy object frames in the dataset and therefore cannot correct the object frames. Specifically, when noisy object frames are present in the dataset, the trained network performs poorly. Summary of the Invention

[0007] The purpose of the present invention is to address the shortcomings of the above-mentioned existing technologies and propose a noise target frame correction method based on uncertainty perception and complementary information to improve the correction quality of the noise target frame and thereby improve the detection accuracy of the target detection network.

[0008] The technical idea for achieving the purpose of the present invention is: to obtain target features across images by constructing a category feature group that can be updated online, provide complementary information for the target in the current image, and alleviate the problem of only covering the local area of ​​the target; by introducing an additional regression branch, obtain the uncertainty value of the network prediction candidate box and optimize the predicted candidate box based on the uncertainty value, alleviate the problem of target box offset, and improve the correction quality of the noisy target box; by improving the correction quality of the noisy target box, the detection accuracy of the target detection network for the target is improved.

[0009] According to the above technical ideas, the technical solution of the present invention includes the following steps:

[0010] (1) Obtain training tensor data and test tensor data:

[0011] 1a) Read the training set data and test set data of the target detection database respectively;

[0012] 1b) Set the normalized mean and standard deviation of the three RGB channels; adjust, flip, and normalize the images in the training set in sequence to obtain the tensor data corresponding to the training set images; adjust and normalize the images in the test set in sequence to obtain the tensor data corresponding to the test images;

[0013] (2) Read the training set annotation file to obtain the noise target box, and generate an initial candidate box set for each noise target box bn: P Ι ={P Ι,1 ,P Ι,2 …P Ι,i …P Ι,N}, where N represents P Ι The number of initial candidate boxes in P Ι,i Indicates P ΙThe i-th initial candidate box in , 1≤i≤N;

[0014] (3) Select an existing image feature extraction network, input the training tensor data into the network, and obtain the bottom-up image pyramid feature X';

[0015] (4) Constructing a noise target frame correction network based on uncertainty perception and complementary information:

[0016] 4a) Establish an uncertainty-aware candidate box optimization module T1, which includes a region of interest alignment network, a fully connected layer, and a regression layer;

[0017] 4b) Establishing a complementary information-based target box correction module T2, which includes a region of interest alignment network, a fully connected layer, a multi-instance learning network, and a category feature group;

[0018] 4c) The uncertainty-aware candidate box optimization module T1 and the complementary information-based target box correction module T2 are connected in series to form a noise target box correction network based on uncertainty perception and complementary information;

[0019] (5) Using image pyramid features X' and initial candidate box set P Ι , based on the noise target frame correction network of uncertainty perception and complementary information, the corrected target frame br ΙΙ :

[0020] 5a) Based on the image pyramid feature X' and the initial candidate box set P Ι , the uncertainty-aware candidate box optimization module T1 optimizes the initial candidate box set P Ι , get the rough candidate box set: Pr Ι ={Pr Ι,1 ,Pr Ι,2 …Pr Ι,i …Pr Ι,N}, where Pr Ι,i Indicates Pr Ι The i-th rough candidate box in ;

[0021] 5b) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn and obtains a corrected target frame br Ι ;

[0022] 5c) Repeat step 5a) to optimize the rough candidate box set The candidate box in the , obtain the refined candidate box set: Pr ΙΙ ={Pr ΙΙ,1 ,Pr ΙΙ,2 …Pr ΙΙ,i…Pr ΙΙ,N}, where Pr ΙΙ,i Indicates Pr ΙΙ The i-th refined candidate box in;

[0023] 5d) Repeat step 5b) according to the fine candidate box set Pr ΙΙ Get the final corrected target frame br ΙΙ ;

[0024] (6) According to the final corrected target frame br ΙΙ , design the overall training loss L.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] First, the present invention optimizes the candidate frame in the constructed uncertainty perception-based candidate frame optimization module, provides a higher quality candidate frame for subsequent correction of the noisy target frame, and improves the position accuracy of the corrected target frame.

[0027] Second, the present invention designs an online updateable category feature group in the target frame correction module based on complementary information, obtains target features across images, and provides complementary information for the target in the current image to guide the correction of the noisy target frame, thereby improving the correction effect of the noisy target frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an implementation flow chart of the present invention;

[0029] Figure 2 Schematic diagram of the noise target frame correction network based on uncertainty perception and complementary information in the present invention;

[0030] Figure 3 yes Figure 2 Schematic diagram of the uncertainty-aware candidate box optimization module;

[0031] Figure 4 yes Figure 2 Schematic diagram of the target frame correction module based on complementary information;

[0032] Figure 5 This is an effect diagram of correcting a noise target frame using the present invention. DETAILED DESCRIPTION

[0033] To help those skilled in the art better understand the present invention, the following will provide a clear and complete description of the technical solutions and effects of the embodiments of the present invention, with reference to the accompanying drawings. It is apparent that the described embodiments are only a portion of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the step numbers in the specification and claims of the present invention are only for the purpose of clearly describing the embodiments of the present invention and facilitating understanding, and the order of the step numbers is not limited.

[0035] Reference Figure 1 , the implementation steps of this embodiment are as follows:

[0036] Step 1: Get the training tensor data and the test tensor data.

[0037] 1.1) Read the training set data and test set data of the target detection database respectively;

[0038] 1.2) Set the normalized mean and standard deviation of the three RGB channels:

[0039] 1.2.1) Set the normalized mean values ​​of the RGB channels to [123.675, 116.28, 103.53] respectively;

[0040] 1.2.2) Set the standard deviations of the three channels to [58.395, 57.12, 57.375] respectively;

[0041] 1.3) The images of the training set are preprocessed by adjusting, flipping and normalizing in sequence to obtain the tensor data corresponding to the training set images:

[0042] 1.3.1) Scale the input image to 1000×600;

[0043] 1.3.2) Randomly flip the scaled image with a probability of 0.5 to obtain the flipped image:

[0044] 1.3.3) Process the image using the mean-standard deviation normalization method to obtain the tensor data corresponding to the training image at this stage;

[0045] 1.4) The images in the test set are adjusted and normalized in turn to obtain the tensor data corresponding to the training set images:

[0046] 1.4.1) Scale the input image to 1000×600;

[0047] 1.4.2) Process the image using the mean-standard deviation normalization method to obtain the tensor data corresponding to the test image at this stage.

[0048] Step 2: Read the training set annotation file to obtain the noise target frame, and generate the initial candidate frame set P for each noise target frame bn Ι .

[0049] 2.1) Convert the noise target frame bn from the [xl, yl, xr, yr] format to the [x, y, w, h] format to obtain the converted noise target frame:

[0050] bn * =[bn x ,bn y ,bn w ,bn h ]

[0051] Where xl and yl represent the x and y coordinates of the upper left corner of the target box, respectively; xr and yr represent the x and y coordinates of the lower right corner of the target box, respectively; x and y represent the x and y coordinates of the center point of the target box after the format conversion, respectively; w and h represent the width and height of the target box after the format conversion, respectively;

[0052] 2.2) Set relevant parameters:

[0053] Set the size adjustment parameter s to modify the noise target box bn * The width and height of , set the parameter value range to: s∈{0.7,0.8,1,1.2,1.3};

[0054] Set the x coordinate adjustment parameter o x , used to modify the noise target box bn * The x coordinate of the center point; set the y coordinate adjustment parameter o y , used to modify the noise target box bn * The y coordinate of the center point;

[0055] Set the x and y coordinate adjustment parameters o x and o y The value range of is:

[0056] (o x ,o y )∈{(0,0),(2,0),(0,2),(-2,0),(-2,-2)}

[0057] 2.3) According to the noise target box bn * =[bn x ,bn y ,bn w ,bn h]Generate initial candidate box

[0058]

[0059] in, Indicates the first generated initial candidate box in the format of [x, y, w, h];

[0060] 2.4) The initial candidate box in the format of [x, y, w, h] Convert the [x,y,w,h] format to the [xl,yl,xr,yr] format to get the first initial candidate box P in the [xl,yl,xr,yr] format Ι,1 ;

[0061] 2.5) Repeat steps 2.3) and 2.4) N times to obtain the initial candidate box set P in the format [xl, yl, xr, yr] Ι .

[0062] Step 3: Extract the training image pyramid feature X' based on the existing image feature extraction network.

[0063] 3.1) Establish image feature extraction network:

[0064] This example uses but is not limited to the ResNet50 network and the Feature Pyramid Network as the image feature extraction network;

[0065] The ResNet50 network contains 5 concatenated convolutional layers, each of which has the same structure and contains several convolution groups; each convolution group contains a convolution operation, a batch normalization layer and a ReLu activation function;

[0066] The feature pyramid network includes 4 convolutional layers with a convolution kernel size of 1×1 and a stride of 1, and 4 convolutional layers with a convolution kernel size of 3×3 and a stride of 1;

[0067] 3.2) Extract the features C of the training image tensor data through the ResNet50 network:

[0068] The training image tensor data preprocessed in step 1 is input into the ResNet50 network for feature extraction. The image features C1, C2, C3, C4 and C5 are extracted respectively through 5 serial convolutional layers. These image features are used to construct the image feature set from bottom to top: C = {C1, C2, C3, C4, C5};

[0069] 3.3) Based on the image feature set C extracted by the ResNet50 network, the image pyramid feature X' is obtained through the feature pyramid network:

[0070] 3.3.1) Image features C2, C3, C4, and C5 in image feature set C are each subjected to four convolutional layers with a kernel size of 1×1 and a stride of 1, so that the number of channels of feature C2 remains at 256, the number of channels of feature C3 decreases from 512 to 256, the number of channels of feature C4 decreases from 1024 to 256, and the number of channels of feature C5 decreases from 2048 to 256;

[0071] 3.3.2) The highest-level feature C5 in the image feature set C is recorded as the highest-level feature C5' in the fused image feature set C'; C5' is upsampled by a factor of 2 and then added to the next-highest-level feature C4 in the image feature set C to obtain the next-highest-level feature C4' in the fused image feature set C'; C4' is upsampled by a factor of 2 and then added to the next-lowest-level feature C3 in the image feature set C to obtain the next-lowest-level feature C3' in the fused image feature set C'; C3' is upsampled by a factor of 2 and then added to the lowest-level feature C2 in the image feature set C to obtain the lowest-level feature C2' in the fused image feature set C';

[0072] 3.3.3) Using the above C5', C4', C3' and C2' features, construct the fused image feature set C' = {C2', C3', C4', C5'} from the bottom up;

[0073] 3.3.4) The features in the fused image feature set C' are respectively passed through four convolutional layers with a kernel size of 3×3 and a stride of 1 to obtain features X2′, X3′, X4′ and X5′. Feature X5′ is downsampled to obtain feature X6′. The features X2′, X3′, X4′, X5′ and X6′ are used to construct an image pyramid feature from bottom to top to obtain the image pyramid feature: X' = {X2', X3', X4', X5', X6'}.

[0074] Step 4: Construct a noise target frame correction network based on uncertainty perception and complementary information.

[0075] Reference Figure 2 , the specific implementation of this step is as follows:

[0076] 4.1) Establishing uncertainty-aware candidate box optimization module T1:

[0077] 4.1.1) Select a ROI alignment network with an output feature size of 7×7 and an output dimension of 256 to sample target features from the image pyramid feature X' and perform feature alignment through interpolation.

[0078] 4.1.2) Select a fully connected layer with an output dimension of 1024 to reduce the dimensionality of the target features;

[0079] 4.1.3) Select two regression layers with output dimensions of 4, which are used to calculate the uncertainty values ​​of the predicted candidate boxes and the network predicted candidate boxes respectively;

[0080] 4.1.4) Connect the region of interest alignment network, the fully connected layer, and the two parallel regression layers in sequence to establish the uncertainty-aware candidate box optimization module T1;

[0081] 4.2) Constructing target frame correction module T2 based on complementary information:

[0082] 4.2.1) Select the region of interest alignment network with an output feature size of 7×7 and an output dimension of 256 to sample the target features from the image pyramid feature X' and perform feature alignment through interpolation;

[0083] 4.2.2) Select a fully connected layer with an output dimension of 1024 to perform target feature dimensionality reduction;

[0084] 4.2.3) A multi-instance learning network consisting of two parallel classification branches and detection branches is used to calculate the multi-instance scores of candidate boxes;

[0085] 4.2.4) Design a category feature set ψ containing C groups of target features, each of which contains U target features, where C represents the total number of target categories.

[0086] 4.2.5) Connect the region of interest alignment network and the fully connected layer in series, and then connect them to the parallel-connected multi-instance learning network and the category feature group in sequence to construct the target bounding box correction module T2 based on complementary information;

[0087] 4.3) The uncertainty-aware candidate box optimization module T1 and the complementary information-based target box correction module T2 are connected in series to form a noise target box correction network based on uncertainty perception and complementary information.

[0088] Step 5: Use the image pyramid feature X' and the initial candidate box set P Ι , based on the noise target frame correction network of uncertainty perception and complementary information, the corrected target frame br ΙΙ :

[0089] 5.1) Optimize the initial candidate box set P using the uncertainty-aware candidate box optimization module T1 Ι , obtain the rough candidate box set Pr Ι :

[0090] Reference Figure 3 , the specific implementation of this step is as follows:

[0091] 5.1.1) Based on the initial candidate box set P obtained in step 2Ι And the image pyramid feature X' obtained in step 3, the initial candidate box P is obtained through the region of interest alignment network and the fully connected layer Ι The feature set of the target: F Ι ={F Ι,1 ,F Ι,2 …F Ι,i …F Ι,N}, where F Ι,i Indicates F Ι The i-th target feature in ;

[0092] 5.1.2) According to the target feature set F Ι , through two parallel regression layers, we obtain the predicted candidate box sets: And the uncertainty value set of the network prediction candidate box: V Ι ={V Ι,1 ,V Ι,2 …V Ι,i …V Ι,N},in express The i-th predicted candidate box, V Ι,i Indicates V Ι The i-th uncertainty value in ;

[0093] 5.1.3) Based on the uncertainty value set V of the candidate box predicted by the network Ι Optimize the prediction candidate box set Get the rough candidate box set Pr Ι :

[0094] 5.1.3.1) For the predicted candidate box set The i-th predicted candidate box Get the predicted candidate box set Center and predict candidate boxes A set of candidate boxes with IoU greater than 0.5: Pn = {Pn1, Pn2, ... Pn j …Pn A}, where A represents the number of candidate boxes in Pn, Pn j Represents the j-th candidate box in Pn, A≤N, 1≤j≤A;

[0095] 5.1.3.2) Based on the uncertainty value set V of the candidate box predicted by the network Ι , obtain the uncertainty value set corresponding to the candidate box set Pn: Vn={Vn1,Vn2…Vn j …Vn A}, Vn j represents the jth uncertainty value in Vn;

[0096] 5.1.3.3) Based on the predicted candidate box Calculate the confidence score set S of the candidate box Pn iou :

[0097]

[0098] in, It's S iou The j-th confidence score in Pn is the j-th candidate box Pn in Pn j Confidence score of

[0099] 5.1.3.4) Based on the candidate box set Pn and the uncertainty value set Vn corresponding to these candidate boxes and the confidence score set S iou , calculate the rough candidate box Pr Ι,i x coordinate of the upper left vertex and y coordinate The x-coordinate of the lower right vertex and y coordinate

[0100]

[0101] in, Represents the candidate box Pn j The confidence score of and Represents the candidate box Pn respectively j The x- and y-coordinates of the upper left vertex, and Represents the candidate box Pn respectively j The x- and y-coordinates of the lower right corner;

[0102] and Respectively represent the network's candidate box Pn j The uncertainty values ​​of the x- and y-coordinate predictions of the upper left vertex, and Respectively represent the network's candidate box Pn j The uncertainty values ​​of the x- and y-coordinate predictions of the lower right vertex;

[0103] 5.1.3.5) Based on the rough candidate box Pr Ι,i x coordinate of the upper left vertex and y coordinate The x-coordinate of the lower right vertex and y coordinate Calculate the rough candidate box Pr Ι,i :

[0104]

[0105] 5.1.3.6) Repeat steps 5.1.3.1) to 5.1.3.5) N times to obtain a rough candidate box set.

[0106] 5.2) Based on the image pyramid feature X' obtained in step 3 and the rough candidate box set Pr obtained in step 5 Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn and obtains a corrected target frame br Ι :

[0107] Reference Figure 4 , the specific implementation of this step is as follows:

[0108] 5.2.1) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , obtain the rough candidate box set Pr through the region of interest alignment network and the fully connected layer Ι The feature set Fr of the target Ι ={Fr Ι,1 ,Fr Ι,2 …Fr Ι,i …Fr Ι,N}, where Fr Ι,i It is Fr Ι The i-th target feature in ;

[0109] 5.2.2) According to the target feature set Fr Ι , calculate the rough candidate box set Pr through the multi-instance learning network Ι Multi-instance score set: S Ι ={S Ι,1 ,S Ι,2 …S Ι,i …S Ι,N}, where S Ι,i It's S Ι The i-th multi-instance score in:

[0110] 5.2.2.1) Set the rough candidate box set Pr Ι The feature set Fr of the target Ι The matrix x is obtained by sending it to the classification branch and detection branch in the multi-instance learning network respectively. cls and x det , for the matrix x cls Normalize to get the matrix

[0111]

[0112] in, Representation matrix The data in row i and column j in Represents the matrix x cls The data in row i and column j in Represents the matrix x cls The data in row i and column m;

[0113] 5.2.2.2) For matrix x det Normalize and obtain the normalized matrix

[0114]

[0115] in, Representation matrix The data in row i and column j in Represents the matrix x det The data in row i and column j in Represents the matrix x det The data in row i and column m;

[0116] 5.2.2.3) Normalized classification branch matrix and the detection branch matrix Perform inner product operation to obtain multi-instance learning matrix

[0117]

[0118] 5.2.2.4) For the rough candidate box set Pr Ι The i-th rough candidate box Pr Ι,i , take the multi-instance learning matrix In the i-th row of data Get data The value corresponding to the category yc is the rough candidate box Pr Ι,i The multi-instance score S Ι,i , yc is the rough candidate box Pr Ι,i The category containing the target;

[0119] 5.2.2.5) Repeat step 5.2.2.4) N times to obtain the rough candidate box set Pr Ι The multi-instance score set S Ι ;

[0120] 5.2.3) According to the rough candidate box set Pr Ι The feature Fr of the target Ι Update the category feature group ψ:

[0121] 5.2.3.1) Obtaining the multi-instance score set Sr Ι The maximum value of Sr Ι,max , obtain the rough candidate box set PrΙ The rough candidate box with the largest multi-instance score is obtained, and the feature Fr of the target contained in the rough candidate box is obtained. Ι,max ;

[0122] 5.2.3.2) Set the rough candidate box set Pr Ι The target category is yc, and the category feature group corresponding to category yc is obtained in Represents ψ yc The u-th feature in , 1≤u≤U, U represents the category feature group ψ yc The number of features in the

[0123] 5.2.3.3) Calculate the feature Fr of the target contained in the rough candidate box with the maximum multi-instance score Ι,max and the category feature group ψ yc Similarity of features in sm m ={sm m,1 ,sm m,2 …sm m,u …sm m,U},

[0124] in It is Fr Ι,max With ψ yc The u-th feature in The similarity of sm m The u-th similarity in , <> represents the inner product operation, |||| represents the two-norm operation;

[0125] 5.2.3.4) According to similarity sm m , from ψ yc Selection and feature Fr Ι,max Most similar features

[0126] 5.2.3.5) Using feature Fr Ι,max and multi-instance score Sr Ι,max Update the category feature group ψ yc Features in

[0127]

[0128] Where r is the momentum coefficient;

[0129] 5.2.3.6) Change the noise target box bn and the target category yc it contains, repeat steps 5.2.3.1) to 5.2.3.5) for a total of Z times to complete the update of the category feature group ψ, where Z represents the total number of targets in the training set.

[0130] 5.2.4) According to the category feature set ψ and the target feature set FrΙ , calculate the rough candidate box set Pr Ι Similarity score set of: Sim Ι ={Sim Ι,1 ,Sim Ι,2 …Sim Ι,i …Sim Ι,N}, where Sim Ι,i It's Sim Ι The i-th similarity score in ;

[0131] 5.2.4.1) Get Pr Ι The i-th rough candidate box Pr Ι,i and its feature Fr containing the target Ι,i , and obtain the rough candidate box Pr Ι,i Contains the target feature Fr Ι,i The category feature group ψ corresponding to category yc yc ;

[0132] 5.2.4.2) Calculate the feature Fr Ι,i and the category feature group ψ yc The u-th feature in Similarity sm i,u :

[0133]

[0134] 5.2.4.3) Repeat step 5.2.4.2) for U times to obtain the feature Fr Ι,i and the category feature group ψ yc Similarity of all features in sm i ={sm i,1 ,sm i,2 …sm i,u …sm i,U};

[0135] 5.2.4.4) Get similarity sm i The maximum value in is the rough candidate box Pr Ι,i Similarity score Sim Ι,i ;

[0136] 5.2.4.5) Repeat steps 5.2.4.1) to 5.2.4.4) N times to obtain the similarity score set Sim Ι .

[0137] 5.2.5) Based on the multi-instance score set S Ι and similarity score set Sim Ι Calculate the rough candidate box set Pr Ι The final score set of Sr Ι={Sr Ι,1 ,Sr Ι,2 …Sr Ι,i …Sr Ι,N}:

[0138] 5.2.5.1) According to S Ι The i-th multi-instance score S Ι,i and similarity score set Sim Ι The i-th similarity score Sim Ι,i , calculate the final score set Sr Ι The final score Sr of the i-th Ι,i :

[0139] Sr Ι,i =α*S Ι,i +β*Sim Ι,i

[0140] Among them, α and β are combination coefficients, α + β = 1;

[0141] 5.2.5.2) Repeat step 5.2.5.1) N times to obtain the final score set Sr Ι ;

[0142] 5.2.6) According to the rough candidate box set Pr Ι and its final score set Sr Ι Calculate the corrected target frame br Ι :

[0143] 5.2.6.1) According to the final score set Sr Ι And the rough candidate box set Pr Ι Get the highest H score sets S top ={S top,1 ,S top,2 ...S top,k …S top,H} and the corresponding candidate box set P top ={P top,1 ,P top,2 …P top,k …P top,H}, where P top,k It's P top The kth candidate box in S top,k It's S top The k-th score in , 1≤k≤H;

[0144] 5.2.6.2) According to the score set S top And the candidate box set P top Calculate the corrected target frame br Ι The x coordinate of the upper left vertex and y coordinate The x-coordinate of the lower right vertex and y coordinate

[0145]

[0146] Among them, S top,k is the score set S top The kth score, P top,k Is the candidate box set P top The kth candidate box in and They are candidate boxes P top,k The x- and y-coordinates of the upper left vertex, and They are candidate boxes P top,k The x- and y-coordinates of the lower right corner;

[0147] 5.2.6.3) According to the target frame after a correction Ι The x coordinate of the upper left vertex and y coordinate The x-coordinate of the lower right vertex and y coordinate Calculate the corrected target frame br Ι :

[0148] br Ι =[br xl ,br yl ,br xr ,br yr ].

[0149] 5.3) Repeat step 5.1) to optimize the rough candidate box set The candidate box in the , obtain the refined candidate box set: Pr ΙΙ ={Pr ΙΙ,1 ,Pr ΙΙ,2 …Pr ΙΙ,i …Pr ΙΙ,N}, where Pr ΙΙ,i Indicates Pr ΙΙ The i-th refined candidate box in .

[0150] 5.4) Repeat step 5.2) and calculate the fine candidate box set Pr ΙΙ Get the final corrected target framebr ΙΙ .

[0151] Step 6: According to the final corrected target frame ΙΙ , design the overall training loss L.

[0152] 6.1) Calculate the total loss L of the uncertainty-aware candidate box optimization module T1 PR :

[0153] 6.1.1) Calculate the loss function corresponding to the x-coordinate and y-coordinate of the upper left corner of a single rough candidate box and And the x-coordinate and y-coordinate corresponding loss function of the lower right corner of a single rough candidate box and

[0154]

[0155] in, and b xl Represents the x coordinate of the upper left corner vertex of the candidate box and the target box, v xl Represents the uncertainty value of the network's prediction result of the x-coordinate of the upper left corner vertex of the candidate box;

[0156] and b yl Represents the y coordinate of the upper left corner vertex of the candidate box and the target box, v yl Represents the uncertainty value of the network's prediction result of the y coordinate of the upper left corner vertex of the candidate box;

[0157] and b xr Represents the x coordinate of the lower right corner vertex of the candidate box and the target box, v xr Represents the uncertainty value of the network's prediction result of the x-coordinate of the vertex in the lower right corner of the candidate box;

[0158] and b yr Represents the y coordinate of the lower right corner vertex of the candidate box and the target box, v yr Represents the uncertainty value of the network's prediction result of the y coordinate of the vertex in the lower right corner of the candidate box;

[0159] 6.1.2) Corresponding loss function based on the x-coordinate and y-coordinate of the upper left corner of a single rough candidate box and The x-coordinate and y-coordinate of the lower right corner of a single rough candidate box correspond to the loss function and Calculate the loss function L corresponding to a single rough candidate box P :

[0160]

[0161] in express and Any candidate box in, b represents the noise target box bn or the target box br after a correction Ι ;

[0162] 6.1.3) According to the loss function L corresponding to a single rough candidate box P , calculate the total loss L of the uncertainty-aware candidate box optimization module T1 PR :

[0163]

[0164] in, Represents the set of predicted candidate boxes in the first candidate box optimization process The i-th predicted candidate box in, Represents the set of predicted candidate boxes in the second candidate box optimization process The i-th predicted candidate box, bn represents the noise target box, br Ι Represents the target frame after a correction, and N represents the total number of predicted candidate frames;

[0165] 6.2) Calculate the loss function L of the target frame correction module T2 based on complementary information BBR :

[0166] 6.2.1) Calculate the loss L of the target frame correction module T2 based on complementary information when performing the first correction Ι :

[0167] According to the multi-instance score matrix at the first correction Calculate the multi-instance learning score for each category during the first correction

[0168]

[0169] in, represents the multi-instance learning score of the i-th rough candidate box in each category during the first correction;

[0170] According to the multi-instance learning scores of the candidate boxes in each category during the first correction Calculate the loss L of the target frame correction module T2 based on complementary information during the first correction Ι :

[0171]

[0172] Among them, CE represents the cross entropy loss function, represents the unique hot code label of the target, C represents the total number of target categories, express Corresponding to the value on category t, express The corresponding value on category t;

[0173] 6.2.2) Calculate the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ :

[0174] According to the second correction multi-instance score matrix Calculate the multi-instance learning score for each category during the second revision

[0175]

[0176] in, represents the multi-instance learning score of the i-th rough candidate box in each category during the second correction;

[0177] According to the multi-instance learning scores of candidate boxes in each category during the second correction Calculate the loss L of the target frame correction module T2 based on complementary information during the second correction ΙΙ :

[0178]

[0179] in, For label Compared with the multi-instance learning scores of candidate boxes in each category during the first correction The inner product of The mean of , FL is the Focal loss function, S neg represents the score of the classification branch of negative samples in multi-instance learning, c neg Indicates its label;

[0180] 6.2.3) Using the target frame correction module T2 loss L based on complementary information during the first correction Ι And the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ , calculate the total loss L of T2 module BBR :

[0181] L BBR =L Ι +L ΙΙ

[0182] 6.3) Modify the target detection network regression loss and use the target frame br after the secondary correction ΙΙ Instead of the noise target box bn, the target detection network is trained, that is, L r (p * ,bn) becomes L r (p * ,brΙΙ ), where L r is the smooth L1 loss function, p * Represents the candidate box generated by the anchor box in the target detection network;

[0183] 6.4) According to the modified regression loss L r (p * ,br ΙΙ ) Get the overall loss L of the target detection network Det :

[0184] L Det =L r (p * ,br ΙΙ )+L RPN +L Cla

[0185] Among them L RPN and L Cla Respectively represent the loss of the RPN module and the classification loss in the existing target detection network;

[0186] 6.5) According to the result of step 6.1) L PR , the result L of step 6.2) BBR and the result L of step 6.4) Det , calculate the overall loss L for training the noisy target box correction network and target detection network based on uncertainty perception and complementary information:

[0187] L=w1L PR +w2L BBR +w3L Det

[0188] Among them, w1, w1 and w1 represent the module T1 loss L PR , module T2 loss L BBR And the overall loss L of the target detection network Det The weight of .

[0189] The effects of the present invention are further described below in conjunction with simulation experiments.

[0190] 1. Experimental Conditions

[0191] The computer processor used is Intel(R) Xeon(R) Gold 6148 CPU @ 2.4 GHz, and the graphics card is two NVIDIA RTX 3090 GPUs with 24 GB of video memory.

[0192] The operating system is 64-bit Ubuntu 22.04, the simulation is performed in Python, and the deep learning framework PyTorch version 1.12.1 is used.

[0193] The evaluation indicator is mAP50, which is the mean average precision under the 50% intersection-over-union threshold.

[0194] All network training uses the back-propagation algorithm to calculate the gradient of each layer, and uses the stochastic gradient descent algorithm with kinetic energy and weight decay terms to update the network parameters, where the kinetic energy term is 0.9 and the weight decay term is 0.0001.

[0195] The noisy dataset used in the experiment was generated by artificially introducing incorrect annotation information into the annotation results of the public dataset. The training set used the noisy annotations, and the test set used the original annotations of the public dataset.

[0196] The noise target box is generated by randomly moving and scaling the real target box, as shown below:

[0197]

[0198] Among them, [xg l ,yg l ,xg r ,yg r ] represents the true target box, xg l and yg l Represents the x-coordinate and y-coordinate of the upper left corner of the real target box, xg r and yg r Indicates the x-coordinate and y-coordinate of the lower right corner of the real target box; [xn l ,yn l ,xn r ,yn r ] represents the noise target box, xn l Heyn l Respectively represent the x-coordinate and y-coordinate of the upper left corner of the noise target box, xn r Heyn r represents the x-coordinate and y-coordinate of the lower right corner of the noise target box; δ obeys the uniform distribution U(-r d ,r d ), r d is the noise rate.

[0199] In the experiment, the target box noise rate ranges from 20%, 30% and 40%.

[0200] The experiment uses the VOC2007 dataset, a classic dataset for object detection tasks. The VOC2007 dataset contains 9,963 images, with 2,501 images in the training set, 2,510 images in the validation set, and 4,952 images in the test set. A total of 24,640 objects are annotated, with an average of 2.5 objects per image.

[0201] 2. Simulation experiment content and results:

[0202] Simulation 1: Under the above experimental conditions, the method of the present invention and the five existing methods are used to correct the noisy target box in the VOC2007 dataset, and the network performance indicators of these methods under different noise rates are compared, as shown in Table 1.

[0203] Table 1 Performance indicators of the present invention and five existing methods under three different noise rates

[0204]

[0205] The five existing methods in Table 1 are:

[0206] Co-teaching: Chadwick et al. proposed a noisy target box correction method based on the Small-loss criterion and a two-headed network;

[0207] SD-LocNet: A noisy object box correction method proposed by Zhang et al. that uses adaptive sampling to measure target localization scores and then identify reliable targets.

[0208] KLLoss: A method proposed by He et al. to correct noisy target frames by estimating the position uncertainty value of the target frame;

[0209] OAMIL: Liu et al. proposed a method for correcting noisy object bounding boxes based on object-aware multi-instance learning.

[0210] SSD-Det: A noisy target box correction method based on spatial self-distillation proposed by Wu et al.

[0211] As can be seen from Table 1, the noise target frame correction network based on uncertainty perception and complementary information proposed by the method of the present invention achieves the best effect when the noise rate is 20%, 30% and 40%. The mAP50 index is improved by 0.1, 0.7 and 2.9 percentage points respectively compared with the SSD-Det method, which proves the effectiveness of the present invention.

[0212] Simulation 2: Under the above experimental conditions, the method of the present invention is used to correct the noisy target frame in the VOC2007 dataset. The results are as follows: Figure 5 As shown, where:

[0213] Figure 5 (a) is the noise target frame,

[0214] Figure 5 (b) is the correction result of the noise target frame.

[0215] Figure 5(c) is the ground truth annotation of the target box.

[0216] contrast Figure 5 (a) and Figure 5 (b) It can be seen that after the noise target frame is corrected by the method of the present invention, the position accuracy of the target frame is improved;

[0217] contrast Figure 5 (b) and Figure 5 (c) It can be seen that the target frame corrected by the method of the present invention can accurately cover the real position of the target.

Claims

1. A noisy target frame correction method based on uncertainty perception and complementary information, characterized in that: The steps include: (1) Obtain training tensor data and test tensor data: 1a) Read the training set data and test set data of the target detection database respectively; 1b) Set the normalized mean and standard deviation of the three RGB channels; adjust, flip, and normalize the images in the training set in sequence to obtain the tensor data corresponding to the training set images; adjust and normalize the images in the test set in sequence to obtain the tensor data corresponding to the test images; (2) Read the training set annotation file to obtain the noise target box, and generate an initial candidate box set for each noise target box bn: P Ι ={P Ι,1 ,P Ι,2 …P Ι,i …P Ι,N }, where N represents P Ι The number of initial candidate boxes in P Ι,i Indicates P Ι The i-th initial candidate box in , 1≤i≤N; (3) Select an existing image feature extraction network, input the training tensor data into the network, and obtain the bottom-up image pyramid feature X'; (4) Constructing a noise target frame correction network based on uncertainty perception and complementary information: 4a) Establish an uncertainty-aware candidate box optimization module T1, which includes a region of interest alignment network, a fully connected layer, and a regression layer; 4b) Establishing a complementary information-based target box correction module T2, which includes a region of interest alignment network, a fully connected layer, a multi-instance learning network, and a category feature group; 4c) The uncertainty-aware candidate box optimization module T1 and the complementary information-based target box correction module T2 are connected in series to form a noise target box correction network based on uncertainty perception and complementary information; (5) Using image pyramid features X' and initial candidate box set P Ι , based on the noise target frame correction network of uncertainty perception and complementary information, the corrected target frame br ΙΙ : 5a) Based on the image pyramid feature X' and the initial candidate box set P Ι , the uncertainty-aware candidate box optimization module T1 optimizes the initial candidate box set P Ι , get the rough candidate box set: Pr Ι ={Pr Ι,1 ,Pr Ι,2 …Pr Ι,i …Pr Ι,N }, where Pr Ι,i Indicates Pr Ι The i-th rough candidate box in ; 5b) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn and obtains a corrected target frame br Ι ; 5c) Repeat step 5a) to optimize the rough candidate box set The candidate box in the , obtain the refined candidate box set: Pr ΙΙ ={Pr ΙΙ,1 ,Pr ΙΙ,2 …Pr ΙΙ ,i…Pr ΙΙ,N }, where Pr ΙΙ,i Indicates Pr ΙΙ The i-th refined candidate box in; 5d) Repeat step 5b) according to the fine candidate box set Pr ΙΙ Get the final corrected target frame br ΙΙ ; (6) According to the final corrected target frame br ΙΙ , design the overall training loss L.

2. The method according to claim 1, characterized in that : In step 1b), the normalized mean and standard deviation of the three RGB channels are set to [123.675, 116.28, 103.53], and the standard deviation of the three channels are set to [58.395, 57.12, 57.375] respectively. In step 1b), the images of the training set are adjusted, flipped, and normalized in sequence. First, the image size is scaled to 1000×600, then the image is randomly flipped with a probability of 0.5, and finally the image is normalized using the mean standard deviation normalization method to obtain the tensor data corresponding to the images of the training set; In step 1b), the images of the test set are adjusted and normalized in sequence, first scaling the image size to 1000×600, and then normalizing the image using the mean standard deviation normalization method to obtain the tensor data corresponding to the images of the test set.

3. The method according to claim 1, characterized in that In step (2), an initial candidate box set P is generated for each noise target box bn Ι , the implementation steps include the following: 2a) Convert the noise target frame bn from the [xl, yl, xr, yr] format to the [x, y, w, h] format to obtain the converted noise target frame: bn * =[bn x ,bn y ,bn w ,bn h ] Where xl and yl represent the x and y coordinates of the upper left corner of the target box, respectively; xr and yr represent the x and y coordinates of the lower right corner of the target box, respectively; x and y represent the x and y coordinates of the center point of the target box after the format conversion, respectively; w and h represent the width and height of the target box after the format conversion, respectively; 2b) Set relevant parameters: Set the size adjustment parameter s to modify the noise target box bn * The width and height of , set the parameter value range to: s∈{0.7,0.8,1,1.2,1.3}; Set the x coordinate adjustment parameter o x , used to modify the noise target box bn * The x coordinate of the center point; set the y coordinate adjustment parameter o y , used to modify the noise target box bn * The y coordinate of the center point; Set the x and y coordinate adjustment parameters o x and o y The value range of is: (the x ,o y )∈{(0,0),(2,0),(0,2),(-2,0),(-2,-2)}; 2c) According to the noise target box bn * =[bn x ,bn y ,bn w ,bn h ]Generate initial candidate box in, Indicates the first generated initial candidate box in the format of [x, y, w, h]; 2d) Initial candidate box in the format of [x, y, w, h] Convert the [x,y,w,h] format to the [xl,yl,xr,yr] format to get the first initial candidate box P in the [xl,yl,xr,yr] format Ι,1 ; 2e) Repeat steps 2c) and 2d) N times to obtain the initial candidate box set P in the format [xl, yl, xr, yr] Ι .

4. The method according to claim 1, wherein Step 4a) Establish an uncertainty-aware candidate box optimization module T1, where the structural parameters and connection methods of each layer are as follows: The region of interest alignment network has an output feature size of 7×7 and an output dimension of 256. The fully connected layer has an output dimension of 1024. The regression layer is set to two layers, and the output dimension of each layer is 4; The region of interest alignment network, the fully connected layer and the two parallel regression layers are connected in sequence to form an uncertainty-aware candidate box optimization module T1.

5. The method according to claim 1, wherein In step 4b), a target frame correction module T2 based on complementary information is established, wherein the structural parameters and connection methods of each layer are as follows: The region of interest alignment network has an output feature size of 7×7 and an output dimension of 256. The fully connected layer has an output dimension of 1024. The multi-instance learning network includes two parallel classification branches and detection branches. The category feature group ψ includes C groups of target features, where C represents the total number of target categories, and each group of target features includes U target features. The region of interest alignment network and the fully connected layer are connected in series, and then connected in parallel with the multi-instance learning network and the category feature group to construct the target box correction module T2 based on complementary information.

6. The method according to claim 1, wherein Step 5a) Based on the image pyramid feature X' and the initial candidate box set P Ι , the uncertainty-aware candidate box optimization module T1 optimizes the initial candidate box set P Ι , obtain the rough candidate box set Pr Ι , the implementation steps include the following: 5a1) Based on the image pyramid feature X' and the initial candidate box set P Ι , obtain the initial candidate box P through the region of interest alignment network and the fully connected layer Ι The feature set of the target: F Ι ={F Ι,1 ,F Ι,2 …F Ι,i …F Ι,N }, where F Ι,i Indicates F Ι The i-th target feature in ; 5a2) According to the target feature set F Ι , through two parallel regression layers, we obtain the predicted candidate box sets: And the uncertainty value set of the network prediction candidate box: V Ι ={V Ι,1 ,V Ι,2 …V Ι,i …V Ι,N },in express The i-th predicted candidate box, V Ι,i Indicates V Ι The i-th uncertainty value in ; 5a3) According to the uncertainty value set V of the network prediction candidate box Ι Optimize prediction candidate boxes Get the rough candidate box Pr Ι,i ; 5a4) Repeat step 5a3) N times to obtain a rough candidate box set Pr Ι .

7. The method according to claim 6, characterized in that Step 5a3) Optimize the predicted candidate box Get the rough candidate box Pr Ι,i , calculated as follows: 5a3.1) Get the predicted candidate box set Center and predict candidate boxes A set of candidate boxes with IoU greater than 0.5: Pn = {Pn1, Pn2, ... Pn j …Pn A }, where A represents the number of candidate boxes in Pn, Pn j Represents the j-th candidate box in Pn, A≤N, 1≤j≤A; 5a3.2) Based on the uncertainty value set V of the network prediction candidate box Ι , obtain the uncertainty value set corresponding to the candidate box set Pn: Vn={Vn1,Vn2…Vn j …Vn A }, Vn j represents the jth uncertainty value in Vn; 5a3.3) Based on the predicted candidate box Calculate the confidence score set S of the candidate box Pn iou : in, It's S iou The j-th confidence score in Pn is the j-th candidate box Pn in Pn j Confidence score of 5a3.4) Based on the candidate box set Pn and the uncertainty value set Vn corresponding to these candidate boxes and the confidence score set S iou , calculate the rough candidate box Pr Ι,i : in, Represents the rough candidate box Pr Ι,i The x coordinate of the upper left vertex, Represents the candidate box Pn j The x coordinate of the upper left vertex, Represents the candidate box Pn j The confidence score of Represents the network's candidate box Pn j The uncertainty value of the x-coordinate prediction of the upper left vertex; Represents the rough candidate box Pr Ι,i The y coordinate of the upper left vertex, Represents the candidate box Pn j The y coordinate of the upper left vertex, Represents the network's candidate box Pn j The uncertainty value of the y-coordinate prediction of the upper left vertex; Represents the rough candidate box Pr Ι,i The x coordinate of the lower right vertex, Represents the candidate box Pn j The x coordinate of the lower right vertex, Represents the network's candidate box Pn j The uncertainty value of the x-coordinate prediction of the lower right vertex; Represents the rough candidate box Pr Ι,i The y coordinate of the lower right vertex, Represents the candidate box Pn j The y coordinate of the lower right vertex, The network is divided into candidate boxes Pn j The uncertainty value of the predicted y-coordinate of the lower right vertex.

8. The method according to claim 1, characterized in that In step 5b), based on the image pyramid feature X' and the rough candidate box set Pr Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn and obtains a corrected target frame br Ι , the implementation steps include the following: 5b1) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , obtain the rough candidate box set Pr through the region of interest alignment network and the fully connected layer Ι The feature set Fr of the target Ι ={Fr Ι,1 ,Fr Ι,2 …Fr Ι,i …Fr Ι,N }, where Fr Ι,i It is Fr Ι The i-th target feature in ; 5b2) According to the target feature set Fr Ι , through the multi-instance learning network, obtain the multi-instance score matrix Calculate the rough candidate box set Pr Ι Multi-instance score set: S Ι ={S Ι,1 ,S Ι,2 …S Ι,i …S Ι,N }, where S Ι,i It's S Ι The i-th multi-instance score in ; 5b3) According to the rough candidate box set Pr Ι The feature Fr of the target Ι Update the category feature group ψ; 5b4) According to the category feature set ψ and the target feature set Fr Ι , calculate the rough candidate box set Pr Ι Similarity score set of: Sim Ι ={Sim Ι,1 ,Sim Ι,2 …Sim Ι,i …Sim Ι,N }, where Sim Ι,i It's Sim Ι The i-th similarity score in ; 5b5) Based on the multi-instance score set S Ι and similarity score set Sim Ι Calculate the rough candidate box set Pr Ι The final score set of Sr Ι ={Sr Ι,1 ,Sr Ι,2 …Sr I,i …Sr Ι,N }, Among them, Sr Ι,i =α*S Ι,i +β*Sim Ι,i is the final score set Sr Ι The final score of the i-th; α and β are combination coefficients, α + β = 1; S Ι,i is the multi-instance score set S Ι The i-th multi-instance score in Sim Ι,i is the similarity score set Sim Ι The i-th similarity score in ; 5b6) According to the rough candidate box set Pr Ι and its final score set Sr Ι Calculate the corrected target frame br Ι : 5b6.1) According to the final score set Sr Ι And the rough candidate box set Pr Ι Get the highest H score sets S top ={S top,1 ,S top,2 …S top,k …S top,H } and the corresponding candidate box set P top ={P top,1 ,P top,2 …P top,k …P top,H }, where P top,k It's P top The kth candidate box in S top,k It's S top The k-th score in , 1≤k≤H; 5b6.2) According to the score set S top And the candidate box set P top Calculate the corrected target frame br Ι : br Ι =[br xl ,br yl ,br xr ,br yr ] in, is the target frame after a correction Ι The x coordinate of the upper left corner, S top,k is the score set S top The kth score, P top,k Is the candidate box set P top The kth candidate box in Is the candidate box P top,k The x coordinate of the upper left vertex; is the target frame after a correction Ι The y coordinate of the upper left vertex, Is the candidate box P top,k The y coordinate of the upper left vertex; is the target frame after a correction Ι The x coordinate of the lower right vertex, Is the candidate box P top,k The x coordinate of the lower right vertex; is the target frame after a correction Ι The y coordinate of the lower right vertex, Is the candidate box P top,k The y coordinate of the lower right vertex.

9. The method according to claim 8, characterized in that: In step 5b3), according to the rough candidate box set Pr Ι The feature Fr of the target Ι Update the category feature group ψ: 5b3.1) Obtain the multi-instance score set Sr Ι The maximum value of Sr Ι,max , obtain the rough candidate box set Pr Ι The rough candidate box with the largest multi-instance score is obtained, and the feature Fr of the target contained in the rough candidate box is obtained. Ι,max ; 5b3.2) Set the rough candidate box set Pr Ι The target category is yc, and the category feature group corresponding to category yc is obtained in Represents ψ yc The u-th feature in , 1≤u≤U, U represents the category feature group ψ yc The number of features in the 5b3.3) Calculate the feature Fr of the rough candidate box with the maximum multi-instance score Ι,max and the category feature group ψ yc Similarity of features in sm m ={sm m,1 ,sm m,2 …sm m,u …sm m,U }, in It is Fr Ι,max With ψ yc The u-th feature in The similarity of sm m The u-th similarity in , < > represents the inner product operation, || || represents the two-norm operation; 5b3.4) According to similarity sm m , from ψ yc Selection and feature Fr Ι,max Most similar features 5b3.5) Using feature Fr Ι,max and multi-instance score Sr Ι,max Update the category feature group ψ yc Features in Where r is the momentum coefficient; 5b3.6) Change the noise target box bn and the target category yc it contains, repeat steps 5b3.1) to 5b3.5) for a total of Z times to complete the update of the category feature group ψ, where Z is the total number of targets in the training set.

10. The method according to claim 8, characterized in that: In step 5b4), according to the category feature set ψ and the target feature set Fr Ι , calculate the rough candidate box set Pr Ι Similarity score set Sim Ι , the implementation steps include the following: 5b4.1) Get Pr Ι The i-th rough candidate box Pr Ι,i and its feature Fr containing the target Ι,i ; 5b4.2) Calculate Pr Ι The i-th rough candidate box Pr Ι,i Contains the target feature Fr Ι,i The category feature group ψ corresponding to category yc yc The similarity of all features in , to obtain the similarity sm i ={sm i,1 ,sm i,2 …sm i,u …sm i,U }, in It is Fr Ι,i With ψ yc The u-th feature in The similarity of sm i The u-th similarity in ; 5b4.3) Get similarity sm i The maximum value in is the rough candidate box Pr Ι,i Similarity score Sim Ι,i ; 5b4.4) Repeat steps 5b4.1) to 5b4.3) N times to obtain the similarity score set Sim Ι .

11. The method according to claim 1, wherein In step (6), the final corrected target frame br ΙΙ , design the overall training loss L, which is implemented as follows: 6a) Design the loss L of the uncertainty-aware candidate box optimization module T1 PR : in, Represents the set of predicted candidate boxes in the first candidate box optimization process The i-th predicted candidate box in, Represents the set of predicted candidate boxes in the second candidate box optimization process The i-th predicted candidate box, bn represents the noise target box, br Ι Represents the target frame after a correction, and N represents the total number of predicted candidate frames; is the loss function corresponding to a single candidate box, express and Any candidate box in, b represents the noise target box bn or the target box br after a correction Ι , Is to calculate the candidate box And the x-coordinate of the upper left corner of the target box b corresponds to the loss function, and b xl Represent candidate boxes respectively and the x coordinate of the upper left corner vertex of the target box b, v xl Represents the network's candidate box The uncertainty value of the x-coordinate prediction result of the upper left vertex; Is to calculate the candidate box And the y coordinate of the upper left corner of the target box b corresponds to the loss function, and b yl Represent candidate boxes respectively and the y coordinate of the upper left corner vertex of the target box b, v yl Represents the network's candidate box The uncertainty value of the y-coordinate prediction result of the upper left vertex; Is to calculate the candidate box And the x-coordinate of the lower right corner of the target box b corresponds to the loss function, and b xr Represent candidate boxes respectively and the x coordinate of the lower right corner vertex of the target box b, v xr Represents the network's candidate box The uncertainty value of the x-coordinate prediction result of the lower right vertex; Is to calculate the candidate box And the y coordinate of the lower right corner of the target box b corresponds to the loss function, and b yr Represent candidate boxes respectively and the y coordinate of the lower right corner vertex of the target box b, v yr Represents the network's candidate box The uncertainty value of the y-coordinate prediction result of the lower right vertex; 6b) Design the loss L of the target box correction module T2 based on complementary information BBR : 6b1) Calculate the loss L of the target frame correction module T2 based on complementary information when performing the first correction Ι : According to the multi-instance score matrix at the first correction Calculate the multi-instance learning score for each category during the first correction in, represents the multi-instance learning score of the i-th rough candidate box in each category during the first correction; According to the multi-instance learning scores of the candidate boxes in each category during the first correction Calculate the loss L of the target frame correction module T2 based on complementary information during the first correction Ι : Among them, CE represents the cross entropy loss function, represents the unique hot code label of the target, C represents the total number of target categories, express Corresponding to the value on category t, express The corresponding value on category t; 6b2) Calculate the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ : According to the second correction multi-instance score matrix Calculate the multi-instance learning score for each category during the second revision in, represents the multi-instance learning score of the i-th rough candidate box in each category during the second correction; According to the multi-instance learning scores of candidate boxes in each category during the second correction Calculate the loss L of the target frame correction module T2 based on complementary information during the second correction ΙΙ : in, For label Compared with the multi-instance learning scores of candidate boxes in each category during the first correction The inner product of The mean of , FL is the Focal loss function, S neg represents the score of the classification branch of negative samples in multi-instance learning, c neg Indicates its label; 6b3) Using the loss L of the target frame correction module T2 based on complementary information during the first correction Ι And the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ , calculate the total loss L of T2 module BBR : L BBR =L Ι +L ΙΙ 6c) Modify the target detection network regression loss and use the target frame br after the secondary correction ΙΙ Instead of the noise target box bn, the target detection network is trained, that is, L r (p * ,bn) becomes L r (p * ,br ΙΙ ), where L r is the smooth L1 loss function, p * Represents the candidate box generated by the anchor box in the target detection network; 6d) According to the updated regression loss L r (p * ,br ΙΙ ) Get the overall loss L of the target detection network Det : L Det =L r (p * ,br ΙΙ )+L RPN +L Cla Among them L RPN and L Cla Respectively represent the loss of the RPN module and the classification loss in the existing target detection network; 6e) Based on the result L of step 6a) PR , the result L of step 6a) BBR and the result L of step 6d) Det , calculate the overall loss L for training the noisy target box correction network and target detection network based on uncertainty perception and complementary information: L=w1L PR +w2L BBR +w3L Det Among them, w1, w1 and w1 represent the module T1 loss L PR , module T2 loss L BBR And the overall loss L of the target detection network Det The weight of .

Images