Noise target frame correction method based on uncertainty perception and complementary information

Abstract

The invention discloses a noise target frame correction method based on uncertainty perception and complementary information. The problem that an existing noise target frame correction technology is low in accuracy is mainly solved. According to the implementation scheme, the method comprises the steps of obtaining image tensor data; extracting image pyramid features; generating an initial candidate frame set according to the noise target frame; constructing a candidate frame optimization module based on uncertainty perception and a target frame correction module based on complementary information, and constructing a noise target frame correction network based on uncertainty perception and complementary information according to the two modules; inputting the image pyramid features and the initial candidate frame set into a target frame correction network, and sequentially correcting a noise target frame once through a candidate frame optimization module and a target frame correction module to obtain a target frame after primary correction; and repeating the process of correcting the noise target frame once to obtain a final corrected target frame, and finishing the correction of the noise target frame. According to the method, the correction effect on the noise target frame is remarkably improved, and the method can be used for obtaining a high-quality labeled data set.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and further designs a noise target frame correction method, which can be used to obtain a high-quality labeled data set. Background Art

[0002] As object detection technology is widely used in various fields, it has developed rapidly. As one of the key elements of artificial intelligence, data plays a vital role in the rapid development of object detection technology. However, manually constructing a dataset is a complex and time-consuming task, and the dataset obtained by combining automatic labeling technology and crawler technology will have problems such as noisy target boxes. Moreover, even manually labeled datasets will inevitably contain noisy target boxes. Noisy target boxes will cause the network to learn incorrect knowledge, affect the accuracy of the model, and have a negative impact on the network's performance.

[0003] With the continuous advancement of technology, more and more fields and industries have begun to realize the importance of data, and have designed relevant algorithms to correct the noisy target boxes in the data set and improve the robustness of the target detection network to noisy target boxes. Wu et al. designed the SSD-Det method to correct the noisy target boxes during network training, and used the corrected target boxes to supervise network training to reduce the impact of the noisy target boxes on network performance. Although this method has achieved good results in correcting noisy target boxes, since it uses multi-instance learning scores to screen candidate boxes, the corrected target boxes obtained have the following two problems:

[0004] (1) Target frame offset problem: When there are multiple targets in an image, the corrected target frame corresponding to a target may overlap another target. This is because multi-instance learning only considers target category information. When the target categories are the same, the selected candidate frames may offset the corresponding targets and overlap another target, resulting in the corrected target frame having a target frame offset problem.

[0005] (2) The problem of only covering the local area of ​​the target, that is, the corrected target box is not accurate and only contains the local area of ​​the target in the image. This is because multi-instance learning tends to "focus" on the salient information of the target, so that the candidate box selected based on the multi-instance learning score may only cover the local area of ​​the target.

[0006] Patent document with application number CN114549863A discloses a light field salient target detection method based on pixel-level noise label supervision, which guides and assists the detection of salient objects by mining rich scene information in high-dimensional light field data, making full use of the salient information available in the noise label and reducing the confidence of the noise information, effectively reducing the pre-experimental cost of salient target detection. Since this method does not take into account the problem of noisy target frames in the data set, it cannot correct the target frame, that is, when there is a noisy target frame in the data set, the network trained by this method does not perform well. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a noisy target frame correction method based on uncertainty perception and complementary information to improve the correction quality of the noisy target frame and thereby improve the detection accuracy of the target detection network.

[0008] The technical idea for achieving the purpose of the present invention is: by constructing a category feature group that can be updated online, target features across images are obtained, complementary information is provided for the target in the current image, and the problem of only covering the local area of ​​the target is alleviated; by introducing an additional regression branch, the uncertainty value of the network prediction candidate box is obtained and the prediction candidate box is optimized based on the uncertainty value, the problem of target box offset is alleviated, and the correction quality of the noisy target box is improved; by improving the correction quality of the noisy target box, the detection accuracy of the target detection network for the target is improved.

[0009] According to the above technical ideas, the technical solution of the present invention includes the following steps:

[0010] (1) Obtain training tensor data and test tensor data:

[0011] 1a) Read the training set data and test set data of the target detection database respectively;

[0012] 1b) Set the normalized mean and standard deviation of the three RGB channels; adjust, flip and normalize the images of the training set in turn to obtain the tensor data corresponding to the training set images; adjust and normalize the images of the test set in turn to obtain the tensor data corresponding to the test images;

[0013] (2) Read the training set annotation file to obtain the noisy target box, and generate an initial candidate box set for each noisy target box bn: Ι = {P Ι,1 ,P Ι,2 …P Ι,i …P Ι,N}, where N represents P Ι The number of initial candidate boxes in P Ι,i Indicates P ΙThe i-th initial candidate box in , 1≤i≤N;

[0014] (3) Select an existing image feature extraction network, input the training tensor data into the network, and obtain a bottom-up image pyramid feature X';

[0015] (4) Constructing a noise target frame correction network based on uncertainty perception and complementary information:

[0016] 4a) Establishing an uncertainty-aware candidate box optimization module T1 including a region of interest alignment network, a fully connected layer, and a regression layer;

[0017] 4b) establishing a target box correction module T2 based on complementary information, which includes a region of interest alignment network, a fully connected layer, a multiple instance learning network and a category feature group;

[0018] 4c) The candidate frame optimization module T1 based on uncertainty perception and the target frame correction module T2 based on complementary information are connected in series to form a noise target frame correction network based on uncertainty perception and complementary information;

[0019] (5) Using the image pyramid feature X' and the initial candidate box set P Ι , based on uncertainty perception and complementary information, the noise target frame correction network is used to obtain the corrected target frame br ΙΙ :

[0020] 5a) Based on the image pyramid feature X' and the initial candidate box set P Ι , the uncertainty-aware candidate box optimization module T1 optimizes the initial candidate box set P Ι , get the rough candidate box set: Pr Ι = {Pr Ι,1 ,Pr Ι,2 …Pr Ι,i …Pr Ι,N}, where Pr Ι,i Indicates Pr Ι The i-th rough candidate box in ;

[0021] 5b) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn to obtain a corrected target frame br Ι ;

[0022] 5c) Repeat step 5a) to optimize the rough candidate box set The candidate box in the box is obtained to obtain a set of refined candidate boxes: Pr ΙΙ = {Pr ΙΙ,1 ,Pr ΙΙ,2 …Pr ΙΙ,i…Pr ΙΙ,N}, where Pr ΙΙ,i Indicates Pr ΙΙ The i-th refined candidate box in;

[0023] 5d) Repeat step 5b) according to the fine candidate box set Pr ΙΙ Get the final corrected target frame br ΙΙ ;

[0024] (6) According to the final corrected target frame br ΙΙ , design the overall training loss L.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] First, the present invention optimizes the candidate frame in the constructed uncertainty-aware candidate frame optimization module, provides a higher-quality candidate frame for subsequent correction of the noisy target frame, and improves the position accuracy of the corrected target frame.

[0027] Secondly, the present invention designs an online updateable category feature group in the target frame correction module based on complementary information, obtains target features across images, provides complementary information for the target in the current image to guide the correction of the noisy target frame, and improves the correction effect of the noisy target frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the implementation of the present invention;

[0029] Figure 2 It is a schematic diagram of a noise target frame correction network based on uncertainty perception and complementary information in the present invention;

[0030] Figure 3 yes Figure 2 Schematic diagram of the candidate box optimization module based on uncertainty perception;

[0031] Figure 4 yes Figure 2 Schematic diagram of the target frame correction module based on complementary information;

[0032] Figure 5 This is an effect diagram of correcting a noise target frame using the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme and effects in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the present invention, not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without creative work should all belong to the protection scope of the present invention.

[0034] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the sequence of the step numbers is not limited.

[0035] Reference Figure 1 , the implementation steps of this embodiment are as follows:

[0036] Step 1: Get training tensor data and test tensor data.

[0037] 1.1) Read the training set data and test set data of the target detection database respectively;

[0038] 1.2) Set the normalized mean and standard deviation of the three RGB channels:

[0039] 1.2.1) Set the normalized mean values ​​of the three RGB channels to [123.675, 116.28, 103.53] respectively;

[0040] 1.2.2) Set the standard deviations of the three channels to [58.395, 57.12, 57.375] respectively;

[0041] 1.3) The images of the training set are adjusted, flipped and normalized in turn to obtain the tensor data corresponding to the images of the training set:

[0042] 1.3.1) Scale the input image to 1000×600;

[0043] 1.3.2) Randomly flip the scaled image with a probability of 0.5 to obtain the flipped image:

[0044] 1.3.3) The image is processed by the mean standard deviation normalization method to obtain the tensor data corresponding to the training image at this stage;

[0045] 1.4) The images of the test set are adjusted and normalized in turn to obtain the tensor data corresponding to the training set images:

[0046] 1.4.1) Scale the input image to 1000×600;

[0047] 1.4.2) The image is processed using the mean standard deviation normalization method to obtain the tensor data corresponding to the test image in this stage.

[0048] Step 2: Read the training set annotation file to obtain the noisy target box, and generate an initial candidate box set P for each noisy target box bn Ι .

[0049] 2.1) Convert the noise target frame bn from [xl, yl, xr, yr] format to [x, y, w, h] format to obtain the noise target frame after conversion:

[0050] bn * =[bn x ,bn y ,bn w ,bn h ]

[0051] Where xl and yl represent the x and y coordinates of the upper left corner vertex of the target box, respectively, and xr and yr represent the x and y coordinates of the lower right corner vertex of the target box, respectively; x and y represent the x and y coordinates of the center point of the target box after the format conversion, respectively, and w and h represent the width and height of the target box after the format conversion;

[0052] 2.2) Set relevant parameters:

[0053] Set the resizing parameter s to modify the noise target box bn * The width and height of the parameter are set to the range of s∈{0.7,0.8,1,1.2,1.3};

[0054] Set the x coordinate adjustment parameter o x , used to modify the noise target box bn * The x coordinate of the center point; set the y coordinate adjustment parameter o y , used to modify the noise target box bn * The y coordinate of the center point;

[0055] Set x and y coordinate adjustment parameters o x and y The value range of is:

[0056] (o x ,o y )∈{(0,0),(2,0),(0,2),(-2,0),(-2,-2)}

[0057] 2.3) According to the noise target box bn * =[bn x ,bn y ,bn w ,bn h] Generate initial candidate box

[0058]

[0059] in, Indicates the first generated initial candidate box in the format of [x, y, w, h];

[0060] 2.4) The initial candidate box in the format of [x, y, w, h] Convert the [x, y, w, h] format to the [xl, yl, xr, yr] format to get the first initial candidate box P in the [xl, yl, xr, yr] format Ι,1 ;

[0061] 2.5) Repeat steps 2.3) and 2.4) N times to obtain the initial candidate box set P in the format of [xl, yl, xr, yr] Ι .

[0062] Step 3: Extract the training image pyramid feature X' based on the existing image feature extraction network.

[0063] 3.1) Establish image feature extraction network:

[0064] This example uses but is not limited to the ResNet50 network and the feature pyramid network as the image feature extraction network;

[0065] The ResNet50 network includes 5 concatenated convolutional layers, each of which has the same structure and includes several convolution groups; each convolution group includes a convolution operation, a batch normalization layer and a ReLu activation function;

[0066] The feature pyramid network includes 4 convolution layers with a convolution kernel size of 1×1 and a stride of 1 and 4 convolution layers with a convolution kernel size of 3×3 and a stride of 1;

[0067] 3.2) Extract the feature C of the training image tensor data through the ResNet50 network:

[0068] The training image tensor data preprocessed in step 1 is input into the ResNet50 network for feature extraction. Image features C1, C2, C3, C4 and C5 are extracted respectively through 5 serial convolutional layers. These image features are used to construct the image feature set from bottom to top: C = {C1, C2, C3, C4, C5};

[0069] 3.3) According to the image feature set C extracted by the ResNet50 network, the image pyramid feature X' is obtained through the feature pyramid network:

[0070] 3.3.1) The image features C2, C3, C4 and C5 in the image feature set C are respectively subjected to 4 convolutional layers with a convolution kernel size of 1×1 and a stride of 1, so that the number of channels of the C2 feature remains at 256, the number of channels of the C3 feature decreases from 512 to 256, the number of channels of the C4 feature decreases from 1024 to 256, and the number of channels of the C5 feature decreases from 2048 to 256;

[0071] 3.3.2) Record the highest level feature C5 in the image feature set C as the highest level feature C5' in the fused image feature set C'; perform a 2-fold upsampling operation on C5' and add it to the second highest level feature C4 in the image feature set C to obtain the second highest level feature C4' in the fused image feature set C'; perform a 2-fold upsampling operation on C4' and add it to the second lowest level feature C3 in the image feature set C to obtain the second lowest level feature C3' in the fused image feature set C'; perform a 2-fold upsampling operation on C3' and add it to the lowest level feature C2 in the image feature set C to obtain the lowest level feature C2' in the fused image feature set C';

[0072] 3.3.3) Using the above C5', C4', C3' and C2' features, construct the fused image feature set C' = {C2', C3', C4', C5'} from bottom to top;

[0073] 3.3.4) The features in the fused image feature set C' are respectively passed through 4 convolutional layers with a kernel size of 3×3 and a step size of 1 to obtain features X2′, X3′, X4′ and X5′. The feature X5′ is downsampled to obtain feature X6′. The image pyramid features are constructed from the bottom up using the features X2′, X3′, X4′, X5′ and X6′ to obtain the image pyramid features: X'={X2',X3',X4',X5',X6'}.

[0074] Step 4: Construct a noise target box correction network based on uncertainty perception and complementary information.

[0075] Reference Figure 2 , the specific implementation of this step is as follows:

[0076] 4.1) Establishing the candidate box optimization module T1 based on uncertainty perception:

[0077] 4.1.1) The output feature size of 7×7 and the output dimension of 256 are selected as the region of interest alignment network, which is used to sample the target features from the image pyramid feature X' and perform feature alignment by interpolation;

[0078] 4.1.2) A fully connected layer with an output dimension of 1024 is selected to reduce the dimension of the target features;

[0079] 4.1.3) Select two regression layers with 4 output dimensions, which are used to calculate the uncertainty values ​​of the predicted candidate boxes and the network predicted candidate boxes respectively;

[0080] 4.1.4) Connect the region of interest alignment network, the fully connected layer and the two parallel regression layers in sequence to establish the uncertainty-aware candidate box optimization module T1;

[0081] 4.2) Construct target frame correction module T2 based on complementary information:

[0082] 4.2.1) The output feature size of 7×7 and the output dimension of 256 are selected as the region of interest alignment network, which is used to sample the target features from the image pyramid feature X' and perform feature alignment by interpolation;

[0083] 4.2.2) A fully connected layer with an output dimension of 1024 is selected to perform dimensionality reduction of target features;

[0084] 4.2.3) A multi-instance learning network consisting of two parallel classification branches and detection branches is selected to calculate the multi-instance scores of the candidate boxes;

[0085] 4.2.4) Design a category feature group ψ containing C groups of target features, each group of target features containing U target features, where C represents the total number of target categories.

[0086] 4.2.5) The region of interest alignment network and the fully connected layer are connected in series, and then connected in sequence with the parallel-connected multi-instance learning network and the category feature group to construct a target box correction module T2 based on complementary information;

[0087] 4.3) The candidate box optimization module T1 based on uncertainty perception and the target box correction module T2 based on complementary information are connected in series to form a noisy target box correction network based on uncertainty perception and complementary information.

[0088] Step 5: Use the image pyramid feature X' and the initial candidate box set P Ι , based on uncertainty perception and complementary information, the noise target frame correction network is used to obtain the corrected target frame br ΙΙ :

[0089] 5.1) Optimize the initial candidate box set P using the uncertainty-aware candidate box optimization module T1 Ι , get the rough candidate box set Pr Ι :

[0090] Reference Figure 3 , the specific implementation of this step is as follows:

[0091] 5.1.1) Based on the initial candidate box set P obtained in step 2Ι The image pyramid feature X' obtained in step 3 is used to obtain the initial candidate box P through the region of interest alignment network and the fully connected layer. Ι The feature set of the target in: F Ι ={F Ι,1 ,F Ι,2 …F Ι,i …F Ι,N}, where F Ι,i Indicates F Ι The i-th target feature in ;

[0092] 5.1.2) According to the target feature set F Ι , through two parallel regression layers, the predicted candidate box sets are obtained respectively: And the uncertainty value set of the network prediction candidate box: V Ι = {V Ι,1 ,V Ι,2 …V Ι,i …V Ι,N},in express The i-th predicted candidate box in V Ι,i Indicates V Ι The i-th uncertainty value in ;

[0093] 5.1.3) Based on the uncertainty value set V of the network prediction candidate box Ι Optimize the prediction candidate box set Get the rough candidate box set Pr Ι :

[0094] 5.1.3.1) For the predicted candidate box set The i-th predicted candidate box in Get the predicted candidate box set Center and predict candidate boxes A series of candidate boxes with IoU greater than 0.5: Pn = {Pn1, Pn2, ... Pn j …Pn A}, where A represents the number of candidate boxes in Pn, Pn j represents the j-th candidate box in Pn, A≤N, 1≤j≤A;

[0095] 5.1.3.2) Based on the uncertainty value set V of the network prediction candidate box Ι , obtain the uncertainty value set corresponding to the candidate box set Pn: Vn = {Vn1, Vn2…Vn j …Vn A},Vn j represents the jth uncertainty value in Vn;

[0096] 5.1.3.3) Based on the predicted candidate box Calculate the confidence score set S of the candidate box Pn iou :

[0097]

[0098] in, YesS iou The j-th confidence score in Pn is the j-th candidate box Pn in Pn j Confidence score of

[0099] 5.1.3.4) Based on the candidate box set Pn and the uncertainty value set Vn corresponding to these candidate boxes and the confidence score set S iou , calculate the rough candidate box Pr Ι,i The x coordinate of the upper left vertex and the y coordinate The x coordinate of the lower right vertex and the y coordinate

[0100]

[0101] in, Represents the candidate box Pn j The confidence score of and Represent the candidate boxes Pn respectively j The x- and y-coordinates of the upper left vertex, and Represent the candidate boxes Pn respectively j The x- and y-coordinates of the lower right vertex;

[0102] and Respectively represent the network for the candidate box Pn j The uncertainty values ​​of the x- and y-coordinates of the upper left vertex are predicted. and Respectively represent the network for the candidate box Pn j The uncertainty values ​​of the x- and y-coordinate predictions of the lower right vertex;

[0103] 5.1.3.5) Based on the rough candidate box Pr Ι,i The x coordinate of the upper left vertex and the y coordinate The x coordinate of the lower right vertex and the y coordinate Calculate the rough candidate box Pr Ι,i :

[0104]

[0105] 5.1.3.6) Repeat steps 5.1.3.1) to 5.1.3.5) N times to obtain a rough candidate box set

[0106] 5.2) Based on the image pyramid feature X' obtained in step 3 and the rough candidate box set Pr obtained in step 5 Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn to obtain a corrected target frame br Ι :

[0107] Reference Figure 4 , the specific implementation of this step is as follows:

[0108] 5.2.1) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , obtain the rough candidate box set Pr through the region of interest alignment network and the fully connected layer Ι The feature set Fr of the target Ι = {Fr Ι,1 ,Fr Ι,2 …Fr Ι,i …Fr Ι,N}, where Fr Ι,i It is Fr Ι The i-th target feature in ;

[0109] 5.2.2) According to the target feature set Fr Ι , through the multi-instance learning network, calculate the rough candidate box set Pr Ι The multi-instance score set: S Ι ={S Ι,1 ,S Ι,2 …S Ι,i …S Ι,N}, where S Ι,i YesS Ι The i-th multi-instance score in :

[0110] 5.2.2.1) Set the rough candidate box set Pr Ι The feature set Fr of the target Ι Send them to the classification branch and detection branch in the multi-instance learning network to obtain the matrix x cls and x det , for the matrix x cls Normalize to get the matrix

[0111]

[0112] in, Representation Matrix The data in row i and column j in Represents the matrix x cls The data in row i and column j in Represents the matrix x cls The data in the i-th row and m-th column;

[0113] 5.2.2.2) For the matrix x det Normalize and obtain the normalized matrix

[0114]

[0115] in, Representation Matrix The data in row i and column j in Represents the matrix x det The data in row i and column j in Represents the matrix x det The data in the i-th row and m-th column;

[0116] 5.2.2.3) Normalized classification branch matrix and the detection branch matrix Perform inner product operation to obtain multi-instance learning matrix

[0117]

[0118] 5.2.2.4) For the rough candidate box set Pr Ι The i-th rough candidate box Pr Ι,i , take the multi-instance learning matrix In the i-th row of data Get data The value corresponding to the category yc is the rough candidate box Pr Ι,i The multi-instance score S Ι,i , yc is the rough candidate box Pr Ι,i The category that contains the target;

[0119] 5.2.2.5) Repeat step 5.2.2.4) N times to get the rough candidate box set Pr Ι The multi-instance score set S Ι ;

[0120] 5.2.3) According to the rough candidate box set Pr Ι The characteristic Fr of the target Ι Update the category feature group ψ:

[0121] 5.2.3.1) Obtaining the multi-instance score set Sr Ι The maximum value of Sr Ι,max , get the rough candidate box set PrΙ The rough candidate box with the largest multi-instance score is obtained, and the feature Fr of the target contained in the rough candidate box is obtained. Ι,max ;

[0122] 5.2.3.2) Set the rough candidate box set Pr Ι The category of the target is yc, and the category feature group corresponding to category yc is obtained in Represents ψ yc The u-th feature in , 1≤u≤U, U represents the category feature group ψ yc The number of features in

[0123] 5.2.3.3) Calculate the feature Fr of the rough candidate box with the maximum multi-instance score Ι,max and the class feature group ψ yc Similarity of features in sm m ={sm m,1 ,sm m,2 …sm m,u …sm m,U},

[0124] in It is Fr Ι,max With ψ yc The uth feature in The similarity of sm m The u-th similarity in , <> represents the inner product operation, and |||| represents the bi-norm operation;

[0125] 5.2.3.4) According to similarity sm m , from ψ yc Selection and feature Fr Ι,max Most similar features

[0126] 5.2.3.5) Using feature Fr Ι,max and the multi-instance score Sr Ι,max Update the class feature set ψ yc Features in

[0127]

[0128] Where r is the momentum coefficient;

[0129] 5.2.3.6) Change the noise target box bn and the target category yc it contains, repeat steps 5.2.3.1) to 5.2.3.5) for a total of Z times to complete the update of the category feature group ψ, where Z represents the total number of targets in the training set.

[0130] 5.2.4) According to the category feature set ψ and the target feature set FrΙ , calculate the rough candidate box set Pr Ι Similarity score set: Sim Ι = {Sim Ι,1 ,Sim Ι,2 …Sim Ι,i …Sim Ι,N}, where Sim Ι,i It's Sim Ι The i-th similarity score in ;

[0131] 5.2.4.1) Get Pr Ι The i-th rough candidate box Pr Ι,i and its feature Fr containing the target Ι,i , and obtain the rough candidate box Pr Ι,i Contains the target feature Fr Ι,i The category feature group ψ corresponding to the category yc yc ;

[0132] 5.2.4.2) Calculate the characteristic Fr Ι,i and the class feature group ψ yc The uth feature in Similarity sm i,u :

[0133]

[0134] 5.2.4.3) Repeat step 5.2.4.2) for U times to obtain the feature Fr Ι,i and the class feature group ψ yc Similarity of all features in sm i ={sm i,1 ,sm i,2 …sm i,u …sm i,U};

[0135] 5.2.4.4) Get similarity sm i The maximum value among them is the rough candidate box Pr Ι,i The similarity score Sim Ι,i ;

[0136] 5.2.4.5) Repeat steps 5.2.4.1) to 5.2.4.4) N times to obtain the similarity score set Sim Ι .

[0137] 5.2.5) Based on the multi-instance score set S Ι and similarity score set Sim Ι Calculate the rough candidate box set Pr Ι The final score of Sr Ι= {Sr Ι,1 ,Sr Ι,2 …Sr Ι,i …Sr Ι,N}:

[0138] 5.2.5.1) According to S Ι The i-th multi-instance score S Ι,i and similarity score set Sim Ι The i-th similarity score Sim Ι,i , calculate the final score set Sr Ι The final score Sr of the i-th Ι,i :

[0139] Sr Ι,i =α*S Ι,i +β*Sim Ι,i

[0140] Among them, α and β are combination coefficients, α+β=1;

[0141] 5.2.5.2) Repeat step 5.2.5.1) N times to obtain the final score set Sr Ι ;

[0142] 5.2.6) According to the rough candidate box set Pr Ι and its final score set Sr Ι Calculate the corrected target frame br Ι :

[0143] 5.2.6.1) According to the final score set Sr Ι And the rough candidate box set Pr Ι Get the highest H score sets S top ={S top,1 ,S top,2 ...S top,k …S top,H} and the corresponding candidate box set P top = {P top,1 ,P top,2 …P top,k …P top,H}, where P top,k YesP top The kth candidate box in S top,k YesS top The kth score in , 1≤k≤H;

[0144] 5.2.6.2) According to the score set S top And the candidate box set P top Calculate the corrected target frame br Ι The x coordinate of the upper left vertex and the y coordinate The x coordinate of the lower right vertex and the y coordinate

[0145]

[0146] Among them, S top,k is the score set S top The kth score, P top,k is the candidate box set P top The kth candidate box in and They are candidate boxes P top,k The x- and y-coordinates of the upper left vertex, and They are candidate boxes P top,k The x and y coordinates of the lower right vertex;

[0147] 5.2.6.3) According to the corrected target frame br Ι The x coordinate of the upper left vertex and the y coordinate The x coordinate of the lower right vertex and the y coordinate Calculate the corrected target frame br Ι :

[0148] br Ι =[br xl ,br yl ,br xr ,br yr ].

[0149] 5.3) Repeat step 5.1) to optimize the rough candidate box set The candidate box in the box is obtained to obtain a set of refined candidate boxes: Pr ΙΙ = {Pr ΙΙ,1 ,Pr ΙΙ,2 …Pr ΙΙ,i …Pr ΙΙ,N}, where Pr ΙΙ,i Indicates Pr ΙΙ The i-th refined candidate box in .

[0150] 5.4) Repeat step 5.2) and calculate the fine candidate box set Pr ΙΙ Get the final corrected target frame br ΙΙ .

[0151] Step 6: According to the final corrected target frame br ΙΙ , design the overall training loss L.

[0152] 6.1) Calculate the total loss L of the uncertainty-aware candidate box optimization module T1 PR :

[0153] 6.1.1) Calculate the x-coordinate and y-coordinate corresponding loss function of the upper left corner of a single rough candidate box and And the x-coordinate and y-coordinate corresponding loss function of the lower right corner of a single rough candidate box and

[0154]

[0155] in, and b xl Represents the x coordinate of the upper left corner vertex of the candidate box and the target box, v xl Represents the uncertainty value of the network's prediction result of the x-coordinate of the upper left corner vertex of the candidate box;

[0156] and b yl Represents the y coordinate of the upper left corner vertex of the candidate box and the target box, v yl Represents the uncertainty value of the network's prediction result of the y coordinate of the upper left corner vertex of the candidate box;

[0157] and b xr Represents the x coordinate of the lower right corner vertex of the candidate box and the target box, v xr Represents the uncertainty value of the network's prediction result of the x-coordinate of the vertex in the lower right corner of the candidate box;

[0158] and b yr Represents the y coordinate of the lower right corner vertex of the candidate box and the target box, v yr Represents the uncertainty value of the network's prediction result of the y coordinate of the vertex in the lower right corner of the candidate box;

[0159] 6.1.2) Corresponding loss function based on the x-coordinate and y-coordinate of the upper left corner of a single rough candidate box and The x-coordinate and y-coordinate of the lower right corner of a single rough candidate box correspond to the loss function and Calculate the loss function L corresponding to a single rough candidate box P :

[0160]

[0161] in express and Any candidate box in, b represents the noise target box bn or the corrected target box br Ι ;

[0162] 6.1.3) According to the loss function L corresponding to a single rough candidate box P , calculate the total loss L of the uncertainty-aware candidate box optimization module T1 PR :

[0163]

[0164] in, Represents the set of predicted candidate boxes in the first candidate box optimization process The i-th predicted candidate box in Represents the set of predicted candidate boxes in the second candidate box optimization process The i-th predicted candidate box in, bn represents the noise target box, br Ι represents the target frame after a correction, and N represents the total number of predicted candidate frames;

[0165] 6.2) Calculate the loss function L of the target frame correction module T2 based on complementary information BBR :

[0166] 6.2.1) Calculate the loss L of the target frame correction module T2 based on complementary information when making the first correction Ι :

[0167] According to the multi-instance score matrix at the first correction Calculate the multi-instance learning score for each category during the first correction

[0168]

[0169] in, represents the multi-instance learning score of the i-th rough candidate box in each category during the first correction;

[0170] According to the multi-instance learning scores of the candidate boxes in each category during the first correction Calculate the loss L of the target frame correction module T2 based on complementary information during the first correction Ι :

[0171]

[0172] Among them, CE represents the cross entropy loss function, represents the unique hot code label of the target, C represents the total number of target categories, express Corresponding to the value on category t, express Corresponding to the value in category t;

[0173] 6.2.2) Calculate the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ :

[0174] According to the second correction, the multi-instance score matrix Calculate the multi-instance learning score for each category during the second revision

[0175]

[0176] in, represents the multi-instance learning score of the i-th rough candidate box in each category during the second correction;

[0177] According to the multi-instance learning scores of the candidate boxes in each category during the second correction Calculate the loss L of the target frame correction module T2 based on complementary information during the second correction ΙΙ :

[0178]

[0179] in, For label Compared with the multi-instance learning scores of candidate boxes in each category during the first correction The inner product of The mean of , FL is the Focal loss function, S neg represents the score of the classification branch of negative samples in multi-instance learning, c neg Indicates its label;

[0180] 6.2.3) Using the loss L of the target frame correction module T2 based on complementary information during the first correction Ι And the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ , calculate the total loss L of T2 module BBR :

[0181] L BBR =L Ι +L ΙΙ

[0182] 6.3) Modify the target detection network regression loss and use the target frame br after the second correction ΙΙ Instead of the noise target box bn, train the target detection network, that is, L r (p * ,bn) becomes L r (p * ,brΙΙ ), where L r is the smooth L1 loss function, p * Represents the candidate box generated based on the anchor box in the target detection network;

[0183] 6.4) According to the modified regression loss L r (p * ,br ΙΙ ) Get the overall loss L of the target detection network Det :

[0184] L Det =L r (p * ,br ΙΙ )+L RPN +L Cla

[0185] Where L RPN and L Cla Respectively represent the loss of the RPN module and the classification loss in the existing target detection network;

[0186] 6.5) According to the result of step 6.1) L PR , the result of step 6.2) L BBR and the result L of step 6.4) Det , calculate the overall loss L for training the noisy target box correction network and the target detection network based on uncertainty perception and complementary information:

[0187] L=w1L PR +w2L BBR +w3L Det

[0188] Among them, w1, w1 and w1 represent the module T1 loss L PR , module T2 loss L BBR And the overall loss L of the target detection network Det The weight of .

[0189] The effects of the present invention are further described below in conjunction with simulation experiments.

[0190] 1. Experimental conditions

[0191] The computer processor used is Intel(R)Xeon(R)Gold 6148CPU@2.4GHz, and the graphics card is two NVIDIA RTX 3090GPUs with 24GB of video memory.

[0192] The operating system is 64-bit Ubuntu 22.04, the simulation adopts Python language, and the deep learning framework PyTorch version 1.12.1 is used.

[0193] The evaluation index is mAP50, which is the mean average precision under the 50% intersection-over-union threshold.

[0194] All network training uses the back-propagation algorithm to calculate the gradients of each layer, and uses the stochastic gradient descent algorithm with kinetic energy and weight decay terms to update the network parameters, where the kinetic energy term is 0.9 and the weight decay term is 0.0001.

[0195] The noisy dataset used in the experiment was generated by artificially introducing erroneous annotation information into the annotation results of the public dataset. The training set used the noisy annotations, and the test set used the original annotations of the public dataset.

[0196] The noisy target box is generated by randomly moving and scaling the real target box, as shown below:

[0197]

[0198] Among them, [xg l ,yg l ,xg r ,yg r ] represents the real target box, xg l and yg l Respectively represent the x-coordinate and y-coordinate of the upper left corner of the real target box, xg r and yg r Indicates the x- and y-coordinates of the lower right corner of the real target box; [xn l ,yn l ,xn r ,yn r ] represents the noise target box, xn l Heyn l Respectively represent the x-coordinate and y-coordinate of the upper left corner of the noise target box, xn r Heyn r represents the x-coordinate and y-coordinate of the lower right corner of the noise target box; δ follows the uniform distribution U(-r d ,r d ), r d is the noise rate.

[0199] In the experiment, the target box noise rate ranges from 20%, 30% and 40%.

[0200] The public dataset used in the experiment is the VOC2007 dataset, which is a classic dataset for object detection tasks. The VOC2007 dataset contains 9963 images, with 2501, 2510, and 4952 images in the training set, validation set, and test set, respectively. A total of 24640 objects are annotated, with an average of 2.5 objects in each image.

[0201] 2. Simulation experiment content and results:

[0202] Simulation 1: Under the above experimental conditions, the method of the present invention and the five existing methods are used to correct the noisy target frame in the VOC2007 data set, and the network performance indicators of these methods under different noise rates are compared, as shown in Table 1.

[0203] Table 1 Performance indicators of the present invention and five existing methods under three different noise rates

[0204]

[0205] The five existing methods in Table 1 are:

[0206] Co-teaching: A noisy target box correction method based on the Small-loss criterion and a two-head network proposed by Chadwick et al.

[0207] SD-LocNet: A noisy target box correction method proposed by Zhang et al. that uses adaptive sampling to measure target localization scores and then identify reliable targets;

[0208] KLLoss: A method proposed by He et al. to correct noisy target frames by estimating the position uncertainty value of the target frame;

[0209] OAMIL: A noisy object box correction method based on object-aware multi-instance learning proposed by Liu et al.

[0210] SSD-Det: A noisy target box correction method based on spatial self-distillation proposed by Wu et al.

[0211] As can be seen from Table 1, the noise target frame correction network based on uncertainty perception and complementary information proposed by the method of the present invention achieves the best effect when the noise rate is 20%, 30% and 40%, and the mAP50 index is improved by 0.1, 0.7 and 2.9 percentage points respectively compared with the SSD-Det method, which proves the effectiveness of the present invention.

[0212] Simulation 2: Under the above experimental conditions, the method of the present invention is used to correct the noisy target frame in the VOC2007 dataset. The results are as follows: Figure 5 As shown, where:

[0213] Figure 5 (a) is the noise target frame,

[0214] Figure 5 (b) is the correction result of the noisy target frame.

[0215] Figure 5(c) is the true annotation of the target box.

[0216] contrast Figure 5 (a) and Figure 5 (b) It can be seen that after the noise target frame is corrected by the method of the present invention, the position accuracy of the target frame is improved;

[0217] contrast Figure 5 (b) and Figure 5 (c) It can be seen that the target frame corrected by the method of the present invention can accurately cover the real position of the target.

Claims

1. A noisy target frame correction method based on uncertainty perception and complementary information, It is characterized in that The steps include: (1) Obtain training tensor data and test tensor data: 1a) Read the training set data and test set data of the target detection database respectively; 1b) Set the normalized mean and standard deviation of the three RGB channels; adjust, flip and normalize the images of the training set in turn to obtain the tensor data corresponding to the training set images; adjust and normalize the images of the test set in turn to obtain the tensor data corresponding to the test images; (2) Read the training set annotation file to obtain the noisy target box, and generate an initial candidate box set for each noisy target box bn: Ι = {P Ι,1 ,P Ι,2 …P Ι,i …P Ι,N }, where N represents P Ι The number of initial candidate boxes in P Ι,i Indicates P Ι The i-th initial candidate box in , 1≤i≤N; (3) Select an existing image feature extraction network, input the training tensor data into the network, and obtain a bottom-up image pyramid feature X'; (4) Constructing a noise target frame correction network based on uncertainty perception and complementary information: 4a) Establishing an uncertainty-aware candidate box optimization module T1 including a region of interest alignment network, a fully connected layer, and a regression layer; 4b) establishing a target box correction module T2 based on complementary information, which includes a region of interest alignment network, a fully connected layer, a multiple instance learning network and a category feature group; 4c) The candidate frame optimization module T1 based on uncertainty perception and the target frame correction module T2 based on complementary information are connected in series to form a noise target frame correction network based on uncertainty perception and complementary information; (5) Using the image pyramid feature X' and the initial candidate box set P Ι , based on uncertainty perception and complementary information, the noise target frame correction network is used to obtain the corrected target frame br ΙΙ : 5a) Based on the image pyramid feature X' and the initial candidate box set P Ι , the uncertainty-aware candidate box optimization module T1 optimizes the initial candidate box set P Ι , get the rough candidate box set: Pr Ι = {Pr Ι,1 ,Pr Ι,2 …Pr Ι,i …Pr Ι,N }, where Pr Ι,i Indicates Pr Ι The i-th rough candidate box in ; 5b) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn to obtain a corrected target frame br Ι ; 5c) Repeat step 5a) to optimize the rough candidate box set The candidate box in the box is obtained to obtain a set of refined candidate boxes: Pr ΙΙ = {Pr ΙΙ,1 ,Pr ΙΙ,2 …Pr ΙΙ ,i…Pr ΙΙ,N }, where Pr ΙΙ,i Indicates Pr ΙΙ The i-th refined candidate box in; 5d) Repeat step 5b) according to the fine candidate box set Pr ΙΙ Get the final corrected target frame br ΙΙ ; (6) According to the final corrected target frame br ΙΙ , design the overall training loss L.

2. The method according to claim 1, Features : In step 1b), the normalized mean and standard deviation of the three RGB channels are set to be [123.675, 116.28, 103.53], and the standard deviation of the three channels is set to be [58.395, 57.12, 57.375]; In step 1b), the images of the training set are adjusted, flipped, and normalized in sequence, firstly scaling the image size to 1000×600, then randomly flipping the image with a probability of 0.5, and finally normalizing the image using the mean standard deviation normalization method to obtain tensor data corresponding to the images of the training set; In step 1b), the images of the test set are adjusted and normalized in sequence, firstly scaling the image size to 1000×600, and then normalizing the image using the mean standard deviation normalization method to obtain the tensor data corresponding to the images of the test set.

3. The method according to claim 1, It is characterized in that In step (2), an initial candidate box set P is generated for each noise target box bn Ι , the implementation steps include the following: 2a) Convert the noise target frame bn from [xl, yl, xr, yr] format to [x, y, w, h] format to obtain the noise target frame after conversion: bn * =[bn x ,bn y ,bn w ,bn h ] Where xl and yl represent the x and y coordinates of the upper left corner vertex of the target box, respectively, and xr and yr represent the x and y coordinates of the lower right corner vertex of the target box, respectively; x and y represent the x and y coordinates of the center point of the target box after the format conversion, respectively, and w and h represent the width and height of the target box after the format conversion; 2b) Set relevant parameters: Set the resizing parameter s to modify the noise target box bn * The width and height of the parameter are set to the range of s∈{0.7,0.8,1,1.2,1.3}; Set the x coordinate adjustment parameter o x , used to modify the noise target box bn * The x coordinate of the center point; set the y coordinate adjustment parameter o y , used to modify the noise target box bn * The y coordinate of the center point; Set x and y coordinate adjustment parameters o x and y The value range of is: (the x ,o y )∈{(0,0),(2,0),(0,2),(-2,0),(-2,-2)}; 2c) According to the noise target box bn * =[bn x ,bn y ,bn w ,bn h ] Generate initial candidate box in, Indicates the first generated initial candidate box in the format of [x, y, w, h]; 2d) The initial candidate box in the format of [x, y, w, h] Convert the [x, y, w, h] format to the [xl, yl, xr, yr] format to get the first initial candidate box P in the [xl, yl, xr, yr] format Ι,1 ; 2e) Repeat operations 2c) and 2d) N times to obtain an initial candidate box set P in the format of [xl, yl, xr, yr] Ι .

4. The method according to claim 1, It is characterized in that Step 4a) Establish a candidate box optimization module T1 based on uncertainty perception, in which the structural parameters and connection methods of each layer are as follows: The region of interest alignment network has an output feature size of 7×7 and an output dimension of 256. The output dimension of the fully connected layer is 1024. The regression layer is set to two layers, and the output dimension of each layer is 4; The region of interest alignment network, the fully connected layer and the two parallel regression layers are connected in sequence to form an uncertainty-aware candidate box optimization module T1.

5. The method according to claim 1, It is characterized in that In step 4b), a target frame correction module T2 based on complementary information is established, wherein the structural parameters and connection methods of each layer are as follows: The region of interest alignment network has an output feature size of 7×7 and an output dimension of 256. The output dimension of the fully connected layer is 1024. The multi-instance learning network comprises two parallel classification branches and detection branches. The category feature group ψ includes C groups of target features, where C represents the total number of target categories, and each group of target features includes U target features. The region of interest alignment network and the fully connected layer are connected in series, and then connected in parallel with the multi-instance learning network and the category feature group to construct a target box correction module T2 based on complementary information.

6. The method according to claim 1, It is characterized in that Step 5a) Based on the image pyramid feature X' and the initial candidate box set P Ι , the uncertainty-aware candidate box optimization module T1 optimizes the initial candidate box set P Ι , get the rough candidate box set Pr Ι , the implementation steps include the following: 5a1) Based on the image pyramid feature X' and the initial candidate box set P Ι , obtain the initial candidate box P through the region of interest alignment network and the fully connected layer Ι The feature set of the target in: F Ι ={F Ι,1 ,F Ι,2 …F Ι,i …F Ι,N }, where F Ι,i Indicates F Ι The i-th target feature in ; 5a2) According to the target feature set F Ι , through two parallel regression layers, the predicted candidate box sets are obtained respectively: And the uncertainty value set of the network prediction candidate box: V Ι = {V Ι,1 ,V Ι,2 …V Ι,i …V Ι,N },in express The i-th predicted candidate box in V Ι,i Indicates V Ι The i-th uncertainty value in ; 5a3) According to the uncertainty value set V of the network prediction candidate box Ι Optimize prediction candidate boxes Get the rough candidate box Pr Ι,i ; 5a4) Repeat step 5a3) N times to obtain a rough candidate box set Pr Ι .

7. The method according to claim 6, It is characterized in that Step 5a3) Optimize the predicted candidate box Get the rough candidate box Pr Ι,i , calculated as follows: 5a3.1) Get the predicted candidate box set Center and predict candidate boxes A series of candidate boxes with IoU greater than 0.5: Pn = {Pn 1 ,Pn 2 ,…Pn j …Pn A }, where A represents the number of candidate boxes in Pn, Pn j represents the j-th candidate box in Pn, A≤N, 1≤j≤A; 5a3.2) According to the uncertainty value set V of the network prediction candidate box Ι , obtain the uncertainty value set corresponding to the candidate box set Pn: Vn = {Vn 1 ,Vn 2 …Vn j …Vn A },Vn j represents the jth uncertainty value in Vn; 5a3.3) Based on the predicted candidate box Calculate the confidence score set S of the candidate box Pn iou : in, YesS iou The j-th confidence score in Pn is the j-th candidate box Pn in Pn j Confidence score of 5a3.4) Based on the candidate box set Pn and the uncertainty value set Vn corresponding to these candidate boxes and the confidence score set S iou , calculate the rough candidate box Pr Ι,i : in, Represents the rough candidate box Pr Ι,i The x coordinate of the upper left vertex, Represents the candidate box Pn j The x coordinate of the upper left vertex, Represents the candidate box Pn j The confidence score of Represents the network's candidate box Pn j The uncertainty value of the x-coordinate prediction of the upper left vertex; Represents the rough candidate box Pr Ι,i The y coordinate of the upper left vertex, Represents the candidate box Pn j The y coordinate of the upper left vertex, Represents the network's candidate box Pn j The uncertainty value of the predicted y coordinate of the upper left vertex; Represents the rough candidate box Pr Ι,i The x coordinate of the lower right vertex, Represents the candidate box Pn j The x coordinate of the lower right vertex, Represents the network's candidate box Pn j The uncertainty value of the predicted x-coordinate of the lower right vertex; Represents the rough candidate box Pr Ι,i The y coordinate of the lower right vertex, Represents the candidate box Pn j The y coordinate of the lower right vertex, The network is divided into candidate boxes Pn j The uncertainty value of the predicted y-coordinate of the lower right vertex.

8. The method according to claim 1, It is characterized in that In step 5b), based on the image pyramid feature X' and the rough candidate box set Pr Ι , the target frame correction module T2 based on complementary information corrects the noise target frame bn to obtain a corrected target frame br Ι , the implementation steps include the following: 5b1) Based on the image pyramid feature X' and the rough candidate box set Pr Ι , obtain the rough candidate box set Pr through the region of interest alignment network and the fully connected layer Ι The feature set Fr of the target Ι = {Fr Ι,1 ,Fr Ι,2 …Fr Ι,i …Fr Ι,N }, where Fr Ι,i It is Fr Ι The i-th target feature in ; 5b2) According to the target feature set Fr Ι , through the multi-instance learning network, obtain the multi-instance score matrix Calculate the rough candidate box set Pr Ι The multi-instance score set: S Ι ={S Ι,1 ,S Ι,2 …S Ι,i …S Ι,N }, where S Ι,i YesS Ι The i-th multi-instance score in ; 5b3) According to the rough candidate box set Pr Ι The characteristic Fr of the target Ι Update the category feature group ψ; 5b4) According to the category feature set ψ and the target feature set Fr Ι , calculate the rough candidate box set Pr Ι Similarity score collection: Sim Ι = {Sim Ι,1 ,Sim Ι,2 …Sim Ι,i …Sim Ι,N }, where Sim Ι,i It's Sim Ι The i-th similarity score in ; 5b5) Based on the multi-instance score set S Ι and similarity score set Sim Ι Calculate the rough candidate box set Pr Ι The final score of Sr Ι = {Sr Ι,1 ,Sr Ι,2 …Sr I,i …Sr Ι,N }, Among them, Sr Ι,i =α*S Ι,i +β*Sim Ι,i is the final score set Sr Ι The final score of the ith one in ; α and β are combination coefficients, α+β=1; S Ι,i is the multi-instance score set S Ι The i-th multi-instance score in Sim Ι,i is the similarity score set Sim Ι The i-th similarity score in ; 5b6) According to the rough candidate box set Pr Ι and its final score set Sr Ι Calculate the corrected target frame br Ι : 5b6.1) According to the final score set Sr Ι And the rough candidate box set Pr Ι Get the highest H score sets S top ={S top,1 ,S top,2 …S top,k …S top,H } and the corresponding candidate box set P top = {P top,1 ,P top,2 …P top,k …P top,H }, where P top,k YesP top The kth candidate box in S top,k YesS top The kth score in , 1≤k≤H; 5b6.2) According to the score set S top And the candidate box set P top Calculate the corrected target frame br Ι : br Ι =[br xl ,br yl ,br xr ,br yr ] in, is the target frame after a correction Ι The x coordinate of the upper left vertex, S top,k is the score set S top The kth score, P top,k is the candidate box set P top The kth candidate box in is the candidate box P top,k The x coordinate of the upper left vertex; is the target frame after a correction Ι The y coordinate of the upper left vertex, is the candidate box P top,k The y coordinate of the upper left vertex; is the target frame after a correction Ι The x coordinate of the lower right vertex, is the candidate box P top,k The x coordinate of the lower right vertex; is the target frame after a correction Ι The y coordinate of the lower right vertex, is the candidate box P top,k The y coordinate of the lower right vertex.

9. The method according to claim 8, Features: In step 5b3), according to the rough candidate box set Pr Ι The characteristic Fr of the target Ι Update the category feature group ψ: 5b3.1) Obtain the multi-instance score set Sr Ι The maximum value of Sr Ι,max , get the rough candidate box set Pr Ι The rough candidate box with the largest multi-instance score is obtained, and the feature Fr of the target contained in the rough candidate box is obtained. Ι,max ; 5b3.2) Set the rough candidate box set Pr Ι The category of the target is yc, and the category feature group corresponding to category yc is obtained in Represents ψ yc The u-th feature in , 1≤u≤U, U represents the category feature group ψ yc The number of features in 5b3.3) Calculate the feature Fr of the rough candidate box with the maximum multi-instance score Ι,max and the class feature group ψ yc Similarity of features in sm m ={sm m,1 ,sm m,2 …sm m,u …sm m,U }, in It is Fr Ι,max With ψ yc The uth feature in The similarity of sm m The u-th similarity in , < > represents the inner product operation, || || represents the two-norm operation; 5b3.4) According to similarity sm m , from ψ yc Selection and feature Fr Ι,max Most similar features 5b3.5) Using feature Fr Ι,max and the multi-instance score Sr Ι,max Update the class feature set ψ yc Features in Where r is the momentum coefficient; 5b3.6) Change the noise target box bn and the target category yc it contains, repeat steps 5b3.1) to 5b3.5) for a total of Z times to complete the update of the category feature group ψ, where Z represents the total number of targets in the training set.

10. The method according to claim 8, Features: In step 5b4), according to the category feature set ψ and the target feature set Fr Ι , calculate the rough candidate box set Pr Ι The similarity score set Sim Ι , the implementation steps include the following: 5b4.1) Get Pr Ι The i-th rough candidate box Pr Ι,i and its feature Fr containing the target Ι,i ; 5b4.2) Calculate Pr Ι The i-th rough candidate box Pr Ι,i Contains the target feature Fr Ι,i The category feature group ψ corresponding to the category yc yc The similarity of all features in , obtain similarity sm i ={sm i,1 ,sm i,2 …sm i,u …sm i,U }, in It is Fr Ι,i With ψ yc The uth feature in The similarity of sm i The u-th similarity in ; 5b4.3) Get similarity sm i The maximum value among them is the rough candidate box Pr Ι,i The similarity score Sim Ι,i ; 5b4.4) Repeat steps 5b4.1) to 5b4.3) N times to obtain the similarity score set Sim Ι .

11. The method according to claim 1, It is characterized in that In step (6), the final corrected target frame br ΙΙ , design the overall training loss L, which is implemented as follows: 6a) Design the loss L of the uncertainty-aware candidate box optimization module T1 PR : in, Represents the set of predicted candidate boxes in the first candidate box optimization process The i-th predicted candidate box in, Represents the set of predicted candidate boxes in the second candidate box optimization process The i-th predicted candidate box in, bn represents the noise target box, br Ι represents the target frame after a correction, and N represents the total number of predicted candidate frames; is the function of the loss corresponding to a single candidate box, express and Any candidate box in, b represents the noise target box bn or the corrected target box br Ι , Is to calculate the candidate box The loss function corresponds to the x-coordinate of the upper left corner of the target box b. and b xl Represent candidate boxes and the x coordinate of the upper left corner vertex of the target box b, v xl Represents the network's candidate box The uncertainty value of the x-coordinate prediction result of the upper left vertex; Is to calculate the candidate box The loss function corresponds to the y coordinate of the upper left corner of the target box b. and b yl Represent candidate boxes and the y coordinate of the upper left corner vertex of the target box b, v yl Represents the network's candidate box The uncertainty value of the y-coordinate prediction result of the upper left vertex; Is to calculate the candidate box The loss function corresponds to the x-coordinate of the lower right corner of the target box b. and b xr Represent candidate boxes and the x coordinate of the lower right corner vertex of the target box b, v xr Represents the network's candidate box The uncertainty value of the x-coordinate prediction result of the lower right vertex; Is to calculate the candidate box The loss function corresponds to the y coordinate of the lower right corner of the target box b. and b yr Represent candidate boxes and the y coordinate of the lower right corner vertex of the target box b, v yr Represents the network's candidate box The uncertainty value of the y-coordinate prediction result of the lower right vertex; 6b) Design the loss L of the target box correction module T2 based on complementary information BBR : 6b1) Calculate the loss L of the target frame correction module T2 based on complementary information when performing the first correction Ι : According to the multi-instance score matrix at the first correction Calculate the multi-instance learning score for each category during the first correction in, represents the multi-instance learning score of the i-th rough candidate box in each category during the first correction; According to the multi-instance learning scores of the candidate boxes in each category during the first correction Calculate the loss L of the target frame correction module T2 based on complementary information during the first correction Ι : Among them, CE represents the cross entropy loss function, represents the unique hot code label of the target, C represents the total number of target categories, express Corresponding to the value on category t, express Corresponding to the value in category t; 6b2) Calculate the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ : According to the second correction, the multi-instance score matrix Calculate the multi-instance learning score for each category during the second revision in, represents the multi-instance learning score of the i-th rough candidate box in each category during the second correction; According to the multi-instance learning scores of candidate boxes in each category during the second correction Calculate the loss L of the target frame correction module T2 based on complementary information during the second correction ΙΙ : in, For label Compared with the multi-instance learning scores of candidate boxes in each category during the first correction The inner product of The mean of , FL is the Focal loss function, S neg represents the score of the classification branch of negative samples in multi-instance learning, c neg Indicates its label; 6b3) Using the loss L of the target frame correction module T2 based on complementary information during the first correction Ι And the loss L of the target frame correction module T2 based on complementary information when performing the second correction ΙΙ , calculate the total loss L of the T2 module BBR : L BBR =L Ι +L ΙΙ 6c) Modify the target detection network regression loss and use the second corrected target frame br ΙΙ Instead of the noise target box bn, train the target detection network, that is, L r (p * ,bn) becomes L r (p * ,br ΙΙ ), where L r is the smooth L1 loss function, p * Represents the candidate box generated based on the anchor box in the target detection network; 6d) According to the updated regression loss L r (p * ,br ΙΙ ) Get the overall loss L of the target detection network Det : L Det =L r (p * ,br ΙΙ )+L RPN +L Cla Where L RPN and L Cla Respectively represent the loss of the RPN module and the classification loss in the existing target detection network; 6e) Based on the result L of step 6a) PR , the result of step 6a) L BBR and the result L of step 6d) Det , calculate the overall loss L for training the noisy target box correction network and the target detection network based on uncertainty perception and complementary information: L=w 1 L PR +w 2 L BBR +w 3 L Det Among them, w 1 、w 1 and w 1 They represent the module T1 loss L PR , module T2 loss L BBR And the overall loss L of the target detection network Det The weight of .

Images