Sample selection network construction method, sample selection method, training method, electronic device, and storage medium

Abstract

The application discloses a sample selection network fusing target feature information and prediction information, which comprises a projection layer, a merging unit and a score calculation unit, and is used for estimating a sample score of each target according to feature information extracted by a detection network and predicted regression information and classification information. i The sample selection method is based on the sample score, and the number of positive samples required by each target is calculated according to the sample score of the candidate sample i Each target candidate sample is sorted according to the score from high to low, the first n The remaining samples are negative samples. The training method first obtains the sample score corresponding to each target by using the sample selection network, then selects the positive samples and the negative samples by using the sample selection method, calculates the sample loss, the classification loss and the regression loss, calculates the sample weight by using the sample score, and uses the sample weight as a weight factor of the classification loss and the regression loss. The sample selection network is applied to a single-stage target detector, and is beneficial to improving the performance of the target detection network.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a sample selection network that integrates target feature information and prediction information. Background Technology

[0002] Sample selection is a crucial component of object detection algorithms, involving choosing suitable positive and negative samples for network training. Currently, sample selection methods used in object detection algorithms are mainly divided into those based on prior information and those based on prediction information.

[0003] Sample selection methods based on prior information mainly include those based on IoU thresholds and those based on centroid distance. The IoU score-based method first calculates the IoU score between a manually set anchor and the target's ground truth bounding box, selecting samples with scores greater than a threshold as positive samples and the rest as negative samples. The centroid distance-based method selects samples falling within the target's ground truth bounding box as positive samples and the rest as negative samples. Both of these prior information-based methods only consider the impact of regression information on network training. However, object detection is a multi-task mechanism involving classification and regression; samples selected while ignoring classification information are often not optimal.

[0004] Sample selection methods based on prediction information include the minimum cost method proposed by OneNet and the optimal transport theory-based method proposed by OTA. The minimum cost method first calculates classification and localization costs based on the network's prediction information, then uses a weighted combination of these two costs as the final cost matrix, selecting the minimum cost sample for each target as a positive sample, and the rest as negative samples. The optimal transport theory-based method introduces optimal transport theory for global label allocation and uses the network's predicted classification and regression information to calculate the cost matrix. Both of these prediction-based methods consider classification and regression information when constructing the sample cost matrix, resulting in more representative samples. However, the rules for calculating the sample cost are still manually set, lacking adaptability to network changes and easily leading to overfitting during network training. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, one objective of this application is to provide a sample selection network that integrates target feature information and prediction information, applied to a single-target detection network. This network takes the feature information learned by the detection network, combined with the predicted regression and classification information, as input, and perceives in real time the changes in target features extracted by the detection network during training. This provides more suitable samples for network training and improves detection performance.

[0006] Another objective of this application is to provide a sample selection method based on sample scores, which calculates training weights for each sample and applies them to the calculation of classification loss and regression loss, thus providing different priorities for training.

[0007] Another objective of this application is to provide a training method that simultaneously trains a sample selection network and an object detection network, while considering the differences between positive and negative samples, the importance of positive samples, etc., to establish an evaluation criterion for the selected samples and to filter out noisy samples, thereby improving the stability of sample selection.

[0008] To achieve the above objectives, the present invention employs the following techniques:

[0009] A sample selection network that integrates target feature information and prediction information is applied to a single-stage target detector. The sample selection network includes:

[0010] The projection layer is used to process the feature information output by the detection network. f The process is performed to obtain a feature map containing sample information. ;

[0011] The merging unit is used to combine the regression information and classification scores predicted by the detection network with the feature map output by the projection layer. Merge into new features ;

[0012] Fraction calculation unit, used for features The process involves calculating the sample score for each target.

[0013] A sample selection method based on sample scores, applied to a single-stage target detector, includes the following steps:

[0014] S11. Use a sample selection network to obtain the sample score of each target corresponding to the sample;

[0015] S12. Filter out candidate samples for each target based on the target's true bounding box;

[0016] S13. Based on the sample scores of the candidate samples, calculate the number of positive samples required for each target. n i ;

[0017] S14. Sort the candidate samples for each target from highest to lowest score, and select the top-scoring samples. n i One sample is designated as a positive sample, and the rest are designated as negative samples.

[0018] A training method for simultaneously training a sample selection network and a single-object detection network including the sample selection network, the training method comprising the steps of:

[0019] S21. First, input the feature information extracted by the detection network, as well as the predicted regression and classification information, into the sample selection network to obtain the sample score corresponding to each target.

[0020] S22. Based on the sample scores obtained in step S21, select appropriate positive and negative samples for each target using a sample selection method based on sample scores.

[0021] S23. Randomly select from the negative samples obtained in step S22 n neg The negative samples are used together with all the positive samples obtained in step S22 to calculate the loss function for sample selection.

[0022] S24. Based on the sample scores of the positive samples in step S22, calculate the training weight of each positive sample, and use it as a weight factor for classification loss and regression loss during training.

[0023] S25. Train the sample selection network and the single-target detection network according to the above method.

[0024] An electronic device includes: at least one processor and a memory; wherein the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to execute the sample selection method based on sample scores or the training method.

[0025] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, controls the device on which the storage medium is located to execute the sample selection method based on sample scores or the training method.

[0026] The beneficial effects of this invention are as follows:

[0027] (1) The sample selection network of this application, which integrates target feature information and prediction information, is a plug-and-play sample selection module suitable for various single-stage detectors. The sample selection network of this application abandons the previous sample selection method based on manually set thresholds. It takes the feature information learned by the detection network, combined with the predicted regression information and classification information as input, and perceives the changes in target features extracted by the detection network in real time during the training process. This provides more suitable samples for network training, thereby improving the detection performance.

[0028] (2) This application uses a sample selection method based on sample scores to dynamically update the number of positive samples for each target and select appropriate positive samples for each target to participate in network training. Since each sample has a different ability to represent the target, training weights are calculated for each sample according to the sample scores, and are applied to the calculation of classification loss and regression loss, providing different priorities for training.

[0029] (3) In the past, positive samples were selected based on manually set thresholds, which lacked evaluation of the selected samples. Compared with the prior art, this application proposes a loss function based on ranking loss, which takes into account the difference between positive and negative samples, the importance between positive samples, etc., establishes an evaluation standard for the selected samples, and filters out noisy samples, which can improve the stability of sample selection. Attached Figure Description

[0030] Figure 1 This is a sample selection network structure diagram of an embodiment of this application.

[0031] Figure 2 This is a flowchart of the sample selection method according to an embodiment of this application.

[0032] Figure 3 This is a diagram of the single-target detection network structure according to an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the implementation methods of the present invention will be described in detail below with reference to the accompanying drawings. However, the embodiments described in this invention are only some embodiments of the present invention, and not all embodiments.

[0034] One aspect of this application provides a sample selection network that integrates target feature information and prediction information, applied to a single-stage target detector. An example scenario of a target detection network in which this sample selection network is used is as follows: Figure 3 As shown.

[0035] In this example, the object detection network includes a backbone network, a feature pyramid, a head network, and the sample selection network provided in this example. The head network includes a classification branch and a regression branch. The backbone network extracts features from the input raw image and outputs multi-layer features; the feature pyramid performs hierarchical processing on the multi-layer features and outputs feature information. f Classification branches are used based on feature information. f The system predicts classification features and outputs the classification score of the target; the regression branch is used to predict classification features based on the feature information. f The feature pyramid predicts regression features and outputs regression information for the target; the sample selection network is used to select features based on the feature information output by the feature pyramid. fThe classification branch outputs the classification score, and the regression branch outputs the regression information. The sample score corresponding to each target is calculated.

[0036] Specifically, such as Figure 1 As shown, the sample selection network in this example includes a projection layer, a merging unit, and a score calculation unit.

[0037] The projection layer consists of two convolutional layers and one activation layer, used to process the feature information output by the detection network. f The process is performed to obtain a feature map containing sample information. Specifically, the projection layer consists of a first convolutional layer with a 3×3 kernel, a first activation layer, and a second convolutional layer with a 1×1 kernel, arranged sequentially. Let the feature information output by the detection network be... f for( f 1 ,f 2 ,…,f 5 The features containing sample information are obtained by sequentially passing through the first convolutional layer, the first activation layer, and the second convolutional layer. .

[0038] The merging unit is used to combine the regression information and classification scores predicted by the detection network with the feature map output by the projection layer. Merge into new features Specifically, the merging unit will feature spliced ​​together N×K × C vector ,in, , l Represents the feature layer. H Indicates the height of the feature map. W Indicates the width of the feature map. N Indicates the number of targets. C This represents the number of channels in the feature map; and based on the regression information predicted by the detection network and the ground truth bounding box of the target, a regression score, i.e., an IoU score, is calculated; and the regression score and classification information are then compared with the vector... Add or merge arrays to obtain characteristics .

[0039] The score calculation unit consists of two fully connected layers and one activation layer, used to generate corresponding sample scores for each target, i.e., for features. The process involves calculating the sample score for each target. Specifically, the score calculation unit includes a first connection layer, a second activation layer, and a second connection layer, and the features obtained from the merging unit are... After passing through the first connection layer, the second activation layer, and the second connection layer in sequence, the sample scores are obtained for subsequent sample selection and network training.

[0040] Another aspect of this application provides a sample selection method based on sample scores, applied to a single-stage target detector. For example... Figure 2 The flowchart shown below illustrates the method steps, which include the following steps:

[0041] S11. Obtain the sample score of each target corresponding to the sample using the sample selection network in the previous embodiment.

[0042] S12. Filter candidate samples for each target based on the true bounding box of the target. Generally, samples within the target bounding box are selected as candidate samples. If the target is too small and there are no samples within the bounding box, samples within 1.5 times the size of the target bounding box are selected as candidate samples.

[0043] S13. Based on the sample scores of the candidate samples, calculate the number of positive samples required for each target. n i .

[0044] Specifically, the sum of the scores for each target candidate sample is first calculated based on the sample scores. S i Then calculate the mean score of the candidate samples for each target. M i and variance V i The number of positive samples required for each target can be calculated using the following formula. n i :

[0045]

[0046] in, α It is a regulatory factor.

[0047] S14. Sort the candidate samples for each target from highest to lowest score, and select the top-scoring samples. n i One sample is designated as a positive sample, and the rest are designated as negative samples.

[0048] This example uses a sample selection method based on sample scores to dynamically update the number of positive samples for each target and select appropriate positive samples for each target to participate in network training. Since each sample has a different ability to represent the target, training weights are calculated for each sample according to its sample score, and these weights are applied to the calculation of both classification and regression losses, providing different priorities for training.

[0049] In another aspect of the embodiments of this application, a training method is provided, applicable to a target detection method that incorporates the sample selection network described in the preceding embodiments, capable of simultaneously training the sample selection network and a single-target detection network containing the sample selection network, comprising the following steps:

[0050] S21. First, input the feature information extracted by the detection network, as well as the predicted regression and classification information, into the sample selection network to obtain the sample score corresponding to each target.

[0051] S22. Based on the sample scores obtained in step S21, select appropriate positive and negative samples for each target using a sample selection method based on sample scores.

[0052] S23. Randomly select from the negative samples obtained in step S22 n neg The negative samples are used together with all the positive samples obtained in step S22 to calculate the loss function for sample selection.

[0053] The loss function for sample selection considers two aspects: First, the scores of all positive samples should be higher than the scores of negative samples. A higher sample score indicates a stronger representation ability of the target; therefore, the scores of positive samples should be higher than those of negative samples. Second, each sample has a different representation ability of the target. When a sample has a higher regression score, it indicates that the sample can better locate the target. Therefore, for positive samples of the same target, samples with higher regression scores should have greater weight.

[0054] In this example, the ranking loss is used to calculate the loss function for sample selection, as shown in the following formula:

[0055]

[0056] in, L sample Indicates sample loss. S p , S n These represent the positive sample scores and the negative sample scores, respectively. O p , O n Let them represent the positive sample set and the negative sample set, respectively. m Indicates the number of targets. Indicates the first i The first goal k The score of the positive sample and the first positive sample score a A number of positive sample scores They represent the first i The first goal kThe regression score of the first sample and the first... a The regression score of each sample.

[0057] S24. Based on the sample scores of the positive samples obtained in step S22, calculate the training weight for each positive sample, which will be used as a weight factor in the classification loss and regression loss during training. The formula for calculating the weight is as follows:

[0058]

[0059] in, s Represents the sample score. γ It is a regulating factor used to adjust the weight of samples.

[0060] S25. Train the sample selection network and the single-target detection network according to the above method.

[0061] In the training method of this example, a loss function based on ranking loss is proposed, which takes into account the difference between positive and negative samples, the importance among positive samples, etc., establishes an evaluation criterion for the selection of samples, and filters out noisy samples, which can improve the stability of sample selection.

[0062] In another aspect of this application, an electronic device is provided, comprising: at least one processor and a memory; wherein the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to execute the sample selection method based on sample scores as described in the preceding embodiments, or to execute the training method as described in the preceding embodiments.

[0063] In another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, controls the device where the storage medium is located to perform the sample selection method based on sample scores as described in the preceding embodiments, or to perform the training method as described in the preceding embodiments.

[0064] The above description is only a preferred embodiment of this application and is not intended to limit this application. Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application.

Claims

1. A method for constructing a sample selection network that integrates target feature information and prediction information, characterized in that, Applied to a single-stage target detector, the target detection network of the single-stage target detector includes a backbone network, a feature pyramid, a head network, and the sample selection network, wherein the head network includes a classification branch and a regression branch; The backbone network is used to extract features from the input raw image and output multi-layer features; Feature pyramids are used to perform hierarchical processing of multiple layers of features and output feature information. f ; Classification branches are used based on feature information f Predict classification features and output the classification score of the target; Regression branches are used based on feature information f It predicts regression features and outputs the regression information of the target. The sample selection network is used to select features based on the feature information output by the feature pyramid. f The classification branch outputs the classification score, the regression branch outputs the regression information, and the sample score corresponding to each target is calculated. The sample selection network construction method includes the following: The projection layer is used to process the feature information output by the object detection network. f The process is performed to obtain a feature map containing sample information. ; The merging unit is used to combine the regression information and classification scores predicted by the object detection network with the feature map output by the projection layer. Merge into new features ; Fraction calculation unit, used for features The process involves calculating the sample score for each target.

2. The sample selection network construction method according to claim 1, characterized in that, The projection layer consists of a first convolutional layer with a 3×3 kernel, a first activation layer, and a second convolutional layer with a 1×1 kernel, arranged sequentially. Let the feature information output by the object detection network be... f for( f 1 ,f 2 ,…,f 5 The features containing sample information are obtained by sequentially passing through the first convolutional layer, the first activation layer, and the second convolutional layer. .

3. The sample selection network construction method according to claim 2, characterized in that, The merging unit is used to combine features spliced ​​together N×K × C vector ,in, , l Represents the feature layer. H Indicates the height of the feature map. W Indicates the width of the feature map. N Indicates the number of targets. C This represents the number of feature channels; it is used to calculate the regression score based on the regression information predicted by the object detection network and the ground truth bounding box of the object; and it is used to combine the regression score and classification information with the vector. Add or merge arrays to obtain characteristics .

4. The sample selection network construction method according to claim 1, characterized in that, The score calculation unit includes a first connection layer, a second activation layer, and a second connection layer. The features obtained by merging units are... After passing through the first connection layer, the second activation layer, and the second connection layer in sequence, the sample scores are obtained for subsequent sample selection and network training.

5. A sample selection method based on sample scores, characterized in that, Applied to a single-stage target detector, the steps include: S11. Obtain the sample score of each target corresponding sample by using the sample selection network constructed by the sample selection network construction method according to any one of claims 1 to 4; S12. Filter out candidate samples for each target based on the target's true bounding box; S13. Based on the sample scores of the candidate samples, calculate the number of positive samples required for each target. n i ; S14. Sort the candidate samples for each target from highest to lowest score, and select the top-scoring samples. n i One sample is designated as a positive sample, and the rest are designated as negative samples.

6. A training method, characterized in that, The training method for simultaneously training a sample selection network constructed by the sample selection network construction method as described in any one of claims 1 to 4 and a single-target detection network containing the sample selection network includes the following steps: S21. First, input the feature information extracted by the target detection network, as well as the predicted regression and classification information, into the sample selection network to obtain the sample score corresponding to each target. S22. Based on the sample scores obtained in step S21, select appropriate positive and negative samples for each target using the sample selection method based on sample scores as described in claim 5. S23. Randomly select from the negative samples obtained in step S22 n neg The negative samples are used together with all the positive samples obtained in step S22 to calculate the loss function for sample selection. S24. Based on the sample scores of the positive samples in step S22, calculate the training weight of each positive sample, and use it as a weight factor for classification loss and regression loss during training. S25. Train the sample selection network and single-target detection network according to the above method.

7. An electronic device, comprising: At least one processor and a memory; wherein the memory stores computer execution instructions; characterized in that, when the at least one processor executes the computer execution instructions stored in the memory, the at least one processor performs the sample selection method based on sample scores as described in claim 5, or performs the training method as described in claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it controls the device where the storage medium is located to perform the sample selection method based on sample scores as described in claim 5, or to perform the training method as described in claim 6.

Images