Infrared small target detection network training method and device, equipment and storage medium

Abstract

The application discloses an infrared small target detection network training method and device, equipment and a storage medium, and relates to the technical field of target detection. The method comprises the following steps: based on the number of positive samples and a preset proportion coefficient, difficult negative sample sets and easy negative sample sets are selected from an initial negative sample set; the preset proportion coefficient is a coefficient that is preset to improve the balance degree of the number of positive and negative samples; a weight matrix for controlling samples in a back propagation process is generated according to the difficult negative sample sets and the easy negative sample sets; a difficult point mining loss function is determined based on the weight matrix, and the infrared small target detection network is point supervised and trained by using the difficult point mining loss function. Through the above scheme, the difficult negative sample sets and the easy negative sample sets are determined based on the number of positive samples and the preset proportion coefficient, the unbalanced situation of the number of positive and negative samples in the target detection network training process is prevented, the balance degree of the number of positive and negative samples is improved, and therefore the training effect of the target detection network is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to an infrared small target detection network training method and device, equipment and a storage medium. BACKGROUND

[0002] The purpose of infrared small target detection is to separate the target from the clutter background and accurately locate it. At present, it is widely used in civil and military fields, such as fire alarm systems, unmanned aerial vehicle vision systems and early warning systems. In recent years, the infrared small target detection method based on deep learning has achieved good results due to its powerful feature learning ability.

[0003] However, since the small target in the infrared image only occupies a few to tens of pixels, the proportion of target pixels in all pixels is very small, which leads to an imbalance in the number of positive and negative samples used in the process of training the infrared small target detection network. This situation will affect the training effect of the target detection network and may cause insufficient feature learning of the network for the infrared small target, thereby making it difficult to improve the network performance. SUMMARY

[0004] Therefore, the purpose of the present application is to provide an infrared small target detection network training method, device, equipment and storage medium, which can prevent the imbalance in the number of positive and negative samples during the training process of the target detection network, improve the balance degree of the number of positive and negative samples, and thus improve the training effect of the target detection network. The specific scheme is as follows:

[0005] In a first aspect, the present application provides an infrared small target detection network training method, comprising:

[0006] Based on the number of positive samples and a preset proportion coefficient, a difficult negative sample set and an easy negative sample set are selected from an initial negative sample set; the preset proportion coefficient is a coefficient preset for improving the balance degree of the number of positive and negative samples;

[0007] A weight matrix for controlling the samples in the back propagation process is generated according to the difficult negative sample set and the easy negative sample set;

[0008] A difficult point mining loss function is determined based on the weight matrix, and the infrared small target detection network is point supervised and trained by using the difficult point mining loss function.

[0009] Optionally, the method of selecting the difficult negative sample set and the easy negative sample set from the initial negative sample set based on the number of positive samples and the preset proportion coefficient comprises:

[0010] The first sample number in the difficult negative sample set and the second sample number in the easy negative sample set are determined based on the number of positive samples and the preset proportion coefficient;

[0011] screening the first sample quantity of negative samples from the initial negative sample set according to a difficult negative sample screening rule constructed based on the initial loss of the pixel sample, to obtain a difficult negative sample set;

[0012] screening the second sample quantity of negative samples from the initial negative sample set, to obtain an easy negative sample set.

[0013] Optionally, the first sample quantity in the difficult negative sample set and the second sample quantity in the easy negative sample set are determined based on the positive sample quantity and a preset proportion coefficient, and the method comprises:

[0014] determining a total quantity of difficult and easy negative samples based on the positive sample quantity and a first preset proportion coefficient for improving the balance degree of the positive and negative sample quantities; the first preset proportion coefficient is a coefficient preset to represent the ratio between the positive sample quantity and the total quantity of difficult and easy negative samples;

[0015] determining the first sample quantity in the difficult negative sample set and the second sample quantity in the easy negative sample set based on the total quantity of difficult and easy negative samples and a second preset proportion coefficient; the second preset proportion coefficient is a coefficient preset to represent the ratio between the quantity of difficult negative samples and the quantity of easy negative samples.

[0016] Optionally, before the first sample quantity of negative samples is screened from the initial negative sample set, the method further comprises:

[0017] determining the sample loss weight of each pixel sample in the network output result of the infrared small target detection network according to a first preset weight determination rule; the first preset weight determination rule is to respectively configure the sample loss weight of the target pixel and the non-target pixel in a protection frame centered on the target pixel as a first non-zero value and zero;

[0018] determining the initial loss of each pixel sample in the network output result according to the class probability corresponding to the sample label of the pixel sample and the sample loss weight.

[0019] Optionally, the difficult negative sample screening rule constructed based on the initial loss of the pixel sample comprises:

[0020] screening a first initial difficult negative sample set from the initial negative sample set; the initial loss of the negative sample in the first initial difficult negative sample set is not less than the initial loss of other negative samples in the initial negative sample set;

[0021] determine a negative sample set to be removed from the first initial difficult negative sample set, and remove the negative sample set to be removed from the first initial difficult negative sample set to obtain a second initial difficult negative sample set; an initial loss of a negative sample in the negative sample set to be removed is not less than an initial loss of a negative sample in the second initial difficult negative sample set;

[0022] randomly select the first sample quantity of negative samples from the second initial difficult negative sample set to obtain a final difficult negative sample set.

[0023] Optionally, the screening the second sample quantity of negative samples from the initial negative sample set to obtain an easy negative sample set comprises:

[0024] randomly screening the second sample quantity of negative samples from the initial negative sample set to obtain an easy negative sample set;

[0025] Or, screening the second sample quantity of negative samples with an initial loss not greater than a preset loss threshold from the initial negative sample set to obtain an easy negative sample set.

[0026] Optionally, the generating the weight matrix for controlling the samples in the back propagation process according to the difficult negative sample set and the easy negative sample set comprises:

[0027] determining a weight coefficient corresponding to each pixel sample in a network output result map of the infrared small target detection network in a previous time based on a second preset weight determination rule to obtain a corresponding weight matrix for controlling the samples in the back propagation process;

[0028] The second preset weight determination rule comprises a weight coefficient determination rule and a weight coefficient constraint condition, the weight coefficient determination rule is to respectively configure a target pixel and a non-target pixel in a protection frame centered on the target pixel with a second non-zero value and zero as the weight coefficients corresponding to the target pixel and the non-target pixel respectively, and the weight coefficient constraint condition is that a sum of all weight coefficients in the weight matrix is equal to a sum of the positive sample quantity, the first sample quantity in the difficult negative sample set and the second sample quantity in the easy negative sample set.

[0029] In a second aspect, the present application provides an infrared small target detection network training device, comprising:

[0030] a difficult and easy negative sample set screening module configured to screen a difficult negative sample set and an easy negative sample set from an initial negative sample set based on a positive sample quantity and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset for improving the balance degree of the positive and negative sample quantities;

[0031] a weight matrix generation module configured to generate a weight matrix for controlling samples in a back propagation process according to the difficult negative sample set and the easy negative sample set;

[0032] a target detection network training module configured to determine a difficult point mining loss function based on the weight matrix, and perform point supervision training on the infrared small target detection network by using the difficult point mining loss function.

[0033] In a third aspect, the present application provides an electronic device, comprising:

[0034] a memory configured to save a computer program;

[0035] a processor configured to execute the computer program to implement the aforementioned infrared small target detection network training method.

[0036] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, the computer program being executed by a processor to implement the aforementioned infrared small target detection network training method.

[0037] In the present application, the difficult negative sample set and the easy negative sample set are selected from the initial negative sample set based on the number of positive samples and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset for improving the balance degree of the number of positive and negative samples; a weight matrix for controlling samples in a back propagation process is generated according to the difficult negative sample set and the easy negative sample set; a difficult point mining loss function is determined based on the weight matrix, and the infrared small target detection network is trained by point supervision using the difficult point mining loss function. In this way, by determining the difficult negative sample set and the easy negative sample set based on the number of positive samples and the preset proportion coefficient, and determining the difficult point mining loss function based on the weight matrix generated according to the difficult negative sample set and the easy negative sample set, the infrared small target detection network is trained by point supervision using the difficult point mining loss function, avoiding the sample imbalance problem caused by the too large number difference between positive and negative samples in the training sample, improving the balance degree of the number of positive and negative samples, and thus improving the training effect of the target detection network. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0039] Figure 1 a flow chart of the infrared small target detection network training method provided by the present application;

[0040] Figure 2 A specific infrared small target detection network training method flowchart is provided for the present application.

[0041] Figure 3 A structure schematic diagram of an infrared small target detection network training device is provided for the present application.

[0042] Figure 4 A structure diagram of an electronic device is provided for the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0044] Considering that a small target in an infrared image only occupies several to tens of pixels, the proportion of target pixels in all pixels is very small, which leads to an unbalanced number of positive and negative samples used in the training of an infrared small target detection network, thereby affecting the training effect of the target detection network. Therefore, the present application discloses an infrared small target detection network training method, which can prevent the unbalanced number of positive and negative samples in the training process of the target detection network, improve the balanced degree of the number of positive and negative samples, and thereby improve the training effect of the target detection network.

[0045] Referring to Figure 1 The embodiments of the present application disclose an infrared small target detection network training method, which comprises:

[0046] Step S11, based on the number of positive samples and a preset proportion coefficient, screening a difficult negative sample set and an easy negative sample set from an initial negative sample set; the preset proportion coefficient is a coefficient preset for improving the balanced degree of the number of positive and negative samples.

[0047] In the present embodiment, based on the number of pre-acquired positive samples and the preset proportion coefficient, the number of negative samples in the initial negative sample set is determined, and difficult negative samples and easy negative samples are screened from the initial negative sample set to obtain the difficult negative sample set and the easy negative sample set.

[0048] It should be noted that before the difficult negative sample set and the easy negative sample set are selected from the initial negative sample set, the method can further include: determining a sample loss weight of each pixel sample in a network output result map of the last time of the infrared small target detection network based on a first preset weight determination rule; the first preset weight determination rule is to configure the sample loss weight of the target pixel and the non-target pixel in the protection frame centered on the target pixel as a first non-zero value and zero respectively, wherein the first non-zero value is usually configured as 1; and determining an initial loss of each pixel sample in the network output result map according to a class probability corresponding to a sample label of the pixel sample and the sample loss weight. That is, considering that in the network output result map, there are non-target samples near the target pixels of some infrared small targets, which can interfere with detection, the sample loss weight of each pixel sample in the network output result map is determined before the difficult and easy negative samples are determined, according to the first preset weight determination rule, the sample loss weight of the target pixel in the protection frame centered on the target pixel is configured as the first non-zero value, and the sample loss weight of the non-target pixel in the protection frame is configured as zero, and then the initial loss of each pixel sample in the network output result map is determined according to the class probability corresponding to the sample label of the pixel sample and the sample loss weight. In this way, the non-target samples belonging to similar interference samples can be avoided from being selected as difficult negative samples to participate in back propagation, the accuracy of target detection is improved, and the performance of the target detection network is improved.

[0049] In step S12, a weight matrix for controlling samples in a back propagation process is generated according to the difficult negative sample set and the easy negative sample set.

[0050] It should be noted that, in the embodiment, the generating the weight matrix for controlling the samples in the back propagation process according to the hard negative sample set and the easy negative sample set can specifically include: determining the weight coefficients corresponding to each pixel sample in the network output result map of the infrared small target detection network in the last time based on a second preset weight determination rule, to obtain a corresponding weight matrix for controlling the samples in the back propagation process; wherein the second preset weight determination rule includes a weight coefficient determination rule and a weight coefficient constraint condition, the weight coefficient determination rule is to respectively configure the weight coefficients corresponding to the target pixels and the non-target pixels in the protection frame centered on the target pixels as a second non-zero value and zero, wherein the second non-zero value is usually configured as 1, and the weight coefficient constraint condition is that the sum of all weight coefficients in the weight matrix is equal to the sum of the number of positive samples, the number of first samples in the hard negative sample set and the number of second samples in the easy negative sample set. That is, the weight coefficients corresponding to each pixel sample in the network output result map are determined based on the second preset weight determination rule, the weight coefficients of the target pixels in the protection frame centered on the target pixels are configured as the second non-zero value, the weight coefficients of the non-target pixels in the protection frame are configured as zero, and the sum of all weight coefficients in the weight matrix is equal to the sum of the number of positive samples, the number of first samples in the hard negative sample set and the number of second samples in the easy negative sample set. In this way, the probability of false learning of the target detection network on the non-target pixels can be reduced, and the detection effect of the target detection network can be improved.

[0051] In step S13, a hard point mining loss function is determined based on the weight matrix, and the infrared small target detection network is point supervised trained by using the hard point mining loss function.

[0052] In the embodiment, the hard negative sample set and the easy negative sample set are filtered from the initial negative sample set based on the number of positive samples and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset for improving the balance degree of the number of positive and negative samples; a weight matrix for controlling the samples in the back propagation process is generated according to the hard negative sample set and the easy negative sample set; a hard point mining loss function is determined based on the weight matrix, and the infrared small target detection network is point supervised trained by using the hard point mining loss function. In this way, by determining the hard negative sample set and the easy negative sample set based on the number of positive samples and the preset proportion coefficient, and determining the hard point mining loss function based on the weight matrix generated according to the hard negative sample set and the easy negative sample set, the infrared small target detection network is point supervised trained by using the hard point mining loss function, the sample imbalance problem caused by the too large number difference between the positive and negative samples in the training samples is avoided, the balance degree of the number of positive and negative samples is improved, and the training effect of the target detection network is improved.

[0053] Referring toFigure 2 As shown, the embodiment of the present application discloses a specific infrared small target detection network training method, comprising:

[0054] Step S21, determining the first sample quantity in the difficult negative sample set and the second sample quantity in the easy negative sample set based on the positive sample quantity and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset for improving the balance degree of the positive and negative sample quantities.

[0055] In the embodiment, it is pointed out that the determination of the first sample quantity in the difficult negative sample set and the second sample quantity in the easy negative sample set based on the positive sample quantity and the preset proportion coefficient can specifically include: determining the total quantity of difficult and easy negative samples based on the positive sample quantity and a first preset proportion coefficient for improving the balance degree of the positive and negative sample quantities; the first preset proportion coefficient is a coefficient preset for representing the ratio between the positive sample quantity and the total quantity of difficult and easy negative samples; determining the first sample quantity in the difficult negative sample set and the second sample quantity in the easy negative sample set based on the total quantity of difficult and easy negative samples and a second preset proportion coefficient; the second preset proportion coefficient is a coefficient preset for representing the ratio between the difficult negative sample quantity and the easy negative sample quantity. The first preset proportion coefficient and the second preset proportion coefficient can both be set by the user according to the user's own needs, for example, the user can set the first preset proportion coefficient to 1 / 39 and the second preset proportion coefficient to 10 / 29, that is, the quantity ratio between the positive samples and the difficult and easy negative samples is 1 / 39, and the quantity ratio between the difficult negative samples and the easy negative samples is 10 / 29. In the traditional case, the quantity deviation between the positive samples and the negative samples is very large, which can usually reach 1 to 100 or even smaller. As can be seen, the first preset proportion coefficient is adjusted based on the balance requirement between the positive and negative samples in advance, which can achieve the purpose of improving the balance degree of the positive and negative sample quantities.

[0056] Step S22, screening the first sample quantity of negative samples from the initial negative sample set by using the difficult negative sample screening rule constructed based on the initial loss of the pixel sample, to obtain the difficult negative sample set.

[0057] It should be noted that, in the embodiment, the difficult negative sample screening rule constructed based on the initial loss of the pixel sample is used to screen the first sample number of negative samples from the initial negative sample set, and specifically can include: screening a first initial difficult negative sample set from the initial negative sample set; the initial loss of the negative sample in the first initial difficult negative sample set is not less than the initial loss of other negative samples in the initial negative sample set; determining a negative sample set to be removed from the first initial difficult negative sample set, and removing the negative sample set to be removed from the first initial difficult negative sample set to obtain a second initial difficult negative sample set; the initial loss of the negative sample in the negative sample set to be removed is not less than the initial loss of the negative sample in the second initial difficult negative sample set; and the first sample number of negative samples are randomly screened from the second initial difficult negative sample set to obtain a final difficult negative sample set. That is, based on the initial loss value of the negative sample in the initial negative sample set, a number of negative samples with high initial loss values are screened to obtain a first initial difficult negative sample set. Considering that the initial loss values of some negative samples may be misjudged, resulting in abnormally high initial loss values of the above-mentioned negative samples, the above-mentioned negative samples will reduce the detection effect of the target detection network when participating in back propagation. Therefore, a number of negative samples with high initial loss values in the first initial difficult negative sample set are determined as a negative sample set to be removed to obtain a negative sample set to be removed, the negative sample set to be removed is removed from the first initial difficult negative sample set to obtain a second initial difficult negative sample set, and then the first sample number of negative samples are randomly screened from the second initial difficult negative sample set to obtain a final difficult negative sample set for back propagation. It can be understood that the number of samples in the first initial difficult negative sample set, the second initial difficult negative sample set and the negative sample set to be removed can be set by the user as required.

[0058] Step S23, screening the second sample number of negative samples from the initial negative sample set to obtain an easy negative sample set.

[0059] In the embodiment, it should be noted that the second sample number of negative samples are screened from the initial negative sample set to obtain an easy negative sample set, which can specifically include but is not limited to the following two specific implementations.

[0060] In the first specific implementation, the second sample number of negative samples are screened from the initial negative sample set to obtain an easy negative sample set, which can specifically include: randomly screening the second sample number of negative samples from the initial negative sample set to obtain an easy negative sample set. That is, the second sample number of easy negative samples are determined from the initial negative sample set by using a random screening method to obtain an easy negative sample set.

[0061] In the second specific implementation, the filtering of the second sample quantity of negative samples from the initial negative sample set to obtain the easy negative sample set can specifically include: filtering the second sample quantity of negative samples with initial loss not greater than the preset loss threshold from the initial negative sample set to obtain the easy negative sample set. That is, a loss threshold is preset, and the negative samples with initial loss not greater than the preset loss threshold in the initial negative sample set are determined as easy negative samples to obtain the easy negative sample set.

[0062] Step S24: generating a weight matrix for controlling samples in a back propagation process according to the difficult negative sample set and the easy negative sample set;

[0063] Step S25: determining a difficult point mining loss function based on the weight matrix, and performing point supervision training on the infrared small target detection network by using the difficult point mining loss function.

[0064] The specific process of steps S24 and S25 can refer to the corresponding content disclosed in the foregoing embodiments, which will not be described here again.

[0065] In this embodiment, the first sample quantity in the difficult negative sample set and the second sample quantity in the easy negative sample set are determined based on the positive sample quantity and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset for improving the balance degree of the positive and negative sample quantities; the difficult negative sample filtering rule constructed based on the initial loss of the pixel sample is used to filter the first sample quantity of negative samples from the initial negative sample set to obtain the difficult negative sample set; the second sample quantity of negative samples is filtered from the initial negative sample set to obtain the easy negative sample set. The weight matrix for controlling samples in a back propagation process is generated according to the difficult negative sample set and the easy negative sample set; the difficult point mining loss function is determined based on the weight matrix, and the point supervision training is performed on the infrared small target detection network by using the difficult point mining loss function. In this way, the sample quantities of the difficult negative samples and the easy negative samples are determined based on the positive sample quantity and the preset proportion coefficient, the sample imbalance problem caused by the too large quantity difference between the positive and negative samples in the training samples is avoided, the balance degree of the positive and negative sample quantities is improved, the quantity relationship of the difficult and easy negative samples is controlled, the difficult negative sample set is determined by using the difficult negative sample filtering rule constructed based on the initial loss of the sample, the influence of the samples with abnormal loss values on the training effect is reduced, and thus the training effect of the target detection network is improved.

[0066] The specific implementation of the present application will be described below in combination with an experiment conducted for the present application.

[0067] In the process of infrared small target detection, the imbalance between the number of positive samples and negative samples is one of the problems that still need to be solved. Considering the low-rank characteristics of negative samples in infrared images, difficult example mining can be used to reduce the repeated learning of a large number of easy negative samples and make the network pay more attention to difficult negative samples in infrared images. Difficult example mining can reduce the imbalance between difficult and easy negative samples and improve the performance of the infrared small target detection network

[0068] Since the current common method for infrared small target detection is designed for general object detection that outputs bounding boxes. The difficult samples used in the detection process are all from the regions of interest generated in the middle of the detection network. However, in the infrared small target detection based on segmentation, each pixel is regarded as a sample, and no box is generated from input to output in the detection process. Therefore, the above existing method cannot be directly applied to the end-to-end infrared small target detection network, and therefore the present application makes some special difficult example mining design for the infrared small target detection network based on segmentation.

[0069] Experimental preparation: This experiment uses 32 sequences of different scenes, and uses point annotation to make target labels. According to the signal-to-noise ratio, 22 real sequences are used as the training set, and 10 sequences are used as the test set. In the test set, 6 sequences are real scenes, which have similar data distribution with the training set, and the other 4 sequences are scenes with simulated targets added to the real background. Among them, the above four simulated sequences with different data distributions can be selected to verify the generalization ability of the deep learning-based method in different data distribution sequences. Most of the 10 sequences in the test set have weak infrared small targets, less than 5x5 pixels, and complex background, so that the performance of the algorithm in complex situations can be more concise and clear. In addition, the gray scale of the real sequence in the experiment is 4096, and the gray scale of the simulated sequence is 256.

[0070] The experiment initializes the weights of the convolutional layer using the Kai Ming method, and uses a uniform distribution of random initialization method to initialize the bias value. In the experiment, the input of the 5 discontinuous frames with long time domain information is 512x512 for each frame, the batch size is set to 2, the initial learning rate is set to 0.001, and all training is stopped after 10 epochs. The evaluation parameters used in the experiment for evaluation include detection rate and false alarm rate, where the detection rate represents the ratio between the number of correctly detected targets and the total number of targets, and the false alarm rate represents the ratio between the number of false alarm pixels and the total number of pixels in the image.

[0071] Experimental process: First, determine the initial loss L1 of all pixel samples in the network output result image using the following formula.

[0072]

[0073] where y e {0, 1} is the sample label, pi e [0, 1] is the model estimated probability of the class with label y = 1, and pi e Pi. wp represents the value of the sample loss weight, which aims to prevent non-target pixel samples around the target pixel sample from being selected as hard examples. Considering that some infrared targets have similar interference objects such as tail flames, etc., which may interfere with detection, causing the detected target centroid to move, a number of protection boxes centered on the target pixel are set to control the loss of the pixels within the box. Specifically, the value of w p for the target pixel samples in the protection box can be set to 1, the value of w p for the non-target pixel samples can be set to 0, and the value of w p for the pixels outside the protection box can be set to 0. In this way, samples belonging to similar interference objects can be avoided from being selected to participate in backpropagation. Therefore, the above-mentioned samples belonging to similar interference objects can be avoided from being learned as background.

[0074] In order to obtain the difficult negative sample set H, a total difficult negative sample set T is generated from the entire negative sample set S containing the difficult negative sample set H.

[0075] Firstly, the initial loss of all negative samples can be sorted to obtain the u negative samples with the highest loss. Generally, a higher loss indicates that the sample is more difficult. These negative samples with high initial loss form a first initial difficult negative sample set. Considering that there may be some negative samples whose initial loss values are misjudged, resulting in abnormally high initial loss values of the above-mentioned negative samples, which will reduce the detection effect of the target detection network when participating in backpropagation, the v samples with the highest loss are removed from the first initial difficult negative sample set to reduce the probability of selecting abnormal values. The second initial difficult negative sample set T is obtained by the following formula.

[0076] T = {S(top(L1, u))} - {S(top(L1, v))}

[0077] Then, b hn samples are randomly selected from T as the final difficult negative sample set H for training to calculate the final loss.

[0078] H = Random(T, b hn )

[0079] In addition to the selected difficult negative sample set H, some easy negative samples can also be selected to participate in backpropagation to improve the robustness of the network. b en samples can be randomly selected from the initial negative sample set as the easy negative sample set E.

[0080] First, according to the hard negative sample set H and the easy negative sample set E, a weight matrix W used in the final loss function is generated to control the samples involved in back propagation. In this embodiment, the weight matrix W determines which samples can participate in loss calculation. In the weight matrix W, the weight coefficients corresponding to the non-zero values (usually configured as 1) can participate in the subsequent loss calculation, and the weight coefficients corresponding to the zero values cannot participate in the subsequent loss calculation. In this embodiment, the HPM loss (i.e. Hard Point Mining loss) function used by the present application is calculated as follows:

[0081] HPM(p t )=∑-αW(1-p t ) γ log(p t )

[0082]

[0083] wherein, b p represents the number of positive samples. α ∈ [0, 1] is the weight value of the sample with a label category of 1, that is, the specific weight value of the sample corresponding to the non-zero weight coefficient in the weight matrix W is assigned when participating in the subsequent loss calculation, 1-α is the weight value of the sample with a label category of 0, and γ represents the adjustable focal point parameter, which can be set to 2. In this embodiment, the ratio of the number of hard and easy negative samples can be set to λ = 10 / 29, that is, b hn / b en = 10 / 29, and the ratio of the number of positive samples to the number of hard and easy negative samples can be set to μ = 1 / 39, that is, b p / (b en +b hn ) = 1 / 39. In this way, the number balance between positive and negative samples can be improved, and the hard samples can be mined.

[0084] Experimental results: The data of the experimental results are shown in Table 1.

[0085] Table 1

[0086]

[0087] Compared with the present application, if the cross-entropy loss function is used, the detection rate in the final total result will decrease by 5.20%, and the false alarm rate will increase by 1.41 x 10 -6This is because the cross-entropy loss function method can guide the target detection network to learn more non-target pixel samples that dominate the loss, resulting in insufficient learning of target pixel samples by the network. If the focal loss is used as the loss function, the detection rate will decrease by 5.30%, and the false alarm rate will increase by 2.11x10 -6 In addition, Table 4 shows that the performance of the focal loss is similar to that of the cross-entropy loss. This indicates that the focal loss method is not suitable for balancing positive and negative samples during the training of the infrared small target detection network.

[0088] Compared with the method provided in the present application, the target detection network trained by the Top-K loss function has a detection rate decrease of 1.43% and a false alarm rate increase of 1.87x10 -6 However, compared with the cross-entropy loss function and the focal loss function, the detection rate of the Top-K method is increased by 3.77% and 3.87%, respectively. In addition to the target pixel samples, the Top-K loss function only focuses on the 39 non-target pixel samples with the highest loss, which indicates that learning difficult samples can improve the performance of the target detection network, as it helps the model to correctly distinguish clutter and targets. However, the Top-K loss function can be affected by outliers in the training set, which in the experiment are mainly non-target pixel samples highly correlated with target pixel samples. Learning the above non-target pixel samples can lead to a decrease in both the detection rate and the false alarm rate. The method provided in the present application reduces the influence of outliers during training and shows the best performance compared with other methods.

[0089] Referring to Figure 3 The embodiment of the present application discloses an infrared small target detection network training device, which comprises:

[0090] The difficult and easy negative sample set screening module 11 is configured to screen a difficult negative sample set and an easy negative sample set from the initial negative sample set based on the number of positive samples and a preset proportion coefficient; the preset proportion coefficient is a coefficient that is preset to improve the balance of the number of positive and negative samples;

[0091] The weight matrix generation module 12 is configured to generate a weight matrix for controlling samples in the back propagation process according to the difficult negative sample set and the easy negative sample set;

[0092] The target detection network training module 13 is configured to determine a difficult point mining loss function based on the weight matrix, and perform point supervision training on the infrared small target detection network using the difficult point mining loss function.

[0093] In the embodiment, the difficult negative sample set and the easy negative sample set are selected from the initial negative sample set based on the number of positive samples and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset for improving the balance degree of the number of positive and negative samples; a weight matrix for controlling samples in the back propagation process is generated according to the difficult negative sample set and the easy negative sample set; and a difficult point mining loss function is determined based on the weight matrix, and the infrared small target detection network is point supervised and trained by using the difficult point mining loss function. In this way, by determining the difficult negative sample set and the easy negative sample set based on the number of positive samples and the preset proportion coefficient, and determining the difficult point mining loss function based on the weight matrix generated according to the difficult negative sample set and the easy negative sample set, the infrared small target detection network is point supervised and trained by using the difficult point mining loss function, the sample imbalance problem caused by the too large number difference between positive and negative samples in the training sample is avoided, the balance degree of the number of positive and negative samples is improved, and the training effect of the target detection network is improved.

[0094] In some specific embodiments, the difficult and easy negative sample set screening module 11 can specifically include:

[0095] The number determination sub-module is configured to determine the first sample number in the difficult negative sample set and the second sample number in the easy negative sample set based on the number of positive samples and a preset proportion coefficient.

[0096] The difficult negative sample screening sub-module is configured to screen the first sample number of negative samples from the initial negative sample set by using a difficult negative sample screening rule constructed based on the initial loss of the pixel sample, to obtain the difficult negative sample set.

[0097] The easy negative sample screening sub-module is configured to screen the second sample number of negative samples from the initial negative sample set, to obtain the easy negative sample set.

[0098] In some specific embodiments, the number determination sub-module can specifically include:

[0099] The total sample number determination unit is configured to determine the total number of difficult and easy negative samples based on the number of positive samples and a first preset proportion coefficient for improving the balance degree of the number of positive and negative samples; the first preset proportion coefficient is a coefficient preset for representing the ratio between the number of positive samples and the total number of difficult and easy negative samples.

[0100] The first sample number and second sample number determination unit is configured to determine the first sample number in the difficult negative sample set and the second sample number in the easy negative sample set based on the total number of difficult and easy negative samples and a second preset proportion coefficient; the second preset proportion coefficient is a coefficient preset for representing the ratio between the number of difficult negative samples and the number of easy negative samples.

[0101] In some embodiments, the infrared small target detection network training apparatus can further include:

[0102] a loss weight determination module configured to determine sample loss weights of each pixel sample in a last network output result map of the infrared small target detection network based on a first preset weight determination rule; the first preset weight determination rule is to configure the sample loss weights of target pixels and non-target pixels in a protection frame centered on the target pixels as a first non-zero value and zero, respectively;

[0103] an initial loss determination module configured to determine initial losses of each pixel sample in the network output result map according to a class probability corresponding to a sample label of the pixel sample and the sample loss weight.

[0104] In some embodiments, the difficult negative sample screening sub-module can specifically include:

[0105] a first initial difficult negative sample determination unit configured to screen a first initial difficult negative sample set from the initial negative sample set; the initial loss of a negative sample in the first initial difficult negative sample set is not less than the initial loss of other negative samples in the initial negative sample set;

[0106] a second initial difficult negative sample determination unit configured to determine a negative sample set to be removed from the first initial difficult negative sample set and remove the negative sample set to be removed from the first initial difficult negative sample set to obtain a second initial difficult negative sample set; the initial loss of a negative sample in the negative sample set to be removed is not less than the initial loss of a negative sample in the second initial difficult negative sample set;

[0107] a difficult negative sample determination unit configured to randomly screen the first sample quantity of negative samples from the second initial difficult negative sample set to obtain a final difficult negative sample set.

[0108] In some embodiments, the easy negative sample screening sub-module can specifically include:

[0109] a first easy negative sample determination unit configured to randomly screen the second sample quantity of negative samples from the initial negative sample set to obtain an easy negative sample set;

[0110] a second easy negative sample determination unit configured to screen the second sample quantity of negative samples with initial losses not greater than a preset loss threshold from the initial negative sample set to obtain an easy negative sample set.

[0111] In some embodiments, the weight matrix generation module 12 can specifically include:

[0112] The weight coefficient determination unit is configured to determine weight coefficients corresponding to each pixel sample in a network output result map of the infrared small target detection network in a previous time based on a second preset weight determination rule, so as to obtain a corresponding weight matrix used for controlling samples in a back propagation process; wherein the second preset weight determination rule comprises a weight coefficient determination rule and a weight coefficient constraint condition, the weight coefficient determination rule is to respectively configure weight coefficients corresponding to target pixels and non-target pixels in a protection frame centered on the target pixels as a second non-zero value and zero, and the weight coefficient constraint condition is that a sum of all weight coefficients in the weight matrix is equal to a sum of the positive sample quantity, a first sample quantity in the difficult negative sample set and a second sample quantity in the easy negative sample set.

[0113] Further, the embodiment of the present application further discloses an electronic device, and the structure diagram of the electronic device 20 is shown according to an example embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0114] Figure 4 A structure schematic diagram of an electronic device 20 provided by the embodiment of the present application is provided. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, the computer program is loaded and executed by the processor 21, so as to realize the related steps in the infrared small target detection network training method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0115] In the embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0116] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, and the resources stored thereon can include an operating system 221, a computer program 222 and the like, and the storage mode can be temporary storage or permanent storage.

[0117] The operating system 221 is configured to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program for implementing the infrared small target detection network training method disclosed in any of the above embodiments, the computer program 222 can further include computer programs for performing other specific tasks.

[0118] Further, the present application also discloses a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the above-mentioned infrared small target detection network training method. For the specific steps of the method, please refer to the corresponding content disclosed in the above embodiments, which will not be repeated here.

[0119] The embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the relevant part.

[0120] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the composition and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0121] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0122] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be intended to distinguish one entity or operation from another entity or operation without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the stated element.

[0123] The above describes the technical solutions provided by the present application in detail, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the examples is only for helping to understand the method and core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description of the specification should not be understood as limiting the present application.

Claims

1. A method for training an infrared small target detection network, characterized in that, The method comprises the steps of: screening a difficult negative sample set and an easy negative sample set from an initial negative sample set based on the number of positive samples and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset for improving the balance of the number of positive and negative samples; generating a weight matrix for controlling samples in a back propagation process according to the difficult negative sample set and the easy negative sample set; determining a difficult point mining loss function based on the weight matrix, and performing point supervision training on the infrared small target detection network by using the difficult point mining loss function; wherein the generating a weight matrix for controlling samples in a back propagation process according to the difficult negative sample set and the easy negative sample set comprises: determining the weight coefficients corresponding to each pixel sample in the network output result map of the infrared small target detection network in the last time based on a second preset weight determination rule, to obtain a corresponding weight matrix for controlling samples in a back propagation process; wherein the second preset weight determination rule comprises a weight coefficient determination rule and a weight coefficient constraint condition, the weight coefficient determination rule is to configure the weight coefficients corresponding to the target pixels and non-target pixels in the protection frame centered on the target pixels as a second non-zero value and zero respectively, and the weight coefficient constraint condition is that the sum of all weight coefficients in the weight matrix is equal to the sum of the number of positive samples, the number of first samples in the difficult negative sample set and the number of second samples in the easy negative sample set; the screening a difficult negative sample set and an easy negative sample set from an initial negative sample set based on the number of positive samples and a preset proportion coefficient comprises: determining the number of first samples in the difficult negative sample set and the number of second samples in the easy negative sample set based on the number of positive samples and a preset proportion coefficient; screening the first sample number of negative samples from the initial negative sample set by using a difficult negative sample screening rule constructed based on an initial loss of pixel samples, to obtain a difficult negative sample set; screening the second sample number of negative samples from the initial negative sample set, to obtain an easy negative sample set.

2. The infrared small target detection network training method according to claim 1, characterized in that, the determining the number of first samples in the difficult negative sample set and the number of second samples in the easy negative sample set based on the number of positive samples and a preset proportion coefficient comprises: determining the total number of difficult and easy negative samples based on the number of positive samples and a first preset proportion coefficient for improving the balance of the number of positive and negative samples; the first preset proportion coefficient is a coefficient preset for representing the ratio between the number of positive samples and the total number of difficult and easy negative samples; determining the number of first samples in the difficult negative sample set and the number of second samples in the easy negative sample set based on the total number of difficult and easy negative samples and a second preset proportion coefficient; the second preset proportion coefficient is a coefficient preset for representing the ratio between the number of difficult negative samples and the number of easy negative samples.

3. The infrared small target detection network training method according to claim 1, characterized in that, before the screening the first sample number of negative samples from the initial negative sample set, the method further comprises the steps of: determine a sample loss weight of each pixel sample in a network output result graph of the infrared small target detection network last time based on a first preset weight determination rule; the first preset weight determination rule is to respectively configure a target pixel and a non-target pixel in a protection frame centered on the target pixel with a first non-zero value and zero as the respective sample loss weights of the target pixel and the non-target pixel; determine an initial loss of each pixel sample in the network output result graph according to a class probability corresponding to a sample label of the pixel sample and the sample loss weight.

4. The infrared small target detection network training method according to claim 1, characterized in that, the hard negative sample screening rule constructed based on the initial loss of the pixel sample is used to screen the first sample number of negative samples from the initial negative sample set, including: screening a first initial hard negative sample set from the initial negative sample set; the initial loss of a negative sample in the first initial hard negative sample set is not less than the initial loss of other negative samples in the initial negative sample set; determine a negative sample set to be removed from the first initial hard negative sample set, and remove the negative sample set to be removed from the first initial hard negative sample set to obtain a second initial hard negative sample set; the initial loss of a negative sample in the negative sample set to be removed is not less than the initial loss of a negative sample in the second initial hard negative sample set; randomly screen the first sample number of negative samples from the second initial hard negative sample set to obtain a final hard negative sample set.

5. The infrared small target detection network training method according to claim 1, characterized in that, the second sample number of negative samples are screened from the initial negative sample set to obtain an easy negative sample set, including: randomly screen the second sample number of negative samples from the initial negative sample set to obtain an easy negative sample set; or, screen the second sample number of negative samples with an initial loss not greater than a preset loss threshold from the initial negative sample set to obtain an easy negative sample set.

6. An infrared small target detection network training apparatus, characterized in that, including: a hard and easy negative sample set screening module configured to screen a hard negative sample set and an easy negative sample set from an initial negative sample set based on a positive sample number and a preset proportion coefficient; the preset proportion coefficient is a coefficient preset to improve the balance degree of the positive and negative sample numbers; a weight matrix generation module configured to generate a weight matrix for controlling samples in a back propagation process according to the hard negative sample set and the easy negative sample set; a target detection network training module configured to determine a hard point mining loss function based on the weight matrix, and perform point supervision training on an infrared small target detection network using the hard point mining loss function; wherein the weight matrix generation module includes: a weight coefficient determination unit configured to determine a weight coefficient corresponding to each pixel sample in a network output result graph of the infrared small target detection network last time based on a second preset weight determination rule to obtain a corresponding weight matrix for controlling samples in a back propagation process. The second preset weight determination rule includes a weight coefficient determination rule and a weight coefficient constraint condition. The weight coefficient determination rule is to configure the weight coefficients corresponding to the target pixels and the non-target pixels in the protection frame centered on the target pixels as the second non-zero value and zero, respectively. The weight coefficient constraint condition is that the sum of all weight coefficients in the weight matrix is equal to the sum of the positive sample quantity, the first sample quantity in the hard negative sample set, and the second sample quantity in the easy negative sample set. The hard and easy negative sample set screening module includes: A quantity determination sub-module configured to determine the first sample quantity in the hard negative sample set and the second sample quantity in the easy negative sample set based on the positive sample quantity and a preset proportion coefficient. A hard negative sample screening sub-module configured to screen the first sample quantity of negative samples from the initial negative sample set by using a hard negative sample screening rule constructed based on the initial loss of the pixel samples, to obtain the hard negative sample set. An easy negative sample screening sub-module configured to screen the second sample quantity of negative samples from the initial negative sample set, to obtain the easy negative sample set.

7. An electronic device, comprising: It includes: A memory configured to save a computer program; A processor configured to execute the computer program to implement the infrared small target detection network training method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer program is saved, and the computer program is executed by a processor to implement the infrared small target detection network training method according to any one of claims 1 to 5.

Images