Semi-supervised target identification method for airborne photoelectric image

Abstract

The invention belongs to the technical field of computer vision and deep learning. The invention provides a semi-supervised target identification method for an airborne photoelectric image. According to the embodiment of the invention, the parameters of the teacher network model are updated through the exponential moving average algorithm, and the quality of the generated pseudo labels is continuously improved, so that the non-labeled data can train the student network more effectively, and the airborne photoelectric image target recognition network model is obtained. Through a pseudo label generation strategy and in combination with a semi-supervised teacher-student network joint training framework, the problem that an unlabeled data sample is difficult to effectively utilize is solved, and after unlabeled sample data is added into training, the target identification capability of an airborne photoelectric image is improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of computer vision and deep learning technology, and in particular to a semi-supervised target recognition method for airborne optoelectronic images. Background Art

[0002] As deep learning technology becomes increasingly mature, target recognition algorithms based on deep learning have achieved remarkable results, but their effectiveness depends largely on the quality and quantity of sample annotations in the application scenario. In the recognition task of airborne optoelectronic images, ground targets need to be identified at high altitudes, and image data is usually taken by airborne optoelectronic sensors. Aircraft accumulate a large amount of image data during flight, but it is very time-consuming and labor-intensive to annotate a large amount of image data, and the annotation cost is extremely high. After acquiring new data, the model needs to be iteratively trained. The speed of manual annotation of collected data cannot meet the speed of new data generation, and a large number of unlabeled samples are still difficult to use. Therefore, through semi-supervised target recognition technology, a part of labeled data and a part of unlabeled data are used for joint training. While reducing manual annotation, the information in unlabeled samples is mined, the recognition model's understanding of the probability of data edge distribution is enriched, and the training samples are further constrained on the decision boundary to achieve effective utilization of unlabeled data, achieving better recognition effect than using only labeled data for supervised learning.

[0003] How to improve the utilization rate of existing labeled samples, how to mine the information of unlabeled samples so that they can participate in model training, and how to quickly make the model have higher recognition accuracy in this scenario with as few annotations as possible are practical problems that need to be considered and solved urgently.

[0004] When a large amount of new unlabeled data is obtained, it is extremely costly to fully label it and then conduct supervised training, and the new unlabeled data cannot be used. Existing methods do not consider using partially labeled data and no data for joint semi-supervised training to improve the performance of existing models; do not consider more sophisticated pseudo-label allocation strategies to improve the effect of semi-supervised training; and do not consider the correlation between target categories and target locations when predicting models. Existing methods use existing models to predict high-confidence pseudo-labels and corresponding unlabeled sample images to optimize the model. Since the existing model has a high confidence in the selected pseudo-labels, these samples have a limited effect on improving the performance of the existing model; existing methods do not consider using relatively low-confidence pseudo-labels and corresponding unlabeled sample images to optimize the model. Directly discarding these pseudo-labels as negative samples will reduce the improvement of the performance of the existing model.

[0005] In summary, when using partly labeled data and partly unlabeled data, it is still a huge challenge to further improve the recognition effect of existing models trained with labeled data on targets in airborne electro-optical images.

[0006] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the invention

[0008] The purpose of the embodiments of the present disclosure is to provide a semi-supervised target recognition method for airborne electro-optical images, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0009] According to an embodiment of the present disclosure, a semi-supervised target recognition method for airborne optoelectronic images is provided, the method comprising: Acquire an image sequence, and divide the image sequence into a training data set and a test data set according to a first preset ratio; Annotating the images in the training data set according to a second preset ratio to obtain a labeled sample set and an unlabeled sample set, and performing data enhancement on the unlabeled sample set and the labeled sample set; Constructing a semi-supervised teacher-student network joint training framework; wherein the semi-supervised teacher-student network joint training framework includes a teacher network and a student network; Inputting the enhanced unlabeled sample set into the teacher network for reasoning to generate pseudo labels, and dividing the pseudo labels according to types; wherein the pseudo labels include reliable pseudo labels, uncertain pseudo labels and erroneous pseudo labels; Constructing a joint loss function based on the supervised loss of the labeled samples and the unsupervised loss of the unlabeled samples; Inputting the enhanced unlabeled sample set, the labeled sample set and the pseudo-label into the student network for training, and combining the joint loss function to obtain a first weight parameter constraining the student network; After processing the first weight parameter using the EMA algorithm, the first weight parameter is loaded into the teacher network to update the second weight parameter of the teacher network; Iteratively training the student network using the updated teacher network until a trained student network is obtained; The test data set is input into the trained student network to obtain the target position and predicted category in the test sample.

[0010] Furthermore, the step of acquiring an image sequence and dividing the image sequence into a training data set and a test data set according to a first preset ratio includes: Use the optoelectronic equipment on the airborne platform to collect optoelectronic videos of different scenes; Performing frame extraction processing on the optoelectronic video to obtain the corresponding image sequence; Adjusting the size of the image sequence so that the image resolution of the image sequence is a preset value; The image sequence is divided into the training data set and the test data set according to a 7:3 ratio.

[0011] Further, the steps of annotating the images in the training data set according to a second preset ratio to obtain an annotated sample set and an unannotated sample set, and performing data enhancement on the unannotated sample set and the annotated sample set include: Divide the training data set into the unlabeled data set and the labeled data set according to an 8:2 ratio, and label the images in the labeled data set; Building a data enhancement library based on image transformation methods; wherein the image transformation methods include contrast transformation, brightness transformation, mirror transformation, image translation, image rotation, image noise addition, motion blur, adding clouds and fog, and adding rain and snow; According to the airborne scene, several corresponding image transformation methods are selected to perform data enhancement on the unlabeled sample set and the labeled sample set.

[0012] Furthermore, the contrast transformation and the brightness transformation are not used at the same time, the addition of clouds and fog and the addition of rain and snow are not used at the same time, the image noise is selected with a probability of 0.1, and the motion model is selected with a probability of 0.2.

[0013] Furthermore, the steps of constructing a semi-supervised teacher-student network joint training framework include: A semi-supervised teacher-student network joint training framework is constructed. The framework consists of the teacher network and the student network with the same structure but different parameter initializations. The initial parameters of the teacher network and the student network are both generated by random initialization, and the network structure adopts YOLOv10.

[0014] Furthermore, the step of inputting the enhanced unlabeled sample set into the teacher network for reasoning to generate pseudo labels, and dividing the pseudo labels according to types includes: Inputting the enhanced unlabeled sample set into the teacher network to generate the pseudo labels; Set high and low thresholds; Defining the pseudo-label whose confidence is greater than the high threshold as the high-quality pseudo-label; defining the pseudo-label with a confidence level between the high threshold and the low threshold as the uncertain pseudo-label; The pseudo label whose confidence is less than the low threshold is defined as the erroneous pseudo label.

[0015] Furthermore, based on the supervised loss of the labeled samples and the unsupervised loss of the unlabeled samples, the step of constructing a joint loss function includes: Constructing a supervised loss according to the target location, category, confidence and annotation information in the labeled sample set; According to the reliable pseudo-label, the uncertain pseudo-label and the erroneous pseudo-label, the CIOU between the corresponding target prediction results of the teacher network and the student network is calculated to obtain a target score; constructing an unsupervised loss based on the target score; The joint loss function is obtained according to the supervised loss and the unsupervised loss.

[0016] Furthermore, the expression of the supervised loss is:

[0017] in, represents the cross entropy loss function, represents the position in the output feature map, Student network feature map The category score output at Student network feature map The position score of the output, Student network feature map The foreground score output at represents the CIOU calculation function, is the true label in the feature map The category score at is the true label in the feature map The position score is is the true label in the feature map Prospect score at The expression of the unsupervised loss is:

[0018]

[0019]

[0020]

[0021] in, is the unsupervised category loss, is the unsupervised position loss, is the unsupervised foreground loss, and After pseudo-label assignment and screening, in the feature map The category score, location score and foreground score of the corresponding feature point target; for Target score for position pseudo-labels; for Classification scores of position pseudo labels; Represents the indicator function. When the condition is met, its value is 1, and when the condition is not met, its value is 0; the confidence is greater than the high threshold Pseudo-label participation Calculation of; confidence is greater than the high threshold Or pseudo labels with a target score greater than 0.9 participate Calculation of; confidence is greater than the high threshold and less than the lower threshold Pseudo-label participation Calculation of whether there is a foreground target in the and low threshold The pseudo-labels between them associate the confidence scores of foreground and background with the consistency of target location prediction; The expression of the joint loss function is:

[0022] in, indicates that there is a supervision loss, and represents the unsupervised loss, is the balance coefficient.

[0023] Furthermore, the step of iteratively training the student network using the updated teacher network until the trained student network is obtained includes: The teacher network generates the pseudo labels, and the pseudo labels are used in the training of the student network to update the first weight parameters; The student network loads the updated first weight parameter into the teacher network after being processed by the EMA algorithm to update the second weight parameter; The student network is iteratively trained until a trained student network is obtained.

[0024] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: In the embodiment of the present disclosure, the above-mentioned semi-supervised target recognition method for airborne optoelectronic images is used to update the parameters of the teacher network model through the exponential moving average algorithm (EMA), continuously improve the quality of the generated pseudo-labels, so that the unlabeled data can be used to train the student network more effectively, and obtain the airborne optoelectronic image target recognition network model. Through the pseudo-label generation strategy, combined with the semi-supervised teacher-student network joint training framework, the problem of the difficulty in effectively utilizing unlabeled data samples is solved. After adding the unlabeled sample data to the training, the target recognition capability of airborne optoelectronic images is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0026] Figure 1 A step diagram showing a semi-supervised target recognition method for airborne electro-optical images in an exemplary embodiment of the present disclosure; Figure 2 A schematic diagram showing the overall structure of a semi-supervised teacher-student network joint training framework in an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic diagram showing an original image of a portion of an optoelectronic image in an airborne scene in an exemplary embodiment of the present disclosure; Figure 4 A first set of experimental result diagrams showing the model before and after training of the semi-supervised target recognition method for airborne optoelectronic images in an exemplary embodiment of the present disclosure; Figure 5 A second set of experimental result graphs showing the model before and after training of the semi-supervised target recognition method for airborne electro-optical images in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0029] This example embodiment provides a semi-supervised target recognition method for airborne optoelectronic images. Figure 1 As shown in , the semi-supervised target recognition method for airborne optoelectronic images may include: steps S101 to S109.

[0030] Step S101: acquiring an image sequence, and dividing the image sequence into a training data set and a test data set according to a first preset ratio; Step S102: annotating the images in the training data set according to a second preset ratio to obtain an annotated sample set and an unannotated sample set, and performing data enhancement on the unannotated sample set and the annotated sample set; Step S103: constructing a semi-supervised teacher-student network joint training framework; wherein the semi-supervised teacher-student network joint training framework includes a teacher network and a student network; Step S104: inputting the enhanced unlabeled sample set into the teacher network for reasoning to generate pseudo labels, and dividing the pseudo labels according to types; wherein the pseudo labels include reliable pseudo labels, uncertain pseudo labels and erroneous pseudo labels; Step S105: constructing a joint loss function based on the supervised loss of the labeled samples and the unsupervised loss of the unlabeled samples; Step S106: inputting the enhanced unlabeled sample set, the labeled sample set and the pseudo-label into the student network for training, and combining the joint loss function to obtain a first weight parameter constraining the student network; Step S107: After processing the first weight parameter using the EMA algorithm, the first weight parameter is loaded into the teacher network to update the second weight parameter of the teacher network. Step S108: iteratively training the student network using the updated teacher network until the trained student network is obtained; Step S109: input the test data set into the trained student network to obtain the target position and predicted category in the test sample.

[0031] Through the above-mentioned semi-supervised target recognition method for airborne optoelectronic images, the parameters of the teacher network model are updated through the exponential moving average algorithm (EMA), the quality of the generated pseudo-labels is continuously improved, and the unlabeled data can be used to train the student network more effectively, and the network model for target recognition of airborne optoelectronic images is obtained. Through the pseudo-label generation strategy and combined with the semi-supervised teacher-student network joint training framework, the problem of the difficulty in effectively utilizing unlabeled data samples is solved. After adding unlabeled sample data to the training, the target recognition ability of airborne optoelectronic images is improved.

[0032] Next, we will refer to Figures 1 to 5 Each step of the above-mentioned semi-supervised target recognition method for airborne electro-optical images in this example implementation is described in more detail.

[0033] In step S101 to step S103, use the visible light / infrared camera carried by the airborne platform to shoot visible light / infrared videos in different scenes, including cities, mountains, deserts, water surfaces, etc.; extract frames from the obtained visible light / infrared video to obtain a corresponding image sequence; resize the obtained image sequence; for example, resize the image to 960×540; divide the data into a training set and a test set according to 7:3, and annotate the test set. Divide the training set into an unlabeled data set and a labeled data set according to 8:2, and annotate the images in the labeled data set. Figure 3 As shown, it is a schematic diagram of the original image of part of the optoelectronic image in the airborne scene.

[0034] Data enhancement methods include contrast transformation, brightness transformation, mirror transformation, image translation, image rotation, image noise addition, motion blur, adding clouds and fog, and adding rain and snow.

[0035] Considering the imaging characteristics of airborne scenes, a motion blur option is added to the data enhancement method. For a single image, the image enhancement method used randomly selects three of the above methods for combination, where contrast transformation and brightness transformation are not used at the same time; considering the impact of different meteorological conditions on imaging, adding clouds and fog and adding rain and snow are used in the augmentation method options, and the two are not used at the same time; considering the impact of noise on recognition effect, image noise is selected with a probability of 0.1; considering that blur is easy to occur in airborne scenes, which affects the imaging effect, the motion blur augmentation method is selected with a probability of 0.2. Through the combined data augmentation method, high-quality expansion of airborne optoelectronic images is achieved.

[0036] More specifically, the optoelectronic equipment on the airborne platform is used to collect optoelectronic videos of different scenes; the obtained videos are processed by frame extraction to obtain the corresponding image sequence; the size of the obtained image sequence is adjusted to obtain an image resolution size suitable for processing; the data is divided into a training set and a test set according to a 7:3 ratio, and the test set is annotated. The training set is divided into an unlabeled data set and a labeled data set according to a 8:2 ratio, and the images in the labeled data set are annotated.

[0037] When airborne photoelectric sensors are used for imaging, the actual imaging images are relatively complex due to factors such as complex meteorological conditions, different shooting scenes, and the status of the airborne platform itself. The constructed data enhancement library contains 9 image transformation methods and includes motion blur to simulate the feature distribution of image data in airborne scenes.

[0038] The constructed data enhancement library includes 9 image transformation methods: contrast transformation, brightness transformation, mirror transformation, image translation, image rotation, image noise addition, motion blur, adding clouds and fog, and adding rain and snow. When performing data augmentation on images, the image augmentation method used randomly selects three of the above methods and combines them, and a combination and selection strategy of different data augmentation methods for airborne scenes is designed.

[0039] The constructed motion blur filter randomly selects a value in the range of 2-6 pixels as the distance of pixel movement, and randomly selects a value in the range of 0-180 degrees as the angle of motion blur. This strategy is used to simulate the motion blur effect consistent with the actual airborne scene.

[0040] In step S104, a semi-supervised teacher-student network joint training framework is built. The framework consists of two target recognition networks with the same structure and different parameter initializations. The initial parameters of the two recognition networks are generated by random initialization, and the network structure adopts YOLOv10.

[0041] In step S105, the pseudo-label generation method generates pseudo-labels for unlabeled samples by the teacher network. The pseudo-label assignment method divides the generated pseudo-labels into reliable pseudo-labels, uncertain pseudo-labels and erroneous pseudo-labels according to different thresholds. Reliable pseudo-labels and uncertain pseudo-labels participate in the calculation of different loss functions respectively, and erroneous pseudo-labels do not participate in the calculation.

[0042] The constructed pseudo-label assignment method provides a more sophisticated pseudo-label generation strategy, setting high and low confidence thresholds to divide the pseudo-labels initially generated by the teacher network into three categories. Pseudo-labels with confidence greater than the high threshold are defined as high-quality pseudo-labels; pseudo-labels with confidence between the high threshold and the low threshold are defined as uncertain pseudo-labels; and pseudo-labels with confidence less than the low threshold are defined as erroneous pseudo-labels. This assignment method improves the fineness of pseudo-label division and can more effectively utilize uncertain pseudo-labels.

[0043] More specifically, a pseudo-label generation and assignment method is adopted to obtain stable and accurate pseudo-labels, and the constructed pseudo-label assignment method provides a refined pseudo-label generation strategy.

[0044] In terms of the number of pseudo labels generated, a distribution adaptation method based on the redistribution method is used. In a batch of data, the average number of targets in each image of the labeled data and the unlabeled data should be similar. The labeled data is supervised by the number of labels in the labeled data, and a certain number of pseudo labels are generated for the unlabeled data using an adaptive label distribution-aware confidence threshold, so that the number of labels on each image of the labeled data and the unlabeled data is as close as possible under the division of the threshold.

[0045] After the teacher network initially generates pseudo labels, the pseudo labels are divided into three categories by setting high and low confidence thresholds. Pseudo labels with confidence greater than the high threshold are defined as high-quality pseudo labels, pseudo labels with confidence between the high and low thresholds are defined as uncertain pseudo labels, and pseudo labels with confidence less than the low threshold are defined as erroneous pseudo labels. The student network then participates in the calculation of different unsupervised partial loss functions according to the different pseudo label categories of the unlabeled data.

[0046] These two thresholds are dynamically updated during network training. In the kth round of training, and The two thresholds are determined as follows:

[0047]

[0048] in, and represents the number of labeled and unlabeled data, represents the number of true annotations for the cth category, represents the pseudo-label score list of the kth round and the cth category, Represents the nth element in the list. If n is not an integer, it is rounded up. and The two thresholds divide the pseudo labels into three categories: high-quality pseudo labels, uncertain pseudo labels, and incorrect pseudo labels.

[0049] In step S106, a joint loss function is constructed, including supervised loss and unsupervised loss. The supervised loss part includes the cross entropy loss for distinguishing foreground and background, the cross entropy loss for distinguishing different target categories, and the CIOU loss for measuring the prediction of the target position. The unsupervised loss part includes the cross entropy loss for distinguishing different target categories, the CIOU loss for measuring the prediction of the target position, and the fine cross entropy loss for distinguishing foreground and background.

[0050] Constructed supervised loss. Constraints are formed between the target location, category, and confidence information contained in the student network output and the labeled information. Through the construction of supervised loss, the model parameters can be optimized in the direction of correctly predicting the target.

[0051] The constructed unsupervised loss. Pseudo labels with high confidence can effectively distinguish the foreground and background, and the corresponding targets are used as positive samples to participate in the training of the model student network; pseudo labels with low confidence are usually backgrounds, and some are randomly selected as negative samples to participate in the training of the model student network; those with confidence between the high threshold and the low threshold are uncertain pseudo labels, and directly discarding them will affect the optimization effect of the student network. The value obtained by calculating the CIOU between the corresponding target prediction results of the teacher network and the student network is the target score. The target score reflects the accuracy of the position of the prediction box output by the student network. The target score is used to improve the ability of the student network to distinguish the foreground and background of uncertain samples, establish a consistent association between the predicted position and the predicted foreground and background, constrain the samples with accurate position prediction to produce accurate category predictions, enable uncertain pseudo labels and corresponding samples to participate in the training optimization process of the student network, and further improve the performance of the auxiliary student network.

[0052] Construct a joint loss function consisting of a supervised loss function and an unsupervised loss function. Balance the contribution of supervised loss and unsupervised loss to the joint loss function through set parameters, and provide support for the training optimization process of the student network.

[0053] By constructing a joint loss function, we can make full use of the relationship between unlabeled data and labeled data, maximize the training efficiency and the utilization of labeled and unlabeled data, and constrain the student network to obtain better optimization results.

[0054] More specifically, the joint loss function consists of a supervised loss part and an unsupervised loss part. The supervised loss part calculates the classification loss and positioning loss of the labeled samples, and the unsupervised loss part calculates different unsupervised loss functions according to the different pseudo-label types. Different types of pseudo-labels and corresponding image samples participate in the calculation of different unsupervised loss functions.

[0055] Constructing the supervised classification loss enables the student network to have a certain target recognition ability under the training of labeled data.

[0056] The loss of the supervised part corresponding to the labeled samples is defined as follows:

[0057] in, represents the cross entropy loss function, represents the position in the output feature map, Student network feature map The category score output at Student network feature map The position score of the output, Student network feature map The foreground score output at represents the CIOU calculation function, is the true label in the feature map The category score at is the true label in the feature map The position score is is the true label in the feature map The foreground score at the location is calculated according to the above formula to calculate the loss of the target, position and target category.

[0058] Construct the unsupervised classification loss This enables the student network to use pseudo labels with high category reliability for the category features of the learning target. Construct the unsupervised regression loss This enables the student network to use pseudo labels with high category reliability or pseudo labels with accurate positioning to learn the location characteristics of the target. Construct the unsupervised target loss This allows the student network to use pseudo-labels with low confidence as negative samples for training, pseudo-labels with high confidence as positive samples for training, and uncertain pseudo-labels as additional information to improve the quality of training. The more accurate the prediction box with the constrained position prediction, the better it can distinguish the foreground and background.

[0059] The loss of the unsupervised part corresponding to the unlabeled samples is defined as follows:

[0060]

[0061]

[0062]

[0063] in, is the unsupervised category loss, is the unsupervised position loss, is the unsupervised foreground loss, and After pseudo-label assignment and screening, in the feature map The category score, location score and foreground score of the corresponding feature point target; for Target score for position pseudo-labels; for Classification scores of position pseudo labels; Represents the indicator function. When the condition is met, its value is 1, and when the condition is not met, its value is 0; the confidence is greater than the high threshold Pseudo-label participation Calculation of; confidence is greater than the high threshold Or pseudo labels with a target score greater than 0.9 participate Calculation of; confidence is greater than the high threshold and less than the lower threshold Pseudo-label participation Calculation of whether there is a foreground target in the and low threshold The pseudo-labels between them associate the confidence scores of the foreground and background with the consistency of the target position prediction, so that the uncertain pseudo-labels participate in the training and optimization process of the model. The overall loss function is defined as follows:

[0064] represents the loss function calculated on the labeled image, and represents the loss function calculated on the unlabeled image, Used to balance supervised loss and unsupervised loss.

[0065] Constructing the joint loss function enables the recognition network to learn the feature distribution in the unlabeled image, realize the recognition of the target in the airborne optoelectronic image, and further improve the recognition effect of the existing model.

[0066] In step S107 and step S108, the prediction results of the teacher network are processed by the pseudo-label generation and distribution method, combined with the corresponding unlabeled images, to assist the training of the student network, and the total loss function is used to update the parameters of the student network. The parameters of the student network are updated by the EMA algorithm to complete the update of the parameters of the teacher network. The two are iterated in turn under the joint training framework to constrain the feedback optimization of the overall recognition network.

[0067] Specifically, the joint training process consists of an initial stage and a joint training stage. In the initial stage, only labeled data is used for training. After the initial stage is completed, the joint training stage begins, which is a joint training stage consisting of labeled data and unlabeled data.

[0068] Generate pseudo labels for unlabeled samples through the teacher network, and use the generated pseudo labels to train the student network; The update of the teacher network parameters is obtained by processing the parameters of the student network through the EMA algorithm. The pseudo-labels generated by the teacher network help improve the recognition effect of the student network. The student network loads the updated weight parameters into the teacher network after processing through the EMA algorithm, which helps the pseudo-labels predicted by the teacher network to be more accurate. The interaction between the two enables the constructed joint training framework to continuously update the weight parameters in the direction of improving model performance.

[0069] More specifically, during the joint training process, the weight parameter optimizer uses the SGD optimizer, the initial learning rate is set to 0.01, the total training rounds are 150 rounds, the confidence threshold of the recognition result when the teacher network generates pseudo labels is 0.1, the IOU threshold is 0.35, the upper and lower limits of the threshold for generating reliable labels are 0.15 and 0.6 respectively, and the EMA update rate is 0.999.

[0070] In this semi-supervised teacher-student network joint training framework, the student network is used for back-propagation to update the weight parameters, and the weight parameters of the teacher network are updated by the weight parameters of the student network through the EMA strategy.

[0071] In step S109, the trained student network is loaded, sample data to be tested is input, and the target position and prediction type in the sample data to be tested are output.

[0072] In a specific embodiment, for the proposed network model, optoelectronic equipment built on an aircraft is used to collect data in urban, mountainous, and water scenes, and recognition experiments are conducted on targets in the images, and the experimental results are displayed.

[0073] In order to explore the recognition effect of this application on various targets in airborne photoelectric sensors, the recognition results of vehicles, ships, oil depots, airplanes, electric wind turbines, airport runways and other targets in photoelectric images after joint training using the method of this application in different scenarios are mainly analyzed. At the same time, the recognition results of the model trained only with labeled data on the test image are shown, and the recognition effects before and after semi-supervised learning are compared. The results are as follows: Figure 5 shown.

[0074] From the comparison in the figure, it can be seen that when using the model trained only on labeled data for recognition, more false detections and missed detections will occur. Using the semi-supervised target recognition method for airborne optoelectronic images of the present application, through the joint semi-supervised training of labeled data and unlabeled data, the overall suppression of false alarms and the recognition effect of missed detection targets of the trained target recognition model are effectively improved, which can improve the performance of existing recognition models.

[0075] The semi-supervised target recognition method for airborne optoelectronic images can achieve better target recognition results. It can effectively identify targets that cannot be recognized after training with labeled images alone, reduce missed detections, and improve model recognition accuracy and recall. The experimental results are as follows: Figure 4 and Figure 5 As shown. Among them, Figure 4 The first set of experimental results of the semi-supervised target recognition method for airborne electro-optical images before and after the model is trained; Figure 5 The second set of experimental results of the model before and after training for the semi-supervised target recognition method for airborne electro-optical images; Figure 4 and Figure 5 The first column is the original image and target annotation; the second column is the recognition result of the model on the test set obtained after training on the labeled data; the third column is the recognition result of the model on the test set obtained after training on the labeled and unlabeled data using the method of the present application.

[0076] Table 1 Quantitative comparison analysis of algorithm performance

[0077] In addition, the performance indicators of the semi-supervised target recognition method for airborne optoelectronic images on the test set are quantitatively analyzed, and the results are shown in Table 1. The model architecture used by Group 1 and Group 2 is the same, but the model used in Group 1 is a model that has not been optimized through semi-supervised learning, and the model used in Group 2 is a model obtained after being processed by the semi-supervised target recognition method for airborne optoelectronic images. From the results, it can be seen that the proposed method can effectively improve the recognition effect of the targets in the test set.

[0078] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: In order to solve the problem of difficulty in acquiring airborne optoelectronic images, the dataset used in this application was constructed by building visible light / infrared cameras on airborne platforms and test towers, and collecting data in different scenarios such as cities, mountains, deserts, and water surfaces.

[0079] When optoelectronic equipment is deployed in actual airborne scenes, it faces complex weather and lighting conditions. The state of the airborne platform itself will also affect the actual imaging picture, and the overall imaging environment is complex. Through the construction of nine image transformation methods, including contrast transformation, brightness transformation, mirror transformation, image translation, image rotation, image noise, motion blur, adding clouds and fog, and adding rain and snow, training data under different lighting and weather conditions are generated. Considering that the actual imaging picture of the airborne platform is prone to motion blur, a motion blur filter is used to simulate the motion blur caused by factors such as the jitter of the airborne platform, providing high-quality data support for improving the training of the recognition model in airborne scenes.

[0080] In view of the difficulties and high cost of manual labeling of airborne electro-optical images, the scarcity of labeled samples compared with actual needs, and the difficulty in using massive unlabeled samples, we constructed a refined pseudo-label generation and distribution method, and used the distribution adaptation and high and low confidence screening strategies based on the redistribution method. Without the need for manual labeling of unlabeled data, we fully utilized the same distribution relationship between unlabeled data and labeled data to obtain stable and accurate pseudo-labels for the training of the student network, enrich the student model's understanding of the probability of data edge distribution, and make full use of previously unusable unlabeled data resources to further optimize the model algorithm and improve the performance of the model.

[0081] In order to solve the problem that some unsupervised samples with low confidence are difficult to be effectively used, through the constraints of different confidence thresholds, in addition to the reliable pseudo-labels and wrong pseudo-labels participating in the student network training, uncertain pseudo-labels are used as input to establish the loss function of the associated predicted position and predicted foreground background. Finally, the training process of the student network is constrained by the constructed joint loss function, so that the supervised loss and unsupervised loss can achieve a good balance.

[0082] In response to the negative optimization problems that may arise during the joint training of teacher-student networks, this application makes full use of the interaction between teacher-student networks, adopts a semi-supervised teacher-student network joint training framework, and uses a phased training method to improve the quality of pseudo-labels at the beginning of training. Then, under the constraints of the joint loss function, the student network is optimized in the direction of better model recognition effect. The knowledge learned by the student network is then loaded into the teacher network after the EMA strategy, so that the teacher network can continuously generate more accurate and reliable pseudo-label data. The pseudo-label data generated by the student network will be used in the student network in the next round of training. Through the interaction between the two under the joint training framework, the student network is assisted to gradually improve the recognition effect of targets in the image.

[0083] It should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. in the above description indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present disclosure and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present disclosure.

[0084] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0085] In the embodiments of the present disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0086] In the embodiments of the present disclosure, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features not being in direct contact but being in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes the first feature being directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes the first feature being directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0087] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0088] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A semi-supervised target recognition method for airborne electro-optical images, characterized in that: The method includes: Acquire an image sequence, and divide the image sequence into a training data set and a test data set according to a first preset ratio; Annotating the images in the training data set according to a second preset ratio to obtain a labeled sample set and an unlabeled sample set, and performing data enhancement on the unlabeled sample set and the labeled sample set; Constructing a semi-supervised teacher-student network joint training framework; wherein the semi-supervised teacher-student network joint training framework includes a teacher network and a student network; Inputting the enhanced unlabeled sample set into the teacher network for reasoning to generate pseudo labels, and dividing the pseudo labels according to types; wherein the pseudo labels include reliable pseudo labels, uncertain pseudo labels and erroneous pseudo labels; Constructing a joint loss function based on the supervised loss of the labeled samples and the unsupervised loss of the unlabeled samples; Inputting the enhanced unlabeled sample set, the labeled sample set and the pseudo-label into the student network for training, and combining the joint loss function to obtain a first weight parameter constraining the student network; After processing the first weight parameter using the EMA algorithm, the first weight parameter is loaded into the teacher network to update the second weight parameter of the teacher network; Iteratively training the student network using the updated teacher network until a trained student network is obtained; The test data set is input into the trained student network to obtain the target position and predicted category in the test sample.

2. The semi-supervised target recognition method for airborne optoelectronic images according to claim 1, characterized in that: The step of acquiring an image sequence and dividing the image sequence into a training data set and a test data set according to a first preset ratio includes: Use the optoelectronic equipment on the airborne platform to collect optoelectronic videos of different scenes; Performing frame extraction processing on the optoelectronic video to obtain the corresponding image sequence; Adjusting the size of the image sequence so that the image resolution of the image sequence is a preset value; The image sequence is divided into the training data set and the test data set according to a 7:3 ratio.

3. The semi-supervised target recognition method for airborne optoelectronic images according to claim 2, characterized in that: The step of annotating the images in the training data set according to a second preset ratio to obtain an annotated sample set and an unannotated sample set, and performing data enhancement on the unannotated sample set and the annotated sample set includes: Divide the training data set into the unlabeled data set and the labeled data set according to an 8:2 ratio, and label the images in the labeled data set; Building a data enhancement library based on image transformation methods; wherein the image transformation methods include contrast transformation, brightness transformation, mirror transformation, image translation, image rotation, image noise addition, motion blur, adding clouds and fog, and adding rain and snow; According to the airborne scene, several corresponding image transformation methods are selected to perform data enhancement on the unlabeled sample set and the labeled sample set.

4. The semi-supervised target recognition method for airborne optoelectronic images according to claim 3, characterized in that: The contrast transformation and the brightness transformation are not used at the same time, the addition of clouds and fog and the addition of rain and snow are not used at the same time, the image noise is selected with a probability of 0.1, and the motion model is selected with a probability of 0.

2.

5. The semi-supervised target recognition method for airborne optoelectronic images according to claim 3, characterized in that: The steps to construct a semi-supervised teacher-student network joint training framework include: A semi-supervised teacher-student network joint training framework is constructed. The framework consists of the teacher network and the student network with the same structure but different parameter initializations. The initial parameters of the teacher network and the student network are both generated by random initialization, and the network structure adopts YOLOv10.

6. The semi-supervised target recognition method for airborne optoelectronic images according to claim 5, characterized in that: The step of inputting the enhanced unlabeled sample set into the teacher network for reasoning to generate pseudo labels, and dividing the pseudo labels according to types includes: Inputting the enhanced unlabeled sample set into the teacher network to generate the pseudo labels; Set high and low thresholds; Defining the pseudo-label whose confidence is greater than the high threshold as the high-quality pseudo-label; defining the pseudo-label with a confidence level between the high threshold and the low threshold as the uncertain pseudo-label; The pseudo label whose confidence is less than the low threshold is defined as the erroneous pseudo label.

7. The semi-supervised target recognition method for airborne optoelectronic images according to claim 6, characterized in that: The step of constructing a joint loss function based on the supervised loss of the labeled samples and the unsupervised loss of the unlabeled samples includes: Constructing a supervised loss according to the target location, category, confidence and annotation information in the labeled sample set; According to the reliable pseudo-label, the uncertain pseudo-label and the erroneous pseudo-label, the CIOU between the corresponding target prediction results of the teacher network and the student network is calculated to obtain a target score; constructing an unsupervised loss based on the target score; The joint loss function is obtained according to the supervised loss and the unsupervised loss.

8. The semi-supervised target recognition method for airborne optoelectronic images according to claim 7, characterized in that: The expression of the supervised loss is: in, represents the cross entropy loss function, represents the position in the output feature map, Student network feature map The category score output at Student network feature map The position score of the output, Student network feature map The foreground score output at represents the CIOU calculation function, is the true label in the feature map The category score at is the true label in the feature map The position score is is the true label in the feature map Prospect score at The expression of the unsupervised loss is: in, is the unsupervised category loss, is the unsupervised position loss, is the unsupervised foreground loss, and After pseudo-label assignment and screening, in the feature map The category score, location score and foreground score of the corresponding feature point target; for Target score for position pseudo-labels; for Category scores of position pseudo labels; Represents the indicator function. When the condition is met, its value is 1, and when the condition is not met, its value is 0; the confidence is greater than the high threshold Pseudo-label participation Calculation of; confidence is greater than the high threshold Or pseudo labels with a target score greater than 0.9 participate Calculation of; confidence is greater than the high threshold and less than the lower threshold Pseudo-label participation Calculation of whether there is a foreground target in the and low threshold The pseudo-labels between them associate the confidence scores of foreground and background with the consistency of target location prediction; The expression of the joint loss function is: in, indicates that there is a supervision loss, and represents the unsupervised loss, is the balance coefficient.

9. The semi-supervised target recognition method for airborne optoelectronic images according to claim 7, characterized in that: The step of iteratively training the student network using the updated teacher network until the trained student network is obtained comprises: The teacher network generates the pseudo labels, and the pseudo labels are used in the training of the student network to update the first weight parameters; The student network loads the updated first weight parameter into the teacher network after being processed by the EMA algorithm to update the second weight parameter; The student network is iteratively trained until a trained student network is obtained.

Images