Aerial image detection and recognition method based on SimROD

Abstract

The present invention provides a SimROD-based aerial image detection and recognition method, comprising the following steps: step S1, collecting a large number of field aerial images; step S2, performing BM3D denoising on the images; step S3, labeling the collected images; step S4, training a source teacher model T; step S5, changing the teacher model parameters to teacher model enhancement parameters θa; step S6, refining the pseudo-label D on the target data; step S7, stabilizing the aerial images. Step S8, detection and recognition; step S9, motion vector classification; step S10, obtaining a detection result. The SimROD-based aerial image detection and recognition method of the present invention can replace traditional recognition detection methods, and achieves the purpose of more accurate and real-time detection and recognition of multiple moving targets in aerial images by improving the efficiency of aerial image detection and recognition and reducing the time consumed in the detection process.

Description

Technical Field

[0001] The present invention relates to the field of automatic visual detection, and in particular to a SimROD-based aerial image detection and recognition method. Background Art

[0002] Aerial images from unmanned aerial vehicles (UAVs) feature high resolution, a wide field of view, and small objects. Camera-carrying UAVs can conduct real-time aerial surveillance of mission areas. Aerial surveillance can be used in areas such as pedestrian detection and tracking, vehicle detection and tracking, human-computer interaction, and intelligent transportation systems. When a UAV carries a fixed camera to detect moving targets, it extracts and identifies them by searching for target changes in each pixel frame. However, in actual operation, UAVs are inevitably affected by environmental factors, making it difficult to detect moving targets. Therefore, achieving real-time, accurate detection and identification of multiple moving targets in UAV aerial images has become a research priority.

[0003] Due to technical reasons of most manufacturers, as well as external factors such as weather changes (such as rain or fog), changes in lighting conditions or image damage (such as motion blur), the existing inspection methods still have the following problems: (1) UAVs are inevitably interfered with by environmental factors in actual working conditions and have difficulty detecting moving targets; (2) Regarding the current hot research topic of UAVs carrying non-fixed cameras to detect moving targets, there is currently no method that can fully prove the reliability of non-fixed cameras in real-time detection of multiple moving targets. Summary of the Invention

[0004] To address the deficiencies of the prior art, the present invention provides an aerial image detection and recognition method based on SimROD. The specific technical solution is as follows:

[0005] A method for detecting and recognizing aerial images based on SimROD includes the following steps:

[0006] Step S1: Collect a large number of aerial images of the field;

[0007] Step S2, performing BM3D denoising on the image;

[0008] Step S3: label the collected image to form target data D, half of which is labeled source data D1 and half is unlabeled target data D2;

[0009] Step S4: Based on the source data, a source teacher model T with a larger capacity than the student model M is trained to obtain the initial pseudo label parameter θs. The source teacher model T is used to generate the initial pseudo label on the target data.

[0010] Step S5: Using a stepwise adaptation algorithm, the parameters of the large source teacher model T are gradually changed from the initial pseudo-label parameters θs to the teacher model enhancement parameters θa;

[0011] Step S6: Use the adaptive teacher model enhancement parameters θa to refine the pseudo-labels of D on the target data, and then use these pseudo-labels to fine-tune the student model M;

[0012] Step S7: stabilizing the aerial image to achieve displacement estimation and displacement compensation;

[0013] Step S8: Detect and identify targets in the foreground of the aerial image;

[0014] Step S9, obtaining motion vectors of two consecutive images;

[0015] Step S10: Detect the unlabeled target data D2 and obtain a detection result.

[0016] Furthermore, the specific steps in step S2 include:

[0017] Step S201: Segment an image into smaller pixel slices and select a reference slice;

[0018] Step S202: Find small pieces similar to the reference piece to form a 3D block;

[0019] Step S203: perform 3D transformation on all similar blocks and perform thresholding;

[0020] Step S204: perform 3D inverse transformation, and restore all 3D blocks to the image through weighted averaging.

[0021] Furthermore, each label of the label processing performed on the collected image in step S3 consists of an object category and bounding box coordinates.

[0022] Furthermore, the specific steps in step S4 include: using the mixed image generated by DomainMix enhancement;

[0023] Step S401: A set of unlabeled images is taken from the target data D for model training. For each image in a batch, three other images are first randomly sampled from the source and target data, and random crops of these images are mixed to create a new domain-mixed image A in a 2×2 template.

[0024] Step S402: Annotate the initial pseudo-label and true label datasets into the mixed image, and calculate the bounding box coordinates of the target based on the relative position of each crop in the new mixed image;

[0025] Step S403: Use a weighted balanced sampler to uniformly sample from the initial pseudo-label and true label datasets, and perform adaptive adjustment of the source teacher model T.

[0026] Furthermore, step S5 adopts a stepwise adaptation method to optimize the parameters of the detection model, and the specific steps include:

[0027] Step S501: Freeze all convolutional layers and only adapt the BN layer in the first w epochs. After the first stage, the parameters of the BN layer are updated.

[0028] Step S502: Use the partially adapted teacher model to generate more accurate pseudo labels.

[0029] Furthermore, the method for realizing displacement estimation in step S7 is:

[0030] For every two consecutive frames, the previous frame image is defined as f(ty) and the current frame image is defined as f(t). In order to reduce the calculation time, the size of the image f(t) is reduced to 75% of the original size and the color is set to grayscale mode to obtain a new image Use SimROD algorithm as feature detector and descriptor to find local features on each frame, f (x,y) (t) is the pixel value of f(t) at position (x, y); the displacement estimation is achieved by using the feature position estimated in homogeneous coordinates using affine transformation; in homogeneous coordinates, the relationship between a pair of feature points in f(t-1) and f(t) is:

[0031]

[0032] Where H is a homogeneous affine matrix:

[0033]

[0034] Among them, a ij is the parameter about the rotation angle θ, T x and T y Double are the parameters of translation T on the x-axis and y-axis respectively;

[0035] The method to achieve displacement compensation is to use a Kalman filter to compensate the translation and rotation trajectories, thereby generating a new transformation for each frame.

[0036] Furthermore, in step S8, the background and foreground of each frame of the aerial image are determined by image stabilization technology, wherein the foreground consists of candidate moving targets in the image, and the foreground can be detected and identified using Haar-like features and enhanced cascade classifiers;

[0037] The specific steps include:

[0038] Step S801: Haar-like features are mainly used to detect objects of different sizes in the image;

[0039] Step S802: Using the AdaBoost algorithm to combine the features of the selected classifiers, the classifier is selected as a value to determine the best classification function for each feature.

[0040] Furthermore, the step S9 uses the Farneback optical flow method to obtain the motion vectors of the two consecutive images.

[0041] Beneficial effects of the present invention:

[0042] The present invention provides an aerial image detection and recognition method based on SimROD. Compared with traditional recognition and detection methods, this method is more effective in realizing real-time detection and recognition of multiple moving targets in aerial images of unmanned aerial vehicles (UAVs). At the same time, the aerial image detection and recognition method based on SimROD can replace traditional recognition and detection methods, and achieve the purpose of more accurate and real-time detection and recognition of multiple moving targets in aerial images by improving the efficiency of aerial image detection and recognition and reducing the time consumed in the detection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the present invention Figure 1 ,in Figure 1 The flowchart of the present invention is shown in FIG. The technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, a SimROD-based aerial image detection and recognition method includes the following specific steps:

[0046] Step S1: Collect a large number of aerial images of the field;

[0047] Step S2, performing BM3D denoising on the image;

[0048] Step S201: Divide an image into small pixel slices of smaller size and select a reference slice.

[0049] Step S202: Find small pieces similar to the reference piece to form a 3D block.

[0050] Step S203: Perform 3D transformation on all similar blocks and perform thresholding.

[0051] Step S204: perform 3D inverse transformation, and restore all 3D blocks to the image through weighted averaging.

[0052] Step S3: label the collected image to form target data D, half of which is labeled source data D1 and half is unlabeled target data D2; each label of the label processing of the collected image in step S3 consists of a target category and a bounding box coordinate.

[0053] Step S4: Based on the source data, train a source teacher model T with a larger capacity than the student model M to obtain the initial pseudo label parameter θs. The source teacher model is used to generate the initial pseudo label on the target data.

[0054] Step S401: A set of unlabeled images is taken from the target data D for model training. For each image in a batch, three other images are first randomly sampled from the source and target data, and random crops of these images are mixed to create a new domain mixed image A in a 2×2 template.

[0055] Step S402: Annotate the initial pseudo labels and the true labels into the mixed image, and calculate the bounding box coordinates of the target according to the relative position of each crop in the new mixed image.

[0056] Step S403: Use a weighted balanced sampler to uniformly sample from the initial pseudo-label and true label datasets, and perform adaptive adjustment of the source teacher model T.

[0057] Step S5: Using a stepwise adaptation algorithm, gradually change the parameters of the large source teacher model T from the initial pseudo-label parameters θs to the teacher model enhancement parameters θa; Step S5 includes: using a stepwise adaptation method to optimize the parameters of the detection model.

[0058] The specific steps are:

[0059] Step S501: Freeze all convolutional layers and only adapt the BN layer in the first w epochs. After the first stage, the parameters of the BN layer are updated.

[0060] Step S502: Use the partially adapted model to generate more accurate pseudo labels.

[0061] Step S6: Use the adaptive teacher model enhancement parameter θa to refine the pseudo-labels D on the target data, and then use these pseudo-labels to fine-tune the student model M to obtain an adaptively adjusted student model M. The target image is input into the adaptively adjusted student model M to obtain the output feature detection result;

[0062] Step S7: Perform aerial image stabilization on the corresponding image that meets the output feature detection result to achieve displacement estimation and displacement compensation.

[0063] The method for implementing displacement estimation in step S7 is as follows: for every two consecutive frames, the previous frame is defined as f(ty) and the current frame is defined as f(t); in order to reduce the calculation time, the size of the image f(t) is reduced to 75% of the original size and the color is set to grayscale mode to obtain a new image The local features on each frame are found using the SimROD algorithm as a feature detector and descriptor. (x,y) (t) is the pixel value of f(t) at position (x,y); displacement estimation is achieved by using affine transformation to estimate the feature position in homogeneous coordinates. In homogeneous coordinates, the relationship between a pair of feature points in f(t-1) and f(t) is:

[0064]

[0065] Where H is a homogeneous affine matrix:

[0066]

[0067] Among them, a ij is the parameter about the rotation angle θ, T x and T y Double are the parameters of translation T on the x-axis and y-axis respectively;

[0068] The method to achieve displacement compensation is to use a Kalman filter to compensate the translation and rotation trajectories, thereby generating a new transformation for each frame.

[0069] Prediction part estimated trajectory exist The status is:

[0070]

[0071] in, is the rotation angle, the initial state is defined by z(0)=[0,0,0], and the covariance of the error can be estimated as:

[0072]

[0073] Where e(t-1) is the optimal state estimate at time t-1, the covariance of the initial error is defined by e(0) = [1,1,1], and Ω p is the covariance of the process noise; the optimal Kalman gain is calculated as:

[0074]

[0075] Step S8, performing target detection and recognition on the foreground of the aerial image; in step S8, each frame of the aerial image is subjected to image stabilization technology to determine the background and foreground, wherein the foreground is composed of candidate moving targets in the image, and the foreground can be detected and recognized using Haar-like features and an enhanced cascade classifier;

[0076] Step S801: Haar-like features are mainly used to detect objects of different sizes in the image.

[0077] Step S802: Use the AdaBoost algorithm to combine the features of the selected classifiers, and select the classifier as a value to determine the best classification function for each feature.

[0078] The training samples are set to s=1,2,...,N, where represents the class label of the sample, β x = 0 or 1 to indicate negative or positive labels respectively. Each sample is converted to grayscale and then scaled down to the detector's baseline resolution. The AdaBoost algorithm creates a weight vector for all training samples in the iteration. For all samples (α1, β1),...,(α N ,β N ) is set to ω1(s) = 1 / N, then the error associated with the selected classifier is:

[0079]

[0080] λ i (α x )=0 or 1 is the selected classifier for negative label or positive label respectively, and i=1,...,I ​​is the number of iterations.

[0081] Step S9: Obtaining the motion vectors of the two consecutive images: Using dense optical flow to calculate the motion vectors of the two consecutive images, the motion direction between the stationary target (background) and the moving target (foreground) is more clearly distinguished.

[0082] Select pixel j within a 10x10 pixel window G(j). By using polynomial expansion, each pixel in G(j) can be approximated by the polynomial at f(t-1):

[0083]

[0084] Where p is the motion vector, A(t-1) is a symmetric matrix, b(t-1) is the motion vector, and c(t-1) is a known scalar parameter.

[0085] Step S10: Detect the unlabeled target data D2 to obtain a detection result, and detect and identify the image expression in the aerial image.

[0086] The present invention improves the detection and recognition efficiency of aerial images, reduces the time consumed in the detection process, and realizes the detection and recognition of multiple moving targets in aerial images by using the adaptive method of the SimROD algorithm and the detection and recognition method based on SimROD.

[0087] The above is only a preferred embodiment of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that several improvements and modifications can be made without departing from the concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A method for detecting and recognizing aerial images based on SimROD, characterized in that: The steps include: Step S1: Collect a large number of aerial images of the field; Step S2, performing BM3D denoising on the image; Step S3: label the collected image to form target data D, half of which is labeled source data D1 and half is unlabeled target data D2; Step S4: Based on the source data, a source teacher model T with a larger capacity than the student model M is trained to obtain the initial pseudo label parameter θs. The source teacher model T is used to generate the initial pseudo label on the target data. The specific steps in step S4 include: using the mixed image generated by DomainMix enhancement; Step S401: A set of unlabeled images is taken from the target data D for model training. For each image in a batch, three other images are first randomly sampled from the source and target data, and random crops of these images are mixed to create a new domain-mixed image A in a 2×2 template. Step S402: Annotate the initial pseudo-label and true label datasets into the mixed image, and calculate the bounding box coordinates of the target based on the relative position of each crop in the new mixed image; Step S403: Use a weighted balanced sampler to uniformly sample from the initial pseudo-label and true label datasets, and perform adaptive adjustment of the source teacher model T; Step S5: Using a stepwise adaptation algorithm, the parameters of the large source teacher model T are gradually changed from the initial pseudo-label parameters θs to the teacher model enhancement parameters θa; Step S6: Use the adaptive teacher model enhancement parameters θa to refine the pseudo labels on the target data D, and then use these pseudo labels to fine-tune the student model M; Step S7: stabilizing the aerial image to achieve displacement estimation and displacement compensation; The method for realizing displacement estimation in step S7 is as follows: For every two consecutive frames, the previous frame image is defined as f(ty) and the current frame image is defined as f(t). In order to reduce the calculation time, the size of the image f(t) is reduced to 75% of the original size and the color is set to grayscale mode to obtain a new image Use SimROD algorithm as feature detector and descriptor to find local features on each frame, f (x,y) (t) is the pixel value of f(t) at position (x, y); the displacement estimation is achieved by using the feature position estimated in homogeneous coordinates using affine transformation; in homogeneous coordinates, the relationship between a pair of feature points in f(t-1) and f(t) is: Where H is a homogeneous affine matrix: Among them, a ij is the parameter about the rotation angle θ, T x and T y Double are the parameters of translation T on the x-axis and y-axis respectively; The method to achieve displacement compensation is: use Kalman filter to compensate the translation and rotation trajectory, thereby generating a new transformation for each frame; Step S8: Detect and identify targets in the foreground of the aerial image; Step S9, obtaining motion vectors of two consecutive images; Step S10: Detect the unlabeled target data D2 and obtain a detection result.

2. The SimROD-based aerial image detection and recognition method according to claim 1, characterized in that: The specific steps in step S2 include: Step S201: Segment an image into smaller pixel slices and select a reference slice; Step S202: Find small pieces similar to the reference piece to form a 3D block; Step S203: perform 3D transformation on all similar blocks and perform thresholding; Step S204: perform 3D inverse transformation, and restore all 3D blocks to the image through weighted averaging.

3. The SimROD-based aerial image detection and recognition method according to claim 2, characterized in that: In step S3, each label of the collected image is labeled and consists of a target category and a bounding box coordinate.

4. The SimROD-based aerial image detection and recognition method according to claim 1, characterized in that: The step S5 adopts a stepwise adaptation method to optimize the parameters of the detection model, and the specific steps include: Step S501: Freeze all convolutional layers and only adapt the BN layer in the first w epochs. After the first stage, the parameters of the BN layer are updated. Step S502: Use the partially adapted teacher model to generate more accurate pseudo labels.

5. The SimROD-based aerial image detection and recognition method according to claim 1, characterized in that: In step S8, the background and foreground of each frame of the aerial image are determined by image stabilization technology, wherein the foreground consists of candidate moving targets in the image, and the foreground is detected and identified using Haar-like features and an enhanced cascade classifier; The specific steps include: Step S801: Haar-like features are mainly used to detect objects of different sizes in the image; Step S802: Using the AdaBoost algorithm to combine the features of the selected classifiers, the classifier is selected as a value to determine the best classification function for each feature.

6. The SimROD-based aerial image detection and recognition method according to claim 5, characterized in that: The step S9 uses the Farneback optical flow method to obtain the motion vectors of the two consecutive images.

Images