Lightweight refueling taper sleeve detection method based on two-stage cascading

Abstract

The invention discloses a two-stage cascade lightweight refueling taper sleeve detection method based on an attention mechanism. The method comprises the following steps: 1, constructing a taper sleeve image data set as a training set of a taper sleeve contour detection method, and preprocessing the data set; the method comprises the following steps: constructing two-stage cascaded lightweight taper sleeve detection combined by a first-stage coarse detection sub-network and a second-stage fine detection sub-network; performing end-to-end offline supervised training on the network by using the constructed taper sleeve image data set to generate a taper sleeve contour detection network with a high-precision taper sleeve contour detection effect, and performing model quantization and conversion on the trained lightweight taper sleeve detection network; and 2, deploying the converted model on an embedded platform to realize taper sleeve contour detection. According to the method, the robustness of taper sleeve detection in a complex scene is greatly improved, the adaptability to illumination variation and environment shielding is enhanced, and the detection precision and the real-time performance are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of refueling cone sleeve contour detection, and particularly relates to a lightweight refueling cone sleeve detection method based on two-stage cascading. BACKGROUND

[0002] Air refueling is a key technology for providing fuel supply to a receiving aircraft in flight through an air refueling device. A visual image navigation system acquires images in a refueling docking process of the aircraft by using a camera, and automatically detects and identifies a refueling cone sleeve by using computer vision and machine learning methods, and outputs attitude, motion and structure information of the refueling cone sleeve in real time, so as to realize accurate guidance of the docking process, and is a core means for realizing automation of air refueling operation.

[0003] Existing refueling cone sleeve detection methods based on machine vision can be divided into two categories: a method based on a traditional computer vision algorithm and a method based on deep learning. The former method realizes detection of a cone sleeve region by designing an image processing operator and using template matching and image segmentation. This method has poor adaptability to environmental changes and complex scenes. The latter method constructs an end-to-end convolutional neural network, trains a feature extractor and a detector by using a large number of labeled samples, and directly predicts a coordinate position or an external rectangle frame of the cone sleeve. The main flowchart of the method is shown in FIG. 1, but the robustness of this method to changes in the shape of the cone sleeve and occlusion is still insufficient. Figure 1

[0004] Therefore, some strategies combine deep networks with traditional algorithms or tracking methods to form a complement. Overall, an end-to-end deep learning method is superior to a traditional computer vision algorithm in detection performance, and is a relatively mainstream technical route at present. However, the performance of the method is restricted by limited scene samples.

[0005] The environmental adaptability and robustness of existing detection methods to complex application scenes still face challenges. Factors such as target occlusion, shape changes, light effects and motion blur can seriously affect the detection performance. At the same time, there are high requirements for detection accuracy, speed and stability. Therefore, to realize stable and robust detection of the refueling cone sleeve under complex conditions and to construct a representative data set are the key to further development of the technology. SUMMARY

[0006] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a lightweight refueling cone sleeve detection method based on two-stage cascading, which greatly improves the robustness of cone sleeve detection under complex scenes, enhances the adaptability to changes in light and environmental occlusion, and significantly improves the detection accuracy and real-time performance. At the same time, the method is designed in software and hardware collaboration and optimized in model compression, realizing efficient and lightweight deployment on an RK3588 embedded processing platform, expanding the application range of the system, and providing a possibility for engineering implementation. ​

[0007] To achieve the above object, the technical scheme adopted by the present application is:

[0008] The two-stage cascaded lightweight refueling cone sleeve detection method based on the attention mechanism establishes a first-stage rough detection sub-network and a second-stage fine detection sub-network of an integrated attention mechanism based on a YOLO (You Only Look Once) detection model, a cascaded tracking method and a cone sleeve contour correction method, and iteratively performs supervised offline training and efficient online inference to generate an optimal refueling cone sleeve contour detection method for the detection of the cone sleeve of an actual air refueling machine, mainly including the following two steps.

[0009] Step one: offline training step,

[0010] A cone sleeve image dataset is constructed as a training set for the cone sleeve contour detection method, and the dataset is preprocessed; a two-stage cascaded lightweight cone sleeve detection network combining the first-stage rough detection sub-network and the second-stage fine detection sub-network is constructed; the network is trained end-to-end offline using the cone sleeve image dataset, and the trained lightweight cone sleeve detection network is model quantized and converted;

[0011] Step two: online inference step,

[0012] The converted model is deployed on an embedded platform, including forward inference and post-processing of the cone sleeve contour detection network, and further enhancing the cone sleeve contour detection effect according to the cascaded tracking and cone sleeve contour correction method.

[0013] The step one is specifically:

[0014] Step 1): Obtain samples and construct a cone sleeve dataset:

[0015] Step 2): Screen and classify the cone sleeve dataset, screen effective samples that meet the model training requirements, and respectively construct the first-stage rough detection sub-network and the second-stage fine detection sub-network corresponding dataset:

[0016] The first-stage rough detection sub-network corresponding dataset contains complete background cone sleeve images;

[0017] The second-stage fine detection sub-network dataset only contains small size images of local cone sleeve regions;

[0018] The second-stage fine detection sub-network corresponding dataset is obtained by cropping the first-stage rough detection sub-network corresponding dataset;

[0019] Step 3): Label the data sets corresponding to the first-level coarse detection sub-network and the second-level fine detection sub-network. For the data set corresponding to the first-level coarse detection sub-network, the image aspect ratio is kept unchanged, and the nearest neighbor interpolation or bilinear interpolation method is used to scale the image. The 4 vertex coordinates of the rotated rectangular frame are labeled;

[0020] Step 4): Perform data augmentation on the data set obtained in step 3). Use HSV color space enhancement, random cropping, rotation scaling, mosaic enhancement, Cutout, and CutMix to enrich data diversity, improve model robustness, and reduce GPU storage requirements;

[0021] Step 5): Train the two-level cascaded lightweight cone detection network using the data augmented data set;

[0022] Step 6): Based on deployment requirements, perform model compression and conversion on an embedded platform with RK3588, which integrates a dedicated neural network processing unit (NPU).

[0023] The step 1) of step one is three ways:

[0024] The first way is to synthesize cone images in different scenes through 3D modeling software. Specifically, 3D models of different types, sizes, and materials are constructed, the position, angle, and lighting parameters of the models are randomly set, and complex backgrounds are added to simulate different weather and actual scenarios of the fuselage.

[0025] The second way is to arrange real cone in the ground environment, adjust the spatial position parameters, and use image equipment to collect static and micro-motion image samples from multiple angles.

[0026] The third way is to collect relevant image or video samples of real in-flight refueling process from the network under the premise of permission and privacy protection.

[0027] The step 3) of step one is as follows:

[0028] For the large size data set of the first level coarse detection sub-network, the synthetic samples are directly labeled by retaining the 3D modeling information; the ground shooting samples are labeled by adding marker lights on the cone to realize positioning and fitting calculation; the network collection samples are manually labeled using labeling tools;

[0029] For the small size data set of the second level fine detection sub-network, region interception is performed based on the labeling of the large size data set of the first level coarse detection sub-network;

[0030] Specifically, the second-level fine detection sub-network dataset is cropped from the horizontal rectangular frame region surrounded by the four vertices of the rotated rectangular frame labeled in the first-level coarse detection sub-network dataset, and the cropped image is scaled using nearest neighbor interpolation or bilinear interpolation while maintaining the aspect ratio. If the object is located at the edge of the input image, resulting in part of the object's region being outside the input image, the missing part is filled with 0 to obtain a completely square image, obtaining the second-level fine detection sub-network dataset image. The label is calculated by the relative relationship between the cropped image region and the first-level coarse detection sub-network dataset image and the scaling ratio, and does not need to be labeled separately.

[0031] In step 4) of step one, the HSV color space is composed of hue (Hue), saturation (Saturation), and brightness (Value); the mosaic enhancement is to randomly arrange and splice four pictures; the Cutout enhancement is to randomly cut part of the region in the sample and fill it with 0 pixel values; and the CutMix enhancement is to cut part of the region but not fill it with 0 pixels but randomly fill the pixel values of the region of other data in the training set.

[0032] In step 5) of step one, the two-level cascaded lightweight cone detection network is composed of a first-level coarse detection sub-network based on an improved YOLO detection model with integrated attention mechanism and a second-level fine detection sub-network. The network structures of the first-level coarse detection sub-network and the second-level fine detection sub-network are the same, except that the training set image sizes used by the two networks are different.

[0033] The improved K-Means++ clustering algorithm is used to obtain the optimal preset anchor box based on the size of the labeled frame; angle classification information is added to realize the conversion from horizontal frame to rotated frame; an Intersection over Union (IOU) calculation method for rotated frame is designed; the loss function is improved to include four parts of loss; and the final model directly predicts the class, position coordinates, scale, direction, etc. of the cone in the image. The network structure improvement will be described in detail later.

[0034] In step 6) of step one, the 32-bit floating-point model is compressed to an 8-bit signed integer model using 8-bit asymmetric dynamic fixed-point quantization; then through the Rockchip Neural Network (RKNN) model conversion tool, it is first converted into an open neural network exchange format model, and then further converted into an RKNN format, finally obtaining the one-two RKNN models of the first-level coarse detection sub-network and the second-level fine detection sub-network suitable for embedded platforms; the simplified 8-bit asymmetric quantization formula is:

[0035]

[0036] Q(x) is the quantized 8-bit signed integer value, x is the input 32-bit floating point number, clip() means limiting the value in the specified interval, round() means rounding the value to the nearest integer, max_range means the maximum value of the floating point number tensor, is a scaling factor for mapping the floating point number to the integer range.

[0037] The step two is specifically:

[0038] Step 1) image preprocessing: initializing the video acquisition interface using the media processing software platform provided by RK3588; obtaining the input image containing the cone sleeve in real time from the camera or video stream; adjusting the size to adapt to the input requirements of the first level coarse detection sub-network and the second level fine detection sub-network respectively, normalizing the pixel value to improve the stability of the model, and adjusting the color space to BGR order to match the training model;

[0039] Step 2) one-two level RKNN model loading, execution and post-processing: calling the general (Application Programming Interface, API) interface to load the model, apply for memory, model inference, and the forward inference result is subjected to the post-processing operation realized by the application to obtain the rotation frame of the cone sleeve, and finally the memory is released;

[0040] Step 3) cascade detection: judging whether the approximate position of the cone sleeve is known, if not, inputting the preprocessed image into the loaded first level coarse detection sub-network to roughly locate the cone sleeve region, and cutting, scaling and filling the located image region, and sending the processed image region into the loaded second level fine detection sub-network for accurate detection; if the target region is known, directly inputting the image into the loaded second level fine detection sub-network for accurate cone sleeve detection using the position information;

[0041] Step 4) cone sleeve contour correction: inputting the detection result of the second level fine detection sub-network into the constructed cone sleeve contour feature correction module, calculating the deviation between the detection frame and the actual cone sleeve contour based on the distance loss function, then updating the detection frame parameters using the gradient descent method, iteratively reducing the frame contour distance, and finally correcting and outputting the detection frame result binding the actual contour;

[0042] Step 5) coordinate conversion: the cone sleeve contour correction module outputs the coordinates of the elliptical detection frame in the intercepted cone sleeve local image region, and through the designed global coordinate conversion module, realizes the mapping and conversion of the local coordinates to the original image coordinate system, and obtains the global coordinate information of the detection frame in the original image space;

[0043] Step 6) tracking feedback: the current frame cone sleeve detection box coordinates converted to the original image coordinate system are output in two ways: one is directly as a detection result input into the subsequent process to complete the downstream task; the other is as a next frame search target area feedback to the secondary detection network to realize the fusion of detection and tracking and improve the detection robustness by using the time sequence information.

[0044] In step 2) of step two, an RKNN model is loaded through a model (load_data) interface, and the loaded RKNN model is initialized through an initialization function (rknn_init), then the frame data structure body (rknn_input) is initialized according to the input image frame, and the input of the model (rknn_inputs_set) is set, then the model inference (rknn_run) is carried out, the three-scale 8-bit feature data (rknn_outputs_get) after inference is obtained by using the parallel computing unit of NPU, the three-scale 8-bit feature data is used to obtain the cone sleeve coordinate information, and the rotation non-maximum suppression (NMS) realized by the application is used to remove redundant boxes, finally, the cone sleeve contour rotation box is obtained, and the memory is released.

[0045] The beneficial effects of the application are as follows:

[0046] 1) The cone sleeve image dataset constructed by the application can cover the cone sleeve images in the actual environment. Different scenes, different postures and different types of cone sleeve images are made and collected through 3D modeling, physical shooting and web crawling, etc. to construct a rich cone sleeve sample dataset, which provides rich training samples for the cone sleeve detection network and improves the detection ability of the network for the refueling cone sleeve contour in different scenes.

[0047] 2) The two-stage cascade network mentioned in the application is light and efficient. The separable convolution block is designed to reduce the parameter amount while the network can also obtain good feature expression ability; the efficient channel attention module is designed to combine spatial and channel attention to filter and fuse more discriminative multi-scale features; and the deformable convolution module is designed to learn the sampling point offset and the weight of each sampling point to reduce irrelevant interference and better capture the details of the cone sleeve in different positions and scales.

[0048] 3) The refueling cone sleeve contour detection method proposed by the application can realize real-time and efficient detection of refueling cone sleeves in the actual environment. Through the design of the rotation box detection method, the cascade tracking method and the cone sleeve contour correction method, the detection ability of the model for the cone sleeve in different scale changes, severe occlusion, image blur, extreme posture deformation and noise interference in complex application scenarios is improved, and high-precision fitting of the pixel-level cone sleeve center coordinates is realized, which provides a basis for accurate control and navigation in the subsequent docking process.

[0049] 4) Through the software and hardware cooperation and optimization of RK3588 embedded processing platform, the high-precision cone sleeve contour detection model trained offline on the computer platform is quantized and accelerated, so that it meets the real-time detection requirements of the embedded platform under the condition of high-speed motion, and is suitable for scenes such as aircraft in-flight refueling, unmanned aerial vehicle docking and other scenes that require strict system response time.

[0050] Compared with the existing detector, the detection accuracy and environmental adaptability of the present application are improved, which lays a foundation for ensuring the visual stability of complex processes such as aircraft in-flight refueling and unmanned aerial vehicle docking. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A deployment flowchart of the existing basic algorithm for cone sleeve detection.

[0052] Figure 2 A development and application schematic diagram of the cone sleeve detection model based on the embedded platform of the present application.

[0053] Figure 3 A detailed flowchart of the method for realizing cone sleeve detection according to the present application.

[0054] Figure 4 A structure schematic diagram of the first-stage coarse detection sub-network and the second-stage fine detection sub-network designed in the present application.

[0055] Figure 5 A detailed structure schematic diagram of the first-stage coarse detection sub-network and the second-stage fine detection sub-network components designed in the present application.

[0056] Figure 6 A schematic diagram of the detection head inside for realizing the calculation of the cone sleeve coordinates and the rotation angle.

[0057] Figure 7 A schematic diagram of the cone sleeve contour geometry correction module designed in the present application. DETAILED DESCRIPTION

[0058] The present application provides a refueling cone sleeve contour detection method, which will be further described in detail below in combination with embodiments. The method steps are described in combination with the accompanying drawings, please refer to Figures 1-7 .

[0059] Figure 1The deployment flowchart of the existing base algorithm for realizing the cone sleeve detection is mainly divided into offline tasks and online tasks, wherein the offline tasks are used for offline training of the designed cone sleeve detection model, saving the model weight information after training, then model conversion according to the selected hardware deployment platform, and generating a model capable of running on the hardware deployment platform; the online tasks are used for loading the converted model on the hardware deployment platform and detecting the input cone sleeve image, and outputting the final cone sleeve frame parameters.

[0060] Figure 2 The application development application diagram of the cone sleeve detection model based on the embedded platform is mainly used for model development through a computer platform end and an embedded platform end; wherein the computer platform end is used for cone sleeve detection model training and model quantization conversion; the embedded platform end is used for forward inference of the cone sleeve detection model and post-processing of the detection result.

[0061] As Figure 3 The deployment flowchart of the method is mainly divided into two processes of offline training tasks of the computer end and online inference tasks of the embedded platform end;

[0062] The offline task of the computer end is used for training the designed integrated attention mechanism two-stage cascaded lightweight cone sleeve detection network model, and performing quantization and format conversion on the trained model to an RK3588 embedded platform capable of loading (Rockchip Neural Network, RKNN) model;

[0063] The online inference task of the embedded platform end is used for real-time processing of the cone sleeve target image to be detected, loading the RKNN model generated by the computer end offline task, and outputting the cone sleeve detection frame parameters for downstream tasks. The embodiment of the application specifically includes the following steps:

[0064] Offline training step:

[0065] Step 1): build a cone sleeve dataset, mainly from three ways to obtain samples:

[0066] First, synthesize cone sleeve images in different scenes through a 3D modeling software, specifically, construct 3D models of cone sleeves of different models, sizes and materials, randomly set the position, angle and illumination parameters of the model, and add a complex background to simulate different actual scenes such as weather and fuselage;

[0067] Second, arrange real cone sleeves in the ground environment, adjust the spatial position parameters, and use image equipment to collect static and micro-motion image samples at multiple angles;

[0068] Third, collect relevant image or video samples of real in-flight refueling process from the network under the premise of permission and privacy protection.

[0069] Step 2): screening and classifying the cone sleeve dataset, screening the effective samples meeting the model training requirements, and respectively constructing the first level coarse detection sub-network and the second level fine detection sub-network corresponding data sets:

[0070] The first level coarse detection sub-network data set contains complete background cone sleeve images, and in the present embodiment, a large size of 640x640 pixels is adopted;

[0071] The second level fine detection sub-network data set only contains small size images of local regions of the cone sleeve, and in the present embodiment, a small size of 224x224 pixels is adopted, wherein the second level fine detection sub-network data set is obtained by cropping the first level coarse detection sub-network data set.

[0072] Step 3): labeling the first level coarse detection sub-network data set and the second level fine detection sub-network data set, for the first level coarse detection sub-network data set cone sleeve image, the image is scaled by using nearest neighbor interpolation or bilinear interpolation method under the condition that the aspect ratio of the image is unchanged, and it is ensured that the longest side size of the scaled image is 640 values, and the 4 vertex coordinates of the rotated rectangular frame are used for labeling format.

[0073] For the large size data set of the first level coarse detection sub-network, the synthetic samples directly obtain the label by retaining the 3D modeling information; the ground real shot samples are positioned and fitted to obtain the label data by adding marker lights on the cone sleeve; the network collection samples manually complete the labeling using the labeling tool.

[0074] For the small size data set of the second level fine detection sub-network, the region is cropped to obtain the second level fine detection sub-network data set based on the labeling of the large size data set of the first level coarse detection sub-network; specifically, the second level fine detection sub-network data set is cropped from the horizontal rectangular frame region surrounded by the 4 vertexes of the rotated rectangular frame labeled in the first level coarse detection sub-network data set, and the cropped image is scaled by using nearest neighbor interpolation or bilinear interpolation method under the condition that the aspect ratio of the image is unchanged. And it is ensured that the longest side size of the scaled image is 224 values, if the object is located at the edge of the input image, resulting in part of the region where the object is located being outside the input image, 0 can be used to fill the missing part to obtain a completely square image, and the second level fine detection sub-network data set image is obtained. Therefore, the second level fine detection sub-network data set image size is 224x224 pixels. The label is calculated by the relative relationship between the cropped image region and the first level coarse detection sub-network data set image and the scaling ratio of the image, and does not need to be labeled separately.

[0075] Step 4): Perform data augmentation on the cone dataset obtained in Step 3), using HSV color space enhancement, random cropping, rotation and scaling, mosaic enhancement, Cutout, and CutMix to enrich data diversity, improve model robustness, and reduce GPU storage requirements.

[0076] The HSV color space is composed of hue, saturation, and value. Mosaic enhancement involves randomly arranging and stitching together four images. Cutout enhancement involves randomly cutting out a portion of the sample and filling it with 0 pixel values. CutMix enhancement involves cropping out a portion of the sample but not filling it with 0 pixels; instead, it randomly fills it with pixel values ​​from other data in the training set.

[0077] Step 5): Use the data-augmented cone-shaped dataset to train the constructed first-level coarse detection subnetwork and the second-level fine detection subnetwork respectively, to obtain the 32-bit floating-point cone-shaped detection model of the first-level coarse detection subnetwork and the second-level fine detection subnetwork.

[0078] The two-stage cascaded lightweight cone detection network of this invention consists of a first-stage coarse detection sub-network and a second-stage fine detection sub-network based on an improved YOLO detection model and an integrated attention mechanism. The first-stage coarse detection sub-network and the second-stage fine detection sub-network have the same network structure. The main improvements include: using an improved K-means++ (K-Means++) clustering algorithm instead of the traditional K-means clustering algorithm to obtain the optimal preset anchor box based on the bounding box size; incorporating angle classification information to achieve the conversion from horizontal to rotated boxes; designing a method for calculating the Intersection over Union (IOU) of the rotated box; and improving the loss function to include four parts. The final model directly predicts the category, location coordinates, scale, and orientation of the cone in the image. The network structure improvements will be described in detail later.

[0079] Step 6): Based on deployment requirements, model compression and conversion are performed on an embedded platform equipped with RK3588, which integrates a dedicated Neural Network Processing Unit (NPU). This invention uses 8-bit asymmetric dynamic fixed-point quantization to compress the 32-bit floating-point model into an 8-bit signed integer model; then, using the Rockchip Neural Network (RKNN) model conversion tool, it is first converted to an open neural network exchange format model, and then further converted to RKNN format, ultimately obtaining a first- and second-level RKNN model suitable for the embedded platform, consisting of a first-level coarse detection subnetwork and a second-level fine detection subnetwork; the simplified 8-bit asymmetric quantization formula is:

[0080]

[0081] Q(x) is the quantized 8-bit signed integer value, x is the input 32-bit floating point number, clip() means limiting the value in the specified interval, round() means rounding the value to the nearest integer, max_range represents the maximum value of the floating point number tensor, is a scaling factor for mapping the floating point number to the integer range.

[0082] Online inference steps:

[0083] Step 1) image preprocessing: initialize the video capture interface using the media processing software platform provided by RK3588; obtain the input image containing the cone sleeve from the camera or video stream in real time; resize to 640x640 and 224x224 to adapt to the input requirements of the first level coarse detection sub-network and the second level fine detection sub-network respectively, normalize the pixel value to improve the stability of the model, and adjust the color space to BGR order to match the training model.

[0084] Step 2) one-two level RKNN model loading, execution and post-processing: call the general API interface for model loading, memory allocation, model inference, etc., and the forward inference result is subjected to post-processing operation to obtain the rotation box of the cone sleeve, and finally complete memory release.

[0085] Specifically, load the RKNN model through the model (load_data) interface, and initialize the loaded RKNN model through the initialization function (rknn_init), then initialize the frame data structure body (rknn_input) according to the input image frame, set the input of the model (rknn_inputs_set), then perform model inference (rknn_run), use the parallel computing unit of NPU to obtain the three-scale 8-bit feature data (rknn_outputs_get) after inference, obtain the cone sleeve coordinate information from the three-scale 8-bit feature data, and remove redundant boxes according to the rotation non-maximum suppression NMS implemented by the present application, finally obtain the cone sleeve contour rotation box, and complete memory release.

[0086] Step 3) cascade detection: determine whether the approximate position of the cone sleeve is known, if not, input the preprocessed image into the loaded first level coarse detection sub-network for rough positioning of the cone sleeve region, and crop, scale and fill the positioned image region, the specific steps are the same as step 3) of the offline training task, and the processed image region is sent into the loaded second level fine detection sub-network for accurate detection; if the target region is known, directly input the image into the loaded second level fine detection sub-network for accurate cone sleeve detection using the position information.

[0087] Step 4) Conical sleeve contour correction: input the second level detection result into the constructed conical sleeve contour feature correction module, calculate the deviation between the detection frame and the actual conical sleeve contour based on the distance loss function, then update the detection frame parameters using gradient descent method, iteratively reduce the frame contour distance, and finally correct the output detection frame result that binds the actual contour.

[0088] Step 5) Coordinate conversion: the conical sleeve contour correction module outputs the coordinates of the elliptical detection frame in the intercepted conical sleeve local image area, and through the designed global coordinate conversion module, realizes the mapping conversion of local coordinates to the original image coordinate system, and obtains the global coordinate information of the detection frame in the original image space.

[0089] Step 6) Tracking feedback: the current frame conical sleeve detection frame coordinates converted to the original image coordinate system are output in two ways: one is directly input as the detection result into the subsequent process to complete the downstream task; the other is fed back to the second level detection network as the target area for searching in the next frame, realizing the fusion of detection and tracking, and improving the detection robustness by using time sequence information.

[0090] The following describes in detail the conical sleeve detection network structure in step 5) in the offline training task:

[0091] The two-level cascaded lightweight conical sleeve detection network is composed of a first level rough detection sub-network and a second level fine detection sub-network based on the YOLO detection model improved with an integrated attention mechanism, wherein the lightweight design of the first level rough detection sub-network uses a separable convolution block structure (Depthwise Convolutional Block Structure, DCBS) and a deformable convolution C3 structure (Deformable Convolutional C3 Structure, DC3) to replace the standard convolution block structure (Convolutional Block Structure, CBS) and the 3 convolution block structure (Three Convolutional Block Structure, C3) modules of the original YOLO model, the integrated attention mechanism design embeds an efficient channel attention (Efficient Channel Attention, ECA) channel attention module in the backbone network, and introduces a convolution attention module (Convolutional Block Attention Module, CBAM) combining a channel attention module (Channel attention module, CAM) and a spatial attention module (Spatial attention module, SAM) in the neck feature fusion layer, which realizes the weighting and selection of effective feature information, and the detection head module adds the prediction of the rotation angle to realize the output of the rotating frame.

[0092] The network structure of the second-level fine detection sub-network is the same as that of the first-level coarse detection sub-network, and the difference lies in that the image sizes of the training sets adopted by the two networks are different, thereby causing different attention degrees of model training. The two-level cascade mainly takes the rotated box region containing the target object output by the first-level coarse detection sub-network as the input of the second-level fine detection sub-network, and finally converts the output of the second-level fine detection sub-network into a global coordinate for a downstream task. The specific cascade process refers to each step of the online inference task.

[0093] As shown in Figure 4 Fig. 1 is a structural schematic diagram of the first-level coarse detection sub-network and the second-level fine detection sub-network designed by the present application, Figure 5 Fig. 2 is a detailed structural schematic diagram of the components of the first-level coarse detection sub-network and the second-level fine detection sub-network designed by the present application, Figure 6 Fig. 3 is a schematic diagram of the internal implementation of the cone sleeve coordinate and the rotation angle calculation of the detection head designed by the present application, and the network detailed design steps are as follows:

[0094] 1) Lightweight design. In order to realize the lightweight of the network structure, in the backbone feature extraction network, the original standard convolution block CBS is replaced by a lighter separable convolution block DCBS. It is inspired by the depth separable convolution idea in MobileNet, which separates the standard convolution operation into depthwise convolution (DepthWise, DW) to learn the spatial features within the channel and pointwise convolution (PointWise, PW) to fuse the features between channels. This strategy greatly reduces the operation amount and parameter size. In the DCBS module, the DW convolution is followed by the PW convolution, the Batch Normalization (BN) layer and the Sigmoid Linear Unit (SiLU) activation function to extract and fuse the features. The module is repeatedly used to build the entire detection network. Experiments prove that the DCBS module also has good feature expression ability under the condition of reducing the parameter amount.

[0095] 2) Attention module. In order to reduce the interference of background information on the detection of the cone sleeve and focus on the cone sleeve related features, the present application embeds a lightweight ECA efficient channel attention module in the backbone network, which focuses on the effective features by recalibrating the correlation between different channels. At the same time, CBAM is connected at the neck, which combines spatial and channel attention to filter and fuse more discriminative multi-scale features, and improves the feature extraction ability of the model.

[0096] 3) Deformable convolution module. In order to process deformation, occlusion and irregular cone sleeve targets, the present application designs a DC3 integrated with a deformable convolution module to better capture the details of the cone sleeve at different positions and scales.

[0097] Specifically, the DCNBS module is defined, and the original standard convolution block CBS in C3 is replaced by a deformable convolution network (Deformable ConvNets v2, DCNv2). The DCNv2 not only learns the sampling point offset, but also learns the weight of each sampling point, reduces irrelevant interference, and improves the model expression ability and adaptability. However, the use of DCNv2 in large quantities will increase the complexity of the model. Through experiments, it is determined that replacing only the 3-layer C3 module in front of the fast spatial pyramid pooling module can achieve a balance between detection performance and computational efficiency, and significantly alleviate the detection difficulty of the cone sleeve occlusion. The DCNv2 can be expressed as:

[0098]

[0099] where Δp k and Δm k are the learned offset and weight of the k-th position, respectively, Δm k ranges from 0 to 1, Δp k is a real number with an unrestricted range, and x(p+p k + Δp k ) is calculated using bilinear interpolation, Δp k and Δm k are obtained by applying a separate convolution layer to the same input feature map x.

[0100] 4) Obtain the optimal prior box for cone sleeve detection based on the improved K-Means++ clustering algorithm. The prior box of the original YOLO model is based on the clustering results of the coco and other data sets, which cannot well adapt to the cone sleeve target.

[0101] The method reconstructs the cone sleeve data set, and uses the improved K-Means++ instead of the original K-Means algorithm for clustering, to avoid the randomness of the initial center selection of the latter and prevent the result from converging to a local optimum. Through multiple experiments and comparisons, when the number of clustering categories is 9, the effect is best. That is, 9 prior box sizes are obtained, and the network 3 different scale output layers are used for regression prediction. The specific steps of the clustering algorithm are as follows

[0102]

[0103] 5) Based on the CSL rotating box detection. The traditional rotating box based on the long side representation causes the regression loss to surge due to the periodicity of the angle, increasing the training difficulty. The method changes the angle prediction from regression to classification, divides the 0-180 degree range into 180 categories, and sets a window function to smooth the label boundary. Each detection anchor predicts the following content: center coordinates, bounding rectangle length and width, detection category, angle category, a total of 5 items. Among them, the detection category is small target top circle and large target cone sleeve, and the angle category number is 180. Details can be seen in Figure 6This design avoids the loss of angle periodicity caused by the shock, which is conducive to network convergence.

[0104]

[0105] g(x) in the formula is a window function, and the window radius is controlled by r. The window function can use impulse function, rectangular function, triangular function, Gaussian function, etc. Due to the ingenious setting of the window function, the model can accurately measure the angle distance between the predicted label and the ground truth label. Specifically, the model converts the angle distance between the predicted value and the true value within a certain range into a loss value, so that the closer the predicted value is to the true value within the range, the smaller the loss generated by the predicted value. It is worth mentioning that by introducing periodicity, the problem of angle periodicity is solved, so that angles such as 89 degrees and -90 degrees become close, better meeting the actual application scenario.

[0106] 6) Loss function improvement. The loss function contains four parts: confidence loss, class loss, angle loss and bounding box regression loss. The first three items use the binary cross entropy loss function with logarithm (Binary Cross Entropy with Logits Loss, BCEWithLogitsLoss), which integrates the advantages of logistic activation function (Logistic Function, Sigmoid) and binary cross entropy (Binary Cross Entropy Loss, BCELoss), realizes numerical stability, efficient calculation, and is suitable for unbalanced and multi-label classification. Its expression is:

[0107]

[0108] where N is the total number of samples y i is the class to which the ith sample belongs, p i is the prediction probability of the ith sample.

[0109] The existing YOLO model uses generalized intersection over union (GIOU) as the regression loss function of the bounding box. Although GIOU can reflect the distance problem of IOU in the case of no overlap, it cannot effectively distinguish the case that the prediction box is in the target box and the size is the same, resulting in difficulty in model training convergence. In order to solve the problem of GIOU and strengthen the detection and adaptability of complex attitude non-positive cone, the present application introduces the prediction of the rotating frame. In the calculation of the rotating frame loss, two parts of angle loss and horizontal frame loss are included, which decouples the angle and frame parameter information, reduces the training time, and promotes the model to converge faster. In order to solve the problem of the loss function of the horizontal frame regression of GIOU, the present application uses complete intersection over union (CIoU) as the loss function of the horizontal frame regression. The CIoU loss function comprehensively considers the overlapping area, center point distance and aspect ratio of the prediction box and the target box, so as to more comprehensively describe the accuracy of the target bounding box. The calculation formula of CIoU is as follows:

[0110]

[0111] Wherein, ρ 2 (b,b gt ) represents the Euclidean distance between the center points of the prediction box and the real box, b and b gt The center points of the prediction box and the real box, c represents the diagonal distance of the smallest closed region that can contain the prediction box and the real box, a is a positive balance parameter, and v is a similarity measure of the aspect ratio, and the corresponding formula is as follows:

[0112]

[0113] Therefore, the loss function of CIoU is defined as:

[0114] L CIoU =1-CIoU (8)

[0115]

[0116] The confidence of the prediction box represents its accuracy, and the greater the value means that the prediction result is more reliable and closer to the true value. The original YOLO method only considers the existence of the target, and sets the confidence label of the grid containing the target to 1, and the rest to 0. This may cause the prediction box to be not accurate enough. The present application provides a method for directly setting the confidence label based on the rotating IOU of the prediction box and the true value box. In addition, the weight coefficient of the confidence loss on different scale feature maps is set, and the loss weight of the large scale feature map for detecting small targets is larger, so that the network pays more attention to small targets in training.

[0117] This invention provides a method for calculating the Intersection over Union (IoU) between two rotated bounding boxes. By extracting key information from the parameters of the rotated boxes, including center coordinates, width, height, and rotation angle, and by employing a center point translation step to improve calculation accuracy, this method effectively handles the geometric relationship between the rotated boxes. Specifically, by calculating the intersection points of the rotated boxes, the Graham scan algorithm is applied to sort the intersection points, thereby constructing a convex hull and calculating its area. Finally, the IoU value between the rotated boxes is obtained by dividing the intersection area by the union area of ​​the rotated boxes. This method not only considers the geometric characteristics of the rotated boxes but also provides efficient and accurate IoU calculation, making it suitable for precise measurement of rotated targets in object detection and computer vision tasks.

[0118] Calculation of intersection of rotating rectangles

[0119] For two given rotating rectangles, with P1, P2, P3, P4 and Q1, Q2, Q3, Q4 representing their four vertices respectively, the intersection point is calculated using the following steps: First, the intersection point of adjacent sides is calculated. For each pair of adjacent sides of the rectangles, the intersection point is calculated using the following formula:

[0120]

[0121] in, and Let be a vector representing the i-th side of rectangle 1 and the j-th side of rectangle 2. t1 and t2 are parameters used to calculate the intersection point of adjacent sides, representing the projection ratio onto the adjacent sides. If both t1 and t2 are within the interval [0, 1], then the intersection point is calculated:

[0122]

[0123] Next, we check the vertices within the projection. For vertex P of rectangle 1... i Check if it lies within the projection of rectangle 2, and calculate P. i On the side and Projection point P on i ':

[0124]

[0125] If P i 'On the side and Between, Inside rectangle 2. For vertex Q of rectangle 2. i Similarly, check whether it is within the projection of rectangle 1.

[0126] Through the above steps, all intersection points between two rotating rectangles can be accurately calculated. This method provides an efficient and reliable rotating rectangle intersection point calculation method, which is suitable for various calculation scenarios.

[0127] Graham scan algorithm for constructing convex hull

[0128]

[0129]

[0130] Intersection area calculation

[0131] The area of a polygon is calculated using the Shoelace Formula. For a given point set Q = {q1,...,q M}, where q i = (x i ,y i ), the area of the polygon can be calculated by the following formula:

[0132]

[0133] where (x0,y0) is the first point in the point set, and m is the number of edges of the polygon. The formula is based on dividing the polygon into multiple triangles, calculating the area of each triangle, and accumulating the total area of the polygon.

[0134] Rotated bounding box IOU calculation

[0135] The final IoU calculation formula for the rotated bounding box can be expressed as follows:

[0136]

[0137] where Area1 and Area2 represent the areas of the rotated bounding boxes 1 and 2 respectively, and Area is the intersection area calculated by the above formula.

[0138] The present application designs a cone sleeve contour correction module to further improve the accuracy of the cone sleeve detection effect. Based on the two-level network detection result, the prediction box parameters are optimized and adjusted to more accurately match the true contour of the cone sleeve target, effectively reduce the boundary error, and improve the accuracy of the center point calibration.

[0139] As shown in Figure 7 , it is a structural schematic diagram of the cone sleeve contour correction module.

[0140] The specific technical steps are as follows:

[0141] 1) First, after the cone sleeve secondary detection sub-network outputs the detection result, the application intercepts the original image region corresponding to the detection frame. In this region of interest (Region of Interest, ROI), the mean filter is used to remove the tiny interference edge, and then the Canny operator is used to extract the initial edge contour. Then, the application connects the edge with the help of morphological processing technique to generate a closed contour, and removes any possible false contour based on prior knowledge.

[0142] 2) Next, the application constructs a cone sleeve frame contour fitting degree optimization model, and the objective function of the model represents the modified Hausdorff distance between the initial edge contour extracted in the previous step and the inscribed maximum ellipse cone sleeve contour in the detection frame. The objective function is represented as:

[0143]

[0144] Where A and B represent the sample point set of the initial edge contour and the sample point set of the inscribed maximum ellipse cone sleeve contour in the detection frame, respectively, and N A represents the number of sample points in set A. The setting of the objective function aims to minimize the error between the two contours. To achieve this goal, the application uses the Stochastic Gradient Descent (SGD) algorithm to iteratively update the detection frame coordinate parameters, gradually reducing the difference with the true contour, so that the detection frame better fits the true contour, improves the accuracy and robustness. The formula of the SGD algorithm is:

[0145] θ new =θ old -α·▽L(θ;x,y) (17)

[0146] Where α is the learning rate, L(θ;x,y) is the loss function, x is the initial edge contour, y is the inscribed maximum ellipse cone sleeve contour in the detection frame, and θ is the model parameter. The gradient of the loss function with respect to the model parameter is denoted as ▽L(θ;x,y), which represents the change direction of the loss function at the current parameter.

[0147] 3) During the iterative optimization process, the application sets a target distance threshold γ as the convergence condition for iteration. Once the distance between the detection frame and the cone sleeve contour is less than the set threshold, i.e. the convergence condition is met, the application obtains the current optimal parameterized detection frame. Finally, after necessary smoothing processing, the optimized and adjusted cone sleeve detection frame is output as the result.

[0148] The application belongs to the technical fields of computer vision, deep learning and industrial detection, and relates to target detection, machine learning, industrial quality control and multiple aspects. In particular, aiming at the key technical problems of docking positioning in the process of aerial in-flight refueling, based on deep learning and image processing methods, the application realizes accurate detection and analysis of the geometric shape of the refueling cone. Compared with existing detection means, the proposed scheme realizes automation, intelligentization and precision of detection based on the latest computer vision technology. And under the optimization design of software and hardware architecture for embedded platform, the detection system has strong environmental adaptability and engineering practicability.

Claims

1. A two-stage cascaded lightweight fueling nozzle sleeve detection method based on an attention mechanism, characterized in that, Comprise the following two steps; Step one: build the cone sleeve image dataset as the training set of the cone sleeve contour detection method, and preprocess the dataset; build a two-level cascaded lightweight cone sleeve detection network combining the first level coarse detection sub-network and the second level fine detection sub-network; use the cone sleeve image dataset to perform end-to-end offline supervised training on the network, and perform model quantization and conversion on the trained lightweight cone sleeve detection network; Step two: deploy the converted model on the embedded platform, including forward inference, post-processing of the cone sleeve contour detection network, and realize cone sleeve contour detection based on cascaded tracking and cone sleeve contour correction.

2. The two-stage cascaded lightweight attention mechanism-based refueling cone sleeve detection method according to claim 1, characterized in that, The step one is specifically: Step 1): Obtain samples and build a cone sleeve dataset: Step 2): Screen and classify the cone sleeve dataset, select valid samples that meet the model training requirements, and build datasets corresponding to the first level coarse detection sub-network and the second level fine detection sub-network respectively: The dataset corresponding to the first level coarse detection sub-network contains cone sleeve images with complete background; The dataset corresponding to the second level fine detection sub-network only contains small size images of local area of cone sleeve; The dataset corresponding to the second level fine detection sub-network is obtained by cropping the dataset corresponding to the first level coarse detection sub-network; Step 3): Label the datasets corresponding to the first level coarse detection sub-network and the second level fine detection sub-network. For the dataset corresponding to the first level coarse detection sub-network, use nearest neighbor interpolation or bilinear interpolation to scale the image while keeping the aspect ratio unchanged, and use the 4 vertex coordinates of the rotated rectangular frame for labeling format. Step 4): Perform data enhancement processing on the dataset obtained in step 3), using HSV color space enhancement, random cropping, rotation scaling, mosaic enhancement, Cutout, and CutMix methods respectively; Step 5): Use the data enhanced dataset to train the two-level cascaded lightweight cone sleeve detection network; Step 6): Based on deployment requirements, perform model compression and conversion on the embedded platform with RK3588.

3. The two-stage cascaded lightweight attention mechanism-based refueling cone sleeve detection method according to claim 2, characterized in that, The step one is specifically: The first way is to synthesize cone sleeve images in different scenes through 3D modeling software, specifically to build 3D models of different types, sizes and materials of cone sleeves, randomly set the position, angle and lighting parameters of the model, and add complex background to simulate different weather and actual scenes of the fuselage; The second way is to arrange real cone sleeves in the ground environment, adjust the spatial position parameters, and use image devices to collect static and micro-motion image samples from multiple angles; The third way is to collect relevant image or video samples of real air refueling process from the network under the premise of permission and privacy protection.

4. The two-stage cascaded lightweight attention mechanism-based refueling cone sleeve detection method according to claim 2, characterized in that, The step three of the step one is specifically: For the large size dataset of the first level coarse detection sub-network, the synthesized samples directly obtain the label by retaining the 3D modeling information; the ground shooting samples realize positioning and fitting calculation by adding marker lights on the cone sleeve to obtain the label data; the network collection samples manually complete the labeling using a labeling tool; For the small size data set of the second level fine detection sub-network, the region is intercepted based on the annotation of the large size data set of the first level coarse detection sub-network; Specifically, the second level fine detection sub-network data set is cropped from the horizontal rectangular frame region surrounded by the four vertices of the annotated rotating rectangular frame in the first level coarse detection sub-network data set, the cropped image is scaled by using the nearest neighbor interpolation or bilinear interpolation method without changing the aspect ratio, if the object is located at the edge of the input image, resulting in part of the region where the object is located being outside the input image, the missing part is filled with 0 to obtain a completely square image, and the second level fine detection sub-network data set image is obtained, and the annotation label is calculated based on the relative relationship between the cropped image region and the first level coarse detection sub-network data set image and the scaling ratio of the image, and does not need to be annotated separately.

5. The two-stage cascaded lightweight attention mechanism-based refueling cone sleeve detection method according to claim 2, characterized in that, In step 4) of step one, the HSV color space is composed of hue, saturation and lightness; the mosaic enhancement is to randomly arrange and splice four pictures; the Cutout enhancement is to randomly cut part of the region in the sample and fill it with 0 pixel value; and the CutMix enhancement is to cut part of the region but not to fill it with 0 pixel value but to randomly fill the pixel value of the region of other data in the training set.

6. The two-stage cascaded lightweight attention mechanism-based refueling cone sleeve detection method according to claim 2, characterized in that, In step 5) of step one, the two-level cascaded lightweight cone detection network is composed of a first level coarse detection sub-network and a second level fine detection sub-network based on a YOLO detection model improved with an integrated attention mechanism, the network structures of the first level coarse detection sub-network and the second level fine detection sub-network are the same, and the difference lies in that the image sizes of the training sets used by the two networks are different; An improved K-Means++ clustering algorithm is used to obtain the optimal preset anchor box based on the size of the labeled frame; Angle classification information is added to realize the conversion of the horizontal frame to the rotating frame; an IOU calculation method of the rotating frame is designed; a loss function is improved to include four parts of loss; and the final model directly predicts the category, position coordinates, scale and direction information of the cone in the image.

7. The two-stage cascaded lightweight attention mechanism-based fuel nozzle sleeve detection method according to claim 6, characterized in that, In step 6) of step one, the 32-bit floating point model is compressed into an 8-bit signed integer model by using 8-bit asymmetric dynamic fixed-point quantization; then through an RKNN model conversion tool, the model is first converted into an open neural network exchange format model, and then further converted into an RKNN format, and finally the one-level and two-level RKNN models of the first level coarse detection sub-network and the second level fine detection sub-network suitable for embedded platforms are obtained; the simplified 8-bit asymmetric quantization formula is: Q(x) is the quantized 8-bit signed integer value, x is the input 32-bit floating point number, clip() means to limit the value in the specified range, round() means to round the value to the nearest integer, max_range means the maximum value of the floating point tensor, is the scaling factor for mapping the floating point number to the integer range.

8. The two-stage cascaded lightweight attention mechanism-based refueling cone sleeve detection method according to claim 1, characterized in that, The step two is specifically: Step 1) initialize the video acquisition interface using the media processing software platform provided by RK3588; obtain the input image containing the cone from the camera or video stream in real time; adjust the size to adapt to the input requirements of the first level coarse detection sub-network and the second level fine detection sub-network respectively, normalize the pixel value to improve the stability of the model, and adjust the color space to BGR order to match the training model; Step 2) calling a general API interface for model loading, memory allocation, model inference, and obtaining the rotating frame of the cone sleeve through the post-processing operation, and finally completing memory release; Step 3) judging whether the approximate position of the cone sleeve is known, if not, inputting the preprocessed image into the loaded first-level coarse detection sub-network for rough positioning of the cone sleeve region, and cutting, scaling and filling the positioned image region, and sending the processed image region into the loaded second-level fine detection sub-network for accurate detection; if the target region is known, directly inputting the image into the loaded second-level fine detection sub-network for accurate cone sleeve detection using the position information; Step 4) inputting the detection result of the second-level fine detection sub-network into the built cone sleeve contour feature correction module, calculating the deviation between the detection frame and the actual cone sleeve contour based on the distance loss function, then updating the detection frame parameters using the gradient descent method, iteratively reducing the frame contour distance, and finally correcting the output detection frame result binding the actual contour; Step 5) the output of the cone sleeve contour correction module is the coordinates of the elliptical detection frame in the intercepted cone sleeve local image region, through the designed global coordinate conversion module, the local coordinates are mapped and converted to the original image coordinate system to obtain the global coordinate information of the detection frame in the original image space; Step 6) the current frame cone sleeve detection frame coordinates converted to the original image coordinate system are output in two ways: one is directly input as the detection result into the subsequent process to complete the downstream task; The second is fed back to the second detection network as the next frame search target region to realize the fusion of detection and tracking.

9. The two-stage cascaded lightweight attention mechanism-based fuel nozzle sleeve detection method according to claim 8, characterized in that, In step 2) of step two, the RKNN model is loaded through the model interface, and the loaded RKNN model is initialized through the initialization function, then the input image frame is initialized according to the frame data structure body, and the input of the model is set, then the model inference is carried out, the parallel computing unit of NPU is used to obtain the three-scale 8-bit feature data after inference, the three-scale 8-bit feature data is used to obtain the cone sleeve coordinate information, and the rotating non-maximum suppression is realized to remove redundant frames, finally the cone sleeve contour rotating frame is obtained, and the memory release is completed.

Images