Target detection method of crossing machine based on heteroocular modal crossing

Abstract

The invention belongs to the technical field of visual detection, and particularly relates to a cross-through machine target detection method based on heteroocular modal crossing, which comprises the following steps of: 1, preprocessing training data, splitting incident light to two sensors by using a spectroscope system, simultaneously acquiring data of an infrared light image and a visible light image, forming a data set, and acquiring the data of the infrared light image and the visible light image; registering the data sets of the visible light images and the infrared light images, labeling positions, categories and confidence labels, and distributing the data sets into a training set, a verification set and a test set; step 2, model training: using an end-to-end multi-modal detection network model to send the data set into the network model for model training and optimization; and 3, model conversion and deployment preparation: optimizing the trained model and converting the model into a traversing machine adaptive format. According to the method, the problems of target blurring, background complexity, shielding and the like in a cross-modal detection task can be solved, the model is helped to understand richer feature information, and high-efficiency target detection is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual detection, in particular to a target detection method for a crossing machine based on cross of different modalities. BACKGROUND

[0002] The crossing machine is a kind of unmanned aerial vehicle with high speed and high maneuverability. In recent years, the crossing machine has developed rapidly and is widely used. For example, security and rescue: searching for missing persons, monitoring fire scenes or earthquake disaster areas; military reconnaissance: quickly reconnaissance enemy targets to avoid personnel exposure; agriculture and inspection: checking crop diseases or power facility defects.

[0003] There are great challenges in using the crossing machine to perform detection tasks. Due to the characteristics of high flight speed and large flight height difference of the crossing machine, the obtained image data has the problems of small target, complex background, large target scale change and motion blur. Moreover, the crossing machine has to face complex scenes all day long. When in low light or dark environment, the target detection task becomes more difficult. Therefore, to overcome these difficulties and improve the target detection capability of the crossing machine is an important research field.

[0004] The Chinese authorized publication number is "CN120259822A", and the name is "Unmanned aerial vehicle visual angle infrared light and visible light fusion target detection method". This method uses two different modalities of images to complement each other and combines residual network, attention mechanism and fusion module to improve the detection performance of the unmanned aerial vehicle. This method only emphasizes a single modality image by adding attention to the residual module, and finally only performs one fusion. In addition, this method separates the fusion and detection tasks and outputs the fusion and detection results at the same time, which leads to a complex network model and increases the amount of calculation, and is not suitable for unmanned aerial vehicle deployment. Therefore, the traditional method is not suitable for the crossing machine detection task.

[0005] In summary, it is an urgent problem for researchers in this field to seek a target detection method suitable for the crossing machine to cope with complex and changing scenes and needs. SUMMARY

[0006] (I) Technical problems to be solved

[0007] In view of the deficiencies of the prior art, the present application provides a crossing machine target detection method based on cross of different modalities, which solves the problems proposed in the above background.

[0008] (II) Technical scheme

[0009] In order to achieve the above purpose, the present application specifically adopts the following technical scheme:

[0010] A crossing machine target detection method based on cross of different modalities, comprising the following steps:

[0011] Step 1: training data preprocessing, using a spectrometer system to divide the incident light into two sensors, while acquiring infrared and visible light image data, forming a data set, registering the visible light image and infrared light image data set, labeling the position, class, confidence label, and assigning it to the training set, validation set and test set;

[0012] Step 2: model training, using an end-to-end multi-modal detection network model, feeding the data set into the network model for model training and optimization;

[0013] Step 3: model conversion and deployment preparation, optimizing and converting the trained model into a format suitable for the crossing machine;

[0014] Step 4: real-time image acquisition and data synchronization, using the crossing machine to synchronously acquire environmental image streams using onboard visible light and infrared cameras;

[0015] Step 5: front-end reasoning and detection execution, feeding the synchronized and fused multi-modal image data into the optimized detection model for forward reasoning;

[0016] Step 6: online updating and model retraining, returning the collected new data to the ground station system after each task, using these field data to perform incremental learning or domain adaptation fine-tuning on the existing model, and continuously optimizing the model's detection accuracy and scene adaptability in real environments.

[0017] Further, the training set, validation set and test set in step 1 are allocated in a ratio of 6:2:2.

[0018] Further, the multi-modal detection network model in step 2 includes a modal cross-fusion part, a deep optimization feature part and a selective feature output part;

[0019] The modal cross-fusion part is composed of a cross-complementary module, which is used to obtain channel information and spatial information of infrared and visible light features, and uses parallel cross processing to cross-complement information defects between different modalities;

[0020] The deep optimization feature part is composed of a pruning module, which is used to retain effective information and remove background noise and redundant information through pruning operations, and cooperates with dense and residual connections to obtain multi-level information;

[0021] The selective feature output part is composed of a scale softening module, a feature selection module and a cross connection, and is used for selecting target information in multi-scale features and enhancing the detail information of the target; the scale softening module is used for performing Gaussian blur processing on feature maps of different scales through Gaussian convolution and pooling operation, so as to relieve the hard fusion of features caused by too large scale transformation; the feature selection module is used for receiving cross feature fusion under different scales, and selecting the most accurate target feature through multi-dimensional attention with adaptive adjustment of feature size, channel and spatial position three-dimensional dynamic change; and the cross connection is used for obtaining rich information through different receptive fields, and improving the generalization ability of the model.

[0022] Further, in step 3, the model is converted into an intermediate representation format, and the model structure is graph optimized, layer fused and precision quantized by using a TensorRT inference framework, so as to compile the optimized model to generate an executable deployment file.

[0023] Further, in step 4, the synchronous acquisition of the environment image is performed by aligning the image based on a hardware trigger signal or a software timestamp, and combining the pre-calibrated internal and external parameters and distortion coefficients to complete image registration and synchronous processing.

[0024] Further, in step 6, the online updating is performed through remote updating and model hot replacement functions deployed on the airborne end; the remote updating is performed by transmitting a new model file from a server to the crossing machine through a wireless network; and the model hot replacement is performed by seamlessly replacing an old model running on the device with a new model file without stopping the running of the main program.

[0025] (Three) beneficial effects

[0026] Compared with the prior art, the present application provides a crossing machine target detection method based on cross-modal fusion, which has the following beneficial effects:

[0027] The present application utilizes the cross-complementary module to simultaneously obtain the channel and spatial information of infrared and visible light images, complement each other's advantages, and dynamically adjust the extraction of infrared and visible light features in the modal cross-fusion part of the network model; the pruning module is used to judge the importance of the features by calculating the L1 norm of the features, and the important features are retained and the redundant information is removed; the scale softening module and the feature selection module are used, the scale softening is performed through Gaussian convolution to soften different scale features so that they are more suitable in multi-scale fusion; the feature selection module generates three-dimensional attention according to the feature size, channel and spatial position by using the dimension attribute of the feature map, enhances the target information, and improves the detection accuracy; therefore, the method of the present application can reduce the model complexity and obtain more feature information, and realize higher precision and faster speed detection. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1This is a flowchart of the method of the present invention;

[0029] Figure 2 This is a diagram of the network model structure of the present invention;

[0030] Figure 3 This is a structural diagram of the cross-complementary module of the present invention;

[0031] Figure 4 This is a structural diagram of the pruning module of the present invention;

[0032] Figure 5 This is a structural diagram of the scale softening module of the present invention;

[0033] Figure 6 This is a structural diagram of the feature selection module of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example

[0036] like Figures 1-6 As shown in the figure, an embodiment of the present invention proposes a target detection method for a racing drone based on heteromorphic modal crosstalk, which includes the following steps:

[0037] Step 1: Training data preprocessing. The incident light is split into two sensors using a beam splitter system, simultaneously acquiring infrared and visible light image data. The visible light image resolution is 1280×960, and the infrared image resolution is 640×480. It includes infrared and visible light images under different weather conditions, lighting conditions, and complex backgrounds, totaling 8354 images. The visible light and infrared image data are registered, labeled with location, category, confidence level, etc., and randomly allocated into training set, validation set, and test set in a 6:2:2 ratio.

[0038] Step 2: Model training. An end-to-end multimodal detection network model is used on the ground workstation. The dataset is fed into this network model for training and optimization. The network model structure is as follows: Figure 2The infrared and visible light images are respectively input into convolution block one and convolution block three, and their outputs are input into cross complementary module one before being input into convolution block two and convolution block four. The feature map generated by the cross complementary module one is added to the original output and then input into convolution block two and convolution block four. The outputs of convolution block two and convolution block four repeat the above operation. Finally, the infrared and visible light feature maps are spliced together and input into the depth optimization feature part. The input feature map is input into three consecutive pruning modules and then input into the selective feature output part. The feature map is input into the scale softening module, and the output is input into feature selection module one, feature selection module two and feature selection module three. The inputs of the scale softening module and the feature selection module are multiple, respectively from P2, P3, P4 and P5, wherein P2 is the feature map after 4 times downsampling, P3 is the feature map after 8 times downsampling, P4 is the feature map after 16 times downsampling, and P5 is the feature map after 32 times downsampling.

[0039] The cross complementary module is as follows: Figure 3 The specific steps are as follows: first, the infrared feature map and the visible light feature map are input into eight 1x1 ordinary convolution outputs in parallel to obtain two groups of and , one group of and is subjected to global average pooling operation to obtain two vectors, and the corresponding point multiplication is performed, and then the corresponding weight value is obtained through the Softmax function, and then the output and of the other infrared feature map and the visible light feature map are weighted and multiplied, and the outputs and are obtained. The and are input into the 1x1 convolution layer and the normalization layer respectively, and the outputs and are obtained. The other group of and is directly subjected to corresponding point multiplication, and then the corresponding weight value is obtained through the Softmax function, and then the output and of the other infrared feature map and the visible light feature map are weighted and multiplied, and the outputs and are obtained. The and are input into the 1x1 convolution layer and the normalization layer respectively, and the outputs and Finally, the infrared features and Adding them together can reveal the characteristics of light. and Add them together and then concatenate them as the output.

[0040] Pruning modules, such as Figure 4 The specific steps are as follows: It consists of four pruned convolutions, accompanied by residual connections and dense connections, extracting deep features while also paying attention to shallow features; the core idea of ​​pruned convolution is to determine the importance of the information brought by the feature map, which is achieved by utilizing... Use norm to calculate importance score The specific formula is as follows:

[0041]

[0042] In the formula, This represents the connection strength between the previous stage output channel i and the next stage input channel j. Representing the The set of output channel indices for the group. This represents the number of channels in the input feature map.

[0043] By retaining the k most important feature maps, the remaining groups are pruned. The weight matrix after pruning is shown below.

[0044]

[0045] In the formula, This represents the coefficients used to select the top k importance coefficients. This represents the number of groups of the feature map.

[0046] Scale softening module, such as Figure 5 The specific steps are as follows: Feature maps from different sizes... , and The results are obtained by inputting the data into a 3×3 Gaussian convolutional layer. and ,Will and Adjust the size to match The same size, then fed into the convolution block and The results after three max pooling operations with a pooling rate of 5 are then concatenated into channels. This module achieves more robust multi-scale feature representation and improved compatibility with both small and large targets through Gaussian softening, multi-scale alignment, and sparse pooling. Figure 6As shown in the structural diagram of the feature selection module of the present application, firstly, the module receives the output feature map of the previous stage, in addition to which it also receives feature maps of different sizes, realizes multi-scale cross fusion, and uses global average pooling operation, full connection operation and Sigmoid to generate attention parameters of three dimensions of feature size, channel and spatial position, to assist the target detection task; through multi-scale cross fusion and three-dimensional attention mechanism, the module realizes intelligent feature selection: automatic focusing on important scales, channels and spatial positions; detection performance improvement: especially significant effect on complex background and small target scene; low computational overhead: conducive to deployment on lightweight devices such as crossing machines.

[0047] Among them, the implementation of convolution, pooling, activation function, channel splicing and feature map addition operation is a method known to those skilled in the art, and the principle thereof will not be described here.

[0048] Step 3: Model conversion and deployment preparation, convert the trained model into an efficient intermediate representation format (such as ONNX), and use inference frameworks such as TensorRT to perform graph optimization, layer fusion and precision quantization on the model structure, to significantly improve inference performance and reduce memory occupancy. Subsequently, according to the actual hardware architecture of the computing platform (such as NVIDIA Jetson series or ARM+GPU embedded platform) carried by the crossing machine, the optimized model is compiled to generate a deployment file that can be efficiently executed, ensuring that it can run stably in a resource-constrained embedded environment.

[0049] Step 4: Real-time image acquisition and data synchronization, the crossing machine synchronously acquires environmental image streams through on-board visible light and infrared cameras. To ensure the consistency and fusibility of multi-modal data in space and time, the system will align the images based on hardware trigger signals or software timestamps, and complete image registration and synchronization processing in combination with pre-calibrated camera parameters (such as internal and external parameters and distortion coefficients), to provide high-quality, aligned multi-modal input for subsequent fusion detection.

[0050] Step 5: Front-end inference and detection execution, the multi-modal image data synchronized and fused is sent into the optimized detection model to perform forward inference. The model outputs the position coordinates, class labels and confidence of potential small targets in the image, and other key detection information. To meet the real-time requirements of the flight control system of the crossing machine, the entire detection process needs to maintain a frame rate processing capability of not less than 10FPS to ensure that the detection results can effectively support the real-time decision and maneuvering control of the platform.

[0051] Step 6: Online update and model retraining, the inference system deployed on the airborne end should support remote update and model hot replacement function, so that the new data collected after each task can be returned to the ground station system. Using these field data, incremental learning or domain adaptation fine-tuning can be performed on the existing model, continuously optimizing the detection accuracy and scene adaptability of the model in the real environment, forming a closed-loop optimization iterative learning process.

[0052] In order to verify the authenticity and effectiveness of the method of the present application, it is compared and analyzed with other target detection methods of the present mainstream, as shown in Table 1.

[0053] Table 1 is a comparison of different algorithms on the data set

[0054] map@0.5 P R Parameters ) Model size (MB) GFLOP / s Faster-RCNN YoloV5s YoloXs 0.3080.3310.346 0.3350.4510.346 0.3640.3350.396 / 7.049.0 / 13.7468.7 / 15.826.8 YoloV7-tiny YoloV8s VAM YoloXmours 0.3580.3920.3820.418 0.4880.5110.5160.514 0.370.3780.3920.404 6.19.8327.05.98 12.319.04207.112.93 13.8428.5151.4122.37

[0055] As can be seen from Table 1, the Faster-RCNN two-stage traditional algorithm is not suitable for cross-modal target detection tasks, and the detection effect is not good; the performance of Yolov5s and YoloXs is close, but the detection accuracy is not high, and their network structure has limitations when facing small targets, and does not fully consider the characteristics of small targets; although the Yolov7-tiny algorithm is lightweight, the detection accuracy is low, and it is not suitable for crossing machines; except for the method of the present application, the detection accuracy of Yolov8s is the best among these algorithms, with a map@0.5 of 39.2%; in summary, the method of the present application is the best, with sufficient advantages in precision, accuracy and recall rate, and the weight and complexity of the model are almost close to the lightweight model, fully meeting the deployment requirements on the crossing machine, and being able to complete the target detection task with high quality.

[0056] Finally, it should be noted that: the above only describes the preferred embodiments of the present application and is not intended to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements for some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A target detection method for racing drones based on heteromorphic modal crosstalk, characterized in that: The method comprises the following steps: Step 1: training data preprocessing, using a spectrometer system to divide the incident light into two sensors, while acquiring the data of the infrared light image and the visible light image, forming a data set, registering the data set of the visible light image and the infrared light image, labeling the position, class, confidence label, and distributing into a training set, a validation set and a test set; Step 2: model training, using an end-to-end multi-modal detection network model, feeding the data set into the network model for model training and optimization; Step 3: model conversion and deployment preparation, optimizing and converting the trained model into a format suitable for the crossing machine; Step 4: real-time image acquisition and data synchronization, using the crossing machine to synchronously acquire the environmental image stream through the on-board visible light and infrared cameras; Step 5: front-end reasoning and detection execution, feeding the synchronized and fused multi-modal image data into the optimized detection model for forward reasoning; Step 6: online updating and model retraining, after each task, the collected new data is returned to the ground station system, and the existing model is incrementally learned or regionally adapted and fine-tuned using these field data, so as to continuously optimize the detection accuracy and scene adaptability of the model in the real environment.

2. The method according to claim 1, wherein the method is characterized by: The training set, the validation set and the test set in step 1 are distributed according to a ratio of 6:2:

2.

3. The method of claim 1, wherein the method is based on the cross-modal interaction of the heteronym. The multi-modal detection network model in step 2 comprises a modal cross-fusion part, a deep optimized feature part and a selective feature output part; The modal cross-fusion part is composed of a cross-complementary module, which is used to acquire channel information and spatial information of infrared and visible light features, and uses parallel cross processing, which can cross-complement information defects between different modalities; The deep optimized feature part is composed of a pruning module, which is used to retain effective information and remove background noise and redundant information through pruning operation, and cooperates with dense and residual connection to acquire multi-level information; The selective feature output part is composed of a scale softening module, a feature selection module and a cross connection, which is used to select target information in multi-scale features and enhance the detail information of the target; the scale softening module is used to perform Gaussian blur processing on feature maps of different scales through Gaussian convolution and pooling operation, to relieve the hard fusion caused by too large scale transformation; The feature selection module is used to dynamically change the multi-dimensional attention of feature size, channel and spatial position through adaptive adjustment, and simultaneously receives cross feature fusion under different scales to select the most accurate target feature; The cross connection is used to acquire rich information through different receptive fields to improve the generalization ability of the model.

4. The method of claim 1, wherein: In step 3, the model is converted into an intermediate representation format, and the model structure is optimized, layer fused and precision quantized by using the TensorRT inference framework, and the optimized model is compiled to generate an executable deployment file.

5. The method of claim 1, wherein: In step 4, the synchronous acquisition of environmental images is performed by aligning the images based on hardware trigger signals or software timestamps, and combining the pre-calibrated internal and external parameters and distortion coefficients to complete image registration and synchronization processing.

6. The method of claim 1, wherein: The step 6 online update is through the remote update and model hot replacement function deployed on the airborne end; the remote update transmits the new model file from the server to the crossing machine through the wireless network; The model hot replacement replaces the old model running on the device with the new model file seamlessly without stopping the main program running.

Images