A dual-channel image fusion target detection method suitable for an optoelectronic pod

Abstract

This invention discloses a dual-channel image fusion target detection method suitable for optoelectronic pods. First, an improved YOLOX-RGBT network is used with feature stacking to fuse the infrared and visible light images acquired by the optoelectronic pod at the feature layer, obtaining a dual-channel target detection result. Then, a two-stage hybrid uniform quantization method is used to reduce the number of parameters in the improved YOLOX-RGBT network, improving the target detection speed of the optoelectronic pod. Finally, the improved and quantized YOLOX-RGBT network is deployed on the Ascend embedded device mounted on the optoelectronic pod, and its detection speed is further improved by calling chip hardware-level acceleration instructions to achieve real-time target detection. This invention utilizes the excellent image processing capabilities and powerful computing power of the Ascend embedded device, combined with the improved dual-channel image fusion target detection algorithm, to achieve a significant advantage in detection accuracy compared to single-channel target detection models while meeting real-time and effectiveness requirements.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent detection of optoelectronic pods, and particularly relates to a dual-channel image fusion target detection method suitable for optoelectronic pods. BACKGROUND

[0002] An optoelectronic pod is an important component of an optoelectronic reconnaissance technology equipment, can carry an infrared visible light image sensor to complete a target detection and tracking task, and is widely applied to manned equipment and unmanned equipment on land, sea and air. Limited by the computing power of an embedded platform, a deployed target detection and tracking algorithm needs to design a lightweight detection and tracking algorithm and a deployment strategy to meet the real-time requirement of the embedded platform.

[0003] At present, many creative algorithms are emerging in the academic field. The Mosaic data enhancement algorithm proposed in Yolov4: Optimal speed and accuracy of object detection effectively solves the problems of uneven data distribution and insufficient data amount. The paper Fast r-cnn proposes a new ROIPooling operator in the network structure, and has a more targeted method in extracting different features. The paper Distilling the knowledge in a neural network proposes a knowledge distillation method, which reduces the network parameter amount and further improves the network running speed. The related content of deep learning is developing vigorously, but the algorithms actually effectively landed in the industrial field are not many. Limited by the actual working scene, it cannot be effectively transformed to the actual application level.

[0004] The deployment of neural networks on embedded devices still has many challenges. The most difficult to solve is the insufficient computing power of edge devices, and many advanced algorithms are difficult to deploy. Secondly, the detection effect in target detection is often affected by environmental factors, and the accuracy is low in the environment with poor light conditions. Therefore, the algorithm development and optimization at the hardware level and the target detection algorithm still have great development space. SUMMARY

[0005] The application proposes a dual-channel image fusion target detection method suitable for optoelectronic pods, which solves the problems of low multi-scale pedestrian detection accuracy in low brightness environment and slow inference speed of embedded platforms.

[0006] The technical solution of the application is a dual-channel image fusion target detection method suitable for optoelectronic pods, characterized in that it comprises the following steps:

[0007] Step 1, the infrared image and the visible light image are collected by the optoelectronic pod respectively, the infrared image and the visible light image are used to establish a FIRE data set, the FIRE data set is put into the improved YOLOX-RGBT network for training, and a trained.pth model file is obtained, and step 2 is entered;

[0008] Step 2, the trained.pth model is processed by using a two-stage mixed uniform quantization method, and a quantized.onnx model file is obtained, and step 3 is entered;

[0009] Step 3, the quantized.onnx model file is deployed to the Atlas 200DK embedded inference acceleration platform, and a real-time target detection task is realized.

[0010] Compared with the prior art, the present application has the following advantages:

[0011] (1) The infrared image is not limited by the light condition, and has good imaging effect in the night or strong light reflection scene, the visible light image has the characteristics of rich feature information and high resolution, the method combines the advantages of the two, and a double-channel image feature fusion method is proposed, which effectively solves the problem of low accuracy of multi-scale pedestrian detection in low brightness environment.

[0012] (2) The target detection algorithm is quantized by two-stage mixed precision, and can be deployed on the optoelectronic pod embedded platform, the target detection speed is improved while the target detection accuracy is ensured, and the real-time target detection task of the optoelectronic pod is realized. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 It is a double-channel image fusion target detection network diagram of the present application.

[0014] Figure 2 It is a whole flow chart of the present application. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0016] The specific embodiments, technical difficulties and points of the present application will be further introduced below in combination with the design examples.

[0017] In combination with Figure 1 and Figure 2The application discloses a dual-channel image fusion target detection method suitable for an optoelectronic pod.

[0018] Step 1, infrared images and visible light images are collected by the optoelectronic pod respectively, the collected infrared images and visible light images are used to establish a FIRE data set, the FIRE data set is placed into an improved YOLOX-RGBT network for training, and a trained.pth model file is obtained.

[0019] Step 1.1, infrared images and visible light images are collected by the optoelectronic pod at different time periods (including day and night) and different places (school, commercial street, etc.), and the collected infrared images and visible light images are used to establish a FIRE data set.

[0020] Step 1.2, the FIRE data set is placed into the improved YOLOX-RGBT network for training, and a.pth model file is obtained.

[0021] The improved YOLOX-RGBT network is based on a light-weight YOLOX-s network structure, uses two YOLOX-s feature extraction backbone networks to perform deep feature extraction on visible light images and infrared images respectively, and uses a feature stacking method to perform feature fusion at the position of a visible light and infrared feature extraction backbone network channel neck, then transmits the feature information to the output feature map through an FPN structure and a decoupling head, and the YOLOX-RGBT convolutional neural network comprises a convolution layer, a pooling layer and an activation function, and is specifically as follows:

[0022] The visible light images and the infrared images are simultaneously input into the YOLOX-RGBT network, the visible light images and the infrared images are processed through a convolution layer, a pooling layer and a SiLU activation function, and feature fusion is performed at the position of each feature extraction backbone network neck, and the specific fusion position is shown in a Fusion module in Figure 1 .

[0023] The convolution operation formula is as follows:

[0024]

[0025] Wherein, O(i,j) is the output result after convolution operation, f(·) is an activation function, the convolution operation result is nonlinearized by using the nonlinearized activation function, the fitting ability of the model is improved, Y represents the output result after nonlinearization operation on O(i,j). I represents an input matrix, I(i+m,j+n) represents an element in the input feature map, K represents a convolution kernel, generally a 1x1, 3x3, 5x5 convolution kernel, K(m,n) represents an element in the mth row and the nth column of the convolution kernel, then the calculated convolution result is added and the final convolution activated feature map is obtained after passing through the activation function. out represents the size of the output feature map, wherein p represents the size of the padding 0, k represents the size of the convolution kernel, s represents the stride, W represents the size of the input feature map, and (i,j) represents the element at the ith row and jth column position of the feature map.

[0026] The SiLU activation function formula is as follows: when the input feature parameter value x is greater than 0, the SiLU activation function will amplify x, and when x is less than 0, the SiLU activation function will reduce x, and overfitting can be suppressed.

[0027]

[0028] f(x)=xσ(x)

[0029] Wherein, σ(x) represents a sigmoid activation function, and f(x) represents a SiLU activation function.

[0030] The dual-channel image feature fusion is achieved by using feature stacking and feature weighting, specifically as follows:

[0031] The feature stacking is to stack the two groups of features extracted from the infrared image and the visible light image by using the Concat method first, then use the convolution layer to perform feature fusion and dimension adjustment, and then stack the features after dimension adjustment through the BN layer and the SiLU layer with the two channels without fusion to complete the overall feature fusion.

[0032] The feature weighting is to fuse the two groups of features by using the Concat method, and different weight parameters are set according to the detection effect of different channels to realize feature fusion.

[0033] In the feature prediction part, multi-scale features are used for target detection, three feature layers are used for classification and position regression prediction, and feature fusion is performed through a feature pyramid, and three detection heads Yolo Head are output, wherein the dimensions of the three detection heads are 13x13x[3x(4+1+cls_num)], 26x26x[3x(4+1+cls_num)] and 52x52x[3x(4+1+cls_num)] respectively.

[0034] Step 2. Process the trained.pth model using a two-stage mixed uniform quantization method to obtain the quantized.onnx model file, as follows:

[0035] Step 2.1. In the first stage, use the uniform quantization method based on truncation to automatically complete the quantization of different layer weight parameters and data of the detection model by roughly setting the quantization parameters, and generate detailed quantization configuration and quantization factor for each layer.

[0036] Convert the.pth model originally trained on the server into an.onnx model using torch.onnx.export.

[0037] Use the model compression software package to generate an evaluator instance and call accuracy_based_auto_calibration to test the precision of the converted.onnx model, and obtain the original model precision. This process simultaneously calls the quantization strategy strategy module in accuracy_based_auto_calibration, and outputs the initialized quantconfig quantization configuration file, which records that all layers can be quantized.

[0038] Use the initial quantization configuration file generated by create_quant_config to perform post-training quantization on the model to obtain the precision of the quantized model.

[0039] The quantization formula for the above process is as follows:

[0040]

[0041] In the formula, V INT8 is the quantized INT8 integer, V Float32 is the Float32 floating point number before quantization. S m is the scaling factor of the floating point number, including the weight scaling factor S w and the data scaling factor S d , O m is the offset of the quantization factor, including the weight quantization offset factor O m and the data quantization offset factor O d , where S d and O d are the two main parameters that affect the quantization effect in the quantization process; round(*) represents rounding, and clip(*) represents the truncation quantization process, as follows:

[0042] Since the eigenvalues of the convolutional neural network are generally not uniformly distributed, the edge data distribution is generally sparse. Further quantization of some low-frequency values can more accurately map the data and reduce the precision loss after quantization.

[0043] An input feature reconstruction algorithm (IFMR) is used to determine the optimal quantization method by searching under a certain data distribution, and to calculate the data quantization scaling factor S d and the data quantization offset factor O d . Since the weight parameters of different channels are distributed differently, calculating the overall channel parameters will result in large errors in individual channels. Therefore, the weight quantization is calculated separately for each channel to improve the accuracy of the final quantization. The quantization calculation process can be represented as:

[0044]

[0045] In the above formula, and represent the maximum Float32 value and the maximum INT8 value in the activation value, respectively. and represent the minimum Float32 value and the minimum INT8 value in the activation value, respectively. Float32 V INT8 represent the activation value before quantization, and V represent the activation value after quantization.

[0046] To obtain the optimal quantization method, the quantization threshold is searched within the feasible interval, and the threshold corresponding to the optimal quantization result is used as the final truncation threshold. Based on the separate calculation of each channel, the upper boundary C max and the lower boundary C min are set at the boundary of the data distribution, and the step size S step is searched within the range of and to obtain different quantization results.

[0047] Step 2.2, based on the implementation of the first stage model weight parameter quantization and activation data quantization, if the precision of the quantized model does not meet the performance requirements, the mixed uniform quantization method uses the quantization configuration file generated in the first stage to adjust the key configuration parameters and the key network layer strategy to improve the performance of the model.

[0048] The FIRE dataset is divided into test dataset D test and calibration dataset D calib according to the ratio of 7:3.

[0049] The first-stage quantized .onnx model was tested on dataset D. test Tests were conducted on the [specific platform], and the quantified performance standard P was set based on the test results. th .

[0050] Input the global quantization configuration file Quant.cfg, and call the create_quant_config() function in the model compression package AMCT (Ascend ModelCompressionToolkit) to generate the quantization configuration file config.json.

[0051] M init The config.json file is then passed to the quantize_model() interface to perform model quantization.

[0052] The optimized model, incorporating quantization algorithms, was used on the calibration dataset D. calib Inference is performed on the above, quantization parameters are calculated and saved to the detailed quantization parameter file Q. par middle.

[0053] Using the precision simulation model M init In D test The performance of the quantized model P was tested. quant .

[0054] If the quantized model performance P obtained from the test is quant P does not satisfy th If so, increase the batch_norm value used for quantization in config.json; a recommended value is 16 or 32.

[0055] If adjusting batch_norm still does not meet the requirements, then adjust quant_enable in config.json, and prioritize dequantizing the first layer, the last layer, and layers with fewer parameters.

[0056] If the model results still do not meet the requirements, manually modify the quantization parameters for network weights and activation data in activation_quant_params and weight_quant_params.

[0057] If the accuracy meets the requirements after quantization according to the above process, then the parameter tuning is complete and the performance standard P is obtained. th If the .onnx model is not used, it indicates that quantization has a significant impact on accuracy and quantization cannot be performed. In this case, the quantization configuration is removed, and the .onnx model that has only undergone one stage of quantization is still used.

[0058] Step 3. Deploy the quantized.onnx model file to the Atlas 200DK embedded inference acceleration platform to realize real-time target detection tasks.

[0059] Step 3.1. Determine the width and height of the images collected by the electro-optical pod in real time, select the YUV420SP format after JPEG image decoding, create and set the channel attribute value, apply for a memory block for image decoding, and use the JPEGD processing in DVPP to decode each frame of JPEG image to obtain a YUV420SP format image.

[0060] Further use the VPC module to create a channel and apply for image input and output memory, complete scaling, matting, superimposition, and color gamut conversion of YUV or RGB format images, and obtain a new resolution image. The output image width is 16-aligned and the height is 2-aligned, realizing optimization in memory and speed. The alignment formula is as follows:

[0061]

[0062] where W align and H align correspond to the width and height of the aligned image, W src and H src correspond to the original width and height of the image.

[0063] Finally, use AIPP to complete data preprocessing on AI Core. After image cropping, color gamut conversion, and image normalization operations, the YUV420SP image with aligned width and height is converted into an RGB image with a resolution that meets the model inference requirements, providing effective input data support for subsequent model inference.

[0064] Step 3.2. Convert the.onnx model trained by the server into the.om format supported by the embedded platform using the Ascend tensor compiler.

[0065] After parsing the open-source framework network model using ONNX Parser, convert it into the intermediate state IR Graph.

[0066] The intermediate state IR goes through four steps of graph preparation, graph splitting, graph optimization, and graph compilation to convert into the.om model format adapted to the Ascend AI processor.

[0067] Upload the converted.om model to the board-side environment, and subsequently load the model file through the Ascend CL interface to realize the inference process.

[0068] Step 3.3. Complete resource scheduling for algorithm runtime on the embedded platform to improve running speed.

[0069] Call acl.init interface to initialize pyACL configuration.

[0070] Call acl.rt.set_device interface to specify Device for operation.

[0071] Call acl.rt.get_run_mode interface to get the running mode of Ascend AI software stack, and the internal processing flow is different according to the running mode.

[0072] Call acl.rt.create_context interface to create Context.

[0073] Call acl.rt.create_stream interface to create Stream.

[0074] Call acl.rt.malloc interface to apply for memory on Device.

[0075] Call acl.mdl.load_from_file interface to load.om model file.

[0076] Call acl.mdl.execute interface to execute model inference, synchronous interface.

[0077] Call acl.mdl.get_dataset_num_buffers interface to get the number num of aclDataBuffer from output aclmdlDataset.

[0078] Loop through the output aclmdlDataset to get aclDataBuffer data, convert the data into a numpy object, use the NMS post-processing algorithm to output, and get the target coordinate and category information.

[0079] After all tasks are completed, call acl.mdl.unload interface to unload the model, call destroy interface to release Stream, Context, Device resources, and end the target detection task.

[0080] Further, the photoelectric pod is a vehicle-mounted photoelectric pod, an airborne photoelectric pod, or a ship-mounted photoelectric pod.

[0081] The effect of the present application is to solve the problem of low accuracy of multi-scale pedestrian detection in low brightness environment. A dual-channel target detection network based on feature layer fusion is designed based on the lightweight YOLOX-s network, which effectively utilizes the feature information of different image sensors, reduces the algorithm parameter quantity by adopting a two-stage model quantization method, and realizes real-time inference on an embedded platform. It can basically meet the needs of real-time and accuracy in real-time target detection tasks of the photoelectric pod, and has important social practical significance and engineering application value in promoting the application of the photoelectric pod in the field of target detection.

Claims

1. A dual-channel image fusion target detection method suitable for an opto-electric pod, characterized in that, Comprise the following steps: Step 1, the infrared image and visible light image are collected by the electro-optical pod respectively, the infrared image and visible light image are used to establish a FIRE data set, and the FIRE data set is put into the improved YOLOX-RGBT network for training to obtain a trained.pth model file, The improved YOLOX-RGBT network is based on a lightweight YOLOX-s network structure, uses two YOLOX-s feature extraction backbone networks to extract deep features of visible light images and infrared images respectively, and uses feature stacking at the neck position of the visible light and infrared feature extraction backbone network channel to fuse the features, and then transmits the feature information to the output feature map through the FPN structure and the decoupling head, the YOLOX-RGBT convolutional neural network comprises a convolution layer, a pooling layer and an activation function; The dual-channel image feature fusion adopts feature stacking and feature weighting methods, which are as follows: Feature stacking is to stack the two groups of features extracted from the infrared image and the visible light image by using the Concat method first, then use the convolution layer for feature fusion and dimension adjustment, and then stack the features after the BN layer and the SiLU layer with the two channels without fusion, to complete the overall feature fusion; Feature weighting is to use the Concat method to fuse the two groups of features, and set different weight parameters according to the detection effect of different channels to realize feature fusion; In the feature prediction part, multi-scale features are used for target detection, three feature layers are used for classification and position regression prediction, feature fusion is performed through a feature pyramid, and three detection heads Yolo Head are output, wherein the dimensions of the three detection heads are 13x13x[3x(4+1+cls_num)], 26x26x[3x(4+1+cls_num)] and 52x52x[3x(4+1+cls_num)] respectively; Go to step 2; Step 2, use a two-stage hybrid uniform quantization method to process the trained.pth model to obtain a quantized.onnx model file, and go to step 3; Step 3, deploy the quantized.onnx model file to the Atlas 200DK embedded inference acceleration platform to realize real-time target detection tasks.

2. The dual-channel image fusion object detection method suitable for opto-electrical pod according to claim 1, characterized in that, In step 1, the following steps are taken: Step 1.1, use the infrared and visible light sensors carried by the electro-optical pod to collect infrared images and visible light images at different time periods and different locations, and establish a FIRE data set from the collected infrared images and visible light images; Step 1.2, put the FIRE data set into the improved YOLOX-RGBT network for training to obtain a.pth model file.

3. The dual-channel image fusion target detection method suitable for opto-electrical pod according to claim 2, characterized in that, Specifically: The visible light image and the infrared image are input into the YOLOX-RGBT network at the same time, and the visible light image and the infrared image are processed by the convolution layer, the pooling layer and the SiLU activation function, and then the features are fused at the neck position of each feature extraction backbone network; Wherein, the convolution operation formula is as follows: Wherein, O(i,j) is the output result after convolution operation, f(·) is an activation function, Y represents the output result after nonlinear operation on O(i,j); I represents an input matrix, I(i+m,j+n) represents an element in the input feature map, K represents a convolution kernel, K(m,n) represents an element in the mth row and the nth column of the convolution kernel, W out represents the size of the output feature map, wherein, p represents the size of padding 0, k represents the size of the convolution kernel, s represents the stride, W represents the size of the input feature map, (i,j) represents the element at the i th row and the j th column position of the feature map; The SiLU activation function formula is as follows: when the input feature parameter value x is greater than 0, the SiLU activation function will amplify x, and when x is less than 0, the SiLU activation function will reduce x to suppress overfitting; f(x) = x sigma(x) Wherein, sigma(x) represents a sigmoid activation function, and f(x) represents a SiLU activation function.

4. The dual channel image fusion object detection method suitable for opto-electrical pod according to claim 1, characterized in that, In step 2, the two-stage hybrid uniform quantization method is used for the.pth model obtained by training to obtain the quantized.onnx model file, as follows: In step 2.1, a uniform quantization method based on truncation is used in the first stage to automatically complete the quantization of the weight parameters and data of different layers of the detection model by roughly setting the quantization parameters, and to generate detailed quantization configurations and quantization factors for each layer: The model compression software package is used to generate an evaluator instance and call accuracy_based_auto_calibration to test the precision of the converted.onnx model, and the original model precision is obtained; the process simultaneously calls the quantization strategy strategy module in accuracy_based_auto_calibration, and outputs the initialized quant config quantization configuration file, which records that all layers can be quantized; The initial quantization configuration file generated by create_quant_config is used to quantize the model after training, and the precision of the quantized model is obtained; In step 2.2, based on the implementation of the first stage model weight parameter quantization and activation data quantization, if the precision of the quantized model does not meet the set performance requirements, the hybrid uniform quantization method uses the quantization configuration file generated in the first stage to adjust the key configuration parameters and network layer strategies to improve the performance of the model in the second stage: The FIRE dataset is split into a test dataset D according to a ratio of 7:3 test and a calibration dataset D calib ; The one-stage quantized.onnx model is tested on the test data set D test and the performance standard P of the quantized model is set according to the test result th ; Input the global quantization configuration Quant.cfg, call the create_quant_config() function in the model compression software package AMCT, and generate the quantization configuration file config.json; M init and config.json into the quantize_model() interface, execute the model quantization operation; Using the model comprising the quantization algorithm after optimization on the calibration dataset D calib Inference is performed on the calibration dataset D par in the detailed quantization parameter file Q Using the precision simulation model M init In D test Test the quantized model performance P quant ; If the test gets the quantized model performance P quant Does not meet P th , increase the data batch_norm used for quantization in config.json, recommended value 16 or 32; If the adjustment of batch_norm still does not meet the requirements, adjust quant_enable in config.json, and preferentially release the quantization of the first layer, the last layer and the layer with less parameter quantity; If the model result still does not meet the requirements, manually modify the quantization parameters of the network weight and activation data in activation_quant_params and weight_quant_params; If the accuracy meets the requirements after quantization according to the above process, the parameter adjustment ends and the performance standard P is met th The.onnx model, otherwise it indicates that quantization has a great influence on accuracy, quantization cannot be performed, and the one-stage quantization only.onnx model is still used.

5. The dual channel image fusion object detection method suitable for opto-electrical pod according to claim 1, wherein, In step 3, the quantized.onnx model file is deployed to the Atlas 200DK embedded inference acceleration platform to realize real-time target detection tasks, as follows: In step 3.1, the image width and height collected by the photoelectric pod in real time are determined, the format of the decoded JPEG image is selected as YUV420SP, the channel attribute value is created and set, and the memory block for image decoding is applied, and the DVPP JPEGD is used to decode each frame of JPEG image to obtain a YUV420SP format image; Step 3.

2. Convert the trained.onnx model on the server side to the.om format supported by the embedded platform using the Ascend tensor compiler: After parsing the open source framework network model using ONNX Parser, convert it to the intermediate state IR Graph; The intermediate state IR goes through four steps of graph preparation, graph splitting, graph optimization, and graph compilation to convert it to the.om model format suitable for Ascend AI processors; Upload the converted.om model to the board-side environment, and later load the model file through the Ascend CL interface to implement the inference process; Step 3.

3. Complete the resource scheduling of the algorithm runtime on the embedded platform to improve the running speed: Call the acl.init interface to initialize the pyACL configuration; Call the acl.rt.set_device interface to specify the Device used for operation; Call the acl.rt.get_run_mode interface to get the running mode of the Ascend AI software stack. The internal processing flow is different depending on the running mode; Call the acl.rt.create_context interface to create a Context; Call the acl.rt.create_stream interface to create a Stream; Call the acl.rt.malloc interface to apply for memory on the Device; Call the acl.mdl.load_from_file interface to load the.om model file; Call the acl.mdl.execute interface to execute model inference and synchronize the interface; Call the acl.mdl.get_dataset_num_buffers interface to get the number of aclDataBuffer from the output aclmdlDataset; Loop through the output aclmdlDataset to get aclDataBuffer data, convert it to a numpy object, and use the NMS post-processing algorithm to output the target coordinate and category information; After all tasks are completed, call the acl.mdl.unload interface to unload the model, call the destroy interface to release the Stream, Context, and Device resources, and end the target detection task.

Images