An infrared image instance segmentation and faint light image background fusion method based on TensorRT accelerated YOLOv5s-seg

Abstract

The present application proposes a method of infrared image instance segmentation and low-light image background fusion based on YOLOv5s-seg accelerated by TensorRT for target optical identification requirements, realizing image registration, target identification, contour highlighting and background fusion of low-light and infrared. Deploy the algorithm to NVIDIA Jetson Orin, collect low-light and infrared images, and perform image registration. Perform instance segmentation on the infrared image to obtain the target mask, and the inference speed is improved by 2 times. Fuse the pseudo-color mask to the low-light image to highlight the contour. Finally, make a ui interface to realize image switching display, save pictures or videos, and display frame rate. The processing time from image acquisition to contour highlighting fusion is within 100ms, combining the background information of the low-light image and the target contour details of the infrared image, improving the display effect and detection speed of the current low-light and infrared image fusion detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to an algorithm for infrared image segmentation and low-light image background fusion based on TensorRT-accelerated yolov5s-seg, deployed on the Jetson Orin platform. Background Technology

[0002] The Jetson Orin platform's Deep Learning Accelerator (DLA), a dedicated deep learning inference engine, provides one-third of the AI ​​computing power on the AGX Orin platform. TensorRT, an acceleration library specifically designed by NVIDIA for its platform, supports forward propagation of deep learning frameworks. TensorRT achieves deployment acceleration through optimization strategies for NVIDIA's own GPUs. cuDLA, on the other hand, is the CUDA runtime interface for DLA, extending the CUDA programming model and integrating DLA and CUDA. Using cuDLA, inference can be implicitly run through the TensorRT runtime or explicitly invoked via the cuDLA API. cuDLA's hybrid and standalone modes differ in synchronization. In hybrid mode, DLA tasks are submitted to the CUDA stream, allowing seamless synchronization with other CUDA tasks. In standalone mode, wait and signal events must be specified; cuDLA must wait for and signal separately, as part of the cudlaSubmitTask. In summary, using cuDLA's hybrid mode allows for rapid integration with other CUDA tasks, while using standalone mode avoids creating a CUDA context, thus saving resources throughout the process and ensuring operation without a CUDA context.

[0003] Low-light imaging refers to the technology of imaging using weak light sources (such as starlight and moonlight). It can acquire clear images in extremely dark environments and is suitable for nighttime or low-light conditions. Low-light imaging equipment is sensitive to light but less sensitive to strong light, and performs better in low-light environments. Compared to low-light imaging, infrared imaging utilizes the infrared radiation emitted by objects. It is less affected by lighting conditions, can penetrate obstacles such as smoke and dust, and is suitable for monitoring and reconnaissance in complex environments, and has stronger anti-interference capabilities. Therefore, to a certain extent, target detection and instance segmentation of infrared images can compensate for the problem of unclear target outlines in low-light images under adverse conditions. Furthermore, background fusion in low-light images helps improve the clarity and visibility of the images, making the details of target objects more prominent while preserving the background characteristics of the original low-light image.

[0004] Infrared image segmentation refers to the process of separating target object regions from background regions by classifying infrared images acquired through infrared radiation imaging technology at the pixel level. In infrared image processing, due to the thermal radiation characteristics of objects, there is a significant difference between their heat and the background, making infrared image segmentation an important task. Traditional infrared image segmentation methods include matched filtering, edge detection, and thresholding, but they can only extract surface features and cannot fully utilize deeper semantic information, resulting in unsatisfactory performance against complex backgrounds. With the development of neural networks in infrared image segmentation, new challenges have emerged. The overall low resolution and limited accuracy of infrared images may affect the learning and recognition performance of neural networks on image features. Furthermore, the preprocessing of infrared images to remove noise and enhance contrast also presents challenges for the training and application of neural networks. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes an instance-segmentation-based fusion technique for target optical recognition applications. This technique enables image registration, target recognition, contour highlighting, and background fusion between low-light and infrared images. The detection and inference speed can be increased by more than two times.

[0006] The technical solution adopted in this invention is: an algorithm for infrared image segmentation and low-light image background fusion based on TensorRT-accelerated YOLOv5S-SEG, deployed on the Jetson Orin platform. It includes the following steps:

[0007] Step 1: Use an infrared camera to collect an infrared image dataset containing five types of targets: people, vehicles, bicycles, motorcycles, and drones, and divide it into a training set, a test set, and a validation set.

[0008] Step 2: Use the training set to train the QAT YOLOv5s-seg model. After obtaining the optimal model, call the TensorRT API to reconstruct the YOLOv5 network architecture and obtain the YOLOv5s instance segmentation engine file.

[0009] Step 3: Combine the engine file with cuDLA to deploy the network using CUDA, test the obtained network model using the test set, and run inference using the validation set;

[0010] Step 4: Use a binocular camera to acquire low-light and infrared images in real time, and after image preprocessing, use affine transformation to perform image registration and align the target in the low-light and infrared images.

[0011] Step 5: Use the deployed engine model to perform instance segmentation on the infrared image, generate a target mask, apply pseudo-color processing to the mask, and then fuse it onto the target in the low-light image to achieve contour highlighting.

[0012] Step 6: Use the Python interface pyside2 to create a UI interface to switch between low-light images, infrared images, fused images, and contour highlighting images fused from object detection, including pause display, resume display, stop display, save video or image, and display frame rate.

[0013] Furthermore, in step 1, an infrared image dataset containing five types of targets—people, vehicles, bicycles, motorcycles, and drones—is constructed and divided into a training set, a test set, and a validation set in a 7:2:1 ratio, as detailed below:

[0014] Images were acquired using the LA6110 high-performance uncooled infrared camera module. After mixing and data cleaning to remove invalid data, the original dataset of 918 images was retained. The dataset was expanded using data augmentation techniques, including flipping the images horizontally and vertically, rotating them clockwise by 90°, 180°, and 270°, and applying Mosaic data augmentation. The dataset was expanded to 4590 images, which were then divided into training, testing, and validation sets in a 7:2:1 ratio: 3213 images for training, 918 for testing, and 458 for validation.

[0015] Furthermore, in step 2, based on the YOLOv5s-seg network architecture, quantitative perceptual training is performed using the training set obtained in step 1. Specifically:

[0016] YOLOv5s-seg was quantized and trained using the TensorRT pytorch-quantization toolkit, and then Q / DQ nodes were placed in the TensorRT Q / DQ processing network. To make it compatible with DLA, additional Q / DQ nodes were added, and a Q / DQ transformer was used to infer the input / output scales of other layers. By extracting the Q / DQ nodes from the QAT model, the quantization scale of the YOLOv5 model can be obtained, and information from adjacent layers can be used to infer the input / output scales of other layers. Finally, the ONNX model without Q / DQ nodes and the (PTQ) calibration cache file were exported so that TensorRT could use them to build a DLA engine; this invention uses FP16 mode.

[0017] The TensorRT engine first uses tensorrtx to obtain YOLOv5s-seg.wts, then uses CMake to compile and generate a YOLOv5 Visual Studio project, and finally compiles YOLOv5s-seg.wts to generate the YOLOv5s-seg.engine file.

[0018] Furthermore, in step 3, the YOLOv5s-seg engine file of DLA accelerated by TensorRT is used in conjunction with cuDLA to deploy the network using CUDA, as detailed below:

[0019] texec is a tool provided by TensorRT for creating engines and performance benchmarks. A DLA loadable program refers to a program that TensorRT can package into its serialization engine after successfully compiling DLA using the DLA compiler and obtaining the results. cuDLA is the CUDA runtime interface for DLA, integrated with the CUDA programming model. It allows implicit or explicit invocation of the cuDLA API for inference operations in both mixed and standalone modes. This invention aims to utilize the explicit invocation of the cuDLA API to perform inference operations in both mixed and standalone modes.

[0020] The cuDLA application deployment process is as follows: The `cudlaCreateDevice` function creates the DLA device. The `cudlaModuleLoadFromMemory` function loads the engine from memory for DLA use. The `cudaMalloc` and `cudlaMemRegister` functions are called to allocate memory on the GPU and then register the CUDA pointer with the DLA (mixed mode only). The `cudlaImportExternalMemory` and `cudlaImportExternalSemaphore` functions are called to import external NvSci buffers and synchronization objects (standalone mode only). The `cudlaModuleGetAttributes` function retrieves module attributes from the loaded module. The `cudlaSubmitTask` function is called to submit the inference task. In mixed mode, the user needs to specify a CUDA stream for the cuDLA task to run on. In standalone mode, the user needs to specify signaling events and wait events for cuDLA to wait and signal when the corresponding fence expires.

[0021] The network model of the engine deployed in step 3 is tested using the test set obtained in step 1, and inference is run on the validation set. The test set provides performance metrics for the model, such as accuracy, recall, and F1 score. Recall represents the proportion of correctly predicted samples among the true samples, precision represents the proportion of correctly predicted results among all predictions, AP is obtained by calculating the area under the curve with recall on the horizontal axis and precision on the vertical axis, and mAP is the average of the sums of AP for each class across different classes. A higher mAP value indicates better model precision. These metrics provide insight into the model's performance on the dataset, allowing for optimization of model design and hyperparameter tuning. Furthermore, to more objectively evaluate the performance of the obtained engine model, inference runs on the validation set are also necessary. The validation set is an independent dataset typically used to evaluate the model's generalization ability, i.e., its ability to handle unseen data. Inference runs on the validation set provide a more comprehensive evaluation of the model's performance in real-world applications.

[0022] Furthermore, in step 4, low-light images and infrared images are acquired in real time using a binocular camera. Image registration is then performed on the infrared images using affine transformation, as detailed below:

[0023] After preprocessing, calibration is performed from a set of selected low-light and infrared images. Next, target matching is performed between the infrared and low-light images, selecting points of interest (POIs) of the target contours and associating or "matching" their corresponding points in the two images. Then, an affine transformation matrix is ​​calculated based on these matched feature points. Next, an affine transformation is performed on the infrared image to be registered, ensuring that the transformed infrared image matches the target in the low-light image. The transformed infrared and low-light images are compared and evaluated. If the result is unsatisfactory, parameters need to be adjusted or feature points rematched, and the affine transformation repeated until a satisfactory registered image is obtained. Image registration aims to achieve accurate registration between infrared and low-light images for better subsequent image processing and analysis.

[0024] Affine transformations refer to operations such as translation, rotation, scaling, and shearing performed on a graphic in two-dimensional or three-dimensional Euclidean space to obtain a new graphic. Affine transformations can be described using matrices. Taking a two-dimensional affine transformation as an example, suppose there is a point P(x, y), which, after translation, rotation, scaling, and shearing, results in a new point P′(x′, y′). 11 a 22 Represents the scaling factor, a 12 a 21 Represents the rotation and shear factors, t x , t y This represents the translation amount. The affine transformation matrix is ​​then expressed by the following formula:

[0025]

[0026] Furthermore, in step 5, the YOLOv5s-seg engine model deployed in step 3 is used to perform instance segmentation on the infrared image, generate a target mask, and perform pseudo-color processing. This mask is then fused to the target in the low-light image, preserving background information and highlighting the contour. Specifically:

[0027] YOLOv5s-seg improves and extends YOLOv5, applying it to image segmentation tasks. The YOLOv5s-seg network model consists of four parts: input, backbone, neck, and head.

[0028] YOLOv5s-seg employs Mosaic data augmentation at its input. It utilizes random scaling, cropping, and arrangement for stitching, improving the detection performance of small objects. Adaptive anchor box calculation and adaptive image scaling are used; during quantization-based training, the optimal anchor box values ​​in the training set are adaptively calculated. The original images are uniformly scaled to a standard size before being fed into the detection network, and YOLOv5s-seg adaptively adds minimal black borders to the original images. First, the scaling ratio is calculated by dividing the standard size by the original image size, and a smaller scaling factor is selected. Then, the length and width of the original image are multiplied by the minimum scaling factor, compared with the standard size, and the padding value is calculated. Finally, the remainder is taken using NumPy's `np.mod` method, divided by 2, to obtain the padding values ​​needed at the length or width ends of the image.

[0029] The YOLOv5s-seg backbone network consists of 6x6 convolutional layers and a CSP structure. Before the infrared image enters the backbone, the 6x6 convolutional layers slice the image, concentrating the W and H information into the channel space. This expands the input channels by four times, meaning the stitched image becomes 12 channels compared to the original RGB three-channel mode. Finally, the resulting new image undergoes another convolution operation, yielding a double-downsampled feature map without information loss. The YOLOv5s network uses the CSP1_X structure in the backbone network, while the CSP2_X structure is used in the neck.

[0030] YOLOv5s-seg's Neck adopts an FPN+PAN structure. The FPN layer conveys strong semantic features from top to bottom, while the PAN tower conveys localization features from bottom to top. The addition of the CSP2_X structure enhances the network's feature fusion capability.

[0031] The YOLOv5-seg head extends and improves upon YOLOv5 by adding a segmentation head, enabling the model to perform object detection and image segmentation simultaneously. A new loss function is used to adapt to image segmentation tasks. The seg model is trained using pixel-level labeled data, not just object bounding box annotations. Appropriate training strategies are employed to balance the weights and loss function between the two tasks, achieving efficient segmentation.

[0032] YOLOv5 uses the CIOU_LOSS loss function from object detection as the bounding box loss function, along with Non-Maximum Suppression (NMS) to search for local maxima, suppress non-maximum elements, and reduce redundant boxes during detection. YOLOv5-Seg introduces the binary cross-entropy loss (BCE) from segmentation tasks to calculate the mask loss function, thereby optimizing segmentation accuracy. The formula for calculating CIOU_LOSS is as follows:

[0033]

[0034] The Intersection over Union (IOU) ratio measures the degree of overlap between two bounding boxes. A is the area of ​​the intersection of the two frames, and B is the area of ​​the union of the two frames. Distance C is the length of the diagonal of the smallest bounding rectangle of the two frames. The Euclidean distance Distance2 between the center points of the graph is used to measure the distance between the center points of two boxes. v is a parameter that measures the consistency of aspect ratio. Where w and h represent the width and height of the block diagram, respectively. The loss function for the segmentation mask is L. mask =BCE(M, M) gt )

[0035] YOLOv5-seg improves and optimizes upon YOLOv5 in terms of network structure, loss function, data annotation, training strategy, and inference algorithm, making it better suited to the needs of image segmentation tasks. The target mask contour of the infrared image is obtained by accelerating the YOLOv5s-seg engine model deployed in step 3 using TensorRT, as shown below:

[0036]

[0037]

[0038]

[0039]

[0040] The OpenCV function `cv.applyColorMap()` is used to add pseudo-color to the obtained target mask, which can convert grayscale images into color images with up to 22 conversion modes. This invention selects the `COLORMAP_JET` mode, which has high contrast and can effectively highlight details in the image. The pseudo-color mask is then blended onto the target in the low-light image, preserving the background of the low-light image and achieving contour enhancement.

[0041] Furthermore, in step 6, a UI interface is created using the Python .pyside2 interface to switch between displaying low-light images, infrared images, fused images, and contour-highlighted images fused from object detection. This includes pausing, resuming, stopping, saving videos or images, and displaying the frame rate. Specifically:

[0042] This program utilizes Qt Creator and the Qt Designer program from the PySide2 library, employing a visual drag-and-drop interface to accelerate development. By dragging and dropping controls on the left, a display interface is built, enabling image switching and frame rate detection. Clicking the save button saves the interface as a .ui file, which can then be dynamically imported into the GUI using the `QUiloader().load` method. The designed .ui file is converted into a Python .py file, and changes made in Qt Designer are dynamically updated in the .py file. When the .ui is interacted with, corresponding signals are triggered. For example, a button click triggers a click signal, and corresponding signal handlers (slots) are defined in the Python code to process the click event. When dynamically importing the .ui file, `QApplication` acts as a GUI manager, monitoring for event triggers. For example, pressing the "low-light image" button displays a real-time low-light image; pressing the "detect fusion" button displays a fused image highlighting the target outline. When using PyInstaller for packaging, .ui resource files cannot be automatically packaged and must be manually packaged. At this point, you can choose to convert the UI file to a Py file to achieve static import. Pyside2 provides a UI to Py conversion function. Resource files can be imported like Python libraries using the import function and can be automatically packaged by PyInstaller.

[0043] Compared with existing technologies, the advantages of this invention are as follows: Based on the YOLOv5s-seg network architecture, it utilizes quantized perceptual training, incorporates TensorRT acceleration, and deploys the network on Jeston Orin using CUDA in conjunction with cuDLA to perform instance segmentation of infrared images. A pseudo-color mask is added to the target in the low-light image after instance segmentation and fused to preserve the original background, thus highlighting the contour. Compared with other deep learning-based infrared image segmentation and traditional low-light and infrared image fusion, this invention has higher detection accuracy and better segmentation results, enabling faster target discovery and labeling. This invention further improves the accuracy and robustness of model detection and segmentation, and increases the overall algorithm speed.

[0044] The present invention will now be further described with reference to the accompanying drawings. Attached Figure Description

[0045] Figure 1 This is a schematic diagram illustrating the implementation process of the network structure of the present invention.

[0046] Figure 2 This is a schematic diagram of the YOLOv5s-seg structure.

[0047] Figure 3 This is a schematic diagram illustrating the QAT training and export of YOLOv5s-seg for DLA.

[0048] Figure 4 Generate FP16 engine configuration and joint cuDLA deployment inference graph for TensorRT to accelerate YOLOv5s-seg.

[0049] Figure 5 This is a schematic diagram of the registration of infrared and low-light images and a schematic diagram of affine transformation.

[0050] Figure 6 This is a diagram showing the final detection and fusion effect of the present invention. Detailed Implementation

[0051] The invention will now be further described with reference to the accompanying drawings.

[0052] This invention provides an algorithm for infrared image segmentation and low-light image background fusion based on TensorRT-accelerated YOLOv5S-SEG, deployed on the Jetson Orin platform, including the following steps:

[0053] Step 1: Construct an infrared image dataset containing five target categories: people, vehicles, bicycles, motorcycles, and drones. Images were acquired using the LA6110 high-performance uncooled infrared camera module. After mixing, invalid data was removed through data cleaning, retaining the original dataset of 918 images. The dataset was then augmented using data augmentation techniques, including flipping the images horizontally and vertically, and rotating them clockwise by 90°, 180°, and 270°. The dataset was expanded to 4590 images, which were then divided into training, testing, and validation sets in a 7:2:1 ratio: 3213 images for the training set, 918 images for the testing set, and 458 images for the validation set.

[0054] Figure 1 This is a schematic diagram of the overall network flow for implementing the method of this invention. The original YOLOv5s-seg network is used for Quantized Awareness Training (QAT), accelerated using TensorRT, and deployed using CUDA in conjunction with cuDLA. The obtained network model is tested using a test set, and inference is run using a validation set. The resulting model file (engine) is then fine-tuned. Low-light and infrared images are acquired in real-time using a binocular camera. After image preprocessing, affine transformation is used for image registration to align the target in the low-light and infrared images. The deployed YOLOv5s-seg.engine model is used to perform instance segmentation on the infrared image, generating a target mask. The mask is then processed with pseudo-color and fused to the target in the low-light image, preserving background information and achieving contour highlighting.

[0055] Figure 2 This is a schematic diagram of the YOLOv5s-seg framework. The YOLOv5 network model structure includes:

[0056] On the input side, Mosaic data augmentation, adaptive anchor box calculation, and minimum black border fill scaling were used.

[0057] The backbone network uses 6x6 convolutions and a CSP structure for image feature extraction. An image is divided into four different slices, which are stacked using CONCAT. These slices then pass through a CBL layer. The CSP structure in YOLOv5 includes CSP1_X for the backbone and CSP2_X for the neck. The CSP module divides the low-level feature maps into two parts and then merges them using the proposed cross-stage hierarchical structure. By segmenting the gradient flow, the gradients propagate through different network paths. The CSP structure can significantly reduce inference speed and computational cost.

[0058] Neck employs a combination of Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) for image downsampling and extracting target features at different scales for use in subsequent prediction layers. In the FPN part, three feature layers of different scales are extracted from the image: 76×76, 38×38, and 19×19. The specific method for fusing these two feature layers is as follows: first, the features of the higher layer are upsampled by a factor of 2; then, the features of the lower layer are modified by a 1×1 convolution to change the number of channels; finally, the upsampled features and the lower-layer features processed by the 1×1 convolution are simply added element-wise at corresponding positions. In the PAN structure, the bottom layer of the feature pyramid in the FPN is copied as the bottom layer of a new feature pyramid. This bottom layer is then upsampled and superimposed with the features of the previous layer in the FPN. Finally, it is processed by a CSP2_1 convolutional structure. The PAN structure primarily improves the performance of object detection and instance segmentation by copying and upsampling the bottom-level features and then superimposing them with the features of the previous layer, thereby achieving better feature information transfer and utilization.

[0059] The Head section extends YOLOv5 by adding a segmentation head, enabling the model to perform object detection and image segmentation simultaneously. A new loss function is used to adapt to image segmentation tasks. CIOU_Loss is used as the loss function, NMS non-maximum suppression is used for bounding box filtering, and binary cross-entropy loss (BCE) is used to calculate the segmentation loss function.

[0060] Multi-scale training is performed using Quantization-Aware Training (QAT). During model pre-training, a small learning rate is used to train for some epochs or steps, followed by warmup training at the pre-set learning rate. A cosine learning rate descent strategy is employed. EMA is used to update weights, essentially giving the parameters momentum during training, resulting in smoother updates. Mixed-precision training (AMP) is used to reduce GPU memory usage and accelerate training, but requires GPU support.

[0061] Step 2: To balance the inference performance and accuracy of YOLOv5s-seg, quantization-aware training must be performed on the model. DLA does not support QAT via TensorRT, so the QAT model must be converted to a post-trained quantization (PTQ) model before inference. Quantizing YOLOv5s-seg using the TensorRT pytorch-quantization toolkit requires adding quantizer modules to the neural network graph. Nodes are placed in the TensorRT quantization / dequantization (Q / DQ) processing network. Following the TensorRT fusion strategy, adjustments are made for GPU inference. These nodes are derived using the scale of adjacent layers via the Q / DQ converter. The scale values ​​of each Q / DQ module are then obtained to calibrate the model. After calibration, the calibrated model is fine-tuned using the training set. The Q / DQ converter is used to convert the trained ONNX graph into a PTQ tensor-scaled ONNX model without Q / DQ nodes, so that TensorRT can use them to build the DLA engine file.

[0062] Figure 3 This diagram illustrates the QAT training and export of YOLOv5s-seg for DLA. YOLOv5 is quantized using TensorRT and the pytorch-quantization toolkit, a technique for optimizing neural network models to run more efficiently on hardware. Quantizer modules are added to the neural network graph, replacing some layers in the model with quantized versions to reduce computational and memory usage. The model is calibrated to obtain the scale values ​​for each quantization / dequantization (Q / DQ) module, which are used in subsequent quantization processes. A training plan is selected, and the calibrated model is fine-tuned using the dataset obtained in step 1 to ensure that the model's accuracy is not reduced after quantization. Q / DQ nodes are placed in the TensorRT Q / DQ processing network, following TensorRT's fusion strategy, primarily for GPU inference. If compatibility with DLA (Deep Learning Accelerator) is required, additional Q / DQ nodes can be added, derived using the scale of adjacent layers via a Q / DQ transformer. The ONNX graph trained using QAT (Quantization-Aware Training) is converted into a PTQ (Post-Training Quantization) tensor scale and an ONNX model without Q / DQ nodes. This process utilizes the quantization scale information extracted by the Q / DQ nodes. Finally, the ONNX model without Q / DQ nodes and the calibration cache file are exported so that TensorRT can build the DLA engine to efficiently run the quantized YOLOv5 model on the accelerator.

[0063] Step 3: Use TensorRT to build the DLA engine file. This allows you to package the DLA loadable program into the serialization engine, seamlessly integrating it with the GPU when needed, and running inference using CUDA via cuDLA. `cuDLACreateDevice` creates the DLA device. Since network quantization training is completed on the GPU and then deployed to the target DLA, validation against the target is crucial. Load the image data and normalize it. Because DLA only supports INT8 / FP16, additional reformatting of the inference input and output is required. This invention uses the FP16 format. After inference, the inference result is decoded and non-maximum suppression (NMS) is performed to obtain the detection result.

[0064] Figure 4This document describes the FP16 engine configuration for accelerating YOLOv5s-seg with TensorRT and the deployment inference graph in conjunction with cuDLA. TensorRT, an SDK from NVIDIA for high-performance deep learning inference, optimizes deep learning models and improves inference speed. DLA (Deep Learning Accelerator) is a hardware accelerator specifically designed to accelerate deep learning inference on the NVIDIA Jetson platform. When building DLA loadable programs using TensorRT, cuDLA provides a set of CUDA runtime interfaces to integrate the DLA and CUDA programming models, simplifying the interaction between developers and DLA. cuDLA's application programming interfaces (APIs), such as cudlaCreateDevice, cudlaModuleLoadFromMemory, cudlaMalloc, cudlaMemRegister, cudlaImportExtemalMemory, cudlaImportExtemalSemaphore, cudlaModuleGetAttributes, and cudlaSubmitTask, are used for operations such as creating DLA devices, loading engine memory, managing memory allocation, importing external buffers and synchronization objects, obtaining module attributes, and submitting inference tasks. In hybrid mode, a CUDA stream needs to be specified to control the execution of cuDLA tasks; in standalone mode, semaphore events and wait events need to be specified to synchronize tasks. In the inference workflow, image data is first loaded and normalized, then the input and output are reformatted according to the INT8 / FP16 data types supported by DLA. After inference, the results are decoded and NMS (non-maximum suppression) is performed, and finally the results are saved and mAP is calculated. This invention uses the YOLOv5s-seg model and performs additional FP16 formatting on the input and output. Since Orin DLA is highly optimized for INT8, there may be some performance degradation when passing FP16 data to DLA, but it is still faster than previous methods.

[0065] Step 4: First, use a binocular camera to acquire low-light and infrared images in real time. After preprocessing, select a set of low-light and infrared images for calibration. Match the target in the infrared and low-light images, selecting points of interest (POIs) on the target contour, which represent important or unique content (corners, edges, etc.). After identifying key feature points in the infrared and low-light images, associate or "match" the corresponding key points in the two images. Calculate the affine transformation matrix based on the matched feature points. Perform an affine transformation on the infrared image to be registered, and compare and evaluate the target in the transformed infrared and low-light images. If the requirements are not met, adjust the parameters or rematch the feature points, and repeat the affine transformation to finally obtain the registered image.

[0066] Figure 5 This diagram illustrates the registration and affine transformation of infrared and low-light images. An affine transformation, in geometry, maps a set of points from one vector space to another while preserving the original linear and parallel properties. Simply put, an affine transformation maintains the relative positional relationships between lines, including parallelism and length ratios. Therefore, by using the same target in both infrared and low-light images for keypoint calibration and performing an affine transformation on the infrared image, a new, registered infrared image is obtained, allowing the target in both images to overlap. The affine transformation... It can be written as

[0067] Step 5: The registered infrared image is processed using the YOLOv5s-seg-engine obtained in Step 3 for target detection and instance segmentation, yielding a mask for the detected target. YOLOv5-seg is an improvement and extension of YOLOv5, adding a segmentation head to achieve image segmentation. To optimize segmentation accuracy, YOLOv5-seg introduces a loss function for the segmentation task. Unlike traditional YOLOv5, the loss function of YOLOv5-seg considers both target detection and image segmentation tasks simultaneously. These two tasks are performed in parallel, designed as a single-stage YOLO network, resulting in faster speed. Then, k mask prototype images are generated in the instance segmentation branch, and the confidence scores of k mask prototype images are generated in the detection branch. The mask prototype images and their confidence scores are then linearly combined. Mask prototype images related to the target have high confidence scores, while other mask prototype images have low confidence scores; this combination forms the instance segmentation result.

[0068] In step 6, a UI interface is created using the Python .pyside2 interface to switch between displaying low-light images, infrared images, fused images, and contour-highlighted images fused from target detection. This includes pausing, resuming, stopping, saving videos or images, and displaying the frame rate. Qt Creator software and the Qt designer program of the PySide2 library are used with a visual drag-and-drop interface to accelerate development. By dragging and dropping controls on the left, a display interface is built to achieve image switching and frame rate display. The designed UI file is converted into a Python .py file, and changes made in Qt Designer are dynamically updated in the .py file. When the UI is operated, corresponding signals are triggered. For example, when a button is clicked, a click signal is triggered, and the corresponding signal processing function (slot) is defined in the Python code to handle the click event. At this point, the UI file is dynamically imported, and QApplication acts as a GUI manager, responsible for monitoring whether events are triggered. For example, when the low-light image button is pressed, a real-time low-light image is displayed. When the detection fusion button is pressed, a fused image highlighting the target contour is displayed. This achieves the final display of the invention.

[0069] Figure 6 This is a diagram showing the final detection and fusion effect of the present invention.

[0070] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for infrared image instance segmentation and low-light image background fusion based on TensorRT-accelerated YOLO5s-seg, deployed on the Jetson Orin platform, is described below: Step 1: Use an infrared camera to collect an infrared image dataset containing five types of targets: people, vehicles, bicycles, motorcycles, and drones, and divide it into a training set, a test set, and a validation set. Step 2: Use the training set to train the QAT YOLOv5s-seg model. After obtaining the optimal model, call the TensorRT API to reconstruct the YOLOv5 network architecture and obtain the YOLOv5s instance segmentation engine file. Step 3: Combine the engine file with cuDLA to deploy the network using CUDA, test the obtained network model using the test set, and run inference using the validation set; Step 4: Use a binocular camera to acquire low-light and infrared images in real time, and after image preprocessing, use affine transformation to perform image registration and align the target in the low-light and infrared images. Step 5: Use the deployed engine model to perform instance segmentation on the infrared image, generate a target mask, perform pseudo-color processing on the mask and then fuse it onto the target in the low-light image, retaining background information and highlighting the outline. Step 6: Use the Python interface pyside2 to create a UI interface to switch between low-light images, infrared images, fused images, and contour highlighting images fused from object detection, including pause display, resume display, stop display, save video or image, and display frame rate.

2. The method according to claim 1, characterized in that: The specific process of step 1 is as follows: use an infrared camera to capture images, obtain a certain number of images by sampling in an average distribution manner, mix them, remove invalid data through data cleaning, and retain the original dataset of 918 images; expand the dataset by using data augmentation, and perform data augmentation by flipping up, down, left, and right, rotating clockwise by 90°, 180°, and 270°, as well as Mosaic data augmentation. The dataset was expanded to 4590 images and divided into training, test, and validation sets in a 7:2:1 ratio. The training set contained 3214 images, the test set contained 918 images, and the validation set contained 458 images.

3. The method according to claim 1, characterized in that: The specific process of step 2 is as follows: In order to balance the inference performance and accuracy of YOLOv5s-seg, quantization-aware training (QAT) must be performed on the model; DLA does not support QAT through TensorRT, so the QAT model must be converted into a post-trained quantization (PTQ) model before inference; to quantize YOLOv5s-seg using the TensorRTpytorch-quantization toolkit, a quantizer module needs to be added to the neural network graph. Place nodes in the TensorRT quantization / dequantization (Q / DQ) processing network; Following the TensorRT fusion strategy, adjustments were made for GPU inference; These nodes can be obtained using the scaling of adjacent layers via a Q / DQ converter; The scale value of each Q / DQ module is then obtained to calibrate the model. After calibration, the training set is selected to fine-tune the calibrated model. The Q / DQ converter is used to convert the trained ONNX graph into a PTQ tensor scale and an ONNX model without Q / DQ nodes, so that TensorRT can use them to build the DLA engine file.

4. The method according to claim 1, characterized in that: Step 3 is as follows: Use TensorRT to build the DLA engine file, which packages the DLA loadable program into the serialization engine, allowing seamless integration with the GPU when needed. Run inference using CUDA via cuDLA; create a DLA device using cudlaCreateDevice; since network quantization training is completed on the GPU and then deployed to the target DLA, validation against the target is crucial; load image data and normalize it; since DLA only supports INT8 / FP16, additional reformatting of the inference input and output is required; after inference, decode the inference results and perform non-maximum suppression (NMS) to obtain the detection results.

5. The method according to claim 1, characterized in that: Step 4 is as follows: First, use a binocular camera to acquire low-light and infrared images in real time; after preprocessing, select a set of low-light and infrared images for calibration, match the targets in the infrared and low-light images, and select the points of interest in the target outline, which represent important or unique content; after identifying the key feature points in the infrared and low-light images, associate or "match" the corresponding key points in the two images; calculate the affine transformation matrix based on the matched feature points; perform an affine transformation on the infrared image to be registered, and compare and evaluate the targets in the transformed infrared image and the low-light image; If the requirements are not met, adjust the parameters or rematch the feature points, and perform the affine transformation again to finally obtain the registered image. Let (v, w) be the coordinates of a pixel in the infrared image, and (x, y) be the coordinates of a pixel in the low-light image. The formula for the affine transformation is as follows: 。 6. The method according to claim 1, characterized in that: Step 5 is as follows: The registered infrared image is processed using the YOLOv5s-seg-engine obtained in Step 3 for target detection and instance segmentation to obtain a mask for the detected target. YOLOv5s-seg is an improvement and extension of YOLOv5, adding a segmentation head to achieve image segmentation. To optimize segmentation accuracy, YOLOv5s-seg introduces a loss function for the segmentation task. Unlike traditional YOLOv5, the loss function of YOLOv5s-seg considers both target detection and image segmentation tasks simultaneously. These two tasks are performed in parallel, designed as a single-stage YOLO network, resulting in faster speed. Then, k mask prototype images are generated in the instance segmentation branch, and the confidence scores of k mask prototype images are generated in the detection branch. The mask prototype images and their confidence scores are then linearly combined. Mask prototype images related to the target have high confidence scores, while other mask prototype images have low confidence scores. This combination forms the instance segmentation result. The calculation of the obtained mask is as follows: Suppose we obtain a bounding box with center coordinates (x, y) and width and height (w, h). The width and height of the image are (w, h, h). i h i The process requires converting (x, y) to coordinates (x′, y′) of a 640x640 image, and converting (w, h) to the width and height of the bounding box (w′, h′) within the 640x640 image. gt ,h gt The specific formula for the mask matrix is: Calculate the grid coordinates (x′, y′) of the target bounding box in an 80*80 grid. g y g Note x g y g They are all floating-point numbers; For x g y g Rounding down, we obtain the integer grid coordinates (x0, y0). To accelerate training convergence, YOLO5s-seg takes one neighboring grid cell to the left, right, top, and bottom of (x0, y0): (x1, y0) and (x0, y1). The formula is as follows: x1=x0+(2*round(x g -x0)-1) y1=y0+(2*round(y g -y0)-1) This means that the target box is considered to be located around the three adjacent cells (x0, y0), (x1, y0), and (x0, y1). When using the K-means clustering algorithm to obtain nine anchor boxes, the anchors with the smallest width and height, anchor0, anchor1, and anchor2, are assigned to each cell in the 80*80 grid. Therefore, (x0, y0), (x1, y0), and (x0, y1) correspond to the three anchor boxes anchor0, anchor1, and anchor2 in the 80*80 grid. The width and height of these three anchor boxes are (w0, h0), (w1, h1), and (w2, h2) respectively. Parsing the JSON tag file, the width and height of the target box are (w0, h0), (w1, h1), and (w2, h2). gt h gt Then calculate (w) separately. gt h gt The ratios of (w0, h0), (w1, h1), and (w2, h2) are used to eliminate anchor boxes that do not meet the requirements. The retained anchor boxes are marked as true, and the eliminated anchor boxes are marked as false. Then, the three anchor boxes are labeled as (m0, m1, m2). and If anchori is selected, then anchori is selected; otherwise, anchori is removed, where i = 0, 1, 2. m0=(t w0 <4&&t h0 <4) m1=(t w1 <4&&t h1 <4) m2=(t w2 <4&&t h2 <4) Assign values ​​to the mask matrix based on the three coordinate positions (x0, y0), (x1, y0), and (x0, y1): For all bounding boxes in an image, the above judgment is performed and the mask matrix is ​​assigned a value to obtain the mask matrix of the image. The model output includes two parts: detection results and segmentation results. The detection results have a dimension of 25200*177, where the first 85 columns are the results of each detection box, and the last 32 columns are the mask coefficients of each detection box. The segmentation result is the prototype mask, with a dimension of 32*160*160. The post-processing process is actually a weighted sum of the mask coefficients in the bounding boxes and the prototype mask to obtain the instance segmentation effect. The post-processing method is transformed into matrix multiplication. A matrix is ​​represented using Mat in OpenCV, and instance segmentation is achieved through matrix operations. Finally, pseudo-color is added to the mask using OpenCV's applyColorMap color transformation and blended onto the target in the low-light image, preserving the background of the low-light image and highlighting the target outline.

7. The method according to claim 1, characterized in that: Step 6 involves the following steps: Using Qt Creator software, design the UI interface; install the PySide2 library and utilize its Qt Designer program to accelerate development efficiency through a visual drag-and-drop interface; by dragging and dropping controls on the left, build a display interface to switch between low-light images, infrared images, fused images, and contour highlighting images fused with target detection, including pausing, resuming, stopping, saving videos or images, and displaying the detection frame rate. Clicking the save button will save the interface as a .ui file, which can then be dynamically imported into the GUI using the QUiloader().load method. The designed ui file will be converted to a Python .py file, and changes made in QTDesigner will be dynamically updated in the .py file to achieve the final interface display.

Images