A remote sensing target detection method, system, medium, device and terminal

Abstract

The application belongs to the technical field of computer vision, and discloses a remote sensing target detection method, system, medium, equipment and terminal, constructs an RGB-infrared image fusion network based on CNN, adopts a task-oriented fusion method, a low-layer and high-layer joint adaptive training strategy, and utilizes a joint loss composed of content loss and semantic loss to constrain the fusion network and the fusion method of salient target detection; angle information is introduced to predict the angle by using Gaussian cycle encoding and rectangular cycle encoding, feature extraction is carried out, and a BIFPN bidirectional feature fusion method is used for feature fusion; a real-time collected multi-modal video stream is used for decoding, algorithm reasoning and boundary box calibration, and the result is encoded and pushed to a streaming media server or a client. The application can quickly and accurately detect the vehicle position and category information under strong light and weak light conditions by combining the image fusion algorithm and the rotating target detection algorithm, and is applied to the rotating target detection in the unmanned aerial vehicle traffic management and disaster rescue scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a remote sensing target detection method, system, medium, device and terminal. Background Technology

[0002] Currently, aerial imagery often contains small objects obscured by complex surrounding scenes; it is cluttered and densely packed. Detected objects are typically densely arranged, such as vehicles and ships. Objects in aerial imagery can appear in different orientations. The large aspect ratio problem common in remote sensing further challenges this. Figure 2 As shown, the SCRDet rotating target detection algorithm addresses the following issues: 1) Small targets: A sampling fusion network (SF-Net) is designed to combine feature fusion and finer anchor sampling; 2) Complex backgrounds: A supervised multidimensional attention network (MDA-Net) is developed, consisting of pixel attention networks and channel attention networks, to suppress noise and highlight the foreground; 3) Cluttered and dense objects in arbitrary directions: An angle-sensitive network is designed by introducing angle-related parameters for estimation.

[0003] To obtain a more accurate rotation estimate, an IoU constant factor is added to the smooth-L1 loss to address the boundary issues of the rotated bounding box angle, such as... Figure 2 As shown. Improvement solutions exist: replacing the small object detection module with FPN or PANet is more effective. Angle regression can be replaced with a CLS-based angle classification task, resulting in more accurate angle prediction.

[0004] However, existing remote sensing target detection methods suffer from inaccurate angle prediction. The paper "Arbitrary-Oriented Object Detection with Circular Smooth Label" argues that the fundamental reason for this inaccuracy is that the ideal angle prediction exceeds the defined range. This paper addresses the periodicity of angles by transforming angle prediction from a regression problem into a classification task, increasing tolerance for adjacent angles, and introduces the impact of four window functions and different window radii on detection performance. Existing remote sensing target detection methods also suffer from low-light detection issues. Currently, all algorithms are designed for RGB images, specifically images captured during normal daytime conditions, significantly reducing their detection capability in low-light environments at night. Furthermore, existing remote sensing target detection methods face the challenge of complex backgrounds. Common solutions include semantic segmentation combined with target detection, dilated convolution to increase the receptive field, and channel attention mechanisms to weaken background information. However, the first two methods suffer from issues such as poor real-time performance and the weakening of contextual information due to sparse feature scale. YOLOv4 utilizes the ModifiedSAM attention mechanism, which effectively alleviates this problem. In existing remote sensing target detection methods, small target samples are easily lost. The emergence of the Feature Pyramid Network (FPN) structure has alleviated this problem to some extent.

[0005] The R3Det algorithm was proposed, and the network was built on top of RetinaNet (see...). Figure 4 To address the challenge of detecting rotating objects with high aspect ratios, dense distribution, and extreme class imbalance, this paper proposes an end-to-end refined single-stage rotation detector. This detector includes a feature refinement module (FRM), which re-encodes the position of the current refined bounding box onto corresponding feature points through feature interpolation, achieving feature reconstruction and alignment. Model decomposition: For objects with high aspect ratios, a fast rotation single-stage detector is proposed, employing an anchor combination strategy to enable the detector to efficiently handle dense scenes. The FRM is specifically designed to address class imbalance, aiming to make the detector's features more accurate and reliable during the refinement stage. Finally, this algorithm proposes SkewIoU loss instead of Smooth L1 loss (see...). Figure 5 Two sets of cuboids have the same angle difference but different aspect ratios. Therefore, the Smooth L1 loss values ​​are the same for both sets (mainly considering the angle difference), but the SkewIoU differs significantly. Thus, the Smooth L1 loss function is not suitable for rotation detection, especially for objects with large aspect ratios that are highly sensitive to skew.

[0006] Based on the above analysis, the existing technologies have the following problems and defects: (1) Complex background in UAV images: UAV images often contain a large number of similar-looking objects in densely populated target areas, leading to problems such as missed detections or false detections. In addition, there is a lot of noise information in the background of UAV images, which can weaken or occlude targets, making them difficult to detect continuously and completely. (2) Small target problem in UAV images: The scale range of targets in UAV images is large, and buildings and pedestrians, mountains and animals often appear in the same image. Small targets account for a very small proportion in the image, providing limited resolution, thus making detection difficult. (3)

[0007] Objects in UAV images may appear at any position and in any direction, and the angle changes of the same type of object are not the same. UAV target detection tasks are therefore difficult, and rotating objects make position regression difficult, resulting in a large number of targets being missed. Text detection has the same characteristics, and some research on improved target detection is inspired by text detection. In recent years, many innovative algorithms have been developed to solve the problem of target rotation. Common detection methods can be divided into horizontal detection and rotation detection according to the form of candidate regions and bounding boxes. (4) Low light detection problem: Existing target detection algorithms still cannot detect all scenes. For example, there are still a large number of missed or undetectable cases in night scenes and extremely low light environments. Summary of the Invention

[0008] To address the problems existing in the prior art, the present invention provides a remote sensing target detection method, system, medium, device and terminal, and particularly relates to a remote sensing target detection method, system, medium, device and terminal based on multimodal image fusion.

[0009] This invention addresses the challenges of multi-target clustering, large scale, and excessive background information in aerial images. Combining the concept of rotating target detection, it proposes a rotating target detection algorithm, YOLOv5-Rotate, based on YOLOv5. This algorithm introduces angle information to represent rotating targets and employs an angle classification method to address the issue of angles exceeding boundary ranges in angle regression methods. The angle classification method utilizes cyclic Gaussian coding and rectangular coding to solve the distance problem in angle classification. Simultaneously, it proposes an improved feature extraction network based on the CBAM (Convolutional Block Attention Module) attention mechanism. By introducing a multi-dimensional attention mechanism into the feature extraction network, this algorithm improves the network's accuracy in locating aerial targets. To address the limited detection capability in low-light environments, this invention introduces an infrared-visible light image fusion method for information complementarity, proposing a detection task-driven image fusion algorithm, DetFusion. First, this image fusion algorithm incorporates the detection loss from the YOLOv5 rotation detector to enhance the semantic information of the fusion network. An adaptive training strategy is also proposed to enable synchronous training of the image fusion algorithm and the detector, allowing for better fusion of images from specific detection scenarios. Second, a salient object detection image fusion method is proposed. Based on object detection information, salient object masks and background masks are constructed, and a target saliency intensity loss function is designed to enhance the pixel intensity of objects in the image while weakening the background pixel intensity. Finally, a fusion network model design is proposed to complete the processes of feature extraction, feature fusion, and image reconstruction, incorporating the Sobel operator for texture gradient extraction.

[0010] Based on the aforementioned rotating target detection algorithm, a target detection edge computing system was designed and implemented. This system utilizes a multimodal camera from a UAV to provide data, and combines it with the Jetson Nano embedded platform to process the video stream and apply the algorithm. Furthermore, it leverages the hardware capabilities of TensorRT and a multi-threaded software architecture for real-time optimization.

[0011] The multimodal image-based rotating target detection algorithm described herein has the following steps:

[0012] Step 1, Multimodal Image Fusion: Determine the data fusion object and construct a fusion network. Use joint loss to constrain the fusion network, use the target information from target detection to construct target and background masks to improve target saliency, and use an image reconstructor to achieve feature aggregation and image reconstruction.

[0013] Step 2, Rotation Target Detection: The rectangle is represented by a five-parameter method, CSP-DarkNet53 is used for feature extraction, and Bi-FPN bidirectional feature fusion is used for feature fusion. Guass encoding is used to handle the rotation angle problem.

[0014] Step 3, video stream edge computing: Real-time acquisition of RGB infrared video stream and infrared video stream, followed by three asynchronous processing steps, and finally video encoding and streaming to streaming media or client.

[0015] Furthermore, the multimodal image fusion described in step one includes:

[0016] (1) The infrared-visible light dataset with multimodal registration is used as the data fusion object;

[0017] (2) The fusion network consists of a feature extractor and an image reconstructor, which utilizes dual-stream feature extraction, gradient residual blocks and feature concatenation to integrate deep features and fine-grained detail features;

[0018] (3) A task-oriented fusion method is adopted, with a joint adaptive training strategy for low and high layers and a joint loss constraint fusion network composed of content loss and detection loss, so that the network contains more semantic information; the fine-grained features of infrared images and visible light images are fused through a cascade strategy, and the results are fed back to the image reconstructor to realize feature aggregation and image reconstruction, and the size of the fused image is consistent with that of the source image.

[0019] (4) Introducing a target mask to guide the network model in detecting salient regions, while ensuring the consistency of the intensity and gradient of the specified target region to achieve the fusion of target region saliency and background texture. Utilizing the bounding box representation in the target label of the infrared image. Use the center point coordinates and width and height to construct a target object mask. The background mask is a mask. .

[0020] (5) Design of loss function for fusion network, including content loss, which is composed of intensity loss. and texture loss It consists of two parts, with the intensity loss consisting of the target intensity and the background intensity.

[0021] Furthermore, the rotating target detection in step two includes:

[0022] (1) Five-parameter method Represents a rectangle in any direction, with a range of... ;

[0023] (2) Feature extraction was performed using the backbone network CSP-DarkNet53 in Yolov5;

[0024] (3) Feature fusion is performed using the Bi-FPN bidirectional feature fusion method;

[0025] (4) The Gaussian cyclic label encoding method is used to detect target information.

[0026] (5) The CBAM attention mechanism is used to enhance network performance in the channel and spatial domains.

[0027] Furthermore, the video stream edge computing in step three includes:

[0028] (1) Real-time acquisition of RGB and infrared video from the high-altitude perspective of the UAV;

[0029] (2) Asynchronous processing decodes the RGB video stream and the infrared video stream to decode the RGB-infrared image pair and stores it in the first buffer queue; if it is in YUV format, it is converted into RGB.

[0030] (3) Extract the RGB-infrared image pairs from the first buffer queue, perform algorithm inference for model reading, obtain multi-layer detection results, and put the RGB-infrared image and inference result pairs into the second buffer queue;

[0031] (4) Extract the dRGB-infrared image pairs and inference results from the second buffer queue, perform bounding box calibration, and put the calibrated image pairs into the third buffer queue;

[0032] (5) Extract the RGB-infrared image pairs from the third buffer queue to obtain the inferred image, and finally encode the video and push it to the streaming media or client.

[0033] Another object of the present invention is to provide a remote sensing target detection system applying the aforementioned remote sensing target detection method, the remote sensing target detection system comprising:

[0034] The image fusion module is used to design an image fusion method based on object detection task and an image fusion method based on salient object detection, and to use a joint adaptive training strategy for low and high layers, and to constrain the fusion network by using a joint loss composed of content loss and semantic loss.

[0035] The rotation detector module incorporates angle information, employing a five-parameter method to represent rectangles in arbitrary directions, and Gaussian encoding to smooth angle classification labels and cyclic labels. Features are extracted using CSP-DarkNet53 and fused using the BIFPN bidirectional feature fusion method.

[0036] The video stream edge computing module is used to perform decoding, algorithm inference, and bounding box calibration tasks using real-time acquired multimodal video streams, and encode and push the results to the streaming media server or client.

[0037] Another object of the present invention is to provide a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the remote sensing target detection method described above.

[0038] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the remote sensing target detection method.

[0039] Another objective of this invention is to provide an information data processing terminal for implementing the aforementioned remote sensing target detection system.

[0040] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0041] First, addressing the technical problems existing in the prior art and the difficulty of solving them, this paper closely analyzes, in conjunction with the technical solution to be protected by this invention and the results and data obtained during the research and development process, how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about by solving these problems. The specific description is as follows:

[0042] This invention provides a remote sensing target detection method based on multimodal image fusion, including the fusion of infrared and visible light in multimodal images, and an improved Yolov5 algorithm for rotating target detection, enabling target detection to maintain good performance even in UAV aerial photography scenarios. Addressing the low efficiency of visible light detection in low-light nighttime scenes and the low efficiency of detecting small targets and targets with high aspect ratios, this invention proposes an algorithm combining a lightweight multimodal image fusion network based on gradient residual blocks with a rotating target detector, employing an adaptive training strategy that combines low-level and high-level layers. Simultaneously, this invention also designs a video stream edge computing system, utilizing real-time multimodal video streams acquired by UAVs and embedded edge devices for algorithm inference, and pushing the results to a streaming media server or client. The algorithm and system of this invention are suitable for use in traffic management and roadside assistance scenarios, and are also suitable for individuals and small teams for other rotating small target detection tasks; this system will be rapidly portable and easy to use.

[0043] In the remote sensing target detection scenario, multimodal fusion technology is introduced. (1) To address the problem of difficulty in capturing vehicle information under low light conditions, a multimodal image fusion algorithm is introduced. This invention proposes an image fusion algorithm DetFusion based on detection task and an image fusion method based on salient target detection. This algorithm combines the target detection algorithm with the traditional CNN-based end-to-end image fusion method and introduces detection loss, so that the fusion algorithm contains more semantic information of advanced vision tasks and can obtain a fusion image that is more in line with the specific detection task scenario. At the same time, the proposed salient target detection image fusion method uses the target information of target detection to generate target salient masks and background masks and applies them to the loss function, so that the model can better distinguish between foreground and background, which is conducive to the performance improvement of advanced vision tasks. (2) For the detection problem of high-density targets and large aspect ratio, this invention proposes a rotation target detection method YOLOv5-Rotate based on YOLOv5, and at the same time proposes an improved YOLOv5 feature extraction network based on attention mechanism. This algorithm introduces angle information to detect targets with rotation angles and uses classification ideas and angle smoothing coding methods to solve the angle prediction bias problem. Rotational object detection requires less background information than horizontal object detection, reducing the learning difficulty for the network. By introducing spatial domain attention and channel-based attention mechanisms into the feature extraction network, the network focuses more on features that effectively improve detection performance. Experimental results show that the rotational object detection based on YOLOv5 exhibits a significant performance advantage over other rotational object detection algorithms on the DroneVeihcle dataset. The proposed attention improvement module also effectively enhances detection performance. Furthermore, among the proposed angle encoding methods, Gaussian encoding outperforms rectangular encoding.

[0044] Second, considering the technical solution as a whole or from a product perspective, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:

[0045] The remote sensing target detection method based on multimodal image fusion provided by this invention is based on deep learning methods in computer vision. By combining image fusion algorithms and rotation target detection algorithms, it can quickly and accurately detect vehicle position and category information under both strong light and low light conditions. It can be applied to rotation target detection algorithms in scenarios such as UAV traffic management and disaster relief.

[0046] Third, as supporting evidence of the inventiveness of this invention, it is also reflected in the following important aspects:

[0047] The technical solution of this invention fills a technical gap in the industry both domestically and internationally: (1) The original intention of task-oriented image fusion is to provide more beneficial inputs for subsequent applications. However, in many image fusion tasks, existing deep learning-based methods do not consider the correlation between fusion and subsequent applications when designing loss functions, which often leads to highly subjective fusion results. This invention considers introducing the accuracy of subsequent tasks into the loss function design in the fusion stage, guiding the fusion process from the decision-making level. (2) The rotating target detection algorithm introduces an angle classification method for angle prediction and Gaussian smoothing encoding labels, compared with traditional single-parameter angle regression based on Smooth L1 loss. The new angle classification idea makes the angle prediction more accurate. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the remote sensing target detection method provided in the embodiments of the present invention;

[0050] Figure 2 This is the overall structure of the image fusion method based on detection task-driven provided in the embodiments of the present invention;

[0051] Figure 3 This is a Sobel gradient residual block provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the salient target detection image fusion method provided by the present invention;

[0053] Figure 5 This is a fusion network structure design diagram provided in an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of Gaussian encoding provided in an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the rectangular encoding provided in an embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of boundary cycle coding provided in an embodiment of the present invention;

[0057] Figure 9 This is the YOLOv5 prediction method with introduced angle classification provided in the embodiments of the present invention;

[0058] Figure 10This is the CBAM attention structure provided in the embodiments of the present invention;

[0059] Figure 11 This is the improved YOLOv5C3 module provided in the embodiments of the present invention;

[0060] Figure 12 This is a structural diagram of the real-time video encoding and decoding system provided in an embodiment of the present invention.

[0061] Figure 13 This is a structural diagram of an edge computing hardware system provided in an embodiment of the present invention.

[0062] Figure 14 This is a structural diagram of the edge computing software system provided in an embodiment of the present invention.

[0063] Figure 15 This is an image fusion effect diagram provided by an embodiment of the present invention.

[0064] Figure 16 This is a diagram showing the target detection effect of angle regression provided in an embodiment of the present invention.

[0065] Figure 17 This is a diagram showing the effect of angle-classified target detection provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0067] To address the problems existing in the prior art, the present invention provides a remote sensing target detection method, system, medium, device, and terminal. The present invention will be described in detail below with reference to the accompanying drawings.

[0068] like Figure 1 As shown, the remote sensing target detection method provided in this embodiment of the invention includes the following steps:

[0069] S101, Construct an RGB-infrared image fusion network based on CNN;

[0070] S102 employs a task-oriented fusion method and a joint adaptive training strategy for low and high layers, utilizing a fusion network constrained by a joint loss consisting of content loss and semantic loss.

[0071] S103 introduces angle information and uses the five-parameter method to represent rectangles in any direction, performs feature extraction, and uses the BIFPN bidirectional feature fusion method for feature fusion.

[0072] S104 uses real-time acquired multimodal video streams for decoding, algorithm inference, and bounding box calibration, and encodes and pushes the results to a streaming media server or client.

[0073] As a preferred embodiment, the remote sensing target detection system provided by this invention includes an image fusion module, a detector module, and a video stream edge computing module.

[0074] The image fusion module designs an RGB-infrared image fusion network using a CNN method. It adopts a task-oriented fusion approach, a joint adaptive training strategy for low and high layers, and uses a joint loss consisting of content loss and semantic loss to constrain the fusion network. It also employs an image fusion method based on salient object detection.

[0075] The rotation detector module incorporates angle information using a five-parameter method. To represent a rectangle in any direction, within a certain range, we use classification task methods and cyclic label encoding methods to address the problem of angle prediction inaccuracy caused by sudden loss increases in rotation detectors based on regression angles. We also utilize the BIFPN bidirectional feature fusion method to solve the problem of small target detection.

[0076] The video stream edge computing module uses real-time acquired multimodal video streams to perform tasks such as decoding, algorithm inference, and bounding box calibration, and then encodes and pushes the results to the streaming media server or client.

[0077] The multimodal image fusion module provided in this embodiment of the invention includes:

[0078] (1) The infrared-visible light dataset with multimodal registration is used as the data fusion object;

[0079] (2) The fusion network consists of a feature extractor and an image reconstructor, which utilizes dual-stream feature extraction, gradient residual blocks and feature concatenation to integrate deep features and fine-grained detail features;

[0080] (3) A task-oriented fusion method is adopted, with a joint adaptive training strategy for low and high layers and a joint loss constraint fusion network composed of content loss and detection loss, so that the network contains more semantic information; the fine-grained features of infrared images and visible light images are fused through a cascade strategy, and the results are fed back to the image reconstructor to realize feature aggregation and image reconstruction, and the size of the fused image is consistent with that of the source image.

[0081] (4) Introducing a target mask to guide the network model in detecting salient regions, while ensuring the consistency of the intensity and gradient of the specified target region to achieve the fusion of target region saliency and background texture. Utilizing the bounding box representation in the target label of the infrared image. Use the center point coordinates and width and height to construct a target object mask. The background mask is a mask. .

[0082] (5) Design of loss function for fusion network, including content loss, which is composed of intensity loss. and texture loss It consists of two parts, with the intensity loss consisting of the target intensity and the background intensity.

[0083] The rotating target detector module provided in this embodiment of the invention includes:

[0084] (1) Five-parameter method Represents a rectangle in any direction, with a range of... ;

[0085] (2) Feature extraction was performed using the backbone network CSP-DarkNet53 in Yolov5;

[0086] (3) Feature fusion is performed using the Bi-FPN bidirectional feature fusion method;

[0087] (4) The Gaussian cyclic label coding method is used to solve the problem of angle prediction inaccuracy caused by sudden increase in loss in angle-based rotation detectors in other algorithms.

[0088] The video stream edge computing module provided in this embodiment of the invention includes:

[0089] (1) Real-time acquisition of RGB and infrared video from the perspective of UAV at high altitude.

[0090] (2) Asynchronous processing decodes the RGB video stream and infrared video stream, decodes the RGB-infrared image pair, and stores it in the first buffer queue (if it is YUV format, the image needs to be converted to RGB first).

[0091] (3) Asynchronously process the RGB-infrared image pairs extracted from the first buffer queue and perform algorithm inference for model reading. Obtain multi-layer detection results. Put the inferred RGB-infrared images and inference result pairs into the second buffer queue.

[0092] (4) Asynchronously process the RGB-infrared image pairs and inference results extracted from the second buffer queue, perform bounding box calibration, and put the calibrated image pairs into the third buffer queue.

[0093] (5) Asynchronously process the extraction of RGB-infrared image pairs from the third buffer queue. These are images that have already been inferred. Perform video encoding and push the stream to the streaming media or client.

[0094] To demonstrate the inventiveness and technical value of the present invention, this section provides specific product or related technology application examples of the technical solution.

[0095] This paper designs and implements an aerial target detection system for real-world target detection scenarios. Addressing the issue of low detection efficiency due to excessive background information in aerial photography, a rotational target detection method based on YOLOv5 is designed. To address issues such as missed detections in low light and occlusion, a visible-infrared image fusion algorithm driven by the detection task is designed and applied to the multimodal preprocessing stage of the algorithm. Furthermore, to further improve the detector's practicality, an edge computing system for aerial target detection is designed and implemented from the perspectives of aerial target detection scenarios and IoT systems.

[0096] The system hardware platform design uses a Jetson Nano device as the core of the edge computing system, a multimodal camera from a drone for scene assistance, and displays and other devices for result visualization. Through communication between the drone ground station and the edge computing device, video streams are received from the network and output, thus establishing a rotating target detection edge computing system. Its hardware process design and construction are as follows: Figure 13 As shown, multimodal video data is collected by drones, and the collected video data is processed by Jetson Nano through a ground station to obtain a new video stream. Finally, the network video stream is transmitted to a cloud platform or streaming media server through a base station for monitoring through a display device.

[0097] The software platform design addresses the efficient operation of the improved YOLOv5 rotating target detection algorithm proposed in this invention. It also considers practical engineering applications, optimizing the target detection system's workflow and real-time performance. The overall design of the video stream edge computing system is based on the embedded Jetson Nano platform. Figure 14As shown, the structure design can be divided into six sub-tasks: FFmpeg video stream decoding, image fusion, object detection, OpenCV object rendering, FFmpeg video stream encoding, and cloud platform / streaming media server communication. FFmpeg video stream decoding typically decodes H264 / H265 encoded video streams into multiple frames of raw YUV images, while simultaneously converting them into RGB images for subsequent algorithm inference. The multimodal image fusion algorithm uses the detection task-driven image fusion algorithm proposed in this invention. As a preprocessing stage for the full-scene object detection algorithm, the image fusion task must be lightweight and real-time. The rotation detection algorithm uses the YOLOv5-based rotation object detection algorithm proposed in this invention, and employs a lighter YOLOv5n network to reduce latency during inference on edge devices, thus meeting the real-time requirements of the entire system. The OpenCV object rendering task renders the object bounding boxes and category labels in the fused image based on the detector's detection results. The FFmpeg video stream encoding task encodes the rendered images frame by frame into new H264 / H265 video streams and pushes them to a cloud platform or streaming media server for other systems to pull. The WebSocket data communication task transmits the statistical data from the detection results to the system for display and other computing tasks.

[0098] The overall design of image fusion and rotating target detection provided by this invention is shown below. Figure 2 It consists of two parts: an RGB-infrared image fusion network and a rotating target detection network. It adopts a task-oriented fusion method, a joint adaptive training strategy for low and high layers, and a joint loss constraint fusion network composed of content loss and semantic loss.

[0099] (1) Design a lightweight fusion network based on gradient residual blocks, see Figures 3-5 The fusion network consists of a feature extractor and an image reconstructor. The feature extractor contains two GRDBs to extract fine-grained features. The feature extractor involves two parallel infrared and visible light feature extraction streams, each containing a common convolutional layer and two GRDBs. Shallow features are extracted using a 3×3 kernel and LeakReLU activation function. Next are two gradient residual blocks for extracting fine-grained features from the shallow features. The gradient residual blocks are a variant of Resblock, where the main stream employs dense connections, and the residual streams integrate gradient operations. The main stream deploys two 3×3 convolutional layers with LReLU and a common convolutional layer with a 1×1 kernel. Dense connections are introduced into the main stream to fully utilize the features extracted by each convolutional layer. The residual streams use gradient operations to compute the gradient magnitude of the features and employ 1×1 regular convolutional layers to eliminate channel dimension differences; element-wise addition is added to the outputs of the main dense stream and the residual gradient streams to integrate deep features and fine-grained detail features.

[0100] Fine-grained features from infrared and visible light images are fused using a cascaded strategy, and the result is fed back into an image reconstructor to achieve feature aggregation and image reconstruction. The image reconstructor consists of three cascaded 3×3 convolutional layers and one 1×1 convolutional layer. Information loss is a catastrophic problem in image fusion tasks. Therefore, the padding settings in the fusion network are identical, and the stride is set to 1. Furthermore, the network does not introduce any downsampling, and the size of the fused image is consistent with the source image.

[0101] (2) Loss function design: The quality of the fused image largely depends on the loss function. Mainstream loss function terms include intensity loss, texture loss, SSIM loss, and perceptual loss, and their weight ratios determine the trend of information fusion. To improve visual quality and quantitative indicators, a content loss is designed, consisting of two parts: intensity loss... and texture loss The definition is as follows:

[0102]

[0103] In the formula, Constrain the overall visual intensity of the fused image. Forced fusion of images contains more texture details. This is used to balance intensity loss and texture loss. Since pixel intensity loss measures the difference between the fused image and the source image at the pixel level, it is desirable for the target region to have a more significant contrast with the background region. This requires preserving the maximum pixel intensity in the target region and ensuring that the background region has a lower maximum pixel intensity. To identify the difference between the target object and the background, the bounding box representation of the target label in the infrared image is utilized. Use the center point coordinates and width and height to construct a target object mask. The background mask is a mask. Based on the above analysis and design, the pixel intensity loss of infrared and visible light images is defined as:

[0104]

[0105]

[0106]

[0107] In the formula, H and W are the height and width of the image, respectively, ‖⋅‖ represents the l1-norm, and the maximum value (⋅) represents the maximum selection of elements. The pixel intensity distributions of infrared and visible light images are integrated using a maximum selection strategy. It is expected that the fused image will maintain the optimal intensity distribution while preserving rich texture details in the source images. However, the intensity loss only provides coarse-grained distribution constraints for model learning. Therefore, a texture loss is introduced to force the fused image to include finer-grained texture information. The texture loss is defined as:

[0108]

[0109] In the formula, ∇ represents the Sobel gradient operator, used to measure fine-grained texture information of the image. Here, it is assumed that the optimal texture of the fused image is the maximum aggregation of the textures from the infrared and visible light images.

[0110] (3) To address the problems existing in rotating target detection, an improved YOLOv5 target detection algorithm is proposed based on the YOLOv5 target detection algorithm. A five-parameter method is employed. To represent a rectangle in any direction, within the range of This paper utilizes the BIFPN bidirectional feature fusion method to address the small object detection problem. A cyclic smooth label encoding (CSL) method is incorporated to resolve the issue of sudden loss spikes leading to inaccurate angle predictions in rotation detectors based on regression angles. CSL achieves more robust angle predictions through classification, unaffected by boundary conditions (including EoE and PoA). CSL involves periodic cyclic label encoding, where the specified label values ​​are smooth and have a certain tolerance. The expression for CSL is as follows:

[0111]

[0112] In the formula, It is a window function. r is the radius of the window function. This represents the angle of the current bounding box. An ideal window function g(x) needs to maintain periodicity and symmetry. Monotonicity. Generally, Gaussian functions, impulse functions, rectangular functions, and trigonometric functions are sufficient. For example... Figure 6 , 7 8. Therefore, the total loss after introducing CSL is:

[0113]

[0114] (4) To reduce the adverse effects of background noise from aerial perspectives and improve network inference speed, a multi-dimensional attention mechanism is introduced into the YOLOv5 network. The CBAM multi-dimensional attention mechanism is used to improve the YOLOv5 backbone network, New CSPDarkNet53. CBAM combines channel domain attention and spatial domain attention mechanisms, achieving better results compared to attention mechanisms that only focus on the channel domain or only on the spatial domain. Its structure is as follows... Figure 10 As shown, CBAM processes the input feature layers using both channel attention and spatial attention mechanisms. Specifically, in the CSPDarkNet53 network structure, the C3 module is chosen for attention. This is primarily because the C3 module is the most crucial module in the feature extraction process; introducing attention enhances the network's ability to extract key information about the target features and reduces background noise. The C3 module structures before and after the improvement are shown below. Figure 11 As shown, the original C3 module extracts features through two paths and three convolutional layers (ConvBNSiLU). The first convolutional layer is concatenated with the second convolutional layer using a BottleNeck residual structure, and then a final convolution operation is performed. An attention mechanism (CMAM) is added after this C3 module to extract key information in both the channel and spatial dimensions from the features extracted by C3.

[0115] (5) Prototype System Design. Based on embedded devices such as Jetson Nano, the original UAV multimodal real-time video stream is decoded into RGB-infrared images using ffmpeg technology. OpenCV is used for algorithm reading and bounding box calibration, and the stream is pushed to a streaming media server in real time. The detection performance of the multimodal image fusion detector in low-light scenes is verified. Verification is performed in a real environment through porting to embedded devices. The structure of the real-time video encoding and decoding system provided in this embodiment is as follows: Figure 12 As shown.

[0116] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0117] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

[0118] Qualitative comparison of different algorithms on the DroneVehicle dataset, such as Figure 15 As shown. By comparing with IFCNN, STDFusionNet, and DetFusion proposed in this invention, it can be seen that IFCNN contains less texture information of the visible light image after fusion compared to DetFusion. This reflects that the Sobel gradient operator introduces more texture information during feature extraction. In the image fused by STDFusionNet, false target ghosts in the infrared image are magnified, and there is too much background information, which weakens the salience of the target to a certain extent. DetFusion, on the other hand, can effectively distinguish between prominent targets and weaken the background, and fully supplement the intensity information of the infrared image into the visible light image.

[0119] A set of images with perpendicular target angles was selected from the DroneVehicle test set for qualitative analysis, such as... Figure 16 , 17 The image shows the detection results of different methods applied to the same set of images. Figure 16 The figure shows the results of angle regression using Smooth L1 loss on the DroneVehicle dataset. The figure reveals poor performance in detecting targets at vertical angles, the root cause of which is the boundary issue discussed earlier. Figure 17This demonstrates the effect of using angle classification and Gaussian function for label encoding, followed by boundary smoothing. Visually, the improved YOLOv5 rotation detection method based on this invention shows a significant improvement over regression methods at angle boundaries.

Claims

1. A remote sensing target detection method, characterized in that, Remote sensing target detection methods include the following steps: Step 1, Multimodal Image Fusion: Determine the data fusion object and construct a fusion network. Use joint loss to constrain the fusion network, use the target information from target detection to construct target and background masks to improve target saliency, and use an image reconstructor to achieve feature aggregation and image reconstruction. Step 2, Rotation Target Detection: The rectangle is represented by a five-parameter method, CSP-DarkNet53 is used for feature extraction, and Bi-FPN bidirectional feature fusion is used for feature fusion. Guass encoding is used to handle the rotation angle problem. Step 3, video stream edge computing: Real-time acquisition of RGB infrared video stream and infrared video stream, followed by three asynchronous processing steps, and finally video encoding and streaming to streaming media or client; The multimodal image fusion in step one includes: (1) The infrared-visible light dataset with multimodal registration is used as the data fusion object; (2) The fusion network consists of a feature extractor and an image reconstructor, which utilizes dual-stream feature extraction, gradient residual blocks and feature concatenation to integrate deep features and fine-grained detail features; (3) A task-oriented fusion method is adopted, with a joint adaptive training strategy for low and high layers and a joint loss constraint fusion network composed of content loss and detection loss, so that the network contains more semantic information; the fine-grained features of infrared images and visible light images are fused through a cascade strategy, and the results are fed back to the image reconstructor to realize feature aggregation and image reconstruction, and the size of the fused image is consistent with that of the source image. (4) Introducing a target mask to guide the network model in detecting salient regions, while ensuring the consistency of the intensity and gradient of the specified target region to achieve the fusion of target region saliency and background texture. Utilizing the bounding box representation in the target label of the infrared image. Use the center point coordinates and width and height to construct a target object mask. The background mask is a mask. ; (5) Design of loss function for fusion network, including content loss, which is composed of intensity loss. and texture loss It consists of two parts, where the intensity loss is composed of the target intensity and the background intensity; Step 3, video stream edge computation, includes: (1) Real-time acquisition of RGB and infrared video from the high-altitude perspective of the UAV; (2) Asynchronous processing decodes the RGB video stream and the infrared video stream to decode the RGB-infrared image pair and stores it in the first buffer queue; if it is in YUV format, it is converted into RGB. (3) Extract the RGB-infrared image pairs from the first buffer queue, perform algorithm inference for model reading, obtain multi-layer detection results, and put the RGB-infrared image and inference result pairs into the second buffer queue; (4) Extract the dRGB-infrared image pairs and inference results from the second buffer queue, perform bounding box calibration, and put the calibrated image pairs into the third buffer queue; (5) Extract the RGB-infrared image pairs from the third buffer queue to obtain the inferred image, and finally encode the video and push it to the streaming media or client.

2. The remote sensing target detection method as described in claim 1, characterized in that, Step two, rotating target detection, includes: (1) Five-parameter method Represents a rectangle in any direction, with a range of... ; (2) Feature extraction was performed using the backbone network CSP-DarkNet53 in Yolov5; (3) Feature fusion is performed using the Bi-FPN bidirectional feature fusion method; (4) The Gaussian cyclic label encoding method is used to detect target information; (5) The CBAM attention mechanism is used to enhance network performance in the channel and spatial domains.

3. A remote sensing target detection system applying the remote sensing target detection method as described in any one of claims 1 to 2, characterized in that, Remote sensing target detection systems include: The image fusion module is used to design an RGB-infrared image fusion network using CNN methods. It adopts a task-oriented fusion method, a joint adaptive training strategy for low and high layers, and utilizes a joint loss consisting of content loss and semantic loss to constrain the fusion network. The rotation detector module is used to introduce angle information. It uses a five-parameter method to represent rectangles in arbitrary directions, extracts features using CSP-DarkNet53, and performs feature fusion using the BIFPN bidirectional feature fusion method. The video stream edge computing module is used to perform decoding, algorithm inference, and bounding box calibration tasks using real-time acquired multimodal video streams, and encode and push the results to the streaming media server or client.

4. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the remote sensing target detection method as described in any one of claims 1 to 2.

5. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the remote sensing target detection method as described in any one of claims 1 to 2.

6. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the remote sensing target detection system as described in claim 3.

Images