A Smart Target Detection Method Based on Multi-Source Heterogeneous Information Fusion

Abstract

This invention proposes an intelligent target detection method based on multi-source heterogeneous information fusion. This method achieves complementary extraction of high-level semantic features from dual-modal infrared and visible light images through a dual-adversarial learning fusion module. Then, it adaptively enhances the fusion of visual texture features and infrared thermal target features under extreme lighting conditions through a dual-adversarial learning fusion module. Finally, it combines a lightweight feature extraction network and an edge dynamic scheduling mechanism to achieve bidirectional optimization of detection accuracy and inference speed. This results in efficient, accurate, and robust all-weather real-time vehicle-mounted target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology based on multi-source heterogeneous data (optical + infrared), and in particular to an intelligent target detection method based on multi-source heterogeneous information fusion. Background Technology

[0002] Intelligent driving is a core direction for the transformation and upgrading of the modern automotive industry, and a key technological support for ensuring road traffic safety and improving the driving experience. Since entering the era of intelligent driving, with the iterative upgrades of in-vehicle perception hardware and artificial intelligence technology, accurate perception capabilities in all weather and all scenarios have become a core prerequisite for the practical application of autonomous driving. As the perception input end of the intelligent driving system, the stability and reliability of in-vehicle visual perception technology directly determine the operating effect of the intelligent driving system. It is also deeply intertwined with the innovative development of embedded artificial intelligence and multimodal data fusion technology, possessing profound engineering value and industrial significance for vehicle driving safety and the large-scale implementation of the intelligent driving industry.

[0003] In recent years, multi-source heterogeneous data fusion technology has made significant progress in the field of target detection. However, vehicle-mounted visual inspection methods still face numerous technical bottlenecks and application challenges in perceiving pedestrians and vehicles. First, the dynamic changes in natural lighting and the complex characteristics of the driving environment can cause exposure lag and loss of detail in visible light imaging equipment, significantly increasing driving safety risks. Second, extreme lighting scenarios are characterized by suddenness, transience, and high interference. Conventional perception modes struggle to quickly adapt to sudden changes in lighting conditions, leading to a significant decrease in the stability and environmental adaptability of visual inspection models, failing to meet the safety perception requirements of high-speed driving. Therefore, how to achieve accurate perception of pedestrians and vehicles under extreme lighting conditions remains a key issue worthy of further research. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the prior art and to propose an intelligent target detection method based on multi-source heterogeneous information fusion.

[0005] This invention is achieved through the following technical solution: This invention proposes an intelligent target detection method based on multi-source heterogeneous information fusion, the method comprising: Step 1: Multi-source data acquisition is completed through a dual-modal acquisition unit. The image stream is acquired in parallel by a visible light camera and an infrared thermal imaging camera. The original images are processed for distortion correction, size normalization and timestamp alignment to provide high-quality raw data support for subsequent semantic fusion and target detection. Step 2: Preprocess and semantically fuse the bimodal image data to complete the input of the image denoising, detail enhancement and scale scaling adaptation network. Introduce a bi-adversarial learning fusion architecture to generate a high-semantic-quality fusion feature map. Step 3: Construct a lightweight detection model to complete target feature parsing and detection inference. Based on the YOLOv5-tiny architecture and paired with the Dense Block backbone extraction network, the model significantly reduces the computational load while retaining core detection features, thus adapting to the inference needs of edge platforms. Step 4: Relying on edge deployment optimization and multi-task scheduling mechanism to achieve intelligent output and human-computer interaction, and based on the Jetson Nano platform combined with the TensorRT engine to accelerate model inference, realize high-precision real-time detection and result feedback of vehicle and pedestrian targets in vehicle scenarios.

[0006] Furthermore, in step 2, let the infrared image input be... Visible light image input is , For parameters Fusion generative network, For parameters Lightweight detection network, To obtain the optimal fused image, a two-layer optimization model is constructed for the fusion and detection tasks: the upper layer aims to minimize the detection loss, while the lower layer constrains the fusion output to be the optimal solution. The formula is as follows:

[0007] in Representing the target detection loss, this modeling approach allows the optimization direction of the fusion network to fully serve the downstream detection task.

[0008] Furthermore, in step 2, the fusion generator uses a multi-layered densely connected module as the feature extraction backbone and receives data in parallel. and Feature encoding completed; target discriminator set up in network. and detail discriminator Dual-branch supervision, the target discriminator extracts the true foreground from the original infrared image. Positive samples, fusion prospects Using pseudo-samples, relying on adversarial loss The constraint generator preserves the outline of the infrared thermal target; the detail discriminator performs gradient calculations on the visible light image. Extract background details as real samples to fuse graph gradients. As a pseudo-sample, relying on The constraint generator inherits visible light texture details; the total fusion loss is a weighted average of structural loss, pixel loss, and bi-branch adversarial loss.

[0009] in: Achieve pixel-level weighted fidelity. Ensure the fusion graph structure and original Figure 1 To, , For hyperparameter weights.

[0010] Furthermore, in step 2, a joint iterative optimization strategy of forward propagation and backpropagation is selected, and the detection loss is used to update the parameters of the fusion network through chain-like differentiation. and detection parameters :

[0011] in, The weights for the fusion loss are used; the optimal fusion graph is obtained after collaborative training. The target is fed into a lightweight detection head, and the final output is the target category and the orientation bounding box. .

[0012] Furthermore, in step 3, YOLOv5 adopts an anchor-free architecture, eliminating the predefined anchor box design and directly completing the target center localization and bounding box offset regression on the feature map; for each feature point, the network outputs the class confidence score. and bounding box parameters The following weighted combined loss function is used during the training phase:

[0013] in: For classification cross-entropy loss; The cross-union loss is the ratio of the predicted bounding box to the ground truth bounding box. The target confidence loss is used to filter out false positives in the background. The weighting hyperparameters are used to assign weights to various losses.

[0014] Furthermore, in step 4, a localized edge inference link is built based on the Jetson Nano hardware platform, coupled with a closed-loop feedback control architecture, to complete the data parsing, target determination and task response of multi-source images in the embedded terminal, thereby reducing the transmission latency and overall power consumption caused by remote data interaction.

[0015] Furthermore, in step 4, the system deploys a lightweight neural network that has undergone pruning and quantization compression to detect and classify vehicle and pedestrian targets within the road scene; let the single-frame input image be... The set of model parameters after training convergence is Then the object detection task is represented as:

[0016] in: for The target category prediction output at each time step; For all pedestrian and vehicle target categories; Output the posterior probability of each category for the model; optimize the network computation graph using TensorRT to achieve millisecond-level response.

[0017] Furthermore, in step 4, to achieve balanced resource allocation for the parallel multi-target detection tasks of pedestrians and vehicles, a resource-driven dynamic feedback scheduling strategy is introduced; the real-time resource load index of the edge hardware is denoted as... Using a constructed scheduling function Dynamically allocate task priority and hardware resource usage:

[0018] in: The resource weighting coefficients corresponding to the current CPU and memory load; This represents the current output data frame queue depth of the inference model. The corresponding resource weight coefficient satisfies ;like , The system automatically reduces the image acquisition frequency or switches to lightweight mode when the system load threshold is reached.

[0019] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the intelligent target detection method based on multi-source heterogeneous information fusion.

[0020] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the intelligent target detection method based on multi-source heterogeneous information fusion.

[0021] The beneficial effects of this invention are: This invention proposes an intelligent target detection method based on multi-source heterogeneous information fusion, relying on the Jetson Nano edge platform to achieve multimodal image fusion and real-time target detection. To address the issues of insufficient adaptability to single visible light scenes and the difficulty in balancing target detection accuracy and real-time edge detection, this invention adopts a three-level closed-loop architecture of perception, processing, and interaction. Combined with a modular design approach, it fully leverages the complementary advantages of infrared image thermal target features and visible light image texture features through dual-modal data acquisition, semantic-level multimodal fusion, lightweight target detection, and edge visualization interaction. Simultaneously, relying on lightweight model optimization and a multi-task scheduling mechanism, it balances detection accuracy, inference speed, and scene adaptability, effectively improving the robustness and practicality of pedestrian and vehicle target detection in complex vehicular road conditions. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 This is a flowchart of an intelligent target detection method based on multi-source heterogeneous information fusion according to the present invention; Figure 2 This is a diagram illustrating the overall structural framework of a dual-adversarial learning fusion network. Figure 3 This is a diagram showing the internal structure of the generator and discriminator; Figure 4 This is a schematic diagram of a lightweight detection head; Figure 5 This is a UI diagram for the software. Figure 6 This is a schematic diagram of the detection results in the example. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The purpose of this invention is to solve the problem of intelligent target detection in complex vehicular scenarios under all weather conditions. It utilizes a lightweight deep learning network to efficiently and accurately detect vehicles and pedestrians in vehicular scenarios, outputting corresponding target location information and detection results. To achieve this goal, this invention provides an intelligent target detection method based on multi-source heterogeneous information fusion, relying on the Jetson Nano edge platform to realize multimodal image fusion and real-time target detection.

[0026] Specifically, in combination Figures 1-6 This invention proposes an intelligent target detection method based on multi-source heterogeneous information fusion, the method comprising: Step 1: Multi-source data acquisition is completed through a dual-modal acquisition unit. The image stream is acquired in parallel by a visible light camera and an infrared thermal imaging camera. The original images are processed for distortion correction, size normalization and timestamp alignment to provide high-quality raw data support for subsequent semantic fusion and target detection. Step 2: Preprocess and semantically fuse the dual-modal image data to complete image denoising, detail enhancement and scale scaling adaptation network input. Introduce a dual adversarial learning fusion architecture to solve problems such as artifact interference, target blurring and lack of detail in traditional fusion methods, and generate a high-semantic-quality fusion feature map. Step 3: Construct a lightweight detection model to complete target feature parsing and detection inference. Based on the YOLOv5-tiny architecture and paired with the Dense Block backbone extraction network, the model significantly reduces the computational load while retaining core detection features, thus adapting to the inference needs of edge platforms. Step 4: Relying on edge deployment optimization and multi-task scheduling mechanism to achieve intelligent output and human-computer interaction, and based on the Jetson Nano platform combined with the TensorRT engine to accelerate model inference, realize high-precision real-time detection and result feedback of vehicle and pedestrian targets in vehicle scenarios.

[0027] Optionally, step 1 adopts a dual-camera parallel acquisition mechanism, coupled with a stable image reading algorithm, to effectively avoid image frame loss and stuttering issues in vehicle motion scenarios, meet the requirements of long-term continuous acquisition, and adapt to data acquisition scenarios in complex driving environments.

[0028] Optionally, the dual-adversarial semantic fusion architecture in step 2 completes the deep alignment of multi-source heterogeneous data from the feature layer, abandons the limitations of traditional pixel-level fusion, accurately preserves the core effective features of dual-modal images, and achieves adaptive optimization of image features under different lighting and weather conditions.

[0029] Optionally, step 3 uses a series of model lightweighting optimization strategies to remove low-contribution parameters and redundant channels in the network, adapt to INT8 quantization accuracy, compress the model size with almost no loss of detection accuracy, and significantly improve the inference efficiency of edge devices.

[0030] Optionally, step 4 adopts a task priority allocation mechanism, setting the core task of detection and inference as the highest scheduling priority to avoid delay and frame drop issues caused by multi-task resource contention. At the same time, the visualization interface supports adjustable parameters such as confidence, IOU, and target size, and can flexibly optimize the detection strategy according to real-time traffic conditions.

[0031] Example This invention proposes an intelligent target detection method based on multi-source heterogeneous information fusion, the method comprising: Step 1: Multi-source data acquisition is completed through a dual-modal acquisition unit. The image stream is acquired in parallel by a visible light camera and an infrared thermal imaging camera. The original images are processed for distortion correction, size normalization and timestamp alignment to provide high-quality raw data support for subsequent semantic fusion and target detection. Step 1 involves multi-source data acquisition using a dual-modal acquisition unit. This unit utilizes a visible light camera and an infrared thermal imaging camera to acquire image streams in parallel, obtaining dual-channel raw image data for all-weather scenarios. To address issues such as image distortion, scale inconsistency, temporal misalignment, and frame drops / stuttering in vehicle-mounted motion scenarios, the acquired raw images undergo distortion correction, size normalization, and timestamp alignment processes sequentially. This eliminates data noise and errors introduced by hardware acquisition and motion conditions, ensuring temporal and spatial consistency between visible light texture information and infrared thermal target information. This provides high-quality, highly stable raw data support for subsequent semantic fusion and target detection.

[0032] Step 2: Preprocess and semantically fuse the dual-modal image data to complete image denoising, detail enhancement and scale scaling adaptation network input. Introduce a dual adversarial learning fusion architecture to solve problems such as artifact interference, target blurring and lack of detail in traditional fusion methods, and generate a high-semantic-quality fusion feature map. Step 2 constructs a fusion network based on target-aware dual-adversarial learning. Using two-layer optimization as the theoretical foundation, it achieves end-to-end feature fusion and target enhancement for detection tasks. The specific network architecture is as follows: Figure 2 As shown.

[0033] Let the infrared image input be Visible light image input is , For parameters Fusion generative network, For parameters Lightweight detection network, To obtain the optimal fused image, a two-layer optimization model is constructed, combining the fusion and detection tasks: the upper layer aims to minimize the detection loss, while the lower layer constrains the fusion output to be the optimal solution. The formula is as follows:

[0034] in Representing the target detection loss, this modeling approach allows the optimization direction of the fusion network to fully serve the downstream detection task, which is different from traditional fusion algorithms that only optimize the visual effects of images.

[0035] The fusion generator uses a multi-layered densely connected module as the feature extraction backbone and receives data in parallel. and Feature encoding is completed. The network is configured with a target discriminator. and detail discriminator Dual-branch supervision, the target discriminator extracts the true foreground from the original infrared image. Positive samples, fusion prospects Using pseudo-samples, relying on adversarial loss The constraint generator preserves the outline of the infrared thermal target; the detail discriminator performs gradient calculations on the visible light image. Extract background details as real samples to fuse graph gradients. As a pseudo-sample, relying on The constraint generator inherits visible light texture details. The total fusion loss is a weighted average of structural loss, pixel loss, and bi-branch adversarial loss.

[0036] in: Achieve pixel-level weighted fidelity. Ensure the fusion graph structure and original Figure 1 To, , For hyperparameter weights.

[0037] A joint iterative optimization strategy combining forward and backward propagation is adopted, and the detection loss is used to update the parameters of the fusion network through chained differentiation. and detection parameters :

[0038] in, These are the weighting coefficients for the fusion loss. The optimal fusion graph after co-training. The target is fed into a lightweight detection head, and the final output is the target category and the orientation bounding box. .

[0039] Step 3: Construct a lightweight detection model to complete target feature parsing and detection inference. Based on the YOLOv5-tiny architecture and paired with the Dense Block backbone extraction network, the model significantly reduces the computational load while retaining core detection features, thus adapting to the inference needs of edge platforms. Step 3 uses a lightweight YOLOv5-Tiny network as the backbone architecture for object detection, and combines it with the MobileNetV3 module to optimize the front-end feature extractor, thereby improving the inference efficiency of edge devices while ensuring detection accuracy.

[0040] For multimodal input scenarios, the system performs dual-branch feature extraction on visible light and infrared images respectively, and achieves unified representation of intermediate semantics through a shared fusion module, thereby enhancing the model's robustness in detecting low-contrast, partially occluded, and blurred targets.

[0041] YOLOv5 employs an anchor-free architecture, eliminating the need for predefined anchor boxes and directly performing target center localization and bounding box offset regression on the feature map. For each feature point, the network outputs a class confidence score. and bounding box parameters The following weighted combined loss function is used during the training phase:

[0042] in: Cross Entropy is used for classification. The cross-union loss is the ratio of the predicted bounding box to the ground truth bounding box. The target confidence loss is used to filter out false positives in the background. The weighting hyperparameters are used to assign weights to each loss.

[0043] Step 4: Relying on edge deployment optimization and multi-task scheduling mechanism to achieve intelligent output and human-computer interaction, and based on the Jetson Nano platform combined with the TensorRT engine to accelerate model inference, realize high-precision real-time detection and result feedback of vehicle and pedestrian targets in vehicle scenarios.

[0044] Step 4 relies on the Jetson Nano hardware platform to build a localized edge inference link, coupled with a closed-loop feedback control architecture, to complete data parsing, target determination and task response of multi-source images in the embedded terminal, reducing the transmission latency and power consumption of the whole machine caused by remote data interaction.

[0045] The system deploys a lightweight neural network that has undergone pruning and quantization compression to detect and classify vehicles and pedestrians in road scenes. Let a single frame input image be... The set of parameters for the model after training convergence is The object detection task can then be represented as:

[0046] in: for The target category prediction output at each time step; For all pedestrian and vehicle target categories; Output the posterior probability of each category for the model; optimize the network computation graph using TensorRT to achieve millisecond-level response.

[0047] To achieve balanced resource allocation for parallel multi-target detection tasks involving pedestrians and vehicles, the system introduces a resource-driven dynamic feedback scheduling strategy. Let the real-time resource load index of the edge hardware be denoted as... Using a constructed scheduling function Dynamically allocate task priority and hardware resource usage:

[0048] in: The resource weighting coefficients corresponding to the current CPU and memory load; This represents the current output data frame queue depth of the inference model. The corresponding resource weight coefficient satisfies ;like , The system automatically reduces the image acquisition frequency or switches to lightweight mode when the system load threshold is reached.

[0049] The software architecture follows the design principles of modularity, lightweight design, and real-time priority. It is divided into four major functional modules: data acquisition layer, perception and inference layer, behavior analysis layer, and system control and feedback layer. Each module works collaboratively as an independent process, and completes efficient and stable cross-module data interaction through a shared cache and asynchronous message queue.

[0050] The system runs on the NVIDIA Jetson Nano embedded platform, and the development environment is as follows: Operating System: Ubuntu 18.04 + Jetpack SDK 4.6 Programming languages: Python 3.8+, C++ Deep learning framework: PyTorch 1.13 + TensorRT 8.4 (model acceleration) Edge Inference Optimization: ONNX+TensorRT Conversion and Deployment Visual Interface: The interface control and status feedback are implemented using PyQt5, with the specific structure as follows... Figure 5 As shown.

[0051] This architecture enables rapid access and asynchronous processing of multi-source data, while ensuring system robustness and good scalability. The designed infrared and visible light image fusion pedestrian and vehicle monitoring software adopts a modular layout with partitioned interfaces, divided into three main functional areas: a parameter adjustment area on the left, a multi-source image visualization area in the middle, and a statistical data display area at the bottom. The interactive logic design is completed around the requirements of multimodal image fusion and target detection, taking into account algorithm adjustability, intuitive results, and real-time data statistical capabilities.

[0052] This invention proposes an intelligent target detection method based on multi-source heterogeneous information fusion for all-weather pedestrian and vehicle target detection in complex vehicle-mounted scenarios. This method achieves complementary extraction of high-level semantic features from dual-modal infrared and visible light images through a dual-modal acquisition structure. Then, an adversarial learning fusion module is used to adaptively enhance the fusion of visual texture features and infrared thermal target features under extreme lighting conditions. Finally, a lightweight feature extraction network and an edge dynamic scheduling mechanism are combined to achieve bidirectional optimization of detection accuracy and inference speed. This results in efficient, accurate, and robust real-time all-weather vehicle-mounted target detection.

[0053] This invention selects Jetson Nano B01 as the edge computing core. Its integrated circuit integrates multiple power supply, communication, image acquisition and expansion peripheral interfaces. The functions and implementation methods of each interface circuit are described below.

[0054] In terms of power supply circuitry, the Jetson Nano B01 integrates dual-power supply hardware: the first is a standard DC power jack, compatible with a 5V / 4A external power adapter, capable of handling high-load computing demands in fused networks and target detection inference; the second is a MicroUSB power supply circuit, suitable for low-power, light-load debugging scenarios. The two power supply paths are switched via onboard jumpers, allowing users to flexibly select the appropriate power path based on the overall system load requirements.

[0055] In terms of wired and wireless communication circuits, the development board integrates gigabit Ethernet hardware circuitry, which can provide high-speed wired data transmission links to meet the big data interaction needs such as remote uploading of large-volume deep learning models and remote AI debugging and control of devices. At the same time, the B01 hardware adds a new M.2 E-Key slot with matching circuitry, which can be plugged into WiFi and Bluetooth communication modules to expand the device's wireless data transmission and reception capabilities and wireless networking capabilities, enriching the system's communication and interaction methods.

[0056] In terms of USB peripheral transmission circuitry, the Jetson Nano B01 integrates multiple USB hardware transmission circuits: four USB 2.0 Type-A interface circuits for low-speed general-purpose peripherals such as keyboards, mice, and USB flash drives; the motherboard is equipped with two USB 3.0 Type-A high-speed transmission circuits, which have high bandwidth data throughput capabilities and can connect to high-definition visible light cameras and high-speed storage devices, achieving a hardware performance upgrade compared to the A01 version.

[0057] In terms of image acquisition circuitry, the development board integrates two MIPI CSI camera interface circuits, supporting synchronous acquisition and transmission of dual image sensors. At the hardware level, it is compatible with parallel input from dual visible light and infrared thermal imaging cameras, providing underlying hardware support for the synchronous acquisition of multi-source heterogeneous images and multi-modal fusion sensing in this invention.

[0058] In addition, the Jetson Nano B01 integrates a 40-pin GPIO expansion hardware circuit, is electrically compatible with the Raspberry Pi peripheral interface specification, and the hardware circuit brings out multiple communication buses such as PWM, I2C, SPI, and UART, which can connect to external peripheral devices such as environmental sensors, actuators, status indicator LEDs, and OLED interactive displays, greatly expanding the system's hardware expansion capabilities and adaptability to complex automotive environments.

[0059] It should be understood that the development board integrated circuit described in this invention is not limited to the Jetson Nano's supporting circuit structure. Any embedded edge computing integrated circuit with multi-channel image acquisition, multi-mode power supply, and multi-bus peripheral expansion can replace and implement the underlying hardware functions of this system. This invention does not limit it to a single type. In the actual hardware implementation process, all or part of the various circuit modules described in this invention can be implemented through dedicated hardware circuits, integrated chips, peripheral modules, or any combination thereof. The hardware paths required for sensor acquisition, edge inference, and human-computer interaction all rely on the aforementioned integrated circuits to complete signal transmission, data interaction, and power supply. The various circuit modules work together to support the stable operation of the core system processes such as dual-modal image acquisition, lightweight network inference, and real-time parameter control.

[0060] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the intelligent target detection method based on multi-source heterogeneous information fusion.

[0061] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the intelligent target detection method based on multi-source heterogeneous information fusion.

[0062] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0063] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0064] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0065] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as execution by a hardware decoding processor, or as a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0066] The above provides a detailed description of the intelligent target detection method based on multi-source heterogeneous information fusion proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for intelligent target detection based on multi-source heterogeneous information fusion, characterized in that, The method includes: Step 1: Multi-source data acquisition is completed through a dual-modal acquisition unit. The image stream is acquired in parallel by a visible light camera and an infrared thermal imaging camera. The original images are processed for distortion correction, size normalization and timestamp alignment to provide high-quality raw data support for subsequent semantic fusion and target detection. Step 2: Preprocess and semantically fuse the bimodal image data to complete the input of the image denoising, detail enhancement and scale scaling adaptation network. Introduce a bi-adversarial learning fusion architecture to generate a high-semantic-quality fusion feature map. Step 3: Construct a lightweight detection model to complete target feature parsing and detection inference. Based on the YOLOv5-tiny architecture and paired with the Dense Block backbone extraction network, the model significantly reduces the computational load while retaining core detection features, thus adapting to the inference needs of edge platforms. Step 4: Relying on edge deployment optimization and multi-task scheduling mechanism to achieve intelligent output and human-computer interaction, and based on the JetsonNano platform combined with the TensorRT engine to accelerate model inference, realize high-precision real-time detection and result feedback of vehicle and pedestrian targets in vehicle scenarios.

2. The method according to claim 1, characterized in that, In step 2, let the infrared image input be... Visible light image input is , For parameters Fusion generative network, For parameters Lightweight detection network, To obtain the optimal fused image, a two-layer optimization model is constructed for the fusion and detection tasks: the upper layer aims to minimize the detection loss, while the lower layer constrains the fusion output to be the optimal solution. The formula is as follows: in Representing the target detection loss, this modeling approach allows the optimization direction of the fusion network to fully serve the downstream detection task.

3. The method according to claim 2, characterized in that, In step 2, the fusion generator uses a multi-layered densely connected module as the feature extraction backbone and receives data in parallel. and Feature encoding completed; target discriminator set up in network. and detail discriminator Dual-branch supervision, the target discriminator extracts the true foreground from the original infrared image. Positive samples, fusion prospects Using pseudo-samples, relying on adversarial loss The constraint generator preserves the outline of the infrared thermal target; The detail discriminator performs gradient calculations on the visible light image. Extract background details as real samples to fuse graph gradients. As a pseudo-sample, relying on The constraint generator inherits visible light texture details; the total fusion loss is a weighted average of structural loss, pixel loss, and bi-branch adversarial loss. in: Achieve pixel-level weighted fidelity. Ensure that the structure of the merged graph is consistent with the original graph. , For hyperparameter weights.

4. The method according to claim 3, characterized in that, In step 2, a joint iterative optimization strategy of forward propagation and backpropagation is selected, and the detection loss is used to update the parameters of the fusion network through chain-like differentiation. and detection parameters : in, The weights for the fusion loss are used; the optimal fusion graph is obtained after collaborative training. The target is fed into a lightweight detection head, and the final output is the target category and the orientation bounding box. .

5. The method according to claim 1, characterized in that, In step 3, YOLOv5 adopts an anchor-free architecture, eliminating the predefined anchor box design and directly performing target center localization and bounding box offset regression on the feature map; for each feature point, the network outputs a class confidence score. and bounding box parameters The following weighted combined loss function is used during the training phase: in: For classification cross-entropy loss; The cross-union loss is the ratio of the predicted bounding box to the ground truth bounding box. The target confidence loss is used to filter out false positives in the background. The weighting hyperparameters are used to assign weights to each loss.

6. The method according to claim 1, characterized in that, In step 4, a localized edge inference link is built based on the Jetson Nano hardware platform. With a closed-loop feedback control architecture, the embedded terminal completes the data parsing, target determination and task response of multi-source images, reducing the transmission latency and power consumption of the whole machine caused by remote data interaction.

7. The method according to claim 6, characterized in that, In step 4, the system deploys a lightweight neural network that has undergone pruning and quantization compression to detect and classify vehicles and pedestrians within the road scene; let the single-frame input image be... The set of parameters for the model after training convergence is Then the object detection task is represented as: in: for The target category prediction output at each time step; For all pedestrian and vehicle target categories; Output the posterior probability of each category for the model; optimize the network computation graph using TensorRT to achieve millisecond-level response.

8. The method according to claim 7, characterized in that, In step 4, to achieve balanced resource allocation for the parallel multi-target detection tasks of pedestrians and vehicles, a resource-driven dynamic feedback scheduling strategy is introduced; the real-time resource load index of the edge hardware is denoted as... Using a constructed scheduling function Dynamically allocate task priority and hardware resource usage: in: These are the resource weighting coefficients corresponding to the current CPU and memory load; This represents the current output data frame queue depth of the inference model. The corresponding resource weight coefficient satisfies ;like , The system automatically reduces the image acquisition frequency or switches to lightweight mode when the system load threshold is reached.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-8.

Images