Lightweight multi-mode infrared small target detection method based on RK3588

Abstract

The invention discloses an RK3588-based lightweight multi-modal infrared small target detection method, and relates to the technical field of image processing, and the method comprises the steps: 1, constructing and training a double-layer cooperative detection architecture composed of a domestic open-source multi-modal large model Qwen 2.5-VL and a target detection expert model YOLO-V11; step 2, designing a double-part Prompt composed of a generalization instruction and expert information; step 3, configuring a PC (Personal Computer) end environment required by Rycore micro RK3588 model conversion and offline reasoning program development; 4, converting the collaborative detection model obtained by training into an off-line model format special for an RK3588 platform; 5, realizing image preprocessing, visual coding, cross-modal fusion, text generation and target analysis; 6, environment configuration is completed on the RK3588 edge end equipment, and the offline model file and the reasoning program are deployed to an end side platform. According to the invention, high-precision real-time detection is carried out on an infrared small target on a domestic edge chip, and the method has the advantages of high recognition rate, low false alarm rate, strong real-time performance and the like.

Description

Technical Field

[0001] This invention relates to the fields of image processing, artificial intelligence and embedded systems, and in particular to a lightweight multimodal infrared small target detection method based on RK3588. Background Technology

[0002] In the fields of national defense and aerospace defense, real-time and accurate detection and identification of small infrared targets are core components of situational awareness and threat response. Infrared imaging technology, with its excellent all-weather operation capability, strong resistance to electromagnetic interference, good concealment, and long range, plays an irreplaceable role at the strategic and tactical levels in missile early warning, space target surveillance, precision guidance, and border and coastal reconnaissance. However, high-frame-rate infrared small target detection faces severe challenges in practical applications: First, the targets are small, typically occupying only a few dozen pixels in an image, resulting in a low signal-to-noise ratio and a lack of obvious shape and texture information, making it difficult to effectively extract key information; second, the background noise is complex and variable, with infrared images often containing complex scenes such as ground, clouds, and buildings, and natural or man-made interference such as background clutter and thermal radiation further reducing the target's identifiability; third, the real-time requirements are stringent, as the motion characteristics of high-speed maneuvering targets necessitate real-time detection, tracking, and identification at extremely high frame rates, placing extreme demands on the computational efficiency of the algorithms.

[0003] To address the aforementioned challenges, existing technological approaches can be broadly categorized into two types. The first type comprises traditional algorithms based on manual feature extraction, including detection methods based on filtering, the human visual system, or image data structures, such as maximum median filtering, two-dimensional minimum mean square filtering, local contrast mechanisms, and sparse representation. These methods offer advantages such as low computational cost and good real-time performance, but their reliance on manually designed features limits their generalization ability in complex dynamic scenes, making it difficult to effectively suppress background interference and prone to high false alarm rates and missed detections. The second type consists of detection algorithms based on deep learning, such as single-stage detection models represented by the YOLO series and two-stage detection models represented by the R-CNN series. These methods can autonomously extract deep semantic features through neural networks, improving detection accuracy and robustness to some extent, but still have limitations: when dealing with extremely small targets, target features are easily lost during deep network propagation; simultaneously, deep learning models rely on large-scale, high-quality labeled data, while infrared samples are scarce and labeling costs are high, restricting further improvements in model performance.

[0004] In recent years, large-scale models, with their massive parameter scale and generalized training on massive datasets, possess full parameterized memory of information, as well as intelligent characteristics such as self-learning, self-reasoning, and self-generation, demonstrating unique advantages in small target detection and recognition technologies with strong scene generalization capabilities. However, deploying large-scale models on edge devices still faces significant bottlenecks. International mainstream large-scale models generally suffer from technical black-box issues and high computational power dependence, while the promotion of domestic large-scale models on edge devices is limited by hardware resource constraints and challenges in domestic adaptation. In edge computing scenarios, the inherent huge computational complexity and memory consumption of models with hundreds of millions of parameters are fundamentally contradictory to the limited computing resources and storage bandwidth of domestic embedded chips. Therefore, while ensuring detection accuracy and generalization capabilities, how to achieve efficient embedded deployment of lightweight domestic large-scale models on domestic AI chip platforms through model quantization, operator optimization, and algorithm-chip co-design has become a core technical challenge that urgently needs to be overcome.

[0005] Existing infrared small target detection technologies suffer from insufficient generalization ability in complex scenarios, traditional deep learning models have limited ability to extract features from small targets, and large-scale multimodal models are difficult to deploy efficiently on domestically produced edge AI chips. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a lightweight multimodal infrared small target detection method based on RK3588. The present invention achieves high-precision and low false alarm detection of infrared small targets, and significantly improves the reliability of situational awareness.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A lightweight multimodal infrared small target detection method based on RK3588 proposed in this invention includes:

[0009] Step 1: Using the Qwen2.5-VL multimodal large model as the base, integrate the YOLO-V11 target detection expert model to construct a two-layer collaborative detection network architecture;

[0010] Step 2: Design a two-part Prompt template consisting of generalized task instructions and expert assistance information. The generalized task instructions include the task definition for infrared small target detection, the set of target categories to be detected, and the output format requirements for the absolute coordinates of the target bounding boxes. The expert assistance information includes the preliminary detection results output by the YOLO-V11 target detection expert model, which includes the target category and the absolute coordinates of the target bounding boxes, as well as prompt text describing the preliminary detection results.

[0011] Step 3: Based on the dual-part Prompt template designed in Step 2, the infrared small target image dataset is converted into an instruction fine-tuning dataset that includes image paths, generalization task instructions, expert auxiliary information, and structured answers. This instruction fine-tuning dataset is used as training samples, and LoRA low-rank adaptation technology is used to fine-tune the Qwen2.5-VL multimodal large model in the dual-layer collaborative detection network architecture. The weight parameters of the LoRA adapter are updated until the Qwen2.5-VL multimodal large model converges, and the trained collaborative detection model is obtained.

[0012] Step 4: Lightweight the trained collaborative detection model and reconstruct it into an offline visual perception and encoding model that integrates the YOLO-V11 object detection expert model and the Qwen2.5-VL visual encoder module, adapted to the RK3588 edge, and an offline language decoder model in the Qwen2.5-VL multimodal large model adapted to the RK3588 edge.

[0013] Step 5: Load the visual perception and encoding offline model and the language decoder offline model at the RK3588 edge and perform two-layer collaborative detection network inference:

[0014] Infrared small target images from the infrared small target image dataset are used to obtain preliminary detection results and visual encoding vectors for infrared small targets using an offline visual perception and encoding model. The obtained preliminary detection results are formatted into a complete text Prompt according to the two-part Prompt template in step 2. The complete text Prompt is combined with the visual encoding vector to construct a multimodal token sequence. The multimodal token sequence is input into the offline language decoder model to generate structured text including target category and absolute coordinates of target bounding box. The structured text is parsed to obtain the final infrared small target detection results.

[0015] As a further optimization of the lightweight multimodal infrared small target detection method based on RK3588 described in this invention, the detection capability of YOLO-V11 is used to assist Qwen2.5-VL in semantic understanding of infrared images.

[0016] As a further optimization of the lightweight multimodal infrared small target detection method based on RK3588 described in this invention, step 1, constructing a two-layer collaborative detection architecture, specifically includes:

[0017] Step 11: Select the Qwen2.5-VL multimodal large model as the base and select the YOLO-V11 target detection expert model;

[0018] Step 12: Configure the YOLO-V11 object detection expert model loaded with pre-trained weights as the front-end visual perception module, perform forward propagation on the input infrared image to extract candidate target regions, and output preliminary detection results including target category and absolute coordinates of target bounding box;

[0019] Step 13: Configure the Qwen2.5-VL multimodal large model as the backend semantic reasoning module, and fuse the visual features of the infrared small target image with the expert prior information provided by the preliminary detection results;

[0020] Step 14: Transfer the domain detection experience of the YOLO-V11 object detection expert model to the Qwen-2.5-VL multimodal large model by feature mapping alignment and causal language modeling loss constraints.

[0021] As a further optimization of the lightweight multimodal infrared small target detection method based on RK3588 described in this invention, step 2 involves designing a two-part Prompt template consisting of generalized task instructions and expert auxiliary information, including:

[0022] Step 21: Construct generalized task instructions: Specify the task type of infrared small target detection in the instruction text, list the preset set of infrared target categories to be detected, define the arrangement order of the bounding box coordinate array and the position of the origin of the image coordinate axis, and standardize the output format of the Qwen2.5-VL multimodal large model;

[0023] Step 22: Construct expert assistance information: Develop a guidance prompt template and format the target category and absolute coordinates of the target bounding box output by the YOLO-V11 object detection expert model into a text sequence. Embed the text sequence into the guidance prompt template to form an explicit visual prompt for the Qwen2.5-VL multimodal large model.

[0024] Step 23: Combine the generalization task instructions and expert assistance information in a preset order to form a complete Prompt input sequence, guiding the Qwen2.5-VL multimodal large model to focus on key detection areas in the infrared image.

[0025] As a further optimization of the lightweight multimodal infrared small target detection method based on RK3588 described in this invention, step 3 includes:

[0026] Step 31: Scale the images in the infrared small target image dataset to a preset resolution and convert the YOLO format relative coordinate labels to an absolute coordinate format that is compatible with the input of the Qwen2.5-VL multimodal large model;

[0027] Step 32: Construct a fine-tuning dataset for instructions based on JSONL format. Each sample in the fine-tuning dataset contains user and assistant fields. The user field encapsulates the image path, generalization task instructions, and expert assistance information. The assistant field encapsulates the real annotation data, including the target category with special word segmentation tags and the absolute coordinates of the target bounding box, to guide the Qwen2.5-VL multimodal large model to generate structured text that meets the format requirements.

[0028] Step 33: Configure the LoRA adapter, inject the low-rank adapter into the attention mechanism projection layer and feedforward network layer of the Qwen2.5-VL multimodal large model, set the rank, scaling factor and dropout rate parameters of the adapter, and freeze the pre-trained weights of the Qwen2.5-VL multimodal large model and the pre-trained weights of the YOLO-V11 object detection expert model.

[0029] Step 34: Input the instruction fine-tuning dataset into the Qwen2.5-VL multimodal large model in the two-layer collaborative detection network to perform mixed-precision training. Use the BF16 mixed-precision strategy to update the parameters of the low-rank adapter by minimizing the loss function of causal language modeling until the Qwen2.5-VL multimodal large model converges, and obtain the trained LoRA adapter weight parameters. The obtained LoRA adapter weights, together with the pre-trained weights of the Qwen2.5-VL multimodal large model and the pre-trained weights of the YOLO-V11 object detection expert model, constitute the weights of the collaborative detection model.

[0030] As a further optimization of the lightweight multimodal infrared small target detection method based on RK3588 described in this invention, the lightweight processing in step 4 includes:

[0031] Step 41: Merge the LoRA adapter weight parameters obtained during the training phase with the pre-trained weight parameters of the Qwen2.5-VL multimodal large model, and save the merged Qwen2.5-VL multimodal large model weights in FP16 half-precision format.

[0032] Step 42: Export the Qwen2.5-VL visual encoder module and the YOLO-V11 object detection expert model from the merged Qwen2.5-VL multimodal large model into a single ONNX general format model. Then, use the RKNN-Toolkit toolchain to convert the ONNX general format model into an RKNN offline model adapted to the RK3588 NPU hardware architecture. This RKNN offline model is the visual perception and encoding offline model that integrates the Qwen2.5-VL visual encoder module and the YOLO-V11 object detection expert model.

[0033] Step 43: Using the RKLLM-Toolkit toolchain, configure the W8A8 quantization strategy for the language decoder module in the merged Qwen2.5-VL multimodal large model, compress the weights and activation values ​​of the language decoder module into an 8-bit fixed-point format, and generate an offline RKLLM language decoder model adapted to the RK3588 NPU hardware architecture.

[0034] As a further optimization of the lightweight multimodal infrared small target detection method based on RK3588 described in this invention, in step 5, the RK3588 edge-end loads a visual perception and encoding offline model and a language decoder offline model to perform two-layer collaborative detection network inference; including:

[0035] Step 51: Construct the RK3588 edge inference main program. First, initialize the RKNN and RKLLM runtime environment and preprocess the input infrared small target image to adapt it to the input dimension requirements of the visual perception and encoding offline model.

[0036] Step 52: Call the RKNN-Runtime interface to load and run the visual perception and encoding RKNN offline model, perform forward computation on the preprocessed infrared small target image, and simultaneously obtain the preliminary detection results and visual encoding vector of the infrared small target;

[0037] Step 53: Format the preliminary detection results obtained in step 52 into a complete text Prompt according to the two-part Prompt template, and then convert the text Prompt into a token sequence using a token segmenter.

[0038] Step 54: Linearly concatenate the visual encoding vector output in step 52 with the token sequence generated in step 53 along the sequence length dimension to construct a multimodal token sequence including real-time expert detection data;

[0039] Step 55: Call the RKLLM-Runtime interface to load and run the RKLLM offline language decoder model, perform autoregressive decoding inference on the multimodal token sequence, and generate a structured text stream including the target category and the absolute coordinates of the target bounding box one by one;

[0040] Step 56: Parse the target category and target bounding box absolute coordinates in the structured text stream, denormalize the absolute coordinate values ​​and map them back to the original infrared image coordinate system, and output the final infrared small target detection result.

[0041] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0042] (1) Significantly improves the detection accuracy and robustness of infrared weak targets. This invention constructs a two-layer collaborative architecture of "multimodal large model base + target detection expert model" and innovatively designs a two-part Prompt strategy of "generalization task instruction + expert auxiliary information". This strategy uses the prior detection results of YOLO-V11 as visual cues to explicitly guide the Qwen2.5-VL large model to focus on key areas with extremely low signal-to-noise ratio in infrared images. This mechanism effectively overcomes the false alarm problem that traditional single-modal models are prone to generate under complex background clutter, and at the same time, it significantly reduces the false alarm rate of small targets by utilizing the semantic reasoning ability of the large model;

[0043] (2) This invention eliminates the additional computational overhead of the inference stage while preserving domain knowledge through LoRA-adjusted weight merging technology; combined with W8A8 deep quantization and operator fusion technology, it successfully adapts a multimodal model with 3 billion parameters to the memory-constrained RK3588 platform. This lightweight solution solves the problem of large models not being able to run or run fast on embedded devices, and meets the practical requirements of low latency and high efficiency processing for defense end-side equipment. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the hardware appearance of the RK3588 edge terminal used in this invention;

[0045] Figure 2 These are sample images of the infrared small target dataset constructed in this invention;

[0046] Figure 3 This is a schematic diagram of the visualization results of infrared small target detection in this invention;

[0047] Figure 4 This is a schematic diagram of the output of the two-layer collaborative detection network inference performed on the PC in this invention;

[0048] Figure 5 This is a schematic diagram of the terminal output of the invention performing two-layer collaborative detection network inference at the edge of the RK3588. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] A lightweight multimodal infrared small target detection method based on RK3588 includes the following steps:

[0051] S1. Using the Qwen2.5-VL multimodal large model as the base, and integrating the YOLO-V11 target detection expert model, a two-layer collaborative detection network architecture is constructed.

[0052] S2. Design a two-part Prompt template consisting of generalized task instructions and expert assistance information. The generalized task instructions include the task definition for infrared small target detection, the set of target categories to be detected, and the output format requirements for the absolute coordinates of the target bounding boxes. The expert assistance information includes the preliminary detection results output by the YOLO-V11 target detection expert model, which includes the target category and the absolute coordinates of the target bounding boxes, as well as the prompt text describing the preliminary detection results.

[0053] S3. Based on the bipart Prompt template designed in S2, the infrared small target image dataset is converted into an instruction fine-tuning dataset that includes image paths, generalization task instructions, expert auxiliary information, and structured answers. This instruction fine-tuning dataset is used as training samples. LoRA low-rank adaptation technology is used to fine-tune the Qwen2.5-VL multimodal large model in the bilayer collaborative detection network architecture. The weight parameters of the LoRA adapter are updated until the Qwen2.5-VL multimodal large model converges, and the trained collaborative detection model is obtained.

[0054] S4. Lightweight the trained collaborative detection model and reconstruct it into an offline visual perception and encoding model that integrates the YOLO-V11 object detection expert model and the Qwen2.5-VL visual encoder module, adapted to the RK3588 edge, and an offline language decoder model in the Qwen2.5-VL multimodal large model adapted to the RK3588 edge.

[0055] S5. Load the visual perception and encoding offline model and the language decoder offline model at the edge of the RK3588 to perform two-layer collaborative detection network inference:

[0056] Infrared small target images from the infrared small target image dataset are used to obtain preliminary detection results and visual encoding vectors for infrared small targets using an offline visual perception and encoding model. The obtained preliminary detection results are formatted into a complete text Prompt according to the two-part Prompt template in step 2. The complete text Prompt is combined with the visual encoding vector to construct a multimodal token sequence. The multimodal token sequence is input into the offline language decoder model to generate structured text including target category and absolute coordinates of target bounding box. The structured text is parsed to obtain the final infrared small target detection results.

[0057] Figure 1 This is an appearance diagram of the edge end of the RK3588 used in this invention. Figure 2 These are infrared small target sample images from the infrared small target image dataset.

[0058] The two-layer collaborative detection network architecture constructed in step S1 combines real-time visual perception capabilities with deep semantic understanding capabilities. Logically, this architecture is divided into two modules: front-end visual perception and back-end semantic reasoning. The specific implementation process is further subdivided into the following sub-steps:

[0059] S101. The Qwen2.5-VL multimodal large model is selected as the base, and the YOLO-V11 object detection expert model is selected. This invention needs to be deployed on the RK3588 edge. Due to the hardware resource constraints of the RK3588 edge, it is necessary to balance inference accuracy and memory usage. The specific selection is as follows: the open-source lightweight version of the Qwen2.5-VL multimodal large model, Qwen2.5-VL-3B-Instruct, is selected as the base, and the object detection expert model is the lightweight model YOLO-V11n from the YOLO-V11 series; the official pre-trained weights of both are loaded respectively to complete the initial configuration of the two-layer collaborative detection network architecture.

[0060] S102. Configure the YOLO-V11 object detection expert model with pre-trained weights as the front-end visual perception module. Perform forward propagation on the input infrared image to extract candidate target regions and output preliminary detection results including target category and absolute coordinates of target bounding boxes. In the two-layer collaborative detection network architecture, the preliminary detection results provided by the YOLO-V11 object detection expert model serve as expert prior information, guiding the back-end semantic reasoning module.

[0061] S103. Configure the Qwen2.5-VL multimodal large model as the backend semantic reasoning module, use the Qwen2.5-VL multimodal large model to extract the visual features of the infrared small target image, and map the preliminary detection results output in step S102 as expert prior information to the visual feature space, so that the backend semantic reasoning module can combine the expert prior information to understand and reason about the infrared small target image.

[0062] S104. By using feature mapping alignment and causal language modeling loss constraints, the domain detection experience of the YOLO-V11 object detection expert model is transferred to the Qwen-2.5-VL multimodal large model.

[0063] Step S2 designs a two-part Prompt template consisting of generalized task instructions and expert-aided information. This step aims to clarify the task definition and guide the Qwen2.5-VL multimodal large model to perform semantic reasoning in conjunction with expert-aided information, ensuring the accuracy and structure of the Qwen2.5-VL multimodal large model output. The specific implementation process is subdivided into the following sub-steps:

[0064] S201. Constructing Generalized Task Instructions: The instruction text should clearly define the task type for infrared small target detection, list the preset set of infrared target categories to be detected, define the order of the bounding box coordinate array and the position of the image coordinate axis origin, and standardize the output format of the Qwen2.5-VL multimodal large model. Specifically, the instruction text content is as follows:

[0065] (1) Task definition: "Please give the quantity of each of the following different targets in the picture and the precise bounding box of each target";

[0066] (2) Set of target categories to be detected: "The target categories in the image are: drone, car, ship, bus, pedestrian, cyclist";

[0067] (3) Output format requirements: "The bounding box is an array of four integers [x1, y1, x2, y2], which represent the coordinates of the upper left and lower right corners of the target, respectively. In the image, the coordinate system is defined as follows: the x-axis is from left to right, with the left side being the minimum value 0 and the right side being the maximum value; the y-axis is from top to bottom, with the top being the minimum value and the bottom being the maximum value."

[0068] S202. Constructing Expert Assistance Information: First, the guiding prompt template is set to "Preliminary detection result:". Second, the preliminary detection result output by the YOLO-V11 object detection expert model, including the object category and the absolute coordinates of the object bounding box, is formatted into a text sequence. For example, when the YOLO-V11 object detection expert model outputs an object category of "car" with absolute bounding box coordinates of [259, 234, 263, 239], the corresponding text sequence is: "car is located at [259, 234, 263, 239]". If multiple objects exist, the corresponding text is generated sequentially and concatenated. Finally, the text sequence is embedded into the guiding prompt template to form complete expert assistance information.

[0069] S203. Concatenate the generalized task instructions and expert auxiliary information in the order of "generalized task instructions + expert auxiliary information" to form a complete Prompt input sequence, ensuring that the Qwen2.5-VL multimodal large model understands the task definition first, and then combines the expert auxiliary information for reasoning.

[0070] Step S3, based on the dual-part Prompt template designed in step S2, converts the infrared small target image dataset into a command fine-tuning dataset, and performs command fine-tuning training on the Qwen2.5-VL multimodal large model. The specific implementation process includes the following sub-steps:

[0071] S301. Scale the images in the infrared small target image dataset to a preset resolution and convert the YOLO format relative coordinate labels to an absolute coordinate format compatible with the Qwen2.5-VL multimodal large model input. Specifically, read the images from the infrared small target image dataset, use bilinear interpolation to scale the images to 392×392 pixels, and simultaneously iterate through the label files in the infrared small target image dataset to convert the YOLO format normalized relative coordinates... Convert to an absolute coordinate format suitable for Qwen2.5-VL multimodal large models. .

[0072] S302. Construct a fine-tuning dataset for instructions based on JSONL format. Each sample in the fine-tuning dataset contains the fields "user" and "assistant". In practice, each sample in the fine-tuning dataset is treated as an independent training row, and its encapsulation logic is as follows:

[0073] (1) The user field encapsulates the relative path of the infrared small target image, the generalized task instruction defined in step S201, and the expert assistance information generated in step S202;

[0074] (2) The assistant field encapsulates the actual labeled data and uses special word segmentation tags to wrap the target category and the absolute coordinates of the target bounding box to guide the Qwen2.5-VL multimodal large model to generate structured text that meets the format requirements. Specifically, the target category is wrapped with <|object_ref_start|> and <|object_ref_end|>, and the absolute coordinates of the target bounding box are wrapped with <|box_start|> and <|box_end|>.

[0075] S303. Configure the LoRA adapter by injecting the low-rank adapter into the attention mechanism projection layer and feedforward network layer of the Qwen2.5-VL multimodal large model. Set the adapter's rank, scaling factor, and dropout rate parameters, and freeze the pre-trained weights of the Qwen2.5-VL multimodal large model and the YOLO-V11 object detection expert model. Specific implementation details are as follows:

[0076] (1) Inject trainable low-rank matrices into the attention mechanism projection layers q_proj, k_proj, v_proj, o_proj and the feedforward network layers gate_proj, up_proj, down_proj of the Qwen2.5-VL multimodal large model;

[0077] (2) Set the rank of the low-rank matrix r=64, the scaling factor lora_alpha=16, and the dropout rate lora_dropout=0.05; (3) Freeze all the original pre-trained weights of the YOLO-V11 object detection expert model and the Qwen2.5-VL multimodal large model, and only update the injected LoRA adapter parameters to reduce memory usage and prevent catastrophic forgetting.

[0078] S304. Input the instruction fine-tuning dataset into the Qwen2.5-VL multimodal large model in the two-layer collaborative detection network to perform mixed-precision training. Specific implementation details are as follows:

[0079] Training was initiated on a server equipped with a high-performance GPU using the Trainer interface of the Transformers library. A BF16 mixed-precision strategy was employed during training, which significantly reduced GPU memory consumption while maintaining numerical stability and accelerating the training process. The loss function used was cross-entropy loss based on causal language modeling, continuing until the Qwen2.5-VL multimodal large model converged, yielding the trained LoRA adapter weights. These LoRA adapter weights, along with the pre-trained weights of the Qwen2.5-VL multimodal large model and the YOLO-V11 object detection expert model, together constitute the weights of the collaborative detection model.

[0080] This training process forces the Qwen2.5-VL multimodal large model to learn and internalize the sensitivity of the YOLO target detection expert model to small infrared targets, thereby achieving effective transfer of domain knowledge and improving detection accuracy.

[0081] Step S4 involves lightweighting the trained collaborative detection model. The specific implementation process includes the following sub-steps:

[0082] S401, Transfer the trained LoRA adapter weights Pre-trained weight parameters of Qwen2.5-VL multimodal large model The layers are added and merged one by one, and the weights of the merged Qwen2.5-VL multimodal large model are: The weights of the merged Qwen2.5-VL multimodal large model are saved in FP16 half-precision format, which retains sufficient numerical accuracy to maintain detection performance and provides a standard input format for subsequent conversion to the ONNX universal format model, eliminating the inference latency caused by dynamically loading the LoRA adapter.

[0083] S402. The Qwen2.5-VL visual encoder module and the YOLO-V11 object detection expert model from the merged Qwen2.5-VL multimodal large model are exported together as a single ONNX general-purpose format model. During this process, the input tensor dimensions are fixed at [1, 392, 392, 3]. The RKNN-Toolkit2 toolchain is called to load this ONNX general-purpose format model. Operator fusion and instruction optimization are performed for the RKNPU2 architecture at the RK3588 edge, compiling it into an RKNN format offline model specific to the RK3588 NPU. This RKNN offline model is the visual perception and encoding offline model integrating the Qwen2.5-VL visual encoder module and the YOLO-V11 object detection expert model.

[0084] S403. Using the RKLLM-Toolkit toolchain, load the language decoder module from the merged Qwen2.5-VL multimodal large model. By configuring the W8A8 quantization strategy, compress the weights and activation values ​​of the language decoder module into an 8-bit fixed-point format, generating an offline RKLLM language decoder model adapted to the RK3588 NPU hardware architecture. This quantization step can compress the size of the language decoder module to about half of the original FP16 half-precision format, significantly reducing memory usage and making it compatible with the physical memory limitations of the RK3588 edge.

[0085] Step S5 executes two-layer collaborative detection network inference on the RK3588 edge, and the specific implementation process includes the following sub-steps:

[0086] S501. Construct the RK3588 edge inference main program. Initialize the context handles of the RKNN Runtime and RKLLM Runtime by calling the rknn_init and rkllm_init interfaces respectively, and allocate the NPU memory pool. Convert the input infrared small target image from its original format to RGB color space, and use bilinear interpolation to scale it to 392×392 pixels to ensure that the tensor of the preprocessed infrared small target image strictly matches the input dimension requirements of the visual perception and encoding offline model.

[0087] S502: The RKNN Runtime interface is called to load the offline RKNN model for visual perception and encoding. The preprocessed infrared small target image data is then passed to the input buffer of the RK3588 NPU, driving the NPU to perform forward inference computation. Simultaneously, the preliminary detection results of the infrared small targets and a visual encoding vector reflecting the global features of the infrared small target image are obtained from the output. In this step, the hardware acceleration capability of the RK3588 NPU is utilized to complete the deep convolutional feature extraction task, which is originally difficult to process in real time on a CPU.

[0088] S503. According to the two-part Prompt template, the preliminary detection result obtained in step S502 is formatted into a complete text Prompt, and then the text Prompt is converted into a token sequence by a token segmenter.

[0089] S504. The visual encoding vector output in step S502 and the token sequence generated in step S503 are linearly concatenated in the sequence length dimension to construct a multimodal token sequence that includes real-time expert detection data. This multimodal token sequence simultaneously contains deep visual features of the infrared small target image and real-time expert detection data from the YOLO-V11 target detection expert model.

[0090] S505: The RKLLM-Runtime interface is called to load and run the RKLLM offline language decoder model. The computing power of the RK3588 NPU is used to perform autoregressive decoding on the multimodal token sequence, predicting and generating the next token one by one. The Qwen2.5-VL multimodal large model, constrained by the fine-tuning training phase, will generate a structured text stream according to the preset instruction fine-tuning format.

[0091] For example, the generated structured text stream might be “<|object_ref_start|>drone<|object_ref_end|> is located at <|box_start|>[356 258 375 275]<|box_end|>”, which contains the target category drone and the corresponding absolute coordinates of the target bounding box [356 258 375 275].

[0092] S506. Design a structured text stream generated by real-time matching of regular expressions, parse the target category and target bounding box absolute coordinates wrapped by special markers in the structured text stream, denormalize the absolute coordinate values ​​and map them back to the original infrared image coordinate system, output the final infrared small target detection results, and visualize them. Figure 3 For input Figure 2 The obtained visualization results of infrared small target detection.

[0093] The results of inference using a two-layer collaborative detection network executed on the RK3588 edge device were statistically analyzed, achieving excellent performance with an average recall rate of 77.0%, precision of 91.8%, and false alarm rate of 8.2%. It also demonstrated stable performance in infrared small target localization accuracy and multi-target detection capabilities. Furthermore, the inference efficiency of the language decoder module in the Qwen2.5-VL multimodal large model was monitored, showing an average output token speed of 12 tokens / second. Figure 5The results of text inference using a two-layer collaborative detection network executed at the edge of the RK3588 are presented, along with... Figure 4 The results of the two-layer collaborative detection network inference executed on the PC are consistent, verifying the correctness of the RK3588 edge inference.

[0094] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight multimodal infrared small target detection method based on RK3588, characterized in that, include: Step 1: Using the Qwen2.5-VL multimodal large model as the base, integrate the YOLO-V11 target detection expert model to construct a two-layer collaborative detection network architecture; Step 2: Design a two-part Prompt template consisting of generalized task instructions and expert assistance information. The generalized task instructions include the task definition for infrared small target detection, the set of target categories to be detected, and the output format requirements for the absolute coordinates of the target bounding boxes. The expert assistance information includes the preliminary detection results output by the YOLO-V11 target detection expert model, which includes the target category and the absolute coordinates of the target bounding boxes, as well as prompt text describing the preliminary detection results. Step 3: Based on the dual-part Prompt template designed in Step 2, the infrared small target image dataset is converted into an instruction fine-tuning dataset that includes image paths, generalization task instructions, expert auxiliary information, and structured answers. This instruction fine-tuning dataset is used as training samples, and LoRA low-rank adaptation technology is used to fine-tune the Qwen2.5-VL multimodal large model in the dual-layer collaborative detection network architecture. The weight parameters of the LoRA adapter are updated until the Qwen2.5-VL multimodal large model converges, and the trained collaborative detection model is obtained. Step 4: Lightweight the trained collaborative detection model and reconstruct it into an offline visual perception and encoding model that integrates the YOLO-V11 object detection expert model and the Qwen2.5-VL visual encoder module, adapted to the RK3588 edge, and an offline language decoder model in the Qwen2.5-VL multimodal large model adapted to the RK3588 edge. Step 5: Load the visual perception and encoding offline model and the language decoder offline model at the RK3588 edge and perform two-layer collaborative detection network inference: Infrared small target images from the infrared small target image dataset are used to obtain preliminary detection results and visual encoding vectors for infrared small targets using an offline visual perception and encoding model. The obtained preliminary detection results are formatted into a complete text Prompt according to the two-part Prompt template in step 2. The complete text Prompt is combined with the visual encoding vector to construct a multimodal token sequence. The multimodal token sequence is input into the offline language decoder model to generate structured text including target category and absolute coordinates of target bounding box. The structured text is parsed to obtain the final infrared small target detection results.

2. The lightweight multimodal infrared small target detection method based on RK3588 as described in claim 1, characterized in that, The detection capabilities of YOLO-V11 are used to assist Qwen2.5-VL in semantic understanding of infrared images.

3. The lightweight multimodal infrared small target detection method based on RK3588 as described in claim 1, characterized in that, Step 1, constructing the two-layer collaborative detection architecture, specifically includes: Step 11: Select the Qwen2.5-VL multimodal large model as the base and select the YOLO-V11 target detection expert model; Step 12: Configure the YOLO-V11 object detection expert model loaded with pre-trained weights as the front-end visual perception module, perform forward propagation on the input infrared image to extract candidate target regions, and output preliminary detection results including target category and absolute coordinates of target bounding box; Step 13: Configure the Qwen2.5-VL multimodal large model as the backend semantic reasoning module, and fuse the visual features of the infrared small target image with the expert prior information provided by the preliminary detection results; Step 14: Transfer the domain detection experience of the YOLO-V11 object detection expert model to the Qwen-2.5-VL multimodal large model by feature mapping alignment and causal language modeling loss constraints.

4. The lightweight multimodal infrared small target detection method based on RK3588 as described in claim 1, characterized in that, In step 2, a two-part Prompt template consisting of generalized task instructions and expert assistance information is designed, including: Step 21: Construct generalized task instructions: Specify the task type of infrared small target detection in the instruction text, list the preset set of infrared target categories to be detected, define the arrangement order of the bounding box coordinate array and the position of the origin of the image coordinate axis, and standardize the output format of the Qwen2.5-VL multimodal large model; Step 22: Construct expert assistance information: Develop a guidance prompt template and format the target category and absolute coordinates of the target bounding box output by the YOLO-V11 object detection expert model into a text sequence. Embed the text sequence into the guidance prompt template to form an explicit visual prompt for the Qwen2.5-VL multimodal large model. Step 23: Combine the generalization task instructions and expert assistance information in a preset order to form a complete Prompt input sequence, guiding the Qwen2.5-VL multimodal large model to focus on key detection areas in the infrared image.

5. The lightweight multimodal infrared small target detection method based on RK3588 as described in claim 1, characterized in that, Step 3 includes: Step 31: Scale the images in the infrared small target image dataset to a preset resolution and convert the YOLO format relative coordinate labels to an absolute coordinate format that is compatible with the input of the Qwen2.5-VL multimodal large model; Step 32: Construct a fine-tuning dataset for instructions based on JSONL format. Each sample in the fine-tuning dataset contains user and assistant fields. The user field encapsulates the image path, generalization task instructions, and expert assistance information. The assistant field encapsulates the real annotation data, including the target category with special word segmentation tags and the absolute coordinates of the target bounding box, to guide the Qwen2.5-VL multimodal large model to generate structured text that meets the format requirements. Step 33: Configure the LoRA adapter, inject the low-rank adapter into the attention mechanism projection layer and feedforward network layer of the Qwen2.5-VL multimodal large model, set the rank, scaling factor and dropout rate parameters of the adapter, and freeze the pre-trained weights of the Qwen2.5-VL multimodal large model and the pre-trained weights of the YOLO-V11 object detection expert model. Step 34: Input the instruction fine-tuning dataset into the Qwen2.5-VL multimodal large model in the two-layer collaborative detection network to perform mixed-precision training. Use the BF16 mixed-precision strategy to update the parameters of the low-rank adapter by minimizing the loss function of causal language modeling until the Qwen2.5-VL multimodal large model converges, and obtain the trained LoRA adapter weight parameters. The obtained LoRA adapter weights, together with the pre-trained weights of the Qwen2.5-VL multimodal large model and the pre-trained weights of the YOLO-V11 object detection expert model, constitute the weights of the collaborative detection model.

6. The lightweight multimodal infrared small target detection method based on RK3588 as described in claim 5, characterized in that, The lightweighting process in step 4 includes: Step 41: Merge the LoRA adapter weight parameters obtained during the training phase with the pre-trained weight parameters of the Qwen2.5-VL multimodal large model, and save the merged Qwen2.5-VL multimodal large model weights in FP16 half-precision format. Step 42: Export the Qwen2.5-VL visual encoder module and the YOLO-V11 object detection expert model from the merged Qwen2.5-VL multimodal large model into a single ONNX general format model. Then, use the RKNN-Toolkit toolchain to convert the ONNX general format model into an RKNN offline model adapted to the RK3588 NPU hardware architecture. This RKNN offline model is the visual perception and encoding offline model that integrates the Qwen2.5-VL visual encoder module and the YOLO-V11 object detection expert model. Step 43: Using the RKLLM-Toolkit toolchain, configure the W8A8 quantization strategy for the language decoder module in the merged Qwen2.5-VL multimodal large model, compress the weights and activation values ​​of the language decoder module into an 8-bit fixed-point format, and generate an offline RKLLM language decoder model adapted to the RK3588 NPU hardware architecture.

7. The lightweight multimodal infrared small target detection method based on RK3588 as described in claim 1, characterized in that, In step 5, the RK3588 edge-end loads the offline visual perception and encoding model and the offline language decoder model to perform two-layer collaborative detection network inference; including: Step 51: Construct the RK3588 edge inference main program. First, initialize the RKNN and RKLLM runtime environment and preprocess the input infrared small target image to adapt it to the input dimension requirements of the visual perception and encoding offline model. Step 52: Call the RKNN-Runtime interface to load and run the visual perception and encoding RKNN offline model, perform forward computation on the preprocessed infrared small target image, and simultaneously obtain the preliminary detection results and visual encoding vector of the infrared small target; Step 53: Format the preliminary detection results obtained in step 52 into a complete text Prompt according to the two-part Prompt template, and then convert the text Prompt into a token sequence using a token segmenter. Step 54: Linearly concatenate the visual encoding vector output in step 52 with the token sequence generated in step 53 along the sequence length dimension to construct a multimodal token sequence including real-time expert detection data; Step 55: Call the RKLLM-Runtime interface to load and run the RKLLM offline language decoder model, perform autoregressive decoding inference on the multimodal token sequence, and generate a structured text stream including the target category and the absolute coordinates of the target bounding box one by one; Step 56: Parse the target category and target bounding box absolute coordinates in the structured text stream, denormalize the absolute coordinate values ​​and map them back to the original infrared image coordinate system, and output the final infrared small target detection result.

Images