Lightweight target detection model generation method, analysis method and electronic device

Abstract

The present application relates to a kind of lightweight target detection model generation method, analysis method and electronic equipment.The method comprises: obtaining infrared image dataset;The image of training set in the infrared image dataset is input into the preset partial convolution lightweight network for training, and image training result is obtained, only the convolution operation of partial channel of input image is carried out in the preset partial convolution lightweight network;Partial convolution lightweight network is continuously iterated and optimized based on the preset loss function, the training set and the image training result, and the partial convolution lightweight network after training is obtained.The present application uses partial convolution as basic convolution unit, utilizes the channel redundancy of feature map, only carries out spatial convolution operation to partial channel, while keeping the rest unchanged, thereby significantly reducing the amount of calculation.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, specifically to a lightweight target detection model generation method, analysis method, and electronic device. Background Technology

[0002] With the rapid development of the low-altitude economy and the widespread adoption of drone technology, low-altitude security threats are becoming increasingly prominent. Illegally intruding "low, slow, and small" targets, such as consumer drones, pose a serious threat to the security of sensitive areas such as airports, military bases, large event venues, and prisons. Infrared imaging technology, due to its advantages such as passive detection, all-weather operation, and strong anti-interference capabilities, has become one of the core sensing methods in the field of low-altitude security. Infrared small target detection, as a key component of infrared security systems, directly determines the performance of the entire system through its detection accuracy and real-time performance. However, due to the extremely small size, weak signal, and lack of texture features of infrared small targets, coupled with complex and variable backgrounds, infrared small target detection has become a recognized challenge in the field of computer vision.

[0003] Currently, several deep learning-based infrared small target detection methods have been proposed, but they generally suffer from the challenge of balancing accuracy and efficiency. Some methods employ complex network structures and attention mechanisms to achieve high detection accuracy, but their large parameter and computational loads make real-time inference on edge devices difficult. Other methods adopt lightweight designs to reduce computational overhead, but their detection accuracy is relatively low, with many false negatives and missed detections. Furthermore, existing methods are generally optimized for GPUs (Graphics Processing Units), without considering the computing power and operator support of edge AI chips, making direct deployment to edge platforms often difficult to leverage hardware performance. Therefore, how to achieve real-time deployment at the edge while ensuring detection accuracy is a pressing issue that needs to be addressed in the field of infrared small target detection. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a lightweight target detection model generation method, analysis method, and electronic device.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: A lightweight object detection model generation method includes: Obtain infrared image dataset; The images in the training set of the infrared image dataset are input into a preset partially convolutional lightweight network for training to obtain image training results. The preset partially convolutional lightweight network only performs convolution operations on a portion of the input images. Based on the preset loss function, the training set, and the image training results, the partially convolutional lightweight network is continuously iteratively optimized to obtain the trained partially convolutional lightweight network.

[0006] The beneficial effects of this invention are: it uses partial convolution as the basic convolution unit, and its core idea is to utilize the channel redundancy of the feature map to perform spatial convolution operation only on some channels while keeping the other channels unchanged, thereby significantly reducing the amount of computation.

[0007] Furthermore, the preset partially convolutional lightweight network includes a feature extraction layer, an encoder, and a decoder. The feature extraction layer is used to receive the input image. The encoder is connected to the feature extraction layer and is skipped between the encoder and the decoder. The encoder includes a partially convolutional module, which performs convolution operations on a portion of the input image.

[0008] Furthermore, the partial convolution module performs convolution operations on a portion of the input image channels, including: According to a preset segmentation ratio, the first number of channels of the first image input to a portion of the convolution module are divided into convolution channels and non-convolution channels, and the sum of the convolution channels and the non-convolution channels is the first number of channels; Convolutional feature maps are obtained by convolving the feature maps within the convolutional channels of the first image; Pooling or keeping the feature map in the non-convolutional channel of the first image unchanged yields a non-convolutional feature map. The convolutional feature map and the non-convolutional feature map are concatenated according to the dimension of the first number of channels to obtain a partial convolutional image.

[0009] Furthermore, the encoder includes a first coding block, a second coding block, a third coding block, a fourth coding block, and a fifth coding block connected in series for downsampling, and the decoder includes a first decoding block, a second decoding block, a third decoding block, a fourth decoding block, and a fifth decoding block connected in series for upsampling. The first coding block is connected to the feature extraction layer, the fifth coding block is connected to the first decoding block, and skip connections are established between the first coding block and the fourth decoding block, between the second coding block and the third decoding block, between the third coding block and the second decoding block, and between the fourth coding block and the first decoding block.

[0010] Furthermore, a local contrast enhancement module is inserted on each connection between the corresponding layers of the encoder and the decoder.

[0011] Furthermore, each of the encoding blocks and each of the decoding blocks includes a fast feature block, which includes partial convolution, 1×1 convolution, batch normalization, an efficient channel attention module, and residual connections.

[0012] Furthermore, the fourth and fifth coding blocks include convolutional block attention modules, and the fifth coding block includes strip attention modules.

[0013] Furthermore, after iteratively optimizing the partially convolutional lightweight network based on a preset loss function, the training set, and the image training results to obtain the trained partially convolutional lightweight network, the process further includes: The trained partially convolutional lightweight network model is exported as an open neural network exchange format; The pre-installed platform processing unit converts the partially convolutional lightweight network model of the open neural network exchange format into the corresponding model format.

[0014] A lightweight target detection model analysis method is applied to a lightweight target detection model generated by any of the lightweight target detection model generation methods described above, the method comprising: Input the image to be detected into the lightweight target detection model; The first decoding block, the second decoding block, the third decoding block, and the fifth decoding block each output a single-channel prediction map, which is then fused after upsampling and alignment to generate a detection result.

[0015] An electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the lightweight target detection model generation method described in any of the preceding claims. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the lightweight target detection model generation method provided in the embodiments of this application; Figure 2 This is a diagram of the overall PCLNet network architecture provided in the embodiments of this application; Figure 3 This is a partial structural diagram of the PConv convolutional module provided in the embodiments of this application; Figure 4 This is a structural diagram of the Local Contrast Enhancement (LCEM) module provided in the embodiments of this application; Figure 5 This is a diagram of the multi-scale feature fusion and deep supervision structure provided in the embodiments of this application; Figure 6This is a flowchart of the RK3588 edge deployment process provided in the embodiments of this application; Figure 7 This is a multi-core NPU parallel inference scheduling graph provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0019] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this technology based on the specific circumstances.

[0020] In the description of this application, spatial relation terms such as "below," "under," "below," "below," "above," "over," etc., are used herein to describe the relationship between one element or feature shown in the figures and other elements or features. It should be understood that, in addition to the orientation shown in the figures, spatial relation terms also include different orientations of the device in use and operation. For example, if the device in the figures is flipped, an element or feature described as "below" or "under" or "below" of other elements or features will be oriented "above" other elements or features. Therefore, the exemplary terms "below" and "under" can include both upper and lower orientations. Furthermore, the device may also include other orientations (e.g., rotated 90 degrees or other orientations), and the spatial descriptive terms used herein are interpreted accordingly.

[0021] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0022] Example 1 Figure 1 A schematic flowchart of a lightweight target detection model generation method provided in this application is shown. This lightweight target detection model generation method includes at least the following steps: S110. Obtain the infrared image dataset; S120. Input the images from the training set of the infrared image dataset into a preset partially convolutional lightweight network for training to obtain image training results. The preset partially convolutional lightweight network only performs convolution operations on a portion of the input images. S130. Based on the preset loss function, the training set, and the image training results, the partially convolutional lightweight network is continuously iteratively optimized to obtain the trained partially convolutional lightweight network.

[0023] Specifically, the process involves acquiring infrared image datasets, including dataset construction, data partitioning, image preprocessing, and data augmentation.

[0024] Dataset Construction. The datasets are all derived from various publicly available infrared small target datasets. Each dataset contains infrared images and their corresponding pixel-level labeled masks. For example, in the labeled mask, the pixel value of the target area is 255, and the pixel value of the background area is 0.

[0025] Data partitioning. The dataset is divided into training, validation, and test sets, using a fixed random seed to ensure deterministic partitioning.

[0026] Image preprocessing. The input image is uniformly resized to a fixed size (e.g., 256×256) and normalized using the following formula:

[0027] Among them, X raw ∈R (C-C / n)×H×W X is the original pixel value matrix of the input image in the dataset. norm∈ R (C -C / n)×H×W is the normalized pixel value matrix, where H and W are the height and width of the image, respectively, C is the number of input channels of the input image, and n is the segmentation ratio.

[0028] Data augmentation. During the training phase, data augmentation strategies such as random horizontal flipping, random vertical flipping, random rotation, random scaling, and random cropping are employed. These operations are performed simultaneously on the image and the labeled mask.

[0029] Images from the training set of the infrared image dataset are input into a pre-defined partially convolutional lightweight network for training, resulting in image training results. The pre-defined partially convolutional lightweight network only performs convolution operations on a portion of the input images.

[0030] Based on the preset loss function, training set, and image training results, the partially convolutional lightweight network is continuously iteratively optimized to obtain the trained partially convolutional lightweight network.

[0031] Loss function design. For example, a combination of BCE (Binary Cross-Entropy) loss and Dice loss can be used as the loss function. BCE loss is calculated independently for each pixel, while Dice loss directly optimizes the region overlap.

[0032] The formula for binary cross-entropy loss is as follows:

[0033] Where y i Let ∈{0, 1} be the true label of the i-th pixel in the input image, and p i ∈(0, 1) represents the predicted probability of the label of the i-th pixel in the input image, and N is the total number of pixels in the input image.

[0034] The Dice loss formula is as follows:

[0035] Where ε = 10 5 This is a smoothing coefficient to prevent the denominator from being zero.

[0036] Total loss L total The sum of primary loss and secondary loss:

[0037] Where λ=0.2 is the auxiliary loss weight, and K is the number of auxiliary outputs. Let Laux^k be the loss of the k-th auxiliary output. In this invention, K=3, meaning there are 3 auxiliary outputs, originating from the three decoder layers Dec2 (1 / 2× scale), Dec3 (1 / 4× scale), and Dec4 (1 / 8× scale). Each auxiliary output generates a single-channel prediction map through a 1×1 convolution. After downsampling the ground truth labels to the corresponding scale, the BCE+Dice loss is calculated as Laux^k. Specifically, Laux^1 is the loss of the prediction map of layer Dec2 at the 1 / 2× scale, Laux^2 is the loss of the prediction map of layer Dec3 at the 1 / 4× scale, and Laux^3 is the loss of the prediction map of layer Dec4 at the 1 / 8× scale.

[0038] Using input images from the training set of the infrared image dataset and the labeled masks on those images, a pre-defined partially convolutional lightweight network is trained using the aforementioned loss function. Then, the performance of the trained partially convolutional lightweight network is evaluated using input images from the validation and test sets of the infrared image dataset, along with the labeled masks on those images. The performance evaluation includes pixel-level and target-level metrics.

[0039] Pixel-level evaluation metrics. Pixel-level segmentation performance is evaluated using metrics such as IoU, Precision, Recall, and F1 score.

[0040] The Intersection over Union (IoU) ratio is calculated as follows:

[0041] Precision, the formula is as follows:

[0042] Recall, the formula is as follows:

[0043] The F1 score is calculated using the following formula:

[0044] Where TP represents the number of true positive pixels, FP represents the number of false positive pixels, and FN represents the number of false negative pixels.

[0045] Target-level evaluation metrics. Target-level detection performance is evaluated using Pd (Probability of Detection) and Fa (False Alarm Rate).

[0046]

[0047] Where, N matchedN is the number of targets that are correctly matched. gt This represents the total number of actual targets.

[0048]

[0049] Among them, S unmatched This represents the total area of ​​pixels that did not match the predicted target.

[0050] In this embodiment, partial convolution is used as the basic convolution unit. The core idea is to take advantage of the channel redundancy of the feature map, perform spatial convolution operation on only some channels, and keep the other channels unchanged, thereby significantly reducing the amount of computation.

[0051] Example 2 like Figure 2 As shown, the preset Partial-Conv Lightweight Network PCLNet adopts a U-Net-style encoder-decoder architecture, including a feature extraction layer (Stem), an encoder, and a decoder. The feature extraction layer is used to receive the input image. The encoder is connected to the feature extraction layer, and the encoder and decoder are skip connections. The encoder includes a Partial Convolution (PConv) module, which performs convolution operations on a portion of the input image.

[0052] The input layer receives an infrared input image of size H×W×C, which is then mapped to a 16-channel feature space using a 3×3 convolution in the Stem layer. The encoder path consists of five sequentially connected coding blocks: the first coding block, the second coding block, the third coding block, the fourth coding block, and the fifth coding block, namely Enc1 to Enc5. The number of channels in the Stem layer to Enc5 are configured as 16, 16, 32, 64, 96, and 128, respectively. Max pooling is used between modules to perform a 2x downsampling to progressively expand the receptive field.

[0053] The corresponding decoder path contains five decoder blocks connected in series: the first decoder block, the second decoder block, the third decoder block, the fourth decoder block, and the fifth decoder block Dec4 to Dec0. The number of channels is configured as 16, 64, 32, 16, and 16. Each module performs upsampling by 2 times through bilinear interpolation to restore the spatial resolution.

[0054] To fully utilize the multi-scale features of the encoder, the network establishes skip connections between corresponding layers of the encoder and decoder, namely between the first and fourth decoding blocks, the second and third decoding blocks, the third and second decoding blocks, and the fourth and first decoding blocks. A Local Contrast Enhancement Module (LCEM) is inserted into each connection to enhance the features of small infrared targets. The network ultimately outputs single-channel prediction maps at four different scales: Dec0, Dec2, Dec3, and Dec4. After upsampling and alignment, these maps are fused to generate the final detection result. The total number of parameters in the entire network is 0.211M, and the computational cost is 0.932 GFLOPs.

[0055] The partially convolutional lightweight network PCLNet includes the Partial Convolution (PConv), Efficient Channel Attention (ECA), FastBlock, Strip Attention, Convolutional Block Attention (CBAM), and Local Contrast Enhancement (LCEM).

[0056] The partially convolutional module PConv is the core module of this invention for reducing computational load. The structure diagram of the partially convolutional module PConv is shown below. Figure 3 As shown, traditional convolution performs a 3×3 convolution operation on all channels, while PConv only performs convolution on one-quarter of the channels, leaving the remaining three-quarters unchanged or only performing simple pooling. Specifically, the input image X is first split into two parts, X1 and X2, along the channel dimension, where X1 occupies C / 4 channels and X2 occupies 3C / 4 channels. Then, a standard 3×3 convolution is performed on X1 to obtain X1_conv, while X2 is either left unchanged or, if downsampling is needed, is averaged to obtain X2_identity. Finally, X1_conv and X2_identity are concatenated along the channel dimension to restore the C-channel output. This design makes PConv's computational cost only about 6.25% of that of traditional convolution, theoretically reducing FLOPs by 93.75%. Experiments show that while significantly reducing computational cost, PConv actually brings a 1.83% improvement in accuracy, thanks to the regularization effect of partial convolution and more efficient feature extraction capabilities.

[0057] The partial convolution module performs convolution operations on a portion of the input image channels, including: S210. According to a preset segmentation ratio, the first number channel of the first image input to a partial convolution module is divided into convolution channels and non-convolution channels, and the sum of the convolution channels and the non-convolution channels is the first number channel. S220. Convolve the feature maps within the convolution channels of the first image to obtain a convolution feature map; S230. Pool the feature map in the non-convolutional channel of the first image or keep it unchanged to obtain a non-convolutional feature map. S240. The convolutional feature map and the non-convolutional feature map are concatenated according to the dimension of the first number of channels to obtain a partial convolutional image.

[0058] First, input feature map X∈R C×H×W Divide into two parts according to the channel dimension, as shown in the following formula:

[0059] Where C is the number of input channels, n is the segmentation ratio (default n=4), and X conv ∈R C / n×H×W For the channels participating in convolution, X identity ∈R (C-C / n)×H×W For channels that remain unchanged.

[0060] Only for X conv Perform a 3×3 spatial convolution, as shown in the following formula:

[0061] Where W∈R C / n×3×3 The weights are the convolution kernel weights, b∈R C / n Here, (i, j) represents the bias term, and (i, j) represents the spatial coordinates. In the formula, m and n are summation variables, and ∑_{m=-1}^{1} represents the summation of m taking the values ​​of -1, 0, and 1 in sequence (corresponding to the spatial offset of the 3×3 convolution kernel). When downsampling (stride > 1) is required, for X identity Average pooling is performed to align the spatial dimensions, as shown in the following formula:

[0062] Where s is the step size.

[0063] Finally, the two parts are concatenated along the channel dimension, as shown in the following formula:

[0064] Among them, X out ∈ R C×H' ×W' H' = H / s, W' = W / s. This design reduces the computational cost of a 3×3 convolution to 1 / n² of the original.

[0065] An Efficient Channel Attention (ECA) module is introduced to enhance the network's ability to focus on important channel features. ECA uses one-dimensional convolution to achieve local cross-channel interaction, avoiding the parameter overhead of fully connected layers. The module structure is as follows: Figure 3 As shown, the specific steps are as follows: First, consider the input feature map X ∈ R. C×H×W Global average pooling is performed to compress the spatial information of each channel into a scalar, as shown in the following formula:

[0066] Where, y ∈ R C Here, c is the global descriptor for each channel, and c ∈ {1,2, ..., C} is the channel index.

[0067] The size of the one-dimensional convolutional kernel is adaptively determined so that it is logarithmically related to the number of channels, as shown in the following formula:

[0068] Where γ=2, b=1, and odd indicates taking the nearest odd number. This design allows for the use of a larger receptive field when there are many channels.

[0069] Then, one-dimensional convolution and sigmoid activation are performed to generate channel attention weights, as shown in the following formula:

[0070] Where σ(·) is the Sigmoid function, , θ ∈ R k Let w be the parameters of a one-dimensional convolution kernel, ∈ R. C This is the channel attention weight vector.

[0071] Finally, the channel weights are multiplied by the original features channel by channel, as shown in the following formula:

[0072] Where X out ∈ R C×H×W This is the output feature map after channel attention weighting.

[0073] Each encoding block and each decoding block includes a FastBlock, which consists of partial convolutions, 1×1 convolutions, batch normalization (BN), an efficient channel attention module (ECA), and residual connections, forming an efficient feature extraction unit. The specific steps are as follows: The main path sequentially performs partial convolution, 1×1 convolution, batch normalization, ECA attention, and ReLU activation, as shown in the following formula:

[0074] Among them, PConv is partial convolution, Conv1×1 is 1×1 point convolution used for inter-channel information interaction, BN is batch normalization, and ECA is efficient channel attention.

[0075] The shortcut branch performs identity mapping or channel / size alignment, as shown in the following formula:

[0076] The final output is the sum of the two branches, as shown in the following formula:

[0077] Residual connections enable gradients to be directly backpropagated, alleviating the gradient vanishing problem in deep networks.

[0078] The fourth and fifth coding blocks include the Convolutional Block Attention Module (CBAM) and the fifth coding block includes the Strip Attention Module.

[0079] To enhance the feature response to extremely small targets, a CBAM module is introduced deep into the network. CBAM combines channel attention and spatial attention in series. The specific steps are as follows: The channel attention branch performs global average pooling and global max pooling on the input image X to capture different types of global information. Then, feature transformation is performed through a two-layer MLP (Multi-Layer Perceptron) network with shared parameters, as shown in the following formula:

[0080]

[0081]

[0082] Where W1∈ R C / r×C and W2∈R C×C / rHere, r = 16 represents the compression ratio, GAP and GMP represent global average pooling and global max pooling, respectively, and Mc ∈ R. C This represents the channel attention weight.

[0083] The spatial attention branch performs average pooling and max pooling along the channel dimension on the features after channel attention weighting, generating two single-channel feature maps, as shown in the following formula:

[0084]

[0085]

[0086]

[0087] Where ⊙ represents element-wise multiplication, M s ∈ R H×W For spatial attention weights.

[0088] The final output is the result of channel attention and spatial attention applied sequentially, as shown in the following formula: .

[0089] Small infrared targets may exhibit weak, continuous responses in the horizontal or vertical directions (such as aircraft contrails). A strip attention module is designed to capture this directional contextual information. The specific steps are as follows: First, the number of channels is halved using a 1×1 convolution to reduce the amount of subsequent computation, as shown in the following formula:

[0090] Among them, W reduce ∈ R C / 2×C To reduce the dimensionality of convolution weights, X reduce ∈ R C / 2×H×W This is the feature map after dimensionality reduction.

[0091]

[0092]

[0093] Among them, W. h ∈ R C / 2×1×3 and W w ∈ R C / 2×3×1 These are strip convolution kernels in the horizontal and vertical directions, respectively.

[0094] The responses from both directions are fused to generate a spatial attention map, as shown in the following formula:

[0095] Where σ(·) is the Sigmoid function, A∈R C / 2×H×W This is the fused spatial attention map.

[0096] Finally, the number of channels is restored using a 1×1 convolution and then connected to the original feature residuals, as shown in the following formula:

[0097] Among them, X out ∈ R C×H×W This is the enhanced output feature map.

[0098] Implementation of the Local Contrast Enhancement Module (LCEM). A core physical characteristic of small infrared targets is "high local contrast," meaning the target's pixel brightness is significantly higher than the surrounding background. The LCEM enhances this contrast feature through explicit calculations, such as... Figure 4 As shown, the local contrast enhancement module first uses 3×3 average pooling to calculate the local average background X_bg = AvgPool(X). Since the neighborhood around the small target pixel is mostly background, the average value can be approximated as the background brightness. Then, the difference between the original feature and the background is calculated and ReLU activation is used to obtain the contrast feature X. contrast = ReLU(X - X bg The ReLU algorithm preserves positive values ​​where the target is brighter than the background, while suppressing potential noise and negative values. Adaptive weights W = Sigmoid(Conv1×1(X) are then generated using a 1×1 convolution and the Sigmoid function. contrast The weights, ranging from 0 to 1, represent the reliability of the contrast features. Finally, the weighted contrast features are superimposed onto the original features, i.e., X. out =X+ W X contrast This residual connection method preserves the original information and enhances the target response.

[0099] The specific steps are as follows: First, the local background intensity is estimated using 3×3 average pooling, as shown in the following formula:

[0100] Among them, X bg ∈ R C×H×W Let c be the estimated local background feature map, c ∈ {1,2, ..., C} be the channel index, and (i,j) be the spatial coordinates.

[0101] Calculate the contrast feature, which is the difference between the original feature and the local background, retaining only the positive difference (the part of the target that is brighter than the background), using the following formula:

[0102] Here, max(0, ·) is the ReLU function, used to suppress negative differences.

[0103] Adaptive boosting weights are generated using 1×1 convolution and sigmoid activation, as shown in the following formula:

[0104] Where, θ∈R C×C The weights are 1 × 1 convolution weights, b ∈ R. C For bias, W∈R C×H×W For adaptive weights.

[0105] The enhanced output features are then added back to the original features as residuals, using the following formula:

[0106] The local contrast enhancement module models the physical characteristics of small infrared targets, effectively enhancing the response of the target area.

[0107] PCLNet employs a U-Net-style architecture with a 5-layer encoder and a 5-layer decoder. The encoder extracts multi-scale features layer by layer, and the decoder restores spatial resolution layer by layer. Skip connections fuse the features from the encoder and decoder. The encoder consists of Stem layers and Enc1-Enc5, while the decoder consists of Dec0-Dec4. Enc4 and Enc5 introduce CBAM modules, Enc5 introduces a strip attention module, and LCEM modules are introduced at the skip connections of each decoder layer.

[0108] A multi-scale feature fusion strategy is employed to fully utilize feature information at different resolutions. For example... Figure 5 As shown, the network outputs single-channel prediction maps in four decoder layers: Dec0, Dec2, Dec3, and Dec4, with resolutions of 1 / 1, 1 / 4, 1 / 8, and 1 / 16 of the original image, respectively. The lower-resolution outputs are upsampled to the original image size using bilinear interpolation and then concatenated with the highest-resolution output along the channel dimension. Finally, a 3×3 convolution is used to fuse the results and generate the final prediction. To further enhance the network's learning ability, an auxiliary loss is applied to the intermediate-scale outputs during training, with a weight α set to 0.2. This deep supervision mechanism allows intermediate layers to receive direct supervision signals. Experiments show that deep supervision not only improves mIoU by 1.73 percentage points but also accelerates convergence by 19.2%, significantly improving training efficiency.

[0109] Specifically, the following steps are included: The four scales Dec0, Dec2, Dec3, and Dec4 are each converted into predicted images through 1 × 1 convolutions, as shown in the following formulas:

[0110] Among them, O i ∈ R 1×H i ×W i This is the prediction map for the i-th scale.

[0111] The predictions at each scale are upsampled to the original resolution and then stitched together, as shown in the following formula:

[0112] in, Indicates upsampling by s times, O concat ∈ R 4×H×W This is a multi-scale fusion feature.

[0113] The final output is obtained through a 3×3 convolution, as shown in the following formula:

[0114] Among them, O final ∈ R 1×H×W This is the final prediction graph. During training, auxiliary supervisory losses are applied to O2, O3, and O4.

[0115] Example 3 Step S130, after continuously iteratively optimizing the partially convolutional lightweight network based on a preset loss function, the training set, and the image training results to obtain the trained partially convolutional lightweight network, further includes: S140. Export the trained partially convolutional lightweight network model as an open neural network exchange format. S150: The processing unit based on the pre-installed platform converts the model of the partially convolutional lightweight network in the open neural network exchange format into the corresponding model format.

[0116] Specifically, the trained partially convolutional lightweight network model is exported as the Open Neural Network Exchange (ONNX) format to enable cross-framework deployment. Then, based on the pre-installed platform processing unit, the partially convolutional lightweight network model in the ONNX format is converted into the corresponding model format.

[0117] Domestic edge AI chips, represented by Rockchip's RK3588, are developing rapidly. The RK3588 integrates a 6 TOPS NPU (Neural Processing Unit), employs a three-core heterogeneous design, supports INT8 / FP16 mixed-precision inference, and boasts advantages such as low power consumption, high energy efficiency, and domestic self-control. Compared to GPUs, edge NPUs differ in operator support, memory bandwidth, and parallel scheduling. Directly deploying GPU-optimized networks to an NPU often fails to fully utilize hardware performance and may even lead to deployment failure due to unsupported operators. Therefore, it is necessary to consider the characteristics of edge NPUs from the network design stage, avoiding NPU-unfriendly operations such as dynamic shapes and custom operators. Simultaneously, designing reasonable quantization strategies and multi-core parallel schemes is crucial to fully leverage the advantages of edge computing platforms.

[0118] like Figure 6 As shown, deploying a trained PCLNet model to the RK3588 edge platform requires a complete model conversion and optimization process. The following description uses the complete deployment chain from PyTorch model training to RK3588 edge inference as an example. The overall process is as follows: PyTorch model (.pth) → ONNX model (.onnx) → RKNN model (.rknn) → RK3588 NPU inference.

[0119] Step 1: Export the PyTorch model to ONNX format. After training, use the `torch.onnx.export` interface to export the PyTorch model to the ONNX (Open Neural Network Exchange) intermediate format for cross-framework deployment. During export, fix the input size to 256×256 or 512×512 to avoid dynamic shapes affecting NPU inference efficiency; select operator version `opset 12` to ensure compatibility with the RKNN toolchain and obtain optimal RK3588 NPU support; and enable constant folding optimization to reduce runtime computation. After export, use ONNX Runtime to verify accuracy and ensure lossless conversion.

[0120] Step 2: Convert the ONNX model to RKNN format. Use Rockchip's official RKNN-Toolkit2 toolkit to convert the ONNX model to the RKNN format optimized for NPU. During conversion, configure the target platform as rk3588 and set the mean / std parameters to allow the NPU hardware to directly perform input normalization, reducing CPU overhead. The build phase supports two quantization strategies: FP16 and INT8. FP16 conversion is simple and has minimal accuracy loss; INT8 requires 50-100 calibration samples, can compress the model by 4 times, and improve inference speed by 2-3 times. Table 1 shows a comparison of model size and accuracy for different quantization strategies.

[0121] Table 1 Comparison of Model Size and Accuracy for Different Quantization Strategies

[0122] The FP16 quantized model size is 2.57MB, with a negligible mIoU loss of only 0.32 percentage points compared to the original FP32 model. INT8 quantization further compresses the model to 2.31MB; although the mIoU loss increases to 1.24 percentage points, it remains within an acceptable range. After loading the model using the RKNNLite runtime library and initializing the NPU core on the board, inference can begin. RK3588 single-core NPU inference performance (input 256...) 256 3) As shown in Table 2.

[0123] Table 2. RK3588 Single-Core NPU Inference Performance

[0124] Test results show that the average inference time of the FP16 quantization model is 55.8 milliseconds, corresponding to a frame rate of 17.9 FPS. The INT8 quantization model reduces the inference time to 38.7 milliseconds and increases the frame rate to 25.8 FPS. Evaluation on 532 NUDT test set images shows that the accuracy loss after INT8 quantization is minimal, with mIoU decreasing by only 0.10 percentage points. The detection probability Pd is even slightly improved, and the false alarm rate Fa is also reduced, fully validating the effectiveness of the INT8 quantization strategy.

[0125] Step 3: Parallel inference using a multi-core NPU. For example... Figure 7As shown, the RK3588 integrates three NPU cores with a total computing power of 6 TOPS. Theoretically, inference throughput can be further improved through multi-core parallelism. This invention adopts a serial initialization strategy, initializing each core sequentially with a 500-millisecond interval to avoid resource contention caused by simultaneous initialization of multiple cores. A pipelined parallel scheduling strategy is designed, allocating consecutive video frames to different NPU cores for processing according to their sequence numbers. Each core independently loads model instances and executes inference through an independent thread to avoid resource contention between cores. During inference, an independent thread is created for each core, using a pipelined scheduling method: when Core0 infers the i-th frame, Core1 and Core2 simultaneously process the i+1 and i+2 frames, which can increase the throughput to 2-2.5 times that of a single core. Preliminary tests show that in the multi-core collaborative working mode, the utilization rates of the three NPU cores reach 68%, 20%, and 20%, respectively, and the total system throughput can be maintained at 34.8 FPS.

[0126] In actual deployment tests, the INT8 quantization model was used to process 256×256 resolution infrared images, and the single inference time remained stable at around 38.7 milliseconds, which can meet the requirements of real-time detection. Tests showed that the system can maintain stable detection performance during continuous operation, with a target false detection rate of less than 2.07% and a false alarm rate controlled at a low level of 4.27×10^-5, fully verifying the practicality of the system on the edge platform.

[0127] Step 4: Post-processing optimization. Utilizing the monotonicity of the Sigmoid function, the network output is directly evaluated as x>0 instead of Sigmoid(x)>0.5, avoiding pixel-by-pixel exponential operations. Subsequently, 8-neighborhood connected component analysis is used to extract the target, and noise points are filtered based on pixel area thresholds.

[0128] Example 4 The NUDT-SIRST dataset contains 1327 256×256 resolution infrared images and their pixel-level labeled masks, divided into training and test sets in a 1:1 ratio. During training, a 256×256 input size and a batch size of 16 were used. The learning rate was gradually decayed from an initial 1e-3 to a minimum of 1e-5 using a cosine annealing scheduling strategy. A warmup strategy was used for the first 10 epochs (a complete model training process based on all training data) to stabilize the initial training phase. The entire training process lasted 300 epochs, using the Adam optimizer with a gradient clipping threshold of 0.5 to prevent gradient explosion. A combination of BCE and Dice Loss was used as the loss function, with an auxiliary loss of 0.2 applied to the multi-scale outputs for deep supervision. Data augmentation strategies included random horizontal flipping, vertical flipping, and rotation to improve the model's generalization ability. The entire training was performed on a host equipped with an NVIDIA RTX 4070ti Super graphics card and took approximately one hour.

[0129] Evaluation on 663 images in the NUDT-SIRST test set showed that PCLNet achieved an mIoU of 83.88% and an F1 score of 91.23%, with pixel-level precision and recall reaching 92.77% and 89.74%, respectively. More importantly, in terms of target-level metrics, the network achieved a detection probability Pd of 96.30%, a false alarm rate Fa of only 2.748 × 10^-5, an average inference time of 4.27 milliseconds, and a frame rate of 234.40 FPS. To comprehensively evaluate the model's robustness, threshold scanning experiments were conducted. The results show that PCLNet can maintain a stable detection probability above 96% within a wide threshold range of 0.1 to 0.9, fully demonstrating that the network output has high confidence and predictive stability.

[0130] Example 5 The IRSTD-1k dataset contains 1001 infrared images with a resolution of 512×512. Compared to NUDT-SIRST, this dataset has smaller target sizes and more complex background clutter, significantly increasing the detection difficulty. The average target size in the dataset is only 5×5 pixels, and it contains a large amount of strong clutter and cloud interference, placing higher demands on the robustness of the algorithm. The training configuration is basically the same as in Example 4, with the main adjustment being to increase the input size to 512×512 to accommodate higher resolution images, while reducing the batch size to 4 due to memory limitations.

[0131] Test results show that PCLNet achieved a 58.70% mIoU and a 73.98% F1 score on this dataset. Although the pixel-level metrics are relatively low, the object-level detection probability Pd remains high at 93.27%, with a false alarm rate Fa of 1.799 × 10^-5 and an inference frame rate of 141.82 FPS. The lower pixel-level metrics are mainly due to the extremely small target size, which amplifies the impact of prediction errors on IoU from boundary pixels. However, in terms of object-level detection probability, PCLNet still maintains excellent detection performance. This fully demonstrates the network's good generalization ability and robustness when facing more challenging scenarios.

[0132] This embodiment achieves excellent performance on both the NUDT-SIRST and IRSTD-1k datasets by combining innovative modules such as partial convolution, local contrast enhancement, and multi-scale feature fusion. In particular, its successful deployment on the RK3588 edge platform demonstrates the feasibility and practical value of this invention in real-world engineering applications.

[0133] In some embodiments of this application, the lightweight target detection model generation device can be implemented as a computer program, and the computer program can be implemented in, for example... Figure 8 The device operates on the electronic device shown. The memory of the electronic device can store various program modules that constitute the lightweight target detection model generation apparatus. The computer program composed of these program modules causes the processor to execute the steps in the image detection methods of the various embodiments of this application described in this specification.

[0134] The electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external electronic devices via a network connection. When the computer program is executed by the processor, it implements a lightweight target detection model generation method.

[0135] Accordingly, this application also provides an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Alternatively, the electronic device can be a server.

[0136] The electronic device includes one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processors of the steps of the multi-sensor-based monitoring method described above. The steps of the multi-sensor-based monitoring method described here can be steps from the lightweight target detection model generation method of the various embodiments described above.

[0137] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the above embodiments are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A lightweight target detection model generation method, characterized in that, include: Obtain infrared image dataset; The images in the training set of the infrared image dataset are input into a preset partially convolutional lightweight network for training to obtain image training results. The preset partially convolutional lightweight network only performs convolution operations on a portion of the input images. Based on the preset loss function, the training set, and the image training results, the partially convolutional lightweight network is continuously iteratively optimized to obtain the trained partially convolutional lightweight network.

2. The lightweight target detection model generation method according to claim 1, characterized in that, The preset partially convolutional lightweight network includes a feature extraction layer, an encoder, and a decoder. The feature extraction layer is used to receive the input image. The encoder is connected to the feature extraction layer and is skipped between the encoder and the decoder. The encoder includes a partially convolutional module, which performs convolution operations on a portion of the input image channels.

3. The lightweight target detection model generation method according to claim 2, characterized in that, The partial convolution module performs convolution operations on a portion of the input image channels, including: According to a preset segmentation ratio, the first number of channels of the first image input to a portion of the convolution module are divided into convolution channels and non-convolution channels, and the sum of the convolution channels and the non-convolution channels is the first number of channels; Convolutional feature maps are obtained by convolving the feature maps within the convolutional channels of the first image; Pooling or keeping the feature map in the non-convolutional channel of the first image unchanged yields a non-convolutional feature map. The convolutional feature map and the non-convolutional feature map are concatenated according to the dimension of the first number of channels to obtain a partial convolutional image.

4. The lightweight target detection model generation method according to claim 2, characterized in that, The encoder includes a first coding block, a second coding block, a third coding block, a fourth coding block, and a fifth coding block connected in series for downsampling. The decoder includes a first decoding block, a second decoding block, a third decoding block, a fourth decoding block, and a fifth decoding block connected in series for upsampling. The first coding block is connected to the feature extraction layer, and the fifth coding block is connected to the first decoding block. Skip connections are established between the first coding block and the fourth decoding block, between the second coding block and the third decoding block, between the third coding block and the second decoding block, and between the fourth coding block and the first decoding block.

5. The lightweight target detection model generation method according to claim 4, characterized in that, Local contrast enhancement modules are inserted into each connection between corresponding layers of the encoder and the decoder.

6. The lightweight target detection model generation method according to claim 4, characterized in that, Each of the coded blocks and each of the decoded blocks includes a fast feature block, which includes partial convolution, 1×1 convolution, batch normalization, efficient channel attention module and residual connection.

7. The lightweight target detection model generation method according to claim 4, characterized in that, The fourth and fifth coding blocks include convolutional block attention modules, and the fifth coding block includes strip attention modules.

8. The lightweight target detection model generation method according to claim 1, characterized in that, After iteratively optimizing the partially convolutional lightweight network based on a preset loss function, the training set, and the image training results to obtain the trained partially convolutional lightweight network, the process further includes: The trained partially convolutional lightweight network model is exported as an open neural network exchange format; The pre-installed platform processing unit converts the partially convolutional lightweight network model of the open neural network exchange format into the corresponding model format.

9. A lightweight target detection model analysis method, characterized in that, A lightweight target detection model generated by the lightweight target detection model generation method according to any one of claims 1 to 8, the method comprising: Input the image to be detected into the lightweight target detection model; The first decoding block, the second decoding block, the third decoding block, and the fifth decoding block each output a single-channel prediction map, which is then fused after upsampling and alignment to generate a detection result.

10. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps of the lightweight target detection model generation method according to any one of claims 1 to 8.

Images