Self-adaptive multispectral pedestrian detection method

Through the adaptive multispectral pedestrian detection network, the multispectral data and adaptive optimization mechanism are used to solve the problem of poor pedestrian detection performance in complex lighting environments, and efficient and accurate pedestrian detection is achieved, which improves detection accuracy and robustness. sex.

CN119068513APending Publication Date: 2024-12-03INNER MONGOLIA UNIV OF SCI & TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411259049.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing pedestrian detection technology performs poorly in complex lighting environments, especially in night or complex lighting conditions, and it is difficult to effectively integrate multimodal features, resulting in unstable detection performance.

Method used

An adaptive multispectral pedestrian detection method was designed. By introducing multispectral data and adaptive optimization mechanisms, the channel attention mechanism and spatial attention mechanism were improved, and an adaptive multispectral pedestrian detection network (ASP) was constructed, using The channel control switch module and the cross-modal fusion module dynamically adjust the characteristic channel weights and monitor the light intensity in real time, and automatically switch the appropriate fusion strategy.

Benefits of technology

It significantly improves the accuracy and robustness of multi-spectral pedestrian detection, realizes efficient image recognition and processing in different lighting environments, improves the accuracy and real-time detection, and ensures the stability and reliability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068513A_ABST
    Figure CN119068513A_ABST
Patent Text Reader

Abstract

The invention relates to a self-adaptive multispectral pedestrian detection method, which comprises the following steps of: in an image preprocessing stage, firstly obtaining a KAIST data set, then adjusting to-be-trained images to be uniform in size and classifying the to-be-trained images according to illumination intensity; in the model building and training stage, the classified images are input into the multispectral pedestrian detection network obtained through construction for training; and in the model test stage, the trained multispectral pedestrian detection network is adopted to generate a pedestrian detection result in real time according to the input image. According to the method, multispectral data and an adaptive optimization mechanism are introduced, a channel attention mechanism and a space attention mechanism are improved, and a multispectral pedestrian detection network (ASP) based on adaptation is designed, so that information of different modes can be utilized more effectively, richness and distinction of features are enhanced, complementation of the information of the different modes is realized, and the accuracy of pedestrian detection is improved. Uniform fusion of multispectral data is realized, the robustness of the model is improved, and the recognition precision in a complex scene is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, in particular to an adaptive multi-spectral pedestrian detection method. Background Art

[0002] Most current pedestrian detection technologies rely on information from a single spectrum and face challenges such as illumination changes and occlusion, especially poor performance at night or in complex lighting environments. Multispectral pedestrian detection based on adaptive fusion aims to effectively fuse multimodal features to improve detection performance. According to the different band information in the multispectral image, it is integrated into a comprehensive model to improve the fusion of different modal data features and improve the accuracy and robustness of detection. However, how to perform effective feature fusion and how to better identify pedestrians in multispectral images are still complex issues that require in-depth research and innovative methods. Among them: The selection of multispectral image fusion strategy is a key issue that needs to be solved in adaptive fusion multispectral pedestrian detection. The inability to access the precise selection of fusion strategy during training results in poor test results of the training model at different light intensities. The appropriate fusion strategy is an important factor that leads to differences in pedestrian image features under different light intensities in the dataset. Summary of the invention

[0003] In view of the defects of the existing technology that are insufficient in processing complex scenes and lack cross-modal consistency, the present invention proposes an adaptive multispectral pedestrian detection method. By introducing multispectral data and an adaptive optimization mechanism and improving the channel attention mechanism and the spatial attention mechanism, an adaptive multispectral pedestrian detection network (ASP) is designed. The method can more effectively utilize information from different modalities, enhance the richness and distinctiveness of features, realize the complementarity of information from different modalities, achieve consistent fusion of multispectral data, improve the robustness of the model, and greatly improve the recognition accuracy in complex scenes.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a multispectral pedestrian detection method based on adaptation. In the image preprocessing stage, the KAIST data set is firstly obtained, and then the images to be trained are adjusted to a uniform size and classified according to the illumination intensity; in the model building and training stage, the classified images are input into a constructed multispectral pedestrian detection network for training; in the model testing stage, the trained multispectral pedestrian detection network is used to generate pedestrian detection results in real time according to the input images.

[0006] The classification by light intensity refers to: based on the illumination unit in the International System of Units (SI): lux (lumens / square meter), 1000 lux or more is strong light, 100-500 lux is weak light, and 0-100 lux is no light range threshold value is divided into three categories: strong light, weak light, and no light.

[0007] The multispectral pedestrian detection network includes: a channel control switch module (SE), a category balanced sampling module (CBS), an efficient long-distance network (ELAN), a first downsampling module (MP-1), a second downsampling module (MP-2), a spatial pyramid pooling and channel pyramid pooling module (SPPCSPC), a connection layer (Concat), an upsampling layer (Upsample), a reconvolution module (RepConv), a detection layer (Detect), a cross-modal fusion module (CMFM) and a feature enhanced attention module (EFAM). Technical Effects

[0008] The present invention can automatically select the optimal fusion strategy in low-light, strong-light and no-light environments through two channel control switch modules, thereby ensuring efficient image recognition and processing effects. The module can dynamically adjust the weights of feature channels, significantly improving the accuracy and robustness of multi-spectral pedestrian detection; through real-time monitoring of light intensity by an adaptive light intensity regulation mechanism, a multi-level fusion strategy is adopted, that is, early fusion is adopted in low-light conditions, mid-term fusion is adopted in strong-light conditions, and late fusion is adopted in no-light conditions, so as to maximize the utilization of multi-spectral image information; through an adaptive multi-spectral detection network, based on ambient light intensity and target detection requirements, the appropriate multi-spectral fusion strategy is automatically switched to ensure the accuracy and real-time performance of image processing.

[0009] Compared with the prior art, the present invention significantly improves the accuracy and robustness of multi-spectral pedestrian detection. First, with the help of two channel control switch SE modules, the optimal fusion strategy can be automatically selected in low-light, strong-light and no-light environments, thereby ensuring that the system can achieve efficient image recognition and processing effects under various lighting conditions. Secondly, through the real-time monitoring of light intensity by the adaptive light intensity regulation mechanism and the adoption of a multi-level fusion strategy, the system can perform early fusion under low-light conditions, mid-term fusion under strong-light conditions, and late fusion under no-light conditions. This design greatly improves the utilization rate of multi-spectral image information. In addition, through the adaptive multi-spectral detection network, the system can automatically switch to a suitable multi-spectral fusion strategy according to the ambient light intensity and target detection requirements, thereby ensuring the accuracy and real-time performance of image processing. The synergistic effect of these technical means enables the present invention to exhibit high stability and reliability under different lighting environments, significantly improving the overall performance of multi-spectral pedestrian detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flow chart of the present invention;

[0011] Figure 2 This is a diagram of the network structure for multi-spectral pedestrian detection;

[0012] Figure 3 This is a schematic diagram of the principle of the channel switch control module;

[0013] Figure 4 It is a schematic diagram of the multi-spectral feature fusion module;

[0014] Figure 5 This is a schematic diagram of a multi-spectral feature fusion enhancement module;

[0015] Figure 6 This is an example effect diagram. DETAILED DESCRIPTION

[0016] like Figure 1 As shown, this embodiment relates to an adaptive multi-spectral pedestrian detection method, including:

[0017] Step 1: Obtain the public multispectral pedestrian detection dataset KAIST and divide it into three types of datasets according to light intensity: strong light, weak light, and no light. At the same time, divide the training set, validation set, and test set of the three types of datasets into 8:2:1; construct Figure 2 The multispectral pedestrian detection network shown.

[0018] Step 2: Input the divided training set into the multispectral pedestrian detection network, train it to obtain an adaptive multispectral pedestrian detection network model, and obtain a trained multispectral pedestrian detection model based on adaptive fusion, which specifically includes:

[0019] Step 2.1: To ensure that the model is fully trained, a total of 200 epochs are set. Each iteration will perform a complete weight update through the training data.

[0020] Step 2.2: In each iteration, the model optimizes its prediction accuracy by minimizing the cross entropy loss function. For multi-classification tasks, the cross entropy loss function is: in: N is the number of samples; C is the number of categories; y i,c is the true label (one-hot encoding) of the cth category corresponding to the ith sample; is the model's predicted probability that the ith sample belongs to the cth category.

[0021] Step 2.2.1. Input the RGB+IR images of three light intensities into the multispectral pedestrian detection network. First, enter the first SE(a) module to determine whether the light intensity is strong light. When the strong light condition is reached, the left path is suppressed, and the RGB+IR directly enters the right path. After four CBSs for feature extraction, it is input into the second SE(b) module. According to the weight bias data of the first SE(a), the right path is maintained downward. Through three groups of ELAN modules and MP-1 modules, the first ELAN module further aggregates and enhances the features and outputs them to the MP-1 module. The MP-1 module downsamples the feature map through pooling operation, retains the main significant features, and outputs them to the fourth ELAN module. Continue downward, where the second ELAN and The third ELAN passes the aggregated and enhanced features to four CBSs in parallel to extract and enhance the features, and finally outputs a richer feature map; if the strong light condition is not met, RGB+IR enters two paths respectively, and is extracted through four CBSs, and is input into the second SE(b) module to determine whether the lighting condition is a no-light condition. When the no-light condition is met, RGB and IR pass through three groups of ELAN modules and MP-1 modules respectively, among which: the first ELAN module performs progressive aggregation and enhancement on the features, and outputs them to the MP-1 module. The MP-1 module downsamples the feature map through pooling operation, retains the main significant features, and outputs them to the CMFM module to splice and fuse the multimodal features, and outputs a more integrated feature map for processing by the next layer of ELAN modules. The EFAM module processes the features output from multiple ELAN modules and CMFM modules, further focuses on and enhances the key features, and ensures that important information is not diluted in the subsequent processing process. The second and third ELANs pass the aggregated and enhanced features to the four CBSs in parallel to extract and enhance the features, and finally output a richer feature map. The features aggregated and enhanced at the two fourth ELAN modules are spliced ​​and passed to the SPPCSPC module; if the light-free condition is not met, RGB+IR enters the two channels respectively. After feature extraction by the four CBSs, RGB and IR are further aggregated and enhanced by the three groups of ELAN modules and the MP-1 module: the ELAN module further aggregates and enhances the features. And output it to the MP-1 module. The MP-1 module downsamples the feature map through pooling operation, retains the main significant features, and outputs it to the fourth ELAN module, and continues to proceed downward. At the same time, the second ELAN and the third ELAN of the two channels respectively pass the aggregated and enhanced features into the two channels in parallel for two CBS modules to extract and enhance the features, and pass them into the CMFM module for RGB and IR feature information fusion to enhance the feature expression ability, and output the fused feature map to the EAEM module to enhance the key feature information, suppress irrelevant information and output the enhanced feature map to the last layer of CBS module for convolution operation and concatenate the two features into the Concat module to prepare for subsequent fusion.

[0022] Step 2.2.2, the SPPCSPC module receives the features from the fourth ELAN module, performs multi-scale feature fusion, and outputs the fused feature map. The feature map is then passed to the next CBS module, where a convolution operation is performed to further extract key features. The extracted feature map then enters the upsampling module, which increases the resolution of the feature map through spatial upsampling, and then outputs it to the Concat module. In the Concat module, the upsampled feature map is spliced ​​with the feature map of the early fusion point to form a richer and more complete feature map. Next, the ELAN module performs deep feature extraction on the feature map spliced ​​by the Concat module, and passes the processed feature map to the RepConv module. The RepConv module further optimizes these features through repeated convolutions and finally outputs a high-quality feature map for detection. At the same time, the feature map output from the ELAN module is also downsampled by the MP-2 module. This process retains key features. Although the resolution is reduced, it prepares for the subsequent Concat module to ensure effective fusion and processing of features.

[0023] Step 2.2.3, the Detect module finally processes the feature maps from multiple RepConv modules to generate the final target detection results, including the location and category of the target.

[0024] Step 3: Input the test set into the trained multispectral pedestrian detection network to generate pedestrian detection results based on the input image to test the model effect.

[0025] like Figure 2 As shown, the multispectral pedestrian detection network includes: a channel control switch module (SE), a category balanced sampling module (CBS), an efficient long distance network (ELAN), a first downsampling module (MP-1), a second downsampling module (MP-2), a spatial pyramid pooling and channel pyramid pooling module (SPPCSPC), a connection layer (Concat), an upsampling layer (Upsample), a reconvolution module (RepConv), a detection layer (Detect), a cross-modal fusion module (CMFM) and a feature enhanced attention module (EFAM), wherein:

[0026] Two SEs and two groups of CBS constitute the channel segmentation and feature extraction part. The first SE acts as a channel control switch to control the input RGB and IR image data to flow to the two groups of CBS branches; the first and second CBS branches extract features from the image respectively; the second SE performs inter-channel feature weight adjustment operation;

[0027] Four ELANs, six MP-1s, two CMFMs and one EFAM constitute the multi-scale feature fusion part. The first to third ELANs aggregate and enhance the extracted features and downsample the feature maps through the pooling operation of MP-1; CMFM performs cross-modal feature fusion operations; and the fourth ELAN performs the final feature integration operation.

[0028] Four groups of CBS, four CMFMs and two EFAMs constitute two identical multi-scale feature fusion parts. Each part processes feature maps of different scales and performs cross-modal feature fusion to further enhance the saliency of features.

[0029] Two Concat modules, one Upsample module, two RepConv modules and one Detect module constitute the final detection and output part. The Concat module concatenates feature maps of different scales, the Upsample module improves the resolution by spatial upsampling, and the RepConv module further optimizes the features. Finally, the Detect module generates the final pedestrian detection result.

[0030] The first MP-2 module performs downsampling to retain key features and reduce resolution in preparation for the subsequent first Concat module; the SPPCSPC module processes features from the fourth ELAN module, outputs feature maps after multi-scale feature fusion, and enters the second CBS module for feature map convolution to further extract features; the first Concat module splices the upsampled features with the feature maps of early fusion points to form a richer feature map; the first Upsample module increases the resolution of the feature map through spatial upsampling; the first RepConv module further optimizes features through repeated convolution and outputs the final detection feature map.

[0031] The second MP-2 module also performs downsampling to retain key features and prepares for the subsequent second Concat module; the second Concat module combines previous feature information for splicing to form a richer feature map; the second Upsample module improves the resolution of the feature map by upsampling; the second RepConv module further optimizes these features; the Detect module processes the detection feature maps from multiple RepConv modules to generate the final target detection results, including the location and category of the target.

[0032] like Figure 3As shown in the figure, the channel control switch module compresses the dimension of the original feature map from H×W×C to 1×1×C through a global average pooling operation, that is, compresses the original feature map into one dimension in the spatial dimension, and then uses a sigmoid gating mechanism to capture the relationship between channels. Finally, the learned activation value of each channel (sigmoid activation, value 0 to 1) is multiplied by the original feature on U, specifically: ① ② ③ in: H, W, and C represent the height, width, and number of channels of the feature map, respectively; is the feature map on channel c Output after global average pooling (GAP); F sq ( ) is the global average pooling operation; is the original feature map on channel c; The feature map The value at position (i, j); is the result of global average pooling, which is a vector of size C (number of channels); s is the scaling factor of each channel obtained by the sigmoid activation function; Fe x为 Compression and expansion operations of features; σ is the sigmoid activation function; is the result of linear transformation and ReLU activation of z; W1 is the dimensionality reduction weight matrix, size is W2 is the weight matrix of the restored dimension, and its size is ReLU is the ReLU activation function; is the feature map on the scaled channel c; Fscsle is the feature map scaling operation; sc is the scaling factor of channel c; is the original feature map on channel c.

[0033] In order to reduce the complexity of the model and improve the generalization ability, the SE module adopts a bottleneck structure including two fully connected layers (FC), in which the first FC layer plays a role of dimensionality reduction, the dimensionality reduction coefficient is r which is a hyperparameter, and then ReLU activation is adopted. The last FC layer restores the original dimension.

[0034] The channel control switch module specifically includes: a channel attention unit, a convolution unit, an activation unit and a pooling unit, wherein: the channel attention unit receives multispectral image information, including data of RGB and IR channels, and extracts light intensity and scene information based on the input image information, and performs preliminary analysis to determine the applicable fusion strategy; the convolution unit receives preliminary suggestions from the channel attention unit, further refines the fusion strategy in combination with ambient lighting conditions and target detection requirements, generates the final fusion strategy and passes it to the activation unit; the activation unit executes the fusion strategy generated by the convolution unit, and outputs the fused image features by controlling the fusion mode of the RGB and IR image data streams (such as early, mid-term or late fusion); finally, the pooling unit performs deep processing on the fusion result of the activation unit, enhances important features and suppresses noise, and outputs the optimized image feature map as the final multispectral image processing result, pedestrian detection and other tasks.

[0035] The multispectral feature fusion module adopts the channel difference weighted method to obtain the difference features of the two modal feature maps, performs a global average pooling operation (GAP) on the difference features and applies the Sigmoid activation function. The different features of different forms are amplified and fused with the features of another form, and the enhanced infrared features and enhanced RGB features are calculated. Then, the feature maps of the two modalities are subjected to channel attention operation and feature fusion, and the fused features after the infrared feature enhancement and the fused features after the RGB feature enhancement are calculated and the feature maps processed by the CMFM module are output, which are specifically: D =F R -F T , infrared signature F TD =σ(GAP(F D ))⊙F T , enhanced RGB feature F RD =σ(GAP(F D ))⊙F R , fusion features after infrared feature enhancement Fusion features after RGB feature enhancement in: is the RGB convolution feature map, It is the infrared convolution feature map. It is obtained by weighting the channel-by-channel difference of two features; GAP is the global average pooling, σ is the s-type activation function, and ⊙ represents element-wise multiplication. and After feature enhancement, suppression and F R Fusion; || indicates channel-level connection operation, represents the element-wise sum, is the residual function yes With F T Fusion after feature enhancement. for With F R Fusion after feature enhancement.

[0036] like Figure 4 As shown, the multi-spectral feature fusion module includes: convolution unit, activation unit, feature selection unit, feature fusion unit, wherein: RGB and IR images extract preliminary features through their own convolution units respectively, and perform nonlinear mapping through activation unit (Sigmoid), after feature selection unit, RGB and IR features respectively select important features through dot multiplication operation, and at the same time, RGB features and IR features are cross-dot multiplied to strengthen cross-spectral feature fusion. Finally, the fused features are further extracted through convolution units, and element-by-element addition operation is performed with the original RGB and IR features at the output stage to generate the final multi-spectral fusion feature map for subsequent pedestrian detection tasks.

[0037] The feature enhancement attention module decomposes the global average pooling operation into horizontal coordinates and vertical coordinates, and uses maximum pooling to collect important clues about the features of different objects. After calculating the feature map generated along the horizontal direction and the feature map generated along the vertical direction, the two transformed features and the features of the two spatial directions are respectively aggregated to generate a pair of direction-aware feature maps, and the generated feature maps are processed by a nonlinear activation function. The generated feature maps are input into the attention mechanism to enhance important features and suppress noise interference, and the output R h and R w They are expanded and used as attention weights respectively, and the feature map processed by the attention mechanism is fused with the original feature map and the feature map processed by the EFAM module is output, specifically: feature map And the feature map generated along the vertical direction f=δ(F1([V h , v w ])), in: and They are the feature maps generated along the horizontal and vertical directions respectively. represents the average pooling of the hth row of the c-th channel. Max(x c (h)) represents max-pooling of the h-th row of the c-th channel. is the average pooling of the wth column of the cth channel. Max(x c(w)) represents the Max-pooling of the c-th channel and the w-th column; [·, ·] indicates that the connection operation along the spatial dimension is a nonlinear activation function, and F1([·]) is a 1×1 convolution function. is the intermediate feature map. and Split f into two independent tensors along the horizontal and vertical directions, and each output element is specifically R h =σ(F h (f h )) and R w =σ(F w (f w )), where: F h (·), F w (·) is a 1×1 convolution function; y c (i, j) is the attention weighted feature value of the cth channel at position (i, j); F' RT is the final feature map after fusion; || represents the channel-level connection operation, Represents element-wise sum.

[0038] like Figure 5 As shown, the feature enhancement attention module includes: an RGB feature inputter, an IR feature inputter, an RGB-IR fuser, and a channel attention calculator, wherein the RGB feature inputter processes the input RGB information to obtain a preliminary RGB feature map; the IR feature inputter processes the input infrared (IR) information to obtain a preliminary IR feature map; the RGB-IR fuser performs fusion processing based on the RGB and IR feature maps to obtain an enhanced multi-spectral feature map; the channel attention calculator calculates the channel attention weight based on the fused feature information, and then adjusts the weight of the feature map to obtain the final output feature map.

[0039] After specific practical experiments, under the specific environment setting of Python3.8, using NVIDIA 2080Ti graphics card for experiments, this embodiment includes obtaining pedestrian images from the multispectral pedestrian dataset KAIST, and adjusting these images to a uniform size (640×512) in the preprocessing stage, and dividing them into three categories according to the light intensity: strong light, weak light, and no light. The classification of light intensity is based on the illuminance unit lux in the International System of Units, where 1000 lux or more is strong light, 100-500 lux is weak light, and 0-100 lux is no light. In the model building stage, a multispectral pedestrian detection network is constructed. The model integrates SE, CBS, ELAN, MP-1, MP-2, SPPCSPC, Concat, Upsample, RepConv, Detect, CMFM and EFAM modules. Among them, the SE module acts as a channel control switch to control the data flow of RGB and infrared (IR) images; the CBS module performs feature extraction; the ELAN module aggregates and enhances features; the MP-1 and MP-2 modules downsample through pooling operations to retain the main significant features; the SPPCSPC module performs multi-scale feature fusion; the Concat module splices the upsampled features with the feature maps of the early fusion points; the Upsample module improves the resolution of the feature map; the RepConv module optimizes the features through repeated convolution; and the Detect module generates the final pedestrian detection results, including the location and category of the target. In the model training stage, images under three lighting conditions are input into the multispectral pedestrian detection network for training in turn to obtain an adaptive multispectral pedestrian detection network model. After the training is completed, the trained multispectral pedestrian detection network is used in the test stage to detect pedestrians in multispectral images under different lighting and environmental conditions to generate accurate detection results.

[0040] This experiment uses accuracy, recall and mean average precision (mAP) as evaluation indicators. After 200 rounds of training, the model of the present invention reaches the optimal standard, with an accuracy of 89.5%, a recall of 88.3% and a mAP of 90.2%.

[0041] The present invention classifies images into three categories according to light intensity, namely strong light, weak light and no light, in a preprocessing stage, so that a multispectral pedestrian detection network can adaptively adjust feature extraction and fusion strategies according to different lighting conditions, thereby significantly improving detection accuracy in a complex lighting environment; a plurality of modules are integrated through the multispectral pedestrian detection network, including SE, CBS, ELAN, MP-1, MP-2, SPPCSPC, Concat, Upsample, RepConv, Detect, CMFM and EFAM modules, wherein the SE module dynamically adjusts the feature extraction strategies of RGB and IR images through a channel attention mechanism, while the CMFM and EFAM modules further improve the detection capability of the model in multispectral images through a multi-scale feature fusion and enhancement method; the adopted CMFM and EFAM modules can effectively fuse features from RGB and infrared (IR) images, and the detection performance of the model under different spectra is enhanced through difference feature weighting and direction-aware feature map generation technology. This significantly improves the pedestrian detection accuracy of the multispectral pedestrian detection network in low-light and no-light environments; through the feature aggregation and enhancement of the ELAN module and the repeated convolution optimization of the RepConv module, the multispectral pedestrian detection network maximizes the retention of key information in the process of feature extraction and processing, while reducing redundancy and improving the overall detection speed and accuracy of the model. These improvements enable this method to achieve higher pedestrian detection accuracy and robustness than traditional methods under complex and changeable lighting conditions and multispectral data. And the adaptive multispectral pedestrian detection method of the present invention has a wide range of application potentials in multiple fields, including security monitoring, autonomous driving, smart cities, military fields and disaster relief. In security monitoring systems, it can significantly improve the ability to detect pedestrians at night and in low-light environments; in the field of autonomous driving, it provides more reliable pedestrian detection under complex lighting conditions and improves vehicle safety; in smart city construction, it supports intelligent traffic management and crowd monitoring, and provides accurate pedestrian data; in military monitoring and battlefield environments, it enhances target detection in low-visibility environments; in disaster relief operations, it improves the efficiency of personnel search and rescue at night or in low-light environments.

[0042] In summary, compared with the prior art, the present invention has significant improvements in adaptive detection of light intensity, multi-module fusion enhancement, multi-spectral feature fusion, and efficient feature extraction and optimization.

[0043] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. An adaptive multi-spectral pedestrian detection method, characterized in that: In the image preprocessing stage, the KAIST dataset is first obtained, and then the images to be trained are adjusted to a uniform size and classified according to light intensity; in the model building and training stage, the classified images are input into the constructed multispectral pedestrian detection network for training; In the model testing phase, the trained multispectral pedestrian detection network is used to generate pedestrian detection results in real time based on the input images; The classification by light intensity refers to: based on the unit of illumination in the International System of Units (SI): lux (lumens / square meter), 1000 lux or more is strong light, 100-500 lux is weak light, and 0-100 lux is no light, which are divided into three categories: strong light, weak light, and no light; The multispectral pedestrian detection network includes: a channel control switch module (SE), a category balanced sampling module (CBS), an efficient long-distance network (ELAN), a first downsampling module (MP-1), a second downsampling module (MP-2), a spatial pyramid pooling and channel pyramid pooling module (SPPCSPC), a connection layer (Concat), an upsampling layer (Upsample), a reconvolution module (RepConv), a detection layer (Detect), a cross-modal fusion module (CMFM) and a feature enhanced attention module (EFAM).

2. The adaptive multi-spectral pedestrian detection method according to claim 1, characterized in that: The image preprocessing stage refers to: obtaining the public multispectral pedestrian detection dataset KAIST and dividing it into three categories of datasets: strong light, weak light, and no light according to the light intensity, and at the same time dividing the training set, verification set, and test set of the three categories of datasets according to 8:2:1; building a multispectral pedestrian detection network.

3. The adaptive multi-spectral pedestrian detection method according to claim 1, characterized in that: The training refers to: inputting the divided training set into the multispectral pedestrian detection network, training to obtain an adaptive multispectral pedestrian detection network model, and obtaining a trained multispectral pedestrian detection model based on adaptive fusion, specifically including: Step 2.1: To ensure that the model is fully trained, a total of 200 iterations are set; each iteration model will perform a complete weight update through the training data; Step 2.

2. In each iteration, the model optimizes its prediction accuracy by minimizing the cross entropy loss function. For multi-classification tasks, the cross entropy loss function is: Where: N is the number of samples; C is the number of categories; y i,c is the true label (one-hot encoding) of the cth category corresponding to the ith sample; is the model's predicted probability that the i-th sample belongs to the c-th category; Step 3: Input the test set into the trained multispectral pedestrian detection network to generate pedestrian detection results based on the input image to test the model effect.

4. The adaptive multi-spectral pedestrian detection method according to claim 3, characterized in that: The step 2.2 specifically includes: Step 2.2.1, input the RGB+IR images of three light intensities into the multispectral pedestrian detection network, first enter the first SE(a) module to determine whether the light intensity is strong light, when the strong light condition is reached, the left path is suppressed, and the RGB+IR directly enters the right path, and is extracted through four CBSs, and input into the second SE(b) module. According to the weight bias data of the first SE(a), the right path is maintained downward, and three groups of ELAN modules and MP-1 modules are passed through. The first ELAN module further aggregates and enhances the features and outputs them to the MP-1 module. The MP-1 module downsamples the feature map through a pooling operation, retains the main significant features, and outputs them to the fourth ELAN module, and continues to proceed downward. The second ELAN and the third ELAN pass the aggregated and enhanced features to the four CBSs in parallel to extract and enhance the features, and finally output a richer feature map; if the strong light condition is not met, RGB+IR enters two paths respectively, and is extracted through four CBSs, and is input into the second SE(b) module to determine whether the lighting condition is a no-light condition. When the no-light condition is met, RGB and IR pass through three groups of ELAN modules and MP-1 modules respectively, where: the first ELAN module performs further aggregation and enhancement of the features, and outputs them to the MP-1 module. The MP-1 module downsamples the feature map through a pooling operation, retains the main significant features, and outputs them to the CMFM module to splice and fuse the multimodal features, and outputs a more concentrated The generated feature map is provided for processing by the next layer of ELAN modules. The EFAM module processes the features output from multiple ELAN modules and CMFM modules, and further focuses on and enhances the key features to ensure that important information is not diluted in the subsequent processing process. The second and third ELANs pass the aggregated and enhanced features to the four CBSs in parallel to extract and enhance the features, and finally output a richer feature map. The aggregated and enhanced features of the two fourth ELAN modules are spliced ​​and passed to the SPPCSPC modules; if the light-free condition is not met, RGB+IR enters the two channels respectively. After feature extraction by the four CBSs, RGB and IR pass through three groups of ELAN modules and the MP-1 module: the ELAN module performs feature extraction on the features. One-step aggregation and enhancement, and output it to the MP-1 module. The MP-1 module downsamples the feature map through pooling operation, retains the main significant features, and outputs it to the fourth ELAN module, and continues to go down. At the same time, the second ELAN and the third ELAN of the two channels respectively pass the aggregated and enhanced features into the two channels in parallel for two CBS modules to extract and enhance the features, and pass them into the CMFM module for RGB and IR feature information fusion to enhance the feature expression ability, output the fused feature map to the EAEM module to enhance the key feature information, suppress irrelevant information and output the enhanced feature map to the last layer of CBS module for convolution operation and concatenate the two features into the Concat module to prepare for subsequent fusion; Step 2.2.2, the SPPCSPC module receives the features from the fourth ELAN module, performs multi-scale feature fusion, and outputs the fused feature map, which is then passed to the next CBS module, where a convolution operation is performed to further extract key features. The extracted feature map then enters the upsampling module, increases the resolution of the feature map through spatial upsampling, and then is output to the Concat module. In the Concat module, the upsampled feature map is spliced ​​with the feature map of the early fusion point to form a richer and more complete feature map. Next, the ELAN module performs deep feature extraction on the feature map spliced ​​by the Concat module, and passes the processed feature map to the RepConv module. The RepConv module further optimizes these features through repeated convolutions, and finally outputs the detected high-quality feature map. At the same time, the feature map output from the ELAN module is also downsampled through the MP-2 module. This process retains key features. Although the resolution is reduced, it prepares for the subsequent Concat module to ensure effective fusion and processing of features. Step 2.2.3, the Detect module finally processes the feature maps from multiple RepConv modules to generate the final target detection results, including the location and category of the target.

5. The adaptive multi-spectral pedestrian detection method according to claim 1, characterized in that: The multispectral pedestrian detection network includes: a channel control switch module (SE), a category balanced sampling module (CBS), an efficient long distance network (ELAN), a first downsampling module (MP-1), a second downsampling module (MP-2), a spatial pyramid pooling and channel pyramid pooling module (SPPCSPC), a connection layer (Concat), an upsampling layer (Upsample), a reconvolution module (RepConv), a detection layer (Detect), a cross-modal fusion module (CMFM) and a feature enhanced attention module (EFAM), wherein: Two SEs and two groups of CBS constitute the channel segmentation and feature extraction part. The first SE acts as a channel control switch to control the input RGB and IR image data to flow to the two groups of CBS branches; the first and second CBS branches extract features from the image respectively; the second SE performs inter-channel feature weight adjustment operation; Four ELANs, six MP-1s, two CMFMs, and one EFAM constitute the multi-scale feature fusion part. The first to third ELANs aggregate and enhance the extracted features, and downsample the feature maps through the pooling operation of MP-1. CMFM performs cross-modal feature fusion operations; the fourth ELAN performs the final feature integration operation; Four groups of CBS, four CMFMs and two EFAMs constitute two identical multi-scale feature fusion parts. Each part processes feature maps of different scales and performs cross-modal feature fusion to further enhance the saliency of features. Two Concat modules, one Upsample module, two RepConv modules and one Detect module constitute the final detection and output part. The Concat module concatenates feature maps of different scales, and the Upsample module improves the resolution by spatial upsampling. RepConv further optimizes the features, and finally the Detect module generates the final pedestrian detection results; The first MP-2 module performs downsampling to retain key features and reduce resolution in preparation for the subsequent first Concat module; the SPPCSPC module processes features from the fourth ELAN module, outputs feature maps after multi-scale feature fusion, and enters the second CBS module for feature map convolution to further extract features; the first Concat module splices the upsampled features with the feature maps of the early fusion points to form a richer feature map; the first Upsample module increases the resolution of the feature map through spatial upsampling; the first RepConv module further optimizes features through repeated convolution and outputs the final detection feature map; The second MP-2 module also performs downsampling to retain key features and prepares for the subsequent second Concat module; the second Concat module combines previous feature information for splicing to form a richer feature map; the second Upsample module improves the resolution of the feature map by upsampling; the second RepConv module further optimizes these features; the Detect module processes the detection feature maps from multiple RepConv modules to generate the final target detection results.

6. The adaptive multi-spectral pedestrian detection method according to claim 5, characterized in that: The channel control switch module includes: a channel attention unit, a convolution unit, an activation unit and a pooling unit, wherein: the channel attention unit receives multispectral image information, including data of RGB and IR channels, and extracts light intensity and scene information based on the input image information, and performs preliminary analysis to determine the applicable fusion strategy; the convolution unit receives preliminary suggestions from the channel attention unit, further refines the fusion strategy in combination with the ambient lighting conditions and target detection requirements, generates the final fusion strategy and passes it to the activation unit; the activation unit executes the fusion strategy generated by the convolution unit, and outputs the fused image features by controlling the fusion method of the RGB and IR image data streams; finally, the pooling unit performs deep processing on the fusion result of the activation unit, enhances important features and suppresses noise, and outputs the optimized image feature map as the final multispectral image processing result, the pedestrian detection task.

7. The adaptive multi-spectral pedestrian detection method according to claim 5, characterized in that: The multi-spectral feature fusion module includes: a convolution unit, an activation unit, a feature selection unit, and a feature fusion unit, wherein: the RGB and IR images are respectively extracted with preliminary features through their respective convolution units, and nonlinear mapping is performed through the activation unit (Sigmoid). After the feature selection unit, the RGB and IR features are respectively selected for important features through dot multiplication operations, and at the same time, the RGB features are cross-dot multiplied with the IR features to strengthen the cross-spectral feature fusion. Finally, the fused features are further extracted with advanced features through the convolution unit, and element-by-element addition operations are performed with the original RGB and IR features at the output stage to generate the final multi-spectral fusion feature map for the subsequent pedestrian detection task.

8. The adaptive multi-spectral pedestrian detection method according to claim 5, characterized in that: The feature enhancement attention module includes: an RGB feature inputter, an IR feature inputter, an RGB-IR fuser, and a channel attention calculator, wherein the RGB feature inputter processes the input RGB information to obtain a preliminary RGB feature map; the IR feature inputter processes the input infrared (IR) information to obtain a preliminary IR feature map; the RGB-IR fuser performs fusion processing based on the RGB and IR feature maps to obtain an enhanced multi-spectral feature map; the channel attention calculator calculates the channel attention weight based on the fused feature information, and then adjusts the weight of the feature map to obtain the final output feature map.

Citation Information

Cited By

  • Marine ship monitoring method and system based on image recognition

    CN120411897A