Pedestrian detection method based on multi-modal detector

Abstract

The invention discloses a pedestrian detection method based on a multi-modal detector, and relates to the field of pedestrian detection, and the method comprises the steps: obtaining a multispectral pedestrian detection image data set comprising an RGB image and an infrared image; establishing a multi-modal detector based on a state space model, and training the multi-modal detector by using the multispectral image data set; and acquiring the RGB image and the infrared image of the detection area at the same moment, aligning and inputting the RGB image and the infrared image into the trained multi-mode detector to obtain a pedestrian detection result. The method provided by the invention not only has excellent detection precision, but also has very high reasoning speed, and can effectively improve the pedestrian detection accuracy under the dark light or night light condition.

Description

Technical Field

[0001] This invention relates to the field of pedestrian detection technology, and more specifically to a pedestrian detection method based on a multimodal detector. Background Technology

[0002] Traditional pedestrian detection mainly focuses on single-modality (pedestrian detection based on RGB images). In dark environments, it is prone to false positives and false negatives due to cluttered backgrounds and poor visibility at night.

[0003] However, current representative solutions for low-light detection take the form of "intensifier + detector," introducing an intensifier as a preprocessing step for conventional detectors to improve detection quality through image enhancement. However, introducing an intensifier can introduce black edges and preserve dark, noisy areas, interfering with subsequent target localization and recognition processes and causing misjudgments of small targets at edges. Furthermore, while enhancing contrast improves overall visual quality, these enhancements can negatively impact target detection performance, reducing detection accuracy. Simultaneously, the additional preprocessing steps increase system complexity and runtime. For applications with high real-time requirements, the increased computational and time costs are a significant burden on resource-constrained platforms. Moreover, research indicates that low-light enhancement methods often lack sufficient flexibility to adapt to diverse lighting conditions; enhancement strategies that perform well under specific conditions may be ineffective or even counterproductive under other circumstances.

[0004] Therefore, how to improve the accuracy of pedestrian detection under low light or nighttime lighting conditions is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a pedestrian detection method based on a multimodal detector, which solves the problem of pedestrian detection under low light or nighttime lighting conditions by introducing multimodal data such as RGB, infrared and thermal images and performing multimodal data fusion.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention discloses a pedestrian detection method based on a multimodal detector, the specific steps of which are as follows:

[0008] Obtain a multispectral pedestrian detection image dataset including RGB and infrared images;

[0009] A multimodal detector based on a state-space model is established, and the multimodal detector is trained using the multispectral image dataset;

[0010] The RGB and infrared images of the detection area are acquired at the same time, aligned, and then input into the trained multimodal detector to obtain the pedestrian detection results.

[0011] Furthermore, it also includes preprocessing the input feature map of the overall architecture of the state space model, flattening the input feature map and adjusting the matrix order, as shown in the following formula:

[0012]

[0013] in, The input feature map is for the RGB modality. The input feature map is the infrared mode; Reshape represents the operation of changing the shape while keeping the elements unchanged, converting the three-dimensional tensor into a two-dimensional matrix; W and H are the height and width of the feature map, respectively, and C is the number of channels; and The two-dimensional matrices representing the RGB and infrared modes are reconciled. After adjusting to two-dimensional matrices, the features of the two modes are concatenated and a learnable positional encoding of dimension C×2HW is added.

[0014] Furthermore, the multimodal detector includes: a backbone network, a neck network, and a detection head;

[0015] The backbone network includes an RGB branch and an infrared branch built based on a fast feature extraction network, with a multispectral feature fusion module embedded between the RGB branch and the infrared branch; both the RGB branch and the infrared branch include a stem module and four fast network stages connected in sequence.

[0016] The fast network stage consists of an embedding layer and a fast network module connected in sequence. The embedding layer maps the input feature data to the embedding space and inputs it into the fast network module. The fast network module extracts features by performing local convolution and pointwise convolution on the input feature data. The second, third, and fourth fast network stages of the RGB branch are embedded with the fast network stages corresponding to the infrared branch. Each of the multispectral feature fusion modules performs cross-modal feature fusion on the RGB modal features extracted by the RGB branch and the infrared modal features extracted by the infrared branch.

[0017] The neck network is composed of a feature pyramid network; through the spatial pyramid pooling feature module, the RGB modal features, the infrared modal features, and the fused features extracted in the second, third, and fourth fast network stages are adaptively output to the neck network, and the neck network performs multi-feature fusion.

[0018] The detection head generates the final detection result based on the output of the neck network.

[0019] Furthermore, the fast network module includes: a local convolution module, a pointwise convolution module, a batch normalization layer, and a pointwise convolution module connected in sequence; after the input feature data undergoes local convolution, pointwise convolution, batch normalization, and pointwise convolution, element-wise addition is performed with the input feature data to obtain the output of the fast network module.

[0020] Furthermore, the multispectral feature fusion module consists of eight repeating state-space models.

[0021] Furthermore, the state space model includes a two-dimensional convolutional layer, a batch normalization layer, an activation function layer, a local space extraction module, a normalization layer, a two-dimensional parallel scanning module, a normalization layer, and a gating module connected in sequence.

[0022] Input feature data Z l-3 The feature data Z is obtained after two-dimensional convolution, batch normalization, and activation function. l-2 Then, using the local spatial extraction module, the feature data Z is processed. l-2 Local spatial features are extracted and normalized. Then, parallel correlation scanning is performed using the two-dimensional parallel scanning module, and finally compared with the feature data Z. l-2 Perform element-wise addition to obtain the feature data Z. l-1 ; for the feature data Z l-1 After normalization, global feature extraction is performed using the gating module, and then compared with the feature data Z. l-1 By performing element-wise addition, the final output Z of the state-space model is obtained. l .

[0023] Furthermore, the local space extraction module includes: a depthwise separable convolutional module, a batch normalization layer, a two-dimensional convolutional layer, an activation function layer, and another two-dimensional convolutional layer connected in sequence;

[0024] The input feature data is first processed by depthwise separable convolution, then batch normalized to obtain an intermediate state, then mixed by 1×1 convolution and the distribution of information is preserved by activation function, and finally convolved with the input feature data element-wise to obtain the output of the local space extraction module.

[0025] Furthermore, the two-dimensional parallel scanning module includes: a linear layer, a depthwise separable convolution module, an activation function layer, a scanning module, a normalization layer, and a linear layer connected in sequence;

[0026] The input feature data is linearly mapped and then subjected to depthwise separable convolution. It is then nonlinearly transformed by an activation function, followed by parallel correlation scanning by a scanning module. Finally, it is normalized and linearly mapped before being output.

[0027] Furthermore, the gating module includes two branches and a two-dimensional convolutional layer;

[0028] The first branch consists of a two-dimensional convolutional layer, with the input feature data F. l-2 Feature data is obtained through 1×1 convolution.

[0029] The second branch consists of sequentially connected 2D convolutional layers, depthwise separable convolutional modules, and activation function layers, with input feature data F. l-2 Feature data is obtained after 1×1 convolution. After further depthwise separable convolution processing, it is combined with the feature data. Perform element-wise addition, then apply a non-linear transformation using an activation function, and finally combine it with the feature data. Element-wise multiplication is used to obtain feature data.

[0030] Final feature data After convolution processing by a two-dimensional convolutional layer, the input feature data F is combined with... l-2 Perform element-wise addition to obtain the final output F of the gating module. l .

[0031] Furthermore, the depthwise separable convolution module includes a depthwise convolution module and a pointwise convolution module; the depthwise convolution module applies an independent convolution kernel to each channel of the input feature data to perform convolution operations, thereby obtaining the feature map of each channel; the pointwise convolution module performs a 1x1 convolution operation on the feature map of each channel, integrates the information of each channel, and generates the final output feature map.

[0032] As can be seen from the above technical solution, compared with the prior art, this invention discloses a pedestrian detection method based on a multimodal detector. Based on the complementary characteristics between RGB and infrared images, a novel fusion strategy is designed to promote effective interaction of information between different modalities. A lightweight architecture is designed using fast network design principles, and techniques such as depthwise separable convolution are combined to reduce the number of parameters and computational complexity, thereby achieving high accuracy and a lightweight model. This invention not only has excellent detection accuracy but also very fast inference speed, effectively improving pedestrian detection accuracy under low light or nighttime lighting conditions. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the overall structure of the multimodal detector according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of the fast network module structure according to an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram of the RMF module structure according to an embodiment of the present invention.

[0038] Figure 5 This is a schematic diagram of the parallel correlation scanning operation according to an embodiment of the present invention.

[0039] Figure 6 This is a schematic diagram of the depth-separable convolution module structure according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] This invention discloses a pedestrian detection method based on a multimodal detector, such as... Figure 1 As shown, the specific steps are as follows:

[0042] Obtain a multispectral pedestrian detection image dataset including RGB and infrared images;

[0043] A multimodal detector based on a state-space model was established, and the multimodal detector was trained using a multispectral image dataset.

[0044] The RGB and infrared images of the detection area are acquired at the same time, aligned, and then input into the trained multimodal detector to obtain the pedestrian detection results.

[0045] In a specific embodiment, the method further includes preprocessing the input feature map of the overall architecture of the state space model, flattening the input feature map and adjusting the matrix order, as follows:

[0046]

[0047] in, The input feature map is for the RGB modality. The input feature map is the infrared mode; Reshape represents the operation of changing the shape while keeping the elements unchanged, converting the three-dimensional tensor into a two-dimensional matrix; W and H are the height and width of the feature map, respectively, and C is the number of channels; and The two-dimensional matrices representing the RGB and infrared modes are reconciled. After adjusting to two-dimensional matrices, the features of the two modes are concatenated and a learnable positional encoding of dimension C×2HW is added.

[0048] The system (a linear time-invariant (LTI) continuous-time dynamic system, emphasizing that the model evolves dynamically, gradually transforming the input sequence into the output sequence) passes through a hidden intermediate state. One-dimensional sequence The mapping is represented by an output sequence y(t), as shown in the formula:

[0049] h'(t) = Ah(t) + Bx(t);

[0050] y(t) = Ch(t);

[0051] in, Represents the state transition matrix. The output represents the weight matrix of the hidden state space. Let h(t) represent the observation matrix that maps the hidden intermediate states to the output, h'(t) represent the hidden state in continuous time, h'(t) represent the reciprocal of h(t), x(t) represent the input signal in continuous time, and y(t) represent the output signal in continuous time.

[0052] The state-space model applies the system to discrete-time series data, using a fixed discretization rule f. A and f B Convert parameters A and B to and The formula is:

[0053]

[0054] Here, Δ represents a time scale parameter used to adjust the time resolution of the model, and I is the identity matrix. The discretized state transition matrix is ​​derived from A using rules such as zero-order preservation. Let ΔA represent the discretized input matrix, where ΔA represents the discretization step size of A, i.e., the time interval during discretization of the continuous-time system, which determines the frequency of state updates; and ΔB represents the discretization step size of B.

[0055] After the state-space model transformation, the system performs efficient autoregressive inference through linear recursion, and the calculation formula is as follows:

[0056]

[0057] y t =Ch t ;

[0058] The formula for calculating the transformation of the entire sequence using convolution is as follows:

[0059]

[0060] in, This represents the structured convolution kernel, where L represents the length of the input sequence, and h represents the length of the input sequence. t x represents the hidden state at discrete time step t. t y represents the input at discrete time step t. t This represents the input for the discrete time step y.

[0061] In one specific embodiment, the multimodal detector (ESM) employs a two-stream backbone network based on a fast feature extraction network and embeds a multispectral feature fusion module to promote modality fusion and interaction. A lightweight neck network is constructed using depthwise separable convolutions and PAFPAN. Finally, in the detection head section, this paper designs a Focal-EIoU loss function that integrates cross-entropy squared error, such as... Figure 2 As shown in the diagram. Here, PAFAPN Neck represents the neck network, DecoupledHead represents the detection head; RFM represents the multispectral feature fusion module, Fasternet represents the fast network stage, DW-conv and DSCconv represent depthwise separable convolution, and conv represents convolution; U represents upsampling, C represents data fusion; Prediction represents the predicted bounding box, and Small, Medium, and Large are three different sizes of predicted bounding boxes.

[0062] The multimodal detector includes: a backbone network, a neck network, and a detection head;

[0063] The backbone network includes an RGB branch and an infrared branch built on a fast feature extraction network, with a multispectral feature fusion module embedded between the RGB and infrared branches; both the RGB and infrared branches include a stem module and four fast network stages connected in sequence.

[0064] The fast network stage consists of an embedding layer and fast network modules connected sequentially. The embedding layer maps the input feature data to the embedding space and inputs it into the fast network module. The fast network module extracts features by performing local convolution and pointwise convolution on the input feature data. Multispectral feature fusion modules are embedded between the 2nd, 3rd, and 4th fast network stages of the RGB branch and the fast network stages corresponding to the infrared branch. Each multispectral feature fusion module performs cross-modal feature fusion on the RGB modal features extracted by the RGB branch and the infrared modal features extracted by the infrared branch.

[0065] The neck network is composed of a feature pyramid network. Through the spatial pyramid pooling feature module, the RGB modal features, infrared modal features, and fused features extracted in the second, third, and fourth fast network stages are adaptively output to the neck network, and the neck network performs multi-feature fusion.

[0066] The detection head generates the final detection result based on the output of the neck network.

[0067] In a specific embodiment, such as Figure 3 As shown, the fast network module includes: a local convolution module, a pointwise convolution module, a batch normalization layer, and a pointwise convolution module connected in sequence; after the input feature data undergoes local convolution, pointwise convolution, batch normalization, and pointwise convolution, it is then element-wise added to the input feature data to obtain the output of the fast network module.

[0068] In a specific embodiment, such as Figure 4 As shown in section (e), the multispectral feature fusion module (RMF) consists of eight repeating state-space models (SSM Blocks) stacked sequentially.

[0069] Specifically, the core of the multispectral feature fusion module is based on eight repeating SSM blocks, learning long-range dependencies and integrating global contextual information during the feature extraction stage. This module is embedded in the feature extraction backbone network to facilitate feature fusion and interaction between different modalities (e.g., RGB and thermal imaging), allowing for simultaneous intra-modal and cross-modal feature fusion. In this way, the RMF module can effectively capture the potential interactions between the RGB and thermal domains, thereby enhancing the expressiveness of the RGB and infrared branch convolutions. The RMF module can perform global context integration, unlike convolutional operations with local receptive fields; therefore, its receptive field is global, allowing it to better integrate global information from different modalities. The RMF module can simultaneously perform intra-modal and cross-modal feature fusion, strengthening features within a single RGB modality while effectively combining information from the infrared modality.

[0070] In a specific embodiment, such as Figure 4 As shown in part (b), the state-space model comprises sequentially connected two-dimensional convolutional layers ( Figure 4 Conv2d), batch normalization layer ( Figure 4 BatchNorm and activation function layers Figure 4 Activation), Local Space Extraction Module Figure 4 LS Block), normalization layer ( Figure 4 LayerNorm and 2D parallel scanning module ( Figure 4 SS2D), normalization layer, gating module ( Figure 4 (middle RG Block);

[0071] Input feature data Z l-3 The feature data Z is obtained after two-dimensional convolution, batch normalization, and activation function. l-2 Then, using the local spatial extraction module, the feature data Z is processed. l-2 Local spatial features are extracted and normalized. Then, a two-dimensional parallel scanning module is used for parallel correlation scanning, followed by comparison with feature data Z. l-2 Perform element-wise addition to obtain the feature data Z. l-1 For feature data Z l-1 After normalization, global feature extraction is performed using a gating module, and then compared with the feature data Z. l-1 By performing element-wise addition, we obtain the final output Z of the state-space model. l .

[0072] Specifically, the SSM Block is the core module of the RMF module. During the input phase, it undergoes a series of processing steps, enabling the network to learn deeper and richer feature representations. Simultaneously, batch normalization maintains the efficiency and stability of the training and inference process. This can be represented as:

[0073] The SSM Block's normalization and residual linking design borrows from the visual transformer-style architecture, enabling efficient information flow and training even in deeply stacked models. The calculation formula is:

[0074] Z l-1 =SS2D(LN(LS(Z) l-2 )))+Z l-2 ;

[0075] Z l =RG(LN(Z) l - 1 ))+Z l - 1 ;

[0076] Wherein, SS2D represents parallel correlation scanning, LN represents normalization operation, LS represents local space extraction module, and RG represents gating module.

[0077] In a specific embodiment, such as Figure 4 As shown in section (c), the Local Spatial Extraction Module (LSBlock) includes: a depthwise separable convolutional module, a batch normalization layer, a two-dimensional convolutional layer, an activation function layer, and another two-dimensional convolutional layer connected in sequence.

[0078] The input feature data is first processed by depthwise separable convolution, then batch normalized to obtain an intermediate state, then mixed by 1×1 convolution and the distribution of information is preserved by activation function, and finally convolved with the input feature data element-wise to obtain the output of the local space extraction module.

[0079] Specifically, for a given input feature (C, H, and W represent the channels, height, and width of the feature map, respectively), X l-1 First, depthwise separable convolution is applied, performing convolution operations on each input channel separately without mixing channel information. This effectively extracts the local spatial information of the input feature map while reducing computational cost and the number of parameters. Then, batch normalization provides a degree of regularization while reducing overfitting, resulting in the intermediate state X. l-1 The formula is: X l-1 =BN(DWConv 3×3 (X l-2 ));

[0080] Where BN represents batch normalization and DWConv represents depthwise separable convolution.

[0081] intermediate state X l-1 By using a 1×1 convolution to mix channel information and a better preservation of information distribution through an activation function, the model can learn more complex feature representations and extract rich multi-scale contextual information from the input feature map. In LSBlock, a non-linear GeLU activation function is used, changing the number of feature channels without altering the spatial dimension, thereby enhancing feature representation capabilities. Finally, residual connections fuse the original input with the processed features, allowing the model to understand and integrate features of different sizes in the image, thus enhancing robustness to scale changes. The formula is:

[0082]

[0083] Among them, X l The output features of the local spatial extraction module are represented by Φ, which denotes the activation function (non-linear GeLU). It represents element-wise addition. LSBlock can effectively capture and represent the local spatial information of the input feature map and fuse it with the original input to enhance the representational power of the features, providing powerful capabilities to handle scale variations and contextual information in visual data.

[0084] In a specific embodiment, such as Figure 4 As shown in part (a), the two-dimensional parallel scanning module (SS2D) includes: a linear layer, a depthwise separable convolutional module, an activation function layer, a scanning module, a normalization layer, and a linear layer connected in sequence;

[0085] The input feature data is linearly mapped and then subjected to depthwise separable convolution. It is then nonlinearly transformed by an activation function, followed by parallel correlation scanning by a scanning module. Finally, it is normalized and linearly mapped before being output.

[0086] Specifically, parallel correlation scanning, such as Figure 5 As shown, scan expansion, S6 blocks, and scan merging are the three main steps of SS2D. The scan expansion operation unfolds the input image into a series of sub-images, each representing a specific direction. From a diagonal perspective, the scan expansion operation proceeds along four symmetrical directions: top-down, bottom-up, left-to-right, and right-to-left. This layout not only comprehensively covers all areas of the input image but also provides a rich, multi-dimensional information foundation for subsequent feature extraction through systematic directional transformations, enhancing the efficiency and comprehensiveness of multi-dimensional image feature capture. Next, feature extraction is performed on these sub-images in the S6 block operation (the S6 module refers to the efficient computation of recursion on the GPU using a parallel scan algorithm, which is part of the state-space model). Finally, the scan merging operation combines these sub-images into an output image of the same size as the input image.

[0087] In a specific embodiment, such as Figure 4 As shown in part (d), the gating module includes two branches and a two-dimensional convolutional layer;

[0088] The first branch consists of a two-dimensional convolutional layer, with the input feature data F. l-2 Feature data is obtained through 1×1 convolution.

[0089] The second branch consists of sequentially connected 2D convolutional layers, depthwise separable convolutional modules, and activation function layers, with input feature data F. l-2 Feature data is obtained after 1×1 convolution. After further depthwise separable convolution processing, it is combined with the feature data. Perform element-wise addition, then apply a non-linear transformation using an activation function, and finally combine it with the feature data. Element-wise multiplication is used to obtain feature data.

[0090] Final feature data After convolution processing by a two-dimensional convolutional layer, the input feature data F is combined with... l-2 Perform element-wise addition to obtain the final output F of the gating module. l .

[0091] Specifically, this invention proposes a concise design for a gating module (RG Block) aimed at improving model performance with lower computational cost. The gating module draws input F... l-2 Create two branches and A fully connected layer is implemented on each branch using a 1×1 convolution, as shown in the formula:

[0092]

[0093] Where Conv 1×1 This represents a 1×1 convolution.

[0094] exist The branch employs depthwise separable convolutions as the location encoding module, using residual connections to more effectively backflow gradients during training. This significantly improves performance while preserving and utilizing the spatial structure information of the image, despite lower computational costs. The RG Block uses a non-linear GeLU as the activation function to control the information flow at each level, and then uses element-wise multiplication and... The branches are merged, then a 1×1 convolution is used to mix channel information to refine the global features, and finally the original input F is processed through residual connections. l-2 The RG Block is added to the features in the hidden layer. It captures more global features while only slightly increasing computational cost, resulting in the output feature F. l The formula is:

[0095]

[0096] Where ⊙ represents element-wise multiplication. In this invention, the gating mechanism in the gating module preserves spatial information while making the model more sensitive to fine-grained features in the image by integrating convolution operations. Compared with traditional MLP, the gating module transfers global dependencies and global features to each pixel to capture the dependencies of neighboring features, thereby enriching the contextual information and further enhancing the model's expressive power.

[0097] In one specific embodiment, the depthwise separable convolution module includes a depthwise convolution module and a pointwise convolution module; the depthwise convolution module applies an independent convolution kernel to each channel of the input feature data to perform convolution operations to obtain the feature map of each channel; the pointwise convolution module performs a 1x1 convolution operation on the feature map of each channel, integrates the information of each channel, and generates the final output feature map.

[0098] Specifically, depthwise separable convolution breaks down the traditional convolution operation into two steps: depthwise convolution and pointwise convolution. In the depthwise convolution stage, an independent convolution kernel is applied to each input channel, meaning each filter processes only a single channel's feature map, resulting in an intermediate feature map with the same number of channels as the original input. The pointwise convolution stage performs 1x1 convolution operations on feature maps with the same number of channels, handling cross-channel information fusion. It applies 1x1 convolutions to integrate information from each channel and generate the final output feature map. In contrast, standard convolution simultaneously filters and combines inputs into a new output set, while depthwise separable convolution decomposes this process into two independent layers: a single channel layer and a single combination layer, such as... Figure 6 As shown, the parallel processing effect brought about by this decomposition significantly reduces the amount of computation and the model size.

[0099] Depthwise separable convolutions effectively split the interaction between the convolutional kernel and different channels into two steps: the first step processes information within a single channel, and the second step fuses information across channels. This significantly reduces model complexity and computational resource consumption, especially when processing high-dimensional feature maps, which is beneficial for model lightweighting and improved computational efficiency. While depthwise separable convolutions cannot extract and express rich feature information as fully as traditional convolutions, and while compressing computation, they reduce the model's ability to learn more complex nonlinear expressions and cause some accuracy loss, in specific pedestrian detection tasks, the impact of feature information compression is reduced due to the simplistic features of pedestrian images and the large sample size of pedestrian datasets. Therefore, depthwise separable convolutions can better balance latency and accuracy, achieving a balance between speed and precision.

[0100] In one specific embodiment, a multispectral pedestrian detection dataset is constructed based on the FLIR dataset, LLVIP dataset, and KAIST dataset to train the multimodal detector.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0102] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pedestrian detection method based on a multimodal detector, characterized in that, The specific steps are as follows: Obtain a multispectral pedestrian detection image dataset including RGB and infrared images; A multimodal detector based on a state-space model is established, and the multimodal detector is trained using the multispectral image dataset; The RGB and infrared images of the detection area are acquired at the same time, aligned, and then input into the trained multimodal detector to obtain the pedestrian detection results.

2. The pedestrian detection method based on a multimodal detector according to claim 1, characterized in that, It also includes preprocessing the input feature map of the overall architecture of the state space model, flattening the input feature map and adjusting the matrix order, as shown in the following formula: in, The input feature map is for the RGB modality. The input feature map is the infrared mode; Reshape represents the operation of changing the shape while keeping the elements unchanged, converting the three-dimensional tensor into a two-dimensional matrix; W and H are the height and width of the feature map, respectively, and C is the number of channels; and The two-dimensional matrices representing the RGB and infrared modes are reconciled. After adjusting to two-dimensional matrices, the features of the two modes are concatenated and a learnable positional encoding of dimension C×2HW is added.

3. The pedestrian detection method based on a multimodal detector according to claim 1, characterized in that, The multimodal detector includes: a backbone network, a neck network, and a detection head; The backbone network includes an RGB branch and an infrared branch built based on a fast feature extraction network, with a multispectral feature fusion module embedded between the RGB branch and the infrared branch; both the RGB branch and the infrared branch include a stem module and four fast network stages connected in sequence. The fast network stage consists of an embedding layer and a fast network module connected in sequence. The embedding layer maps the input feature data to the embedding space and inputs it into the fast network module. The fast network module extracts features by performing local convolution and pointwise convolution on the input feature data. The second, third, and fourth fast network stages of the RGB branch are embedded with the fast network stages corresponding to the infrared branch. Each of the multispectral feature fusion modules performs cross-modal feature fusion on the RGB modal features extracted by the RGB branch and the infrared modal features extracted by the infrared branch. The neck network is composed of a feature pyramid network; through the spatial pyramid pooling feature module, the RGB modal features, the infrared modal features, and the fused features extracted in the second, third, and fourth fast network stages are adaptively output to the neck network, and the neck network performs multi-feature fusion. The detection head generates the final detection result based on the output of the neck network.

4. The pedestrian detection method based on a multimodal detector according to claim 3, characterized in that, The fast network module includes: a local convolution module, a pointwise convolution module, a batch normalization layer, and a pointwise convolution module connected in sequence; after the input feature data undergoes local convolution, pointwise convolution, batch normalization, and pointwise convolution, element-wise addition is performed with the input feature data to obtain the output of the fast network module.

5. The pedestrian detection method based on a multimodal detector according to claim 3, characterized in that, The multispectral feature fusion module consists of eight repeating state-space models.

6. The pedestrian detection method based on a multimodal detector according to claim 5, characterized in that, The state space model includes a two-dimensional convolutional layer, a batch normalization layer, an activation function layer, a local space extraction module, a normalization layer, a two-dimensional parallel scanning module, a normalization layer, and a gating module connected in sequence. Input feature data Z l-3 The feature data Z is obtained after two-dimensional convolution, batch normalization, and activation function. l-2 Then, using the local spatial extraction module, the feature data Z is processed. l-2 Local spatial features are extracted and normalized. Then, parallel correlation scanning is performed using the two-dimensional parallel scanning module, and finally compared with the feature data Z. l-2 Perform element-wise addition to obtain the feature data Z. l-1 ; for the feature data Z l-1 After normalization, global feature extraction is performed using the gating module, and then compared with the feature data Z. l-1 By performing element-wise addition, the final output Z of the state-space model is obtained. l .

7. The pedestrian detection method based on a multimodal detector according to claim 6, characterized in that, The local space extraction module includes: a depthwise separable convolutional module, a batch normalization layer, a two-dimensional convolutional layer, an activation function layer, and another two-dimensional convolutional layer connected in sequence. The input feature data is first processed by depthwise separable convolution, then batch normalized to obtain an intermediate state, then mixed by 1×1 convolution and the distribution of information is preserved by activation function, and finally convolved with the input feature data element-wise to obtain the output of the local space extraction module.

8. The pedestrian detection method based on a multimodal detector according to claim 6, characterized in that, The two-dimensional parallel scanning module includes: a linear layer, a depthwise separable convolution module, an activation function layer, a scanning module, a normalization layer, and a linear layer connected in sequence; The input feature data is linearly mapped and then subjected to depthwise separable convolution. It is then nonlinearly transformed by an activation function, followed by parallel correlation scanning by a scanning module. Finally, it is normalized and linearly mapped before being output.

9. A pedestrian detection method based on a multimodal detector according to claim 6, characterized in that, The gating module includes two branches and a two-dimensional convolutional layer; The first branch consists of a two-dimensional convolutional layer, with the input feature data F. l-2 Feature data is obtained through 1×1 convolution. The second branch consists of sequentially connected 2D convolutional layers, depthwise separable convolutional modules, and activation function layers, with input feature data F. l-2 Feature data is obtained after 1×1 convolution. After further depthwise separable convolution processing, it is combined with the feature data. Perform element-wise addition, then apply a non-linear transformation using an activation function, and finally combine it with the feature data. Element-wise multiplication is used to obtain feature data. Final feature data After convolution processing by a two-dimensional convolutional layer, the input feature data F is combined with... l-2 Perform element-wise addition to obtain the final output F of the gating module. l .

10. A pedestrian detection method based on a multimodal detector according to any one of claims 7, 8, and 9, characterized in that, The depthwise separable convolution module includes a depthwise convolution module and a pointwise convolution module. The depthwise convolution module applies an independent convolution kernel to each channel of the input feature data to perform convolution operations, thereby obtaining the feature map of each channel. The pointwise convolution module performs a 1x1 convolution operation on the feature map of each channel, integrates the information of each channel, and generates the final output feature map.

Images