Infrared single-pixel imaging method and device based on hybrid neural network model

By combining CNN-Mamba hybrid neural network model, combining CNN local feature extraction and Mamba's global receptive field, the problem of low reconstruction quality and efficiency in single-pixel imaging is solved, and efficient and high-quality infrared single-pixel imaging is achieved. .

CN118900366APending Publication Date: 2024-11-05LINYI UNIVERSITY

Patent Information

Application Number
CN202411104778.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The existing single-pixel imaging reconstruction method based on deep learning has insufficient reconstruction quality and efficiency. Especially when processing long-sequence signals, CNN lacks global perception capabilities, while the Transformer network has high computational complexity, resulting in imaging time resolution. Low rate.

Method used

Using the CNN-Mamba hybrid neural network model, combined with the convolutional neural network (CNN) as an encoder and Mamba-based decoder, the multi-scale attention fusion module uses the local feature extraction of CNN and the global receptive field of Mamba to realize Efficient signal reconstruction.

Benefits of technology

It improves the resolution and detail quality of the reconstructed images, enhances the model's adaptability to complex scenes, reduces computing costs, and is suitable for real-time imaging applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118900366A_ABST
    Figure CN118900366A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of optical imaging and deep learning, and provides an infrared single-pixel imaging method and device based on a hybrid neural network model, and the method comprises the following steps: obtaining a one-dimensional undersampling infrared single-pixel detection signal corresponding to a to-be-imaged target according to a binary coding pattern obtained through training; inputting the obtained infrared single-pixel detection signal into the trained CNN-Mama hybrid neural network model to obtain a reconstructed image of the imaging target; the model based on the CNN-Mama hybrid neural network comprises a CNN-based encoder, a Mama-based decoder and a multi-scale attention fusion module. According to the model, high-quality, high-efficiency and high-robustness infrared single-pixel imaging can be realized under the condition of undersampling, and the problems of low reconstruction quality and low reconstruction efficiency of the existing reconstruction method are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of optical imaging and deep learning technology, and in particular to an infrared single-pixel imaging method and device based on a hybrid neural network model. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Single-pixel imaging, also known as correlation imaging, is a new computational optical imaging technology inspired by quantum optics in the field of physics and compressed sensing theory in the field of signal processing. Single-pixel imaging first compresses and encodes a two-dimensional or three-dimensional imaging target into a one-dimensional single-pixel detection signal by using a series of modulated light fields with different spatial distributions, and then reconstructs the high-dimensional spatial information of the imaging target from the one-dimensional detection signal through a specific algorithm. Compared with the imaging method of directly collecting the target light field intensity using traditional focal plane array detectors (such as CCD, CMOS), single-pixel imaging has the following advantages: First, the photosensitive surface area of ​​a single-pixel detector is much larger than the pixel area of ​​a two-dimensional array detector, and it can receive more photons, which is conducive to weak light or long-distance imaging; second, a single-pixel detector has a wide spectral response range and can achieve imaging in the non-visible light band; third, the manufacturing process of a single-pixel detector is mature, stable and low-cost, which can reduce the cost of the imaging system. Based on the above hardware advantages and the flexibility of the single-pixel imaging mechanism, single-pixel imaging has been verified to have unique advantages and significant performance in non-visible light bands such as X-rays, ultraviolet, infrared, and terahertz, as well as non-optical bands such as atoms, electrons, and neutrons. It also has great application value and potential in many fields such as three-dimensional imaging, underwater imaging, phase imaging, non-field of view imaging, fluorescence lifetime imaging, microscopic imaging, and information encryption. However, due to the need for multiple encoding and detection in the single-pixel imaging process, its imaging time resolution is low and it is not suitable for dynamic target imaging, which limits its application scope to a certain extent.

[0004] In recent years, with the widespread popularization of artificial intelligence machine learning and the super advantages of deep learning in image processing and signal processing, the use of deep neural networks to complete information extraction and image reconstruction under undersampling has become a mainstream choice. Similarly, in the field of single-pixel imaging, some works have been proposed to use deep learning networks to reduce the number of sampling times in the imaging process, so as to shorten the imaging time and improve the quality of reconstructed images. However, the network models used in these works are all based on CNN (Convolutional Neural Network) or Transformer networks. CNN has extremely strong local perception ability, but lacks the perception of global information, and is generally effective when processing long sequences of single-pixel detection signals. The Transformer network can better capture the long-distance dependency of single-pixel detection signals through the self-attention mechanism, but it requires the computational complexity of the square of the signal length, so it has high requirements for hardware equipment and reconstruction time. In summary, the existing single-pixel imaging reconstruction methods based on deep learning have problems with low reconstruction quality and efficiency. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes an infrared single-pixel imaging method and device based on a hybrid neural network model. Based on the CNN-Mamba hybrid neural network model, high-quality, high-efficiency and high-robustness infrared single-pixel imaging can be achieved under undersampling conditions.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: The first aspect of the present invention provides an infrared single-pixel imaging method based on a hybrid neural network model, comprising the following steps: According to the binary coding pattern obtained through training, a one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged is obtained; The acquired infrared single-pixel detection signal is input into the trained CNN-Mamba hybrid neural network model to obtain a reconstructed image of the imaging target; The CNN-Mamba hybrid neural network model includes a CNN-based encoder, a Mamba-based decoder, and a multi-scale attention fusion module connecting the CNN-based encoder and the Mamba-based decoder. The multi-scale attention fusion module is jump-connected to the corresponding encoding layer in the CNN-based encoder.

[0007] The second aspect of the present invention provides an infrared single-pixel imaging device based on a hybrid neural network model, comprising an infrared single-pixel imaging system and a processor; An infrared single-pixel imaging system is used to obtain a one-dimensional under-sampled infrared single-pixel detection signal corresponding to a target to be imaged; The processor is configured to execute the steps of the infrared single-pixel imaging method based on the hybrid neural network model described in the first aspect of the present invention.

[0008] The third aspect of the present invention provides an infrared single-pixel imaging device based on a hybrid neural network model, comprising: Data acquisition module: configured to acquire a one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged according to the binary coding pattern obtained through training; Reconstruction module: configured to input the acquired infrared single-pixel detection signal into the trained CNN-Mamba hybrid neural network model to obtain a reconstructed image of the imaging target; The CNN-Mamba hybrid neural network model includes a CNN-based encoder, a Mamba-based decoder, and a multi-scale attention fusion module connecting the CNN-based encoder and the Mamba-based decoder. The multi-scale attention fusion module is jump-connected to the corresponding encoding layer in the CNN-based encoder.

[0009] Compared with the prior art, the present invention has the following beneficial effects: The hybrid neural network model constructed by the present invention uses a convolutional neural network (CNN) as an encoder, which can effectively capture local feature information in infrared single-pixel detection signals, thereby improving the representation ability of the signal; the Mamba-based decoder has a low computational cost and a large global receptive field. This enables the effective aggregation and integration of global information during the decoding process, thereby improving the quality and accuracy of the reconstructed image. At the same time, through a multi-scale attention fusion module, the high-level features output by each layer of the encoder are weighted and compressed, and combined with the corresponding decoding layer features in the decoder. This multi-scale feature fusion method can make full use of feature information at different levels, improve the model's adaptability to complex scenes, and enhance the imaging effect. Through the combination of the above-mentioned technical means, the imaging method based on the CNN-Mamba hybrid neural network model can effectively improve the reconstruction quality of the imaging target, and the obtained reconstructed image has a significant improvement in resolution and details.

[0010] The advantages of the present invention and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention but do not constitute a limitation of the present invention.

[0012] Figure 1 is a flow chart of a single pixel imaging method according to Embodiment 1 of the present invention; Figure 2It is a schematic diagram of the structure of the CNN-Mamba hybrid neural network model of Example 1 of the present invention; FIG3 (a) is a schematic diagram of the structure of a residual module in a CNN-Mamba hybrid neural network model according to Example 1 of the present invention; FIG3( b ) is a schematic diagram of the structure of a multi-scale attention fusion module in the CNN-Mamba hybrid neural network model of Example 1 of the present invention; FIG3( c ) is a schematic diagram of the structure of the Mamba module in the CNN-Mamba hybrid neural network model of Example 1 of the present invention; FIG3( d ) is a schematic diagram of the structure of the SSM block of Embodiment 1 of the present invention; Figure 4 It is a structural schematic diagram of an infrared single-pixel imaging system for realizing one-dimensional under-sampling infrared single-pixel detection signal acquisition according to Embodiment 1 of the present invention; FIG5 (a) is a comparison diagram of the peak signal-to-noise ratios of the method of the present invention and the prior art imaging method at different sampling rates in the experiment of Example 1 of the present invention; FIG5( b ) is a comparison diagram of the structural similarity between the method of the present invention and the imaging method of the prior art at different sampling rates in the experiment of Example 1 of the present invention; Figure 6 is a comparison diagram of the imaging effects of the method of the present invention and the imaging method of the prior art in the experiment of Example 1 of the present invention; Among them, 1. digital micromirror device, 2. example binary coding pattern, 3. infrared light source, 4. infrared lens, 5. infrared reflector, 6. infrared single pixel detector, 7. data acquisition card, 8. example imaging target, 9. computer. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0014] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0015] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments of the present invention and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0016] Example 1 In the technical solutions disclosed in one or more embodiments, Figures 1 to 4 As shown, an infrared single-pixel imaging method based on a hybrid neural network model includes the following steps: Step 1: According to the binary coding pattern obtained through training, a one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged is obtained; Step 2: Input the acquired infrared single pixel detection signal into the trained CNN-Mamba hybrid neural network model to obtain a reconstructed image of the imaging target; The CNN-Mamba hybrid neural network model includes a CNN-based encoder, a Mamba-based decoder, and a multi-scale attention fusion module connecting the encoder and the decoder. The multi-scale attention fusion module is jump-connected with the corresponding encoding layer in the encoder.

[0017] In this embodiment, the constructed hybrid neural network model uses CNN (convolutional neural network) as an encoder, which can give full play to its advantages in local feature extraction. The convolution operation of CNN has a strong local perception ability and can effectively capture the local feature information in the infrared single-pixel detection signal, thereby improving the representation ability of the signal. The Mamba-based decoder has a low computational cost and a large global receptive field. This enables the effective aggregation and integration of global information during the decoding process, improving the quality and accuracy of the reconstructed image. The use of Mamba to achieve low computational cost helps to improve the operating efficiency of the system and is suitable for real-time imaging application scenarios. At the same time, through the multi-scale attention fusion module, the high-level features output by each layer of the encoder are weighted and compressed, and combined with the corresponding decoding layer features in the decoder. This multi-scale feature fusion method can make full use of feature information at different levels, improve the model's adaptability to complex scenes, and enhance the imaging effect.

[0018] Through the combination of the above-mentioned technical means, the imaging method based on the CNN-Mamba hybrid neural network model can effectively improve the reconstruction quality of the imaging target, and the reconstructed image obtained has a significant improvement in resolution and details.

[0019] As a further technical solution, the CNN-based encoder adopts the residual module as the feature extraction unit to extract the local feature information of the infrared single-pixel detection signal. The multi-scale attention fusion module compresses the high-level features output by each layer of the encoder into weighted low-level features, and then combines them with the features of the corresponding decoding layer in the decoder to obtain the global information. The Mamba-based decoder is used to aggregate and integrate the global information.

[0020] In step 1, the one-dimensional under-sampled infrared single-pixel detection signal can be obtained by building an infrared single-pixel imaging system to collect the one-dimensional under-sampled infrared single-pixel detection signal corresponding to the imaging target; In some embodiments, Figure 4 As shown, the infrared single pixel imaging system includes an infrared light source 3, an infrared reflector 5, a digital micromirror device 1 (DMD), an infrared lens 4, an infrared single pixel detector 6, a data acquisition card 7, a computer 9 and a number of signal lines.

[0021] The digital micromirror device 1 (DMD) is used to load the trained binary coding pattern, and each time a pattern is refreshed, a trigger signal is sent to the data acquisition card 7; Figure 4 As shown, the leftmost QR code image is the loaded example binary code pattern 2; The infrared lens 4 is an infrared band imaging lens, which is used to project each binary coded pattern onto the area to be imaged, thereby completing active spatial encoding of the imaging target; Figure 4 In the figure, the imaging target is the example imaging target 8 on the far right; The infrared single pixel detector 6 is used to detect the total intensity of the light field after the imaging target is actively spatially encoded and generate a corresponding analog signal value; The data acquisition card 7 receives a trigger signal from a digital micromirror device (DMD), reads the analog signal strength of the infrared single pixel detector 6 at the current moment, and converts it into a digital signal; The computer 9 acquires a digital signal with a length equal to the number of coding patterns through the data acquisition card 7, that is, the one-dimensional under-sampled infrared single-pixel detection signal to be acquired.

[0022] Specifically, the working process of the above infrared single pixel imaging system is as follows: When collecting the one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged, the infrared light source 3 emits a continuous parallel light beam to illuminate the infrared reflector 5; the infrared reflector 5 makes the infrared parallel light beam enter the DMD working area at an angle of 12°; the DMD loads all the trained binary coding patterns at one time through the computer 9; after the loading is completed, all the binary coding patterns are displayed in sequence according to the set display time of the loaded binary coding patterns; each binary coding pattern displayed by the DMD is projected by the infrared camera lens to the area to be imaged, completing the active spatial coding of the imaging target.

[0023] The total intensity of the light field after encoding of the example imaging target 8 is sensed by the infrared single-pixel detector 6 and generates a corresponding analog signal value; at the same time, the DMD sends a trigger signal to the data acquisition card 7 at the beginning of each refresh of a pattern. After receiving the trigger signal, the data acquisition card 7 reads the analog signal intensity of the infrared single-pixel detector 6 at the current moment and converts it into a digital signal; after the DMD displays all the encoded patterns, the computer 9 obtains a digital signal with a length equal to the number of encoded patterns through the data acquisition card 7, that is, the one-dimensional under-sampled infrared single-pixel detection signal that needs to be collected.

[0024] Furthermore, before the infrared single-pixel detection signal is input into the CNN-based encoder, the feature dimension of the infrared single-pixel detection signal is adjusted so that the feature dimension is the same as the dimension of the reconstructed target image, that is, the feature dimension is enlarged to , is the number of channels of the target image, , The spatial resolution size of the target image; Through consistent dimensional processing, the uncertainty caused by differences in input signals can be reduced and the generalization ability of the model can be enhanced.

[0025] A further technical solution, based on a CNN-based encoder, detects local feature information of infrared single-pixel signals, including the following steps: The CNN-based encoder includes multiple residual modules connected in sequence. The cascaded residual modules downsample the input image in sequence. Each layer of residual modules performs convolution operations, activation operations, and identity mapping on the input image to obtain the feature image output by each layer of residual modules, which is the local feature information of different scales. The structure of the residual module is shown in Figure 3 (a). A further technical solution, the multi-scale attention fusion module compresses the high-level features output by each encoding layer of the encoder into weighted low-level features, including the following process: Step 21: Get the feature map output by the encoding layer, change the number of channels of the feature map through the first convolution operation, and obtain the feature dimension ;Then The feature map is subjected to three parallel convolution operations to obtain feature maps of different scales for weighted fusion; Specifically, The feature map is processed through three parallel convolution operations, namely 3×3, 5×5, and 7×7 convolutions to obtain three feature maps of different scales, and then the three feature maps of different scales are fused; In the above step 21, multiple convolution operations, specifically convolution operations with three different sizes (3×3, 5×5, 7×7) of convolution kernels, are performed on the feature map to extract information of different scales. This multi-scale feature extraction method can capture features of different sizes, which helps improve the model's ability to recognize complex structures and details. Fusion of feature maps of multiple scales effectively integrates information of different scales and improves the richness and diversity of feature expression.

[0026] Step 22: Perform the maximum pooling operation, convolution operation and activation operation on the feature map after the weighted fusion in step 21 in sequence to obtain a feature map of the same size as the feature map output by the encoding layer as the output feature map of path one; Specifically, the shape of the feature map tensor after the maximum pooling operation is , the pooled feature map passes through a 5×5 convolution operation, and then passes through a Sigmoid activation function, the output tensor is feature map, thereby keeping the output size unchanged; In the above step 22, the maximum pooling operation can reduce the size of the feature map while retaining important feature information, reducing the amount of calculation while maintaining important features. The 5×5 convolution operation is used to further extract features, and combined with the Sigmoid activation function, the output feature map has nonlinear characteristics and maintains the same size as the input feature map. This step effectively enhances and compresses the fused features, ensuring that the size of the output feature map is the same as the input feature map. Figure 1 To facilitate subsequent operations.

[0027] Step 23: Perform average pooling, convolution and activation operations on the feature map output by the coding layer to obtain a path 2 output feature map; Specifically, first, an average pooling operation is performed, and the size of the feature map tensor after pooling is , the feature map after average pooling passes through a 1×1 convolution operation, and the output shape remains unchanged. The convolution output passes through a ReLU activation function, and the output shape is The second path outputs a feature map.

[0028] In the technical solution of step 23 above, the average pooling operation can smooth the feature map, reduce noise, and obtain a smoother feature representation. Through the 1×1 convolution operation, the number of channels of the feature map can be reduced while keeping the spatial dimension of the feature map unchanged. Combined with the ReLU activation function, the output feature map has nonlinear characteristics, which enhances the feature representation capability. This step effectively generates the output feature map of path two, providing another feature representation of the input feature.

[0029] Step 24: Perform a Hadamard product operation on the output of path one and the output of path two, and add the feature map after the Hadamard product to the feature map of the input of the multi-scale fusion module. The output tensor size is Feature map, which is consistent with the tensor of the input feature map; Among them, the dimensions of the feature map tensor include batch size, number of channels, height, and width.

[0030] The Hadamard product, also known as the element-wise product, is a matrix operation that generates a new matrix by multiplying the elements in corresponding positions of two matrices element-by-element.

[0031] In the above step 24, the output feature maps of path 1 and path 2 are subjected to the Hadamard product operation, which can fuse the feature information of two different paths and improve the expressiveness and accuracy of the features. The feature map after the Hadamard product is added to the feature map input by the multi-scale fusion module to generate the final output feature map, ensuring that the dimension of the output feature map is consistent with the input feature map. Figure 1 This operation can integrate the feature information of different paths while keeping the size and shape of the feature map unchanged, which is convenient for subsequent network processing.

[0032] The above process is realized through the multi-scale attention fusion module in Figure 3 (b). The multi-scale attention fusion module with the illustrated structure performs multi-scale feature extraction, fusion, pooling, convolution and activation operations, which effectively enhances the feature representation capability and improves the robustness and generalization ability of the model.

[0033] In some embodiments, the structure of the Mamba-based decoder may be as shown in FIG. 3( c ). Based on the method of aggregating and integrating global information in the Mamba-based decoder, the Mamba-based decoder includes multiple layers of Mamba modules. The output of the Mamba module of the previous layer of the Mamba-based decoder is used as the input of the Mamba module of the next layer. Each Mamba module performs the following steps: Step 201: normalize the input feature map inside each layer of the Mamba module, and perform an activation operation based on the normalized feature map to obtain a first feature map; The Mamba module first undergoes layer normalization. Layer normalization is a regularization technique used to reduce the differences between layers within the model to improve training efficiency and generalization ability. Its principle is to normalize the feature map within each layer so that the output distribution of each layer remains stable. Step 202: multiply the feature map after the activation operation in step 201 by the weight matrix and add a bias term to realize a linear transformation of the input data; Step 203: Perform DW convolution on the linearly transformed feature map; Among them, DW convolution is depth convolution; Step 204, scan expansion: scan the feature map output by the DW convolution in four different directions to obtain a one-dimensional sequence in four directions; Step 205: Adjust the SSM according to the one-dimensional sequence in four directions. Parameters; SSM is the state space model, which is a processing module in SS2D, the core module of Mamba module; Step 206: Summarize and merge the sequences in the four directions based on the scan merging operation, and restore the output image to the same size as the input image as a feature map after scan merging; Step 207, performing layer normalization processing on the scanned and merged feature maps; Step 208: Perform a Hadamard product operation on the layer-normalized feature map and the first feature map, then perform a linear transformation, and then add the feature map input in step 201 to obtain a decoded global feature.

[0034] Further technical solutions, the CNN-Mamba hybrid neural network model mainly includes: a CNN-based encoder, a Mamba-based decoder, and a multi-scale attention fusion module; Optionally, a CNN-Mamba hybrid neural network model has a U-shaped network structure, where the left side of the U-shaped network structure is an encoder composed of a multi-layer CCN-based residual module; the middle of the U-shaped network structure is different from the traditional skip connection, and a multi-scale attention fusion module is added; the right side of the U-shaped network structure is a decoder composed of multiple Mamba modules; The number of layers of the U-shaped network structure can be set. Figure 2 As shown, the encoder includes 4 layers of residual modules and the decoder consists of 4 Mamba modules; The following is a detailed introduction to the encoder's residual module, the multi-scale attention fusion module, and the decoder's main component, the Mamba module.

[0035] In some embodiments, the structure diagram of the residual module may be as shown in FIG3(a), including an input layer, multiple convolutional layers, a nonlinear activation layer, and an output layer, wherein multiple convolutional layers are cascaded after the input layer, the activation layer is arranged between the convolutional layers and after the output layer, and the input layer and the output layer are connected by an identity mapping; Among them, the input layer is used to receive input data ; Set the activation function for each activation layer; The purpose of introducing the residual module is to enable very deep networks to be effectively trained, thereby solving the degradation problem in deep neural networks and the problem of gradient vanishing due to the network being too deep. The output of each layer depends not only on the input of the current layer, but also directly on the input of the previous layers; The operation process of the residual module can be specifically expressed as the following formula:

[0036] in, Represents the residual function to be learned, which is usually composed of several convolutional layers and nonlinear activation functions; express The parameters of the convolutional layer; is the input and is directly added to the output of the residual function through the identity mapping.

[0037] The encoder of the model of this embodiment adopts four layers of residual modules, including a first residual module, a second residual module, a third residual module and a fourth residual module. The input image is downsampled after each layer of residual module, thereby realizing multi-scale feature extraction of the input image. Correspondingly, the decoder includes a first Mamba module, a second Mamba module, a third Mamba module and a fourth Mamba module, and is also provided with a fifth Mamba module, through which the fourth residual module is connected to the fourth Mamba module; In some embodiments, the structure of the multi-scale attention fusion module may be as shown in FIG3( b ), including a path one network and a path two network; The path one network includes a first convolutional layer connected in sequence, three second convolutional layers connected in parallel, a fusion module, a maximum pooling module, a third convolutional layer, and a path one activation layer; The path 2 network includes an average pooling layer, a path 2 convolution layer I, a path 2 activation layer, and a path 2 convolution layer II; the output ends of the path 1 network and the path 2 network are connected to the Hadamard product operation module and the sum operation module in sequence, and are summed with the input after the Hadamard product operation; The multi-scale attention fusion module of this embodiment can capture features at different scales, helping the model to better understand complex contextual relationships. This multi-scale feature extraction makes the model more robust when processing objects or features of different sizes.

[0038] Specifically, the data processing process of the multi-scale attention fusion module is as follows: The input data shape is ,in is the input channel, Represents height and width respectively. The multi-scale attention fusion module is divided into two paths. Path 1: First, the shape of the output of the 1×1 convolution operation of the first convolutional layer is ; then this Through the three parallel convolution operations of the second convolutional layer, 3×3, 5×5, and 7×7 convolutions are used to obtain three feature maps of different scales, and then the three feature maps of different scales are added and fused based on the fusion module; the output after addition and fusion undergoes a maximum pooling operation, and the shape of the feature map tensor after pooling is , the pooled feature map passes through the convolution operation of the third convolution layer (convolution 5×5), and then passes through the Sigmoid activation function of the path one activation layer, and the output tensor is feature map, thus keeping the output size unchanged; Path 2: First, through an average pooling operation, the size of the tensor after pooling is , the average pooled tensor passes through the 1×1 convolution operation of the path 2 convolution layer I, and the output shape remains unchanged. The convolution output passes through the ReLU activation function of the path 2 activation layer, and then passes through the 1×1 convolution operation of the path 2 convolution layer II. The shape of the output feature map is ; The output of path 1 is Hadamard-producted with the output of path 2 to generate a new tensor. The tensor after Hadamard-product is added to the input tensor of the multi-scale fusion module, and the output tensor size is , consistent with the shape of the input tensor, and then used as part of the input to the Mamba module in the decoder.

[0039] The Mamba module of this embodiment, as shown in FIG3( c ), can realize efficient processing and prediction on sequence data.

[0040] The Mamba module improves the limitations of traditional state space models by introducing Selective State Space Models (SSM), thereby enhancing the model's ability to express complex sequence data while maintaining high time efficiency. The structure of the SSM block is shown in Figure 3 (d), and the output processing can be expressed by the following formula:

[0041] Among them, the first formula is the state quantity at the previous moment, is the input at the current moment, is the updated state quantity at the current moment; in the second formula is the output at the current moment, is the updated current state quantity and the current time input jointly decided, are trainable parameters.

[0042] The Mamba-based decoder includes multiple layers of Mamba modules. Five Mamba modules are provided in this embodiment. Except for the highest layer, each Mamba module input includes the output of the multi-scale fusion module and fuses the output of the Mamba module of the previous layer of the Mamba-based decoder to form a new Mamba module input.

[0043] A feasible technical solution, the Mamba module includes: The layer normalization module is used to normalize the input feature map inside each layer of the Mamba module, and perform activation operations based on the normalized feature map to obtain the first feature map, which can keep the output distribution of each layer stable; Layer normalization uses regularization technology to reduce the differences between layers within the model to improve training efficiency and generalization ability. The feature maps are normalized within each layer so that the output distribution of each layer remains stable.

[0044] The linear layer is used to multiply the first feature map output by the layer normalization module with the weight matrix and add the bias term to achieve a linear transformation of the input data; DW convolution, convolution operation is performed on the feature maps output by the linear layer and then Hadamard product operation is performed; DW convolution is a special convolution operation. It first performs convolution on each channel and then performs Hadamard product operation to obtain the final result. This method can greatly reduce the number of parameters while retaining the ability to extract features.

[0045] The SS2D module includes a scan extension module, a state space model (abbreviated as SSM), and a scan merging module.

[0046] Scanning expansion module: Scan the feature map output by DW convolution in four different directions to obtain four one-dimensional sequences in different directions; SSM: Adjust SSM according to input Parameters.

[0047] Scan merging module: The sequences in four directions are summarized and merged based on the scan merging operation, and the output image is restored to the same size as the input image.

[0048] The whole process can be expressed by the following formula:

[0049] in, is a sequence of four directions, is the scan extension function, To input Information is filtered by SSM. Scan merge function, yes The final output after combination.

[0050] Fusion unit: used to multiply the scanned merged feature map output by the SS2D module with the data after layer normalization by the layer normalization module. The multiplied feature data is added to the original input after passing through the linear layer to obtain the fused output feature map. The processing of the fusion unit in this embodiment can facilitate better propagation of gradients, thereby making it easier to train deeper neural networks.

[0051] This embodiment constructs a CNN-Mamba hybrid neural network model for infrared single-pixel imaging. The network model adopts a U-shaped structure, with four CNN-based residual modules as encoders for extracting local feature information, and four Mamba modules as decoders for aggregating and integrating global information. This enables the imaging network to have powerful modeling capabilities for both local and global features, thereby effectively improving the reconstruction quality of infrared target scenes under undersampling conditions.

[0052] In addition, by designing a multi-scale attention fusion module in the imaging network, the module can learn the attention between different channels based on learning local features. This makes the encoding layer and the decoding layer in the U-shaped network structure no longer a simple splicing, but compresses the high-level features into weighted low-level features and then combines them with the features of the decoding layer, further ensuring the image reconstruction performance of the CNN-Mamba hybrid neural network.

[0053] A further technical solution also includes a method for training a CNN-Mamba hybrid neural network model, comprising the following steps: S1. Based on the principle of single-pixel imaging, a supervised infrared single-pixel imaging training dataset is created; S101, single pixel imaging principle, which can be expressed by the following formula: ; in, is a one-dimensional column vector, representing a single pixel measurement signal; is a two-dimensional matrix representing a binary coding pattern, is the imaging target, which is also a one-dimensional column vector.

[0054] S102. Create a supervised infrared single-pixel imaging training dataset as follows: Select images from the public data set as training imaging scenes, process each training imaging scene according to the single-pixel imaging principle, generate a one-dimensional single-pixel measurement signal corresponding to each image, and construct a training data set; Optionally, 100,000 images in the public dataset STL-10 can be used as imaging scenes for training; then, these 100,000 images can be deformed to the required spatial resolution as needed; finally, a one-dimensional single-pixel measurement signal corresponding to each image is generated according to the single-pixel imaging principle, thereby constructing a supervised infrared single-pixel imaging training dataset containing 100,000 pairs of samples.

[0055] S2, constructing an infrared single-pixel imaging model based on a CNN-Mamba hybrid neural network, and training the constructed CNN-Mamba hybrid neural network model through a training data set; The overall structure of the infrared single-pixel imaging model of the CNN-Mamba hybrid neural network is as follows: Figure 2 As shown, the training of the hybrid neural network model mainly includes the training of two parameters, one is the binary coding pattern training, and the other is the training of the single-pixel detection signal to the target image reconstruction network.

[0056] S21. Training of binary coding patterns: S211, initialize a set of random grayscale coding patterns Before the coding pattern is multiplied by the target image, the random grayscale coding pattern needs to be binarized to obtain a binary coding pattern , the specific binarization operation is as follows:

[0057] S212, calculate binary coding pattern Coded pattern with random grayscale The difference , the random grayscale coding pattern and Difference Add together and calculate the simulated one-dimensional single pixel detection signal value based on the single pixel imaging principle ; Since the above binarization process cannot calculate the gradient through back propagation during model training, and in order to ensure that back propagation is not interrupted, we first calculate and The difference between , then and Add, and then do forward propagation. Specifically, it can be expressed as follows:

[0058] in, Represents the binarization operation process; represents the backpropagation truncation, which ensures Parameter updates can be performed via back-propagation. Then Multiplying with the target image matrix, we can get the simulated one-dimensional single pixel detection signal value .

[0059] S22, based on the obtained simulated one-dimensional single pixel detection signal value , input to the CNN-Mamba hybrid neural network to reconstruct the target image: S221, simulate one-dimensional single pixel detection signal value , feature dimension amplification based on the fully connected layer; Specifically, the input is a one-dimensional single-pixel detection signal value collected by simulation , first connected to it is a fully connected layer, which is used to enlarge the feature dimension to , is the number of channels of the target image, , The spatial resolution size of the target image.

[0060] S222, input the dimensionally enlarged features into the encoder of the CNN-Mamba hybrid neural network for multi-layer encoding to output feature maps of different scales; S223, compressing the high-level features output by each layer of the encoder into weighted low-level features, and inputting them into the Mamba module of the corresponding layer of the decoder of the Mamba base; S224, the output of the Mamba module of the upper layer of the Mamba-based decoder is input to the Mamba module of the lower layer; S225, the Mamba module aggregates and integrates to generate decoded global information to obtain a reconstructed image; S23, calculating the value of the loss function according to the obtained reconstructed image, iteratively updating the model parameters and the parameters of the binary coding pattern until the iteration termination condition is met, and obtaining the trained CNN-Mamba hybrid neural network and the trained binary coding pattern; Optional, the loss function used by the entire network model It is expressed by the following formula:

[0061] Here is the label image, This is the image reconstructed by the proposed CNN-Mamba hybrid neural network model. The gradient values ​​of each parameter in the entire network model are calculated through the loss value, including the parameters of the generated binary coding pattern that can be trained. Finally, the Adam optimizer is used to update each parameter in the entire network model.

[0062] The binary coding patterns are trained together during the training process of the CNN-Mamba hybrid neural network model. After the training, the set of binary coding patterns is fixed and applicable to any imaging target.

[0063] In the above training scheme, each image is acted on the same set of binary coding patterns to obtain a simulated one-dimensional single-pixel detection signal, and then a CNN-Mamba hybrid neural network between the one-dimensional single-pixel detection signal and the target image is trained as a reconstruction network based on the simulated one-dimensional single-pixel detection signal and the corresponding target image. After the training is completed, a set of trained coding patterns are projected onto the imaged target in the experiment, so that the infrared single-pixel detector 6 collects a real one-dimensional single-pixel detection signal, and the real single-pixel measurement signal obtained by the infrared single-pixel detector 6 is input into the trained CNN-Mamba hybrid neural network model, and the target image corresponding to the collected signal can be output, thereby obtaining an infrared high-quality target reconstructed image corresponding to the actual measurement value output by the network.

[0064] In this embodiment, the undersampled infrared single-pixel detection signal can be directly used as the input for training the network model, and the reconstructed high-quality target scene image can be directly output by running the network model, thereby avoiding the cumbersome and complex iterative steps of traditional reconstruction based on optimization algorithms such as compressed sensing theory. At the same time, the number of sampling times required for infrared single-pixel imaging also becomes lower, thereby improving the sampling speed and reconstruction speed of infrared single-pixel imaging.

[0065] In order to verify the effectiveness of the infrared single-pixel imaging method based on the CNN-Mamba hybrid neural network model, numerical simulations and real physical experiments were carried out based on the method of the present invention, and the relevant experimental results were compared with the imaging results of other existing single-pixel imaging methods. The existing methods used are shown in Table 1: Table 1 Existing methods for experimental comparison

[0066] Specifically, in this embodiment, the same experimental conditions are first set to perform a numerical simulation experiment: In the experiment, the public dataset Set12 was used as the target scene, the resolution was set to 64×64, and the sampling rates were set to 0.05, 0.1, 0.20, and 0.30, respectively. In order to quantitatively compare the imaging effects of different single-pixel imaging methods, the peak signal to noise ratio (PSNR) and structural similarity (SSIM) were used as evaluation criteria for spatial quality and visual effects. For each method, the peak signal to noise ratio (PSNR) and structural similarity (SSIM) of all reconstructed images in the Set12 dataset were averaged to obtain intuitive comparison results as shown in Figures 5 (a) and 5 (b). It can be seen that the near-infrared single-pixel imaging method based on the CNN-Mamba hybrid neural network model of the present invention is significantly superior to other existing single-pixel imaging methods in terms of PSNR and SSIM evaluation.

[0067] In order to further verify the effectiveness of the infrared single-pixel imaging method based on the CNN-Mamba hybrid neural network model of this embodiment, this method is compared with other existing infrared single-pixel imaging methods. In the real experiment, a plaster statue is used as the target scene, and the sampling rate is set to 0.1. The imaging comparison results are shown in Figure 2. Figure 6 As shown in the figure, it can be seen that in the imaging result images corresponding to other related imaging methods, the reconstructed images are relatively blurred or have obvious noise points, resulting in the inability to clearly see the image content. However, the near-infrared single-pixel imaging method of the present invention achieves the best imaging effect, and the reconstructed image has finer image details and lower noise.

[0068] Example 2 Based on Example 1, this embodiment provides an infrared single-pixel imaging device based on a hybrid neural network model, including an infrared single-pixel imaging system and a processor; An infrared single-pixel imaging system is used to obtain a one-dimensional under-sampled infrared single-pixel detection signal corresponding to a target to be imaged; The processor is configured to execute the steps of the infrared single-pixel imaging method based on the hybrid neural network model described in Example 1.

[0069] In some embodiments, Figure 4 As shown, the infrared single pixel imaging system includes an infrared light source 3, an infrared reflector 5, a digital micromirror device 1 (DMD), an infrared lens 4, an infrared single pixel detector 6, a data acquisition card 7, a computer 9 and a number of signal lines; The digital micromirror device 1 (DMD) is used to load the trained binary coding pattern, and each time a pattern is refreshed, a trigger signal is sent to the data acquisition card 7; Figure 4 As shown, the leftmost QR code image is the loaded example binary code pattern 2; The infrared lens 4 is an infrared band imaging lens, which is used to project each binary coded pattern onto the area to be imaged, thereby completing active spatial encoding of the imaging target; Figure 4 In the figure, the imaging target is the example imaging target 8 on the far right; The infrared single pixel detector 6 is used to detect the total intensity of the light field after the imaging target is actively spatially encoded and generate a corresponding analog signal value; The data acquisition card 7 receives a trigger signal from a digital micromirror device (DMD), reads the analog signal strength of the infrared single pixel detector 6 at the current moment, and converts it into a digital signal; The computer 9 acquires a digital signal with a length equal to the number of coding patterns through the data acquisition card 7, that is, the one-dimensional under-sampled infrared single-pixel detection signal to be acquired.

[0070] Specifically, the working process of the above infrared single pixel imaging system is as follows: When collecting the one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged, the infrared light source 3 emits a continuous parallel light beam to illuminate the infrared reflector 5; the infrared reflector 5 makes the infrared parallel light beam enter the DMD working area at an angle of 12°; the DMD loads all the trained binary coding patterns at one time through the computer 9; after the loading is completed, all the binary coding patterns are displayed in sequence according to the set display time of the loaded binary coding patterns; each binary coding pattern displayed by the DMD is projected by the infrared camera lens to the area to be imaged, completing the active spatial coding of the imaging target.

[0071] The total intensity of the light field after encoding of the example imaging target 8 is sensed by the infrared single-pixel detector 6 and generates a corresponding analog signal value; at the same time, the DMD sends a trigger signal to the data acquisition card 7 at the beginning of each refresh of a pattern. After receiving the trigger signal, the data acquisition card 7 reads the analog signal intensity of the infrared single-pixel detector 6 at the current moment and converts it into a digital signal; after the DMD displays all the encoded patterns, the computer 9 obtains a digital signal with a length equal to the number of encoded patterns through the data acquisition card 7, that is, the one-dimensional under-sampled infrared single-pixel detection signal that needs to be collected.

[0072] The processor of this embodiment executes the steps in Embodiment 1, and the specific implementation process is the same, which will not be described again here.

[0073] Example 3 Based on Example 1, this embodiment provides an infrared single-pixel imaging device based on a hybrid neural network model, including: Data acquisition module: configured to acquire a one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged according to the binary coding pattern obtained through training; Reconstruction module: configured to input the acquired infrared single-pixel detection signal into the trained CNN-Mamba hybrid neural network model to obtain a reconstructed image of the imaging target; The CNN-Mamba hybrid neural network model includes a CNN-based encoder, a Mamba-based decoder, and a multi-scale attention fusion module connecting the encoder and the decoder. The multi-scale attention fusion module is jump-connected with the corresponding encoding layer in the encoder.

[0074] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An infrared single-pixel imaging method based on a hybrid neural network model, characterized in that: The steps include: According to the binary coding pattern obtained through training, a one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged is obtained; The acquired infrared single-pixel detection signal is input into the trained CNN-Mamba hybrid neural network model to obtain a reconstructed image of the imaging target; The CNN-Mamba hybrid neural network model includes a CNN-based encoder, a Mamba-based decoder, and a multi-scale attention fusion module connecting the CNN-based encoder and the Mamba-based decoder. The multi-scale attention fusion module is jump-connected to the corresponding encoding layer in the CNN-based encoder.

2. The infrared single pixel imaging method based on the hybrid neural network model as claimed in claim 1, characterized in that: The CNN-based encoder uses the residual module as the feature extraction unit to extract the local feature information of the infrared single-pixel detection signal. The multi-scale attention fusion module compresses the high-level features output by each layer of the encoder into weighted low-level features, and then combines them with the features of the corresponding decoding layer in the Mamba-based decoder to obtain global information. The Mamba-based decoder is used to aggregate and integrate global information.

3. The infrared single pixel imaging method based on the hybrid neural network model as claimed in claim 1, characterized in that: Before inputting the infrared single-pixel detection signal into the CNN-based encoder, the feature dimension of the infrared single-pixel detection signal is adjusted so that the feature dimension is the same as the dimension of the reconstructed target image.

4. The infrared single pixel imaging method based on the hybrid neural network model according to claim 1, characterized in that: The CNN-based encoder includes multiple residual modules connected in sequence. The cascaded residual modules downsample the input image in sequence. Each layer of residual modules performs convolution operations, activation operations, and identity mapping on the input image to obtain the feature image output by each layer of residual modules, which is the local feature information of different scales.

5. The infrared single pixel imaging method based on the hybrid neural network model as claimed in claim 1, characterized in that: The multi-scale attention fusion module compresses the high-level features output by each encoding layer of the encoder into weighted low-level features, including the following process: Get the feature map output by the encoding layer, change the number of channels through the first convolution operation, and then obtain feature maps of different scales through three parallel convolution operations and perform weighted fusion; The weighted fused features are subjected to the maximum pooling operation, the convolution operation, and the activation operation in sequence to obtain a feature map of the same size as the feature map output by the encoding layer, which is used as the output feature map of path one; The feature map output by the encoding layer is subjected to average pooling, convolution, and activation operations in sequence to obtain the output feature map of path 2; A Hadamard product operation is performed on the output feature map of path one and the output feature map of path two, and the feature map after the Hadamard product is added to the input feature map of the multi-scale fusion module.

6. The infrared single pixel imaging method based on the hybrid neural network model according to claim 1, characterized in that: The Mamba-based decoder includes multiple layers of Mamba modules. The output of the upper layer of the Mamba module of the Mamba-based decoder is used as the input of the lower layer of the Mamba module. Each layer of the Mamba module performs the following steps: The input feature map is normalized inside each layer of the Mamba module, and an activation operation is performed based on the normalized feature map to obtain a first feature map; Performing a linear transformation on the first feature map; Perform DW convolution on the linearly transformed feature map; Scan the feature map output by the DW convolution in four different directions to obtain a one-dimensional sequence in four directions; Adjust the parameters of the state-space model in the Mamba module according to the one-dimensional sequence in four directions; Based on the scan merging operation, the one-dimensional sequences in four directions are summarized and merged, and the output image is restored to the same size as the input image as the feature map after scan merging; Perform layer normalization on the scanned and merged feature maps; The layer-normalized feature map is subjected to a Hadamard product operation with the first feature map, and then a linear transformation is performed, and then added to the feature map originally input to the Mamba module to obtain the decoded global feature.

7. The infrared single pixel imaging method based on the hybrid neural network model according to claim 1, characterized in that: The method for training the CNN-Mamba hybrid neural network model includes the following steps: Based on the principle of single-pixel imaging, a supervised infrared single-pixel imaging training dataset is created; Construct an infrared single-pixel imaging model based on the CNN-Mamba hybrid neural network, and train the constructed CNN-Mamba hybrid neural network model through the training data set as follows: Initialize a set of random grayscale coding patterns, and before multiplying the coding patterns with the target image, perform a binarization operation on the random grayscale coding patterns to obtain a binary coding pattern P ; Calculate binary coded pattern P Coded pattern with random grayscale The difference , the random grayscale coding pattern and Difference Add together and calculate the simulated one-dimensional single pixel detection signal value based on the single pixel imaging principle ; Will simulate a one-dimensional single pixel detection signal value , to enlarge the feature dimension; The dimensionally enlarged features are input into the encoder of the CNN-Mamba hybrid neural network for multi-layer encoding to output feature maps of different scales; Compress the high-level features output by each layer of the encoder into weighted low-level features and input them into the Mamba module of the corresponding layer of the decoder of the Mamba base; The output of the Mamba module on the previous layer of the Mamba-based decoder is input to the Mamba module on the next layer; The Mamba module aggregates and integrates the decoded global information to obtain the reconstructed image; The value of the loss function is calculated according to the obtained reconstructed image, and the model parameters and the parameters of the binary coding pattern are iteratively updated until the iteration termination condition is met, thereby obtaining the trained CNN-Mamba hybrid neural network and the binary coding pattern.

8. An infrared single-pixel imaging device based on a hybrid neural network model, characterized in that: Including infrared single pixel imaging system and processor; An infrared single-pixel imaging system is used to obtain a one-dimensional under-sampled infrared single-pixel detection signal corresponding to a target to be imaged; A processor is configured to execute the steps of the infrared single-pixel imaging method based on a hybrid neural network model as described in any one of claims 1 to 7.

9. The infrared single-pixel imaging device based on the hybrid neural network model as claimed in claim 8, characterized in that: The infrared single pixel imaging system includes an infrared light source, a digital micromirror device, an infrared lens, an infrared single pixel detector, a data acquisition card and a computer; A digital micromirror device is used to load the trained binary coding pattern and send a trigger signal to the data acquisition card every time a pattern is refreshed; The infrared lens is used to project each binary coded pattern onto the imaging area under the action of the infrared light source, so as to complete the active spatial coding of the imaging target; Infrared single-pixel detector, used to detect the total intensity of the light field after active spatial encoding of the imaging target and generate a corresponding analog signal value; The data acquisition card receives the trigger signal sent by the digital micromirror device, reads the analog signal strength of the infrared single-pixel detector at the current moment, and converts it into a digital signal; The computer obtains a digital signal with a length equal to the number of coded patterns through a data acquisition card, namely a one-dimensional under-sampled infrared single-pixel detection signal.

10. An infrared single-pixel imaging device based on a hybrid neural network model, characterized in that: include: Data acquisition module: configured to acquire a one-dimensional under-sampled infrared single-pixel detection signal corresponding to the target to be imaged according to the binary coding pattern obtained through training; Reconstruction module: configured to input the acquired infrared single-pixel detection signal into the trained CNN-Mamba hybrid neural network model to obtain a reconstructed image of the imaging target; The CNN-Mamba hybrid neural network model includes a CNN-based encoder, a Mamba-based decoder, and a multi-scale attention fusion module connecting the CNN-based encoder and the Mamba-based decoder. The multi-scale attention fusion module is jump-connected to the corresponding encoding layer in the CNN-based encoder.

Citation Information

Patent Citations

  • Terahertz single-pixel imaging method and system

    CN114894740A

  • Defogging method based on cross-layer attention feature interaction and multi-scale channel attention

    CN117541505A

  • Image restoration method based on state space model

    CN118195905A

  • Image segmentation method based on hybrid convolution and multi-scale attention gate

    CN118212415A

  • Optical remote sensing image segmentation method based on VMama model

    CN118365882A

Cited By

  • Compression imaging method and system based on Swin Transform network

    CN119417918A

  • Super-resolution terahertz single-pixel imaging method, system, equipment and medium

    CN119861050A

  • A super-resolution terahertz single-pixel imaging method, system, device and medium

    CN119861050B

  • Lane line detection method and device based on global feature aggregation

    CN121147863A