Degraded stripe image enhancement and fusion method and device based on self-similar mapping mechanism

Abstract

The present invention belongs to the field of image processing technology and relates to a method and device for enhancing and fusing degraded stripe images based on a self-similar mapping mechanism. The method comprises: estimating brightness to obtain dark features and bright features; generating a pseudo-brightness feature sequence based on the dark features and bright features; calculating a first modulation distribution offset and a second modulation distribution offset; calculating a self-similar matrix in each self-similar mapping path; fusing the input first modulation distribution offset with the pseudo-brightness feature sequence and the second modulation distribution offset with the pseudo-brightness feature sequence through matrix multiplication; and combining the input with the pseudo-brightness feature sequence after batch normalization to obtain a modulation-enhanced stripe image sequence. The present invention uses a self-similar mapping mechanism to adjust the modulation offsets of overexposed and underexposed areas, thereby generating an enhanced stripe image. Through these steps, the present invention significantly improves the modulation distribution of the stripe image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and device for enhancing and fusing degraded stripe images based on a self-similar mapping mechanism. Background Art

[0002] Traditional image fusion methods have significant drawbacks when processing HDR (High Dynamic Range) streak images. Multi-exposure fusion methods, which fuse multiple images with different exposure times, can improve dynamic range to a certain extent. However, they require cumbersome data acquisition and extensive computation time, resulting in low efficiency and difficulty meeting real-time requirements. Furthermore, these methods rely on adjusting physical exposure parameters, which can damage delicate or inaccessible objects, limiting their applicability.

[0003] Physics-driven algorithms, such as those using developers, are not only complex to operate but can also damage the surface of objects, making them unsuitable for scenarios such as industrial inspection and cultural relic preservation, which require high surface integrity. Furthermore, most existing methods rely on fixed image processing logic and lack the ability to adaptively process local image features, making it difficult to accurately address complex degradation scenarios in different regions.

[0004] In recent years, deep learning methods have been introduced to the field of streak image enhancement, but existing approaches still have significant shortcomings in image fusion. While traditional convolutional neural networks (such as UNet) can extract image features, their fixed-shape convolution kernels make it difficult to flexibly adapt to the local spatial variations of modulation information and dynamic changes in modulation intensity in streak images. When processing overexposed or underexposed areas, they are unable to accurately estimate and correct for shifts in modulation distribution across different regions, resulting in problems such as streak distortion and frequency interference in the fused image. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a method and device for enhancing and fusing degraded stripe images based on a self-similar mapping mechanism.

[0006] In a first aspect, the present invention provides a method for enhancing and fusing degraded stripe images based on a self-similar mapping mechanism, comprising:

[0007] The UNet network is used to extract background intensity feature information from long-exposure images, and dark and bright features are obtained through brightness estimation.

[0008] A pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features;

[0009] Using the pseudo brightness characteristic sequence as reference information, calculating a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculating a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence;

[0010] Constructing a modulation intensity fusion mapping model, taking the first modulation distribution offset and the second modulation distribution offset as inputs for modulation intensity fusion mapping, the modulation intensity fusion mapping model assigns independent self-similar mapping paths to the inputs, and calculating a self-similar matrix in each self-similar mapping path;

[0011] The self-similarity matrix is ​​symmetrized and normalized, each row of the self-similarity matrix is ​​used as a spatial attention map, and the attention matrix is ​​used as the weight of the spatial attention map. The first modulation distribution offset of the input is combined with the pseudo brightness feature sequence, and the second modulation distribution offset is combined with the pseudo brightness feature sequence through matrix multiplication to perform feature fusion;

[0012] The feature fusion results are batch normalized and combined with the pseudo brightness feature sequence to obtain the modulation enhanced stripe image sequence.

[0013] In a second aspect, the present invention provides a degraded streak image enhancement and fusion device based on a self-similar mapping mechanism, comprising an extraction unit, a generation unit, a calculation processing unit, a model building and processing unit, a feature fusion unit, and an output unit;

[0014] The extraction unit is used to extract background intensity feature information from the long exposure image using the UNet network, and obtain dark and bright features through brightness estimation;

[0015] A generating unit, configured to generate a pseudo brightness feature sequence according to the dark feature and the bright feature using a pseudo feature generator;

[0016] a calculation processing unit, configured to calculate a first modulation distribution offset between a bright modulation characteristic sequence and a pseudo brightness characteristic sequence, and calculate a second modulation distribution offset between a dark modulation characteristic sequence and a pseudo brightness characteristic sequence, using the pseudo brightness characteristic sequence as reference information;

[0017] a model building and processing unit, configured to build a modulation intensity fusion mapping model, perform modulation intensity fusion mapping using the first modulation distribution offset and the second modulation distribution offset as inputs, wherein the modulation intensity fusion mapping model assigns independent self-similar mapping paths to the inputs and calculates a self-similar matrix in each self-similar mapping path;

[0018] A feature fusion unit is used to symmetrize and normalize the self-similarity matrix, use each row of the self-similarity matrix as a spatial attention map, use the attention matrix as the weight of the spatial attention map, and perform feature fusion of the first modulation distribution offset and the pseudo brightness feature sequence, as well as the second modulation distribution offset and the pseudo brightness feature sequence, by matrix multiplication;

[0019] The output unit is used to batch normalize the feature fusion results and combine them with the pseudo brightness feature sequence to obtain a modulation-enhanced stripe image sequence.

[0020] On the basis of the above technical solution, the present invention can also be improved as follows.

[0021] Furthermore, the UNet network is used to extract background intensity feature information of overexposed and underexposed areas from the long-exposure image, and the dark and bright features are obtained through brightness estimation, including:

[0022] Set the exposure time threshold, the long exposure time is greater than the exposure time threshold, and the short exposure time is less than the exposure time threshold. Use the four-step phase-shifted image sequences taken under long exposure time and short exposure time as the input of the UNet network;

[0023] Extract background intensity feature information of each four-step phase-shifted image sequence;

[0024] According to the background intensity characteristic information, the bright modulation feature sequence corresponding to the four-step phase-shift image sequence shot under long exposure time is calculated to obtain the dark feature, and the dark modulation feature sequence corresponding to the four-step phase-shift image sequence shot under short exposure time is calculated to obtain the bright feature.

[0025] Furthermore, a pseudo feature generator is used to generate a pseudo brightness feature sequence according to the dark feature and the bright feature, including: setting is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , the pseudo brightness feature sequence is , , represents the pseudo brightness feature map, then the pseudo brightness feature sequence is expressed as:

[0026] .

[0027] Furthermore, the four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are spliced ​​in the channel dimension and input into the mapping network. The weights of the four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are calculated, and the weight of each channel is extracted from the output weight map. The weight of each channel is multiplied by the input corresponding to the channel, and the final pseudo-brightness feature sequence is obtained by weighted summation.

[0028] Furthermore, using the pseudo brightness characteristic sequence as reference information, calculating a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculating a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence, including:

[0029] set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are:

[0030] ;

[0031] .

[0032] Furthermore, a modulation intensity fusion mapping model is constructed, and the first modulation distribution offset and the second modulation distribution offset are used as inputs for modulation intensity fusion mapping, including: inputting the first modulation distribution offset and the second modulation distribution offset into the modulation intensity fusion mapping model, and the modulation intensity fusion mapping model uses two independent convolution kernels to extract the spatial offset and the modulation offset, and obtains the stripe image sequence after the modulation intensity fusion mapping; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, , the first modulation distribution offset is , , the second modulation distribution offset is , , Represents the modulation intensity fusion mapping, and the fringe image sequence after the modulation intensity fusion mapping is ,but:

[0033] .

[0034] Further, suppose represents matrix multiplication, and the self-similar matrix is , and represents the convolution kernel, is through The feature map obtained, is through The feature map obtained, represents the brightness offset feature map, represents the darkness offset feature map, represents the pseudo brightness feature map, , then the self-similar matrix is:

[0035] .

[0036] Furthermore, the self-similarity matrix is ​​symmetrized and normalized, each row of the self-similarity matrix is ​​used as a spatial attention map, the attention matrix is ​​used as the weight of the spatial attention map, and the first modulation distribution offset of the input is combined with the pseudo brightness feature sequence and the second modulation distribution offset is combined with the pseudo brightness feature sequence through matrix multiplication, including:

[0037] Let the self-similarity matrix be , the attention matrix is , the self-similar matrix Symmetric and normalized, self-similar matrix Each row of is used as a spatial attention map, and the attention matrix is As the weight of the spatial attention map, the first modulation distribution offset of the input is connected with the second modulation distribution offset and the pseudo brightness feature sequence through matrix multiplication, and the connection result is added to the self-similar feature. Let and To pass The weight matrix extracted by convolution, is the feature fusion operation, The attention map weights for the brightness offset and darkness offset extracted by applying the convolution kernel, , is the self-similar matrix extracted by applying the convolution kernel, for The attention map weight is:

[0038] .

[0039] Furthermore, the feature fusion results are batch normalized and combined with the pseudo brightness feature sequence to obtain a modulation-enhanced stripe image sequence, including:

[0040] Assume that the fused modulation enhanced stripe image sequence is , is the pseudo brightness feature sequence, represents batch normalization, and To pass The weight matrix extracted by convolution, is the brightness feature fusion operation, is the dark feature fusion operation, is the pseudo brightness feature sequence The self-similar matrix of , then:

[0041] .

[0042] The beneficial effects of the present invention are as follows: the present invention uses a Unet network to extract intensity feature maps of overexposed and underexposed areas from input long-exposure and short-exposure images; then, a pseudo-feature generator is used to generate a pseudo-luminance feature map based on these extracted dark and bright features; a first modulation distribution offset between the bright modulation feature sequence and the pseudo-luminance feature sequence is calculated, as is a second modulation distribution offset between the dark modulation feature sequence and the pseudo-luminance feature sequence; and an innovative modulation intensity fusion mapping model is proposed, which uses a self-similar mapping mechanism to adjust the modulation offsets of overexposed and underexposed areas, thereby generating an enhanced streak image. Through these steps, the present invention significantly improves the modulation distribution of the streak image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the degraded stripe image enhancement and fusion method based on the self-similar mapping mechanism provided in Example 1 of the present invention;

[0044] Figure 2 This is a flow chart of the degraded stripe image enhancement and fusion method based on the self-similar mapping mechanism;

[0045] Figure 3 This is the principle block diagram of the self-similar mapping mechanism;

[0046] Figure 4 This is a schematic diagram of the principle of the degraded stripe image enhancement and fusion device based on the self-similar mapping mechanism provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0048] Example 1

[0049] As an example, as shown in the attached Figure 1 As shown, to solve the above technical problems, this embodiment provides a degraded stripe image enhancement and fusion method based on a self-similar mapping mechanism, including:

[0050] The UNet network is used to extract background intensity feature information from long-exposure images, and dark and bright features are obtained through brightness estimation.

[0051] A pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features;

[0052] Using the pseudo brightness characteristic sequence as reference information, calculating a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculating a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence;

[0053] Constructing a modulation intensity fusion mapping model, taking the first modulation distribution offset and the second modulation distribution offset as inputs for modulation intensity fusion mapping, the modulation intensity fusion mapping model assigns independent self-similar mapping paths to the inputs, and calculating a self-similar matrix in each self-similar mapping path;

[0054] The self-similarity matrix is ​​symmetrized and normalized, each row of the self-similarity matrix is ​​used as a spatial attention map, and the attention matrix is ​​used as the weight of the spatial attention map. The first modulation distribution offset of the input is combined with the pseudo brightness feature sequence, and the second modulation distribution offset is combined with the pseudo brightness feature sequence through matrix multiplication to perform feature fusion;

[0055] The feature fusion results are batch normalized and combined with the pseudo brightness feature sequence to obtain the modulation enhanced stripe image sequence.

[0056] As attached Figure 2As shown, the present invention uses a Unet network to extract intensity feature maps of overexposed and underexposed areas from the input long-exposure and short-exposure images. Subsequently, a pseudo-feature generator is used to generate a pseudo-luminance feature map based on these extracted dark and bright features. A first modulation distribution offset is calculated between the bright modulation feature sequence and the pseudo-luminance feature sequence, as is a second modulation distribution offset between the dark modulation feature sequence and the pseudo-luminance feature sequence. The present invention also proposes an innovative modulation intensity fusion mapping model that uses a self-similar mapping mechanism to adjust the modulation offsets of overexposed and underexposed areas, thereby generating an enhanced streak image. Through these steps, the present invention significantly improves the modulation distribution and dynamic range of the streak image.

[0057] Optionally, use the UNet network to extract background intensity feature information of overexposed and underexposed areas from the long-exposure image, and obtain dark and bright features through brightness estimation, including:

[0058] Set the exposure time threshold, the long exposure time is greater than the exposure time threshold, and the short exposure time is less than the exposure time threshold. Use the four-step phase-shifted image sequences taken under long exposure time and short exposure time as the input of the UNet network;

[0059] Extract background intensity feature information of each four-step phase-shifted image sequence;

[0060] According to the background intensity characteristic information, the bright modulation feature sequence corresponding to the four-step phase-shift image sequence shot under long exposure time is calculated to obtain the dark feature, and the dark modulation feature sequence corresponding to the four-step phase-shift image sequence shot under short exposure time is calculated to obtain the bright feature.

[0061] Optionally, using a pseudo feature generator to generate a pseudo brightness feature sequence based on dark features and bright features, including: setting is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , the pseudo brightness feature sequence is , , represents the pseudo brightness feature map, then the pseudo brightness feature sequence is expressed as:

[0062] .

[0063] Optionally, using the pseudo brightness characteristic sequence as reference information, calculating a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculating a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence, includes:

[0064] set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are:

[0065] ;

[0066] .

[0067] Optionally, the four-step phase-shift image sequence, dark features, and bright features taken under long exposure time and short exposure time are spliced ​​in the channel dimension and input into the mapping network, the weights of the four-step phase-shift image sequence, dark features, and bright features taken under long exposure time and short exposure time are calculated, the weight of each channel is extracted from the output weight map, the weight of each channel is multiplied by the input corresponding to the channel, and the final pseudo-brightness feature sequence is obtained by weighted summation.

[0068] Optionally, using the pseudo brightness characteristic sequence as reference information, calculating a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculating a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence, includes:

[0069] set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are:

[0070] ;

[0071] .

[0072] Optionally, a modulation intensity fusion mapping model is constructed, and the first modulation distribution offset and the second modulation distribution offset are used as input for modulation intensity fusion mapping, including: inputting the first modulation distribution offset and the second modulation distribution offset into the modulation intensity fusion mapping model, and the modulation intensity fusion mapping model uses two independent convolution kernels to extract the spatial offset and the modulation offset to obtain a stripe image sequence after modulation intensity fusion mapping; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, , the first modulation distribution offset is , , the second modulation distribution offset is , , Represents the modulation intensity fusion mapping, and the fringe image sequence after the modulation intensity fusion mapping is ,but:

[0073] .

[0074] Optional, set represents matrix multiplication, and the self-similar matrix is , and represents the convolution kernel, is through The feature map obtained, is through The feature map obtained, represents the brightness offset feature map, represents the darkness offset feature map, represents the pseudo brightness feature map, , then the self-similar matrix is:

[0075] .

[0076] As attached Figure 3 The principle block diagram of the self-similar mapping mechanism is shown in the figure. The frame line V represents the self-similar mapping process, and N1 is the first modulation distribution offset. , N2 is the pseudo brightness feature sequence , N3 is the second modulation distribution offset , C1 represents , C2 represents , C3 represents , T1 represents the second modulation distribution offset pass The brightness feature map obtained, T2 represents the pseudo brightness feature sequence pass The obtained characteristic map, T3 represents the first modulation distribution offset pass The obtained characteristic map, T4 represents the second modulation distribution offset pass The obtained feature map, T5 represents the pseudo brightness feature sequence pass The obtained characteristic map, T6 represents the first modulation distribution offset pass The obtained characteristic map, T7 represents the second modulation distribution offset The corresponding self-similarity matrix, T8 represents the pseudo brightness feature sequence The corresponding self-similar matrix, T9 represents the second modulation distribution offset The corresponding self-similar matrix, X represents matrix multiplication, q1 represents , q2 represents , q3 represents , q4 means , P represents batch normalization, N4 represents the fused modulated enhanced stripe image sequence .

[0077] Optionally, the self-similarity matrix is ​​symmetrized and normalized, each row of the self-similarity matrix is ​​used as a spatial attention map, the attention matrix is ​​used as a weight of the spatial attention map, and the first modulation distribution offset of the input is combined with the pseudo brightness feature sequence and the second modulation distribution offset is combined with the pseudo brightness feature sequence through matrix multiplication, including:

[0078] Let the self-similarity matrix be , the attention matrix is , the self-similar matrix Symmetric and normalized, self-similar matrix Each row of is used as a spatial attention map, and the attention matrix is As the weight of the spatial attention map, the first modulation distribution offset of the input is connected with the second modulation distribution offset and the pseudo brightness feature sequence through matrix multiplication, and the connection result is added to the self-similar feature. Let and To pass The weight matrix extracted by convolution, is the feature fusion operation, The attention map weights for the brightness offset and darkness offset extracted by applying the convolution kernel, , is the self-similar matrix extracted by applying the convolution kernel, for The attention map weight is:

[0079] .

[0080] Optionally, the feature fusion result is batch normalized and then combined with the pseudo brightness feature sequence to obtain a modulation-enhanced fringe image sequence, including:

[0081] Assume that the fused modulation enhanced stripe image sequence is , is the pseudo brightness feature sequence, represents batch normalization, and To pass The weight matrix extracted by convolution, is the brightness feature fusion operation, is the dark feature fusion operation, is the pseudo brightness feature sequence The self-similar matrix of , then:

[0082] .

[0083] In the above process, by performing self-similarity mapping, the self-similarity matrix and attention matrix of each input sequence are extracted, so that the fusion process can highlight features with stronger modulation correlation while suppressing redundant or irrelevant information.

[0084] Example 2

[0085] Based on the same principle as the method shown in Example 1 of the present invention, as shown in the attached Figure 3 As shown, an embodiment of the present invention further provides a degraded stripe image enhancement and fusion device based on a self-similar mapping mechanism, comprising an extraction unit, a generation unit, a calculation processing unit, a model building and processing unit, a feature fusion unit and an output unit;

[0086] The extraction unit is used to extract background intensity feature information from the long exposure image using the UNet network, and obtain dark and bright features through brightness estimation;

[0087] A generating unit, configured to generate a pseudo brightness feature sequence according to the dark feature and the bright feature using a pseudo feature generator;

[0088] a calculation processing unit, configured to calculate a first modulation distribution offset between a bright modulation characteristic sequence and a pseudo brightness characteristic sequence, and calculate a second modulation distribution offset between a dark modulation characteristic sequence and a pseudo brightness characteristic sequence, using the pseudo brightness characteristic sequence as reference information;

[0089] a model building and processing unit, configured to build a modulation intensity fusion mapping model, perform modulation intensity fusion mapping using the first modulation distribution offset and the second modulation distribution offset as inputs, wherein the modulation intensity fusion mapping model assigns independent self-similar mapping paths to the inputs and calculates a self-similar matrix in each self-similar mapping path;

[0090] A feature fusion unit is used to symmetrize and normalize the self-similarity matrix, use each row of the self-similarity matrix as a spatial attention map, use the attention matrix as the weight of the spatial attention map, and perform feature fusion of the first modulation distribution offset and the pseudo brightness feature sequence, as well as the second modulation distribution offset and the pseudo brightness feature sequence, by matrix multiplication;

[0091] The output unit is used to batch normalize the feature fusion results and combine them with the pseudo brightness feature sequence to obtain a modulation-enhanced stripe image sequence.

[0092] Optionally, use the UNet network to extract background intensity feature information of overexposed and underexposed areas from the long-exposure image, and obtain dark and bright features through brightness estimation, including:

[0093] Set the exposure time threshold, the long exposure time is greater than the exposure time threshold, and the short exposure time is less than the exposure time threshold. Use the four-step phase-shifted image sequences taken under long exposure time and short exposure time as the input of the UNet network;

[0094] Extract background intensity feature information of each four-step phase-shifted image sequence;

[0095] According to the background intensity characteristic information, the bright modulation feature sequence corresponding to the four-step phase-shift image sequence shot under long exposure time is calculated to obtain the dark feature, and the dark modulation feature sequence corresponding to the four-step phase-shift image sequence shot under short exposure time is calculated to obtain the bright feature.

[0096] Optionally, using a pseudo feature generator to generate a pseudo brightness feature sequence based on dark features and bright features, including: setting is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , the pseudo brightness feature sequence is , , represents the pseudo brightness feature map, then the pseudo brightness feature sequence is expressed as:

[0097] .

[0098] Optionally, the four-step phase-shift image sequence, dark features, and bright features taken under long exposure time and short exposure time are spliced ​​in the channel dimension and input into the mapping network, the weights of the four-step phase-shift image sequence, dark features, and bright features taken under long exposure time and short exposure time are calculated, the weight of each channel is extracted from the output weight map, the weight of each channel is multiplied by the input corresponding to the channel, and the final pseudo-brightness feature sequence is obtained by weighted summation.

[0099] Optionally, using the pseudo brightness characteristic sequence as reference information, calculating a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculating a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence, includes:

[0100] set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are:

[0101] ;

[0102] .

[0103] Optionally, a modulation intensity fusion mapping model is constructed, and the first modulation distribution offset and the second modulation distribution offset are used as input for modulation intensity fusion mapping, including: inputting the first modulation distribution offset and the second modulation distribution offset into the modulation intensity fusion mapping model, and the modulation intensity fusion mapping model uses two independent convolution kernels to extract the spatial offset and the modulation offset to obtain a stripe image sequence after modulation intensity fusion mapping; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, , the first modulation distribution offset is , , the second modulation distribution offset is , , Represents the modulation intensity fusion mapping, and the fringe image sequence after the modulation intensity fusion mapping is ,but:

[0104] .

[0105] Optional, set represents matrix multiplication, and the self-similar matrix is , and represents the convolution kernel, is through The feature map obtained, is through The feature map obtained, represents the brightness offset feature map, represents the darkness offset feature map, represents the pseudo brightness feature map, , then the self-similar matrix is:

[0106] .

[0107] Optionally, the self-similarity matrix is ​​symmetrized and normalized, each row of the self-similarity matrix is ​​used as a spatial attention map, the attention matrix is ​​used as a weight of the spatial attention map, and the first modulation distribution offset of the input is combined with the pseudo brightness feature sequence and the second modulation distribution offset is combined with the pseudo brightness feature sequence through matrix multiplication, including:

[0108] Let the self-similarity matrix be , the attention matrix is , the self-similar matrix Symmetric and normalized, self-similar matrix Each row of is used as a spatial attention map, and the attention matrix is As the weight of the spatial attention map, the first modulation distribution offset of the input is connected with the second modulation distribution offset and the pseudo brightness feature sequence through matrix multiplication, and the connection result is added to the self-similar feature. Let and To pass The weight matrix extracted by convolution, is the feature fusion operation, The attention map weights for the brightness offset and darkness offset extracted by applying the convolution kernel, , is the self-similar matrix extracted by applying the convolution kernel, for The attention map weight is:

[0109] .

[0110] Optionally, the feature fusion result is batch normalized and then combined with the pseudo brightness feature sequence to obtain a modulation-enhanced fringe image sequence, including:

[0111] Assume that the fused modulation enhanced stripe image sequence is , is the pseudo brightness feature sequence, represents batch normalization, and To pass The weight matrix extracted by convolution, is the brightness feature fusion operation, is the dark feature fusion operation, is the pseudo brightness feature sequence The self-similar matrix of , then:

[0112] .

[0113] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A degraded stripe image enhancement and fusion method based on self-similar mapping mechanism, characterized by: include: The UNet network is used to extract background intensity feature information from long-exposure images, and dark and bright features are obtained through brightness estimation. A pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features; Using the pseudo brightness characteristic sequence as reference information, calculating a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculating a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence; Constructing a modulation intensity fusion mapping model, taking the first modulation distribution offset and the second modulation distribution offset as inputs for modulation intensity fusion mapping, the modulation intensity fusion mapping model assigns independent self-similar mapping paths to the inputs, and calculating a self-similar matrix in each self-similar mapping path; A modulation intensity fusion mapping model is constructed, and the first modulation distribution offset and the second modulation distribution offset are used as input for modulation intensity fusion mapping, including: inputting the first modulation distribution offset and the second modulation distribution offset into the modulation intensity fusion mapping model, and the modulation intensity fusion mapping model uses two independent convolution kernels to extract the spatial offset and the modulation offset, and obtains the stripe image sequence after the modulation intensity fusion mapping; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, , the first modulation distribution offset is , , the second modulation distribution offset is , , Represents the modulation intensity fusion mapping, and the fringe image sequence after the modulation intensity fusion mapping is ,but: ; The modulation intensity fusion mapping model assigns independent self-similar mapping paths to the input, and calculates the self-similar matrix in each self-similar mapping path, including: each self-similar mapping path contains three convolution kernels, and the self-similar matrix is ​​calculated in each self-similar mapping path; suppose represents matrix multiplication, and the self-similar matrix is , and represents the convolution kernel, is through The feature map obtained, is through The feature map obtained, represents the brightness offset feature map, represents the darkness offset feature map, represents the pseudo brightness feature map, , then the self-similar matrix is: ; The self-similarity matrix is ​​symmetrized and normalized, each row of the self-similarity matrix is ​​used as a spatial attention map, and the attention matrix is ​​used as the weight of the spatial attention map. The first modulation distribution offset of the input is combined with the pseudo brightness feature sequence, and the second modulation distribution offset is combined with the pseudo brightness feature sequence through matrix multiplication to perform feature fusion; The feature fusion results are batch normalized and combined with the pseudo brightness feature sequence to obtain the modulation enhanced stripe image sequence.

2. The degraded stripe image enhancement and fusion method based on the self-similar mapping mechanism according to claim 1 is characterized in that: The UNet network is used to extract background intensity feature information of overexposed and underexposed areas from long-exposure images, and dark and bright features are obtained through brightness estimation, including: Set the exposure time threshold, the long exposure time is greater than the exposure time threshold, and the short exposure time is less than the exposure time threshold. Use the four-step phase-shifted image sequences taken under long exposure time and short exposure time as the input of the UNet network; Extract background intensity feature information of each four-step phase-shifted image sequence; According to the background intensity characteristic information, the bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is calculated to obtain the dark feature, and the dark modulation feature sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is calculated to obtain the bright feature.

3. The degraded stripe image enhancement and fusion method based on the self-similar mapping mechanism according to claim 1 is characterized in that: The pseudo feature generator is used to generate a pseudo brightness feature sequence based on the dark feature and the bright feature, including: setting is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , the pseudo brightness feature sequence is , , represents the pseudo brightness feature map, then the pseudo brightness feature sequence is expressed as: 。 4. The degraded stripe image enhancement and fusion method based on the self-similar mapping mechanism according to claim 1 is characterized in that: The four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are spliced ​​in the channel dimension and input into the mapping network. The weights of the four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are calculated. The weight of each channel is extracted from the output weight map, and the weight of each channel is multiplied by the input corresponding to the channel. The final pseudo-brightness feature sequence is obtained by weighted summation.

5. The degraded stripe image enhancement and fusion method based on the self-similar mapping mechanism according to claim 1 is characterized in that: The method uses the pseudo brightness characteristic sequence as reference information, calculates a first modulation distribution offset between the bright modulation characteristic sequence and the pseudo brightness characteristic sequence, and calculates a second modulation distribution offset between the dark modulation characteristic sequence and the pseudo brightness characteristic sequence, including: set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shift image, and the bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are: ； 。 6. The degraded stripe image enhancement and fusion method based on the self-similar mapping mechanism according to claim 1 is characterized in that: The self-similarity matrix is ​​symmetrized and normalized, each row of the self-similarity matrix is ​​used as a spatial attention map, and the attention matrix is ​​used as the weight of the spatial attention map. The first modulation distribution offset of the input is combined with the pseudo brightness feature sequence, and the second modulation distribution offset is combined with the pseudo brightness feature sequence through matrix multiplication, including: Let the self-similarity matrix be , the attention matrix is , the self-similar matrix Symmetric and normalized, self-similar matrix Each row of is used as a spatial attention map, and the attention matrix is As the weight of the spatial attention map, the first modulation distribution offset of the input is connected with the second modulation distribution offset and the pseudo brightness feature sequence through matrix multiplication, and the connection result is added to the self-similar feature. Let and To pass The weight matrix extracted by convolution, is the feature fusion operation, The attention map weights for the brightness offset and darkness offset extracted by applying the convolution kernel, , is the self-similar matrix extracted by applying the convolution kernel, for The attention map weight is: 。 7. The degraded stripe image enhancement and fusion method based on self-similar mapping mechanism according to claim 1 is characterized in that: The feature fusion results are batch normalized and combined with the pseudo-brightness feature sequence to obtain the modulation-enhanced stripe image sequence, including: Assume that the fused modulation enhanced stripe image sequence is , is the pseudo brightness feature sequence, represents batch normalization, and To pass The weight matrix extracted by convolution, is the brightness feature fusion operation, is the dark feature fusion operation, is the pseudo brightness feature sequence The self-similar matrix of , then: 。 8. A degraded stripe image enhancement and fusion device based on a self-similar mapping mechanism, characterized in that: It includes extraction unit, generation unit, calculation processing unit, model building and processing unit, feature fusion unit and output unit; The extraction unit is used to extract background intensity feature information from the long exposure image using the UNet network, and obtain dark and bright features through brightness estimation; A generating unit, configured to generate a pseudo brightness feature sequence according to the dark feature and the bright feature using a pseudo feature generator; a calculation processing unit, configured to calculate a first modulation distribution offset between a bright modulation characteristic sequence and a pseudo brightness characteristic sequence, and calculate a second modulation distribution offset between a dark modulation characteristic sequence and a pseudo brightness characteristic sequence, using the pseudo brightness characteristic sequence as reference information; a model building and processing unit, configured to build a modulation intensity fusion mapping model, perform modulation intensity fusion mapping using the first modulation distribution offset and the second modulation distribution offset as inputs, wherein the modulation intensity fusion mapping model assigns independent self-similar mapping paths to the inputs and calculates a self-similar matrix in each self-similar mapping path; A modulation intensity fusion mapping model is constructed, and the first modulation distribution offset and the second modulation distribution offset are used as input for modulation intensity fusion mapping, including: inputting the first modulation distribution offset and the second modulation distribution offset into the modulation intensity fusion mapping model, and the modulation intensity fusion mapping model uses two independent convolution kernels to extract the spatial offset and the modulation offset, and obtains the stripe image sequence after the modulation intensity fusion mapping; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, , the first modulation distribution offset is , , the second modulation distribution offset is , , Represents the modulation intensity fusion mapping, and the fringe image sequence after the modulation intensity fusion mapping is ,but: ; The modulation intensity fusion mapping model assigns independent self-similar mapping paths to the input, and calculates the self-similar matrix in each self-similar mapping path, including: each self-similar mapping path contains three convolution kernels, and the self-similar matrix is ​​calculated in each self-similar mapping path; suppose represents matrix multiplication, and the self-similar matrix is , and represents the convolution kernel, is through The feature map obtained, is through The feature map obtained, represents the brightness offset feature map, represents the darkness offset feature map, represents the pseudo brightness feature map, , then the self-similar matrix is: ; A feature fusion unit is used to symmetrize and normalize the self-similarity matrix, use each row of the self-similarity matrix as a spatial attention map, use the attention matrix as the weight of the spatial attention map, and perform feature fusion of the first modulation distribution offset and the pseudo brightness feature sequence, as well as the second modulation distribution offset and the pseudo brightness feature sequence, by matrix multiplication; The output unit is used to batch normalize the feature fusion results and combine them with the pseudo brightness feature sequence to obtain a modulation-enhanced stripe image sequence.

Images