Image enhancement and three-dimensional reconstruction integration method, device, equipment and medium

Abstract

The invention provides an image enhancement and three-dimensional reconstruction integration method and device, equipment and a medium, and the method comprises the steps: inputting a to-be-processed image into an encoder 1, and obtaining a feature image 1 to a feature image N through layer-by-layer recursion processing; obtaining a refined image from the feature image N through feature change and a refining layer; inputting the refined image into a decoder N, fusing the current image to be decoded with a corresponding feature image in the current decoder through layer-by-layer recursion processing, optimizing the fused image into a current adjustment image through a local attention mechanism, and transmitting the current adjustment image to a next layer of decoder until an adjustment image N-1 is output; inputting the adjusted image N-1 into a decoder 1, and fusing the adjusted image N-1 with the feature image 1 to obtain a fused image 1; and inputting the fused image 1 into a decoder 0 to obtain a target image, and performing three-dimensional reconstruction to obtain a target three-dimensional model. According to the method, efficient multi-scale feature extraction and fusion and an improved channel attention mechanism are realized based on deep learning, and the generalization ability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement and three-dimensional reconstruction, and in particular to an image enhancement and three-dimensional reconstruction integrated method, device, equipment and medium. Background Art

[0002] 3D reconstruction is a key technology that extracts the shape and spatial information of objects from 2D images through computer algorithms. It has wide application value in the fields of virtual reality, augmented reality, medical imaging, robot navigation and cultural heritage protection. The mainstream 3D reconstruction methods can be roughly divided into three categories: geometric modeling technology, active 3D reconstruction and passive 3D reconstruction. Among them, the passive 3D reconstruction method relies on multi-view image registration and feature extraction, which is a hot spot in current technical research and application. However, the 3D reconstruction process is extremely sensitive to the quality of the input image. Due to the complex imaging environment (such as dim lighting conditions, dynamic blur, etc.) and the performance limitations of the acquisition equipment, the input image is often accompanied by low light, noise and blur. The degradation of these low-quality images will significantly affect the detection and matching accuracy of feature points, thereby reducing the final 3D reconstruction effect.

[0003] In order to improve the applicability of low-quality images in 3D reconstruction tasks, researchers have proposed using image enhancement techniques as a preprocessing step to improve the brightness, contrast, and clarity of the input image. Traditional low-light image enhancement techniques are generally divided into two categories: (1) value mapping-based methods, such as histogram equalization and gamma correction, which adjust grayscale values ​​to improve overall brightness; (2) model-based methods, such as Retinex theory, optimization algorithms, or deep learning models to correct illumination distribution. However, these methods can usually only solve a single degradation problem and perform poorly for complex scenes where "low light" and "blur" coexist. In addition, existing image enhancement networks often directly handle low-light or deblurring tasks with independent modules, lacking methods to coordinate and deal with multiple degradation problems. This fragmented strategy severely limits the uniformity of its effects in 3D reconstruction.

[0004] At the same time, the existing network structure for image enhancement also faces many technical bottlenecks. Specifically: (1) Insufficient fusion of multi-scale features: Multi-scale feature fusion technology is crucial to enhancing the model's ability to restore details. However, most existing enhancement networks usually use simple concatenation or weighted average strategies when processing features of different scales. This results in insufficient modeling and enhancement of information at different scales, especially high-frequency details are easily ignored, while low-frequency features are often over-adjusted, which ultimately affects the model's ability to restore details.

[0005] (2) Inefficient feature selection and channel attention mechanism: Images in natural scenes are diverse, and the information carried by features in different channels is not equally important. Although some models have introduced channel attention mechanisms to dynamically adjust the feature weights of each channel, these attention mechanisms are often too simple and cannot effectively distinguish between redundant information and important information, resulting in problems such as overfitting of irrelevant features and redundant information, thereby reducing the enhancement effect.

[0006] (3) Problems of training complexity and model generalization ability: Training deep networks to process high-resolution images usually consumes a lot of computing resources, and the existing network structure is slow to train and prone to computing bottlenecks when dealing with high-resolution scenes. In addition, the data distribution in different scenes is significantly different, and some enhanced networks lack flexibility in model architecture, so their performance on new data sets will appear unstable, thus limiting their practical application value.

[0007] In view of the above problems, in the traditional 3D reconstruction process, using a single preprocessing step or using an existing enhancement network is often not enough to meet actual needs. Specifically, the problem of insufficient optimization of the enhancement network will result in a failure to significantly improve the quality of the input image, which in turn affects the accuracy of feature point extraction or matching; in addition, the lack of comprehensive consideration of blur repair, uneven illumination correction, and noise suppression will lead to deviations in the results of key steps of 3D reconstruction (such as stereo matching or depth estimation). Therefore, an innovative integrated method is needed to combine image enhancement technology and 3D reconstruction, improve image quality while adapting to multiple degradation problems in complex imaging environments, and provide more robust input data for 3D reconstruction tasks. Summary of the invention

[0008] In order to overcome the defects of the above-mentioned prior art, the present invention provides an integrated method, device, equipment and medium for image enhancement and three-dimensional reconstruction, which realizes efficient multi-scale feature extraction and fusion and an improved channel attention mechanism based on deep learning, improves the network's sensitivity to details and global information, improves the accuracy of image restoration, optimizes the ability to capture image details, and improves generalization ability.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions: According to a first aspect of an embodiment of the present invention, a method for integrating image enhancement and three-dimensional reconstruction is provided, comprising: Get the image to be processed; The image to be processed is input into encoder 1, and through layer-by-layer recursive processing, feature images 1 to feature images N are obtained after being processed by encoders 1 to encoders N connected in sequence, so that feature images 1 to feature images N are respectively input into decoders 1 to decoders N connected in a one-to-one jump manner with encoders 1 to encoders N; the size of the feature map output by encoders 1 to encoders N decreases layer by layer and the number of channels increases layer by layer, and the size of the feature map output by decoders N to decoder 0 connected in sequence increases layer by layer and the number of channels decreases layer by layer; The feature image N is processed by feature change and refinement layer to obtain a refinement image; The refined image is input into the decoder N, and after being recursively processed layer by layer, the current image to be decoded is fused with the corresponding feature image in the current decoder, and then optimized into the current adjusted image through the local attention mechanism, and the current adjusted image is passed to the decoder of the next layer until the adjusted image N-1 is finally output; Input the adjusted image N-1 into the decoder 1, and fuse it with the feature image 1 input by the jump connection to obtain a fused image 1; Input the fused image 1 into decoder 0 to obtain a target image; A three-dimensional reconstruction is performed based on the target image to obtain a three-dimensional model of the target.

[0010] In an exemplary embodiment, the method further includes a processing step inside the encoder, specifically including: Get the current image to be encoded; Performing convolution and activation processing on the current image to be encoded to obtain a first encoded image; Downsampling the first coded image to obtain a second coded image; Performing pyramid pooling processing on the second encoded image to obtain a third encoded image; The third encoded image is processed by the curve channel attention mechanism to obtain the current feature image and output it.

[0011] In an exemplary embodiment, the method further includes a processing step inside a decoder, specifically including: Get the current image to be decoded; Performing convolution and activation processing on the current image to be decoded to obtain a decoded image; Performing selective feature fusion processing on the decoded image and the feature image input by the jump connection to obtain a current fused image; The current image to be decoded and the current fused image are added to obtain a current updated image.

[0012] In an exemplary embodiment, the optimization of the current adjusted image by the local attention mechanism includes: Obtaining the current update image; Calculating the correlation of each feature in the currently updated image, and determining the attention allocation corresponding to each feature; Scaling and normalizing the attention allocations corresponding to the features to determine the attention weights corresponding to the features; A weighted summation process is performed based on the attention weights corresponding to the respective features to obtain the current adjusted image.

[0013] In an exemplary embodiment, the selective feature fusion processing of the decoded image and the feature image input by the jump connection to obtain the current fused image includes: Performing feature splicing processing on the decoded image and the feature image input by the jump connection to obtain a spliced ​​feature map; Performing feature summation processing on the spliced ​​feature map to obtain global information; Calculate attention based on the global information to obtain a target attention weight; According to the target attention weight, the feature map is weighted fused to obtain the current fused image.

[0014] In an exemplary embodiment, the value of N is 3.

[0015] In an exemplary embodiment, when N=3, the method includes: Acquire the image to be processed; Input the image to be processed into encoder 1 to obtain the feature image 1; Inputting the feature image 1 into the encoder 2 to obtain the feature image 2; Input the feature image 2 into the encoder 3 to obtain the feature image 3; The feature image 3 is processed by feature change and refinement layer to obtain the refinement image; Input the refined image into the decoder 3, and fuse it with the feature image 3 input by the jump connection to obtain a fused image 3; The fused image 3 is optimized by the local attention mechanism 1 to obtain an adjusted image 1; Input the adjusted image 1 into the decoder 2, and fuse it with the feature image 2 input by the jump connection to obtain a fused image 2; The fused image 2 is optimized by the local attention mechanism 2 to obtain an adjusted image 2; Input the adjusted image 2 into the decoder 1, and fuse it with the feature image 1 input by the jump connection to obtain the fused image 1; Input the fused image 1 into the decoder 0 to obtain the target image; A three-dimensional reconstruction is performed based on the target image to obtain the target three-dimensional model.

[0016] According to a second aspect of an embodiment of the present invention, there is provided an image enhancement and three-dimensional reconstruction integrated device, which is implemented by using any of the above-mentioned image enhancement and three-dimensional reconstruction integrated methods, and the device includes: An acquisition module, used for acquiring an image to be processed; The encoding module is used to input the image to be processed into encoder 1, and obtain feature images 1 to feature images N respectively processed by the encoders 1 to N connected in sequence through recursive processing layer by layer, so that the feature images 1 to N are respectively input into decoders 1 to N connected in a one-to-one jump manner with the encoders 1 to N; the size of the feature map output by the encoders 1 to N decreases layer by layer and the number of channels increases layer by layer, and the size of the feature map output by the decoders N to 0 connected in sequence increases layer by layer and the number of channels decreases layer by layer; A refinement module, used for processing the feature image N through feature changes and refinement layers to obtain a refined image; A decoding module, which is used to input the refined image into the decoder N, and through layer-by-layer recursive processing, fuse the current image to be decoded with the corresponding feature image in the current decoder, optimize it into a current adjusted image through a local attention mechanism, and pass the current adjusted image to the decoder of the next layer until the adjusted image N-1 is finally output; A fusion module, used for inputting the adjusted image N-1 into the decoder 1, and fusing it with the feature image 1 input by the jump connection to obtain a fused image 1; A target module, used for inputting the fused image 1 into a decoder 0 to obtain a target image; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the target image to obtain a target three-dimensional model.

[0017] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any of the above-mentioned image enhancement and three-dimensional reconstruction integration methods.

[0018] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement any of the above-mentioned image enhancement and three-dimensional reconstruction integration methods.

[0019] By adopting the above technical solution, the present invention has the following beneficial effects: (1) Efficient multi-scale feature extraction and fusion: By introducing the pyramid pooling module and the selective convolution feature fusion module, the present invention can effectively extract image features at multiple scales and adaptively fuse these features. The pyramid pooling module captures image features of different sizes through multi-scale pooling operations, and the selective feature fusion module fuses features of different scales through dynamic weighting, thereby improving the network's sensitivity to details and global information and the accuracy of image restoration.

[0020] (2) Improved channel attention mechanism: By introducing the curve channel attention layer and the local attention mechanism module, the present invention improves the selective weighting of feature channels. The curve channel attention mechanism selects features based on multiple scales and dynamically adjusts different feature channels, so that the model can more accurately identify and enhance key areas while suppressing unimportant features, thereby optimizing the model's ability to capture image details.

[0021] (3) Improving the generalization ability of the model: While maintaining the high efficiency of the model, the present invention improves the generalization ability of different image features, making the model more stable in different data sets and different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A schematic diagram of a flow chart of an integrated method for image enhancement and three-dimensional reconstruction provided by an embodiment of the present invention; Figure 2 A detailed diagram of a process of an encoder provided by an embodiment of the present invention; Figure 3 A detailed flowchart of a decoder provided by an embodiment of the present invention; Figure 4 A general flow chart of an implementation of an image enhancement and 3D reconstruction integrated method provided by an embodiment of the present invention Figure 5 A structural block diagram of an image enhancement and three-dimensional reconstruction integrated device provided by an embodiment of the present invention; Figure 6 A hardware structure block diagram of an electronic device for running an integrated method of image enhancement and three-dimensional reconstruction provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. In the description of the embodiments of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "top", "bottom", etc. is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. Moreover, the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0026] See also Figure 1 , which is a flow chart of an image enhancement and three-dimensional reconstruction integration method provided by an embodiment of the present invention, the image enhancement and three-dimensional reconstruction integration method comprises the following steps: Step S1: Obtain an image to be processed; Step S2: Input the image to be processed into encoder 1, and obtain feature images 1 to feature images N after being processed by encoders 1 to encoders N connected in sequence, so that feature images 1 to feature images N are respectively input into decoders 1 to decoder N connected to encoders 1 to encoders N in a one-to-one jump manner; the size of the feature map output by encoders 1 to encoders N decreases layer by layer and the number of channels increases layer by layer, and the size of the feature map output by decoders N to decoder 0 connected in sequence increases layer by layer and the number of channels decreases layer by layer; Step S3: The feature image N is processed by feature change and refinement layer to obtain a refined image; Step S4: Input the refined image into decoder N, and through layer-by-layer recursive processing, fuse the current image to be decoded with the corresponding feature image in the current decoder, optimize it into the current adjusted image through the local attention mechanism, and pass the current adjusted image to the next layer of decoder until the final output is the adjusted image N-1; Step S5: input the adjusted image N-1 into decoder 1, and fuse it with the feature image 1 input by the jump connection to obtain a fused image 1; Step S6: input the fused image 1 into the decoder 0 to obtain the target image; Step S7: Perform three-dimensional reconstruction based on the target image to obtain a target three-dimensional model.

[0027] In an optional embodiment, the encoder-decoder structure can adopt LEDNet or a U-Net variant.

[0028] Specifically, LEDNet maps low-light blurry images to corresponding normal-light clear images. This network is built on an encoder-decoder architecture with filter-adaptive skip connections to solve this joint spatial variation task.

[0029] LEDNet consists of an encoder for light enhancement and a decoder for deblurring. They are connected through three filter adaptive jump connections. Inserting the pyramid pooling module PPM and the curve nonlinear unit CurveNLU in the encoder makes the light enhancement more stable and powerful. LEDNet uses the filter adaptive jump connection FASC to perform spatial adaptive transformation on the decoder from the filter generated from the enhanced features. The processing ability of different areas of the image is enhanced, thereby improving the brightness, contrast and deblurring effect of the image at the same time.

[0030] The encoder is designed for low-light enhancement with supervision of intermediate enhancement loss. It consists of three scale blocks, each of which contains a residual block, a residual downsampling block, a pyramid pooling module, and a curvilinear nonlinear unit. To facilitate intermediate supervision, the enhanced image is output through a convolutional layer at the smallest scale. This design enables the encoder to embed the input image into the feature space of the normal light image, so that the subsequent decoder can focus more on the deblurring task.

[0031] Pyramid Pooling Module. The output of a typical light enhancement network is often prone to local artifacts, which can be significantly corrected by injecting global contextual priors into the network. To achieve this goal, PPM is introduced into the encoder. PPM uses multi-scale region pooling layers to extract hierarchical global priors and aggregates them in the last convolutional layer. Using the original design of PPM, it has four average pooling branches, namely 1, 2, 3, and 6. This is the first time that PPM has been used in a low-light enhancement network. This is crucial to suppress artifacts that may be caused by the coexistence of other blurs and noise degradation Curve Nonlinear Unit. Local illumination, such as light sources, is often observed in nighttime environments. Past studies tend to over- or under-enhance these local regions. To address this problem, this module proposes a learnable nonlinear activation function CurveNLU for feature transformation using estimated curve parameters. The curve estimation module consists of three convolutional layers and a Sigmoid function, and the number of iterations n is set to 3.

[0032] The blur decoder is able to focus more on deblurring. It contains three convolutional blocks, each with two residual blocks, a residual upsampling block, and a filter adaptive convolution layer FAC to bridge the encoder and decoder.

[0033] Both low-light enhancement and deblurring are spatially varying problems. Deblurring of dynamic scenes is challenging due to spatial blur caused by object motion and depth variation. While nonlinear activation functions apply pixel-by-pixel adjustments in the encoder, this is insufficient for deblurring tasks that typically require dynamic spatial kernels to handle motion blur. This paper designs a filter-adaptive skip connection (FASC) that leverages the enhancement information from the encoder to solve the deblurring problem.

[0034] U-Net is a classic encoder-decoder structure, widely used in image segmentation and enhancement tasks. If you don't need so many complex feature fusion and attention mechanisms, you can use a simpler U-Net variant: U-Net: The standard U-Net structure consists of a downsampling encoder part and an upsampling decoder part with skip connections in between.

[0035] Attention U-Net: Introducing the attention mechanism based on U-Net, it can focus on more critical areas.

[0036] ResUNet: combines U-Net and ResNet, adds residual connections, and improves the training stability of the network.

[0037] If the task is relatively basic and you want a simpler network structure, you can choose the U-Net variant.

[0038] In an optional embodiment, the channel attention mechanism adopts a complex channel attention mechanism similar to KPAEnc and CurveCALayer, or a Transformer-based network.

[0039] Specifically, a channel attention mechanism, a knowledge prior attention mechanism, and feature fusion based on selective attention are added. The combination of these mechanisms enables the model to perform more sophisticated feature selection and weighting at the three levels of channel, space, and scale, greatly enhancing the ability to capture key information, thereby improving the overall performance of the model. In addition, the present invention combines the improved model with three-dimensional reconstruction to achieve better modeling results.

[0040] Transformer and its variants (such as Vision Transformer, DETR) have achieved good results in image processing tasks.

[0041] Vision Transformer (ViT): Divides the image into blocks and performs self-attention on each block. ViT tends to show stronger learning capabilities for larger images and complex tasks.

[0042] Swin Transformer: Similar to ViT, but with a hierarchical structure that can perform better in processing images of different resolutions more efficiently.

[0043] If you want the network to better capture global information and the task is more complex, you can choose the Transformer model.

[0044] like Figure 2 As shown, in an optional embodiment, the method further includes a processing step inside the encoder, specifically including: Get the current image to be encoded; Performing convolution and activation processing on the current image to be encoded to obtain a first encoded image; Downsampling the first encoded image to obtain a second encoded image; Performing pyramid pooling processing on the second encoded image to obtain a third encoded image; The third encoded image is processed by the curve channel attention mechanism to obtain the current feature image and output it.

[0045] like Figure 3 As shown, in an optional embodiment, the method further includes a processing step inside the decoder, specifically including: Get the current image to be decoded; Perform convolution and activation processing on the current image to be decoded to obtain a decoded image; Performing selective feature fusion processing on the decoded image and the feature image input by the skip connection to obtain the current fused image; The current image to be decoded and the current fused image are added to obtain the current updated image.

[0046] In an optional embodiment, the above optimization of the local attention mechanism into the current adjusted image includes: Get the current update image; Calculate the correlation of each feature in the current updated image and determine the attention allocation corresponding to each feature; Scaling and normalizing are performed based on the attention allocation corresponding to each feature to determine the attention weight corresponding to each feature; A weighted summation process is performed based on the attention weights corresponding to each feature to obtain the current adjusted image.

[0047] In an optional embodiment, the above-mentioned selective feature fusion processing of the decoded image and the feature image input by the skip connection to obtain the current fused image includes: Perform feature concatenation processing on the decoded image and the feature image input by the jump connection to obtain a concatenated feature map; Perform feature summation processing on the spliced ​​feature map to obtain global information; Calculate attention based on global information and obtain the target attention weight; According to the target attention weight, the feature map is weighted fused to obtain the current fused image.

[0048] like Figure 4 As shown, in a preferred embodiment, the value of N is 3. When N=3, the above method includes: Get the image to be processed; Input the image to be processed into encoder 1 to obtain feature image 1; Input feature image 1 into encoder 2 to obtain feature image 2; Input feature image 2 into encoder 3 to obtain feature image 3; Feature image 3 is processed through feature change and refinement layer to obtain a refined image; Input the refined image into decoder 3 and fuse it with the feature image 3 input by the jump connection to obtain a fused image 3; After the fused image 3 is optimized by the local attention mechanism 1, the adjusted image 1 is obtained; Input the adjusted image 1 into decoder 2 and fuse it with the feature image 2 input by the jump connection to obtain the fused image 2; After the fused image 2 is optimized by the local attention mechanism 2, the adjusted image 2 is obtained; The adjusted image 2 is input into the decoder 1, and fused with the feature image 1 input by the jump connection to obtain the fused image 1; Input the fused image 1 into decoder 0 to obtain the target image; Three-dimensional reconstruction is performed based on the target image to obtain a three-dimensional model of the target.

[0049] like Figure 2-Figure 4 As shown, in a specific embodiment, the model established by the image enhancement and three-dimensional reconstruction integrated method includes: (1) Input layer The input is an image of shape (B, 3, H, W) (i.e., batch size B, number of channels 3, image height H, width W).

[0050] (2) Encoder part The encoder is composed of multiple basic blocks, which gradually reduce the size of the feature map and increase the number of channels. ch1, ch2, ch3, and ch4 represent the number of channels in each layer of the network. They are set according to the experimental design or model architecture requirements. In this model, they are set to [32, 64, 128, 256], which is the number of channels that gradually increases from the input layer to the last layer (in the encoder and decoder).

[0051] Encoder 1: The input is an image of (3, H, W), which is mapped to the ch1 channel through a convolutional layer, convolved twice, merged through a residual connection, and finally downsampled. The output feature map size becomes (ch2, H / 2, W / 2).

[0052] Encoder 2: The output of encoder 1 is passed in, and after convolution and downsampling, the number of output channels becomes ch3, and the output size is (ch3, H / 4, W / 4).

[0053] Encoder 3: The input is the output of encoder 2. After convolution, residual block, pooling, etc., the output size becomes (ch4, H / 8, W / 8).

[0054] (3) Pyramid Pooling (PPM) and Curve Channel Attention (CurveCA) At each stage of the encoder, PPM and CurveCA are used to enhance the expressiveness of feature maps. Pyramid pooling performs multi-scale pooling on feature maps through different pooling layers to fuse information of different scales. Then it is transformed through convolution and activation functions. This can capture global information at different scales. Different PPMs are used in encoder 1, encoder 2, and encoder 3. Their role is to enhance semantic information at different levels. CurveCA is a channel attention mechanism used to enhance the features of specific channels. It learns the importance of each channel through a series of convolution operations and Sigmoid activation functions, and applies it to the feature map. Different CurveCAs are applied in the three encoders.

[0055] (4) The feature transformation and refinement layer is a key module in the decoder part. It is used to process high-level features and further enhance the expressiveness of the network. Through two consecutive convolution operations and residual connections, they help restore the details of the image and prepare for subsequent upsampling and feature fusion. This module processes the features transmitted from encoder 3 to ensure the gradual recovery from high-level features to low-level features.

[0056] (5) Decoder The decoder part consists of multiple basic modules, each of which performs upsampling, convolution and fusion of feature maps to gradually restore the resolution of the image.

[0057] Decoder 3: Input size: (ch4, H / 8, W / 8), convolution, residual block processing, the output channel number remains ch4, the size is still (ch4, H / 8, W / 8). Then, it is selectively fused with the feature map in encoder 3 (through jump connection 3). These operations ensure that the network can pay attention to features of different scales during reconstruction.

[0058] Decoder 2: Input size: (ch4, H / 8, W / 8), output channel number becomes ch3, size becomes (ch3,H / 4, W / 4). Selective attention fusion is performed with the feature map in encoder 2 (via skip connection 2).

[0059] Decoder 1: Input size: (ch3, H / 4, W / 4), the number of output channels becomes ch2, the size becomes (ch2, H / 2, W / 2), after feature fusion, it is selectively fused with the feature map in encoder 1 (through skip connection 1).

[0060] Decoder 0: Input size: (ch2, H / 2, W / 2), convolution and residual block processing are performed through decoder 0, and the final output image is generated. Output size: (3, H, W), that is, restored to the same size and number of channels as the input image (RGB image).

[0061] (6) In the decoding stage, selective feature fusion (SKFusion) is used to perform multi-scale feature fusion, fusing features at different levels by weighted averaging to achieve cross-scale fusion. This module achieves weighted fusion of features by adaptively assigning weights to each input feature map. It does this through the following steps: Feature concatenation: concatenate multiple input feature maps into one large feature map.

[0062] Feature summation: Sum the concatenated feature maps to obtain global information.

[0063] Compute attention: Generate weights for each input feature map via global average pooling and a small MLP.

[0064] Weighted fusion: According to the learned attention weights, the input feature maps are weighted summed to obtain the final fused feature map.

[0065] (7) Local Attention Mechanism Module (KPAEnc): This is a module used to learn local feature enhancement.

[0066] Input Query (Q): query vector, Key (K): key vector, Value (V). Calculation process: Phase 1: Calculate the correlation (similarity) and determine the attention allocation by calculating the correlation or similarity between the query and the key. The formula is: .

[0067] Stage 2: Scaling and Normalization. In order to stabilize the training, we need to scale the dot product. is the dimension of the Key vector. The formula is: Then, we apply the softmax operation to the score to get the weight coefficient. Softmax will normalize the attention score into a probability distribution so that the sum of all weights is 1. The formula is: .

[0068] Phase 3: Weighted summation. In this phase, the obtained attention weights are used to perform weighted summation on the values ​​to obtain the final output. This step weights the value of each element according to its attention weight, thereby emphasizing those "important" elements and ignoring those "irrelevant" elements. The formula is: .

[0069] When calculating attention, the KPAEnc module introduces an attention mask, which emphasizes areas at certain locations by adding a mask matrix. This can be seen as introducing spatial context information in the attention calculation, making the module more "intelligent" when performing spatial perception. This operation helps the network to adaptively adjust attention, making the model more flexible when focusing on important areas. When calculating attention, the relationship between local blocks is emphasized, not just the global relationship. This enables the model to perform more fine-grained attention adjustments in space, which is suitable for processing tasks with complex spatial structures.

[0070] (8) 3D reconstruction module (opensfm), which is an open source image-based 3D reconstruction project. The output image of the above network is input into opensfm to obtain a clearer model.

[0071] As can be seen from the above, the present invention is a convolutional neural network that combines multiple modules and techniques to perform joint low-light enhancement and deblurring, mainly used for image processing tasks, and combined with three-dimensional reconstruction code to obtain a clearer model. The network is specially designed to consider the synergy between two interrelated tasks. A large number of experiments have demonstrated the effectiveness of this method on synthetic and real-world datasets.

[0072] It can be seen from the above technical solutions of the embodiments of the present invention that the key to the present invention is: (1) Fusion of multiple attention mechanisms The model uses three different attention mechanisms to enhance feature extraction and fusion: A nonlinear attention mechanism based on curve fitting is provided. This mechanism is applied at different stages to help improve the quality of feature maps and model the complex dependencies between channels through a Sigmoid function. This mechanism can significantly increase the model's attention to key channels and suppress unimportant channels, thereby improving the representation ability of features.

[0073] The selective feature fusion module performs selective feature fusion through an adaptive attention mechanism, combining features at different levels to improve the expressiveness of local and global features.

[0074] The local attention module performs adaptive attention weighting on the local areas of the input feature map. This module mainly calculates the correlation of local areas and dynamically adjusts the importance of each area, thereby enhancing the model's ability to perceive features in different areas. This attention mechanism can help the model focus on more meaningful local information in the image and improve the quality of feature expression.

[0075] (2) Effective skip connections and feature fusion The use of skip connections combined with selective fusion layers ensures that key information from early layers can be effectively passed to subsequent layers, improving the model's ability to fuse local details and global semantic features.

[0076] In summary, the innovation of the model lies in the creation of a powerful feature extraction and enhancement mechanism by combining multiple attention mechanisms (such as channel attention, local attention, and selective fusion) with pyramid pooling, residual blocks, adaptive convolution and other technologies. This enables the model to perform well in complex image processing tasks, effectively capture multi-scale information, and retain detailed features.

[0077] Corresponding to the image enhancement and three-dimensional reconstruction integrated method provided in the above-mentioned embodiment, an embodiment of the present invention further provides an image enhancement and three-dimensional reconstruction integrated device. Since the image enhancement and three-dimensional reconstruction integrated device provided in the embodiment of the present invention corresponds to the image enhancement and three-dimensional reconstruction integrated method provided in the above-mentioned embodiment, the implementation method of the aforementioned image enhancement and three-dimensional reconstruction integrated method is also applicable to the image enhancement and three-dimensional reconstruction integrated device provided in this embodiment, and will not be described in detail in this embodiment.

[0078] See also Figure 5 , which is a structural block diagram of an image enhancement and three-dimensional reconstruction integrated device provided by an embodiment of the present invention; the device comprises: 01: Acquisition module, used to acquire the image to be processed; 02: An encoding module, which is used to input the image to be processed into encoder 1, and obtain feature images 1 to feature images N which are processed by encoders 1 to N connected in sequence through recursive processing layer by layer, so that feature images 1 to feature images N are respectively input into decoders 1 to decoder N which are jump-connected to encoders 1 to encoder N one by one; the size of the feature map output by encoders 1 to encoder N decreases layer by layer and the number of channels increases layer by layer, and the size of the feature map output by decoders N to decoder 0 connected in sequence increases layer by layer and the number of channels decreases layer by layer; 03: Refinement module, used to process the feature image N through feature changes and refinement layers to obtain a refined image; 04: Decoding module, which is used to input the refined image into decoder N. Through layer-by-layer recursive processing, the current image to be decoded is fused with the corresponding feature image in the current decoder, and then optimized into the current adjusted image through the local attention mechanism. The current adjusted image is then passed to the next layer of decoder until the final output is the adjusted image N-1; 05: Fusion module, used to input the adjusted image N-1 into decoder 1, and fuse it with the feature image 1 input by the jump connection to obtain a fused image 1; 06: target module, used to input the fused image 1 into decoder 0 to obtain the target image; 07: 3D reconstruction module, used to perform 3D reconstruction based on the target image to obtain a target 3D model.

[0079] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0080] An embodiment of the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the image enhancement and three-dimensional reconstruction integration method provided in the above method embodiment.

[0081] The memory can be used to store software programs and modules. The processor executes various functional applications and realizes high-level autonomous driving by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, application programs required for functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0082] The method embodiments provided in the embodiments of the present invention may be executed in a computer terminal, a server or a similar computing device, that is, the above-mentioned electronic device may include a computer terminal, a server or a similar computing device. Figure 6 is a hardware structure block diagram of an electronic device for running an image enhancement and three-dimensional reconstruction integrated method provided by an embodiment of the present invention, such as Figure 6 As shown, the internal structure of the electronic device may include but is not limited to: a processor, a network interface and a memory. The processor, network interface and memory in the electronic device may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0083] Among them, the processor (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device. The network interface may optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.). Memory is a memory device in an electronic device for storing programs and data. It is understandable that the memory here can be a high-speed RAM storage device or a non-volatile memory device, such as at least one disk storage device; optionally, it can also be at least one storage device located away from the aforementioned processor. The memory provides a storage space, which stores the operating system of the electronic device, which may include but is not limited to: Windows system (an operating system), Linux (an operating system), Android (Android, a mobile operating system) system, IOS (a mobile operating system) system, etc., and the present invention is not limited to this; and, in the storage space, one or more instructions suitable for being loaded and executed by the processor are also stored, and these instructions may be one or more computer programs (including program codes). In the embodiment of this specification, the processor loads and executes one or more instructions stored in the memory to implement the image enhancement and three-dimensional reconstruction integration method provided in the above method embodiment.

[0084] An embodiment of the present invention also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the image enhancement and three-dimensional reconstruction integration method provided by the method embodiment.

[0085] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0086] It should be noted that the order of the above embodiments of the present invention is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes a specific embodiment of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-small sample image classification and parallel processing are also possible or may be advantageous.

[0087] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0088] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0089] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An integrated method of image enhancement and three-dimensional reconstruction, characterized in that: include: Get the image to be processed; The image to be processed is input into encoder 1, and through layer-by-layer recursive processing, feature images 1 to feature images N are obtained after being processed by encoders 1 to encoders N connected in sequence, so that feature images 1 to feature images N are respectively input into decoders 1 to decoders N connected in a one-to-one jump manner with encoders 1 to encoders N; the size of the feature map output by encoders 1 to encoders N decreases layer by layer and the number of channels increases layer by layer, and the size of the feature map output by decoders N to decoder 0 connected in sequence increases layer by layer and the number of channels decreases layer by layer; The feature image N is processed by feature change and refinement layer to obtain a refinement image; The refined image is input into the decoder N, and after being recursively processed layer by layer, the current image to be decoded is fused with the corresponding feature image in the current decoder, and then optimized into the current adjusted image through the local attention mechanism, and the current adjusted image is passed to the decoder of the next layer until the adjusted image N-1 is finally output; Input the adjusted image N-1 into the decoder 1, and fuse it with the feature image 1 input by the jump connection to obtain a fused image 1; Input the fused image 1 into decoder 0 to obtain a target image; A three-dimensional reconstruction is performed based on the target image to obtain a three-dimensional model of the target.

2. The method for integrating image enhancement and three-dimensional reconstruction according to claim 1, characterized in that: The method also includes processing steps inside the encoder, specifically including: Get the current image to be encoded; Performing convolution and activation processing on the current image to be encoded to obtain a first encoded image; Downsampling the first coded image to obtain a second coded image; Performing pyramid pooling processing on the second encoded image to obtain a third encoded image; The third encoded image is processed by the curve channel attention mechanism to obtain the current feature image and output it.

3. The method for integrating image enhancement and three-dimensional reconstruction according to claim 2, characterized in that: The method also includes processing steps within the decoder, specifically including: Get the current image to be decoded; Performing convolution and activation processing on the current image to be decoded to obtain a decoded image; Performing selective feature fusion processing on the decoded image and the feature image input by the jump connection to obtain a current fused image; The current image to be decoded and the current fused image are added to obtain a current updated image.

4. The method for integrating image enhancement and three-dimensional reconstruction according to claim 3, characterized in that: The optimization of the local attention mechanism to the current adjusted image includes: Obtaining the current update image; Calculating the correlation of each feature in the currently updated image, and determining the attention allocation corresponding to each feature; Scaling and normalizing the attention allocations corresponding to the features to determine the attention weights corresponding to the features; A weighted summation process is performed based on the attention weights corresponding to the respective features to obtain the current adjusted image.

5. The method for integrating image enhancement and three-dimensional reconstruction according to claim 3, characterized in that: The selective feature fusion processing of the decoded image and the feature image input by the jump connection to obtain the current fused image includes: Performing feature splicing processing on the decoded image and the feature image input by the jump connection to obtain a spliced ​​feature map; Performing feature summation processing on the spliced ​​feature map to obtain global information; Calculate attention based on the global information to obtain a target attention weight; According to the target attention weight, the feature map is weighted fused to obtain the current fused image.

6. The method for integrating image enhancement and three-dimensional reconstruction according to any one of claims 1 to 5, characterized in that: The value of N is 3.

7. The method for integrating image enhancement and three-dimensional reconstruction according to claim 6, characterized in that: When N=3, the method includes: Acquire the image to be processed; Input the image to be processed into encoder 1 to obtain the feature image 1; Inputting the feature image 1 into the encoder 2 to obtain the feature image 2; Input the feature image 2 into the encoder 3 to obtain the feature image 3; The feature image 3 is processed by feature change and refinement layer to obtain the refinement image; Input the refined image into the decoder 3, and fuse it with the feature image 3 input by the jump connection to obtain a fused image 3; The fused image 3 is optimized by the local attention mechanism 1 to obtain an adjusted image 1; Input the adjusted image 1 into the decoder 2, and fuse it with the feature image 2 input by the jump connection to obtain a fused image 2; The fused image 2 is optimized by the local attention mechanism 2 to obtain an adjusted image 2; Input the adjusted image 2 into the decoder 1, and fuse it with the feature image 1 input by the jump connection to obtain the fused image 1; Input the fused image 1 into the decoder 0 to obtain the target image; A three-dimensional reconstruction is performed based on the target image to obtain the target three-dimensional model.

8. An image enhancement and three-dimensional reconstruction integrated device, implemented by the image enhancement and three-dimensional reconstruction integrated method according to any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; The encoding module is used to input the image to be processed into encoder 1, and obtain feature images 1 to feature images N respectively processed by the encoders 1 to N connected in sequence through recursive processing layer by layer, so that the feature images 1 to N are respectively input into decoders 1 to N connected in a one-to-one jump manner with the encoders 1 to N; the size of the feature map output by the encoders 1 to N decreases layer by layer and the number of channels increases layer by layer, and the size of the feature map output by the decoders N to 0 connected in sequence increases layer by layer and the number of channels decreases layer by layer; A refinement module, used for processing the feature image N through feature changes and refinement layers to obtain a refined image; A decoding module, which is used to input the refined image into the decoder N, and through layer-by-layer recursive processing, fuse the current image to be decoded with the corresponding feature image in the current decoder, optimize it into a current adjusted image through a local attention mechanism, and pass the current adjusted image to the decoder of the next layer until the adjusted image N-1 is finally output; A fusion module, used for inputting the adjusted image N-1 into the decoder 1, and fusing it with the feature image 1 input by the jump connection to obtain a fused image 1; A target module, used for inputting the fused image 1 into a decoder 0 to obtain a target image; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the target image to obtain a target three-dimensional model.

9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image enhancement and three-dimensional reconstruction integration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the image enhancement and three-dimensional reconstruction integration method as described in any one of claims 1 to 7.

Images