Real-time dynamic super-resolution image reconstruction method and reconstruction system

Abstract

The present invention provides a real-time dynamic super-resolution image reconstruction method and a reconstruction system thereof, which belong to the field of image reconstruction technology. By constructing a meta-learning lightweight model, the computational cost is significantly reduced, making the image reconstruction process more efficient. The introduction of a hybrid attention mechanism effectively improves the accuracy of feature extraction. By weighting and fusing features at different scales, the reconstructed image can retain more detail information. The application of a deep convolutional network further enhances the expression ability of the feature map, providing richer information for subsequent upsampling and detail recovery. Finally, the combination of a bicubic interpolation method and a deep refinement network not only realizes the rapid enlargement of the image size, but also ensures that the loss of details in the image during the enlargement process is minimized, thereby significantly improving the overall quality of the reconstructed image.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a real-time dynamic super-resolution image reconstruction method and a reconstruction system thereof. Background Art

[0002] With the rapid development of digital imaging technology, the demand for high-resolution images in medical diagnosis, satellite remote sensing, security monitoring and other fields is increasing. However, due to the physical limitations of imaging equipment, network transmission bandwidth and other factors, the images actually acquired are often low-resolution, which is difficult to meet the requirements of image details and clarity in the above fields. Therefore, studying how to reconstruct high-resolution images from low-resolution images, that is, super-resolution image reconstruction technology, has important theoretical significance and practical value.

[0003] Traditional super-resolution image reconstruction methods mainly rely on interpolation, reconstruction filtering and other technologies. Although these methods are simple and easy to use, the reconstructed high-resolution images often lack sufficient detail information and have poor visual effects. In recent years, with the rise of deep learning technology, super-resolution image reconstruction methods based on convolutional neural networks (CNNs) have made significant progress. However, while pursuing high-resolution reconstruction effects, these methods often face the problems of large computational complexity and high model complexity, making them difficult to apply in real-time dynamic scenes.

[0004] Therefore, it is necessary to provide a real-time dynamic super-resolution image reconstruction method and a reconstruction system thereof to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a real-time dynamic super-resolution image reconstruction method and a reconstruction system thereof. By constructing a meta-learning lightweight model, the complexity of the model is effectively reduced and the reconstruction speed is improved. At the same time, a hybrid attention mechanism and a deep refinement network are introduced, so that the model can better capture the detailed information of the image and improve the quality of the reconstructed image.

[0006] The present invention provides a real-time dynamic super-resolution image reconstruction method, the reconstruction method comprising the following steps:

[0007] S1: Acquire a low-resolution image to be reconstructed, and preprocess the low-resolution image;

[0008] S2: using a pre-built meta-learning lightweight model to extract shallow features of the pre-processed low-resolution image at different scales to obtain a scale feature map;

[0009] S3: Use the hybrid attention mechanism to weight the shallow features in all scale feature maps, and fuse all weighted scale feature maps to obtain a fused feature map;

[0010] S4: Performing feature enhancement processing on the fused feature map using a deep convolutional network to obtain an enhanced feature map;

[0011] S5: Use the bicubic interpolation method to upsample the enhanced feature map, and use the deep refinement network to restore the details of the upsampled low-resolution image to obtain the target high-resolution image.

[0012] Preferably, in step S1, preprocessing of the low-resolution image includes size standardization, denoising and grayscale conversion.

[0013] Preferably, step S2 specifically includes the following steps:

[0014] S201: obtaining a meta-learning lightweight model based on a set of low-resolution sample images in the same field as the low-resolution image, wherein each low-resolution sample image in the set of low-resolution sample images is different from the low-resolution image to be reconstructed in terms of texture, lighting conditions and scene content;

[0015] S202: obtaining low-resolution images at different scales based on convolution kernels of different sizes;

[0016] S203: Using the trained meta-learning lightweight model, shallow features in low-resolution images at different scales are extracted to obtain scale feature maps corresponding to different scales.

[0017] Preferably, in step S201, the steps of constructing the meta-learning lightweight model include:

[0018] Obtain a low-resolution sample image set in the same field as the low-resolution image, and divide the low-resolution sample image set into a training set and a validation set;

[0019] Construct a meta-learning model based on a neural network architecture and initialize the meta-parameter φ of the meta-learning model according to the meta-learning principle;

[0020] Define a meta-learning task, and iteratively update the parameter θ of the meta-learning model according to the output of the meta-learning task, while recording the performance value of the meta-learning task, wherein the meta-learning task is to extract features from the training set, and the update formula of the parameter θ is:

[0021]

[0022] In the formula, θ * is the parameter of the updated meta-learning model, α is the learning rate, L is the loss function, D is the low-resolution sample image set, is the gradient of the parameter θ, f θ Feature extraction function based on parameter θ, x is a low-resolution sample image in D, x∈D;

[0023] After the iteration of the parameter θ is completed, the meta-parameter φ of the meta-learning model is iteratively adjusted based on the evaluation index values ​​of all meta-learning tasks on the validation set until the meta-parameter φ converges, thereby obtaining a meta-learning model;

[0024] The meta-learning model is refined and compressed into a lightweight network through the knowledge distillation method to obtain a meta-learning lightweight model.

[0025] Preferably, step S3 specifically includes the following steps:

[0026] S301: Based on the channel attention mechanism and the spatial attention mechanism, the channel attention weight and the position attention weight of each scale feature map are calculated respectively, and used as the weighting coefficient of each element in the scale feature map;

[0027] S302: performing element-by-element weighting on each scale feature map and the corresponding weighting coefficient to obtain a weighted scale feature map;

[0028] S303: performing weighted summation on the weighted scale feature maps to form a fused feature map.

[0029] Preferably, in step S301, the step of calculating the channel attention weight includes:

[0030] Perform a global average pooling operation on each scale feature map to compress the features of each channel into a feature vector;

[0031] The first fully connected layer is used to perform a linear transformation on the feature vector, and the ReLU activation function is used to perform a nonlinear transformation on the feature vector after the linear transformation. The second fully connected layer is then used to perform a linear transformation on the feature vector after the nonlinear transformation to obtain the weight of each channel.

[0032] The obtained weights are applied to each channel of the scale feature map through a channel-by-channel multiplication operation to obtain the channel attention weights of each channel in each scale feature map.

[0033] Preferably, in step S301, the step of calculating the position attention weight includes:

[0034] The 3x3 convolution is used to fuse the channel information of the scale feature map, and the position attention weight of each position in the scale feature map is calculated by the sigmoid activation function.

[0035] Preferably, step S4 specifically includes the following steps:

[0036] S401: Input the fused feature map to the initial layer of the deep convolutional network, and map the features in the fused feature map from the fusion space to the feature representation space;

[0037] S402: gradually extracting deep features in the fused feature map after feature mapping based on continuous convolutional layers and activation functions;

[0038] S403: Input the deep features into the feature enhancement layer, perform feature enhancement operation using residual connection, and finally output an enhanced feature map.

[0039] The present invention also provides a real-time dynamic super-resolution image reconstruction system, which is applied to a real-time dynamic super-resolution image reconstruction method. The reconstruction system comprises:

[0040] A preprocessing module, used for acquiring a low-resolution image to be reconstructed and preprocessing the low-resolution image;

[0041] A scale feature map acquisition module is used to extract shallow features of the preprocessed low-resolution image at different scales using a pre-built meta-learning lightweight model to obtain a scale feature map;

[0042] The fusion feature map acquisition module is used to weight the shallow features in all scale feature maps using the hybrid attention mechanism, and fuse all the weighted scale feature maps to obtain a fusion feature map;

[0043] An enhanced feature map acquisition module is used to perform feature enhancement processing on the fused feature map using a deep convolutional network to obtain an enhanced feature map;

[0044] The target high-resolution image acquisition module is used to upsample the enhanced feature map using the bicubic interpolation method, and to restore the details of the upsampled low-resolution image using the deep refinement network to obtain the target high-resolution image.

[0045] Compared with the related art, the real-time dynamic super-resolution image reconstruction method and reconstruction system provided by the present invention have the following beneficial effects:

[0046] The present invention significantly reduces the computational cost by constructing a meta-learning lightweight model, making the image reconstruction process more efficient. The introduction of the hybrid attention mechanism effectively improves the accuracy of feature extraction. By weighting and fusing features at different scales, the reconstructed image can retain more detail information. The application of deep convolutional networks further enhances the expressiveness of feature maps and provides richer information for subsequent upsampling and detail recovery. Finally, the combination of the bicubic interpolation method and the deep refinement network not only achieves rapid enlargement of the image size, but also ensures that the loss of details in the image during enlargement is minimized, thereby significantly improving the overall quality of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1A flow chart of a real-time dynamic super-resolution image reconstruction method provided by the present invention;

[0048] Figure 2 A schematic diagram of the module structure of a real-time dynamic super-resolution image reconstruction system provided by the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0050] It should also be noted that, for ease of description, only the part relevant to the present invention but not all the content is shown in the accompanying drawings. It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as the processing or method described as a flow chart. Although the flow chart describes each operation (or step) as being processed sequentially, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0051] Embodiment 1

[0052] The present invention provides a real-time dynamic super-resolution image reconstruction method, referring to Figure 1 As shown, the reconstruction method includes the following steps:

[0053] S1: Acquire a low-resolution image to be reconstructed, and preprocess the low-resolution image.

[0054] In this embodiment, the low-resolution image to be reconstructed is read from different sources (such as cameras, network downloads, databases, etc.), and since low-resolution images are often accompanied by noise, blur and other problems, preprocessing is a necessary basic step to eliminate these unfavorable factors and provide clean input for subsequent feature extraction. The preprocessing includes size standardization, denoising and grayscale to preliminarily improve the image quality and reduce interference in subsequent processing.

[0055] S2: Using a pre-built meta-learning lightweight model, shallow features of the pre-processed low-resolution image at different scales are extracted to obtain a scale feature map.

[0056] In this embodiment, the meta-learning lightweight model can quickly adapt to new tasks through the common knowledge learned from multiple related tasks, reduce the dependence on large amounts of labeled data, and then quickly learn from a small number of samples and extract features of different scales to enhance the generalization ability of the model. At the same time, the meta-learning lightweight model can reduce the number of model parameters and increase the computing speed, making it suitable for real-time applications.

[0057] S3: Use the hybrid attention mechanism to weight the shallow features in all scale feature maps, and fuse all weighted scale feature maps to obtain a fused feature map.

[0058] In this embodiment, spatial attention (focusing on what is important in the image) and channel attention (focusing on which features are important) are combined to more finely control feature fusion, so that features of different scales can be intelligently weighted to ensure that important features are enhanced and the reconstruction quality is improved. Finally, all weighted scale feature maps are merged to form a global feature representation that integrates multi-scale information, that is, a fused feature map.

[0059] S4: Perform feature enhancement processing on the fused feature map using a deep convolutional network to obtain an enhanced feature map.

[0060] In this embodiment, the complex relationship between features is mined and amplified through deep structure, so as to further refine the image details, enhance the image details and make the feature map richer.

[0061] Specifically, the fused feature map is processed with a deeper convolutional layer and a residual connection is included to optimize the learning process. The details of the obtained enhanced feature map are more obvious, which is conducive to subsequent upsampling and detail recovery.

[0062] S5: Use the bicubic interpolation method to upsample the enhanced feature map, and use the deep refinement network to restore the details of the upsampled low-resolution image to obtain the target high-resolution image.

[0063] In this embodiment, bicubic interpolation keeps the image edges smooth, and the deep refinement network supplements the details to enhance the image realism.

[0064] Specifically, the feature map is first upsampled to the target resolution through bicubic interpolation, and then a deep refinement network is used to restore details based on the upsampled map to obtain a target high-resolution image. The generated target high-resolution image is not only larger in size, but also rich in details and close to reality.

[0065] Specifically, in step S1, the preprocessing of the low-resolution image includes size standardization, denoising and grayscale conversion.

[0066] In this embodiment, the input low-resolution image is adjusted to the input size required by the meta-learning lightweight model through size normalization; since low-resolution images are often accompanied by noise, the acquired low-resolution image is denoised by the median filtering method; the color image is converted into a grayscale image through grayscale conversion to simplify the image information, remove the interference of color on subsequent processing, and reduce the computational burden.

[0067] Specifically, step S2 specifically includes the following steps:

[0068] S201: A meta-learning lightweight model is obtained based on training a low-resolution sample image set in the same field as the low-resolution image, wherein each low-resolution sample image in the low-resolution sample image set is different from the low-resolution image to be reconstructed in terms of texture, lighting conditions and scene content.

[0069] In this embodiment, in order to build a meta-learning lightweight model with strong generalization ability and adaptability, the model can learn generally applicable feature representations from other low-resolution sample images in the same field as the low-resolution image, and select samples with different textures, lighting conditions and scene contents. This can increase the diversity and robustness of the model and ensure that it can effectively extract useful information in various situations. This is especially important for real-time dynamic super-resolution reconstruction because the images in actual application scenarios vary widely.

[0070] Specifically, first, a large number of low-resolution sample image sets are collected from the same field to ensure that the samples cover a wide range of texture types (such as smooth, rough), different lighting conditions (bright, dim, backlit, etc.) and rich scene content (city, natural scenery, people, etc.) to reflect the potential diversity.

[0071] The model is then trained using this diverse set of low-resolution sample images through a meta-learning strategy, which allows the model to learn how to quickly adapt to new tasks on a range of related tasks by optimizing model parameters so that it can make accurate predictions or feature extraction after seeing a small number of examples.

[0072] During training, the model not only learns how to extract features from low-resolution images, but also learns how to effectively transfer this learning ability to new, unseen images, even if these images differ from the training samples in texture, lighting, or content.

[0073] S202: Obtain low-resolution images at different scales based on convolution kernels of different sizes.

[0074] In this embodiment, convolution kernels of different sizes are used to capture and analyze image features at different scales. Features at different scales are crucial to understanding the image content. Small convolution kernels are suitable for capturing detailed information, while large convolution kernels can capture more macroscopic structural features.

[0075] Specifically, the low-resolution image is scanned by applying convolution kernels of different sizes. Each convolution kernel size corresponds to a feature extractor of a certain scale. Small convolution kernels focus on the detailed features of the image, such as texture and edges; large convolution kernels can capture larger contextual information, such as the overall shape and layout of the object. Finally, the extracted features of different scales are integrated together to form a comprehensive feature representation, which includes both microscopic details and macroscopic structures.

[0076] S203: Using the trained meta-learning lightweight model, shallow features in low-resolution images at different scales are extracted to obtain scale feature maps corresponding to different scales.

[0077] In this embodiment, the trained meta-learning lightweight model is used to extract shallow features of low-resolution images at different scales in order to more effectively utilize the knowledge learned by the model at the feature level, especially those general features that are independent of resolution.

[0078] Specifically, the low-resolution images of different scales processed by S202 are input into the meta-learning lightweight model. The model will automatically identify and extract the most relevant features at each scale. After the features of each scale are processed by the model, a corresponding scale feature map will be output. These feature maps reflect the information of the image at different levels of abstraction. By separating features of different scales, different structures and detail levels in the image can be more clearly identified, which facilitates precise control during the fusion and enhancement process, and ultimately improves the clarity and realism of the reconstructed image.

[0079] Specifically, in step S201, the steps of constructing the meta-learning lightweight model include:

[0080] A set of low-resolution sample images in the same field as the low-resolution images is obtained, and the low-resolution sample image set is divided into a training set and a validation set.

[0081] In this embodiment, the low-resolution sample image set can provide sufficient and diverse learning materials for the meta-learning model, ensuring that the model can learn feature representations that are generally applicable in a specific field. At the same time, the training set and the validation set are divided to ensure that there is an independent data set to evaluate the model performance during the model training process, prevent overfitting, and ensure generalization ability.

[0082] Specifically, low-resolution images are collected from the target application field, covering a variety of textures, lighting, and content conditions. The entire dataset is randomly divided into training and validation sets, with a general ratio of 80% training and 20% validation to maintain the adequacy of the training data and the independence of the validation data.

[0083] A meta-learning model based on a neural network architecture is constructed, and the meta-parameters φ of the meta-learning model are initialized according to the meta-learning principles.

[0084] In this embodiment, the constructed meta-learning model can solve the adaptation problem of new tasks. By initializing meta-parameters, the model can be quickly adjusted to adapt to new low-resolution image feature extraction tasks, reducing the time and amount of data required for training from scratch.

[0085] Specifically, a convolutional neural network is selected as the deep learning architecture for image feature extraction, and a meta-learning mechanism is integrated. At the same time, the meta-learning principle is used to set the initial state of the model. These parameters can guide the model to quickly adapt to new tasks. The initialized meta-parameters help the model converge to the optimal parameter configuration for new tasks more quickly, reduce dependence on large data sets, and improve the efficiency and flexibility of model training.

[0086] Define a meta-learning task, and iteratively update the parameter θ of the meta-learning model according to the output of the meta-learning task, while recording the performance value of the meta-learning task, wherein the meta-learning task is to extract features from the training set, and the update formula of the parameter θ is:

[0087]

[0088] In the formula, θ * is the parameter of the updated meta-learning model, α is the learning rate, L is the loss function, D is the low-resolution sample image set, is the gradient of the parameter θ, f θ Feature extraction function based on parameter θ, x is a low-resolution sample image in D, x∈D.

[0089] In this embodiment, the meta-learning task defined is to extract low-resolution image features from the training set, optimize the model parameters to minimize the loss function, and use the gradient descent method to update the model parameters according to the gradient calculated by the loss function. The above formula expresses the parameter update rule. The iterative update process continuously optimizes the model parameters and improves the feature extraction capability. At the same time, the performance values ​​are recorded to provide a basis for model tuning.

[0090] After the iteration of the parameter θ is completed, the meta-parameter φ of the meta-learning model is iteratively adjusted based on the evaluation index values ​​of all meta-learning tasks on the validation set until the meta-parameter φ converges, thereby obtaining a meta-learning model.

[0091] In this embodiment, in order to find the optimal model initialization state in the entire meta-learning process, this step can ensure that the model quickly achieves good performance on the new task.

[0092] Specifically, the meta-parameters are adjusted based on the performance indicators on the validation set (such as accuracy, loss value, etc.), and the process is repeated until the meta-parameters no longer change significantly, i.e., converge. The converged meta-parameters can more effectively guide the model to quickly achieve high performance on new tasks, and after repeated testing on the validation set, the model becomes more stable and reliable.

[0093] The meta-learning model is refined and compressed into a lightweight network through the knowledge distillation method to obtain a meta-learning lightweight model.

[0094] In this embodiment, in order to achieve miniaturization of the model, reduce the demand for computing resources, and accelerate the reasoning process, the meta-learning model is used as the teacher model and the lightweight network is used as the student model. By allowing the student model to imitate the output of the teacher model and combining it with soft labels (the probability distribution of the teacher model) for training, a meta-learning lightweight model is finally obtained. While maintaining high accuracy, the computing cost and storage requirements of the model are reduced, making it suitable for real-time applications.

[0095] Specifically, step S3 specifically includes the following steps:

[0096] S301: Based on the channel attention mechanism and the spatial attention mechanism, the channel attention weight and the position attention weight of each scale feature map are calculated respectively, and used as the weighting coefficient of each element in the scale feature map.

[0097] In this embodiment, the combination of the channel attention mechanism and the spatial attention mechanism is to more finely filter out the information in each scale feature map that is critical to the super-resolution reconstruction task, ensure that the model can focus on the key features and areas in the image and ignore redundant information. At the same time, by considering the importance of channels and space at different scales respectively, the model can more flexibly cope with various complex image contents and improve performance on unknown images.

[0098] Specifically, for each scale feature map, the correlation between channels is calculated, and a weight is assigned to each channel to indicate its relative importance relative to other channels; within each scale feature map, the relative importance of each pixel position is evaluated to form a spatial attention map, indicating which areas in the image are the most critical.

[0099] S302: performing element-by-element weighting on each scale feature map and the corresponding weighting coefficient to obtain a weighted scale feature map.

[0100] In this embodiment, the weights generated by the attention mechanism are directly applied to each feature map, ensuring that important features are fully emphasized before fusion, making the subsequent fusion process more efficient.

[0101] Specifically, the obtained channel and spatial attention weights are applied to the corresponding positions or channels of each scale feature map, and weighting is completed through element-by-element multiplication to enhance key features and weaken secondary features, which effectively improves the expression of key information in the feature map and makes subsequent fusion pay more attention to the main content of the image. At the same time, more direct weights guide the model on how to allocate computing resources and optimize the feature extraction process.

[0102] S303: performing weighted summation on the weighted scale feature maps to form a fused feature map.

[0103] In this embodiment, feature maps of different scales are aggregated to integrate multi-level feature information and provide a comprehensive and rich feature basis for the final super-resolution image.

[0104] Specifically, all weighted scale feature maps are summed up, and the feature information under the guidance of attention at different scales is fused into one to form a comprehensive feature representation, which confirms the feature information at different scales, provides comprehensive support for the structure and details of the image, and provides a rich and structured feature basis for generating high-resolution images, which is conducive to restoring more delicate and realistic image details.

[0105] Specifically, in step S301, the step of calculating the channel attention weight includes:

[0106] A global average pooling operation is performed on each scale feature map to compress the features of each channel into a feature vector.

[0107] In this embodiment, global average pooling is performed on the feature map of each scale to achieve dimensionality reduction. Specifically, global average pooling simplifies the two-dimensional feature map of each channel into a scalar value, which represents the overall activation of the channel. In this way, the original scale feature map is converted into a feature vector, in which each element corresponds to the average activation value of a channel.

[0108] The first fully connected layer is used to perform linear transformation on the feature vector, and the ReLU activation function is used to perform nonlinear transformation on the feature vector after linear transformation. The second fully connected layer is then used to perform linear transformation on the feature vector after nonlinear transformation to obtain the weight of each channel.

[0109] In this embodiment, the first fully connected layer is used to perform a linear transformation on the obtained feature vector, the purpose of which is to learn the complex relationship between channels and introduce nonlinearity. Specifically, this is done through the multiplication and addition of the weight matrix and the bias term. The transformed feature vector is then nonlinearly mapped through the ReLU (Rectified Linear Unit) activation function, and negative values ​​are removed to introduce nonlinear features and enhance the model's expressiveness. This helps the model capture more complex feature patterns.

[0110] The obtained weights are applied to each channel of the scale feature map through a channel-by-channel multiplication operation to obtain the channel attention weights of each channel in each scale feature map.

[0111] In this embodiment, the nonlinearly transformed feature vector is fed into the second fully connected layer to further refine the distribution of weights between channels. After this linear transformation, the Sigmoid activation function is used to squeeze the output range to (0, 1), ensuring that the weight value obtained by each channel is a probability value, expressing the importance of the channel relative to other channels.

[0112] Specifically, in step S301, the step of calculating the position attention weight includes:

[0113] The 3x3 convolution is used to fuse the channel information of the scale feature map, and the position attention weight of each position in the scale feature map is calculated by the sigmoid activation function.

[0114] Specifically, step S4 specifically includes the following steps:

[0115] S401: Input the fused feature map into the initial layer of the deep convolutional network, and map the features in the fused feature map from the fusion space to the feature representation space.

[0116] In this embodiment, this step transforms the features in the fused feature map from a lower-level fusion space to a higher-dimensional feature representation space, which can reveal and enhance more high-level features and patterns, providing richer information for subsequent detail recovery.

[0117] Specifically, the fused feature map is used as the input of the deep convolutional network. Through the convolution operation of the initial layer of the deep convolutional network, the pixels in the feature map are converted into a series of feature maps. Each feature map represents a different type of feature pattern in the original image. The combination of these feature maps forms a higher-dimensional feature representation space. The mapped feature representation space contains higher-level features, such as texture, shape, and structural information, which are crucial in super-resolution reconstruction. At the same time, through mapping, more useful information for reconstruction is provided to subsequent layers, enhancing the expressive power of the model.

[0118] S402: gradually extracting deep features in the fused feature map after feature mapping based on continuous convolutional layers and activation functions.

[0119] In this embodiment, in order to further refine and abstract features, continuous convolutional layers and activation functions are used to deeply mine the potential information in the fused feature map and extract more complex structures and details.

[0120] Specifically, each layer of convolution operation extracts more abstract features based on the previous layer, gradually going deeper, and at the same time using the ReLU activation function to introduce nonlinearity, maintain the network's learning ability, enable the model to fit complex data distribution, and make each layer perform a deeper level of abstraction on the features, so that the model can capture more delicate structures of the image. The addition of activation functions and multi-layer convolutions enhance the network's ability to express complex features, and provide more accurate feature support for reconstructing high-resolution images.

[0121] S403: Input the deep features into the feature enhancement layer, perform feature enhancement operation using residual connection, and finally output an enhanced feature map.

[0122] In this embodiment, the residual connection of the feature enhancement layer can enhance important detail information, reduce information loss, and optimize reconstruction quality while maintaining the original features.

[0123] Specifically, the deep feature map is input into the feature enhancement layer, and the difference between the new feature and the original feature is calculated and added back to the original feature. This can directly enhance the useful features and suppress the useless parts while retaining the original information. Finally, after this series of operations, the output feature map contains richer and optimized details, preparing for upsampling and detail recovery.

[0124] The working principle of a real-time dynamic super-resolution image reconstruction method provided by the present invention is as follows: first, a low-resolution image to be reconstructed is obtained, and necessary preprocessing operations are performed to eliminate interference factors such as noise and distortion in the image. Then, a pre-built meta-learning lightweight model is used to extract shallow features at different scales from the preprocessed low-resolution image to obtain a scale feature map. Next, a hybrid attention mechanism is used to weight the shallow features in all scale feature maps. The hybrid attention mechanism can assign different weights to feature maps of different scales according to their importance, thereby highlighting important feature information and suppressing irrelevant information. The weighted scale feature maps are fused into a fused feature map, which contains the image at each scale. The important information at each scale is obtained, which provides a basis for the subsequent feature enhancement processing. In the feature enhancement stage, the present invention uses a deep convolutional network to further process the fused feature map. The deep convolutional network can extract the deep features in the fused feature map through multi-layer convolution and pooling operations, and further enhance the detail information of the image. The enhanced feature map obtained after processing by the deep convolutional network contains richer and more accurate image information. Finally, the enhanced feature map is upsampled by the bicubic interpolation method to expand the size of the low-resolution image to the target size. The upsampled image is restored in detail by the deep refinement network. The deep refinement network can further improve the clarity and detail expression of the image, and finally obtain the target high-resolution image.

[0125] Embodiment 2

[0126] The present invention also provides a real-time dynamic super-resolution image reconstruction system, which is applied to a real-time dynamic super-resolution image reconstruction method, referring to Figure 2 As shown, the reconstruction system includes:

[0127] The preprocessing module is used to obtain a low-resolution image to be reconstructed and preprocess the low-resolution image.

[0128] The scale feature map acquisition module is used to extract shallow features of the preprocessed low-resolution image at different scales using a pre-built meta-learning lightweight model to obtain a scale feature map.

[0129] The fused feature map acquisition module is used to weight the shallow features in all scale feature maps using the hybrid attention mechanism, and fuse all the weighted scale feature maps to obtain a fused feature map.

[0130] The enhanced feature map acquisition module is used to use a deep convolutional network to perform feature enhancement processing on the fused feature map to obtain an enhanced feature map.

[0131] The target high-resolution image acquisition module is used to upsample the enhanced feature map using the bicubic interpolation method, and to restore the details of the upsampled low-resolution image using the deep refinement network to obtain the target high-resolution image.

[0132] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0134] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or includes elements that are inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

Claims

1. A real-time dynamic super-resolution image reconstruction method, characterized in that: The reconstruction method includes the following steps: S1: Acquire a low-resolution image to be reconstructed, and preprocess the low-resolution image; S2: using a pre-built meta-learning lightweight model to extract shallow features of the pre-processed low-resolution image at different scales to obtain a scale feature map; S3: Use the hybrid attention mechanism to weight the shallow features in all scale feature maps, and fuse all weighted scale feature maps to obtain a fused feature map; S4: Performing feature enhancement processing on the fused feature map using a deep convolutional network to obtain an enhanced feature map; S5: Use the bicubic interpolation method to upsample the enhanced feature map, and use the deep refinement network to restore the details of the upsampled low-resolution image to obtain the target high-resolution image; Step S2 specifically includes the following steps: S201: obtaining a meta-learning lightweight model based on a set of low-resolution sample images in the same field as the low-resolution image, wherein each low-resolution sample image in the set of low-resolution sample images is different from the low-resolution image to be reconstructed in terms of texture, lighting conditions and scene content; S202: obtaining low-resolution images at different scales based on convolution kernels of different sizes; S203: extracting shallow features in low-resolution images at different scales using the trained meta-learning lightweight model to obtain scale feature maps corresponding to different scales; In step S201, the steps of constructing a meta-learning lightweight model include: Obtain a low-resolution sample image set in the same field as the low-resolution image, and divide the low-resolution sample image set into a training set and a validation set; Construct a meta-learning model based on a neural network architecture and initialize the meta-parameters of the meta-learning model according to the meta-learning principles ; Define a meta-learning task and iteratively update the parameters of the meta-learning model based on the output of the meta-learning task , and record the performance value of the meta-learning task at the same time, where the meta-learning task is feature extraction from the training set, and the parameters The update formula is: In the formula, are the parameters of the updated meta-learning model, learning rate, is the loss function, is a set of low-resolution sample images, For parameters The gradient of Based on parameters The feature extraction function is for A low-resolution sample image in ; In the parameters After the iteration is completed, the meta-parameters of the meta-learning model are iteratively adjusted based on the evaluation index values ​​of all meta-learning tasks on the validation set. , until the meta parameter Converge to obtain a meta-learning model; The meta-learning model is refined and compressed into a lightweight network through the knowledge distillation method to obtain a meta-learning lightweight model.

2. A real-time dynamic super-resolution image reconstruction method according to claim 1, characterized in that: In step S1, preprocessing of the low-resolution image includes size standardization, denoising and grayscale conversion.

3. The real-time dynamic super-resolution image reconstruction method according to claim 1, characterized in that: Step S3 specifically includes the following steps: S301: Based on the channel attention mechanism and the spatial attention mechanism, the channel attention weight and the position attention weight of each scale feature map are calculated respectively, and used as the weighting coefficient of each element in the scale feature map; S302: performing element-by-element weighting on each scale feature map and the corresponding weighting coefficient to obtain a weighted scale feature map; S303: performing weighted summation on the weighted scale feature maps to form a fused feature map.

4. A real-time dynamic super-resolution image reconstruction method according to claim 3, characterized in that: In step S301, the step of calculating the channel attention weight includes: Perform a global average pooling operation on each scale feature map to compress the features of each channel into a feature vector; The first fully connected layer is used to perform a linear transformation on the feature vector, and the ReLU activation function is used to perform a nonlinear transformation on the feature vector after the linear transformation. The second fully connected layer is then used to perform a linear transformation on the feature vector after the nonlinear transformation to obtain the weight of each channel. The obtained weights are applied to each channel of the scale feature map through a channel-by-channel multiplication operation to obtain the channel attention weights of each channel in each scale feature map.

5. The real-time dynamic super-resolution image reconstruction method according to claim 3, characterized in that: In step S301, the step of calculating the position attention weight includes: The 3x3 convolution is used to fuse the channel information of the scale feature map, and the position attention weight of each position in the scale feature map is calculated by the sigmoid activation function.

6. The real-time dynamic super-resolution image reconstruction method according to claim 1, characterized in that: Step S4 specifically includes the following steps: S401: Input the fused feature map to the initial layer of the deep convolutional network, and map the features in the fused feature map from the fusion space to the feature representation space; S402: gradually extracting deep features in the fused feature map after feature mapping based on continuous convolutional layers and activation functions; S403: Input the deep features into the feature enhancement layer, perform feature enhancement operation using residual connection, and finally output an enhanced feature map.

7. A real-time dynamic super-resolution image reconstruction system, applied to a real-time dynamic super-resolution image reconstruction method according to any one of claims 1 to 6, characterized in that: Rebuilding the system includes: A preprocessing module, used for acquiring a low-resolution image to be reconstructed and preprocessing the low-resolution image; A scale feature map acquisition module is used to extract shallow features of the preprocessed low-resolution image at different scales using a pre-built meta-learning lightweight model to obtain a scale feature map; The fusion feature map acquisition module is used to weight the shallow features in all scale feature maps using the hybrid attention mechanism, and fuse all the weighted scale feature maps to obtain a fusion feature map; An enhanced feature map acquisition module is used to perform feature enhancement processing on the fused feature map using a deep convolutional network to obtain an enhanced feature map; The target high-resolution image acquisition module is used to upsample the enhanced feature map using the bicubic interpolation method, and to restore the details of the upsampled low-resolution image using the deep refinement network to obtain the target high-resolution image.

Images