Super-resolution image processing method and device based on artificial intelligence

Abstract

The invention relates to the technical field of artificial intelligence and image processing, in particular to a super-resolution image processing method and device based on artificial intelligence, and the method comprises the steps: obtaining a first target image meeting a first preset resolution condition, segmenting the first target image, and obtaining a second target image meeting a preset pixel condition; aligning pixels of the second target image by using the extracted depth features of the second target image to obtain a third target image; and performing feature fusion processing on the third target image by using a preset target neural network, and obtaining an RGB image meeting a second preset resolution condition from the first target image according to the fused image. Therefore, the problem of limitation of an image super-resolution processing mode in related technologies is solved, the applicability of the image super-resolution method is improved, the complexity of image super-resolution operation is reduced, and the efficiency of image super-resolution processing is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and image processing technology, and in particular to a super-resolution image processing method and device based on artificial intelligence. Background Art

[0002] With the development of computer vision technology, high-resolution image processing has become increasingly important in various fields. Traditional image super-resolution technology usually relies on image interpolation and enhancement algorithms. However, these methods often cause image blur and increased noise when processing image details. In recent years, image super-resolution technology based on deep learning has received widespread attention. Through the learning and optimization of neural networks, image details can be restored more effectively.

[0003] Deep learning can effectively use the interconnected information between pictures to improve clarity, or enhance the quality of photos such as denoising. Among them, since the input of the multi-frame synthesis algorithm is multiple pictures, it is very important to align multiple pictures, and artificial intelligence can detect pixel-level deviations in the pictures and align them. After aligning each picture, artificial intelligence can add different details obtained from multiple photos to one photo, thereby achieving the function of increasing resolution. In addition, artificial intelligence will also use the information learned in previous training to predict some details, and can learn through the database, so that this type of algorithm can better deal with some special input pictures, such as different light, color or virtual shadows.

[0004] However, in the related technologies, since the resolution and quality of the image will be affected by the shooting equipment and noise, the image super-resolution processing method based on deep learning has limitations, which reduces the applicability of image super-resolution processing. In particular, the calculation process of deep learning is relatively complicated, which reduces the efficiency of image super-resolution processing and needs to be solved urgently. Summary of the invention

[0005] The present application provides a super-resolution image processing method and device based on artificial intelligence to solve the limitations of image super-resolution processing methods in related technologies, improve the applicability of image super-resolution methods, reduce the complexity of image super-resolution operations, and improve the efficiency of image super-resolution processing.

[0006] The first aspect of the present application provides an artificial intelligence-based super-resolution image processing method, comprising the following steps: acquiring a first target image that meets a first preset resolution condition, and segmenting the first target image to obtain a second target image that meets a preset pixel condition; extracting depth features of the second target image, and using the extracted depth features to align pixels of the second target image to obtain a third target image; using a pre-set target neural network to perform feature fusion processing on the third target image to obtain a fused image, and obtaining an RGB image that meets the second preset resolution condition of the first target image based on the fused image.

[0007] Optionally, in one embodiment of the present application, the acquiring of a first target image that satisfies a first preset resolution condition, and segmenting the first target image to obtain a second target image that satisfies a preset pixel condition, includes: acquiring a continuous multi-frame image of the target scene that satisfies the first preset resolution condition; determining a target segmentation method for the continuous multi-frame images, and segmenting the continuous multi-frame images using the target segmentation method to obtain the second target image that satisfies the preset pixel condition.

[0008] Optionally, in one embodiment of the present application, the extracting of depth features of the second target image and using the extracted depth features to align pixels of the second target image to obtain a third target image includes: encoding each frame image in the second target image to extract the depth features of the second target image; using the first frame image in the continuous multiple frame images as a base frame image, and using the depth features to predict offsets of other frame images and harmonic scalars of the other frame images relative to the base frame image; and aligning the current frame image with the base frame image according to the offset and the harmonic scalar to obtain the third target image.

[0009] Optionally, in one embodiment of the present application, the use of a preset target neural network to perform feature fusion processing on the third target image to obtain a fused image includes: predicting a fusion weight for each pixel in the third target image based on the target content and noise level of the third target image and the preset neural network, and normalizing the fusion weight to obtain a normalized weight; using the normalized weight to perform weighted summation on the deep features of the second target image to obtain a fused feature map, so as to obtain the fused image based on the fused feature map.

[0010] Optionally, in one embodiment of the present application, obtaining an RGB image of the first target image that meets a second preset resolution condition based on the fused image includes: determining an upsampling rate of the continuous multiple frame images based on the fused image; upsampling the continuous multiple frame images using the upsampling rate to obtain an upsampled image; and reconstructing and decoding the upsampled images, respectively, to obtain the RGB image of the first target image that meets the second preset resolution condition.

[0011] The second aspect of the present application provides an artificial intelligence-based super-resolution image processing device, including: an acquisition module, used to acquire a first target image that meets a first preset resolution condition, and segment the first target image to obtain a second target image that meets a preset pixel condition; a determination module, used to extract depth features of the second target image, and use the extracted depth features to align pixels of the second target image to obtain a third target image; a processing module, used to perform feature fusion processing on the third target image using a preset target neural network to obtain a fused image, and obtain an RGB image that meets the second preset resolution condition of the first target image based on the fused image.

[0012] Optionally, in one embodiment of the present application, the acquisition module includes: an acquisition unit, used to acquire the continuous multi-frame images of the target scene that meet the first preset resolution condition; a first determination unit, used to determine the target segmentation method of the continuous multi-frame images, so as to segment the continuous multi-frame images using the target segmentation method to obtain the second target image that meets the preset pixel condition.

[0013] Optionally, in one embodiment of the present application, the determination module includes: an extraction unit, used to encode each frame image in the second target image to extract the depth features of the second target image; a prediction unit, used to take the first frame image in the continuous multiple frame images as the base frame image, and use the depth features to predict the offsets of other frame images and the harmonic scalars of the other frame images relative to the base frame image; a second determination unit, used to align the current frame image with the base frame image according to the offset and the harmonic scalar to obtain the third target image.

[0014] Optionally, in one embodiment of the present application, the processing module includes: a first processing unit, used to predict the fusion weight of each pixel in the third target image based on the target content and noise level of the third target image and the pre-set neural network, and normalize the fusion weight to obtain a normalized weight; a third determination unit, used to use the normalized weight to perform weighted summation on the depth features of the second target image to obtain a fused feature map, so as to obtain the fused processed image according to the fused feature map.

[0015] Optionally, in one embodiment of the present application, the processing module includes: a fourth determination unit, used to determine the upsampling rate of the continuous multiple frame images based on the image after the fusion processing; a fifth determination unit, used to upsample the continuous multiple frame images using the upsampling rate to obtain the upsampled image; and a second processing unit, used to reconstruct and decode the upsampled images respectively to obtain the RGB image of the first target image that meets the second preset resolution condition.

[0016] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the artificial intelligence-based super-resolution image processing method as described in the above embodiment.

[0017] The fourth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned artificial intelligence-based super-resolution image processing method.

[0018] The embodiment of the present application can obtain a second target image that meets a certain pixel condition by segmenting a first target image that meets a first resolution condition, and align the pixels of the second target image using the extracted depth features of the second target image to obtain a third target image, and perform feature fusion processing on the third target image using a pre-set target neural network to obtain an RGB image that meets the second resolution condition of the first target image. As a result, the limitation problem of the image super-resolution processing method in the related art is solved, the applicability of the image super-resolution method is improved, the complexity of the image super-resolution operation is reduced, and the efficiency of the image super-resolution processing is improved.

[0019] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a super-resolution image processing method based on artificial intelligence provided according to an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of the principle of a sliding window (Swin) Transformer group of a specific embodiment of the present application;

[0023] Figure 3 A schematic diagram of a deformable convolutional network according to a specific embodiment of the present application;

[0024] Figure 4 A schematic diagram of the principle of a stereoscopic cross attention module according to a specific embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an artificial intelligence-based super-resolution image processing device provided according to an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0028] The following describes the artificial intelligence-based super-resolution image processing method and device of the embodiment of the present application with reference to the accompanying drawings. In view of the limitation of the image super-resolution processing method in the related art mentioned in the background technology center, which reduces the applicability of image super-resolution processing, especially the problem that the operation process is relatively complicated, which reduces the efficiency of image super-resolution processing, the present application provides a super-resolution image processing method based on artificial intelligence, in which a first target image that meets a first resolution condition can be segmented to obtain a second target image that meets a certain pixel condition, and the pixels of the second target image are aligned using the extracted depth features of the second target image to obtain a third target image, and the third target image is subjected to feature fusion processing using a pre-set target neural network to obtain an RGB image that meets the second resolution condition of the first target image. As a result, the limitation of the image super-resolution processing method in the related art is solved, the applicability of the image super-resolution method is improved, the complexity of the image super-resolution operation is reduced, and the efficiency of the image super-resolution processing is improved.

[0029] Specifically, Figure 1 A flowchart of an artificial intelligence-based super-resolution image processing method provided in an embodiment of the present application.

[0030] like Figure 1 As shown, the super-resolution image processing method based on artificial intelligence includes the following steps:

[0031] In step S101, a first target image satisfying a first preset resolution condition is acquired, and the first target image is segmented to obtain a second target image satisfying a preset pixel condition.

[0032] In the embodiment of the present application, the first preset resolution condition is the condition of continuously shot low-resolution images; the preset pixel condition is the condition for dividing a large-pixel image into small-pixel images; wherein, the specific low resolution and pixel requirements can be set by technical personnel in this field and are not specifically limited here; the first target image is multiple frames of low-resolution images shot continuously; the second target image is each frame of the image obtained after segmentation.

[0033] It can be understood that the embodiment of the present application can obtain a first target image that meets the first resolution condition. For example, the embodiment of the present application can use a binocular camera to capture continuous images of a scene, and based on the image characteristics and processing requirements, the large picture obtained in each frame is divided into small images to obtain a second target image that meets the pixel condition, thereby effectively improving the executability of image processing.

[0034] Among them, in one embodiment of the present application, a first target image that meets a first preset resolution condition is obtained, and the first target image is segmented to obtain a second target image that meets a preset pixel condition, including: obtaining a continuous multi-frame image of the target scene that meets the first preset resolution condition; determining a target segmentation method for the continuous multi-frame images, and segmenting the continuous multi-frame images using the target segmentation method to obtain a second target image that meets the preset pixel condition.

[0035] During the actual implementation process, the embodiment of the present application can fix the binocular camera in a stable position to ensure that the relative position between the two camera modules does not change, so as to ensure the consistency of the captured image; then adjust the parameters such as the focal length, aperture and exposure time of the binocular camera to adapt to the lighting conditions of the current shooting scene to ensure the clarity and quality of the captured image.

[0036] Furthermore, the embodiments of the present application can set the software or hardware parameters of the camera so that it can capture multiple frames of images continuously and quickly. The time interval between each frame of the image should be short enough to reduce the image differences caused by camera jitter and scene changes. During the shooting process, the binocular camera simultaneously captures multiple frames of images of the left view and the right view to form a set of image sequences with temporal continuity; then, the embodiments of the present application can select a suitable image segmentation method and size based on the image characteristics and processing requirements, divide the large picture obtained in each frame into small pictures, and select a suitable neural network parameter scale based on the size of the picture after slicing and the processing requirements, thereby effectively improving the applicability of image super-resolution processing.

[0037] Among them, the embodiment of the present application can analyze each frame of the large picture taken to determine its resolution, size, content complexity and other characteristics; and determine the appropriate segmentation method according to the characteristics and processing requirements of the large picture, such as the number of segments, the size of each segment, etc.

[0038] It should be noted that the image segmentation modes include uniform segmentation and adaptive segmentation. Uniform segmentation can evenly divide a large image into several small images of equal size, such as dividing a large image of 1024*1024 pixels into 16 small images of 256*256 pixels; adaptive segmentation can perform adaptive segmentation according to the complexity and detail of the image content. For areas with more details, it can be divided into smaller blocks, and for areas with fewer details, it can be divided into larger blocks. In addition, the boundaries of the divided small images should not cause obvious damage to the image content. The small images obtained by segmentation are then numbered and stored for subsequent processing.

[0039] For example, the embodiments of the present application may use the image peak signal-to-noise ratio or the image structure similarity index as an indicator for judging image processing. Among them, PSNR (Peak Signal-to-Noise Ratio) is an indicator commonly used to evaluate image quality. PSNR quantifies image quality by comparing the difference between the original signal (reference image) and the processed signal (target image). The calculation of PSNR is based on the MSE (Mean Squared Error) of the signal peak and noise, usually expressed in logarithmic form, in decibels (dB). The higher the PSNR value, the smaller the difference between the two images, thereby indicating that the image quality is better. The calculation formula of PSNR is as follows:

[0040]

[0041] Among them, R represents the maximum possible value of the pixel value, usually 255 for RGB images; M and N represent the length and width of the image respectively, X(i, j) and Y(i, j) represent the pixel values ​​of the reference image and the target image at the same position respectively, MSE represents the mean square error, and PSNR represents the peak signal-to-noise ratio.

[0042] Furthermore, SSIM (Structural Similarity Index) is also an indicator commonly used to evaluate image quality. Compared with the peak signal-to-noise ratio, SSIM pays more attention to the subjective perception of the human eye on the image and can more accurately evaluate the structural similarity and quality of the image. SSIM quantifies image quality by comparing the differences in structure, brightness, and contrast between the reference image and the target image. The calculation of SSIM is based on three key factors: brightness similarity, contrast similarity, and structure similarity. Its range is between 0 and 1. The closer the value is to 1, the better the image quality. The calculation formula of SSIM is as follows:

[0043] SSIM=l(x,y) α *c(x,y) β *s(x,y) γ

[0044]

[0045] Among them, SSIM represents the structural similarity index, x and y represent the reference image and the target image respectively; l(x, y), c(x, y), s(x, y) represent the similarity of brightness, contrast and structure respectively; α, β, γ represent adjustable parameters; μ and σ represent the mean and standard deviation respectively; σ xyrepresents the covariance of the two, c1, c2, c3 are all constants used to avoid the situation where the denominator is 0. In general, c3 = c2 / 2, α = β = γ = 1 can be taken, then we can get:

[0046]

[0047] It should be noted that those skilled in the art can select appropriate image processing evaluation indicators according to actual conditions, and no specific limitation is made here.

[0048] Secondly, the embodiment of the present application can select a neural network model for the image processing task, and determine the image size input to the neural network according to the size of the small image after segmentation, thereby determining the number of layers of the neural network and the parameter scale of each layer. For example, commonly used models include convolutional neural networks, recurrent neural networks, residual networks, fully connected networks, attention mechanism networks, and Transformer networks. Different neural network models have different parameter scales and computational complexities, and should be selected according to specific processing requirements.

[0049] Among them, the input image size of the neural network should be consistent with the size of the training data to obtain the best processing effect. If the size of the input image is inconsistent with the training data, it can be adjusted through preprocessing operations such as image scaling.

[0050] Again, the embodiment of the present application can determine the number of layers of the neural network and the parameter scale of each layer, that is, according to the complexity of image processing and the limitation of hardware resources, select the appropriate number of layers to achieve a compromise between model expression ability and computational complexity; the parameter scale of each layer includes the size of the convolution kernel, the number of convolution kernels, the number of nodes in the fully connected layer, etc. The parameter scale should be set according to the size of the input image and the processing requirements. Technical personnel in this field can make a choice based on actual conditions, and no specific restrictions are made here.

[0051] In step S102, the depth features of the second target image are extracted, and the pixels of the second target image are aligned using the extracted depth features to obtain a third target image.

[0052] In the embodiment of the present application, the third target image is an image that achieves pixel-level alignment.

[0053] It can be understood that the embodiments of the present application can extract the depth features of the second target image. For example, the embodiments of the present application can be based on the second target image, that is, each frame of the image obtained by segmentation can be encoded through a neural network to extract depth features, and the pixels of multiple frames of images can be aligned using the extracted depth features to obtain the third target image, so that the quality of the captured image is improved, the computational complexity of the artificial intelligence algorithm is reduced, while retaining more image detail information, thereby increasing the overall visual experience of the image.

[0054] Among them, in one embodiment of the present application, the depth features of the second target image are extracted, and the pixels of the second target image are aligned using the extracted depth features to obtain a third target image, including: encoding each frame image in the second target image to extract the depth features of the second target image; taking the first frame image in a plurality of consecutive frame images as a base frame image, and using the depth features to predict the offsets of other frame images and the harmonic scalars of the other frame images relative to the base frame image; and aligning the current frame image with the base frame image according to the offset and the harmonic scalar to obtain the third target image.

[0055] As a possible implementation method, the embodiment of the present application can independently map each input image into a deep feature based on each frame image in the second target image through the encoding module of the neural network, take the first frame image in each continuous multi-frame image as the basic frame image, and use the alignment module of the neural network to predict the offset of other frame images and the harmonic scalar for the basic frame image, so as to align the current frame image with the basic frame image based on the current frame, offset and modulation scalar.

[0056] For example, an embodiment of the present application can use a convolutional neural network to independently map each input image into a deep feature, wherein the hierarchical structure of the neural network is designed to include convolutional layers, pooling layers, activation layers, and fully connected layers, etc. Multiple convolutional layers can be used to extract features at different levels, pooling layers can be used to reduce the spatial dimension of features, activation functions (such as ReLU, Sigmoid, etc.) can be used to increase nonlinear mapping capabilities, and fully connected layers can be used to integrate and classify features.

[0057] In the actual execution process, the embodiment of the present application can input a small area of ​​each frame of the image into the encoding module of the neural network, and extract the deep features of the image through convolution operations. The convolution kernel of each layer can select an appropriate size and number according to the different image features, such as 3*3, 5*5, etc. During the convolution operation, a pooling layer (such as maximum pooling and average pooling) is applied to reduce the size of the feature map and maintain the spatial relationship of the features. The step size and size of the pooling layer should be adjusted according to the specific application scenario.

[0058] In addition, during the feature extraction process, the embodiments of the present application can use batch normalization and Dropout technology to enhance the training stability and generalization ability of the neural network model. Among them, batch normalization can accelerate model training and reduce the gradient vanishing problem; Dropout reduces the overfitting phenomenon of the model by randomly discarding some neurons.

[0059] For another example, in the process of aligning the current frame image with the base frame image in the embodiment of the present application, a convolutional neural network can be used to learn the transformation relationship between different frames to achieve accurate alignment at the pixel level. In addition, an attention mechanism or a convolutional attention network can be used to enhance the performance of the alignment module and improve the adaptability to different scenes.

[0060] Specifically, the embodiment of the present application can take a small area of ​​each frame image as input, process it through the neural network of the alignment module, and predict the displacement and transformation parameters of each frame relative to the basic frame image. The output of the alignment module is an offset vector and a transformation matrix, which represent the correspondence between the current frame image and the basic frame image. The predicted offset vector and transformation matrix are used to transform the pixels of each frame image to ensure that the spatial correspondence of the pixels of all frames is consistent. Among them, specific transformation methods include affine transformation, perspective transformation, etc., and the specific selection depends on the characteristics and requirements of the scene.

[0061] In an embodiment of the present application, in order to improve the alignment quality, the embodiment of the present application can use an iterative optimization algorithm, such as a gradient descent method, Newton's method, etc., to optimize the alignment process and reduce errors and drifts in pixel alignment. Furthermore, a loss function, such as mean square error, structural similarity, etc., can be introduced to evaluate and optimize the alignment results to ensure that the aligned image pixels have high similarity and consistency.

[0062] In step S103, a preset target neural network is used to perform feature fusion processing on the third target image to obtain a fused image, and an RGB image that satisfies a second preset resolution condition of the first target image is obtained based on the fused image.

[0063] In the embodiment of the present application, the second preset resolution condition is a condition of a high-resolution image obtained by super-resolution image processing. The specific high resolution can be set by those skilled in the art and is not specifically limited here.

[0064] It can be understood that the embodiment of the present application can utilize a pre-set target neural network to perform feature fusion processing on the third target image, that is, use a pre-selected neural network to perform feature fusion on the aligned multi-frame images to obtain a fused image, perform cross-view interaction on the binocular image, and adaptively upsample based on the fused image, and use a decoding neural network to obtain a high-resolution RGB image, which effectively improves the efficiency of image super-resolution processing and improves the applicability and robustness of image super-resolution processing.

[0065] Optionally, in one embodiment of the present application, a pre-set target neural network is used to perform feature fusion processing on the third target image to obtain a fused image, including: based on the target content and noise level of the third target image and a pre-set neural network, predicting the fusion weight of each pixel in the third target image, and normalizing the fusion weight to obtain a normalized weight; using the normalized weight to perform weighted summation on the deep features of the second target image to obtain a fused feature map, so as to obtain a fused image based on the fused feature map.

[0066] In some embodiments, the embodiments of the present application can use a fusion module of a neural network to predict pixel-level fusion weights based on the image content and noise level of a third target image, and normalize the fusion weights. Based on the normalized fusion weights, weighted summation is performed on the depth features of multiple frames of images to obtain a fused feature map. Then, based on the depth features of the fused images of each binocular camera, an interaction module of a neural network is used to perform cross-view interaction on the binocular images to obtain stereo information and three-dimensional features of the images.

[0067] In the embodiment of the present application, the embodiment of the present application can perform content analysis on the captured multi-frame images, extract the edge, texture, shape and other features of the image, and can use the Sobel operator, Canny edge detection and other methods to obtain the edge features of the image, and use noise estimation methods, such as noise estimation based on wavelet transform, noise estimation based on variance, etc., to evaluate the noise level of the image, and provide a basis for subsequent fusion weight prediction. In addition, the fusion module of the neural network can be implemented using a convolutional neural network, using a pooling layer to reduce the feature dimension, and then using a fully connected layer to predict the weight.

[0068] In the actual implementation process, the embodiment of the present application can input a small area of ​​​​a plurality of frame images into a neural network fusion module, extract features through a convolutional layer, and use a fully connected layer to predict the fusion weight of each pixel. In the weight prediction process, the fusion module can dynamically adjust the weight distribution according to the image content and noise level of the third target image to ensure that important features and low-noise areas have higher weights. The predicted fusion weights need to be normalized to ensure that the sum of the weights of each pixel is 1. The normalization method includes Softmax normalization, L1 normalization, etc., which can be specifically selected by technicians in this field according to actual conditions and are not specifically limited here.

[0069] Secondly, the embodiment of the present application can perform weighted processing on the depth features of each frame image at the pixel level. The weight is determined by the weight value predicted and normalized by the fusion module. The depth features of all frame images are weightedly summed pixel by pixel to obtain a fused feature map.

[0070] Specifically, the fused feature map retains the main information of multiple frame images and effectively suppresses the influence of noise. The fused feature map should have high resolution and detail retention to facilitate subsequent image processing and analysis.

[0071] Again, the embodiment of the present application can use a dual-stream network to implement a neural network for cross-view interaction. During the interaction process, the interaction module dynamically adjusts the weight of feature fusion according to the feature differences between the left and right views to ensure the accurate extraction of stereo information and three-dimensional features. Among them, the embodiment of the present application can use the output of the interaction module to obtain the stereo information and three-dimensional features of the image, wherein the stereo information includes a depth map, a disparity map, etc.; the three-dimensional features include the spatial structure of the image, the shape of the object, etc. In the process of obtaining stereo information and three-dimensional features, a comprehensive analysis is performed in combination with the depth features of the left and right views to ensure the accuracy of the stereo information and the integrity of the three-dimensional features.

[0072] Optionally, in one embodiment of the present application, an RGB image of the first target image that meets a second preset resolution condition is obtained based on the fused image, including: determining an upsampling rate of multiple consecutive frames of images based on the fused image; upsampling the multiple consecutive frames of images using the upsampling rate to obtain an upsampled image; and reconstructing and decoding the upsampled images, respectively, to obtain an RGB image of the first target image that meets the second preset resolution condition.

[0073] In some embodiments, the embodiments of the present application can select an appropriate upsampling rate according to the ratio of the resolution of the fused binocular image features to the target image resolution, such as bilinear interpolation, bicubic interpolation, convolution upsampling, sub-pixel convolution, etc., based on the upsampling rate and the continuous multi-frame image features obtained in the fusion module, and adaptively and gradually merge multiple low-resolution features. Then, based on the merged image features, the high-resolution image is reconstructed and decoded using the decoding module of the neural network to obtain a high-resolution RGB image.

[0074] During actual implementation, the embodiments of the present application may determine the resolution of the target image based on actual application requirements. For example, in applications such as high-definition display and detail reconstruction, the target image resolution may need to reach 4K or higher.

[0075] Optionally, the embodiment of the present application can design a decoding module for high-resolution image reconstruction. The decoding module usually uses a deep learning model such as a convolutional neural network or a generative adversarial network, inputs the merged image features into the decoding module, and performs high-resolution reconstruction through a deconvolution layer. The function of the deconvolution layer is to restore low-resolution features to a high-resolution space through an upsampling operation. During the reconstruction process, activation functions such as ReLU, Leaky ReLU, etc. can be used to increase nonlinear mapping capabilities and improve image detail performance.

[0076] Furthermore, based on the high-resolution reconstruction, a decoding module is used for further processing to generate a final high-resolution RGB image. The decoding process includes steps such as color adjustment and detail enhancement to ensure color restoration and detail retention of the reconstructed image. Among them, the embodiment of the present application can use a generative adversarial network based on adversarial learning for image decoding, which can effectively improve the naturalness and realism of the image. The generative adversarial network generates high-quality images through adversarial training of the generator and the discriminator.

[0077] For example, combined with Figure 2 and Figure 3 As shown, in the embodiment of the present application, when a monocular camera is used, a sliding window Transformer (Swin Transformer) group is used for feature extraction, and a deformable convolutional network is used for multi-frame image alignment, the specific method is as follows:

[0078] (1) Use a monocular camera to take multiple frames of a scene. Based on the characteristics of the image and processing requirements, the large image obtained from each frame is divided into small images. Based on the size of the sliced ​​images and processing requirements, the size of the neural network parameters is determined.

[0079] First, the monocular camera can be fixed in a stable position to ensure that the camera position does not change to ensure the consistency of the captured image; then adjust the camera parameters such as focal length, aperture and exposure time to adapt to the light conditions of the current shooting scene to ensure the clarity and quality of the captured image; in addition, set the software or hardware parameters of the camera so that it can capture multiple frames of images continuously and quickly, and the time interval between each frame should be short enough to reduce image differences caused by camera jitter and scene changes.

[0080] Among them, the embodiments of the present application can use the image peak signal-to-noise ratio or the image structure similarity index as an indicator for judging image processing; in addition, the embodiments of the present application can determine the image size input to the neural network according to the size of the small picture after segmentation; determine the number of layers of the neural network and the parameter scale of each layer.

[0081] In the actual implementation process, the input image size of the neural network should be consistent with the size of the training data to obtain the best processing effect. If the size of the input image is inconsistent with the training data, it can be adjusted through preprocessing operations such as image scaling.

[0082] (2) Based on each frame of the image obtained by segmentation, the deep features are extracted through the Swin Transformer group, and the pixels of multiple frames are aligned using a deformable convolutional network.

[0083] Specifically, the embodiment of the present application can flatten the segmented low-resolution image input into a single channel, and then use a 3x3 convolution layer to convert it into three channels, so that the data can be input into the SpyNet network and three levels of optical flow are calculated for each frame relative to the reference frame. These flows help understand how different parts of the image move or change in the sequence. At the same time, the original four-channel raw image will be sent to several Swin Transformer modules to obtain features.

[0084] In the actual implementation process, the Swin Transformer module divides each frame of the segmented image into several fixed-size image patches, such as 16*16 pixel image patches. Each image patch is used as an input unit for feature extraction, and then the image patch is embedded into a high-dimensional feature space through linear projection to form a feature vector of the image patch.

[0085] Furthermore, if Figure 2 As shown in the figure, the feature vector is input into the Transformer network, and the deep features of each image block are extracted through a multi-layer self-attention mechanism and a feedforward neural network. The position information of the image block is introduced into the feature vector using position encoding, ensuring that the Transformer network can capture the spatial relationship of the image blocks, thereby aggregating relevant high-frequency information.

[0086] Among them, the pyramidal optical flow predicted by SpyNet can be regarded as a rough alignment prior, based on which the deformable convolutional network can learn more precise and detailed alignment feature offsets.

[0087] like Figure 3 As shown, given feature F i , and the corresponding optical flow f i , we can get the rough deformation characteristics It is then concatenated with the reference feature to predict a fine local offset, and then the fine offset is added to the optical flow to obtain a more precise offset. Based on these offsets, the input feature is deformed through the original deformable convolutional network alignment module to obtain the aligned feature Af iIn addition, the network also includes a three-level pyramid structure to further improve the alignment. The predicted offset and aligned features are upsampled and concatenated with the offset and alignment features of the next level. This allows the output features to be refined using multi-scale information and improves the ability to reduce noise. In addition, a feature enhancement network is added before the deformable convolutional network to mitigate the negative effects of noise.

[0088] (3) Use a pre-selected neural network to fuse the features of multiple aligned image frames.

[0089] In an embodiment of the present application, a loop fusion module can be used to fuse the aligned features. In order to make full use of the base frame, a loop module corresponding to the key frame is designed for the loop fusion module. Its function is to use the base frame as a guide for the spatial position. Since the non-base frame modules processed by the alignment module generally cannot be perfectly aligned with the base frame, it is necessary for the fusion module to consider the base frame as a guide to reduce the offset and artifacts in the merged feature position. The loop fusion scheme will propagate information from the base frame to other frames to achieve the purpose of guidance. This process can also be regarded as the gradual recovery of the base frame.

[0090] Specifically, the loop fusion module uses a loop network combined with a residual learning strategy, which can accept two inputs: one is the current aligned feature, and the other is the previous and merged feature. Each new feature will enter the loop fusion module and merge with the previous feature. During the merging process, the features of the base frame, the current aligned features, and the previously merged features will be connected along the channel dimension to avoid the loss of key information in the base frame and make full use of the base frame. When all features are merged, the loop fusion module outputs the fused features for the next step of processing.

[0091] It should be noted that, since a monocular camera is used for shooting in this embodiment, cross-view interaction is not involved.

[0092] (4) Adaptively upsample the fused image features and use a decoding neural network to obtain a high-resolution RGB image.

[0093] In the actual implementation process, the embodiment of the present application can decode the residual network. The decoding module first maps the input H×W to 128 channels and passes it through a residual network. Then it is upsampled by sub-pixel convolution to change its dimension to sH×sW. After another convolution layer, the feature dimension is increased to 2 2 s 2 D′, the shape is H / 2×W / 2×2 2The feature vector at each position is then rearranged into a 2s×2s×D′ tensor to obtain a higher resolution feature map of shape H×W×D′, where H and W are the feature map height and width, s is the upsampling rate, and D′ is the sub-pixel convolutional layer output feature size.

[0094] In an embodiment of the present application, the sub-pixel convolution layer can be initialized using the ICNR initialization method to optimize its output, and Gaussian smoothing can be performed on the output feature map to improve the quality of the image. Subsequently, these smoothed feature maps are upsampled and further processed through a series of residual blocks, and finally, these feature maps are passed through a convolution layer to obtain a high-resolution RGB image.

[0095] For example, Figure 4 As shown, the embodiment of the present application can use the SCAM (Stereo Cross Attention Mechanism) module to perform cross-view fusion, and the specific method is as follows:

[0096] (1) Use a binocular camera to take continuous multiple-frame photos of a scene. Based on the characteristics of the image and processing requirements, the large image obtained in each frame is divided into small images to determine the size of the neural network parameters.

[0097] During the actual implementation process, the binocular camera is fixed in a stable position to ensure that the relative position between the two camera modules does not change to ensure the consistency of the captured image; then the binocular camera parameters such as focal length, aperture and exposure time are adjusted to adapt to the lighting conditions of the current shooting scene to ensure the clarity and quality of the captured image.

[0098] Furthermore, the software or hardware parameters of the camera are set so that it can capture multiple frames of images continuously and quickly. The time interval between each frame of the image should be short enough to reduce the image difference caused by camera shaking and scene changes. During the shooting process, the binocular camera simultaneously captures multiple frames of images of the left view and the right view to form a set of image sequences with temporal continuity.

[0099] Among them, the embodiment of the present application can select the appropriate neural network parameter scale based on the size of the image after slicing and the processing requirements. That is to say, in the embodiment of the present application, each frame of the large image taken should be analyzed to determine its resolution, size, content complexity and other characteristics; and according to the characteristics of the image and the processing requirements, determine the appropriate segmentation method, such as the number of segments, the size of each segment, etc. In addition, the boundaries of the segmented small images should not cause obvious damage to the image content, and then the segmented small images are numbered and stored for subsequent processing.

[0100] Secondly, the present application embodiment may use the image peak signal-to-noise ratio or the image structure similarity index as an index for judging image processing. It should be noted that those skilled in the art may select appropriate image processing evaluation indexes according to actual conditions, and no specific limitation is made here.

[0101] In one embodiment of the present application, the embodiment of the present application can determine the number of layers of the neural network and the parameter scale of each layer, that is, according to the complexity of image processing and the limitation of hardware resources, select the appropriate number of layers to achieve a compromise between the model expression ability and the computational complexity; the parameter scale of each layer includes the size of the convolution kernel, the number of convolution kernels, the number of nodes in the fully connected layer, etc. The parameter scale should be set according to the size of the input image and the processing requirements, and those skilled in the art can make a selection according to the actual situation, and no specific limitation is made here.

[0102] (2) Based on each frame of the image obtained by segmentation, it is encoded through a neural network to extract deep features and align the pixels of multiple frames.

[0103] Specifically, the embodiment of the present application can independently map each input image into a deep feature through the encoding module of the neural network based on each frame of the image obtained by segmentation; take the first frame in each continuous shot as the base frame, and use the alignment module of the neural network to predict the offset of other frames and the harmonic scalar for the base frame; align the current frame with the base frame based on the current frame, the offset and the modulation scalar.

[0104] In an embodiment of the present application, two networks stacked by NAF (Nonlinear Activation Free) blocks with shared weights can be used to extract the inner view features of the left image and the right image respectively. First, the network uses a 3 by 3 convolution layer to map the input image space to a higher dimensional feature space, and then uses N NAF blocks to perform deep inner view feature extraction. Since these steps are independent for the two input photos, this step is the inner view feature extraction. After the extraction is completed, a 3 by 3 convolution and a pixel reorganization layer upsample the features with a scaling factor s. In addition, in order to reduce the burden of feature learning, the network uses global residual learning to predict the residual between the bilinearly upsampled low-resolution image and the true high-resolution image.

[0105] Secondly, the embodiment of the present application can take a small area of ​​each frame image as input, process it through the neural network of the alignment module, and predict the displacement and transformation parameters of each frame relative to the base frame. The output of the alignment module is an offset vector and a transformation matrix, which represents the correspondence between the current frame and the base frame. Using the predicted offset vector and transformation matrix, the pixels of each frame image are transformed to ensure that the spatial correspondence of the pixels of all frames is consistent. Specific transformation methods include affine transformation, perspective transformation, etc., and the specific selection depends on the characteristics and requirements of the scene.

[0106] (3) Use the stereo cross-attention module (SCAM) to fuse the features of multiple aligned frames and perform cross-view interaction for binocular images.

[0107] Specifically, in order to interact with the cross-view information, each NAF block is followed by the following Figure 4 The SCAM module shown uses the stereo features generated by the previous NAF block as input, performs bidirectional cross-view interaction, and outputs interaction features fused with the input intra-view features.

[0108] In the embodiment of the present application, the NAF block consists of a mobile convolution module and a feedforward network module, and each module is preceded by normalization to stabilize the learning process by normalizing the input. In addition, both modules are added with residual connections to promote smoother gradient flow and allow deeper network architectures to be built without increasing gradient risk.

[0109] Among them, Figure 4 As shown in Figure 1, SCAM is a stereoscopic cross attention module based on scaled dot product attention, which can calculate the dot product of the query vector and all key vectors, and apply a Softmax function to obtain the weights of the corresponding values:

[0110]

[0111] In the above formula, Q is a matrix mapped by the feature of the source internal view (such as the left view), K and V are the key matrix and value matrix mapped by the feature of the target internal view, respectively. Figure 4 H, W, and C are the feature map height, width, and number of channels, respectively. Since the stereo images are highly symmetrical under the horizontal line constraint, the same Q and K are used to represent each internal view feature, and the cross-view correlation is calculated along the horizontal line. Specifically, given the input internal view feature X L , X R , we can obtain the normalized stereo features, and then calculate the bidirectional cross attention of the left and right views:

[0112] F RL =Attention(W L1 XL ,W R1 X R ,W R2 X R )

[0113] F LR =Attention(W R1 X R ,W L1 X L ,W L2 X L )

[0114] F L =γ L F RL +X L

[0115] F R =γ R F LR +X R

[0116] Among them, W L1 , W R1 , W L2 , W R2 is the projection matrix, and only one calculation is needed to get the left and right attention matrices F LR and F RL Finally, the cross-view information and the intra-view information are fused by element-by-element addition, where F L and F R are the bidirectional cross attention for the left and right views, respectively, L and γ R are trainable channel-by-channel scaling factors.

[0117] (4) Adaptively upsample the fused image features and use a decoding neural network to obtain a high-resolution RGB image.

[0118] In the actual implementation process, the embodiment of the present application can decode the residual network. The decoding module first maps the input H×W to 128 channels and passes it through a residual network. Then it is upsampled by sub-pixel convolution to change its dimension to sH×sW. After another convolution layer, the feature dimension is increased to 2 2 s 2 D′, the shape is H / 2×W / 2×2 2 The feature vector at each position is then rearranged into a 2s×2s×D′ tensor to obtain a higher resolution feature map of shape H×W×D′, where H and W are the feature map height and width, s is the upsampling rate, and D′ is the sub-pixel convolutional layer output feature size.

[0119] In an embodiment of the present application, the sub-pixel convolution layer can be initialized using the ICNR initialization method to optimize its output, and Gaussian smoothing can be performed on the output feature map to improve the quality of the image. Subsequently, these smoothed feature maps are upsampled and further processed through a series of residual blocks, and finally, these feature maps are passed through a convolution layer to obtain a high-resolution RGB image.

[0120] According to the artificial intelligence-based super-resolution image processing method proposed in the embodiment of the present application, a second target image that meets a certain pixel condition can be obtained by segmenting a first target image that meets a first resolution condition, and the pixels of the second target image are aligned using the extracted depth features of the second target image to obtain a third target image, and a pre-set target neural network is used to perform feature fusion processing on the third target image to obtain an RGB image that meets the second resolution condition of the first target image. As a result, the limitation problem of the image super-resolution processing method in the related art is solved, the applicability of the image super-resolution method is improved, the complexity of the image super-resolution operation is reduced, and the efficiency of the image super-resolution processing is improved.

[0121] Next, an artificial intelligence-based super-resolution image processing device proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0122] Figure 5 It is a block diagram of an artificial intelligence-based super-resolution image processing device according to an embodiment of the present application.

[0123] like Figure 5 As shown, the artificial intelligence-based super-resolution image processing device 10 includes: an acquisition module 100, a determination module 200 and a processing module 300.

[0124] Specifically, the acquisition module 100 is used to acquire a first target image that meets a first preset resolution condition, and segment the first target image to obtain a second target image that meets a preset pixel condition.

[0125] The determination module 200 is used to extract the depth features of the second target image, and align the pixels of the second target image using the extracted depth features to obtain a third target image.

[0126] The processing module 300 is used to perform feature fusion processing on the third target image using a preset target neural network to obtain a fused image, and obtain an RGB image that meets a second preset resolution condition for the first target image based on the fused image.

[0127] Optionally, in one embodiment of the present application, the acquisition module 100 includes: an acquisition unit and a first determination unit.

[0128] The acquisition unit is used to acquire a plurality of continuous frames of images of the target scene that meet a first preset resolution condition.

[0129] The first determination unit is used to determine a target segmentation method for the continuous multiple frames of images, so as to segment the continuous multiple frames of images using the target segmentation method to obtain a second target image that meets a preset pixel condition.

[0130] Optionally, in one embodiment of the present application, the determination module 200 includes: an extraction unit, a prediction unit and a second determination unit.

[0131] The extraction unit is used to encode each frame of the second target image to extract the depth feature of the second target image.

[0132] The prediction unit is used to use the first frame image of the continuous multiple frame images as the basic frame image, and use the depth feature to predict the offset of other frame images and the harmonic scalar of other frame images relative to the basic frame image.

[0133] The second determining unit is used to align the current frame image with the base frame image according to the offset and the harmonization scalar to obtain a third target image.

[0134] Optionally, in one embodiment of the present application, the processing module 300 includes: a first processing unit and a third determining unit.

[0135] Among them, the first processing unit is used to predict the fusion weight of each pixel in the third target image based on the target content and noise level of the third target image and a preset neural network, and normalize the fusion weight to obtain a normalized weight.

[0136] The third determining unit is used to perform weighted summation on the depth features of the second target image using the normalized weights to obtain a fused feature map, so as to obtain a fused image according to the fused feature map.

[0137] Optionally, in one embodiment of the present application, the processing module 300 includes: a fourth determining unit, a fifth determining unit and a second processing unit.

[0138] The fourth determination unit is used to determine the upsampling rate of the continuous multiple frame images based on the fused image.

[0139] The fifth determining unit is used to upsample the continuous multiple frame images using the upsampling rate to obtain the upsampled images.

[0140] The second processing unit is used to reconstruct and decode the upsampled image respectively to obtain an RGB image of the first target image that meets a second preset resolution condition.

[0141] It should be noted that the aforementioned explanation of the embodiment of the super-resolution image processing method based on artificial intelligence is also applicable to the super-resolution image processing device based on artificial intelligence in this embodiment, and will not be repeated here.

[0142] According to the artificial intelligence-based super-resolution image processing device proposed in the embodiment of the present application, the first target image that meets the first resolution condition can be segmented to obtain a second target image that meets a certain pixel condition, and the pixels of the second target image are aligned using the extracted depth features of the second target image to obtain a third target image, and the third target image is subjected to feature fusion processing using a pre-set target neural network to obtain an RGB image that meets the second resolution condition of the first target image. As a result, the limitation problem of the image super-resolution processing method in the related art is solved, the applicability of the image super-resolution method is improved, the complexity of the image super-resolution operation is reduced, and the efficiency of the image super-resolution processing is improved.

[0143] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0144] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0145] When the processor 602 executes the program, the super-resolution image processing method based on artificial intelligence provided in the above embodiment is implemented.

[0146] Furthermore, the electronic device further comprises:

[0147] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0148] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0149] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0150] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0151] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0152] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0153] This embodiment also provides a computer program product, including a computer program, which, when executed, is used to implement the above artificial intelligence-based super-resolution image processing method.

[0154] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0155] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0156] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0157] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0158] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0159] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0161] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A super-resolution image processing method based on artificial intelligence, characterized in that: The following steps are involved: Acquire a first target image that meets a first preset resolution condition, and segment the first target image to obtain a second target image that meets a preset pixel condition; Extracting depth features of the second target image, and aligning pixels of the second target image using the extracted depth features to obtain a third target image; The third target image is subjected to feature fusion processing by using a preset target neural network to obtain a fused image, and an RGB image satisfying a second preset resolution condition of the first target image is obtained based on the fused image.

2. The method according to claim 1, characterized in that The step of acquiring a first target image that meets a first preset resolution condition and segmenting the first target image to obtain a second target image that meets a preset pixel condition includes: Acquire a plurality of continuous frames of images of the target scene that meet the first preset resolution condition; A target segmentation method for the continuous multiple frames of images is determined, and the continuous multiple frames of images are segmented using the target segmentation method to obtain the second target image that meets the preset pixel condition.

3. The method according to claim 2, characterized in that The step of extracting the depth feature of the second target image and aligning pixels of the second target image using the extracted depth feature to obtain a third target image includes: Encoding each frame of the second target image to extract depth features of the second target image; Taking the first frame image of the continuous multiple frame images as the basic frame image, and using the depth feature to predict the offset of other frame images and the harmonic scalar of the other frame images relative to the basic frame image; The current frame image and the base frame image are aligned according to the offset and the harmonization scalar to obtain the third target image.

4. The method according to claim 3, characterized in that The step of performing feature fusion processing on the third target image by using a preset target neural network to obtain a fused image includes: Based on the target content and noise level of the third target image and the preset neural network, predict the fusion weight of each pixel in the third target image, and normalize the fusion weight to obtain a normalized weight; The normalized weights are used to perform weighted summation on the depth features of the second target image to obtain a fused feature map, so as to obtain the fused image according to the fused feature map.

5. The method according to claim 2, characterized in that: The step of obtaining an RGB image of the first target image that meets a second preset resolution condition according to the fused image includes: Determining an upsampling rate of the continuous multiple frames of images based on the fused image; Upsampling the continuous multiple frames of images using the upsampling rate to obtain an upsampled image; The upsampled image is reconstructed and decoded respectively to obtain the RGB image of the first target image that meets a second preset resolution condition.

6. A super-resolution image processing device based on artificial intelligence, characterized in that: include: An acquisition module, used for acquiring a first target image satisfying a first preset resolution condition, and segmenting the first target image to obtain a second target image satisfying a preset pixel condition; A determination module, configured to extract depth features of the second target image, and align pixels of the second target image using the extracted depth features to obtain a third target image; A processing module is used to perform feature fusion processing on the third target image using a preset target neural network to obtain a fused image, and to obtain an RGB image that satisfies a second preset resolution condition for the first target image based on the fused image.

7. The device according to claim 6, characterized in that The acquisition module comprises: An acquisition unit, configured to acquire a plurality of continuous frames of images of a target scene that meet a first preset resolution condition; The first determination unit is used to determine a target segmentation method for the continuous multiple frames of images, so as to segment the continuous multiple frames of images using the target segmentation method to obtain the second target image that meets the preset pixel condition.

8. The device according to claim 7, characterized in that The determination module comprises: an extraction unit, configured to encode each frame of the second target image to extract a depth feature of the second target image; A prediction unit, configured to use a first frame image of the continuous multiple frame images as a base frame image, and use the depth feature to predict the offset of other frame images and the harmonic scalar of the other frame images relative to the base frame image; The second determining unit is used to align the current frame image with the basic frame image according to the offset and the harmonization scalar to obtain the third target image.

9. The device according to claim 8, characterized in that The processing module comprises: A first processing unit is used to predict the fusion weight of each pixel in the third target image based on the target content and noise level of the third target image and the preset neural network, and normalize the fusion weight to obtain a normalized weight; The third determining unit is used to perform weighted summation on the depth features of the second target image using the normalized weights to obtain a fused feature map, so as to obtain the fused image according to the fused feature map.

10. The device according to claim 7, characterized in that The processing module comprises: A fourth determining unit, configured to determine an upsampling rate of the continuous multiple frames of images based on the fused image; a fifth determining unit, configured to upsample the continuous multiple frames of images using the upsampling rate to obtain an upsampled image; The second processing unit is used to reconstruct and decode the upsampled image respectively to obtain the RGB image of the first target image that meets the second preset resolution condition.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the artificial intelligence-based super-resolution image processing method as described in any one of claims 1 to 5.

12. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the artificial intelligence-based super-resolution image processing method as described in any one of claims 1 to 5.

Images