Image interpolation method based on high dimension

Abstract

The invention relates to the technical field of computer vision and image processing, and discloses an image interpolation method based on high dimensionality. The method comprises the following steps: establishing a video decoder module, a timestamp analysis module, a feature extraction module, a time sequence correlation modeling module, a time sequence correlation modeling module, a convolutional neural network module, an edge smoothing module, an edge detection module, an edge enhancement module, an image output module and a quality evaluation module; the time stamp analysis module records time points and positions; the feature extraction module extracts image edge feature data; the time sequence correlation modeling module captures time and spatial correlation data; the convolutional neural network module performs pixel-level interpolation on image frames; the edge enhancement module is used for enhancing edge pixel points, the image output module is used for generating high-quality interpolation image frames, and the quality evaluation module is used for carrying out quality inspection on the generated image.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to an image interpolation method based on high dimension. Background Art

[0002] In the field of image interpolation, although high-dimensional image interpolation methods can improve the resolution and detail restoration capabilities of images to a certain extent, there are still some significant defects in the existing technologies. Among them, the most prominent problem is that when the image magnification is high, these methods may introduce artifacts and noise, thereby reducing the quality and readability of the image.

[0003] Artifacts and noise are mainly caused by inaccurate estimation of image pixel values ​​during the interpolation process. In high-dimensional image interpolation, the algorithm needs to infer the values ​​of unknown pixels based on the values ​​of known pixels, a process that often involves complex mathematical models and a lot of calculations. However, due to the complexity and diversity of image data, as well as the limitations of the interpolation algorithm itself, it is difficult to ensure that the estimation of all pixels is completely accurate. Therefore, in the interpolated image, some pixel value changes that did not originally exist may appear, i.e., artifacts. At the same time, since some random errors or rounding errors may be introduced during the interpolation process, noise may also appear in the image.

[0004] The appearance of artifacts and noise not only affects the visual effect of the image, making it look more blurred or distorted, but may also bring difficulties to subsequent image processing and analysis. For example, in medical image analysis, artifacts and noise may interfere with the doctor's judgment of lesions; in remote sensing image processing, they may affect the recognition and classification accuracy of objects. Therefore, how to effectively suppress or eliminate artifacts and noise is one of the important issues that high-dimensional image interpolation methods need to solve. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides an image interpolation method based on high dimension, which has the advantage of being able to effectively suppress or eliminate artifacts and noise, and solves the problem that the prior art cannot effectively suppress or eliminate artifacts and noise when used.

[0006] (II) Technical solution To achieve the above object, the present invention provides the following technical solution: a high-dimensional image interpolation method, comprising the following steps: Step 1: Establish a video decoder module, a timestamp analysis module, a feature extraction module, a temporal association modeling module, a temporal association modeling module, a convolutional neural network module, an edge smoothing processing module, an edge detection module, an edge enhancement module, an image output module and a quality assessment module; Step 2: Use the video decoder module to extract images frame by frame from the original video file; Step 3: Record the specific time point and position of each frame through the timestamp analysis module to provide a reference for subsequent interpolation operations; Step 4: Using the feature extraction module, extract the image edge feature data of the edge and texture from each frame of the image; Step 5: Use the temporal correlation modeling module to build a multi-dimensional temporal correlation model based on the image edge feature data in the previous step to capture the temporal and spatial correlation data between adjacent frames; Step 6: Use the convolutional neural network module to perform pixel-level interpolation on the image frame to predict and fill in the values ​​of unknown pixels; Step 7: The edge smoothing processing module further optimizes the interpolation result to ensure smooth transition and detail restoration of edge pixels; Step 8: Use the edge detection module to identify blurred or distorted edge pixels generated during the interpolation process; Step 9: Apply edge enhancement module to enhance edge pixels; Step 10: Use the image output module to generate high-quality interpolated image frames; Step 11: Perform quality check on the generated image through the quality assessment module.

[0007] Preferably, the feature extraction module extracts 9 sets of image edge feature data about edges and textures from each frame of image, and counts and numbers the 9 sets of image edge feature data, wherein the 9 sets of image edge feature data are numbered as follows: , based on the 9 sets of numbered image edge feature data, an original image matrix is ​​formed ,as follows: ; Original image matrix is 3x3, Represents the first set of edge feature data extracted from the image, represents the second set of edge feature data extracted from the image, Represents the third set of edge feature data extracted from the image.

[0008] Preferably, the original image matrix Calculate the edge strength , and its calculation formula is as follows: ;In the formula, represents the generated edge strength, Respectively represent 9 groups of image edge feature data, represents the gradient in the horizontal direction, Represents the gradient in the vertical direction.

[0009] Preferably, the edge strength It can reflect the edge degree of the acquired image. When the edge strength When it is high, it corresponds to the edge area in the image; When it is low, it corresponds to the non-edge area in the image.

[0010] Preferably, the temporal correlation modeling module constructs a multi-dimensional temporal correlation model based on the image edge feature data of the previous step, captures the temporal and spatial correlation data between adjacent frames, records and counts the temporal and spatial correlation data between adjacent frames, and performs data numbering, wherein the temporal correlation data between adjacent frames is numbered as follows: , the spatial correlation data between adjacent frames is numbered as .

[0011] Preferably, the temporal correlation data between adjacent frames and the spatial correlation data between adjacent frames are jointly calculated to generate pixel values. , and its calculation formula is: ; In the formula, represents the pixel value, represents the temporal correlation data between adjacent frames, Represents the spatial correlation data between adjacent frames, represents the temporal and spatial correlation coefficients between adjacent frames, Indicates the amount of statistical data.

[0012] Preferably, the pixel value The calculation can be dynamically adjusted according to the changes in image content to adapt to different image sequences and scenes.

[0013] Preferably, the convolutional neural network module performs pixel-level interpolation on the image frame, predicts and fills the value of unknown pixel points, records and counts the pixel-level interpolation and predicted unknown pixel points of the image frame, and performs data numbering, and the pixel-level interpolation number of the image frame is , the predicted unknown pixel value number is .

[0014] Preferably, the image frame pixel-level interpolation and the predicted unknown pixel value are jointly calculated to generate a high-dimensional prediction model pixel difference index , and its calculation formula is as follows: ; In the formula, Represents the pixel difference index of the high-dimensional prediction model, represents the pixel-level interpolation of the image frame, represents the pixel-level interpolation of the i-th image frame, Represents the predicted unknown pixel value, represents the unknown pixel value of the i-th prediction, Indicates the number of values ​​​​to be counted.

[0015] Preferably, the high-dimensional prediction model pixel difference index , can be used as a feedback signal to dynamically adjust the interpolation strategy or parameters to adapt to different image content and complexity.

[0016] Compared with the prior art, the present invention provides a high-dimensional image interpolation method, which has the following beneficial effects: 1. The present invention calculates the edge strength , through this threshold and edge strength By comparison, it can effectively suppress or eliminate some artifacts and noise. After identifying the edge points, filtering techniques (such as Gaussian filtering, median filtering, etc.) are further applied to smooth the image and reduce residual noise. At the same time, morphological operations (such as dilation, erosion, opening and closing operations) are used to clean up the details in the image, remove isolated noise points, and maintain the integrity of the edges. Finally, the processed image is visually inspected to verify the effect of edge detection. If there are still artifacts or noise that has not been completely eliminated, a more complex post-processing technique will be switched.

[0017] 2. The present invention calculates pixel values , this value can be dynamically adjusted according to the changes in image content, so as to adapt to different image sequences and scenes to improve the universality of interpolation. When it is higher, it means that the pixel has a higher importance in the image, and it also means that the pixel is located at the edge of the image, in the texture area, or in an area with large color changes. It helps the model to reconstruct key areas more accurately during the interpolation process, thereby improving the overall quality of the image. On the contrary, when the calculated pixel value When it is low, it means that the pixel is less important in the image, and it also means that the pixel is located in the background area or in an area with small color changes. Pixels with values ​​of 0 do not need to be paid much attention to during the interpolation process because they have little impact on the overall image quality. The calculation method can not only improve the accuracy of interpolation, but also retain the key features of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See also Figure 1 , a high-dimensional image interpolation method, comprising the following steps: Step 1: Establish a video decoder module, a timestamp analysis module, a feature extraction module, a temporal association modeling module, a temporal association modeling module, a convolutional neural network (CNN) module, an edge smoothing processing module, an edge detection module, an edge enhancement module, an image output module and a quality assessment module; Step 2: Use the video decoder module to extract images frame by frame from the original video file; Step 3: Record the specific time point and position of each frame through the timestamp analysis module to provide a reference for subsequent interpolation operations; Step 4: Using the feature extraction module, extract the image edge feature data of the edge and texture from each frame of the image; Step 5: Use the temporal correlation modeling module to build a multi-dimensional temporal correlation model based on the image edge feature data in the previous step to capture the temporal and spatial correlation data between adjacent frames; Step 6: Use the convolutional neural network module to perform pixel-level interpolation on the image frame to predict and fill in the values ​​of unknown pixels; Step 7: The edge smoothing processing module further optimizes the interpolation result to ensure smooth transition and detail restoration of edge pixels; Step 8: Use the edge detection module to identify blurred or distorted edge pixels generated during the interpolation process; Step 9: Apply the edge enhancement module to enhance the edge pixels (to improve the clarity and readability of the overall image); Step 10: Use the image output module to generate high-quality interpolated image frames (ensuring that the resolution is improved while maintaining the edge clarity and detail richness of the original image); Step 11: Perform a quality check on the generated image through the quality assessment module (to ensure that it meets the requirements of subsequent processing and analysis tasks and output the interpolated image frame).

[0021] The feature extraction module extracts 9 sets of image edge feature data about edges and textures from each frame of image, and counts and numbers the 9 sets of image edge feature data. The 9 sets of image edge feature data are numbered as follows: , based on the 9 sets of numbered image edge feature data, an original image matrix is ​​formed ,as follows: ; Original image matrix is 3x3, Represents the first set of edge feature data extracted from the image, represents the second set of edge feature data extracted from the image, Represents the third set of edge feature data extracted from the image.

[0022] Original image matrix Calculate the edge strength , and its calculation formula is as follows: ;In the formula, represents the generated edge strength, Respectively represent 9 groups of image edge feature data, represents the gradient in the horizontal direction, Represents the gradient in the vertical direction.

[0023] The advantage is: by calculating the edge strength , and use a single gradient amplitude as the edge strength, and form an original image matrix with the numbered 9 sets of image edge feature data to generate multi-dimensional feature data. Can fully reflect the edge information of the image, according to the edge strength Substitute the data features and historical model data into the edge strength The threshold value is when the edge strength When the edge intensity is higher than the threshold, it corresponds to the edge area in the image and the pixel is considered to be an edge point. When it is lower than the threshold, it corresponds to the non-edge area in the image and is considered a non-edge point. By comparison, it can effectively suppress or eliminate some artifacts and noise. After identifying the edge points, filtering techniques (such as Gaussian filtering, median filtering, etc.) are further applied to smooth the image and reduce residual noise. At the same time, morphological operations (such as dilation, erosion, opening and closing operations) are used to clean up the details in the image, remove isolated noise points, and maintain the integrity of the edges. Finally, the processed image is visually inspected to verify the effect of edge detection. If there are still artifacts or noise that has not been completely eliminated, a more complex post-processing technique will be switched.

[0024] Edge Strength It can reflect the edge degree of the acquired image. When the edge strength When it is high, it corresponds to the edge area in the image; When it is low, it corresponds to the non-edge area in the image.

[0025] The temporal correlation modeling module builds a multi-dimensional temporal correlation model based on the image edge feature data in the previous step, captures the temporal and spatial correlation data between adjacent frames (the temporal correlation between adjacent frames refers to the similarity between consecutive frames in an image sequence. Since objects in a video or image sequence will not change dramatically in a short period of time, the pixel values ​​between adjacent frames will be similar), records and counts the temporal and spatial correlation data between adjacent frames, and numbers the data. The temporal correlation data between adjacent frames are numbered as , the spatial correlation data between adjacent frames is numbered as .

[0026] The temporal correlation data between adjacent frames and the spatial correlation data between adjacent frames are jointly calculated to generate pixel values , and its calculation formula is: ; In the formula, represents the pixel value, represents the temporal correlation data between adjacent frames, Represents the spatial correlation data between adjacent frames, represents the temporal and spatial correlation coefficients between adjacent frames, Indicates the amount of statistical data.

[0027] The advantage is: by calculating the pixel value , this value can be dynamically adjusted according to the changes in image content, so as to adapt to different image sequences and scenes to improve the universality of interpolation. When it is higher, it means that the pixel has a higher importance in the image, and it also means that the pixel is located at the edge of the image, in the texture area, or in an area with large color changes. It helps the model to reconstruct key areas more accurately during the interpolation process, thereby improving the overall quality of the image. On the contrary, when the calculated pixel value When it is low, it means that the pixel is less important in the image, and it also means that the pixel is located in the background area or in an area with small color changes. Pixels with values ​​of 0 do not need to be paid much attention to during the interpolation process because they have little impact on the overall image quality. The calculation method can not only improve the accuracy of interpolation, but also retain the key features of the image.

[0028] Pixel Value The calculation can be dynamically adjusted according to changes in image content to adapt to different image sequences and scenes (thereby improving the universality and effect of interpolation).

[0029] Convolutional neural network module, convolutional neural network module, performs pixel-level interpolation on image frames, predicts and fills in the values ​​of unknown pixels, records and counts the pixel-level interpolation and predicted unknown pixel values ​​of image frames, and performs data numbering. The pixel-level interpolation number of the image frame is , the predicted unknown pixel value number is .

[0030] The pixel-level interpolation of the image frame and the predicted unknown pixel value are jointly calculated to generate the pixel difference index of the high-dimensional prediction model , and its calculation formula is as follows: ; In the formula, Represents the pixel difference index of the high-dimensional prediction model, represents the pixel-level interpolation of the image frame, represents the pixel-level interpolation of the i-th image frame, Represents the predicted unknown pixel value, represents the unknown pixel value of the i-th prediction, Indicates the number of values ​​​​to be counted.

[0031] The advantages are: by calculating the pixel difference index of the high-dimensional prediction model The interpolation method can better cope with various types of image sequences and scenes, and can obtain high-quality interpolation effects in both static and dynamic image scenes, thereby enhancing the universality of the method of the present invention. It can help reconstruct the quality of the frame and dynamically adjust the interpolation parameters to reduce artifacts and noise, thereby retaining more detail information to improve the visual effect of the image, while using the high-dimensional prediction model pixel difference index It can also enhance the robustness of the interpolation model, enabling the model to maintain stable performance in different environments, thereby processing complex and changeable image content to improve the reliability of the system.

[0032] High Dimensional Prediction Model Pixel Difference Index , can be used as a feedback signal to dynamically adjust the interpolation strategy or parameters to adapt to different image content and complexity.

[0033] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-dimensional image interpolation method, characterized in that: The following steps are involved: Step 1: Establish a video decoder module, a timestamp analysis module, a feature extraction module, a temporal association modeling module, a temporal association modeling module, a convolutional neural network module, an edge smoothing processing module, an edge detection module, an edge enhancement module, an image output module and a quality assessment module; Step 2: Use the video decoder module to extract images frame by frame from the original video file; Step 3: Record the specific time point and position of each frame through the timestamp analysis module to provide a reference for subsequent interpolation operations; Step 4: Using the feature extraction module, extract the image edge feature data of the edge and texture from each frame of the image; Step 5: Use the temporal correlation modeling module to build a multi-dimensional temporal correlation model based on the image edge feature data in the previous step to capture the temporal and spatial correlation data between adjacent frames; Step 6: Use the convolutional neural network module to perform pixel-level interpolation on the image frame to predict and fill in the values ​​of unknown pixels; Step 7: The edge smoothing processing module further optimizes the interpolation result to ensure smooth transition and detail restoration of edge pixels; Step 8: Use the edge detection module to identify blurred or distorted edge pixels generated during the interpolation process; Step 9: Apply edge enhancement module to enhance edge pixels; Step 10: Use the image output module to generate high-quality interpolated image frames; Step 11: Perform quality check on the generated image through the quality assessment module.

2. The high-dimensional image interpolation method according to claim 1, characterized in that: The feature extraction module extracts 9 sets of image edge feature data about edges and textures from each frame of image, and counts and numbers the 9 sets of image edge feature data. The 9 sets of image edge feature data are numbered as follows: , based on the 9 sets of numbered image edge feature data, an original image matrix is ​​formed ,as follows: ； Original image matrix is 3x3, Represents the first set of edge feature data extracted from the image, represents the second set of edge feature data extracted from the image, Represents the third set of edge feature data extracted from the image.

3. The high-dimensional image interpolation method according to claim 2, characterized in that: The original image matrix Calculate the edge strength , and its calculation formula is as follows: ;In the formula, represents the generated edge strength, Respectively represent 9 groups of image edge feature data, represents the gradient in the horizontal direction, Represents the gradient in the vertical direction.

4. The high-dimensional image interpolation method according to claim 3, characterized in that: The edge strength It can reflect the edge degree of the acquired image. When the edge strength When it is high, it corresponds to the edge area in the image; When it is low, it corresponds to the non-edge area in the image.

5. The high-dimensional image interpolation method according to claim 1, characterized in that: The temporal correlation modeling module constructs a multi-dimensional temporal correlation model based on the image edge feature data of the previous step, captures the temporal and spatial correlation data between adjacent frames, records and counts the temporal and spatial correlation data between adjacent frames, and performs data numbering. The temporal correlation data between adjacent frames is numbered as follows: , the spatial correlation data between adjacent frames is numbered as .

6. The high-dimensional image interpolation method according to claim 5, characterized in that: The temporal correlation data between adjacent frames and the spatial correlation data between adjacent frames are jointly calculated to generate pixel values. , and its calculation formula is: ； In the formula, represents the pixel value, Represents the temporal correlation data between adjacent frames, Represents the spatial correlation data between adjacent frames, represents the temporal and spatial correlation coefficients between adjacent frames, Indicates the amount of statistical data.

7. The high-dimensional image interpolation method according to claim 6, characterized in that: The pixel value The calculation can be dynamically adjusted according to the changes in image content to adapt to different image sequences and scenes.

8. The high-dimensional image interpolation method according to claim 1, characterized in that: The convolutional neural network module performs pixel-level interpolation on the image frame, predicts and fills the value of unknown pixel points, records and counts the pixel-level interpolation and predicted unknown pixel points of the image frame, and performs data numbering. The pixel-level interpolation number of the image frame is , the predicted unknown pixel value number is .

9. The high-dimensional image interpolation method according to claim 8, characterized in that: The image frame pixel-level interpolation and the predicted unknown pixel value are jointly calculated to generate a high-dimensional prediction model pixel difference index , and its calculation formula is as follows: ； In the formula, Represents the pixel difference index of the high-dimensional prediction model, represents the pixel-level interpolation of the image frame, represents the pixel-level interpolation of the i-th image frame, Represents the predicted unknown pixel value, represents the unknown pixel value of the i-th prediction, Indicates the number of values ​​being counted.

10. The high-dimensional image interpolation method according to claim 9, characterized in that: The high-dimensional prediction model pixel difference index , can be used as a feedback signal to dynamically adjust the interpolation strategy or parameters to adapt to different image content and complexity.

Images