Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium, and is applied to the technical field of image processing. According to the scheme, the method comprises the following steps: detecting angular points in a target image to be interpolated based on an angular point detection algorithm to obtain feature points in the target image; for the texture complexity of the neighborhood image of each feature point, selecting a target interpolation algorithm for processing the feature point; and performing interpolation processing on the feature points based on a target interpolation algorithm to generate an interpolation result image. For angular points with complex textures, a high-precision interpolation algorithm is adopted to ensure that image details are finely restored; and for the angular points with slightly simple textures, a low-calculation-amount algorithm is used to reduce resource occupation, so that the limitation that a single algorithm is uniformly adopted for the whole image in the prior art is avoided, and the redundant calculation amount is remarkably reduced while the interpolation precision of the key feature region is ensured and the quality of the whole image is maintained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium based on corner detection. Background Art

[0002] Interpolation is a mathematical method for estimating the value of an unknown area using known discrete data points. Its core principle is to construct a continuous function that strictly passes through all known data points and smoothly transitions between them. This technology is widely used in image processing, scientific computing, geographic information analysis, and other fields, such as adding pixels when scaling images and reconstructing sensor data.

[0003] Existing interpolation algorithms include nearest neighbor interpolation, linear interpolation, bilinear interpolation, bicubic interpolation, Lagrange interpolation, Newton interpolation, Hermite interpolation, cubic spline interpolation, and GPU-based KRIGING interpolation. These algorithms have low precision due to low computational effort, and high precision due to high computational effort. For example, the nearest neighbor interpolation algorithm has a low computational effort but low interpolation accuracy. It simply assigns the value of the nearest known point to the target point without considering the influence of other surrounding points, resulting in discontinuities such as jagged edges in the interpolation result where brightness changes. High-precision interpolation algorithms such as cubic spline interpolation and bicubic interpolation require complex calculations based on multiple surrounding points, constructing high-order polynomials to approximate the target point value. This results in a cumbersome and computationally intensive process. Summary of the Invention

[0004] In view of this, the present application is committed to providing an image processing method, apparatus, device and storage medium that can significantly reduce the amount of calculation while ensuring the interpolation accuracy of key feature areas and maintaining the overall image quality.

[0005] According to a first aspect of the present application, there is provided an image processing method, comprising:

[0006] Obtain the target image to be interpolated;

[0007] Detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image;

[0008] For each of the feature points, a target interpolation algorithm for processing the feature point is selected based on the texture complexity of a first neighborhood image; the first neighborhood image is a neighborhood image of the feature point; and the calculation accuracy of the target difference algorithm is positively correlated with the texture complexity;

[0009] Based on the target interpolation algorithm, the feature points are interpolated to generate an interpolation result image.

[0010] Optionally, for each of the feature points, selecting a target interpolation algorithm for processing the feature point according to the texture complexity of the first neighborhood image includes:

[0011] Calculating a local brightness average and a local brightness standard deviation of the first neighborhood image;

[0012] Calculating a weighted sum of the local brightness average and the local brightness standard deviation to obtain a local eigenvalue for texture complexity of the first neighborhood image;

[0013] By comparing the local eigenvalue with a first threshold, a target interpolation algorithm for processing the feature point is determined; the first threshold is determined based on a global statistic of the target image and is used to represent the overall texture complexity of the target image.

[0014] Optionally, detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image includes:

[0015] Calculating a structural tensor matrix of a target pixel point according to a second neighborhood image of the target pixel point; the target pixel point is any pixel point in the target image;

[0016] Calculating a corner point response value of the target pixel point according to the corner point response function and the structure tensor matrix;

[0017] When the corner point response value is not less than a second threshold, the target pixel point is marked as a feature point in the target image; the second threshold is determined based on the global statistics of the target image and is used to represent the overall texture complexity of the target image.

[0018] Optionally, the detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image further includes:

[0019] Obtaining a global brightness average and a global brightness standard deviation of the target image;

[0020] A weighted sum of the global brightness average value and the global brightness standard deviation is calculated to obtain the second threshold.

[0021] Optionally, calculating the structure tensor matrix of the target pixel point based on the second neighborhood image of the target pixel point includes:

[0022] For each neighborhood pixel in the second neighborhood image, calculating a brightness gradient value at the neighborhood pixel;

[0023] Calculating a target matrix corresponding to the neighborhood pixel points according to the brightness gradient value at the neighborhood pixel points; the structure tensor matrix has the same shape as the target matrix;

[0024] Determining a first weight value corresponding to the neighborhood pixel point according to a brightness difference between the neighborhood pixel point and the target pixel point; the brightness difference between the neighborhood pixel point and the target pixel point is negatively correlated with the first weight value;

[0025] Calculating a structure tensor matrix of the target pixel point based on the first weight value of each of the neighborhood pixels and the target matrix;

[0026] Each element in the structure tensor matrix is ​​obtained by weighted summing the elements at the same position in the target matrix according to the first weight value of each of the neighborhood pixels.

[0027] Optionally, determining the first weight value corresponding to the neighborhood pixel point according to the brightness difference between the neighborhood pixel point and the target pixel point includes:

[0028] The first weight value corresponding to the neighborhood pixel point is calculated based on the following formula:

[0029]

[0030] Wherein, w(x,y) is the first weight value of the neighborhood pixel; I (x,y) is the brightness value of the neighborhood pixel; c is the brightness value of the target pixel; α is a preset coefficient.

[0031] Optionally, the method further includes:

[0032] The interpolation result image is segmented to obtain sub-images of the interpolation result image:

[0033] Inputting the sub-image into a preset neural network model to obtain a first prediction result; the first prediction result is used to represent second weight values ​​corresponding to various preset interpolation algorithms;

[0034] Based on the second weight values ​​respectively corresponding to the various preset interpolation algorithms, weighted summation is performed on the interpolation results obtained by performing interpolation processing on the sub-image using the various preset interpolation algorithms to obtain an optimized sub-image;

[0035] An optimized interpolation result image is obtained according to the optimized sub-image.

[0036] Optionally, before inputting the sub-image into a preset neural network model, the method further includes:

[0037] Obtaining a training sample set including a plurality of sample picture groups and first label data corresponding to the sample picture groups; the sample picture groups including a first sample image and a second sample image obtained by performing image degradation processing on the sample image; the first label data being used to represent target weight values ​​corresponding to each of the preset interpolation algorithms; performing a weighted summation of interpolation results of the sub-images obtained by the various preset interpolation algorithms based on the target weight values, and obtaining a minimum error relative to the first sample image;

[0038] The initial model is trained based on the training sample set, with the goal of minimizing the value of the loss function, and the model parameters of the initial model are adjusted to obtain a preset neural network model; the loss function is used to represent the difference between the first prediction result output by the initial model and the first label data.

[0039] Optionally, inputting the sub-image into a preset neural network model to obtain a first prediction result includes:

[0040] The sub-image is input into the preset neural network model to obtain the first prediction result and the second prediction result; the second prediction result is used to represent the prediction errors corresponding to the various preset interpolation algorithms respectively; the prediction error corresponding to the preset interpolation algorithm is used to represent the error of the interpolation result obtained by interpolating the sub-image based on the preset interpolation algorithm.

[0041] Optionally, the method further includes:

[0042] Obtaining second label data corresponding to the sample image group; the second label data is used to represent an error of an interpolated image obtained by interpolating the second sample image based on various preset interpolation algorithms relative to the first sample image;

[0043] The preset neural network model is also used to output a second prediction result; the second prediction result is used to represent the prediction errors corresponding to the various preset interpolation algorithms respectively; the loss function is used to represent the weighted sum of the first loss and the second loss; the first loss is used to represent the difference between the first prediction result output by the initial model and the first label data; the second loss is used to represent the difference between the second prediction result output by the initial model and the second label data.

[0044] According to a second aspect of the present application, there is provided an image processing apparatus, comprising:

[0045] An acquisition module, used for acquiring a target image to be interpolated;

[0046] A detection module, configured to detect corner points in the target image based on a corner detection algorithm to obtain feature points in the target image;

[0047] a selection module configured to select, for each feature point, a target interpolation algorithm for processing the feature point based on the texture complexity of a first neighborhood image; the first neighborhood image being a neighborhood image of the feature point; and a calculation accuracy of the target interpolation algorithm being positively correlated with the texture complexity;

[0048] The interpolation module is used to perform interpolation processing on the feature points based on a target interpolation algorithm to generate an interpolation result image.

[0049] According to a third aspect of the present application, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor is configured to execute the method described in any one of the above embodiments.

[0050] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above embodiments.

[0051] Existing interpolation algorithms have minimal performance differences in areas with simple textures. Because pixel brightness changes gradually, even algorithms with lower computational effort and moderate accuracy can still achieve approximate results. However, interpolation quality degrades significantly in areas with complex textures. Therefore, the overall performance of image interpolation algorithms is essentially limited by their ability to process complex textures.

[0052] The present application provides an image processing method, apparatus, device and storage medium. The solution: based on a corner detection algorithm, detects corner points in a target image to be interpolated to obtain feature points in the target image; selects a target interpolation algorithm for processing the feature point based on the texture complexity of the neighborhood image of each feature point; and then, based on the target interpolation algorithm, interpolates the feature points to generate an interpolation result image.

[0053] This enables the use of high-precision interpolation algorithms for corner points with complex textures to ensure precise restoration of image details and maintain interpolation accuracy in key feature areas to maintain overall image quality. For corner points with simpler textures, a low-computation algorithm is used to reduce resource usage. Compared to existing approaches that use a single algorithm for the entire image, this solution significantly reduces redundant computation while ensuring interpolation accuracy in key feature areas and maintaining overall image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Shown is a flow chart of an image processing method provided in one embodiment of the present application.

[0055] Figure 2Shown is a block diagram of an image processing device provided by one embodiment of the present application.

[0056] Figure 3 Shown is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] Application Overview

[0059] Image interpolation algorithms have multiple uses in vehicles. For example, in the vehicle's 360-degree surround view function, the image captured by the surround-view camera is a fisheye image. During the correction process, an interpolation algorithm is needed to restore image information at non-integer pixel positions. When converting the corrected image into a bird's-eye view, an interpolation algorithm is also needed to establish a coordinate mapping relationship after perspective transformation to fill the target pixels. When splicing several bird's-eye views into a surround view image, an interpolation algorithm is needed to perform weighted interpolation on the overlapping areas to eliminate splicing traces.

[0060] Existing interpolation algorithms include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, nine-point interpolation, etc. The following describes their general processes:

[0061] Nearest neighbor interpolation is the simplest interpolation method. It simply takes the brightness value of the nearest neighbor pixel based on the distance between the target pixel and the original image pixel and assigns it to the pixel. This method requires less computation but has low accuracy and is prone to jagged or mosaic artifacts.

[0062] Bilinear interpolation uses linear interpolation of the four neighboring points around the pixel to be determined, calculating the brightness value in two directions. This method requires slightly more computation than nearest neighbor interpolation, but it also improves accuracy, but may blur image edges.

[0063] Bicubic interpolation is more complex. It considers the weighted average of the surrounding 16 sampling points and calculates the weights through a cubic polynomial. The amount of calculation is large, but it can produce smoother edges and the processed image effect is better.

[0064] Nine-point interpolation combines the advantages of bicubic interpolation and Hermite interpolation to a certain extent, and uses the information of nine adjacent points for interpolation calculation. Its computational complexity is between bilinear interpolation and bicubic interpolation, and its accuracy is also between the two. It can balance the computational complexity and accuracy to a certain extent.

[0065] In general, the interpolation algorithm with small computational complexity has relatively low accuracy, while the interpolation algorithm with high accuracy has relatively large computational complexity.

[0066] In automotive image processing scenarios, there's often a trade-off between computational efficiency and interpolation quality. To ensure computational speed, a low-computation, but lower-precision, interpolation algorithm is often required for the entire image. However, to achieve high-quality interpolation, a high-computation, full-image interpolation algorithm must be employed. In this scenario, achieving both smooth and clear image processing is difficult within the computational performance limitations of vehicle chipsets.

[0067] Engineers discovered that image pixels can be categorized into corners, edges, and flat areas. Corners, where multi-directional edges converge, exhibit the most discrete grayscale gradient distribution and significantly higher texture complexity than other areas. Because the geometric characteristics of corner regions are crucial for expressing image structure, their interpolation accuracy directly determines the overall accuracy of the resulting image. However, the texture complexity of different corners within the same image varies, requiring different interpolation algorithms to achieve optimal processing results.

[0068] In order to solve the above problems, in the embodiment of the present application, based on the corner detection algorithm, the corner points in the target image to be interpolated are detected to obtain the feature points in the target image; according to the texture complexity of the neighborhood image of each feature point, the target interpolation algorithm for processing the feature point is selected; and then based on the target interpolation algorithm, the feature points are interpolated to generate an interpolation result image. Thus, by accurately locating the feature points (corner points) in the target image based on corner detection, and dynamically matching the target interpolation algorithm according to the texture complexity of the neighborhood of each feature point, a high-precision interpolation algorithm is used for corner points with complex textures to ensure that the image details are finely restored; and for corner points with slightly simpler textures, a low-computational algorithm is used to reduce resource usage, avoiding the limitation of the prior art of uniformly using a single algorithm for the entire image, while ensuring the interpolation accuracy of key feature areas and maintaining the overall image quality, the redundant calculation amount is significantly reduced.

[0069] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0070] Exemplary Methods

[0071] Figure 1 It is a flowchart of an image processing method provided by an embodiment of the present application. Figure 1 The method described is executed by a computing device, but the embodiment of the present application is not limited thereto. Figure 1 As shown, the method includes the following contents:

[0072] Step S110: Acquire a target image to be interpolated.

[0073] Step S120: detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image.

[0074] In an embodiment of the present application, the corner detection algorithm is used to identify pixel areas in an image with significant brightness changes and obvious gradient responses in multiple directions, and may include Harris corner detection based on brightness variance, Shi-Tomasi algorithm based on gradient extreme value judgment, and FAST feature detection based on accelerated segmentation test, etc.

[0075] In an embodiment of the present application, the corner point is the intersection point of two or more edges in the image, which is manifested as a pixel coordinate position with significant brightness change characteristics in at least two orthogonal directions. Its geometric characteristics include but are not limited to L-shaped corners, T-shaped connection points, and local extreme points with stable rotation invariance.

[0076] In the embodiment of the present application, the feature points in the target image are corner points in the target image.

[0077] Step S130: for each of the feature points, select a target interpolation algorithm for processing the feature point according to the texture complexity of a first neighborhood image; the first neighborhood image is the neighborhood image of the feature point.

[0078] In an embodiment of the present application, in an embodiment of the present application, the first neighborhood image is a neighborhood image of the feature point, which can be specifically defined as a local pixel area within a preset size range centered on the feature point, and is used to analyze texture complexity.

[0079] In an embodiment of the present application, the texture complexity of the first neighborhood image can be determined by calculating its brightness value distribution, brightness value change, gradient information or using other image analysis methods to evaluate the detail richness of the area.

[0080] In an embodiment of the present application, at least two candidate interpolation algorithms are preset, and different candidate interpolation algorithms correspond to different texture complexity threshold ranges and computing resource requirements. Candidate interpolation algorithms may include, but are not limited to, nearest neighbor interpolation, linear interpolation, bilinear interpolation, bicubic interpolation, Lagrange interpolation, Newton interpolation, Hermite interpolation, cubic spline interpolation, GPU-based KRIGING interpolation, etc.

[0081] In this embodiment of the present application, the computational complexity of the target interpolation algorithm is commensurate with the texture complexity of the first neighborhood image. A high-precision interpolation algorithm, such as bicubic interpolation, is used for corner points with complex textures. A low-computation interpolation algorithm, such as bilinear interpolation, can be used for corner points with relatively simple textures.

[0082] Step S140: performing interpolation processing on the feature points based on a target interpolation algorithm to generate an interpolation result image.

[0083] In the embodiment of the present application, the interpolation result image is an interpolation result image obtained by performing interpolation processing on the target image.

[0084] In the embodiments of the present application, for pixels other than feature points, an interpolation algorithm with lower precision and less computational complexity than the target interpolation algorithm can be uniformly used. For the latter, an interpolation algorithm can also be selected that is appropriate for the texture complexity of the neighboring image. For example, if the Harris corner detection algorithm is used, it is necessary to calculate a corner response value for each pixel in the target image, and the interpolation algorithm can be selected based on the corner response value.

[0085] It should be noted that interpolation processing is not required for pixels other than feature points in all cases. For example, when stitching several bird's-eye view images into a ring image, interpolation processing can be performed only on the feature points in the bird's-eye view images.

[0086] Existing interpolation algorithms have minimal performance differences in areas with simple textures. Because pixel brightness changes gradually, even algorithms with lower computational effort and moderate accuracy can still achieve approximate results. However, interpolation quality degrades significantly in areas with complex textures. Therefore, the overall performance of image interpolation algorithms is essentially limited by their ability to process complex textures.

[0087] In the embodiment of the present application, based on the corner detection algorithm, the corner points in the target image to be interpolated are detected to obtain the feature points in the target image; according to the texture complexity of the neighborhood image of each feature point, the target interpolation algorithm for processing the feature point is selected; and then based on the target interpolation algorithm, the feature points are interpolated to generate an interpolation result image. Thus, by accurately locating the feature points (corner points) in the target image based on corner detection, and dynamically matching the target interpolation algorithm according to the texture complexity of the neighborhood of each feature point, for corner points with complex textures, a high-precision interpolation algorithm is used to ensure that the image details are finely restored and the interpolation accuracy of the key feature areas is ensured to maintain the overall image quality; and for corner points with slightly simpler textures, a low-computational algorithm is used to reduce resource usage. Compared with the solution of uniformly using a single algorithm for the entire image in the prior art, the embodiment of the present application significantly reduces the amount of redundant calculations while ensuring the interpolation accuracy of the key feature areas and maintaining the overall image quality.

[0088] based on Figure 1 The method in this specification also provides some specific implementation plans of the method, which are described below.

[0089] Optionally, for each of the feature points, selecting a target interpolation algorithm for processing the feature point according to the texture complexity of the first neighborhood image includes:

[0090] Calculating a local brightness average and a local brightness standard deviation of the first neighborhood image;

[0091] Calculating a weighted sum of the local brightness average and the local brightness standard deviation to obtain a local eigenvalue for texture complexity of the first neighborhood image;

[0092] By comparing the local eigenvalue with a first threshold, a target interpolation algorithm for processing the feature point is determined; the first threshold is determined based on a global statistic of the target image and is used to represent the overall texture complexity of the target image.

[0093] In the embodiment of the present application, the local brightness average is the arithmetic mean of the brightness values ​​of all pixels in the first neighborhood image, and the calculation formula can be: Where W p Represents the first neighborhood image of feature point p; μ p Represents the first neighborhood image W p The local brightness average of I(x,y) represents the first neighborhood image W p The brightness value of the pixel (x, y) in the image; M represents the first neighborhood image W p The number of pixels in the image.

[0094] In the embodiment of the present application, the local brightness standard deviation is used to represent the degree of dispersion of the brightness values ​​of all pixels in the first neighborhood image relative to the local brightness average value, and the calculation formula is: Where W p Represents the first neighborhood image of feature point p; σ p The first neighborhood image W p The local brightness standard deviation; μ p Represents the first neighborhood image W p The local brightness average of I(x,y) represents the first neighborhood image W p The brightness value of the pixel (x, y) in the image; M represents the first neighborhood image W p The number of pixels in the image.

[0095] In the embodiment of the present application, the local eigenvalue is used to measure the texture complexity of the first neighborhood image, and can be set as the weighted sum of the local brightness average and the local brightness standard deviation. p +k p ·σ p , where k p is a constant that can be adjusted according to the characteristics of the local area.

[0096] In the embodiment of the present application, the first threshold is used to represent the overall texture complexity of the target image; the first threshold can be obtained based on the weighted sum of the global brightness mean and the global brightness standard deviation of the target image. If the weighted sum of the global brightness mean and the global brightness standard deviation of the target image is called t1 (i.e., the second threshold below), the first threshold can be set to k1*t1, where k1 can be selected based on actual conditions, for example, set to 1.5.

[0097] In an embodiment of the present application, determining the target interpolation algorithm for processing the feature point by comparing the local eigenvalue with a first threshold may include: if the local eigenvalue is greater than the first threshold, selecting a first interpolation algorithm with higher precision; otherwise, selecting a second interpolation algorithm with lower precision. The first interpolation algorithm and the second interpolation algorithm may be set based on actual conditions. For example, the first interpolation algorithm may be set to a bicubic interpolation algorithm, and the second interpolation algorithm may be set to a bilinear interpolation algorithm.

[0098] In an embodiment of the present application, determining the target interpolation algorithm for processing the feature point by comparing the local eigenvalue with a first threshold value may include: calculating a ratio or difference between the local eigenvalue t2 and the first threshold value t1; and selecting the target interpolation algorithm based on the ratio or difference. For example, when multiple candidate interpolation algorithms are provided, each candidate interpolation algorithm corresponds to a numerical interval. If the ratio or difference between the local eigenvalue t2 and the first threshold value t1 falls within a certain numerical interval, the target interpolation algorithm is set to the candidate interpolation algorithm corresponding to the numerical interval.

[0099] In an embodiment of the present application, for each of the feature points, the weighted sum of the local brightness average and the local brightness standard deviation of its first neighborhood image is calculated to obtain a local eigenvalue of the texture complexity of the first neighborhood image; the local eigenvalue is compared with a first threshold representing the texture complexity of the target image as a whole, thereby selecting a target interpolation algorithm for processing the feature point. Thus, based on the texture complexity at the feature point and the texture complexity of the target image as a whole, a target interpolation algorithm for processing the feature point is selected. At corners with more complex textures, detail blurring or aliasing introduced by low-order algorithms is avoided, and at places with slightly simpler textures, the computing resources required for the interpolation algorithm are reduced, and computing resources are preferentially allocated to corners with more complex textures, ultimately achieving a dynamic optimization balance between computing resource allocation and interpolation accuracy, while maintaining image quality while improving processing efficiency.

[0100] Optionally, detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image includes:

[0101] Calculating a structural tensor matrix of a target pixel point according to a second neighborhood image of the target pixel point; the target pixel point is any pixel point in the target image;

[0102] Calculating a corner point response value of the target pixel point according to the corner point response function and the structure tensor matrix;

[0103] When the corner point response value is not less than a second threshold, the target pixel point is marked as a feature point in the target image; the second threshold is determined based on the global statistics of the target image and is used to represent the overall texture complexity of the target image.

[0104] In an embodiment of the present application, the corner detection algorithm may include a Harris corner detection algorithm.

[0105] In existing technologies, the second threshold is typically set to a fixed value, which fails to account for variations in image characteristics. When the overall image texture is complex, this fixed threshold can lead to the detection of too many feature points, resulting in a large number of false corners and increased computational complexity in subsequent processing. On the other hand, when the image texture is simple, important corner features may be missed.

[0106] In this embodiment of the present application, the second threshold is determined based on the global statistics of the target image and is used to represent the overall texture complexity of the target image. When the overall texture complexity of the target image is relatively complex, the second threshold is automatically increased to more strictly filter feature points and avoid screening out too many feature points. When the overall texture of the target image is relatively simple, the second threshold is adaptively decreased to detect more feature points.

[0107] Specifically, the second threshold can be determined based on a global statistic such as the mean, standard deviation, or variance of the brightness of each pixel in the target image, or the standard deviation or variance of the brightness gradient. In another embodiment, the second threshold can be determined based on a histogram distribution of corner point response values, selecting a critical value that can distinguish corner points from non-corner points as the second threshold, for example, the minimum value of the top 5% high response values.

[0108] In this embodiment of the present application, the second neighborhood image is a neighborhood image of the target pixel. Specifically, it can be defined as a local pixel region within a preset size range centered on the feature point, used for analyzing texture complexity. Its size can be the same as or different from that of the first neighborhood image, without specific limitation.

[0109] In the embodiment of the present application, the structure tensor matrix represents the spatial correlation and anisotropy of the brightness gradient distribution within the second neighborhood window of the target pixel point, which can be calculated by the brightness gradient within the second neighborhood window of the target pixel point. The calculation formula of the structure tensor matrix M is:

[0110]

[0111] Where W is the second neighborhood image of the target pixel; (x, y) is the neighborhood pixel in the second neighborhood image; I x , I y are the brightness gradient values ​​of the neighboring pixel (x, y) in the second neighborhood image in the x and y directions, respectively. These can be calculated using operators such as the Sobel operator, Prewitt operator, or Scharr operator. w(x, y) is the weight value corresponding to each neighboring pixel (x, y) and can be set to a rectangular window function or a Gaussian function.

[0112] When w(x, y) is a rectangular window function, all neighboring pixels in the second neighborhood image are assigned the same weight, i.e., w(x, y) = 1 / M, where M is the number of neighboring pixels in the second neighborhood image. Pixels outside the second neighborhood image are assigned a weight of 0. This method is simple and straightforward, requiring minimal computation. However, it fails to consider the distance relationship between different pixels in the neighborhood and the target pixel, treating all pixels equally and potentially being affected by noise.

[0113] When w(x,y) uses Gaussian weighting, the weight coefficient decays according to the Euclidean distance from the center pixel, and its expression is: (x c ,y c ) are the coordinates of the target pixel; (x, y) are the coordinates of the neighboring pixel in the second neighboring image. σ is the standard deviation of the Gaussian kernel, which is used to control the weight decay rate. Gaussian weighting can suppress noise interference far from the center pixel and enhance the robustness of corner detection.

[0114] In the embodiment of the present application, the corner response function is defined as R=det(M)-k*trace(M) 2 ; In the formula, R is the corner point response value, det(M) refers to the determinant of the structure tensor matrix M, which represents the discrete degree of gradient distribution; trace(M) 2 It refers to the square of the trace of the matrix M, reflecting the concentration of the gradient amplitude; k is a constant, usually ranging from 0.04 to 0.06, which can be used to adjust the sensitivity threshold of corner point screening.

[0115] In the embodiment of the present application, the corner point response value is an indicator value used to measure whether the target pixel point is a corner point. The larger the value, the higher the possibility that the target pixel point is a corner point.

[0116] In the embodiment of the present application, the feature point is a target pixel point whose corner point response value is not less than a second threshold.

[0117] In this embodiment of the present application, the second threshold for filtering the corner point response values ​​is determined based on the global statistics of the target image. Specifically, the second threshold corresponding to the corner point response value is adapted to the overall texture complexity of the target image. When the overall image texture is complex, a higher second threshold can prevent excessive feature point detection, reducing the burden of subsequent processing. When the image texture is simple, a lower second threshold ensures that key corner points are not missed, ensuring that the feature point screening results are neither redundant nor missed, thereby improving the accuracy of feature point detection.

[0118] Furthermore, in order to achieve adaptive determination of the above-mentioned second threshold, the present application also provides a method for calculating the second threshold.

[0119] Optionally, the detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image further includes:

[0120] Obtaining a global brightness average and a global brightness standard deviation of the target image;

[0121] A weighted sum of the global brightness average value and the global brightness standard deviation is calculated to obtain the second threshold.

[0122] In an embodiment of the present application, the second threshold is determined based on the global brightness average and the global brightness standard deviation of the target image; specifically, the second threshold is the weighted sum of the global brightness average and the global brightness standard deviation.

[0123] The calculation formula of the second threshold can be set to: t1=μ+k2*σ; where μ represents the global brightness average of the target image; σ represents the global brightness standard deviation of the target image; k2 is a constant and can be adjusted according to actual needs.

[0124] In the embodiment of the present application, the calculation formula of the global brightness average value is: The calculation formula of the global brightness standard deviation is: Where I(i) is the brightness value of pixel i in the target image; N is the number of pixels in the target image.

[0125] In an embodiment of the present application, the weighted sum of the global brightness average and the global brightness standard deviation is used as the second threshold corresponding to the corner response value for screening; the global brightness average characterizes the overall brightness level of the image, and the global brightness standard deviation reflects the severity of the image brightness change. By combining the two in a weighted sum manner, the overall texture complexity of the image can be comprehensively evaluated. When the overall texture of the image is complex, a higher global brightness standard deviation will cause the second threshold to increase, thereby strictly screening out significant corner features and avoiding over-detection; and when the image texture is simple, a lower global brightness standard deviation will reduce the second threshold to fully capture potential corner information and prevent missed detection. This method effectively avoids the limitations of fixed thresholds in different image scenes, ensures that the feature point screening results are neither too redundant nor omitted, and provides a reliable feature point set for the subsequent selection of differential interpolation algorithms.

[0126] In existing techniques, the weight coefficients w(x,y) of the structure tensor matrix M are typically generated using rectangular window functions or Gaussian functions. While these methods can reflect the spatial distribution characteristics of pixels, they fail to fully consider the brightness correlation between neighboring pixels and the target pixel. Specifically, rectangular window weighting ignores the impact of brightness differences on texture representation, while Gaussian weighting assigns weights based solely on geometric distance. Both methods can lead to insufficient feature differentiation of corner response values.

[0127] To this end, this application further proposes a dynamic weight calculation method based on brightness difference.

[0128] Optionally, calculating the structure tensor matrix of the target pixel point based on the second neighborhood image of the target pixel point includes:

[0129] For each neighborhood pixel in the second neighborhood image, calculating a brightness gradient value at the neighborhood pixel;

[0130] Calculating a target matrix corresponding to the neighborhood pixel points according to the brightness gradient value at the neighborhood pixel points; the structure tensor matrix has the same shape as the target matrix;

[0131] Determining a first weight value corresponding to the neighborhood pixel point according to a brightness difference between the neighborhood pixel point and the target pixel point; the brightness difference between the neighborhood pixel point and the target pixel point is negatively correlated with the first weight value;

[0132] Calculating a structure tensor matrix of the target pixel point based on the first weight value of each of the neighborhood pixels and the target matrix;

[0133] Each element in the structure tensor matrix is ​​obtained by weighted summing the elements at the same position in the target matrix according to the first weight value of each of the neighborhood pixels.

[0134] In the embodiment of the present application, the brightness gradient value includes the brightness gradient value I of the neighborhood pixel point (x, y) in the x and y directions. x , I y .

[0135] In the embodiment of the present application, the target matrix is ​​a 2×2 symmetric matrix, and its specific form is:

[0136] In an embodiment of the present application, the first weight value is determined based on the brightness difference between the neighborhood pixel point and the target pixel point; the greater the difference between the two, the smaller the first weight value corresponding to the neighborhood pixel point; the smaller the difference between the two, the larger the first weight value corresponding to the neighborhood pixel point.

[0137] If the brightness of the neighboring pixels is close to that of the target pixel, the target matrix corresponding to these neighboring pixels with similar brightness will be included in the summation with a larger weight when calculating the structure tensor matrix M. Because these points have similar brightness to the target pixel, they are more likely to belong to the same structure or the same object, and their gradient information is more valuable for describing the local features of the target pixel. The larger weight allows the gradient information of these points to be more fully reflected in the structure tensor matrix, thereby enhancing the local feature representation of the target pixel.

[0138] Conversely, if the brightness of the neighboring pixels differs significantly from the target pixel, their weights are much less than 1. When calculating the structure tensor matrix M, the target matrix corresponding to these neighboring pixels with large brightness differences will be included in the summation with extremely small weights. These points with large brightness differences may be noise points or belong to different structures or objects, and their gradient information may interfere with the accurate description of the local features of the target pixel. By assigning smaller weights, the influence of these points on the structure tensor matrix can be suppressed, reducing the interference of noise and irrelevant information.

[0139] For a true corner point, there are usually multiple pixels in its neighborhood that have similar brightness to the target pixel and have different gradient directions. Since the adaptive weight increases the weight of these pixels with similar brightness, the structure tensor matrix det(M) can more accurately reflect the local characteristics of the corner point. At the same time, trace(M) 2 It will also reasonably reflect the strength of the local gradient. In summary, the corner response value R will increase, making it easier to be detected as a corner point.

[0140] In an embodiment of the present application, the first weight value corresponding to the neighborhood pixel point is determined based on the brightness difference between the neighborhood pixel point and the target pixel point, the first weight value corresponding to the neighborhood pixel point whose brightness is close to that of the target pixel point is increased, and the first weight value corresponding to the neighborhood pixel point whose brightness difference is large with that of the target pixel point is reduced; thereby, the corner point response value R at the true corner point is increased, the discrimination of the corner point response value is improved, the false detection and missed detection phenomena are reduced, and ultimately the accuracy of the corner point detection algorithm is improved.

[0141] Optionally, determining the first weight value corresponding to the neighborhood pixel point according to the brightness difference between the neighborhood pixel point and the target pixel point includes:

[0142] The first weight value corresponding to the neighborhood pixel point is calculated based on the following formula:

[0143]

[0144] Wherein, w(x,y) is the first weight value of the neighborhood pixel; I (x,y) is the brightness value of the neighborhood pixel; c is the brightness value of the target pixel; α is a preset coefficient.

[0145] In the embodiment of the present application, the preset coefficient is a constant used to adjust the sensitivity of the weight to the brightness difference, and is usually taken as 0.01 to 0.1. In the embodiment of the present application, if the brightness of the neighborhood pixel is close to that of the target pixel, its corresponding value is close to 1, and the target matrix corresponding to these neighborhood pixels with similar brightness will participate in the summation with a larger weight. On the contrary, if the brightness of the neighborhood pixel is significantly different from that of the target pixel, its weight is much less than 1. When calculating the structural tensor matrix M, the target matrix corresponding to these neighborhood pixels with large brightness differences will participate in the summation with a very small weight.

[0146] In the embodiment of the present application, the weighting function is improved from rectangular window function and Gaussian function to This allows the true corner features to be highlighted, while reducing false detections and missed detections, providing a more accurate feature point foundation for subsequent image processing tasks.

[0147] In practical applications, the corner points detected by the corner detection algorithm may have adjacent feature points or multiple repeated feature points in a small area. The aggregation of feature points interferes with image details.

[0148] Based on this, in the embodiment of the present application, a neural network model is used to further process the interpolation result image to eliminate the above defects.

[0149] Optionally, the method further includes:

[0150] The interpolation result image is segmented to obtain sub-images of the interpolation result image:

[0151] Inputting the sub-image into a preset neural network model to obtain a first prediction result; the first prediction result is used to represent second weight values ​​corresponding to various preset interpolation algorithms;

[0152] Based on the second weight values ​​respectively corresponding to the various preset interpolation algorithms, weighted summation is performed on the interpolation results obtained by performing interpolation processing on the sub-image using the various preset interpolation algorithms to obtain an optimized sub-image;

[0153] An optimized interpolation result image is obtained according to the optimized sub-image.

[0154] In the embodiment of the present application, the segmentation process is used to divide the original image into sub-image blocks of a specified size according to a grid, such as a regular area of ​​32*32 pixels.

[0155] In an embodiment of the present application, the sub-image is an image block obtained by segmenting the interpolation result image, and is used to input a preset neural network model for processing.

[0156] In an embodiment of the present application, the preset neural network model may be a convolutional neural network model, specifically adopting a lightweight CNN structure including a residual block and an attention mechanism.

[0157] In an embodiment of the present application, the preset interpolation algorithm includes at least two interpolation algorithms, which may specifically include the candidate interpolation algorithms mentioned above. The candidate interpolation algorithms may include, but are not limited to, nearest neighbor interpolation, linear interpolation, bilinear interpolation, bicubic interpolation, Lagrange interpolation, Newton interpolation, Hermite interpolation, cubic spline interpolation, GPU-based KRIGING interpolation, etc.

[0158] In an embodiment of the present application, the first prediction result is used to represent the second weight value corresponding to each of the preset interpolation algorithms. The specific form can be a normalized weight vector, and one element in the vector is used for the second weight value of a preset interpolation algorithm.

[0159] In the embodiment of the present application, the magnitude of the second weight value is negatively correlated with the prediction error of the preset interpolation algorithm within the sub-image area.

[0160] In an embodiment of the present application, each pixel value of the optimized sub-image is a weighted sum of the processing results of each preset interpolation algorithm at the corresponding position, and the weight is dynamically allocated by the second weight value.

[0161] In one embodiment of the present application, if the number of feature points in a sub-image is greater than a preset threshold, the sub-image is input into a preset neural network model for processing. In another embodiment, all sub-images are input into a preset neural network model for processing.

[0162] In the embodiment of the present application, the reconstruction results of the sub-images are spliced ​​and reassembled according to the spatial positions at the time of segmentation to generate a complete optimized interpolation result image.

[0163] In an embodiment of the present application, the interpolation result image is divided into sub-images and input into a preset neural network model, the complex feature distribution of the sub-images is learned by the preset neural network model, and the second weight value of each preset interpolation algorithm is adaptively generated based on the local features of the image, so as to identify the applicability of different interpolation algorithms in local areas, thereby achieving fine adjustment of the local structure of the feature point clustering area, so that the optimized sub-image can still maintain clear edges and details in the feature point dense area, effectively avoiding blurring or interference of image details caused by feature point clustering, and ultimately improving the overall quality and visual effect of the optimized interpolation result image.

[0164] Optionally, before inputting the sub-image into a preset neural network model, the method further includes:

[0165] Obtaining a training sample set including a plurality of sample picture groups and first label data corresponding to the sample picture groups; the sample picture groups including a first sample image and a second sample image obtained by performing image degradation processing on the sample image; the first label data being used to represent target weight values ​​corresponding to each of the preset interpolation algorithms; performing a weighted summation of interpolation results of the sub-images obtained by the various preset interpolation algorithms based on the target weight values, and obtaining a minimum error relative to the first sample image;

[0166] The initial model is trained based on the training sample set, with the goal of minimizing the value of the loss function, and the model parameters of the initial model are adjusted to obtain a preset neural network model; the loss function is used to represent the difference between the first prediction result output by the initial model and the first label data.

[0167] In an embodiment of the present application, the sample picture group includes a first sample image and a second sample image obtained by performing image degradation processing on the sample image.

[0168] In an embodiment of the present application, the first sample image is an undegraded, high-definition image containing complete texture details and spatial distribution characteristics. The second sample image is generated by applying a degradation process to the first sample image. The degradation process is used to simulate image degradation processes in actual applications, including but not limited to: one or more of: noise injection and blurring. The degradation process is adapted to the use scenario of the interpolation algorithm; for example, if the interpolation algorithm is used to stitch several bird's-eye views into a ring image, the degradation process is used to simulate defects such as light spots and streaks in the ring image.

[0169] In an embodiment of the present application, the target weight value is a weight value calculated by an optimization algorithm, which satisfies the following conditions: when the target weight value is used to perform weighted fusion on the processing results of each preset interpolation algorithm, the mean square error between the weighted summation result and the first sample image reaches a minimum value.

[0170] In this embodiment of the present application, the first label data is a vector of the same dimension as the first prediction result, whose elements correspond one-to-one to a preset interpolation algorithm, with each element value representing the optimal weight distribution ratio for the corresponding interpolation algorithm in the sub-image region. During the label generation process, the error contribution of each interpolation algorithm on the degraded image block can be solved using the least squares method, ultimately obtaining a set of weight values ​​that minimizes the weighted fusion error.

[0171] In an embodiment of the present application, the loss function is used to represent the difference between the first prediction result output by the initial model and the first label data, and is specifically calculated using a cross-entropy loss function. The cross-entropy loss function quantifies the degree of deviation between the initial model output and the label data by measuring the difference between the predicted second weight value distribution of each preset interpolation algorithm and the target weight value distribution in the first label data, providing an optimization direction for model parameter adjustment.

[0172] In an embodiment of the present application, the model parameters of the initial model are adjusted with the goal of minimizing the value of the loss function, including: inputting the second sample image into the initial model, generating a second weight value prediction result corresponding to each preset interpolation algorithm through a forward propagation process; based on the one-to-one correspondence between the first prediction result and the target weight value in the first label data, using the cross entropy loss function to calculate the loss function value output by the current model; through the back propagation algorithm, the gradient of the loss function is transferred layer by layer to the parameters of each layer of the initial model, and the model parameters are iteratively updated according to the gradient descent rule, gradually reducing the loss function value until the loss function converges to a preset threshold or reaches the maximum number of training rounds, and finally obtaining a preset neural network model that can accurately predict the target weight value.

[0173] In an embodiment of the present application, by constructing a training sample set containing a sample image group and corresponding first label data, a preset neural network model can be trained for image degradation scenarios that may occur in actual applications, thereby accurately capturing the error characteristics and weight distribution patterns of different interpolation algorithms in various image regions. The model training process based on the cross-entropy loss function effectively adjusts the initial model parameters through an iterative optimization mechanism of forward propagation, loss calculation, and backpropagation, enabling the preset neural network model to adaptively output the optimal second weight value based on the local features of the sub-image, ensuring that the error between the weighted summed interpolation result and the original high-definition image is minimized.

[0174] Optionally, inputting the sub-image into a preset neural network model to obtain a first prediction result includes:

[0175] The sub-image is input into the preset neural network model to obtain the first prediction result and the second prediction result; the second prediction result is used to represent the prediction errors corresponding to the various preset interpolation algorithms respectively; the prediction error corresponding to the preset interpolation algorithm is used to represent the error of the interpolation result obtained by interpolating the sub-image based on the preset interpolation algorithm.

[0176] In an embodiment of the present application, the preset neural network model adopts a multi-task learning architecture and simultaneously outputs a first prediction result of the interpolation algorithm weight allocation and a second prediction result of the algorithm performance evaluation.

[0177] In an embodiment of the present application, the second prediction result is used to represent the prediction errors corresponding to various preset interpolation algorithms, and is specifically in the form of a feature vector with a dimension equal to the number of interpolation algorithms, and each element in the vector corresponds to the prediction error of a preset interpolation algorithm in the sub-image area.

[0178] In an embodiment of the present application, the prediction error corresponding to the preset interpolation algorithm is used to represent the error of the interpolation result obtained by interpolating the sub-image based on the preset interpolation algorithm, and is used to quantify the degree of deviation between the interpolation result of the preset interpolation algorithm in the sub-image area and the true value.

[0179] In this embodiment of the present application, the preset interpolation algorithm may include a target interpolation algorithm; correspondingly, the second prediction result includes a prediction error of the target interpolation algorithm on the sub-image. When the error value exceeds a preset threshold, the system automatically switches to a multi-algorithm fusion mode: based on the second weight values ​​corresponding to each of the preset interpolation algorithms, the interpolation results obtained by the various preset interpolation algorithms on the sub-image are weighted summed to obtain an optimized sub-image.

[0180] In this embodiment of the present application, the preset neural network model employs a multi-task learning architecture, simultaneously outputting a first prediction result for the interpolation algorithm weight assignment and a second prediction result for the interpolation algorithm performance evaluation. By quantitatively evaluating the local errors of each preset interpolation algorithm, objective data support can be provided for the assignment of the second weight value, effectively improving the accuracy of the weight assignment strategy and thus significantly enhancing the reliability and accuracy of the optimized sub-images. Outputting the second prediction result also enhances the interpretability of the first prediction result.

[0181] Optionally, the method further includes:

[0182] Obtaining second label data corresponding to the sample image group; the second label data is used to represent an error of an interpolated image obtained by interpolating the second sample image based on various preset interpolation algorithms relative to the first sample image;

[0183] The preset neural network model is also used to output a second prediction result; the second prediction result is used to represent the prediction errors corresponding to the various preset interpolation algorithms respectively; the loss function is used to represent the weighted sum of the first loss and the second loss; the first loss is used to represent the difference between the first prediction result output by the initial model and the first label data; the second loss is used to represent the difference between the second prediction result output by the initial model and the second label data.

[0184] In an embodiment of the present application, the second label data is generated by calculating the pixel-level difference between the result of interpolating the second sample image using each preset interpolation algorithm and the first sample image. Specifically, the interpolation deviation can be measured using perceptual loss or mean square error (MSE). The second label data is a feature vector equal to the number of preset interpolation algorithms, where each element represents the error of a preset interpolation algorithm.

[0185] In an embodiment of the present application, the loss function is used to represent the weighted sum of the first loss and the second loss. By balancing the optimization objectives of the two tasks, the preset neural network model can simultaneously improve the prediction accuracy of the second weight value and the prediction error.

[0186] In an embodiment of the present application, the first loss is used to measure the difference between the first prediction result (second weight value) output by the initial model and the first label data (target weight value).

[0187] In an embodiment of the present application, the second loss is used to measure the deviation between the second prediction result (prediction error) output by the initial model and the second label data (actual interpolation error).

[0188] In this embodiment of the present application, the calculation formula of the loss function can be:

[0189]

[0190] Where, It is used to represent the second prediction result output by the initial model, that is, the prediction error corresponding to various preset interpolation algorithms; error Used to represent the second tag data; It is used to represent the first prediction result output by the initial model, that is, the second weight value corresponding to each of the preset interpolation algorithms; weights It is used to represent the first label data, that is, the target weight values ​​corresponding to various preset interpolation algorithms. MSE() is the mean squared error function, which is used to calculate the regression loss between the predicted error and the actual error. CrossEntropy() is the cross entropy loss function, which is used to calculate the classification loss between the predicted weight value and the target weight value. λ1 and λ2 are balancing hyperparameters used to adjust the weight of the second loss and the first loss in the total loss, adjusting the optimization intensity of the weight prediction task and the error prediction task. They can be set according to the actual situation, such as λ1 = 1.0 and λ2 = 0.5.

[0191] In an embodiment of the present application, the first loss ensures that the second weight value output by the model is close to the target weight value that minimizes the weighted fusion error, and the second loss constrains the model's prediction results of the interpolation errors of each algorithm to be consistent with the actual error. The synergistic effect of the two prompts the model to generate a more reasonable weight allocation strategy and error evaluation results. Therefore, by introducing the second label data and constructing a weighted sum loss function that includes the first loss and the second loss, the preset neural network model can simultaneously optimize the prediction ability of the second weight value and the interpolation error during the training process. The dual-task loss function mechanism avoids the model prediction deviation that may be caused by single-task training by balancing the optimization objectives of the two key tasks of weight allocation and error evaluation, so that the model can more accurately capture the actual performance of different preset interpolation algorithms in the sub-image area.

[0192] Exemplary devices

[0193] The device embodiments of this application can be used to execute the method embodiments of this application. For details not disclosed in the device embodiments of this application, please refer to the method embodiments of this application.

[0194] Figure 2 FIG. 1 is a block diagram of an image processing device provided by an embodiment of the present application. Figure 2 As shown, the device 200 includes:

[0195] An acquisition module 210 is used to acquire a target image to be interpolated;

[0196] A detection module 220 is configured to detect corner points in the target image based on a corner detection algorithm to obtain feature points in the target image;

[0197] A selection module 230 is configured to select, for each feature point, a target interpolation algorithm for processing the feature point based on the texture complexity of a first neighborhood image; the first neighborhood image is a neighborhood image of the feature point; and the calculation accuracy of the target interpolation algorithm is positively correlated with the texture complexity.

[0198] The interpolation module 240 is configured to perform interpolation processing on the feature points based on a target interpolation algorithm to generate an interpolation result image.

[0199] Optionally, the selection module 230 is configured to:

[0200] Calculating a local brightness average and a local brightness standard deviation of the first neighborhood image;

[0201] Calculating a weighted sum of the local brightness average and the local brightness standard deviation to obtain a local eigenvalue for texture complexity of the first neighborhood image;

[0202] By comparing the local eigenvalue with a first threshold, a target interpolation algorithm for processing the feature point is determined; the first threshold is determined based on a global statistic of the target image and is used to represent the overall texture complexity of the target image.

[0203] Optionally, the corner detection algorithm includes a Harris corner detection algorithm; and the detection module 220 includes:

[0204] A structure tensor matrix unit, configured to calculate a structure tensor matrix of a target pixel point based on a second neighborhood image of the target pixel point; the target pixel point is any pixel point in the target image;

[0205] A response value calculation unit, configured to calculate a corner point response value of the target pixel point according to a corner point response function and the structure tensor matrix;

[0206] A screening unit is used to mark the target pixel point as a feature point in the target image when the corner point response value is not less than a second threshold; the second threshold is determined based on the global statistics of the target image and is used to represent the overall texture complexity of the target image.

[0207] Optionally, the detection module 220 further includes:

[0208] A statistical unit, configured to obtain a global brightness average and a global brightness standard deviation of the target image;

[0209] A calculation unit is used to calculate a weighted sum of the global brightness average value and the global brightness standard deviation to obtain the second threshold.

[0210] Optionally, the structure tensor matrix unit includes:

[0211] a gradient calculation subunit, configured to calculate, for each neighborhood pixel in the second neighborhood image, a brightness gradient value at the neighborhood pixel;

[0212] A matrix calculation subunit, configured to calculate a target matrix corresponding to the neighborhood pixel points according to the brightness gradient values ​​at the neighborhood pixel points; the structure tensor matrix has the same shape as the target matrix;

[0213] a weight calculation subunit, configured to determine a first weight value corresponding to the neighborhood pixel point based on a brightness difference between the neighborhood pixel point and the target pixel point; the brightness difference between the neighborhood pixel point and the target pixel point is negatively correlated with the first weight value;

[0214] A tensor calculation subunit is configured to calculate a structure tensor matrix of the target pixel point based on the first weight value of each of the neighborhood pixel points and the target matrix;

[0215] Each element in the structure tensor matrix is ​​obtained by weighted summing the elements at the same position in the target matrix according to the first weight value of each of the neighborhood pixels.

[0216] Optionally, the weight calculation subunit is used to:

[0217] The first weight value corresponding to the neighborhood pixel point is calculated based on the following formula:

[0218]

[0219] Wherein, w(x,y) is the first weight value of the neighborhood pixel; I (x,y) is the brightness value of the neighborhood pixel; c is the brightness value of the target pixel; α is a preset coefficient.

[0220] Optionally, the apparatus 200 further includes:

[0221] A segmentation unit is used to segment the interpolation result image to obtain sub-images of the interpolation result image:

[0222] A neural network unit is configured to input the sub-image into a preset neural network model to obtain a first prediction result; the first prediction result is used to represent second weight values ​​corresponding to various preset interpolation algorithms;

[0223] a weighted calculation unit, configured to perform weighted summation on interpolation results obtained by performing interpolation processing on the sub-images using the various preset interpolation algorithms based on second weight values ​​respectively corresponding to the various preset interpolation algorithms, to obtain an optimized sub-image;

[0224] The splicing unit is used to obtain an optimized interpolation result image according to the optimized sub-image.

[0225] Optionally, before inputting the sub-image into a preset neural network model, the method further includes:

[0226] A sample acquisition unit is configured to acquire a training sample set comprising a plurality of sample picture groups and first label data corresponding to the sample picture groups; the sample picture groups comprising a first sample image and a second sample image obtained by performing image degradation processing on the sample image; the first label data being used to represent target weight values ​​corresponding to each of the preset interpolation algorithms; and performing a weighted summation of interpolation results of the sub-images by the various preset interpolation algorithms based on the target weight values, thereby minimizing the error relative to the first sample image.

[0227] A model training unit is used to train the initial model based on the training sample set, adjust the model parameters of the initial model with the goal of minimizing the value of the loss function, and obtain a preset neural network model; the loss function is used to represent the difference between the first prediction result output by the initial model and the first label data.

[0228] Optionally, the neural network unit is used to:

[0229] The sub-image is input into the preset neural network model to obtain the first prediction result and the second prediction result; the second prediction result is used to represent the prediction errors corresponding to the various preset interpolation algorithms respectively; the prediction error corresponding to the preset interpolation algorithm is used to represent the error of the interpolation result obtained by interpolating the sub-image based on the preset interpolation algorithm.

[0230] Optionally, the device sample acquisition unit is further configured to:

[0231] Obtaining second label data corresponding to the sample image group; the second label data is used to represent an error of an interpolated image obtained by interpolating the second sample image based on various preset interpolation algorithms relative to the first sample image;

[0232] The preset neural network model is also used to output a second prediction result; the second prediction result is used to represent the prediction errors corresponding to the various preset interpolation algorithms respectively; the loss function is used to represent the weighted sum of the first loss and the second loss; the first loss is used to represent the difference between the first prediction result output by the initial model and the first label data; the second loss is used to represent the difference between the second prediction result output by the initial model and the second label data.

[0233] Exemplary electronic devices

[0234] Below, reference Figure 3 To describe the electronic device according to the embodiment of the present application. Figure 3 The figure shows a block diagram of an electronic device according to an embodiment of the present application.

[0235] like Figure 3 As shown, electronic device 300 includes one or more processors 310 and memory 320 .

[0236] The processor 310 may have other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 300 to perform desired functions.

[0237] Specifically, the processor 310 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the solution of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor 310 may also include a main processor, a baseband chip, a modem, etc.

[0238] The memory 320 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 310 may execute the program instructions to implement the image processing methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as category correspondences may also be stored in the computer-readable storage medium.

[0239] In one example, the electronic device 300 may further include an input device 330 and an output device 340 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0240] In addition, the input device 330 may also be a device that receives data and information input by the user, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor. The output device 340 may output various information to the outside. The output device 340 may include, for example, a display, speaker, printer, communication network and its connected remote output device, etc.

[0241] Of course, to simplify, Figure 3 Only some of the components related to the present application in the electronic device 300 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 300 may further include any other appropriate components according to specific application scenarios.

[0242] Exemplary computer program products and computer-readable storage media

[0243] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the image processing method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0244] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0245] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the image processing method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0246] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0247] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0248] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0249] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0250] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0251] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0252] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0253] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0254] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0255] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0256] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0257] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0258] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: include: Obtain the target image to be interpolated; Detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image; For each of the feature points, a target interpolation algorithm for processing the feature point is selected based on the texture complexity of a first neighborhood image; the first neighborhood image is a neighborhood image of the feature point; and the calculation accuracy of the target difference algorithm is positively correlated with the texture complexity; Based on the target interpolation algorithm, the feature points are interpolated to generate an interpolation result image.

2. The method according to claim 1, characterized in that The step of selecting, for each of the feature points, a target interpolation algorithm for processing the feature point according to the texture complexity of the first neighborhood image comprises: Calculating a local brightness average and a local brightness standard deviation of the first neighborhood image; Calculating a weighted sum of the local brightness average and the local brightness standard deviation to obtain a local eigenvalue for representing the texture complexity of the first neighborhood image; By comparing the local eigenvalue with a first threshold, a target interpolation algorithm for processing the feature point is determined; the first threshold is determined based on a global statistic of the target image and is used to represent the overall texture complexity of the target image.

3. The method according to claim 1, characterized in that The detecting corner points in the target image based on a corner detection algorithm to obtain feature points in the target image includes: Calculating a structural tensor matrix of a target pixel point according to a second neighborhood image of the target pixel point; the target pixel point is any pixel point in the target image; Calculating a corner point response value of the target pixel point according to the corner point response function and the structure tensor matrix; When the corner point response value is not less than a second threshold, the target pixel point is marked as a feature point in the target image; the second threshold is determined based on the global statistics of the target image and is used to represent the overall texture complexity of the target image.

4. The method according to claim 3, characterized in that The method further comprises: Obtaining a global brightness average and a global brightness standard deviation of the target image; A weighted sum of the global brightness average value and the global brightness standard deviation is calculated to obtain the second threshold.

5. The method according to claim 3, characterized in that The step of calculating a structure tensor matrix of the target pixel point based on the second neighborhood image of the target pixel point includes: For each neighborhood pixel in the second neighborhood image, calculating a brightness gradient value at the neighborhood pixel; Calculating a target matrix corresponding to the neighborhood pixel points according to the brightness gradient value at the neighborhood pixel points; the structure tensor matrix has the same shape as the target matrix; Determining a first weight value corresponding to the neighborhood pixel point according to a brightness difference between the neighborhood pixel point and the target pixel point; the brightness difference between the neighborhood pixel point and the target pixel point is negatively correlated with the first weight value; Calculating a structure tensor matrix of the target pixel point based on the first weight value of each of the neighborhood pixels and the target matrix; Each element in the structure tensor matrix is ​​obtained by weighted summing the elements at the same position in the target matrix according to the first weight value of each of the neighborhood pixels.

6. The method according to claim 1, characterized in that The method further comprises: The interpolation result image is segmented to obtain sub-images of the interpolation result image: Inputting the sub-image into a preset neural network model to obtain a first prediction result; the first prediction result is used to represent second weight values ​​corresponding to various preset interpolation algorithms; Based on the second weight values ​​respectively corresponding to the various preset interpolation algorithms, weighted summation is performed on the interpolation results obtained by performing interpolation processing on the sub-image using the various preset interpolation algorithms to obtain an optimized sub-image; An optimized interpolation result image is obtained according to the optimized sub-image.

7. The method according to claim 6, characterized in that The step of inputting the sub-image into a preset neural network model to obtain a first prediction result includes: The sub-image is input into the preset neural network model to obtain the first prediction result and the second prediction result; the second prediction result is used to represent the prediction errors corresponding to the various preset interpolation algorithms respectively; the prediction error corresponding to the preset interpolation algorithm is used to represent the error of the interpolation result obtained by interpolating the sub-image based on the preset interpolation algorithm.

8. An image processing device, characterized in that: include: An acquisition module, used for acquiring a target image to be interpolated; A detection module, configured to detect corner points in the target image based on a corner detection algorithm to obtain feature points in the target image; a selection module configured to select, for each feature point, a target interpolation algorithm for processing the feature point based on the texture complexity of a first neighborhood image; the first neighborhood image being a neighborhood image of the feature point; and a calculation accuracy of the target interpolation algorithm being positively correlated with the texture complexity; The interpolation module is used to perform interpolation processing on the feature points based on a target interpolation algorithm to generate an interpolation result image.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images