Image processing method and device, storage medium and electronic equipment

Abstract

The invention provides an image processing method and device, a storage medium and electronic equipment, and the method comprises the steps: determining a to-be-processed first image and filtering kernel parameters corresponding to a plurality of scales during image processing; based on the first image and the filtering kernel parameters respectively corresponding to the plurality of scales, determining sharpening gain information respectively corresponding to the plurality of scales; and then based on the sharpening gain information corresponding to the plurality of scales, performing enhancement processing on the first image to obtain an enhanced second image. According to the image sharpening method and device, the first image is filtered through the filtering kernel parameters of the multiple scales, and the sharpening gain information of the multiple scales can be obtained, so that textures and edges of the first image under different scales can be enhanced, the image sharpening effect is more hierarchical, and naked eye perception and machine visual perception are better facilitated.

Description

Technical Field

[0001] This disclosure relates to image processing technology, and in particular to an image processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Image edge enhancement is an important image enhancement technique in Image Signal Processing (ISP). It primarily enhances image details and edges to avoid blurring caused by noise reduction and improve image clarity. The Unsharp Mask (USM) algorithm is a major component in this field, extracting and enhancing high-frequency information to achieve image sharpening. However, the USM algorithm has a limited scale of enhancement, resulting in relatively poor sharpening effects. Summary of the Invention

[0003] This disclosure provides an image processing method to enhance texture and edges at different scales, making the image sharpening effect more layered and more conducive to visual perception and machine vision perception.

[0004] A first aspect of this disclosure provides an image processing method, comprising: determining a first image to be processed, and filtering kernel parameters corresponding to multiple scales respectively; determining sharpening gain information corresponding to the multiple scales respectively based on the first image and the filtering kernel parameters corresponding to the multiple scales respectively; and performing enhancement processing on the first image based on the sharpening gain information corresponding to the multiple scales respectively to obtain an enhanced second image.

[0005] A second aspect of this disclosure provides an image processing apparatus, comprising: a determining module, configured to determine a first image to be processed and filter kernel parameters corresponding to a plurality of scales respectively; a first processing module, configured to determine sharpening gain information corresponding to the plurality of scales respectively based on the first image and the filter kernel parameters corresponding to the plurality of scales respectively; and a second processing module, configured to perform enhancement processing on the first image based on the sharpening gain information corresponding to the plurality of scales respectively, to obtain an enhanced second image.

[0006] A third aspect of this disclosure is to provide a computer-readable storage medium storing a computer program that is executed by a processor to perform the image processing method described in any of the above embodiments of this disclosure.

[0007] A fourth aspect of this disclosure provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory, the processor executing the executable instructions to implement the image processing method described in any of the above embodiments of this disclosure.

[0008] A fifth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs the image processing method provided in any of the above embodiments of this disclosure.

[0009] The image processing method, apparatus, storage medium, and electronic device provided in the above embodiments of this disclosure can determine a first image to be processed and filter kernel parameters corresponding to multiple scales during image processing; then, based on the first image and the filter kernel parameters corresponding to multiple scales, determine sharpening gain information corresponding to multiple scales; and then, based on the sharpening gain information corresponding to multiple scales, perform enhancement processing on the first image to obtain an enhanced second image. This disclosure filters the first image using filter kernel parameters at multiple scales, thereby obtaining sharpening gain information at multiple scales. This allows for enhancement of textures and edges at different scales of the first image, making the image sharpening effect more layered and more conducive to visual perception and machine vision. Attached Figure Description

[0010] Figure 1 This is an exemplary application scenario of the image processing method provided in this disclosure;

[0011] Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure;

[0012] Figure 3 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure;

[0013] Figure 4 This is a schematic flowchart of an image processing method provided in yet another exemplary embodiment of this disclosure;

[0014] Figure 5 This is a schematic diagram illustrating the principle of determining the filter result according to an exemplary embodiment of the present disclosure;

[0015] Figure 6 This is a schematic diagram illustrating the principle of determining the filter sub-result provided in another exemplary embodiment of this disclosure;

[0016] Figure 7 This is a schematic diagram of filter kernel parameters provided in an exemplary embodiment of this disclosure;

[0017] Figure 8 This is a schematic flowchart of an image processing method provided in yet another exemplary embodiment of this disclosure;

[0018] Figure 9 This is a schematic flowchart of an image processing method provided in yet another exemplary embodiment of this disclosure;

[0019] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure;

[0020] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of the present disclosure;

[0021] Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in yet another exemplary embodiment of the present disclosure;

[0022] Figure 13 This is a structural diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0024] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0025] Application Overview

[0026] In developing this disclosure, the inventors discovered that image edge enhancement is an important image enhancement technique in Image Signal Processing (ISP). Image edge enhancement primarily enhances image details, textures, and edges to avoid blurring caused by noise reduction and improve image clarity. Related technologies mainly employ the Unsharp Mask (USM) algorithm, which extracts and enhances high-frequency information in the image to achieve a sharpening effect. However, the USM algorithm has a single enhancement scale, resulting in poor sharpening effects.

[0027] Exemplary Overview

[0028] Figure 1 This is an exemplary application scenario of the image processing method provided in this disclosure. For example... Figure 1As shown, in an intelligent driving scenario, vehicle 11 can perceive its surrounding environment through various sensors, providing effective perception results for path planning and control. Sensors include, but are not limited to, one or more of the following: cameras, LiDAR, millimeter-wave radar, and ultrasonic radar. Cameras include, but are not limited to, […]. Figure 1 The forward-facing camera 12 is used in the vehicle. Images captured by the camera under low-light conditions such as rain, fog, and nighttime often suffer from blurriness. The image processing method provided in this embodiment can enhance the edges of the image, effectively restoring the clarity of lane lines, pedestrians, and other surrounding vehicles (as shown in Figures 13 and 14), thereby improving the recognition capabilities of the Advanced Driver Assistance System (ADAS) of the vehicle 11. Specifically, a first image to be processed and filter kernel parameters corresponding to multiple scales can be determined; based on the first image and the filter kernel parameters corresponding to multiple scales, sharpening gain information corresponding to multiple scales is determined; based on the sharpening gain information corresponding to multiple scales, the first image is enhanced to obtain an enhanced second image.

[0029] The image processing method disclosed herein filters a first image using filter kernel parameters at multiple scales, thereby obtaining sharpening gain information at multiple scales. This enables the enhancement of textures and edges at different scales of the first image, resulting in a more layered image sharpening effect that is more conducive to visual perception and machine vision.

[0030] It should be noted that the image processing method disclosed herein is not limited to intelligent driving scenarios, but can also be applied to any other application scenarios that require image processing, such as, but not limited to, medical image processing scenarios, industrial visual inspection scenarios, security and monitoring scenarios, photography and digital image processing scenarios, etc.

[0031] Exemplary methods

[0032] Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of the image processing method provided in this disclosure. The image processing method provided in this embodiment can be applied to electronic devices and chips. The electronic devices include, but are not limited to, in-vehicle computing platforms (or in-vehicle terminals), embody intelligent agents, mobile phones, tablets, wearable devices, etc., and the chips include, but are not limited to, intelligent driving chips, intelligent cockpit chips, cockpit-driver integrated chip chips, and chips in other electronic devices, such as… Figure 2 As shown, the image processing method of this disclosure embodiment may include the following steps:

[0033] Step 210: Determine the first image to be processed, and the filter kernel parameters corresponding to multiple scales.

[0034] Here, the first image is any image to be sharpened. Optionally, the first image can be an image captured by a camera, or it can be an intermediate image that has undergone other processing during image signal processing. Multiple scales refer to multiple scales of different sizes.

[0035] In some optional embodiments, multiple scales are used, for example, but not limited to, 3x3, 5x5, 7x7, 9x9, etc. The filter kernel parameters are weight parameters used to filter the first image. The filter kernel parameters are, for example, but not limited to, one or more of the following: Laplacian operator, Gaussian operator, Sobel operator, Laplacian Gaussian operator, mean filter operator, etc.

[0036] In some optional embodiments, the filter kernel parameters corresponding to multiple scales may include multiple kernel weights of different sizes. For example, 3*3 kernel weights, 5*5 kernel weights, 7*7 kernel weights, etc.

[0037] Optionally, the filter kernel parameters at multiple scales can be pre-set preset parameters, or they can be dynamically determined based on certain influencing factors. For example, in intelligent driving scenarios, the filter kernel parameters can be dynamically determined based on one or more factors such as the weather (e.g., rainy, foggy, sunny) and vehicle speed. For instance, different mapping relationships between factors and filter kernel parameters can be pre-set, and during vehicle operation, the current filter kernel parameters can be determined based on the real-time status of the corresponding factors through the mapping relationship. The specific method for determining the filter kernel parameters is not limited.

[0038] Step 220: Based on the first image and the filter kernel parameters corresponding to multiple scales, determine the sharpening gain information corresponding to multiple scales.

[0039] In some optional embodiments, the sharpening gain information is gain information used to enhance the texture and edges of the first image. Sharpening gain information corresponding to multiple scales can be used to enhance the texture and edges of the first image at different scales.

[0040] Optionally, for each of the multiple scales, the first image can be processed based on the filter kernel parameters corresponding to that scale and pre-configured rules for determining sharpening gain information to determine the sharpening gain information at that scale. The rules for determining the sharpening gain information may include, but are not limited to, filtering, function mapping, etc.; or the rules for determining the sharpening gain information may include, but are not limited to, filtering, lookup tables, etc.

[0041] Step 230: Based on the sharpening gain information corresponding to multiple scales, the first image is enhanced to obtain the enhanced second image.

[0042] This process involves fusing the first image with sharpening gain information corresponding to multiple scales to enhance the edges of the first image and obtain the second image. The fusing method can be set according to actual needs.

[0043] The image processing method provided in this disclosure can determine a first image to be processed and filter kernel parameters corresponding to multiple scales during image processing; then, based on the first image and the filter kernel parameters corresponding to multiple scales, determine sharpening gain information corresponding to multiple scales; and then, based on the sharpening gain information corresponding to multiple scales, perform enhancement processing on the first image to obtain an enhanced second image. This disclosure filters the first image using filter kernel parameters at multiple scales, thereby obtaining sharpening gain information at multiple scales. This allows for enhancement of textures and edges at different scales of the first image, making the image sharpening effect more layered and more conducive to human visual perception and machine vision perception.

[0044] In some optional embodiments, the filter kernel parameters corresponding to multiple scales may include kernel weights of a preset size and multiple sampling intervals. The preset kernel weights may be, for example, 3x3 kernel weights or kernel weights of other sizes. Taking a 3x3 kernel weight as an example, a 3x3 kernel weight includes 9 valid weight values. The multiple sampling intervals are used to expand the preset kernel weights to kernel weights of multiple scales. For example, if the sampling interval is 0, it represents the original 3x3 kernel weight; if the sampling interval is 1, the 3x3 kernel weight is expanded to a 5x5 kernel weight; if the sampling interval is 2, the 3x3 kernel weight is expanded to a 7x7 kernel weight. The 5x5 and 7x7 kernel weights include the 9 valid weight values ​​from the preset kernel weights according to the corresponding sampling intervals, and the other expanded portions are invalid weight values, such as 0 or other values ​​indicating invalidity. Alternatively, multiple sampling intervals can be used during filtering to sample a number of pixel values ​​corresponding to a preset kernel weight within a region centered on the pixel in the first image, achieving the same filtering effect as the extended multi-scale kernel weights described above. For example, taking a 3*3 kernel weight and a sampling interval of 1, for pixel a11 in the first row and first column of the first image, 9 pixel values ​​are sampled at the sampling interval within a region centered on pixel a11. This is equivalent to sampling the pixel values ​​of the four corner pixels, the center pixels of the four sides, and the 9 pixels of pixel a11 within a 5*5 region centered on pixel a11. The 9 sampled pixel values ​​are then weighted using the 9 effective weight values ​​in the 3*3 kernel weight to achieve filtering, thereby extracting information of different scales from the first image. Since a11 is the pixel in the upper left corner of the first image, there are actually no pixels to its left and top, which can be satisfied by padding to meet the pixel sampling requirements of a11.

[0045] Figure 3 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure.

[0046] In some alternative embodiments, in the above... Figure 2 Based on the illustrated embodiments, as Figure 3 As shown, step 220, which determines the sharpening gain information corresponding to multiple scales based on the first image and the filter kernel parameters corresponding to multiple scales, may include:

[0047] Step 2210: For any first scale among multiple scales, determine the filtering result corresponding to the first scale based on the first image and the filtering kernel parameters corresponding to the first scale.

[0048] In this process, any one of the multiple scales is taken as the first scale, and the first image is filtered based on the filter kernel parameters corresponding to the first scale to obtain the filtering result corresponding to the first scale.

[0049] Optionally, the filter kernel parameters corresponding to the first scale may include kernel weights of the size corresponding to the first scale, or the filter kernel parameters corresponding to the first scale may include kernel weights of a preset size and the sampling interval corresponding to the first scale.

[0050] Optionally, the filtering result can be a feature map of the same size as the first image. Taking a 5*5 kernel weight as an example, the filtering result corresponding to the first scale can be obtained by performing a convolution operation with a stride of 1 on the first image based on the 5*5 kernel weight. To keep the size of the filtering result the same as the first image, the first image can be padded before performing the convolution operation.

[0051] Step 2220: Based on the filtering results corresponding to the first scale and the pre-configured gain mapping rules, determine the sharpening gain information corresponding to the first scale.

[0052] The gain mapping rule is used to map the filtering result to sharpening gain information at the corresponding scale. The gain mapping rule is implemented, for example, but not limited to, based on a mapping function or lookup table. The gain mapping rule can be set based on the correlation between the filter value and noise, texture, and edges in the filtering result. For example, the setting principles of the gain mapping rule include, for example, but not limited to, one or more of the following: 1. When the filtering result is high-frequency information extracted from the first image, if the filter value is small, the current pixel corresponding to that filter value may be noise at the first scale, and excessive gain should be avoided to prevent noise enhancement; when the filter value is large, the current pixel may be a strong edge at the first scale, and excessive gain should be avoided to prevent edge overshoot; 2. When the filtering result is low-frequency information extracted from the first image, if the filter value is small, the current pixel is in a dark area, and excessive gain should be avoided to prevent noise enhancement; if the filter value is large, the current pixel is in a bright area, and a larger gain should be used to enhance image texture and edges to improve image clarity; 3. When the filtering result includes information from multiple frequency bands of the extracted first image, the characteristics of the filter values ​​of multiple frequency bands can be combined to avoid noise enhancement and ensure that image texture and edges are enhanced with a larger gain.

[0053] After obtaining the filtering result corresponding to the first scale, the filtering value of each pixel in the filtering result corresponding to the first scale can be mapped to the corresponding gain value based on the gain mapping rule. Based on the gain value of each pixel, the sharpening gain information corresponding to the first scale can be obtained.

[0054] Alternatively, the sharpening gain information can be represented as a gain map with the same size as the first image.

[0055] In the embodiments of this disclosure, the first image can be filtered by filtering kernel parameters corresponding to multiple scales to extract information (i.e., filtering results) of the first image at multiple scales. Then, based on the gain mapping rule, the filtering results are mapped to the corresponding gain to enhance the texture and edges of the first image at multiple scales. Thus, the texture and edges of the first image at multiple scales are enhanced, effectively improving the image clarity.

[0056] Figure 4 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of the present disclosure.

[0057] In some optional embodiments, the filter kernel parameters corresponding to any scale include filter kernel sub-parameters corresponding to multiple frequency bands respectively.

[0058] The multiple frequency bands include, but are not limited to, multiple frequency bands from high frequency, mid frequency, and low frequency. The high-frequency filter kernel parameters are used to extract high-frequency information from the image (e.g., the first image). High-frequency information refers to areas in the image where grayscale changes are abrupt, such as object edges, texture details, and noise. The mid-frequency filter kernel parameters are used to extract mid-frequency information from the image. Mid-frequency information reflects moderate levels of detail in the image, such as leaves, grass, and other non-edge information with certain regularities. The low-frequency filter kernel parameters are used to extract low-frequency information from the image. Low-frequency information reflects areas in the image where grayscale changes are slow, such as large areas of sky and background. Optionally, the multiple frequency bands may include two frequency bands: high frequency (or the first preset frequency band) and low frequency (or the second preset frequency band). Alternatively, the multiple frequency bands may include three frequency bands: high frequency, low frequency, and mid frequency; the specific number of frequency bands is not limited.

[0059] The filter kernel parameters for any frequency band at any scale can include the kernel weights of that frequency band corresponding to the size of that scale, or the filter kernel parameters for any frequency band at any scale can include the kernel weights of a preset size corresponding to that frequency band and the sampling interval corresponding to that scale. For the same scale, the weight values ​​of the filter kernel parameters for different frequency bands are different.

[0060] like Figure 4 As shown, step 2210, which determines the filtering result corresponding to the first scale based on the first image and the filtering kernel parameters corresponding to the first scale, may include:

[0061] Step 22110: Based on the first image and the filter kernel parameters corresponding to the multiple frequency bands at the first scale, determine the filter sub-results corresponding to the multiple frequency bands at the first scale.

[0062] Specifically, for each frequency band of the first scale, the first image can be filtered based on the filter kernel parameters corresponding to that frequency band, and the information of the corresponding frequency band of the first scale of the first image can be extracted as the filtering sub-result.

[0063] Taking a first scale of 3*3 and a high frequency band as an example, the high-frequency filtering kernel parameters of 3*3 can include 3*3 high-frequency kernel weights. Convolution operations are performed on the first image based on these 3*3 high-frequency kernel weights to extract the high-frequency information of the first image at the first scale, which is then used as the high-frequency filtering result at the first scale. The filtering result can be a feature map with the same size as the first image. For example, if the first image is an H*W image, the filtering result is an H*W feature map, which includes the filter values ​​corresponding to each pixel in the first image, where H represents the height of the first image and W represents the width of the first image.

[0064] Step 22120: Based on the filtering sub-results corresponding to multiple frequency bands at the first scale, determine the filtering result corresponding to the first scale.

[0065] In this process, after obtaining the filtering sub-results for multiple frequency bands, the filtering sub-results for multiple frequency bands at the first scale can be combined to form the filtering result corresponding to the first scale.

[0066] Step 2220, based on the filtering result corresponding to the first scale and the pre-configured gain mapping rules, determines the sharpening gain information corresponding to the first scale, which may include:

[0067] Step 22210: Based on the filtering sub-results and gain mapping rules corresponding to multiple frequency bands at the first scale, determine the sharpening gain information corresponding to the first scale.

[0068] After obtaining the filtering result corresponding to the first scale, the filtering values ​​of the filtering sub-results corresponding to multiple frequency bands in the filtering result can be mapped to the sharpening gain information corresponding to the first scale based on the gain mapping rule.

[0069] In some optional embodiments, the gain mapping rule can be a mapping function of filter values ​​across multiple frequency bands, where the variables in the mapping function are the filter values ​​of multiple frequency bands, and the output is the gain. For each pixel in the first image, based on the filter values ​​of the multiple frequency bands corresponding to that pixel, the gain corresponding to that pixel can be obtained through the mapping function. This allows obtaining the gain corresponding to each pixel in the first image, and the gains of all pixels constitute the gain map corresponding to the first scale, serving as the sharpening gain information for the first scale.

[0070] In the embodiments of this disclosure, for each scale, information (i.e., filtering sub-results) of multiple frequency bands of the first image at that scale can be extracted through filtering kernel parameters of multiple frequency bands. Then, the filtering sub-results of multiple frequency bands are mapped to sharpening gain information at that scale. This can take into account the correlation characteristics between the filtering values ​​of different frequency bands and noise, texture, and edges. While effectively enhancing texture and edges at different scales, it prevents the enhancement of noise and avoids overshoot effect under strong edges, further improving the enhancement effect and improving image clarity.

[0071] In some optional embodiments, the filter kernel parameters corresponding to any frequency band of the first scale include the kernel weight and sampling interval of a preset size corresponding to that frequency band; the sampling intervals are different for different scales.

[0072] The kernel weight of the preset size is a basic kernel weight that only includes valid weight values. The preset sizes corresponding to different scales can be the same or different. For example, the preset size corresponding to scale 1 and scale 2 is 3*3, or the preset size corresponding to scale 1 is 3*3 and the preset size corresponding to scale 2 is 5*5.

[0073] The sampling interval is a personalized parameter reflecting different scales. Different sampling intervals represent different scales. The sampling interval represents the interval at which pixel values ​​of the first image are sampled during filtering. The kernel weight of the preset size determines the number of effective weight values ​​participating in the calculation. The sampling interval determines the actual receptive field of the kernel weight. For example, with a kernel weight of 3*3 and a sampling interval of 1, the receptive field of the kernel weight is expanded from 3*3 to 5*5; with a sampling interval of 2, the receptive field of the kernel weight is expanded from 3*3 to 7*7.

[0074] Optionally, when the preset size is the same at different scales, the specific weight values ​​of the kernel weights for the preset sizes at different scales can be the same or different. When the preset sizes are different at different scales, the receptive field reflected by the kernel weights and sampling intervals of the larger-scale preset size must be greater than that of the smaller-scale preset size. For example, if scale 1 is a smaller scale and scale 2 is a larger scale, then the size after expanding the kernel weights of the preset size at scale 1 based on the sampling interval is smaller than the expanded size at scale 2, to ensure that large-scale filtering and parameter extraction of large-scale information of the image are performed. For example, the preset size of scale 1 is 3*3 with a sampling interval of 0, and the preset size of scale 2 is 5*5, with sampling intervals of 0, 1, 2, etc.

[0075] Step 22110, based on the filter kernel parameters corresponding to multiple frequency bands at the first image and the first scale, determines the filter sub-results corresponding to multiple frequency bands at the first scale, including:

[0076] For any one of the multiple frequency bands, based on the kernel weight and sampling interval of the first image and the preset size corresponding to the frequency band, the filter value corresponding to each pixel in the first image in that frequency band is determined; based on the filter value corresponding to each pixel in the first image in that frequency band, the filter sub-result corresponding to that frequency band is determined.

[0077] Optionally, for pixel aij in the first image, a preset number of valid pixel values ​​satisfying the sampling interval in the region centered on pixel aij in the first image can be weighted based on a preset number of valid weight values ​​in a kernel weight of a preset size to obtain the filtered value of the pixel, where aij represents the pixel in the i-th row and j-th column of the first image. Thus, the filtered value of each pixel in the first image in that frequency band can be obtained, and a filtered sub-result corresponding to that frequency band can be constructed based on the filtered values ​​corresponding to all pixels.

[0078] In the embodiments of this disclosure, by sharing a preset kernel weight across different scales, the number of effective weight values ​​participating in filtering at different scales is the same. Different sampling intervals determine the receptive field of the kernel weights at different scales. Based on extracting multi-scale information from the image, this effectively reduces computational complexity and avoids the significant increase in computational complexity caused by large-scale kernel weights. Thus, it achieves edge and texture enhancement at multiple scales of the image with less computation. Taking 3*3 and 7*7 as examples, if all kernel weights in 7*7 are effective weight values, the number of effective weight values ​​participating in the multiplication-addition calculation is 49. During filtering, for each pixel of the image, 49 weight values ​​need to be multiplied and added to 49 pixel values. In this embodiment, using a 3*3 kernel weight as the effective kernel weight and setting the sampling interval to 2 achieves the receptive field of the 7*7 scale. However, the effective weight value in this disclosure is always 9, meaning that at the 7*7 scale, the number of effective weight values ​​participating in the calculation is still 9, balancing a larger receptive field with a smaller computational load.

[0079] In some optional embodiments, based on the first image and the kernel weight and sampling interval of a preset size corresponding to the frequency band, the filter value corresponding to each pixel in the first image in the frequency band is determined, including:

[0080] For any first pixel in the first image, a sampling window centered on the first pixel is determined based on a preset size and sampling interval; a pixel value of a preset size is determined from the sampling window based on the sampling interval; and the pixel value of the preset size is weighted based on the kernel weight of the preset size to obtain the filter value corresponding to the first pixel.

[0081] In this process, any pixel in the first image is taken as the first pixel. Based on a preset-size kernel sampling interval, a sampling window centered on the first pixel can be determined. The size of the sampling window represents the size of the receptive field at the first scale. Pixel values ​​of a preset size (or a pixel tensor of a preset size) are sampled from the sampling window based on the sampling interval. These preset-size pixel values ​​include pixel values ​​that correspond one-to-one with each effective weight value of the preset-size kernel weight. Then, the preset-size pixel values ​​are weighted based on the preset-size kernel weight to obtain the filtered value corresponding to the first pixel. Thus, the filtered value corresponding to each pixel in the first image can be obtained, and a filtering sub-result is obtained based on the filtered values ​​of each pixel.

[0082] In some optional examples, Figure 5 This is a schematic diagram illustrating the principle of determining the filter sub-result provided in an exemplary embodiment of this disclosure. For example... Figure 5 As shown, taking a kernel weight W of a preset size of 3*3 and a sampling interval S of 1 as an example, the kernel weight includes 9 effective weight values ​​wij, i=1, 2, 3, j=1, 2, 3. The sampling window is a 5*5 region centered on the first pixel in the first image. The size of the first image is 6 rows and 6 columns, and the pixel value at each row and column position sk is represented as ask, s=1, 2, ..., 6, k=1, 2, ..., 6. Taking the pixel at position 11 as the first pixel, a 5*5 sampling window C11 centered on a11 can be determined based on the preset size kernel sampling interval. Based on the sampling interval, a preset size pixel value D11 of 3*3 is sampled from the sampling window, including 9 effective pixel values: p11, p13, p15, p31, a11, a13, p51, a31, and a33. Where p represents the padding portion of the first image, p11, p15, p51, and a33 are the pixel values ​​of the four corner pixels within the sampling window, p13, p31, a13, and a31 are the pixel values ​​of the center pixels of the four sides within the sampling window, and a11 is the pixel value of the center region of the sampling window. Further, the pixel values ​​of the preset size are weighted based on a kernel weight of a preset size to obtain the filter value b11 corresponding to a11. When the pixel at position 44 is taken as the first pixel, the pixel value of the first pixel is a44, and the corresponding sampling window is a sampling window C44 with a22, a26, a62, and a66 as the four corners. Pixel values ​​D44 of the preset size are sampled from C44 at sampling intervals. D44 is then weighted based on a kernel weight W of the preset size to obtain the filter value b44 corresponding to a44.

[0083] It should be noted that, Figure 5 This is just one exemplary filtering process. In practical applications, the preset size, the size of the first image, the sampling interval, etc., are not limited to... Figure 5 As shown in the image.

[0084] In the embodiments of this disclosure, during filtering, for any pixel in the first image, a sampling window of the corresponding scale centered on that pixel is determined based on a preset size and sampling interval. Pixel values ​​corresponding to each effective weight value are sampled from the sampling window at the sampling interval, and the filtered value is obtained through weighted summation. This achieves skip-point sampling of the first image, making the computational cost of extracting large-scale information comparable to that of extracting small-scale information, avoiding an increase in computational cost with increasing scale, and effectively reducing computational complexity. Furthermore, this also effectively saves chip area and power consumption when designing chips used for related calculations.

[0085] In some optional embodiments, based on the first image and the kernel weight and sampling interval of a preset size corresponding to the frequency band, the filter value corresponding to each pixel in the first image in the frequency band is determined, including:

[0086] Based on the kernel weights of the preset size and the sampling interval, the kernel weights of the extended size are determined; for any first pixel in the first image, based on the kernel weights of the extended size, the pixel values ​​within the window of the extended size centered on the first pixel are weighted to obtain the filter value corresponding to the first pixel; based on the filter values ​​corresponding to each pixel in the first image, the filter values ​​corresponding to each pixel in the first image in the frequency band are determined.

[0087] The extended-size kernel weight refers to the kernel weight obtained by extending the preset-size kernel weight based on the sampling interval. For example, the preset-size kernel weight can be extended to a larger size by inserting holes or zero-padding according to the sampling interval. For example, extending a 3*3 kernel weight by a sampling interval of 1 results in a 5*5 kernel weight. Its effective weight value is still the effective weight value in the preset-size kernel weight, while the inserted part is an invalid weight value and does not participate in subsequent weighting calculations. After obtaining the extended-size kernel weight, for any first pixel in the first image, the pixel values ​​within the extended-size window centered on the first pixel can be weighted based on the extended-size kernel weight to obtain the filter value corresponding to the first pixel. During weighting, only the effective weight values ​​in the extended-size kernel weight and the pixel values ​​within the extended-size window corresponding to the effective weight values ​​are weighted; it is not necessary to weight the invalid weight values ​​in the kernel weight or the pixel values ​​within the window corresponding to the invalid weight values.

[0088] In some optional examples, Figure 6 This is a schematic diagram illustrating the principle of determining the filter sub-result provided in another exemplary embodiment of this disclosure. For example... Figure 6As shown, the kernel weights W of the preset size and the first image are described above. Based on the kernel weights of the preset size and the sampling interval S (taking S=1 as an example), the determined extended-size kernel weights W' are 5*5 kernel weights, and their effective weight values ​​are still the 9 effective weight values ​​in W. The 0s inserted according to the sampling interval do not participate in subsequent convolution operations or weighted operations, but are used to expand the receptive field of the kernel weights. The extended-size window is as follows: Figure 6 Taking C11 and C44 as examples, during weighting, the effective weight values ​​in the kernel weights based on the extended size are weighted with the corresponding pixel values ​​in C11. That is, the 9 effective weight values ​​are weighted with their corresponding 9 pixel values ​​to obtain the filter value b11 corresponding to a11. Similarly, C44 is weighted based on the kernel weights of the extended size to obtain the filter value b44 corresponding to a44. Other symbols are described above. Figure 5 The content shown.

[0089] In this embodiment, the kernel weights of a preset size are expanded based on the sampling interval to increase the receptive field of the kernel weights. This ensures that during filtering, the number of effective weight values ​​for the expanded, larger-scale kernel weights is comparable to the number of effective weight values ​​for the preset-size kernel weights. Consequently, the computational cost of extracting large-scale information from the image is comparable to that of extracting small-scale information, preventing the computational cost from increasing with scale and effectively reducing computational complexity. Furthermore, this design also effectively saves chip area and power consumption when designing chips used for related calculations.

[0090] In some optional embodiments, the multiple frequency bands include a first preset frequency band and a second preset frequency band.

[0091] The first preset frequency band is high frequency, and the second preset frequency band is low frequency.

[0092] In some optional examples, Figure 7 This is a schematic diagram of filter kernel parameters provided in an exemplary embodiment of this disclosure. For example... Figure 7 As shown, taking multiple scales including scale 1 and scale 2 as an example, the filter kernel parameters for scale 1 include high-frequency kernel weights with a sampling interval S of 0 and low-frequency kernel weights with a sampling interval S of 0. The filter kernel parameters for scale 2 include high-frequency kernel weights with a sampling interval S of 1 and low-frequency kernel weights with a sampling interval S of 1. For any scale, the weight values ​​of its high-frequency kernel weights and low-frequency kernel weights are not limited to... Figure 7 As shown, for example, for the high-frequency kernel weights of scale 1, the weight value of 8 can be adjusted to 12, and the weight value of -1 can be adjusted to -2. The specific weight values ​​can be set according to the requirements and the characteristics of the operators used. The operators used include, but are not limited to, the Laplace operator, the Gaussian operator, the Sobel operator, the Gaussian Laplace operator, the mean filter operator, etc.

[0093] Optionally, the high-frequency filter kernel parameter at any scale can be, but is not limited to, one of the following: Laplacian operator, Sobel operator, Gaussian Laplacian operator of any size. The low-frequency filter kernel parameter at any scale can be Gaussian operator, mean filter operator, etc. of any size. The specific settings can be made according to actual needs.

[0094] The Laplace operator operates at edges where grayscale values ​​change drastically, causing its second derivative to cross zeros. By detecting these zeros, edge locations can be precisely pinpointed. The Laplace operator can be implemented using discrete templates of four or eight neighborhoods. An eight-neighborhood is an example... Figure 7 The high-frequency kernel weights of mesoscale 1, four-neighborhood refers to replacing 8 with 4 in eight-neighborhood.

[0095] The Sobel operator is a first-order differential operator that combines smoothing and gradient calculation to estimate the magnitude and direction of image gradients. Its discrete template includes a weighted average, which is equivalent to first Gaussian smoothing and then differentiation.

[0096] The mechanism of the Laplacian Gaussian operator is to smooth the image and suppress high-frequency noise.

[0097] The mechanism of the Gaussian operator is to perform a weighted average of each pixel and its neighborhood, with the weights distributed in a Gaussian manner as the pixel distance increases.

[0098] The mean filter operator is essentially a linear average filter that can blur edges and is used for noise reduction.

[0099] Based on actual image enhancement needs, and combined with the characteristics of various operators, filter kernel parameters for multiple scales and multiple frequency bands can be set.

[0100] In some optional embodiments, step 22210, which determines the sharpening gain information corresponding to the first scale based on the filtering sub-results and gain mapping rules corresponding to multiple frequency bands at the first scale, may include:

[0101] Determine the gain parameter corresponding to the first scale; based on the first filter sub-result corresponding to the first preset frequency band, the second filter sub-result corresponding to the second preset frequency band, the gain parameter corresponding to the first scale, and the gain mapping rule, determine the sharpening gain information corresponding to the first scale; the gain mapping rule is that the sharpening gain first increases and then decreases as the first filter sub-result increases, and increases as the second filter sub-result increases.

[0102] Among them, the gain parameter is the parameter required by the gain mapping rule. The gain parameters of different scales can be the same or different. The gain parameter can be obtained in advance through empirical debugging.

[0103] Optionally, for the first and second filter values ​​corresponding to the same pixel A in the first and second filter sub-results, the first filter value, the second filter value, and the gain parameter corresponding to the first scale can be mapped using a gain mapping rule to obtain the gain corresponding to pixel A. Thus, the gain of each pixel constitutes the sharpening gain information corresponding to the first scale. The gain mapping rule is that the gain first increases and then decreases as the first filter sub-result increases, and increases as the second filter sub-result increases. Specifically, the gain first increases and then decreases as the filter value in the first filter sub-result increases, and increases as the filter value in the second filter sub-result increases.

[0104] Since the first filtering sub-result is a high-frequency filtering sub-result, extracting high-frequency information from the first image, when the first filter value in the first filtering sub-result is small, the current pixel in the first image may be noise at the first scale, and excessive gain should be avoided to prevent noise enhancement. When the first filter value is large, the current pixel in the first image may be a strong edge at the first scale, and excessive gain should be avoided to prevent edge overshoot. The second filtering sub-result is a low-frequency filtering sub-result, extracting low-frequency information from the first image. When the second filter value in the second filtering sub-result is small, the current pixel in the first image is in a dark area, and excessive gain should be avoided to prevent noise enhancement. When the second filter value is large, the current pixel in the first image is in a bright area, and a larger gain should be used to enhance image texture and edges to improve image clarity. Based on this, the gain mapping rule is set as follows: it first increases and then decreases as the first filter value in the first filtering sub-result increases, and it increases as the second filter value in the second filtering sub-result increases.

[0105] In some optional embodiments, the gain mapping rule can be implemented based on a mapping function. For example, the gain mapping rule can be expressed as a mapping function as follows:

[0106] Formula (1)

[0107] in, This represents the gain corresponding to any pixel A. This represents the first filtered value (i.e., the high-frequency filtered value) corresponding to pixel A in the first filtered sub-result corresponding to the first preset frequency band. This represents the second filter value (i.e., the low-frequency filter value) corresponding to pixel A in the second filter sub-result corresponding to the second preset frequency band. , , , and This represents the gain parameter, which has corresponding gain parameters for different scales. In other words, for different scales... , , , and The values ​​are the parameter values ​​for each scale. This indicates taking the minimum value. Represents the sine function. It is represented in 180 degrees (radians).

[0108] Substituting the first filter value, the second filter value, and the gain parameter corresponding to the first scale for any pixel A into formula (1) yields the gain of pixel A. Gain The gain increases first and then decreases as the first filter value increases, and increases as the second filter value increases, in order to avoid enhancing noise and edge overshoot, and to enhance image texture and edges with a large gain, thereby improving image clarity.

[0109] In the embodiments of this disclosure, by setting corresponding gain parameters for different scales, the high-frequency and low-frequency filter values ​​at different scales are mapped to corresponding gains according to the gain parameters of their respective scales through a gain mapping rule. The gain mapping rule satisfies that the gain first increases and then decreases as the first filter value in the first filter sub-result increases, and increases as the second filter value in the second filter sub-result increases. This allows the mapped gain to avoid enhancing noise, avoid causing edge overshoot, and enhance image texture and edges with a larger gain, effectively improving image clarity.

[0110] In some optional embodiments, step 22210, which determines the sharpening gain information corresponding to the first scale based on the filtering sub-results and gain mapping rules corresponding to multiple frequency bands at the first scale, may include:

[0111] Based on the first filter sub-result corresponding to the first preset frequency band, the first gain corresponding to the first preset frequency band is determined through the first preset lookup table; based on the second filter sub-result corresponding to the second preset frequency band, the second gain corresponding to the second preset frequency band is determined through the second preset lookup table; based on the first gain and the second gain, the sharpening gain information corresponding to the first scale is determined.

[0112] The gain mapping rule is implemented based on a lookup table (LUT) and post-processing. The lookup table is a pre-stored mapping table of input and output correspondences. During application, the output is obtained by looking up the input in the lookup table. The first preset lookup table, the second preset lookup table, and the post-processing rule following the lookup tables satisfy the condition that the gain first increases and then decreases as the first filter value in the first filter sub-result increases, and then increases as the second filter value in the second filter sub-result increases.

[0113] For each first filter value in the first filtering sub-result, the high-frequency gain corresponding to the first filter value is determined using a first preset lookup table. For each second filter value in the second filtering sub-result, the low-frequency gain corresponding to the second filter value is determined using a second preset lookup table. Taking the first preset lookup table as an example, the first preset lookup table includes the high-frequency gains corresponding to multiple preset filter values. The first filter value is input into the first preset lookup table, which compares the first filter value with the multiple preset filter values. If a target filter value identical to the first filter value exists among the multiple preset filter values ​​included in the first preset lookup table, the high-frequency gain corresponding to the target filter value is used as the gain corresponding to the first filter value. If the first filter value falls between two adjacent preset filter values, the gain corresponding to the first filter value is obtained by interpolation based on these two adjacent preset filter values. It should be noted that this is only an exemplary lookup table; in actual applications, the implementation of the lookup table is not limited to this example, and the working principle of the specific lookup table will not be elaborated further. Through the first preset lookup table, the gain corresponding to each first filter value in the first filtering sub-result can be determined, and then, based on the gain corresponding to each first filter value, the first gain corresponding to the first preset frequency band can be obtained. The first gain is the gain of the high-frequency component corresponding to the first image. The working principle of the second lookup table is similar to that of the first lookup table, and will not be described in detail here. Through the second preset lookup table, the second gain corresponding to the second preset frequency band can be determined. The second gain is the gain of the low-frequency component of the first image. Then, based on the first gain and the second gain, the sharpening gain information corresponding to the first scale is determined. Optionally, the first gain and the second gain can be represented as gain maps with the same height and width as the first image. The result of multiplying the corresponding elements of the first gain and the second gain can be used as the sharpening gain information corresponding to the first scale. For example, in a gain map with a first gain of H*W and a gain map with a second gain of H*W, if the gain value of the first gain in the i-th row and j-th column is y1ij, and the gain value of the second gain in the i-th row and j-th column is y2ij, then the gain value of the sharpening gain information in the i-th row and j-th column is y3ij = y1ij * y2ij.

[0114] In the embodiments of this disclosure, two different lookup tables are used to look up high-frequency gain and low-frequency gain respectively, and then the high-frequency gain and low-frequency gain are fused to obtain the sharpening gain information corresponding to the first scale. This avoids noise enhancement, avoids edge overshoot, and can enhance image texture and edges with a larger gain. Furthermore, the lookup tables can reduce the amount of computation, quickly obtain sharpening gain information, and improve image processing efficiency.

[0115] Figure 8 This is a schematic flowchart of an image processing method provided in yet another exemplary embodiment of this disclosure.

[0116] In some alternative embodiments, based on any of the above embodiments, such as Figure 8 As shown, in step 230, the first image is enhanced based on the sharpening gain information corresponding to multiple scales to obtain the enhanced second image, including:

[0117] Step 2301: Determine the first edge information based on the first image.

[0118] The first edge information is obtained by edge extraction of the first image. Optionally, a pre-configured edge extraction operator can be used to extract the first edge information of the first image. The edge extraction operator can be any implementable operator. Edge extraction operators include, but are not limited to, the Sobel operator and the Prewitt operator.

[0119] Optionally, the first edge information can be represented as an edge information map with the same height and width as the first image. Each pixel in the first image can correspond to a set of information in the edge information map, which includes, but is not limited to, edge intensity value and edge direction. The magnitude of the edge intensity value in the edge information map indicates whether the pixel is an edge, or the degree to which the pixel is likely to be an edge. For example, a larger edge intensity value indicates that the pixel is more likely to be an edge. The specific information content included in the first edge information can be determined according to the edge extraction operator, and is not limited to the aforementioned edge intensity value and edge direction.

[0120] Step 2302: Determine edge gain information based on the first edge information.

[0121] Among them, the mapping rule between edge information and edge gain (or edge gain mapping rule) can be preset, and the edge gain information can be determined based on the first edge information and the edge gain mapping rule.

[0122] Optionally, the edge gain mapping rule may be implemented, for example, but not limited to, based on a mapping function or a lookup table.

[0123] Step 2303: Based on the sharpening gain information and edge gain information corresponding to multiple scales, the first image is enhanced to obtain the second image.

[0124] Specifically, for any pixel A in the first image, the pixel value of pixel A can be enhanced based on the sharpening gain value corresponding to pixel A in the sharpening gain information of multiple scales and the edge gain value corresponding to pixel A in the edge gain information, resulting in the enhanced pixel value corresponding to pixel A. The second image is constructed based on the enhanced pixel values ​​corresponding to each pixel in the first image.

[0125] Optionally, enhancement algorithms for sharpening gain values ​​and edge gain values ​​at multiple scales can be pre-set. Based on the sharpening gain information and edge gain information corresponding to each scale, the first image is enhanced using the enhancement algorithms to obtain the second image. The enhancement algorithms can be set according to actual needs.

[0126] In some optional examples, the enhancement algorithm can be represented as follows:

[0127] Formula (2)

[0128] in, This represents the pixel value of pixel i in the first image. This represents the edge gain value of pixel i. Indicates that pixel i is at the 1st dpole. Sharpening gain value at different scales Indicates the first The filtered value corresponding to pixel i in the scale-based filtering result. N represents the number of scales. Optionally, the first scale can refer to the smallest scale. For example, the first scale can be a scale that represents a receptive field of 3*3 for the filter kernel, and the second scale can be a scale that represents a receptive field of 5*5 for the filter kernel, etc. It can be a high-frequency filter value or a low-frequency filter value.

[0129] In some optional examples, the augmentation algorithm can be represented as follows:

[0130] Formula (3)

[0131] in, These represent the preset minimum enhancement threshold and maximum enhancement threshold, respectively. This represents a clamping function used to limit the boost value to a specific value. arrive Between. For other symbols, see formula (2) above.

[0132] In practical applications, the enhancement algorithm is not limited to the algorithms shown in formulas (2) and (3). Other enhancement algorithms can also be set according to the actual image enhancement requirements.

[0133] In the embodiments of this disclosure, edge gain information of the image is further determined, and the image is enhanced by combining sharpening gain information and edge gain information, thereby simultaneously enhancing image texture and edges at different scales.

[0134] Figure 9 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of the present disclosure.

[0135] In some alternative embodiments, based on any of the above embodiments, such as Figure 9 As shown, step 230, based on the sharpening gain information corresponding to multiple scales, enhances the first image to obtain the enhanced second image, which may include:

[0136] Step 2310: Determine the first edge information based on the first image.

[0137] Step 2320: Determine edge gain information based on the first edge information.

[0138] The specific operations for steps 2310 and 2320 can be referred to steps 2301 and 2302 above, and will not be repeated here.

[0139] Step 2330: Based on the filtering sub-results corresponding to the first preset frequency band in multiple frequency bands at each scale, the sharpening gain information and edge gain information corresponding to multiple scales respectively, the first image is enhanced by using preset edge enhancement rules to obtain the second image.

[0140] In this scenario, where the filtering results at each scale include filtering sub-results from a first preset frequency band (i.e., high frequency) and filtering sub-results from other frequency bands (e.g., low frequency filtering sub-results), the first image can be enhanced based on the filtering sub-results from the first preset frequency band, sharpening gain information, and edge gain information at each scale, using a preset edge enhancement rule (or enhancement algorithm). The edge enhancement rule is, for example, but not limited to, a preset mapping function relating high-frequency filtering values, sharpening gain values, and edge gain values. The mapping function includes an original pixel value part and an enhancement value part, wherein the enhancement value part determines the enhancement value used to enhance the original pixel value based on the high-frequency filtering value, sharpening gain value, and edge gain value.

[0141] In some optional examples, the edge enhancement rule (i.e., the enhancement algorithm) can be represented as follows:

[0142] Formula (4)

[0143] in, This represents the pixel value of pixel i in the first image. This represents the edge gain value of pixel i. Indicates that pixel i is at the 1st dpole. Sharpening gain value at different scales Indicates that pixel i is at the 1st dpole. High-frequency filter value at the scale, N represents the number of scales. Optionally, the first scale can refer to the smallest scale. For example, the first scale can be a scale that represents a receptive field of 3*3 for the filter kernel, and the second scale can be a scale that represents a receptive field of 5*5 for the filter kernel, etc.

[0144] In some optional examples, the edge enhancement rule can be represented as follows:

[0145] Formula (5)

[0146] in, These represent the preset minimum enhancement threshold and maximum enhancement threshold, respectively. This represents a clamping function used to limit the boost value to a specific value. arrive Between. For other symbols, see formula (4) above.

[0147] In the embodiments of this disclosure, when extracting information from multiple frequency bands at each scale in the image, the sharpening gain information comprehensively considers the gain of the high-frequency part and the gain of the low-frequency part in the image. It further combines the high-frequency filter value and the edge gain to enhance the image. This can prevent the enhancement of noise, avoid the overshoot effect under strong edges, and enhance the texture and edges of the image with a large gain. Compared with image enhancement based on single frequency band information, the image enhancement based on multi-frequency band information in the embodiments of this disclosure can further improve the image enhancement effect and effectively improve the image clarity.

[0148] In some optional embodiments, determining edge gain information based on the first edge information may include:

[0149] Based on the first edge information, the edge gain information is determined through a third preset lookup table.

[0150] The third preset lookup table can include multiple different mapping tables between preset edge information and edge gain. After obtaining the first edge information of the first image, the edge gain information corresponding to the first edge information can be found through the third preset lookup table.

[0151] In the embodiments of this disclosure, the mapping from edge information to edge gain information is achieved through a lookup table, which helps to further improve the efficiency of determining edge gain information, thereby further improving image processing efficiency.

[0152] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0153] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0154] Exemplary device

[0155] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be used to implement the image processing method provided in any of the above embodiments of the present disclosure, such as... Figure 10 The apparatus shown may include: a determining module 51, a first processing module 52, and a second processing module 53.

[0156] The determination module 51 is used to determine the first image to be processed, and the filter kernel parameters corresponding to multiple scales respectively.

[0157] The first processing module 52 is used to determine the sharpening gain information corresponding to multiple scales based on the first image and the filter kernel parameters corresponding to multiple scales respectively.

[0158] The second processing module 53 is used to enhance the first image based on sharpening gain information corresponding to multiple scales to obtain the enhanced second image.

[0159] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of the present disclosure.

[0160] In some alternative embodiments, in the above... Figure 10 Based on the illustrated embodiments, as Figure 11 As shown, the first processing module 52 includes: a first determining unit 521 and a second determining unit 522.

[0161] The first determining unit 521 is used to determine the filtering result corresponding to the first scale based on the first image and the filtering kernel parameters corresponding to the first scale for any first scale among multiple scales.

[0162] The second determining unit 522 is used to determine the sharpening gain information corresponding to the first scale based on the filtering result corresponding to the first scale and the pre-configured gain mapping rules.

[0163] In some optional embodiments, the filter kernel parameters corresponding to any scale include filter kernel sub-parameters corresponding to multiple frequency bands respectively.

[0164] The first determining unit 521 is specifically used for:

[0165] Based on the filter kernel parameters corresponding to multiple frequency bands at the first image and the first scale, the filter sub-results corresponding to the multiple frequency bands at the first scale are determined. Based on the filter sub-results corresponding to the multiple frequency bands at the first scale, the filter result corresponding to the first scale is determined.

[0166] The second determining unit 522 is specifically used to: determine the sharpening gain information corresponding to the first scale based on the filtering sub-results and gain mapping rules corresponding to multiple frequency bands of the first scale.

[0167] In some optional embodiments, the filter kernel parameters corresponding to any frequency band of the first scale include the kernel weight and sampling interval of a preset size corresponding to that frequency band; the sampling intervals are different for different scales.

[0168] The first determining unit 521 is specifically used for:

[0169] For any one of the multiple frequency bands, based on the kernel weight and sampling interval of the first image and the preset size corresponding to the frequency band, the filter value corresponding to each pixel in the first image in that frequency band is determined; based on the filter value corresponding to each pixel in the first image in that frequency band, the filter sub-result corresponding to that frequency band is determined.

[0170] In some optional embodiments, the first determining unit 521 is specifically used for:

[0171] For any first pixel in the first image, a sampling window centered on the first pixel is determined based on a preset size and sampling interval; a pixel value of a preset size is determined from the sampling window based on the sampling interval; and the pixel value of the preset size is weighted based on the kernel weight of the preset size to obtain the filter value corresponding to the first pixel.

[0172] In some optional embodiments, the first determining unit 521 is specifically used for:

[0173] Based on the kernel weights of the preset size and the sampling interval, the kernel weights of the extended size are determined; for any first pixel in the first image, based on the kernel weights of the extended size, the pixel values ​​within the window of the extended size centered on the first pixel are weighted to obtain the filter value corresponding to the first pixel; based on the filter values ​​corresponding to each pixel in the first image, the filter values ​​corresponding to each pixel in the first image in the frequency band are determined.

[0174] In some optional embodiments, the multiple frequency bands include a first preset frequency band and a second preset frequency band.

[0175] In some optional embodiments, the second determining unit 522 is specifically used for:

[0176] Determine the gain parameter corresponding to the first scale; based on the first filter sub-result corresponding to the first preset frequency band, the second filter sub-result corresponding to the second preset frequency band, the gain parameter corresponding to the first scale, and the gain mapping rule, determine the sharpening gain information corresponding to the first scale; the gain mapping rule is that the sharpening gain first increases and then decreases as the first filter sub-result increases, and increases as the second filter sub-result increases.

[0177] In some optional embodiments, the second determining unit 522 is specifically used for:

[0178] Based on the first filter sub-result corresponding to the first preset frequency band, the first gain corresponding to the first preset frequency band is determined through the first preset lookup table; based on the second filter sub-result corresponding to the second preset frequency band, the second gain corresponding to the second preset frequency band is determined through the second preset lookup table; based on the first gain and the second gain, the sharpening gain information corresponding to the first scale is determined.

[0179] Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of the present disclosure.

[0180] In some alternative embodiments, based on any of the above embodiments, such as Figure 12 As shown, the second processing module 53 may include: a first processing unit 531, a second processing unit 532 and a third processing unit 533.

[0181] The first processing unit 531 is used to determine first edge information based on the first image.

[0182] The second processing unit 532 is used to determine edge gain information based on the first edge information.

[0183] The third processing unit 533 is used to enhance the first image based on sharpening gain information and edge gain information corresponding to multiple scales to obtain the second image.

[0184] In some alternative embodiments, based on any of the above embodiments, such as Figure 12 As shown, the second processing module 53 may include: a first processing unit 531, a second processing unit 532 and a third processing unit 533.

[0185] The first processing unit 531 is used to determine first edge information based on the first image.

[0186] The second processing unit 532 is used to determine edge gain information based on the first edge information.

[0187] The third processing unit 533 is used to enhance the first image based on the filtering sub-results corresponding to the first preset frequency band in multiple frequency bands at each scale, the sharpening gain information and edge gain information corresponding to multiple scales respectively, and adopt a preset edge enhancement rule to obtain the second image.

[0188] In some optional embodiments, the second processing unit 532 is specifically used to: determine edge gain information based on the first edge information and through a third preset lookup table.

[0189] The exemplary embodiments of this device correspond to the exemplary method section described above in terms of implementation. The corresponding content between the two can be referenced, combined, and cited, and will not be repeated here. The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section described above, and will not be repeated here.

[0190] Exemplary electronic devices

[0191] Figure 13 A structural diagram of an electronic device provided in this disclosure includes at least one processor 91 and a memory 92.

[0192] The processor 91 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0193] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the image processing methods and / or other desired functions of the various embodiments of this disclosure described above.

[0194] In one example, the electronic device 90 may also include an input device 93 and an output device 94, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0195] The input device 93 may also include, for example, a touchscreen, a microphone, various sensors, etc. Sensors may include, for example, image sensors (e.g., cameras, webcams), LiDAR, millimeter-wave radar, ultrasonic radar, positioning sensors, pressure sensors, air quality sensors, temperature sensors, etc. Image sensors, LiDAR, millimeter-wave radar, ultrasonic radar, etc., can be used for environmental perception, i.e., detecting moving and static objects in the surrounding environment. Moving and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to locate the mobile device (e.g., a bicycle, a robot, etc.) where the electronic device is located. Positioning sensors may include, for example, an Inertial Measurement Unit (IMU), a Global Positioning System (GPS), etc. Pressure sensors can be used to detect seat pressure. Temperature sensors can be used to detect the temperature inside the vehicle cabin. Air quality sensors can be used to detect the air quality inside the vehicle cabin.

[0196] The output device 94 can output various information to the outside, including, for example, a display, a speaker, a communication network and its connected remote output devices, etc.

[0197] Of course, for the sake of simplicity, Figure 13 Only some of the components of the electronic device 90 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 90 may include any other suitable components depending on the specific application.

[0198] Exemplary computer program products and computer-readable storage media

[0199] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the image processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0200] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0201] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the image processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0202] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0204] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An image processing method, comprising: Determine the first image to be processed, and the filter kernel parameters corresponding to multiple scales respectively; Based on the first image and the filter kernel parameters corresponding to the multiple scales respectively, the sharpening gain information corresponding to the multiple scales is determined; Based on the sharpening gain information corresponding to the multiple scales, the first image is enhanced to obtain the enhanced second image.

2. The method according to claim 1, wherein, The step of determining the sharpening gain information corresponding to the multiple scales based on the first image and the filter kernel parameters corresponding to the multiple scales respectively includes: For any first scale among the multiple scales, the filtering result corresponding to the first scale is determined based on the first image and the filtering kernel parameters corresponding to the first scale; Based on the filtering result corresponding to the first scale and the pre-configured gain mapping rule, the sharpening gain information corresponding to the first scale is determined.

3. The method according to claim 2, wherein, The filter kernel parameters corresponding to any scale include filter kernel sub-parameters corresponding to multiple frequency bands respectively; The step of determining the filtering result corresponding to the first scale based on the first image and the filtering kernel parameters corresponding to the first scale includes: Based on the first image and the filter kernel parameters corresponding to the multiple frequency bands at the first scale, the filter sub-results corresponding to the multiple frequency bands at the first scale are determined respectively; Based on the filtering sub-results corresponding to the multiple frequency bands of the first scale, the filtering result corresponding to the first scale is determined; The step of determining the sharpening gain information corresponding to the first scale based on the filtering result corresponding to the first scale and the pre-configured gain mapping rule includes: Based on the filtering sub-results and gain mapping rules corresponding to the multiple frequency bands at the first scale, the sharpening gain information corresponding to the first scale is determined.

4. The method according to claim 3, wherein, The multiple frequency bands include a first preset frequency band and a second preset frequency band; Based on the filtering sub-results and gain mapping rules corresponding to the multiple frequency bands at the first scale, the sharpening gain information corresponding to the first scale is determined, including: Determine the gain parameter corresponding to the first scale; Based on the first filter sub-result corresponding to the first preset frequency band, the second filter sub-result corresponding to the second preset frequency band, the gain parameter corresponding to the first scale, and the gain mapping rule, the sharpening gain information corresponding to the first scale is determined; The gain mapping rule is that the sharpening gain first increases and then decreases as the first filter result increases, and increases as the second filter result increases; or, Based on the first filter sub-result corresponding to the first preset frequency band, the first gain corresponding to the first preset frequency band is determined through a first preset lookup table. Based on the second filter sub-result corresponding to the second preset frequency band, the second gain corresponding to the second preset frequency band is determined through the second preset lookup table; Based on the first gain and the second gain, the sharpening gain information corresponding to the first scale is determined.

5. The method according to claim 3, wherein, The filter kernel parameters corresponding to any frequency band of the first scale include the kernel weight and sampling interval of the preset size corresponding to that frequency band; Different scales correspond to different sampling intervals; The step of determining the filtering sub-results corresponding to the multiple frequency bands at the first scale based on the filtering kernel parameters corresponding to the first image and the multiple frequency bands at the first scale includes: For any one of the multiple frequency bands, based on the first image and the kernel weight of the preset size corresponding to the frequency band and the sampling interval, the filter value corresponding to each pixel in the first image in that frequency band is determined; Based on the filter value corresponding to each pixel in the first image in that frequency band, the filter sub-result corresponding to that frequency band is determined.

6. The method according to claim 5, wherein, The step of determining the filter value corresponding to each pixel in the first image in the frequency band based on the kernel weight of the preset size and the sampling interval corresponding to the first image and the frequency band includes: For any first pixel in the first image, a sampling window centered on the first pixel is determined based on the preset size and the sampling interval; The pixel value of the preset size is determined from the sampling window based on the sampling interval; Based on the kernel weights of the preset size, the pixel values ​​of the preset size are weighted to obtain the filtered value corresponding to the first pixel; or, Based on the kernel weight of the preset size and the sampling interval, determine the kernel weight of the extended size; For any first pixel in the first image, the pixel values ​​within the window of the extended size centered on the first pixel are weighted based on the kernel weight of the extended size to obtain the filtered value corresponding to the first pixel. Based on the filter values ​​corresponding to each pixel in the first image, the filter values ​​corresponding to each pixel in the first image in that frequency band are determined.

7. The method according to any one of claims 1-6, wherein, The enhancement process, based on the sharpening gain information corresponding to the multiple scales, is applied to the first image to obtain the enhanced second image, including: Based on the first image, determine the first edge information; Based on the first edge information, determine the edge gain information; Based on the sharpening gain information and edge gain information corresponding to the multiple scales respectively, the first image is enhanced to obtain the second image.

8. The method according to claim 7, wherein, The step of determining edge gain information based on the first edge information includes: Based on the first edge information, the edge gain information is determined through a third preset lookup table.

9. The method according to claim 3, wherein, The enhancement process, based on the sharpening gain information corresponding to the multiple scales, is applied to the first image to obtain the enhanced second image, including: Based on the first image, determine the first edge information; Based on the first edge information, determine the edge gain information; Based on the filtering sub-results corresponding to the first preset frequency band in the multiple frequency bands at each scale, the sharpening gain information and the edge gain information corresponding to the multiple scales respectively, the first image is enhanced using preset edge enhancement rules to obtain the second image.

10. An image processing apparatus, comprising: The determination module is used to determine the first image to be processed, and the filter kernel parameters corresponding to multiple scales respectively; The first processing module is used to determine the sharpening gain information corresponding to the multiple scales based on the first image and the filter kernel parameters corresponding to the multiple scales respectively; The second processing module is used to enhance the first image based on the sharpening gain information corresponding to the multiple scales, so as to obtain the enhanced second image.

11. A computer-readable storage medium storing a computer program, which, when executed, implements the image processing method according to any one of claims 1-9.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image processing method according to any one of claims 1-9.

Images