Image processing method and device, computer readable storage medium and terminal equipment

Abstract

The invention provides an image processing method and device and a computer readable storage medium, and a terminal device comprises the steps that a first video and a second video are acquired, and the second video is a video obtained after the first video is enhanced; determining a first overlapping region with the maximum structural similarity in two adjacent frames in the first video, wherein the two adjacent frames comprise a first current frame and a previous frame thereof; calculating a first pixel difference between the pixel variation of the first video and the second video in the current frame and the pixel variation of the first video and the second video in the previous frame; and calculating a correction degree according to the first pixel difference, and correcting each pixel point of a second current frame in the second video in the first overlapping region according to the correction degree and the first pixel difference. According to the technical scheme, the flicker correction effect can be improved, and the video quality is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and apparatus, a computer-readable storage medium, and a terminal device. Background Art

[0002] Deep learning technology has been used in more and more scenarios, and most deep learning is supervised. For example, in the task of image dynamic range enhancement, images need to be used as supervisory labels. These supervisory labels are often obtained by manual image retouching. After manual image retouching, the modified image will introduce problems such as color cast, local detail loss, and video flicker due to changes in the brightness and contrast of different areas of the input image. Such problematic images are used as supervisory labels for training models, and the resulting models often have problems such as color cast, local detail loss, and video flicker. The problem of flicker amplification in videos after image enhancement is mainly because it is sensitive to changes in the input values ​​of the approximate scene, and most of the image retouching processes require manual participation. It is not easy for humans to ensure that the degree of brightening after retouching of each approximate scene is similar.

[0003] However, the flicker correction scheme in the prior art mainly considers the brightness average of the entire image, and the correction effect is poor. Summary of the invention

[0004] The technical problem to be solved by this application is the problem of video flickering in the video after image enhancement.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, an image processing method is provided, the image processing method comprising: acquiring a first video and a second video, the second video being a video after enhancing the first video; determining a first overlapping area with the greatest structural similarity between two adjacent frames in the first video, the two adjacent frames including a first current frame and a previous frame; calculating a first pixel difference between a pixel change in a current frame and a pixel change in a previous frame between the first video and the second video; calculating a correction degree based on the first pixel difference, and correcting each pixel point of the second current frame in the second video within the first overlapping area based on the correction degree and the first pixel difference.

[0007] Optionally, the calculation of the first pixel difference between the pixel change in the current frame and the pixel change in the previous frame of the first video and the second video includes: calculating the second pixel difference between the second current frame and the first current frame in the first overlapping area; calculating the third pixel difference between the previous frame of the second current frame and the previous frame of the first current frame in the first overlapping area; calculating the difference between the second pixel difference and the third pixel difference as the first pixel difference.

[0008] Optionally, calculating the correction degree according to the first pixel difference includes: calculating the correction degree using the first pixel difference and a smoothing function, and the greater the first pixel difference, the greater the correction degree.

[0009] Optionally, correcting each pixel of the second current frame in the second video in the first overlapping area according to the correction degree and the first pixel difference includes: calculating parameters of the two-dimensional Gaussian surface according to the position of each pixel in the first overlapping area to obtain the weight value of each pixel in the first overlapping area; weighting according to the weight value of each pixel in the second current frame, the correction degree corresponding to each pixel and the first pixel difference corresponding to each pixel for correcting each pixel in the second current frame.

[0010] Optionally, correcting each pixel of the second current frame in the second video within the first overlapping area according to the correction degree and the first pixel difference includes: calculating a third pixel difference between a previous frame of the second current frame and a previous frame of the first current frame within the first overlapping area; weighting the first pixel difference corresponding to each pixel point according to the correction degree of each pixel point in the second current frame to obtain a weighted result; and calculating the pixel value of each pixel point in the second current frame, the third pixel difference and the sum of the weighted results to obtain a corrected pixel point.

[0011] Optionally, determining the overlapping area with the greatest structural similarity between two adjacent frames in the first video includes: sliding the first current frame on its previous frame and calculating the structural similarity of the overlapping area after each sliding; and selecting the overlapping area with the greatest structural similarity as the first overlapping area.

[0012] Optionally, the image processing method also includes: respectively calculating the ratios of the values ​​of each color channel of each reference pixel in the first current frame to the values ​​of each color channel to obtain multiple sets of ratios; respectively weighting the ratios of the values ​​of each pixel in the second current frame in each color channel to the values ​​corresponding to the color channel to obtain multiple sets of correction values; and correcting the values ​​of each pixel in each color channel using at least the multiple sets of correction values.

[0013] In a second aspect, the present application also discloses an image processing device, which includes: an acquisition module, used to acquire a first video and a second video, wherein the second video is a video after enhancing the first video; an overlapping area determination module, used to determine a first overlapping area with the greatest structural similarity between two adjacent frames in the first video, wherein the two adjacent frames include a first current frame and its previous frame; a pixel difference calculation module, used to calculate a first pixel difference between a pixel change amount in a current frame and a pixel change amount in a previous frame between the first video and the second video; a correction module, used to calculate a correction degree based on the first pixel difference, and correct each pixel point of the second current frame in the second video within the first overlapping area based on the correction degree and the first pixel difference.

[0014] Optionally, the smoothing function is a S-type function.

[0015] Optionally, the ratio of the value of each color channel of each pixel to the values ​​of all color channels is calculated as the weighted weight of the color channel; the value of each color channel of the corrected pixel is weightedly calculated using the weighted weights of each color channel and the corresponding correction value.

[0016] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is executed by a processor to execute any one of the methods provided in the first aspect.

[0017] According to a fourth aspect, a computer program product is provided, on which a computer program is stored, and the computer program is executed by a processor to execute any one of the methods provided in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application further provides a chip (or a data transmission device) on which a computer program is stored, and when the computer program is executed by the chip, the steps of the above method are implemented.

[0019] In the sixth aspect, an embodiment of the present application also provides a system chip, which is applied to a terminal, and the chip system includes at least one processor and an interface circuit, the interface circuit and the at least one processor are interconnected through lines, and the at least one processor is used to execute instructions to execute any one of the methods provided in the first aspect.

[0020] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0021] In the technical solution of the present application, a first video and a second video are obtained, and the second video is a video after the first video is enhanced; the first overlapping area with the largest structural similarity between two adjacent frames in the first video is determined, and the two adjacent frames include the first current frame and its previous frame; the first pixel difference between the two adjacent frames in the first video and the corresponding two adjacent frames in the second video in the overlapping area is calculated; the degree of correction is calculated according to the first pixel difference, and the various pixel points of the second current frame in the second video in the first overlapping area are corrected according to the degree of correction and the first pixel difference. The technical solution of the present application refers to the degree of change of the corresponding positions in the two adjacent frames of the first video before image enhancement, and corrects the degree of change of the corresponding positions in the two adjacent frames of the second video after image enhancement, thereby realizing the correction of each pixel point in the second current frame. In addition, considering the changes in the scene in the adjacent frames of the video, only the pixel points in the first overlapping area are corrected, which not only takes into account the overall picture, but also takes into account the local details, thereby improving the effect of flicker correction and improving the video quality.

[0022] Furthermore, the parameters of the two-dimensional Gaussian surface are calculated according to the size of the first overlapping area to obtain the weight value of each pixel in the first overlapping area; weighting is performed according to the weight value of each pixel in the second current frame and the correction degree corresponding to each pixel and the first pixel difference corresponding to each pixel, so as to correct each pixel in the second current frame. The technical solution of the present application uses the two-dimensional Gaussian surface parameters to perform weighted calculation on the pixels during flicker correction, which can prevent artifacts from appearing in the edge area of ​​the first overlapping area when calculating the correction value, thereby further improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present application;

[0024] Figure 2 is a framework diagram of an image processing system provided in an embodiment of the present application;

[0025] Figure 3 is a flowchart of another image processing method provided by an embodiment of the present application;

[0026] Figure 4 is a framework diagram of another image processing system provided in an embodiment of the present application;

[0027] Figure 5 is a flowchart of another image processing method provided by an embodiment of the present application;

[0028] Figure 6 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0029] Figure 7It is a schematic diagram of the hardware structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] As described in the background art, the flicker correction scheme in the prior art mainly considers the average brightness of the entire image, and the correction effect is poor.

[0031] The technical solution of the present application refers to the degree of change of the corresponding positions in two adjacent frames of the first video before image enhancement, and corrects the degree of change of the corresponding positions in two adjacent frames of the second video after image enhancement, thereby achieving correction of each pixel in the second current frame. In addition, considering the changes in the picture scenes in adjacent frames of the video, only the pixels in the first overlapping area are corrected, which not only takes into account the overall picture, but also takes into account the local details, thereby improving the effect of flicker correction and improving the video quality. In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] The first video and the second video in the embodiment of the present invention may both include multiple video frames.

[0033] The second current frame referred to in the embodiment of the present invention refers to a video frame in the second video that needs to be flicker corrected, and the first video frame refers to a video frame before enhancement corresponding to the second current frame in the first video.

[0034] See also Figure 1 The method provided in this application specifically includes the following steps:

[0035] Step 101: Obtain a first video and a second video, where the second video is a video obtained by enhancing the first video;

[0036] Step 102: determining a first overlapping region having the greatest structural similarity between two adjacent frames in the first video, where the two adjacent frames include a first current frame and a previous frame;

[0037] Step 103: Calculate a first pixel difference between a pixel change amount in a current frame and a pixel change amount in a previous frame of the first video and the second video;

[0038] Step 104: Calculate the correction degree according to the first pixel difference, and correct each pixel point in the second current frame in the second video within the first overlapping area according to the correction degree and the first pixel difference.

[0039] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0040] It is understandable that, in a specific implementation, the image processing method can be implemented in the form of a software program, and the software program runs in a processor integrated inside a chip or a chip module. The method can also be implemented in the form of software combined with hardware, which is not limited in this application.

[0041] In this embodiment, refer to Figure 2 The image enhancement device 20 can enhance the first video and output a second video, and each video frame in the second video corresponds to each video frame in the first video.

[0042] The image correction device 30 can perform Figure 1 The first video and the second video are input to the image correction device 30, and the image correction device 30 outputs the corrected second video. Specifically, the image correction device 30 can correct the video flicker problem in the second video. Compared with the second video before correction, the corrected second video has no flicker and has higher quality.

[0043] Continue to refer to Figure 1 In the specific implementation of step 101, the image correction device 30 obtains a first video and a second video. Each video frame in the second video is a video frame after performing an image enhancement operation on each video frame in the first video. There may be a video flicker problem in the second video.

[0044] In the specific implementation of step 102, the first overlapping region with the largest structural similarity between two adjacent frames in the first video can be determined. Specifically, the structural similarity can be represented by a structural similarity index (SSIM). The structural similarity can be used to measure the similarity between two given video frames. In other words, the first overlapping region represents an area in which the pictures between two adjacent frames have basically not changed.

[0045] The embodiment of the present application takes into account the changes in the picture scenes in adjacent frames of the video and only corrects the pixels in the first overlapping area. It not only takes into account the overall picture, but also takes into account the local details, thereby improving the flicker correction effect and improving the video quality.

[0046] In a specific embodiment, the first current frame is slid on the previous frame, and the structural similarity of the overlapping area is calculated after each sliding; the overlapping area with the largest structural similarity is selected as the first overlapping area.

[0047] For example, the previous frame of the first current frame is org_img_pre, and the first current frame is org_img_cur. The first current frame org_img_cur is slid up, down, left, and right on the previous frame org_img_pre of the first current frame, and the structural similarity of the overlapping area, such as SSIM, is calculated, and finally the coordinates of the overlapping area with the largest SSIM are obtained, that is, the first overlapping area is obtained.

[0048] Continue to refer to Figure 1 In the specific implementation of step 103, the pixel change amount between the first video and the second video in the current frame, and the first pixel difference between the pixel change amount between the first video and the second video in the previous frame are calculated. In other words, the first pixel difference represents the difference in the pixel change amount between the first video and the second video in two adjacent frames.

[0049] In a specific embodiment, the first pixel difference can be calculated in the following manner: calculate the second pixel difference between the second current frame and the first current frame in the first overlapping area; calculate the third pixel difference between the previous frame of the second current frame and the previous frame of the first current frame in the first overlapping area; calculate the difference between the second pixel difference and the third pixel difference as the first pixel difference.

[0050] Specifically, the second pixel difference represents the difference between the pixel value of the pixel point in the second current frame and the pixel value of the corresponding pixel point in the first current frame within the first overlapping area. Correspondingly, the third pixel difference represents the difference between the pixel value of the pixel point in the previous frame of the second current frame and the pixel value of the corresponding pixel point in the previous frame of the first current frame within the first overlapping area.

[0051] More specifically, the previous frame of the second video frame is org_label_img_pre, and the second current frame is org_label_img_cur. The coordinates of the first overlapping area are used to intercept the overlapping area pictures of the previous frame org_img_pre of the first current frame, the first current frame org_img_cur, the previous frame org_label_img_pre of the second current frame, and the second current frame org_label_img_cur, respectively. The above intercepted pictures are represented by org_img_pre_cut, org_img_cur_cut, org_label_img_pre_cut, and org_label_img_cur_cut, respectively.

[0052] The specific calculation formula for the first pixel difference delta_img is as follows:

[0053] delta_img = (org_label_cur_cut-org_img_cur_cut)-(org_label_pre_cut-org_img_pre_cut)(1)

[0054] Continue to refer to Figure 1 In the specific implementation of step 104, different correction degrees are calculated for different first pixel differences, and the first pixel differences and their corresponding correction degrees are used to correct each pixel point of the second current frame in the first overlapping area.

[0055] Specifically, the correction degree is calculated using the first pixel difference and the smoothing function. The first pixel difference is positively correlated with the correction degree. That is, the larger the first pixel difference, the greater the correction degree; conversely, the smaller the first pixel difference, the smaller the correction degree.

[0056] In a specific embodiment, the larger the absolute value of the first pixel difference delta_img is, the closer the correction degree is to 1. The smaller the absolute value of the first pixel difference is, the closer the correction degree is to zero.

[0057] In a specific embodiment, the smoothing function is a sigmoid function, the first difference first pixel difference delta_img is an input of the sigmoid function, and the correction degree rectify_k is an output of the sigmoid function.

[0058] For example, the formula for calculating the correction degree rectify_k using the S-type function is as follows:

[0059]

[0060] Wherein, k and b are empirical constants, for example, k=0.5 and b=-5.

[0061] The embodiment of the present invention calculates the correction degree through a smoothing function, so that the correction degree can change smoothly with the first pixel difference, so that the pixel value in the corrected second current frame changes naturally, further improving the quality of the corrected second video.

[0062] In another specific embodiment, the specific steps of correction are as follows: calculating the third pixel difference between the previous frame of the second current frame and the previous frame of the first current frame in the first overlapping area; weighting the first pixel difference corresponding to each pixel point according to the correction degree of each pixel point in the second current frame to obtain a weighted result. Calculating the sum of the pixel value of each pixel point in the second current frame, the third pixel difference and the weighted result to obtain a corrected pixel point.

[0063] The calculation formula for the pixel value rectify_value of the corrected pixel point in the first overlapping area of ​​the second current frame is as follows:

[0064] rectify_value=org_img_cur_cut+(org_label_img_pre_cut-org_img_pre_cut)+delta_img×rectify_k(3)

[0065] Since the flicker correction is for the pixels of the second current frame in the first overlapping area, the non-first overlapping area in the second current frame can be spliced ​​with the corrected first overlapping area to obtain a complete corrected second current frame.

[0066] It should be noted that the specific method for splicing the non-first overlapping area with the corrected first overlapping area can refer to the existing technology, and the present application does not limit this.

[0067] In another specific embodiment, in order to avoid artifacts appearing at the edge of the first overlapping area during correction, the correction degree may be adjusted according to the position of each pixel in the first overlapping area.

[0068] Specifically, the parameters of the two-dimensional Gaussian surface are calculated according to the positions of each pixel in the first overlapping area to obtain the weight value of each pixel in the first overlapping area. The weight value is related to the position of the pixel. More specifically, the weight value of the pixel located at the center of the first overlapping area is the highest value 1, and as the position of the pixel moves from the center to the edge of the first overlapping area, its corresponding weight value gradually decays to 0.

[0069] Then, during correction, weighting is performed according to the weight value of each pixel point in the second current frame, the correction degree corresponding to each pixel, and the first pixel difference corresponding to each pixel point to correct each pixel point in the second current frame.

[0070] Specifically, the specific steps of correction are as follows: calculate the third pixel difference between the previous frame of the second current frame and the previous frame of the first current frame in the first overlapping area; weight the first pixel difference corresponding to each pixel point according to the weight value and correction degree of each pixel point in the second current frame to obtain a weighted result; calculate the pixel value of each pixel point in the second current frame, the third pixel difference and the sum of the weighted results to obtain the corrected pixel point.

[0071] The calculation formula for the pixel value rectify_value of the corrected pixel point in the first overlapping area of ​​the second current frame is as follows:

[0072] rectify_value=org_img_cur_cut+(org_label_img_pre_cut-org_img_pre_cut)+delta_img×(posk×rectify_k)(4)

[0073] Compared with formula (3), when calculating the pixel value rectify_value, formula (4) uses the weight value posk of the pixel point to adjust the correction degree rectify_k of the pixel point.

[0074] The embodiment of the present application calculates weights related to the positions of pixels, thereby preventing artifacts from appearing in the edge area of ​​the first overlapping area when fusing correction values, thereby further improving the quality of the second video.

[0075] Please refer to Figure 3 , Figure 3 The flowchart of an image processing method is shown.

[0076] The input of the method is a first current frame and its previous frame in the first video, and a second current frame and its previous frame in the second video. The output is the corrected second current frame.

[0077] In step 301, a first overlapping region having the greatest structural similarity between two adjacent frames in a first video is determined.

[0078] In step 302, a first pixel difference and a correction degree between a pixel change amount in a current frame and a pixel change amount in a previous frame of the first video and the second video are calculated.

[0079] In step 303, the weight value of each pixel in the first overlapping area is calculated.

[0080] In step 304, the corrected pixel points in the first overlapping area are calculated and merged with the non-first overlapping area.

[0081] Specifically, step 302 and step 303 may be executed in parallel, or step 302 may be executed first and then step 303, or step 303 may be executed first and then step 302. This application does not impose any restrictions on this.

[0082] For more specific implementation methods of the embodiments of the present application, please refer to the aforementioned embodiments, which will not be repeated here.

[0083] Refer to Figure 4 In addition to performing flicker correction 403 on the second video, the image correction device 30 can also perform color cast correction 402 on each video frame in the second video to correct the color cast problem in the second image.

[0084] Please refer to Figure 5 In step 501, the ratios of the values ​​of each color channel of each reference pixel to the values ​​of each color channel are calculated respectively to obtain multiple sets of ratios.

[0085] Taking RGB as an example, the ratio of the color channel R, G, B values ​​of each reference pixel to the color channel R is calculated to obtain a set of ratios ratio_rr, ratio_gr, ratio_br. The specific formula is as follows:

[0086] ratio_rr = 1 (5)

[0087] ratio_gr = org_g / org_r (6)

[0088] ratio_br = org_b / org_r (7)

[0089] If the denominator is zero, the ratio is not calculated.

[0090] Accordingly, the ratios of the values ​​of the color channels R, G, and B of each reference pixel to the color channel G are calculated respectively to obtain a set of ratios ratio_rg, ratio_gg, and ratio_bg.

[0091] Calculate the ratio of the color channel R, G, B values ​​of each reference pixel to the color channel B to obtain a set of ratios ratio_rb, ratio_gb, ratio_bb.

[0092] In step 502, the value of each pixel point in each color channel in the second current frame and the ratio corresponding to the color channel are weighted to obtain multiple groups of correction values.

[0093] For a set of ratios ratio_rr, ratio_gr, ratio_br, weight them with the value of each pixel in the color channel R to obtain a set of correction values ​​rectify_rr, rectify_gr, rectify_br. The specific formula is as follows:

[0094] rectify_rr = org_label_r×ratio_rr (8)

[0095] rectify_gr = org_label_r×ratio_gr (9)

[0096] rectify_br = org_label_r×ration_br (10)

[0097] Among them, org_label_r represents the value of the pixel point in the second current frame in the R channel.

[0098] For a set of ratios ratio_rg, ratio_gg, ratio_bg, weight them with the value of each pixel in the color channel G to obtain a set of correction values ​​rectify_rg, rectify_gg, rectify_bg.

[0099] For a set of ratios ratio_rb, ratio_gb, ratio_bb, weight them with the value of each pixel in color channel B to obtain a set of correction values ​​rectify_rb, rectify_gb, rectify_bb.

[0100] Furthermore, the correction values ​​that exceed the preset range are modified. For example, the first picture and the second picture are both 8-bit pictures, and their data representation range is 0 to 255. If there are correction values ​​exceeding 255 in a set of correction values, they need to be modified so that the correction values ​​fall within the range of 0 to 255. Taking a set of correction values ​​rectify_rr, rectify_gr, rectify_br as an example, the specific formula is as follows:

[0101] maxv = max(rectify_rr,rectify_gr,rectify_br) (11)

[0102] rectify_rr = rectify_rr / maxv×255 (12)

[0103] rectify_gr = rectify_gr / maxv×255 (13)

[0104] rectify_br = rectify_br / maxv×255 (14)

[0105] In step 503, at least a plurality of sets of correction values ​​are used to correct the value of each pixel in each color channel.

[0106] Specifically, a weighted sum of multiple groups of correction values ​​and corresponding weights is calculated to obtain the corrected value of the pixel point on each color channel.

[0107] Specifically, the ratio of the value of each color channel of each pixel to the value of all color channels is calculated as the weighted weight of the color channel. The weighted weights of each color channel and the corresponding correction value are used to weight the value of the corrected pixel in each color channel.

[0108] For example, the weighted weight of each color channel is calculated according to the proportion of each color channel org_label_r, org_label_g, and org_label_b of the pixel. The specific formula is as follows:

[0109] pr = org_label_r / (org_label_r+org_label_g+org_label_b) (15)

[0110] pg = org_label_g / (org_label_r+org_label_g+org_label_b) (16)

[0111] pb = org_label_b / (org_label_r+org_label_g+org_label_b) (17)

[0112] If a set of correction values ​​rectify_rr, rectify_gr, rectify_br corresponding to the color channel R are not calculated, the weighted weight pr corresponding to the color channel R is zero.

[0113] Correspondingly, if a set of correction values ​​rectify_rg, rectify_gg, rectify_bg corresponding to the color channel G are not calculated, the weighted weight pg corresponding to the color channel G is zero.

[0114] If a set of correction values ​​rectify_rg, rectify_gg, rectify_bg corresponding to color channel B are not calculated, the weighted weight pb corresponding to color channel B is zero.

[0115] In a non-limiting embodiment, the average difference between the reference pixel and the pixel is calculated as the first correction value; and the value of each pixel in each color channel is corrected using multiple sets of correction values ​​and the first correction value.

[0116] Specifically, by calculating the mean difference between the RGB value of the second image org_label_img and the RGB value of the first image org_img at each pixel, the first correction value mean_diffs is obtained to represent the average difference of each pixel in all color channels. The first correction value mean_diffs is added to the second image org_label_img to obtain a correction image rectify_mean_diffs. This correction will not cause the ratio to be unable to be calculated due to the zero division problem.

[0117] Accordingly, in step 503, the values ​​of each pixel in each color channel are corrected using the multiple groups of correction values ​​and the first correction value.

[0118] The weighted weight corresponding to the first correction value is the absolute value of the difference between the sum of the weighted weights corresponding to each color channel and 1, that is, pmean_diff=(1-pr-pg-pb).

[0119] Assume that the first correction values ​​of each color channel on the correction map rectify_mean_diffs are rectify_mean_diffs_r, rectify_mean_diffs_g, rectify_mean_diffs_b. The weights of rectify_mean_diffs_r, rectify_mean_diffs_g, rectify_mean_diffs_b are pmean_diff.

[0120] The correction value of the pixel in each color channel is shown in the following formula:

[0121] rectify_r=pr×rectify_rr+pg×rectify_rg+pb×rectify_rb+(1-pr-pg-pb)×rectify_mean_diffs_r(18)

[0122] rectify_g=pr×rectify_gr+pg×rectify_gg+pb×rectify_gb+(1-pr-pg-pb)×rectify_mean_diffs_g(19)

[0123] rectify_b=pr×rectify_br+pg×rectify_bg+pb×rectify_bb+(1-pr-pg-pb)×rectify_mean_diffs_b(20)

[0124] The embodiments of the present invention consider more comprehensive color channel ratio relationships, such as r / g, b / g, g / r, g / b, b / r, b / g and other ratio relationships, and perform weighted calculation of the values ​​of pixel points in each color channel. In this way, the color cast correction in areas with obvious color tendencies can be better handled without introducing obvious artifacts, thereby improving the effect of image color cast correction.

[0125] In addition, the image correction device 30 can also perform detail correction on the second video to improve the problem of local detail loss in each video frame in the second video.

[0126] It should be noted that the image correction device 30 can selectively perform detail correction 401, color cast correction 402 and / or flicker correction 403, and the above three correction processes can be independently executed in parallel or in series, and the present application does not impose any limitation on this.

[0127] Please refer to Figure 6 , Figure 6 An image processing device 60 is shown, and the image processing device 60 may include:

[0128] An acquisition module 601 is used to acquire a first video and a second video, where the second video is a video obtained by enhancing the first video;

[0129] An overlapping region determining module 602 is used to determine a first overlapping region having the greatest structural similarity between two adjacent frames in the first video, where the two adjacent frames include a first current frame and a previous frame;

[0130] The pixel difference calculation module 603 is used to calculate a first pixel difference between a pixel change amount in a current frame and a pixel change amount in a previous frame of the first video and the second video;

[0131] The correction module 604 is configured to calculate a correction degree according to the first pixel difference, and correct each pixel point in the first overlapping area of ​​the second current frame in the second video according to the correction degree and the first pixel difference.

[0132] In a specific implementation, the above-mentioned image processing device 60 may correspond to a chip with an image processing function in a terminal device, such as a system-on-a-chip (SOC), a baseband chip, etc.; or correspond to a chip module with an image processing function in a terminal device; or correspond to a chip module with a data processing function chip, or correspond to a terminal device.

[0133] For other related descriptions about the image processing device 60 , reference may be made to the related descriptions in the aforementioned embodiments, which will not be repeated here.

[0134] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component of the chip module (such as a chip, circuit module, etc.) or in different components, or at least some of the modules / units can be implemented in the form of software programs. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0135] The present application also discloses a storage medium, which is a computer-readable storage medium and stores a computer program, which can be executed when the computer program is run. Figures 1 to 3 The steps of the method shown in . The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. The storage medium may also include a non-volatile memory (non-volatile) or a non-transitory memory, etc.

[0136] Please refer to Figure 7 The embodiment of the present application also provides a hardware structure diagram of a communication device. The device includes a processor 701, a memory 702 and a transceiver 703.

[0137] Processor 701 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. Processor 701 may also include multiple CPUs, and processor 701 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The processor here may refer to one or more devices, circuits, or processing cores for processing data (such as computer program instructions).

[0138] The memory 702 may be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, and the embodiments of the present application do not impose any restrictions on this. The memory 702 may exist independently (in this case, the memory 702 may be located outside the device or inside the device), or it may be integrated with the processor 701. Among them, the memory 702 may contain a computer program code. The processor 701 is used to execute the computer program code stored in the memory 702, thereby realizing the method provided in the embodiments of the present application.

[0139] The processor 701, the memory 702 and the transceiver 703 are connected via a bus. The transceiver 703 is used to communicate with other devices or communication networks. Optionally, the transceiver 703 may include a transmitter and a receiver. The device used to implement the receiving function in the transceiver 703 can be regarded as a receiver, and the receiver is used to perform the receiving step in the embodiment of the present application. The device used to implement the sending function in the transceiver 703 can be regarded as a transmitter, and the transmitter is used to perform the sending step in the embodiment of the present application.

[0140] when Figure 7 The schematic diagram of the structure shown in FIG. 1 is used to illustrate the structure of the terminal device involved in the above embodiment. The processor 701 is used to control and manage the actions of the terminal device. For example, the processor 701 is used to support the terminal device to execute Figure 1 , Figure 3 and Figure 5 The processor 701 may communicate with other network entities, for example, the network devices described above, through the transceiver 703. The memory 702 is used to store program codes and data of the terminal device.

[0141] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0142] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0143] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0144] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0145] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means.

[0146] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0150] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present application.

[0151] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application shall be subject to the scope defined by the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first video and a second video, where the second video is a video obtained by enhancing the first video; Determine a first overlapping region having the greatest structural similarity between two adjacent frames in the first video, the two adjacent frames including a first current frame and a previous frame; Calculating a first pixel difference between a pixel change amount in a current frame and a pixel change amount in a previous frame of the first video and the second video; A correction degree is calculated according to the first pixel difference, and each pixel point of the second current frame in the second video within the first overlapping area is corrected according to the correction degree and the first pixel difference.

2. The image processing method according to claim 1, characterized in that: The calculating of a first pixel difference between a pixel change amount of the first video and the second video in a current frame and a pixel change amount in a previous frame comprises: Calculating a second pixel difference between the second current frame and the first current frame in the first overlapping area; Calculating a third pixel difference between a previous frame of the second current frame and a previous frame of the first current frame in the first overlapping area; A difference between the second pixel difference and the third pixel difference is calculated as the first pixel difference.

3. The image processing method according to claim 1, characterized in that: Calculating the correction degree according to the first pixel difference comprises: The correction degree is calculated using the first pixel difference and a smoothing function. The greater the first pixel difference, the greater the correction degree.

4. The image processing method according to claim 1, characterized in that: The correcting each pixel of the second current frame in the second video in the first overlapping area according to the correction degree and the first pixel difference comprises: Calculate the parameters of the two-dimensional Gaussian surface according to the position of each pixel in the first overlapping area to obtain the weight value of each pixel in the first overlapping area; Weighting is performed according to the weight value of each pixel point in the second current frame, the correction degree corresponding to each pixel, and the first pixel difference corresponding to each pixel point to correct each pixel point in the second current frame.

5. The image processing method according to claim 1, characterized in that: The correcting each pixel of the second current frame in the second video in the first overlapping area according to the correction degree and the first pixel difference comprises: Calculating a third pixel difference between a previous frame of the second current frame and a previous frame of the first current frame in the first overlapping area; weighting the first pixel difference corresponding to each pixel point according to the correction degree of each pixel point in the second current frame to obtain a weighted result; Calculate the sum of the pixel value of each pixel point in the second current frame, the third pixel difference and the weighted result to obtain a corrected pixel point.

6. The image processing method according to claim 1, characterized in that: The step of determining the overlapping region with the greatest structural similarity between two adjacent frames in the first video comprises: Slide the first current frame on its previous frame, and calculate the structural similarity of the overlapping area after each slide; The overlapping region with the greatest structural similarity is selected as the first overlapping region.

7. The image processing method according to claim 1, characterized in that: Also includes: Respectively calculating the ratios of the values ​​of each color channel of each reference pixel in the first current frame to the values ​​of each color channel to obtain multiple sets of ratios; respectively weighting the value of each pixel point in each color channel in the second current frame and the ratio corresponding to the color channel to obtain multiple groups of correction values; The value of each pixel in each color channel is corrected by at least using the multiple groups of correction values.

8. An image processing device, characterized in that: include: An acquisition module, used to acquire a first video and a second video, where the second video is a video obtained by enhancing the first video; An overlapping region determination module, configured to determine a first overlapping region having the greatest structural similarity between two adjacent frames in the first video, wherein the two adjacent frames include a first current frame and a previous frame; A pixel difference calculation module, used to calculate a first pixel difference between a pixel change amount in a current frame and a pixel change amount in a previous frame of the first video and the second video; A correction module is used to calculate a correction degree according to the first pixel difference, and correct each pixel point of the second current frame in the second video within the first overlapping area according to the correction degree and the first pixel difference.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are executed.

10. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the image processing method according to any one of claims 1 to 7 are performed.

Images