Image processing method and device and electronic equipment
By acquiring and fusing global and local motion vectors in image matching and motion estimation algorithms, the problem of unsmoothing data of the same pixel point is solved, and the quality of images and videos is improved.
Patent Information
- Application Number
- CN202310707024.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-14
- Publication Date
- 2024-12-20
AI Technical Summary
In image matching and motion estimation algorithms, the data of the same pixel point is not smooth enough, resulting in the results of global alignment and local alignment that cannot be fused, and the image quality is poor.
The plane calibration matrix is obtained by obtaining the current frame image and the reference frame image for global alignment processing, and local alignment processing is performed to obtain the local motion vector. These results are used for motion compensation processing to obtain the global and local match degrees, and according to the two Relationships combine global motion vectors and local motion vectors.
Improves the smoothness and accuracy of image processing results and improves image and video quality.
Smart Images
Figure CN119168846A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and electronic equipment. Background Art
[0002] In current image matching and motion estimation algorithms, one of the two processing strategies, global alignment or local alignment, is usually used. The global alignment solution can solve the problem of inaccuracy between image sequences caused by camera motion. Although the local alignment process is more accurate in processing local details, the overall effect is poor and the frame is discontinuous. However, in the specific implementation process, the data at the same pixel point is often not smooth enough during the global and local matching process, resulting in the inability to merge the results of global alignment and local alignment, and poor image quality. Summary of the invention
[0003] In view of this, the present application provides an image processing method, device and electronic device, the main purpose of which is to solve the problem that the data at the same pixel point is not smooth enough, which often occurs in the current matching process.
[0004] In a first aspect, the present application provides an image processing method, comprising:
[0005] Acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame;
[0006] Performing global alignment processing on the current frame image and the reference frame image to obtain a plane calibration matrix, and performing local alignment processing on the current frame image and the reference frame image to obtain a local motion vector;
[0007] Performing motion compensation processing on the current frame image according to the plane calibration matrix and the local motion vector to obtain a first compensation result and a second compensation result;
[0008] Using the first compensation result and the second compensation result to perform stereo matching processing with the reference frame image respectively to obtain a global matching degree and a local matching degree;
[0009] According to the global matching degree and the local matching degree, a fusion process is performed on the global motion vector and the local motion vector; wherein the global motion vector is obtained through the plane calibration matrix.
[0010] In a second aspect, the present application provides an image processing device, comprising:
[0011] An acquisition unit is configured to acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame;
[0012] a processing unit configured to perform global alignment processing on the current frame image and the reference frame image to obtain a plane calibration matrix, and perform local alignment processing on the current frame image and the reference frame image to obtain a local motion vector;
[0013] a compensation unit configured to perform motion compensation processing on the current frame image according to the plane calibration matrix and the local motion vector to obtain a first compensation result and a second compensation result;
[0014] A calculation unit is configured to use the first compensation result and the second compensation result to perform stereo matching processing with the reference frame image respectively to obtain a global matching degree and a local matching degree;
[0015] A fusion unit is configured to fuse the global motion vector and the local motion vector according to the global matching degree and the local matching degree; wherein the global motion vector is obtained through the plane calibration matrix.
[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method described in the first aspect.
[0017] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the image processing method described in the first aspect when executing the computer program.
[0018] By means of the above technical scheme, the present application provides an image processing method, device and electronic device, firstly obtaining the current frame image and the reference frame image, in the time sequence, the reference frame image includes one or more image frames before the current frame. The motion vector and the subsequent fusion between the two frames can be calculated through the current frame image and the reference frame image. Then the current frame image and the reference frame image are globally aligned to obtain a plane calibration matrix, and the local alignment is performed to obtain a local motion vector, and then motion compensation is performed according to the plane calibration matrix and the local motion vector to obtain a first compensation result and a second compensation result, and the first compensation result and the second compensation result are used to perform stereo matching with the reference frame image respectively to obtain a global matching degree and a local matching degree. Finally, according to the global matching degree and the local matching degree, the global motion vector and the local motion vector are fused, wherein the global motion vector is obtained by the plane calibration matrix. The present application is based on the matching degree of the image after motion compensation, and according to the relationship between the global matching degree and the local matching degree, the global motion vector and the local motion vector are fused, so that the image processing result is smoother and more accurate, and the image and video quality are improved.
[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0022] Figure 1 A schematic diagram of a process flow of an image processing method provided in an embodiment of the present application is shown;
[0023] Figure 2 A schematic diagram showing a flow chart of another image processing method provided in an embodiment of the present application is shown;
[0024] Figure 3 A schematic diagram showing a flow chart of another image processing method provided in an embodiment of the present application is shown;
[0025] Figure 4 A schematic diagram showing a flow chart of another image processing method provided in an embodiment of the present application is shown;
[0026] Figure 5 A schematic structural diagram of an image processing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0027] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0028] The main scenario of the solution proposed in this embodiment is as a key module of the image signal processing (ISP) in the image processor, in the multi-frame processing and video processing process of the image pipeline. In this process, high-quality global alignment selection and high-quality fusion of local motion estimation are required, and the image processing method proposed in this embodiment can be used, and this method is open to all scenes, including image processing in single-frame shooting mode, image processing in multi-frame shooting mode, image processing in preview mode, image processing in video recording mode, night scene shooting, image processing in video mode, and other shooting and video usage scenarios. All of these methods can adopt this method.
[0029] In current image matching and motion estimation algorithms, one of the two processing strategies, global alignment or local alignment, is usually used. The global alignment solution can solve the problem of inaccuracy between image sequences caused by camera motion. Although the local alignment process is more accurate in processing local details, the overall effect is poor and the frame is discontinuous. However, in the specific implementation process, the data at the same pixel point is often not smooth enough during the global and local matching process, resulting in the inability to merge the results of global alignment and local alignment, and poor image quality.
[0030] In order to improve the problem that the data at the same pixel point is not smooth enough, which often occurs in the current matching process, this embodiment provides an image processing method, such as Figure 1 As shown, the method includes:
[0031] S101, obtaining a current frame image and a reference frame image.
[0032] First, the basic idea of motion estimation technology is to divide each frame of the image sequence into non-overlapping macroblocks, and assume that the displacement of all pixels in the macroblock is the same. Then, in this step, for each macroblock to a given specific search range of the reference frame, a block that is most similar to the current block is found according to a certain matching criterion, that is, a matching block. The relative displacement between the matching block and the current block is the motion vector. In this step, its main purpose is to determine the two frames of frame images used to infer the motion vector, so as to facilitate the subsequent scheme to process and merge the motion vector.
[0033] In terms of time sequence, the reference frame image includes one or more image frames before the current frame.
[0034] S102 , performing global alignment processing on the current frame image and the reference frame image to obtain a plane calibration matrix, and performing local alignment processing on the current frame image and the reference frame image to obtain a local motion vector.
[0035] First, the current frame image and the reference frame image are globally aligned to obtain a plane calibration matrix, where the plane calibration matrix can be selected as a Homography matrix. More specifically, the judgment of the quality and confidence of the global alignment of the image can actually be divided into quantitative analysis on a data set with annotated data Ground Truth and actual application scenarios without Ground Truth. Then, the current frame image and the reference frame image are locally aligned to obtain a local motion vector.
[0036] Through the above steps, the global alignment and local alignment of the current frame image and the reference frame image are completed, and the plane calibration matrix and local motion vector are obtained. In addition, by obtaining the confidence level, the entire image processing result can also be based on higher accuracy and authenticity.
[0037] S103 , performing motion compensation processing on the current frame image according to the plane calibration matrix and the local motion vector to obtain a first compensation result and a second compensation result.
[0038] The plane calibration matrix and the local motion vector are used to perform motion compensation on the current frame image and the reference frame image respectively, to obtain a first compensation result and a second compensation result. The compensation result here is usually the image of the corresponding frame after compensation.
[0039] S104 , using the first compensation result and the second compensation result to perform stereo matching processing with the reference frame image respectively to obtain a global matching degree and a local matching degree.
[0040] The first compensation result and the second compensation result are used to perform stereo matching processing with the reference frame image respectively, and then the global matching degree and the local matching degree are calculated. There are multiple options for the matching degree, such as the SAD value of the sum of the absolute values of the differences between the corresponding pixel blocks of the two frames of images calculated after the stereo matching processing, or the matching degree can be calculated by collecting image features.
[0041] S105 , fusing the global motion vector and the local motion vector according to the global matching degree and the local matching degree.
[0042] Among them, the global motion vector is obtained through the plane calibration matrix.
[0043] The global matching degree and the local matching degree are used as the basis for judging the fusion of the image alignment results, and the global motion vector and the local motion vector are fused according to the relationship between the global matching degree and the local matching degree, so that the image processing result is smoother and more accurate.
[0044] In this embodiment, the current frame image and the reference frame image are first obtained, and the motion vector between the two frames and the subsequent fusion can be calculated through the current frame image and the reference frame image. Then, the current frame image and the reference frame image are globally aligned to obtain a plane calibration matrix, and local alignment is performed to obtain a local motion vector, and motion compensation is performed based on the plane calibration matrix and the local motion vector to obtain a global matching degree and a local matching degree. Finally, based on the global matching degree and the local matching degree, the global motion vector and the local motion vector are fused, wherein the global motion vector is obtained through the plane calibration matrix. This application is based on the matching degree of the image after motion compensation, and based on the relationship between the global matching degree and the local matching degree, the global motion vector and the local motion vector are fused, so that the image processing result is smoother and more accurate, and the image and video quality are improved.
[0045] Optionally, motion compensation is performed using a plane calibration matrix and a local motion vector to obtain a global match and a local match, including: performing motion compensation processing on the current frame image according to the plane calibration matrix to obtain a first compensation result; performing motion compensation processing on the current frame image according to the local motion vector to obtain a second compensation result; and using the first compensation result and the second compensation result to perform stereo matching processing with the reference frame image respectively to obtain a global match and a local match.
[0046] In this embodiment, the motion compensation process is introduced. First, the current frame image needs to be motion compensated according to the plane calibration matrix to obtain a first compensation result. Then, the current frame image needs to be motion compensated according to the local motion vector to obtain a second compensation result. The first compensation result and the second compensation result are used to perform stereo matching with the reference frame image to obtain a global matching degree and a local matching degree. The advantages and disadvantages and trade-offs of global alignment and local alignment are judged to improve image accuracy.
[0047] Optionally, the matching degree may include a SAD value; and according to the global matching degree and the local matching degree, the global motion vector and the local motion vector are fused, including: according to the size relationship between the global SAD value and the local SAD value, the global motion vector and the local motion vector are fused.
[0048] In this embodiment, taking the SAD value as the matching degree as an example, the global motion vector and the local motion vector are fused according to the global matching degree and the local matching degree, that is, the global motion vector and the local motion vector are fused according to the size relationship between the global SAD value and the local SAD value. Among them, stereo matching is performed on the reference frame image to obtain the global SAD value and the local SAD value, which means that the reference frame image is processed by the SAD algorithm based on the plane calibration matrix and the local motion vector, so as to obtain the global SAD value and the local SAD value. The idea is the sum of the absolute values of the differences between the corresponding pixel blocks between the two frames of images, and the SAD value is the sum of the absolute values of the differences between the corresponding pixel blocks between the two frames of images. It is generally believed that the smaller the SAD is, the higher the similarity between the two pixels (or small image areas), which means that the alignment effect at this position is better, so the comparison of SAD can be used to judge the pros and cons and trade-offs between global alignment and local alignment.
[0049] Optionally, the global motion vector and the local motion vector are fused according to the size relationship between the global SAD value and the local SAD value, including: when the sum of the global SAD value and a preset compensation threshold is less than the local SAD value, and the global SAD value is less than the preset compensation threshold, determining the fusion weight and fusing the global motion vector and the local motion vector.
[0050] In this embodiment, when the global SAD value plus a threshold is still less than the local SAD value, and the global SAD value itself is less than a threshold, it can be said that the effect of the global alignment result is better than the effect of the current local alignment. It is generally believed that the smaller the SAD, the higher the similarity between two pixels (or small image areas), which means that the alignment effect at this position is better, so the comparison of SAD can be used to judge the pros and cons and trade-offs between global alignment and local alignment, thereby improving image accuracy.
[0051] At the same time, there are some special cases in the above embodiments, such as completely overexposed areas and completely flat areas. In fact, the SAD calculated for these two areas will be smaller due to the details and content of the image itself. In this case, choosing global alignment or local alignment will not make any special difference in the end.
[0052] Optionally, determining a fusion weight to fuse the global motion vector and the local motion vector includes: performing weighted fusion of the global motion vector and the local motion vector based on a lookup table and a weight table.
[0053] In this embodiment, when the global motion vector and the local motion vector are fused, the global motion vector and the local motion vector may be weightedly fused based on a lookup table and a weight table, so that the result is more accurate.
[0054] Optionally, after obtaining the global matching degree and the local matching degree, the method further comprises: performing morphological processing on the global matching degree and the local matching degree.
[0055] In this embodiment, morphological processing is performed on the matching degree to obtain a better morphological contour in preparation for the subsequent fusion step.
[0056] Optionally, morphological processing is performed on the global matching degree and the local matching degree, including: dilation processing and erosion processing are performed on the global matching degree and the local matching degree.
[0057] In this embodiment, dilation (usually ⊕) is one of the basic operations in mathematical morphology. Erosion (usually Representation) is one of the two basic operations in morphological image processing, which was originally developed for binary images, but has now been extended to grayscale images and then to complete lattices. All other morphological operations are based on this. It was originally defined for binary images, later extended to grayscale images, and later to complete lattices. Erosion operations usually use structural elements to detect and reduce shapes contained in the input image. The matching degree is expanded and eroded to obtain a better morphological outline in preparation for the subsequent fusion step.
[0058] Furthermore, in order to better reflect the specific steps of the technical solution proposed in this embodiment, a specific implementation method is given. This embodiment is consistent with the idea of the previous embodiment, but the steps and details of the solution are introduced more comprehensively. In this example, the matching degree takes the SAD value as an example, and includes 4 parts. In the first part, the global alignment processing and local alignment processing of the current frame image and the reference frame image are performed, such as Figure 2 As shown, including:
[0059] S201, performing global alignment processing on the current frame image and the reference frame image to obtain a Homograhy matrix.
[0060] In this step, while obtaining the Homograohy matrix, the confidence can also be obtained at the same time to judge its reliability. For example, the following global image confidence metric can be constructed:
[0061] Global Image Alignment Confidence=Feature Confidence*FeatureCoverage
[0062] Through the above steps, the Homography matrix can be obtained through global alignment processing and the confidence can be obtained at the same time as a guarantee of image reliability. Among them, the global motion vector of the image can also be calculated through the Homography matrix.
[0063] S202, completing local alignment processing on the current frame image and the reference frame image to obtain a local motion vector.
[0064] After the above discussion on global image alignment, the same is true for local alignment. For local alignment, the following confidence metric can be constructed:
[0065] Local Image Alignment Confidence=Feature Confidence*MatchingConfidence*Aggregation Confidence*Refinement Confidence
[0066] Through the above steps, the local motion vector can be obtained by completing the local alignment process on the current frame image and the reference frame image, and the local confidence can be obtained.
[0067] In the second part, motion compensation processing is required, such as Figure 3 As shown, including:
[0068] S301, performing motion compensation processing on the current frame image according to the plane calibration matrix to obtain a first compensation result.
[0069] This part performs global compensation on the current frame image according to the plane calibration matrix obtained by global alignment to obtain the first compensation result. The input data frames are the target image and the reference image. Here is the plane calibration matrix Homography for global alignment of the reference to the target image. First, a small area is obtained on the reference image. Here, 3*3 is taken as an example. The motion vector of each pixel point of the 3*3 patch is calculated according to the coordinate information and the homography matrix. This is done by calculating the conversion from homography to motion vector, where x_in&y_in is the xy coordinate information of the pixel in the patch, and the conversion from homography to motion vector is performed. Finally, use x_out and y_out to extract reference_image_patch_global_3x3 from reference, the extraction method is: reference_image_patch_global_3x3 = warp (reference_image, x_out, y_out), where warp is a conventional operation of image warping using motion vectors. Image warping is the process of affine and transmission transformation of images by motion vectors or image perspective transformation matrix Homography during digital processing of images. Image warping can correct the affine and transmission distortion of images, and the same technology is also applicable to video.
[0070] The most obvious way to transform a digital image is forward mapping. This applies the transformation directly to the source image, typically generating unevenly spaced points, which are then interpolated to generate the required regularly spaced pixels. For injective transformations, reverse mapping is also available. This applies the inverse transformation to the target pixels to find the locations in the source image that caused them to be unevenly spaced. Estimating them from the source image pixels requires interpolating the source image. To get an idea of what kind of distortion occurs between consecutive images, optical flow estimation techniques can be used to generate motion vectors or image projection transformation matrices such as Homography.
[0071] The principle of backward mapping Inverse Warping is: traverse each point p_destination in the destination image, multiply it by the affine matrix from the destination image to the source image, and get the corresponding point p_source of this point in the source image, and make the pixel value of p_destination equal to the value of p_source. If the coordinates of p_source are not integers, they are approximated by interpolation approximation, so there will be no Forward Warping problem.
[0072] S302, performing motion compensation processing on the current frame image according to the local motion vector to obtain a second compensation result.
[0073] In this step, motion compensation is performed based on the local motion vector obtained by local alignment to obtain the second compensation result. The input data frames are target image and reference image. Here we have motion vectors mvx_all and mvy_all for local alignment of reference to targetimage; by directly using motion vectors mvx_all and mvy_all for motion compensation, we obtain the motion compensated image blocks target_image_patch_local_3x3 and reference_image_patch_local_3x3 of target image and reference on the target size patch. The compensation method here is:
[0074] reference_image_patch_global_3x3=warp(reference_image,x_out,y_out)
[0075] Among them, x_out&y_out are the motion vectors corresponding to the compensation position 3*3, which come from the pixel values after compensation based on mvx_all&mvy_all; the warp processing method is also a conventional operation, that is, the process of image warping interpolation based on the coordinates after motion compensation.
[0076] S303 , performing stereo matching processing with the reference frame image using the first compensation result and the second compensation result to obtain a global SAD and a local SAD.
[0077] Since the matching degree in this example uses SAD information as a measure, the SAD value is calculated based on the first compensation result (such as target_image_patch_global_3x3) and the second compensation result (such as reference_image_patch_global_3x3) obtained above. First, the global matching degree, that is, the global SAD value, is calculated.
[0078] SAD_global=SAD_calc(target_image_patch_global_3x3,
[0079] reference_image_patch_global_3x3)
[0080] The SAD calculation is a general calculation process and can be found here if needed: In digital image processing, the sum of absolute differences (SAD) is a measure of similarity between image patches. It is calculated by taking the absolute difference between each pixel in the original patch and the corresponding pixel in the patch used for comparison. These differences are summed to create a simple block similarity measure, namely the L1 norm of the difference image or the Manhattan distance between the two image patches. The sum of absolute differences can be used for various purposes such as object recognition, generation of disparity maps for stereo images, and motion estimation for video compression. This example uses the sum of absolute differences to identify which part of the search image is most similar to the template image.
[0081] Then comes the calculation of the local SAD value. The core calculation method of this part is the same as above, and the input is the locally aligned image information; based on the target_image_patch_local_3x3 and reference_image_patch_local_3x3 obtained above, we can calculate the SAD of the global alignment of the image:
[0082] SAD_local=SAD_calc(target_image_patch_local_3x3,
[0083] reference_image_patch_local_3x3)
[0084] In the third part, the global SAD and local SAD are morphologically processed. The global SAD_global and local SAD_local information of each pixel position obtained by the previous calculation can be used to obtain better morphological contours through morphological processing. Specifically, morphological processing may include erosion and expansion operations; its enablement is controlled by enabling SAD morphological processing through gf_sad_morph_en; the expansion and erosion (erosion) operations in image morphological operations are the two main operations we use here, and the specific calculation process is as follows: Dilation (usually used ⊕) is one of the basic operations in mathematical morphology. It was originally developed for binary images, but has now been extended to grayscale images and then to complete grids. The dilation operation usually uses structural elements to detect and expand the shapes contained in the input image.
[0085] Erosion (usually Representation) is one of the two fundamental operations in morphological image processing (the other is dilation), and all other morphological operations are based on it. It was originally defined for binary images, later extended to grayscale images, and later to complete lattices. Erosion operations usually use a structural element to detect and reduce shapes contained in the input image.
[0086] In engineering practice, we can perform morphological processing of SAD_global and SAD_local by performing dilation and erosion operations on SAD_global and SAD_local, or erosion and dilation operations. The dilation and erosion operations will first fill the discontinuous areas of the map, and then use erosion to reduce the edges. The erosion and dilation operations will first erode and delete the discontinuous areas, and then use dilation to fill the edges. The two combinations can be adjusted according to the actual engineering and effect tuning conditions. The window size of the erosion and dilation operations can be defined downward within the defined maximum size range. For example, a 13*13 window is defined, and dilation and erosion operations less than or equal to 13*13 can be adjusted during actual operation. This part is also flexibly performed in combination with the effect.
[0087] After the above expansion and erosion operations, we perform better morphological processing on the SAD map obtained by SAD calculation to prepare for subsequent fusion.
[0088] In the fourth part, the size relationship between the global SAD and the local SAD calculated in the previous order is judged, so as to realize the fusion of the global motion vector and the local motion vector. Figure 4 As shown, including:
[0089] S401, obtaining a global SAD value after morphological processing and a local SAD value after morphological processing.
[0090] S402. Determine the fusion weight according to the magnitude relationship between the global SAD value and the local SAD value.
[0091] S403. Based on the fusion weight, fuse the global motion vector and the local motion vector based on the look-up table and the weight table.
[0092] Based on the previous SAD calculation results, through the previous results, combined with the threshold and the formula, perform the fusion process of global and local alignment. When the global SAD (global SAD value) plus a threshold is still less than the local SAD (local SAD value), and the global SAD itself is less than a threshold, at this time, it can be considered that the effect of the global alignment result is better than the current local alignment effect. The global alignment result should be preferentially selected as the alignment result of this pixel at this pixel position. The core calculation formula is as follows:
[0093] When (global SAD + sad_detector_comparison_thr < local SAD) && (global SAD < sad_detect_small_global_thr), select the global alignment result, that is, when using sad_detect_global_blend_weight, give a greater weight to the global alignment (Global mvx & Global mvy).
[0094] Among them, global SAD represents the SAD calculation result of the current position pixel (or small image area) after motion compensation using the motion vectors global mvx and global mvy at the current pixel position converted by the homography matrix of global alignment and the current position pixel (or small image area) of the reference frame. Among them, the local SAD identification number is the SAD calculation result of the current position pixel (or small image area) after motion compensation using the local motion vectors local mvx and local mvy of local alignment and the current position pixel (or small image area) of the reference frame.
[0095] Among them, the smaller the SAD is, the higher the similarity between the two pixels (or small image areas), which means that the alignment effect at this position is better, so the SAD comparison is used to judge the pros and cons and trade-offs between global alignment and local alignment. There are also some special cases here, such as completely overexposed areas and completely flat areas. In fact, the SAD calculated for these two areas will be smaller due to the details and content of the image itself. In this case, choosing global alignment or local alignment will not make any special difference in the end. Therefore, it can be considered that such scenes with insufficient image content can also be effectively compatible by relying on SAD, that is, by judging the following thresholds to confirm whether to select the global alignment result:
[0096] global SAD+sad_detector_comparison_thr <local SAD)&&(global SAD<sad_detect_small_global_thr)
[0097] Range:[0,1024]
[0098] Default:0
[0099] After completing the above comparison, the global alignment and local alignment can be fused based on the weight sad_detect_global_blend_weight, where sad_detect_global_blend_weight is the weight of the fusion of global alignment and local alignment; the fusion form is:
[0100] fusion_mvx=(sad_detect_global_blend_weight*mvx_global+(1024-sad_detect_global_blend_weight)*mvx_local) / 1024;
[0101] fusion_mvy=(sad_detect_global_blend_weight*mvy_global+(1024-sad_detect_global_blend_weight)*mvy_local) / 1024;
[0102] Where fusion_mvx & fusion_mvy are the motion vectors in the x & y directions after the fusion of global alignment and local alignment;
[0103] Among them, mvx_global&mvy_global is the global alignment motion vector information of the current pixel converted by the global alignment homography matrix;
[0104] Among them, mvx_local&mvy_local is the local alignment motion vector information of the current pixel;
[0105] After fusion, it is divided by 1024 for normalization. For the convenience of hardware implementation, it can also be replaced by right shifting 10 bits. If different weight fixed-point precision is used, such as 16384 corresponding to 14 bits, then here is:
[0106] fusion_mvx=(sad_detect_global_blend_weight*mvx_global+(16384-sad_detect_global_blend_weight)*mvx_local) / 16384;
[0107] fusion_mvy=(sad_detect_global_blend_weight*mvy_global+(16384-sad_detect_global_blend_weight)*mvy_local) / 16384;
[0108] The corresponding final step is to divide by 16384 for normalization, or use right shift 14 bits to achieve better hardware friendliness.
[0109] After completing the above comparison, the global alignment and local alignment can be fused based on the lookup table, where SADDetect Global Blend Weight LUT is the weight table for the fusion of global alignment and local alignment. The above threshold-based selection is a binary selection, but in practice, global alignment and local alignment need to be weighted fused to obtain smooth global alignment and local alignment selection and fused motion vector information. The SAD Detect Global Blend WeightLUT lookup table is used to record the weights of the weighted fusion of global alignment and local alignment.
[0110] Through the above four-part image processing process, this embodiment fuses the global motion vector and the local motion vector based on the matching degree of the motion compensated image and the relationship between the global matching degree and the local matching degree, so that the image processing result is smoother and more accurate, thereby improving the image and video quality.
[0111] Further, as Figures 1 to 4The specific implementation of the method shown in the embodiment provides an image processing device, such as Figure 5 As shown, the device includes: an acquisition unit 51, a processing unit 52, a compensation unit 53, a calculation unit 54, and a fusion unit 55.
[0112] An acquisition unit 51 is configured to acquire a current frame image and a reference frame image, wherein the reference frame image includes one or more image frames before the current frame in terms of time sequence;
[0113] The processing unit 52 is configured to perform global alignment processing on the current frame image and the reference frame image to obtain a plane calibration matrix, and perform local alignment processing on the current frame image and the reference frame image to obtain a local motion vector;
[0114] A compensation unit 53 is configured to perform motion compensation processing on the current frame image according to the plane calibration matrix and the local motion vector to obtain a first compensation result and a second compensation result;
[0115] A calculation unit 54 is configured to use the first compensation result and the second compensation result to perform stereo matching processing with the reference frame image respectively to obtain a global matching degree and a local matching degree;
[0116] The fusion unit 55 is configured to fuse the global motion vector and the local motion vector according to the global matching degree and the local matching degree; wherein the global motion vector is obtained through the plane calibration matrix.
[0117] In a specific application scenario, the compensation unit 53 is specifically configured to perform motion compensation processing on the current frame image according to the plane calibration matrix to obtain the first compensation result; and perform motion compensation processing on the current frame image according to the local motion vector to obtain the second compensation result.
[0118] In a specific application scenario, the fusion unit 55 is further configured to fuse the global motion vector and the local motion vector according to the magnitude relationship between the global SAD value and the local SAD value.
[0119] In a specific application scenario, the fusion unit 55 is further configured to determine a fusion weight and fuse the global motion vector and the local motion vector when the sum of the global SAD value and the preset compensation threshold is less than the local SAD value, and the global SAD value is less than the preset compensation threshold.
[0120] In a specific application scenario, the fusion unit 55 is further configured to perform weighted fusion on the global motion vector and the local motion vector based on a lookup table and a weight table.
[0121] In a specific application scenario, the processing unit 52 is further configured to perform morphological processing on the global matching degree and the local matching degree.
[0122] In a specific application scenario, the processing unit 52 is further configured to perform expansion processing and erosion processing on the global matching degree and the local matching degree.
[0123] It should be noted that for other corresponding descriptions of the functional units involved in the image processing device provided in this embodiment, reference can be made to Figures 1 to 4 The corresponding description in will not be repeated here.
[0124] Based on the above Figures 1 to 4 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figures 1 to 4 The method shown.
[0125] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.
[0126] Based on the above Figures 1 to 4 The method shown, and Figure 5 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which can be configured on a computer terminal, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figures 1 to 4 The method shown.
[0127] Based on the above Figures 1 to 4 The method shown, and Figure 5 In order to achieve the above-mentioned purpose, the embodiment of the virtual device shown in the embodiment of the present application further provides a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, the signal including the computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the above-mentioned Figures 1 to 4 The method shown.
[0128] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
[0129] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.
[0130] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.
[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the solution of this embodiment, compared with the related art, the global motion vector and the local motion vector are fused based on the matching degree of the image after motion compensation and the relationship between the global matching degree and the local matching degree, so that the image processing result is smoother and more accurate, and the image and video quality is improved.
[0132] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0133] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.
Claims
1. An image processing method, characterized in that: include: Acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame; Performing global alignment processing on the current frame image and the reference frame image to obtain a plane calibration matrix, and performing local alignment processing on the current frame image and the reference frame image to obtain a local motion vector; Performing motion compensation processing on the current frame image according to the plane calibration matrix and the local motion vector to obtain a first compensation result and a second compensation result; Using the first compensation result and the second compensation result to perform stereo matching processing with the reference frame image respectively to obtain a global matching degree and a local matching degree; According to the global matching degree and the local matching degree, a fusion process is performed on the global motion vector and the local motion vector; wherein the global motion vector is obtained through the plane calibration matrix.
2. The method according to claim 1, characterized in that The performing motion compensation processing on the current frame image according to the plane calibration matrix and the local motion vector to obtain the first compensation result and the second compensation result includes: Performing motion compensation processing on the current frame image according to the plane calibration matrix to obtain the first compensation result; Perform motion compensation processing on the current frame image according to the local motion vector to obtain the second compensation result.
3. The method according to claim 1, characterized in that The matching degree includes a SAD value; The fusing process of the global motion vector and the local motion vector according to the global matching degree and the local matching degree includes: The global motion vector and the local motion vector are fused according to the magnitude relationship between the global SAD value and the local SAD value.
4. The method according to claim 3, characterized in that The fusing process of the global motion vector and the local motion vector according to the magnitude relationship between the global SAD value and the local SAD value comprises: When the sum of the global SAD value and the preset compensation threshold is smaller than the local SAD value, and the global SAD value is smaller than the preset compensation threshold, a fusion weight is determined to fuse the global motion vector and the local motion vector.
5. The method according to claim 4, characterized in that Determining a fusion weight to fuse the global motion vector and the local motion vector includes: Based on the lookup table and the weight table, the global motion vector and the local motion vector are weightedly fused.
6. The method according to any one of claims 1 to 5, characterized in that After obtaining the global matching degree and the local matching degree, the method further includes: Morphological processing is performed on the global matching degree and the local matching degree.
7. The method according to claim 6, characterized in that Performing morphological processing on the global matching degree and the local matching degree includes: The global matching degree and the local matching degree are subjected to dilation processing and erosion processing.
8. An image processing device, characterized in that: include: An acquisition unit is configured to acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame; a processing unit configured to perform global alignment processing on the current frame image and the reference frame image to obtain a plane calibration matrix, and perform local alignment processing on the current frame image and the reference frame image to obtain a local motion vector; a compensation unit configured to perform motion compensation processing on the current frame image according to the plane calibration matrix and the local motion vector to obtain a first compensation result and a second compensation result; A calculation unit is configured to use the first compensation result and the second compensation result to perform stereo matching processing with the reference frame image respectively to obtain a global matching degree and a local matching degree; A fusion unit is configured to fuse the global motion vector and the local motion vector according to the global matching degree and the local matching degree; wherein the global motion vector is obtained through the plane calibration matrix.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
11. A chip, characterized in that: The invention comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 7.