Image processing method and device and electronic equipment
The confidence is obtained through global and local alignment processing and the image alignment input information is determined, which solves the problem of insufficient quality confirmation of multi-source alignment information in image processing, and improves processing effect and smoothness.
Patent Information
- Application Number
- CN202310707040.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-14
- Publication Date
- 2024-12-20
Smart Images
Figure CN119168847A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and electronic equipment. Background Art
[0002] In current image matching and motion estimation algorithms, one of the two processing strategies is usually adopted: global alignment or local alignment. The global alignment solution can solve the problem of inaccuracy between image sequences caused by camera motion. Although the local alignment processing is more accurate in processing local details, it has poor overall effect and frame discontinuity. In addition, in the face of multi-source global or local alignment information, there is no unified measurement standard to confirm their quality and selection strategy. Therefore, no matter which processing strategy is adopted in the current image processing method, there will be problems such as insufficient smoothness, switching jitter and discontinuous image quality, resulting in poor processing effect. Summary of the invention
[0003] In view of this, the present application provides an image processing method, device and electronic device, the main purpose of which is to solve the current technical problem that the quality of multi-source global or local alignment information cannot be confirmed, resulting in poor image processing effect.
[0004] In a first aspect, the present application provides an image processing method, comprising:
[0005] Acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame;
[0006] Performing global alignment processing on the current frame image and the reference frame image to obtain a global confidence;
[0007] Performing local alignment processing on the current frame image and the reference frame image to obtain a local confidence;
[0008] Input information for image alignment is determined according to the global confidence and the local confidence.
[0009] In a second aspect, the present application provides an image processing device, comprising:
[0010] An acquisition unit is configured to acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame;
[0011] A first processing unit is configured to perform global alignment processing on the current frame image and the reference frame image to obtain a global confidence;
[0012] A second processing unit is configured to perform local alignment processing on the current frame image and the reference frame image to obtain a local confidence;
[0013] A determination unit is configured to determine input information for image alignment according to the global confidence and the local confidence.
[0014] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method described in the first aspect.
[0015] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the image processing method described in the first aspect when executing the computer program.
[0016] By means of the above technical scheme, the present application provides an image processing method, device and electronic device, which first obtains the current frame image and the reference frame image, and then performs global alignment processing on the current frame image and the reference frame image to obtain the global confidence, and performs local alignment processing on the current frame image and the reference frame image to obtain the local confidence. Finally, according to the global confidence and the local confidence, the input information for image alignment is determined. The present application introduces confidence as a measure of multi-source image alignment information, performs quality confirmation and selection on multi-source global or local image alignment information, obtains better alignment information, and thus makes the image processing results smoother, so that the alignment information can be effectively combined in all scenarios, and solves the technical problem that the global or local alignment information of multiple sources cannot be quality confirmed, resulting in poor image processing effects.
[0017] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0020] Figure 1A schematic diagram of a process flow of an image processing method provided in an embodiment of the present application is shown;
[0021] Figure 2 A schematic diagram showing a flow chart of another image processing method provided in an embodiment of the present application is shown;
[0022] Figure 3 A schematic diagram showing a flow chart of another image processing method provided in an embodiment of the present application is shown;
[0023] Figure 4 A schematic diagram showing a flow chart of another image processing method provided in an embodiment of the present application is shown;
[0024] Figure 5 A schematic structural diagram of an image processing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0025] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0026] The main scenario of the solution proposed in this embodiment is as a key module of image signal processing (ISP) in the image processor, in the multi-frame processing and video processing process of the image pipeline. In this process, high-quality global alignment selection and high-quality fusion of local motion estimation are required, and the image processing method proposed in this embodiment can be adopted, and this method is open to all scenes, including image processing in single-frame shooting mode, image processing in multi-frame shooting mode, image processing in preview mode, image processing in video recording mode, night scene shooting, image processing in video mode, and other shooting and video usage scenarios. All of these methods can adopt this method.
[0027] The existing image matching and motion estimation (Image Alignment & Motion Estimation) algorithms include global multi-frame image alignment algorithms based on feature extraction and feature matching. Among them, the global alignment solution can solve the global image alignment problem between image sequences caused by camera motion; it also includes local alignment algorithms based on block matching or feature matching and cost aggregation. However, in practical applications, there is no unified measurement standard to confirm the quality and selection strategy of global or local alignment information from multiple sources. Therefore, no matter which processing strategy is adopted in the current image processing method, there will be problems such as poor processing effect due to insufficient smoothness, switching jitter and discontinuous image quality. There are problems such as the inability to effectively compare pixel-level quality for image-based global alignment and image-based local alignment information, the inability to effectively select and smoothly switch pixel-level for image-based global alignment information and image-based local alignment information, switching jitter and discontinuous image quality within the frame.
[0028] In order to improve the current technical problem that the quality of multi-source global or local alignment information cannot be confirmed, resulting in poor image processing effect, this embodiment provides an image processing method, such as Figure 1 As shown, the method includes:
[0029] S101, obtaining a current frame image and a reference frame image.
[0030] The current frame image and the reference frame image here are also the two frames of inferred images in the motion estimation algorithm. The relevant information required for image alignment can be obtained by performing global alignment processing and local alignment processing on the two frames of inferred images.
[0031] S102: Perform global alignment processing on the current frame image and the reference frame image to obtain a global confidence.
[0032] First, the current frame image and the reference frame image are globally aligned to obtain the global confidence. During the global alignment process, a plane calibration matrix can also be obtained. The plane calibration matrix here can be selected as a Homography matrix. In the subsequent processing process, the plane calibration matrix will be combined to process the relevant information.
[0033] In terms of time sequence, the reference frame image includes one or more image frames before the current frame;
[0034] S103: Perform local alignment processing on the current frame image and the reference frame image to obtain local confidence.
[0035] The current frame image and the reference frame image are locally aligned to obtain local confidence, wherein the local motion vectors of the two frames of inferred images can also be obtained during the local alignment process.
[0036] S104: Determine input information for image alignment according to the global confidence and the local confidence.
[0037] In this embodiment, the current frame image and the reference frame image are first obtained, and then the current frame image and the reference frame image are globally aligned to obtain the global confidence, and the current frame image and the reference frame image are locally aligned to obtain the local confidence. Finally, according to the global confidence and the local confidence, the input information for image alignment is determined. This embodiment introduces confidence as a measure of multi-source image alignment information, and performs quality confirmation and selection on multi-source global or local image alignment information to obtain better alignment information, thereby making the image processing result smoother, and enabling the alignment information to be effectively combined in all scenarios, thereby solving the technical problem that the global or local alignment information of multiple sources cannot be quality confirmed, resulting in poor image processing effects.
[0038] It should be noted that, in the present embodiment, the input information for image alignment may be the input information of the image alignment module. The working mode of the image alignment module is generally selected as the Auto calculate mode. The working modes of this module include three types. The first is to force the use of the image-based global alignment algorithm without using the gyroscope Gyro-based image alignment algorithm information; the second is to force the use of the Gyro-based global alignment algorithm information without using the image-based alignment algorithm information; the third is the automatic calculation mode. When the image-based alignment algorithm function fails or fails, it switches to the Gyro-based alignment algorithm information, and selects and decides local alignment and global alignment based on the confidence information. The Autocalculate mode is also the third mode.
[0039] Optionally, based on the global confidence, input information for image alignment is determined, including: comparing the global confidence with a global confidence threshold, and when the global confidence is less than the global confidence threshold, obtaining input information for image alignment based on a gyroscope data source; when the global confidence is greater than or equal to the global confidence threshold, obtaining a plane calibration matrix obtained through global alignment processing, and determining the input information for image alignment based on the plane calibration matrix.
[0040] In this embodiment, how to determine the input information for image alignment according to the global confidence, that is, how to determine the input information of the image alignment module is described. The global confidence is compared with the global confidence threshold. If the global confidence is less than the global confidence threshold, it means that the result of the current global alignment of the image is not good, and it is necessary to obtain the input information for image alignment based on the gyroscope Gyro data source. Otherwise, the plane calibration matrix Homography obtained by the global alignment process is obtained, and the input information for image alignment is determined according to the plane calibration matrix. By setting the global confidence threshold, the result of the global alignment process is tested, so as to clarify the selection strategy for the input information.
[0041] Optionally, according to the gyroscope data source, input information for image alignment is obtained, including: querying a supplementary plane calibration matrix in the gyroscope data source, and determining the input information for image alignment based on the supplementary plane calibration matrix; when the supplementary plane calibration matrix cannot be queried in the gyroscope data source, obtaining a plane calibration matrix obtained through global alignment processing, and performing an identity transformation on the plane calibration matrix as the input information for image alignment.
[0042] In this embodiment, according to the gyroscope Gyro data source, the step of obtaining the input information for image alignment is to obtain the supplementary plane calibration matrix based on the Gyro data source. The idea of this step is to discard the original plane calibration matrix after global alignment processing, and calculate the supplementary plane calibration matrix based on the Gyro data source. Then, the input information for image alignment is determined from the supplementary plane calibration matrix. If the Gyro data source cannot be obtained, it is necessary to process the plane calibration matrix obtained after global alignment processing, such as setting the matrix to an identity transformation with the diagonal of 1, so as to clarify the selection strategy for the input information.
[0043] Optionally, based on the local confidence, input information for image alignment is determined, including: comparing the local confidence with a local confidence threshold, and when the local confidence is less than the local confidence threshold, querying the gyroscope data source for supplementary alignment information as input information for image alignment; when the local confidence is greater than or equal to the local confidence threshold, obtaining a local motion vector obtained through local alignment processing as input information for image alignment.
[0044] In this embodiment, how to determine the input information for image alignment based on the local confidence is described. Similarly, the local confidence is compared with the local confidence threshold. If it is less than, the corresponding supplementary alignment information is queried in the Gyro data source as the input information for image alignment. If it is greater than or equal to the local confidence threshold, the local motion vector obtained by the local alignment process is obtained as the input information for image alignment. By setting the global confidence threshold, the result of the global alignment process is tested, thereby clarifying the selection strategy for the input information.
[0045] Furthermore, if the corresponding supplementary alignment information cannot be obtained from the Gyro data source, the local motion vector obtained by the local alignment process still needs to be processed, specifically, the motion vector of each pixel in the local motion vector needs to be set to zero.
[0046] Optionally, after determining the input information for image alignment, the method further comprises: when the local confidence is less than the local confidence threshold, performing hysteresis processing on the local alignment module according to the local confidence.
[0047] In this embodiment, the above is only a selection method when multiple data sources are determined. In the further local alignment module, it is also necessary to ensure that the noise estimation of the local alignment algorithm module is effectively and correctly calibrated. That is, when it is determined that the local confidence is less than the local confidence threshold, it is necessary to check the quality of the local motion vector.
[0048] Optionally, according to the local confidence, the local alignment module is subjected to hysteresis processing, including: obtaining the local confidence Q; when Q<Q1, not performing hysteresis control, and turning off the output of the local alignment module; when Q1<Q<Q2, performing hysteresis control, and after completing the hysteresis control, turning off the output of the local alignment module; when Q>Q3, performing hysteresis control, and turning on the output of the local alignment module. Wherein, Q1 is the first judgment parameter, Q2 is the second judgment parameter, Q3 is the third judgment parameter, and Q1<Q2<Q3.
[0049] In this embodiment, three judgment parameters are set to determine the interval of local confidence. When Q<Q1, it means that the quality is low at this time, so the output of the local alignment module will be turned off. When Q1<Q<Q2, hysteresis control is performed, and after the hysteresis control is completed, the output of the local alignment module is turned off. When Q>Q3, it means that the image quality is good and relatively stable at this time, hysteresis control is performed, and the output of the local alignment module is turned on. The above settings can improve the stability of the local alignment area.
[0050] Furthermore, in order to better reflect the specific steps of the technical solution proposed in this embodiment, a specific implementation method is given. This embodiment is consistent with the idea of the previous embodiment, but the steps and details of the solution are introduced more comprehensively, including 5 parts, respectively. Figures 2 to 5 shown.
[0051] In the first part, the global alignment and local alignment of two frames of inference images are processed, such as Figure 2 As shown, including:
[0052] S201, performing global alignment processing on the current frame image and the reference frame image to obtain a global confidence.
[0053] The judgment of the quality and confidence of global image alignment can actually be divided into quantitative analysis on datasets with Ground Truth and practical application scenarios without Ground Truth. On datasets with Ground Truth, such as the KITTI dataset, the evaluation table of the KITTI dataset ranks all methods according to the number of erroneous pixels. All methods that provide less than 100% density use simple background interpolation, and the specific evaluation method is as follows:
[0054] D1: Percentage of stereo disparity outliers in first frame, that is, the percentage of outliers in the first frame stereo disparity (or optical flow motion vector)
[0055] D2: Percentage of stereo disparity outliers in second frame, that is, the percentage of stereo disparity constant in the second frame
[0056] Fl: Percentage of optical flow outliers, that is, the percentage of optical flow outliers
[0057] SF: Percentage of scene flow outliers (= outliers in either D0, D1 or Fl), that is, the percentage of scene flow outliers (= outliers in D0, D1 or Fl)
[0058] bg:Percentage of outliers averaged only over background regions, that is, the percentage of outliers averaged only over background regions
[0059] fg:Percentage of outliers averaged only over foreground regions, that is, the percentage of outliers averaged only over foreground regions
[0060] all:Percentage of outliers averaged over all ground truth pixels, that is, the average percentage of outliers on all real pixels
[0061] In engineering practice, the following method is used to evaluate the alignment effect: after obtaining the Homography matrix of global image alignment or the motion vector information of local alignment, the image is reconstructed and interpolated according to the global alignment Homography matrix or the motion vector information of local alignment by using the remap algorithm, and the alignment effect of the remapped image is evaluated as an evaluation indicator, such as Diff, SSIM, etc. It is recommended to use the dHash method for evaluation, and attention should be paid to alignment in brightness and color. Specifically, there are the following standards:
[0062] 1. Moving distance (End point error, EPE, the smaller the better)
[0063] 2. Angular Error (AE, the smaller the better)
[0064] 3. The ratio of pixels with optical flow information to the entire image (Valid Pixel Ratio, VPR, the larger the better)
[0065] 4. Outliers with errors more than 3px in non-occluded regions, or over 5% length of GT flow (Out-Noc-3 / F1-all, the smaller the better)
[0066] 5. dHash (Difference Hash)
[0067] Reduce the image to (size+1)*size and calculate the similarity between the two. Scan row by row, if the left pixel is larger than the right pixel, count it as 1, otherwise count it as 0, to form a hash sequence of size*size, and calculate the Hamming distance of the hash sequences of the two images. Size is commonly 8 or 16.
[0068] 6. Diff image (the darker the better), the difference between the warped image and the ref image
[0069] In actual engineering, the more challenging scenario is the second one, that is, when there is no Ground Truth. Since there is no Ground Truth, there is no method to construct a metric to evaluate the difference between the remapped result image and the true value. There are two ways: one is to comprehensively consider the quality metric Feature Extraction Confidence of feature point selection in the feature extraction and optimization calculation process and the iterative optimization quality metric Homography Calculation Confidence in the calculation process such as RANSAC; the construction of Feature Extraction Confidence here can be measured from the strength of the feature itself Feature Strength and the uniformity of the feature distribution Feature Coverage.
[0070] If the response obtained by various operators during feature extraction is high, the feature at this position is considered to be a strong feature, and its confidence Feature Confidence is considered to be relatively high; conversely, if the response obtained during feature extraction is low, the feature at this position is considered to be a weak feature, and its feature confidence Feature Confidence is relatively low. Another important measure is to confirm the uniformity of feature distribution across the entire image. Since the scene is changeable, images with strong corners and edges are not evenly distributed in the scene. In the image, areas with more strong features are naturally easier to align. In global optimization such as global alignment, it is easier to obtain better optimization. In the final remapping image, it is also seen in previous engineering experience that these positions have better global alignment effects, so it is believed that these areas of the image actually correspond to better Feature Coverage. Therefore, the following global image confidence metric is constructed:
[0071] Global Image Alignment Confidence=Feature Confidence*FeatureCoverage
[0072] Through the above steps, the Homography matrix can be obtained through global alignment processing and the global confidence can be obtained at the same time as a guarantee of image reliability.
[0073] S202: Perform local alignment processing on the current frame image and the reference frame image to obtain local confidence.
[0074] After the above discussion on global image alignment, the local alignment process is the same. For the case with GroundTruth, you can refer to the above information and will not repeat it here. Without Ground Truth, local alignment is constructed using SGM 2D, which completes the construction of two-dimensional dense motion vectors through Census feature extraction, feature matching, cost aggregation and post-processing. Among these steps, there are several steps that need to be paid attention to. The first is feature extraction and matching. Here, we can actually obtain feature-based confidence information based on the strength of the feature and the matching score. The higher the feature strength and the higher the matching score, the more reason to believe that the final motion vector confidence is higher. Here, we refer to Feature Confidence and Matching Confidence. The second is the cost aggregation position. By aggregating the matching costs of each position, we obtain the matching cost after cost aggregation. The higher the matching degree, that is, the lower the matching cost, and the matching costs of the left and right are similar after the LR Consistency Check, the higher the confidence of this position is. The confidence here is recorded as Aggregation Confidence. Finally, consider the interpolation process from semi-dense to dense through interpolation and other methods in the refinement process. Here, the preamble is to use superpixel-based information to guide the model fitting and motion vector interpolation. The higher the confidence of the model construction, the higher the confidence of the dense motion vector generated by the semi-dense vector, which is recorded as Refinement Confidence. Therefore, for local alignment, the following confidence metric is constructed:
[0075] Local Image Alignment Confidence=Feature Confidence*MatchingConfidence*Aggregation Confidence*Refinement Confidence
[0076] Through the above steps, the local motion vector can be obtained through local alignment processing, and the local confidence can be acquired.
[0077] In the second part, you need to set the working state of the image alignment module. The image alignment module usually has three working states, including the first one is to force the use of the image-based global alignment algorithm, and not use the gyroscope-based image alignment algorithm information; the second one is to force the use of the gyro-based global alignment algorithm information, and not use the image-based alignment algorithm information; the third one is the automatic calculation mode. When the image-based alignment algorithm function fails or fails, it switches to the gyro-based alignment algorithm information, and selects and decides local alignment and global alignment based on confidence information. Here you need to set the working state to Auto calculate mode.
[0078] In the third part, it is necessary to screen the image information from multiple sources. Specifically, the steps for checking the results after global alignment are as follows: Figure 3 As shown, including:
[0079] S3031, determine whether the global confidence is less than the global confidence threshold. If not, proceed to S3021; if so, proceed to S3022.
[0080] S3021, continue to use the image-based alignment algorithm information.
[0081] S3022, query whether alignment information based on the Gyro data source can be obtained. If not, proceed to S3031; if so, proceed to S3032.
[0082] S3031, obtaining a plane calibration matrix obtained by global alignment processing, and determining input information of an image alignment module after transforming the matrix.
[0083] S3032, obtaining input information of the image alignment module based on the Gyro data source.
[0084] After the global confidence and the plane calibration matrix Homography obtained after the global alignment processing based on the image are calculated in the previous order, the result of the global alignment processing needs to be judged next. First, as in S301, the global confidence and the global confidence threshold need to be compared. If the global confidence is greater than or equal to the global confidence threshold, it means that the image processing result is reliable, then as in S3021, continue to use the image-based alignment algorithm information. If the global confidence is less than the global confidence threshold, as in S3022, it means that the image processing result is not good, and it is necessary to obtain the alignment information based on Gyro (gyroscope data source). Furthermore, if the alignment information of Gyro can be obtained, the image alignment information based on Gyro is used for subsequent processing; if the alignment information based on Gyro cannot be found, it is also necessary to modify the image result based on the global alignment processing, such as the plane calibration matrix obtained by the global alignment processing, and the plane calibration matrix is transformed by identity, and then used as the input information of the image alignment module.
[0085] In the fourth part, it is necessary to screen the image information from multiple sources. Specifically, the steps for checking the results after local alignment are as follows: Figure 4 As shown, including:
[0086] S401, determining whether the local confidence is less than a local confidence threshold.
[0087] S4021, continue to use the image-based alignment algorithm information.
[0088] S4022, query whether alignment information based on the Gyro data source can be obtained.
[0089] S4031, obtaining a local motion vector obtained by the local alignment process, and performing a zeroing process on the local motion vector.
[0090] S4032, obtaining input information of the image alignment module based on the Gyro data source.
[0091] Based on the local confidence and local motion vector obtained after the local alignment processing based on the image, it is necessary to judge the result of the local alignment processing. As in S401, the local confidence is still compared with the local confidence threshold. If the local confidence is greater than or equal to the local confidence threshold, the image-based alignment algorithm information continues to be used. Otherwise, it means that the processing result after the local alignment processing is not good, and it is necessary to obtain the Gyro-based alignment information. Furthermore, if the Gyro alignment information can be obtained, subsequent processing is performed based on the Gyro image alignment information; if the Gyro-based alignment information cannot be found, it will be necessary to perform all zero processing on the local motion vectors obtained by the local alignment processing.
[0092] In the first four parts, we determined how to select from multiple sources of image alignment information to ensure quality. In the fifth part, we will further make judgments based on confidence in the local alignment process. Specifically, this part mainly completes the selection of global alignment algorithm and local alignment algorithm through modeDecisionLocalGlobal(); this module is used to complete the local alignment algorithm enable control; it is important to ensure that the noise estimation of the local alignment algorithm module is effectively and correctly calibrated.
[0093] In this section, the first judgment parameter Q1, the second judgment parameter Q2 and the third judgment parameter Q3 are set, and Q1<Q2<Q3. When the local confidence is less than the first judgment parameter, the local alignment will be forced to close, and when only the image-based alignment information is used, the hysteresis time constraint will be skipped and the local alignment information will be directly closed. When the local confidence is greater than the first judgment parameter and less than the second judgment parameter, the algorithm will wait for the hysteresis strategy to be completed, and then it will turn off the output of the local alignment information. When the local confidence is greater than the third judgment parameter, the decision algorithm will turn on the output of the local alignment information after the hysteresis strategy is completed. The above settings can improve the stability of the local alignment area.
[0094] Furthermore, hysteresis control also includes three situations. The first one is the Enabling SGM Min AgeThreshold compare module. The local alignment switches from off to on. The hysteresis control module means that the algorithm uses parameters to know the minimum number of frames that need to be waited before the local motion estimation module SGM is enabled. During the control process, the SGM confidence needs to be continuously monitored to ensure that the confidence is higher than the minimum threshold and the number of frames that last is greater than or equal to the minimum number of frames that need to be waited. After that, the control algorithm starts to enable the local alignment algorithm module; this part is mainly to ensure the stability of the manual cut-in of the local alignment algorithm, and avoid the switching jitter and unstable system image quality output caused by the direct cut-in of a frame of data that meets the requirements, and the next frame does not meet the requirements.
[0095] Enabling SGM Min Age ThresholdDefault value:3
[0096] Enabling SGM Min Age Threshold data range: Range: [0,4294967295]
[0097] The second is the Ignoring SGM Min Age Threshold compare module, which is the hysteresis control module for switching local alignment from on to off. Here, the decision algorithm uses this parameter to know the minimum number of frames to wait before SGM is turned off; that is, Ignoring SGM Min Age Threshold, after which the control algorithm starts to turn off the local alignment algorithm module; this part is mainly to ensure the stability of manual switching of the local alignment algorithm, and avoid switching jitter and unstable system image output caused by the direct switching out of a frame of data that does not meet the confidence requirements, and the next frame meets the requirements.
[0098] Ignoring SGM Min Age ThresholdDefault value:3
[0099] Ignoring SGM Min Age Threshold data range: Range: [0,4294967295]
[0100] The third is the Consecutive Good SGM Frames Threshold compare module. Here, the decision algorithm uses this parameter to confirm the number of consecutive good local alignment frames (greater than or equal to 1) required before returning to the local alignment (SGM) function. The parameter should be less than or equal to 0xFFFFFFFD, and the frame sequence information with a confidence level higher than the SGM confidence high threshold is required. This part is mainly to ensure the stability of the automatic cut-in of the local alignment algorithm, and avoid the switching jitter and unstable system image quality output caused by the direct cut-in due to the confidence level of one frame data not meeting the requirements, and the next frame not meeting the requirements.
[0101] Consecutive Good SGM Frames ThresholdDefault value:3
[0102] Consecutive Good SGM Frames Threshold data range: Range: [0,4294967295]
[0103] Through the above five-part image processing process, this embodiment introduces confidence as a measurement standard for multi-source image alignment information, and performs quality confirmation and selection on multi-source global or local image alignment information to obtain better alignment information, thereby making the image processing result smoother, and enabling the alignment information to be effectively combined in all scenarios, thereby solving the current technical problem that the quality of multi-source global or local alignment information cannot be confirmed, resulting in poor image processing effects.
[0104] Further, as Figures 1 to 4 The specific implementation of the method shown in the embodiment provides an image processing device, such as Figure 5 As shown, the device includes: an acquisition unit 51, a first processing unit 52, a second processing unit 53, and a determination unit 54.
[0105] An acquisition unit 51 is configured to acquire a current frame image and a reference frame image, wherein the reference frame image includes one or more image frames before the current frame in terms of time sequence;
[0106] A first processing unit 52 is configured to perform global alignment processing on the current frame image and the reference frame image to obtain a global confidence;
[0107] A second processing unit 53 is configured to perform local alignment processing on the current frame image and the reference frame image to obtain a local confidence;
[0108] The determination unit 54 is configured to determine input information for image alignment according to the global confidence and the local confidence.
[0109] In a specific application scenario, the determination unit 54 is specifically configured to compare the global confidence with the global confidence threshold, and when the global confidence is less than the global confidence threshold, obtain the input information for image alignment according to the gyroscope data source; when the global confidence is greater than or equal to the global confidence threshold, obtain the plane calibration matrix obtained by the global alignment processing, and determine the input information for image alignment according to the plane calibration matrix.
[0110] In a specific application scenario, the determination unit 54 is further configured to query a supplementary plane calibration matrix in the gyroscope data source, and determine the input information for image alignment based on the supplementary plane calibration matrix; when the supplementary plane calibration matrix cannot be queried in the gyroscope data source, the plane calibration matrix obtained by the global alignment processing is obtained, and the plane calibration matrix is subjected to an identity transformation and used as the input information for image alignment.
[0111] In a specific application scenario, the determination unit 54 is further configured to compare the local confidence with a local confidence threshold, and when the local confidence is less than the local confidence threshold, query the supplementary alignment information in the gyroscope data source as the input information for image alignment; and when the local confidence is greater than or equal to the local confidence threshold, obtain the local motion vector obtained by the local alignment processing as the input information for image alignment.
[0112] In a specific application scenario, the second processing unit 53 is further configured to perform hysteresis processing on the local alignment module according to the local confidence when the local confidence is less than the local confidence threshold.
[0113] In a specific application scenario, the second processing unit 53 is further configured to obtain the local confidence Q; when Q<Q1, the hysteresis control is not performed, and the output of the local alignment module is turned off; when Q1<Q<Q2, the hysteresis control is performed, and after completing the hysteresis control, the output of the local alignment module is turned off; when Q>Q3, the hysteresis control is performed, and the output of the local alignment module is turned on; wherein Q1 is the first judgment parameter, Q2 is the second judgment parameter, Q3 is the third judgment parameter, and Q1<Q2<Q3.
[0114] It should be noted that for other corresponding descriptions of the functional units involved in the image processing device provided in this embodiment, reference can be made to Figures 1 to 4 The corresponding description in will not be repeated here.
[0115] Based on the above Figures 1 to 4 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figures 1 to 4 The method shown.
[0116] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.
[0117] Based on the above Figures 1 to 4 The method shown, and Figure 5In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which can be configured on a computer terminal, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figures 1 to 4 The method shown.
[0118] Based on the above Figures 1 to 4 The method shown, and Figure 5 In order to achieve the above-mentioned purpose, the embodiment of the virtual device shown in the embodiment of the present application further provides a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, the signal including the computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the above-mentioned Figures 1 to 4 The method shown.
[0119] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
[0120] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.
[0121] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.
[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the solution of this embodiment, compared with the related art, by introducing confidence as a measure of multi-source image alignment information, the quality of multi-source global or local image alignment information is confirmed and selected, and better alignment information is obtained, thereby making the image processing result smoother, and enabling the alignment information to be effectively combined in all scenarios, solving the technical problem that the quality of the global or local alignment information of multiple sources cannot be confirmed, resulting in poor image processing effects.
[0123] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0124] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.
Claims
1. An image processing method, characterized in that: include: Acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame; Performing global alignment processing on the current frame image and the reference frame image to obtain a global confidence; Performing local alignment processing on the current frame image and the reference frame image to obtain a local confidence; Input information for image alignment is determined according to the global confidence and the local confidence.
2. The method according to claim 1, characterized in that Determining input information for image alignment according to the global confidence and the local confidence, including: Comparing the global confidence with a global confidence threshold, and acquiring the input information for image alignment according to a gyroscope data source when the global confidence is less than the global confidence threshold; When the global confidence is greater than or equal to the global confidence threshold, a plane calibration matrix obtained through the global alignment process is acquired, and the input information for image alignment is determined according to the plane calibration matrix.
3. The method according to claim 2, characterized in that According to the gyroscope data source, the input information for image alignment is obtained, including: querying a supplementary plane calibration matrix in the gyroscope data source, and determining the input information for image alignment according to the supplementary plane calibration matrix; When the supplementary plane calibration matrix cannot be found in the gyroscope data source, the plane calibration matrix obtained by the global alignment process is obtained, and after performing identity transformation on the plane calibration matrix, the matrix is used as the input information for image alignment.
4. The method according to claim 1, characterized in that: Determining input information for image alignment according to the local confidence, including: Comparing the local confidence with a local confidence threshold, and when the local confidence is less than the local confidence threshold, querying the gyroscope data source for supplementary alignment information as the input information for image alignment; When the local confidence is greater than or equal to the local confidence threshold, a local motion vector obtained through the local alignment process is acquired as the input information for image alignment.
5. The method according to any one of claims 1 to 4, characterized in that After determining the input information for image alignment, the method further includes: When the local confidence is less than the local confidence threshold, a hysteresis process is performed on the local alignment module according to the local confidence.
6. The method according to claim 5, characterized in that According to the local confidence, a hysteresis process is performed on the local alignment module, including: Obtaining the local confidence Q; When Q<Q1, the hysteresis control is not performed, and the output of the local alignment module is turned off; In the case of Q1<Q<Q2, the hysteresis control is performed, and after the hysteresis control is completed, the output of the local alignment module is turned off; When Q>Q3, the hysteresis control is performed to turn on the output of the local alignment module; Among them, Q1 is the first judgment parameter, Q2 is the second judgment parameter, Q3 is the third judgment parameter, and Q1<Q2<Q3.
7. An image processing device, characterized in that: include: An acquisition unit is configured to acquire a current frame image and a reference frame image, wherein in time sequence, the reference frame image includes one or more image frames before the current frame; A first processing unit is configured to perform global alignment processing on the current frame image and the reference frame image to obtain a global confidence; A second processing unit is configured to perform local alignment processing on the current frame image and the reference frame image to obtain a local confidence; A determination unit is configured to determine input information for image alignment according to the global confidence and the local confidence.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
10. A chip, characterized in that: The electronic device comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 6.