Image processing method and device, equipment and storage medium
By acquiring and filtering feature points of the image acquisition device, using feature point tracking and displacement matrix correction methods, the image blur problem caused by vibration of the image acquisition device is solved, and the accuracy and user experience of image correction are improved.
Patent Information
- Application Number
- CN202410034785.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-09
- Publication Date
- 2025-07-11
AI Technical Summary
When high-frequency vibration occurs in the image acquisition device, the OIS of the image acquisition device will cause image blur, and it is difficult for the prior art to effectively correct the image blur problem.
By acquiring feature points of processed images and to be processed images of the image acquisition device, using feature point tracking and displacement matrix correction methods, multiple feature point filtering and fitting are performed to obtain an accurate displacement matrix to correct image blur.
Improves the accuracy of image correction and improves the user experience.
Smart Images
Figure CN120298252A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, device, and storage medium. Background Art
[0002] In related technologies, when an image acquisition device vibrates at a high frequency, the OIS (Optical Image Stabilizer) of the image acquisition device undergoes abnormal position changes, resulting in blurred images being captured. At this time, it is necessary to correct the blurred images based on the displacement amount between two adjacent frames of images. Summary of the Invention
[0003] To overcome the problems existing in related technologies, the present disclosure provides an image processing method, apparatus, device, and storage medium.
[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including: obtaining a processed image and an image to be processed collected by an image acquisition device; where the processed image and the image to be processed are consecutive frame images; obtaining first feature points in the processed image; obtaining second feature points of the image to be processed and first processed feature points of the processed image based on the first feature points; obtaining a first displacement matrix based on the second feature points and the first processed feature points; obtaining second processed feature points of the processed image based on the first displacement matrix, the second feature points, and the first processed feature points; obtaining a second displacement matrix based on the second processed feature points and the second feature points; and processing the image to be processed based on the second displacement matrix.
[0005] In one implementation, the obtaining the first feature points in the processed image includes: performing histogram equalization processing on the processed image, and performing fast FAST corner detection on the processed image after the processing to obtain a plurality of initial corner points; obtaining at least one corner point from the plurality of initial corner points based on the positions of the plurality of initial corner points in the processed image; and obtaining the first feature points based on the at least one corner point.
[0006] In an optionally implementation, the obtaining at least one corner point from the plurality of initial corner points based on the positions of the plurality of initial corner points in the processed image includes: obtaining the initial corner points located within a preset region in the processed image as the at least one corner point based on the positions of the plurality of initial corner points in the processed image.
[0007] In an alternative implementation, obtaining the first feature points based on the at least one corner point includes: obtaining second feature points corresponding to the processed image when processing the processed image; merging the second feature points corresponding to the processed image and the at least one corner point to obtain a set of feature points; obtaining a first pixel distance between each of the corner points and adjacent points in the set of feature points; removing the corner points with the first pixel distance less than a distance threshold from the set of feature points, and taking the remaining points in the set of feature points as the first feature points.
[0008] In an implementation, obtaining the second feature points of the image to be processed and the first processed feature points of the processed image based on the first feature points includes: performing forward sparse optical flow tracking on the image to be processed based on the first feature points to obtain the second feature points corresponding to the first feature points; performing backward sparse optical flow tracking on the processed image based on the second feature points to obtain third feature points corresponding to the first feature points; and screening the first feature points based on the third feature points to obtain the first processed feature points.
[0009] In an alternative implementation, screening the first feature points based on the third feature points to obtain the first processed feature points includes: obtaining a second pixel distance between the first feature points and the corresponding third feature points; and screening the first feature points based on the second pixel distance to obtain the first processed feature points.
[0010] Optionally, screening the first feature points based on the second pixel distance to obtain the first processed feature points includes: arranging the first feature points based on the numerical relationship between the second pixel distances to obtain a sequence of feature points; and obtaining the first feature points within a preset sequence interval in the sequence of feature points as the first processed feature points.
[0011] In an implementation, obtaining a first displacement matrix based on the second feature points and the first processed feature points includes: obtaining a first average displacement amount in a first direction and a second average displacement amount in a second direction of the processed image and the image to be processed based on the second feature points and the first processed feature points; and obtaining the first displacement matrix based on the first displacement amount in the first direction and the second displacement amount in the second direction of the first processed feature points.
[0012] In one implementation, obtaining the secondarily processed feature points of the processed image based on the first displacement matrix, the second feature points, and the primarily processed feature points includes: reprojection of the second feature points based on the first displacement matrix to obtain fourth feature points corresponding to the first feature points; screening the primarily processed feature points based on the fourth feature points to obtain the to-be-processed feature points of the processed image; obtaining rolling shutter correction (RSC) information of the image acquisition device; and correcting the to-be-processed feature points based on the RSC information to obtain the secondarily processed feature points.
[0013] In one implementation, the processed image and the to-be-processed image are captured by the image acquisition device using a telephoto lens.
[0014] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including: an acquisition module configured to acquire a processed image and a to-be-processed image captured by an image acquisition device, where the processed image and the to-be-processed image are consecutive frame images; a feature extraction module configured to acquire first feature points in the processed image; a feature tracking module configured to acquire second feature points of the to-be-processed image and primarily processed feature points of the processed image based on the first feature points; a first processing module configured to obtain a first displacement matrix based on the second feature points and the primarily processed feature points; a second processing module configured to obtain secondarily processed feature points of the processed image based on the first displacement matrix, the second feature points, and the primarily processed feature points; a third processing module configured to obtain a second displacement matrix based on the secondarily processed feature points and the second feature points; and a fourth processing module configured to process the to-be-processed image based on the second displacement matrix.
[0015] In one implementation, the feature extraction module is specifically configured to: perform gray level equalization processing on the processed image, and perform fast FAST corner detection on the processed image after the processing to obtain a plurality of initial corner points; obtain at least one corner point from the plurality of initial corner points based on the positions of the plurality of initial corner points in the processed image; and obtain the first feature points based on the at least one corner point.
[0016] In an alternative implementation, the feature extraction module is specifically configured to: obtain the initial corner points within a preset region in the processed image as the at least one corner point based on the positions of the plurality of initial corner points in the processed image respectively.
[0017] In an alternative implementation, the feature extraction module is specifically configured to: obtain second feature points corresponding to the processed image when processing the processed image; merge the second feature points corresponding to the processed image and the at least one corner point to obtain a set of feature points; obtain a first pixel distance between each of the corner points and adjacent points in the set of feature points; remove the corner points with the first pixel distance less than a distance threshold from the set of feature points, and use the remaining points in the set of feature points as the first feature points.
[0018] In one implementation, the feature tracking module is specifically configured to: perform forward sparse optical flow tracking on the to-be-processed image based on the first feature points to obtain the second feature points corresponding to the first feature points; perform backward sparse optical flow tracking on the processed image based on the second feature points to obtain third feature points corresponding to the first feature points; screen the first feature points based on the third feature points to obtain the once-processed feature points.
[0019] In an alternative implementation, the feature tracking module is specifically configured to: obtain a second pixel distance between the first feature points and the corresponding third feature points; screen the first feature points based on the second pixel distance to obtain the once-processed feature points.
[0020] Optionally, the feature tracking module is specifically configured to: arrange the first feature points based on the numerical relationship between the second pixel distances to obtain a sequence of feature points; obtain the first feature points within a preset sequence interval in the sequence of feature points as the once-processed feature points.
[0021] In one implementation, the first processing module is specifically configured to: obtain a first average displacement amount in a first direction and a second average displacement amount in a second direction of the processed image and the to-be-processed image based on the second feature points and the once-processed feature points; obtain the first displacement matrix based on the first displacement amount in the first direction and the second displacement amount in the second direction of the once-processed feature points.
[0022] In one implementation, the second processing module is specifically configured to: reproject the second feature points based on the first displacement matrix to obtain fourth feature points corresponding to the first feature points; screen the once-processed feature points based on the fourth feature points to obtain the to-be-processed feature points of the processed image; obtain the rolling shutter correction (RSC) information of the image acquisition device; correct the to-be-processed feature points based on the RSC information to obtain the twice-processed feature points.
[0023] In one implementation, the processed image and the image to be processed are captured by the image acquisition device based on a telephoto lens.
[0024] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the foregoing first aspect.
[0025] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the foregoing first aspect.
[0026] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0027] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: the second feature points in the image to be processed and the primary processed feature points in the processed image can be obtained based on the first feature points in the processed image, so as to obtain the first feature matrix by fitting based on the second feature points and the primary processed feature points, and the primary processed feature points are corrected based on the first feature matrix and the second feature points to obtain the secondary processed feature points, and finally the second feature matrix is obtained by fitting the second feature points and the secondary processed feature points, so as to process the image to be processed according to the second feature matrix. More accurate feature points can be obtained through multiple screenings of the acquired feature points, so as to obtain a more accurate displacement matrix representing the displacement change between two consecutive images, and the image is corrected based on the displacement matrix, thereby improving the user experience.
[0028] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.
[0030] Figure 1 is a flowchart of an image processing method shown according to an exemplary embodiment.
[0031] Figure 2 is a flowchart of another image processing method shown according to an exemplary embodiment.
[0032] Figure 3 It is a schematic diagram of a corner point screening scheme shown according to an exemplary embodiment.
[0033] Figure 4 It is a flowchart of another image processing method shown according to an exemplary embodiment.
[0034] Figure 5 It is a schematic diagram of a feature point screening shown according to an exemplary embodiment.
[0035] Figure 6 It is a flowchart of another image processing method shown according to an exemplary embodiment.
[0036] Figure 7 It is a flowchart of another image processing method shown according to an exemplary embodiment.
[0037] Figure 8 It is a schematic diagram of an image processing scheme shown according to an exemplary embodiment.
[0038] Figure 9 It is a block diagram of an image processing device shown according to an exemplary embodiment.
[0039] Figure 10 It is a block diagram of an electronic device for image processing shown according to an exemplary embodiment. Detailed implementation manners
[0040] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0041] The various digital numbers such as the first and second involved in the present disclosure are only for convenience of description and are not used to limit the scope of the embodiments of the present disclosure, nor do they represent the order of precedence.
[0042] Figure 1 It is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 1 shown, the method may include but is not limited to the following steps.
[0043] Step S101: Obtain the processed image and the image to be processed collected by the image acquisition device.
[0044] Among them, in the embodiments of the present disclosure, the above-mentioned processed image and the image to be processed are consecutive frame images.
[0045] For example, obtain two consecutive frames of images collected by an image acquisition device, use the previous frame of the two consecutive frames of images as the processed image, and use the latter frame of images as the image to be processed.
[0046] Step S102: Obtain the first feature points in the processed image.
[0047] For example, perform feature point detection on the processed image to obtain the first feature points in the processed image.
[0048] Step S103: Based on the first feature points, obtain the second feature points of the image to be processed and the first-time processed feature points of the processed image.
[0049] For example, perform feature point tracking in the image to be processed based on the first feature points to obtain the second feature points corresponding to the first feature points in the image to be processed, perform feature point tracking in the processed image based on the second feature points to obtain the tracking feature points of the first feature points in the processed image, and perform screening processing on the first feature points based on the tracking feature points to obtain the first-time processed feature points of the processed image.
[0050] Step S104: Based on the second feature points and the first-time processed feature points, obtain the first displacement matrix.
[0051] For example, based on the second feature points and the first-time processed feature points corresponding to the second feature points, obtain the offset between the second feature points and the corresponding first-time processed feature points, so as to obtain the first displacement matrix based on this offset.
[0052] Step S105: Based on the first displacement matrix, the second feature points and the first-time processed feature points, obtain the second-time processed feature points of the processed image.
[0053] For example, map the second feature points to the processed image based on the first displacement matrix to obtain the mapped feature points of the processed image, and process the first-time processed feature points corresponding to the same second feature points based on the mapped feature points to obtain the second-time processed feature points of the processed image.
[0054] Step S106: Based on the second-time processed feature points and the second feature points, obtain the second displacement matrix.
[0055] For example, obtain the offset between the second feature points and the corresponding second-time processed feature points, so as to obtain the second displacement matrix based on the above offset.
[0056] Step S107: Process the image to be processed based on the second displacement matrix.
[0057] For example, the position of each pixel point in the image to be processed is reversely corrected based on the second displacement matrix.
[0058] In an embodiment of the present disclosure, the second displacement matrix can be filtered, and the image to be processed is processed based on the filtered second displacement matrix.
[0059] By implementing the embodiments of the present disclosure, the second feature points in the image to be processed and the first - processed feature points in the processed image can be obtained based on the first feature points in the processed image, so as to obtain the first feature matrix by fitting based on the second feature points and the first - processed feature points, and the first - processed feature points are corrected based on the first feature matrix and the second feature points to obtain the second - processed feature points. Finally, the second feature matrix is obtained by fitting the second feature points and the second - processed feature points, so as to process the image to be processed according to the second feature matrix. It is possible to obtain more accurate feature points through multiple screenings of the acquired feature points, thereby obtaining a more accurate displacement matrix representing the displacement change between two consecutive images, and correcting the image based on this displacement matrix, so as to improve the user experience.
[0060] In one implementation, the first feature points in the processed image can be obtained by means of corner detection. As an example, please refer to Figure 2 , Figure 2 which is a flowchart of another image - processing method shown according to an exemplary embodiment. As shown in Figure 2 , the method may include but is not limited to the following steps.
[0061] Step S201: Obtain the processed image and the image to be processed collected by the image acquisition device.
[0062] In an embodiment of the present disclosure, step S201 can be implemented in any one of the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail again.
[0063] Step S202: Perform gray - level equalization processing on the processed image, and perform FAST corner detection on the processed image after the processing to obtain a plurality of initial corner points.
[0064] Specifically, perform gray - level equalization processing on the processed image, and perform FAST corner detection on the processed image after the gray - level equalization processing to obtain a plurality of initial corner points.
[0065] Step S203: Obtain at least one corner point from the plurality of initial corner points based on the positions of the plurality of initial corner points in the processed image.
[0066] For example, at least one corner point that meets the preset requirements is selected from the multiple initial corner points based on the positions of the multiple initial corner points in the processed image.
[0067] In an alternative implementation, obtaining at least one corner point from the multiple initial corner points based on the positions of the multiple initial corner points in the processed image includes: obtaining the initial corner points within a preset area in the processed image as the at least one corner point based on the positions of the multiple initial corner points in the processed image respectively.
[0068] As an example, please refer to Figure 3 , Figure 3 which is a schematic diagram of a corner point screening scheme shown according to an exemplary embodiment. As shown in Figure 3 , the initial corner points within the 5% boundary in the processed image can be obtained as corner points based on the positions of the initial corner points.
[0069] Step S204: Obtain the first feature points based on at least one corner point.
[0070] In an alternative implementation, obtaining the first feature points based on at least one corner point may include the following steps:
[0071] A1: Obtain the second feature points corresponding to the processed image when the processed image is being processed.
[0072] For example, when the processed image is used as the original image to be processed and the previous adjacent frame image of the processed image is used as the original processed image, the second feature points obtained by tracking the feature points of the processed image based on the feature points of the previous adjacent frame image of the processed image are obtained when the processed image is being processed.
[0073] A2: Combine the second feature points corresponding to the processed image and at least one corner point to obtain a set of feature points.
[0074] For example, the second feature points corresponding to the processed image and at least one corner point are combined on the processed image to obtain a set of feature points.
[0075] A3: Obtain the first pixel distance between each corner point and its adjacent point in the set of feature points.
[0076] For example, based on the positions of the second feature points and at least one corner point on the processed image, the pixel distance between the nearest adjacent point (e.g., the nearest second feature point or the nearest other corner point) of each corner point in the set of feature points is obtained as the first pixel distance corresponding to each corner point.
[0077] A4: Remove the corner points with the first pixel distance less than the distance threshold from the set of feature points, and use the remaining points in the set of feature points as the first feature points.
[0078] For example, remove the corner points with the corresponding first pixel distance less than the preset distance threshold (e.g., 3 pixels) from the set of feature points, and use the remaining corner points in the set of feature points and the second feature points corresponding to the processed image as the first feature points.
[0079] In some other embodiments of the present disclosure, if the processed image has not been processed as a to-be-processed image (e.g., the processed image is the first frame image in a video stream), then the second feature points corresponding to the processed image do not exist. In this case, steps A1 and A2 may not be executed. That is, directly obtain the pixel distance between each corner point and the adjacent corner point with the closest distance as the first pixel distance, and screen the corner points based on the first pixel distance to obtain the first feature points.
[0080] Step S205: Obtain the second feature points of the to-be-processed image and the first processed feature points of the processed image based on the first feature points.
[0081] In the embodiments of the present disclosure, step S205 may be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0082] Step S206: Obtain the first displacement matrix based on the second feature points and the first processed feature points.
[0083] In the embodiments of the present disclosure, step S206 may be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0084] Step S207: Obtain the second processed feature points of the processed image based on the first displacement matrix, the second feature points, and the first processed feature points.
[0085] In the embodiments of the present disclosure, step S207 may be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0086] Step S208: Obtain the second displacement matrix based on the second processed feature points and the second feature points.
[0087] In the embodiments of the present disclosure, step S208 may be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0088] Step S209: Process the to-be-processed image based on the second displacement matrix.
[0089] In an embodiment of the present disclosure, step S209 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not be elaborated further.
[0090] By implementing the embodiments of the present disclosure, FAST corner detection can be performed on the processed image to obtain FAST corners, and the FAST corners can be screened in combination with the second feature points corresponding to the processed image as the image to be processed, so as to obtain more accurate first feature points in the processed image, thereby improving the effectiveness of the subsequent obtained feature points and improving the accuracy of the second displacement matrix. A more accurate displacement matrix representing the displacement change between two consecutive images can be obtained to correct the image, thereby improving the user experience.
[0091] In one implementation, feature point tracking can be performed on the first feature points to obtain second feature points and once-processed feature points. As an example, please refer to Figure 4 , Figure 4 is a flowchart of another image processing method shown according to an exemplary embodiment. As shown in Figure 4 , the method may include but is not limited to the following steps.
[0092] Step S401: Obtain the processed image and the image to be processed collected by the image acquisition device.
[0093] In an embodiment of the present disclosure, step S401 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not be elaborated further.
[0094] Step S402: Obtain the first feature points in the processed image.
[0095] In an embodiment of the present disclosure, step S402 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not be elaborated further.
[0096] Step S403: Perform forward sparse optical flow tracking on the image to be processed based on the first feature points to obtain second feature points corresponding to the first feature points.
[0097] Step S404: Perform backward sparse optical flow tracking on the processed image based on the second feature points to obtain third feature points corresponding to the first feature points.
[0098] Step S405: Screen the first feature points based on the third feature points to obtain once-processed feature points.
[0099] In an alternative implementation, the above-mentioned screening of the first feature points based on the third feature points to obtain the once-processed feature points may include: obtaining the second pixel distance between the first feature points and the corresponding third feature points; screening the first feature points based on the second pixel distance to obtain the once-processed feature points.
[0100] For example, obtain the second pixel distance value between the first feature point and the corresponding third feature point; and obtain the first feature points whose corresponding second pixel distance values are within a preset pixel distance range as the once-processed feature points.
[0101] Optionally, the above-mentioned screening of the first feature points based on the second pixel distance to obtain the once-processed feature points includes: arranging the first feature points based on the numerical relationship between the second pixel distances to obtain a feature point sequence; obtaining the first feature points within a preset sequence range in the feature point sequence as the once-processed feature points.
[0102] As an example, please refer to Figure 5 , Figure 5 which is a schematic diagram of feature point screening shown according to an exemplary embodiment. As Figure 5 shown, the first feature points can be sorted based on the pixel distance (i.e., Figure 5 distance in Figure 5 ) between the first feature points and the corresponding third feature points to obtain a feature point sequence, and the first feature points within the range of 10% - 90% in the sequence are selected as the once-processed feature points (i.e.,
[0103] Selected Feature Points in
[0104] Step S406: Obtain the first displacement matrix based on the second feature points and the once-processed feature points.
[0105] In the embodiments of the present disclosure, step S406 can be implemented in any of the ways in the respective embodiments of the present disclosure. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further.
[0106] In the embodiments of the present disclosure, step S407 can be implemented in any of the ways in the respective embodiments of the present disclosure. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further.
[0107] Step S408: Obtain the second displacement matrix based on the secondarily processed feature points and the second feature points.
[0108] In an embodiment of the present disclosure, step S408 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0109] Step S409: Process the image to be processed based on the second displacement matrix.
[0110] In an embodiment of the present disclosure, step S409 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0111] By implementing the embodiments of the present disclosure, the second feature points corresponding to the first feature points and the first-pass processed feature points can be obtained based on sparse flow tracking, so as to obtain the first feature matrix by fitting based on the second feature points and the first-pass processed feature points, and correct the first-pass processed feature points based on the first feature matrix and the second feature points to obtain the second-pass processed feature points. Finally, the second feature matrix is obtained by fitting the second feature points and the second-pass processed feature points, so as to process the image to be processed according to the second feature matrix. It is possible to more efficiently obtain the displacement matrix representing the displacement change between two consecutive images to correct the image, thereby improving the user experience.
[0112] In one implementation, the first displacement matrix can be obtained based on the displacement amount between the second feature points and the corresponding first-pass processed feature points. As an example, please refer to Figure 6 , Figure 6 is a flowchart of another image processing method shown according to an exemplary embodiment. As shown in Figure 6 , the method may include but is not limited to the following steps.
[0113] Step S601: Obtain the processed image and the image to be processed collected by the image acquisition device.
[0114] In an embodiment of the present disclosure, step S601 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0115] Step S602: Obtain the first feature points in the processed image.
[0116] In an embodiment of the present disclosure, step S602 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not elaborate further.
[0117] Step S603: Obtain the second feature points of the image to be processed and the first-pass processed feature points of the processed image based on the first feature points.
[0118] In an embodiment of the present disclosure, step S603 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further.
[0119] Step S604: Based on the second feature points and the once-processed feature points, obtain the first average displacement amount in the first direction and the second average displacement amount in the second direction of the processed image and the image to be processed.
[0120] Wherein, in an embodiment of the present disclosure, the above-mentioned first direction can be any one of the X-axis direction or the Y-axis direction, and the second direction is the other direction different from the first direction among the X-axis direction or the Y-axis direction.
[0121] As an example, the first direction is the X-axis direction and the second direction is the Y-axis direction.
[0122] As another example, the first direction is the Y-axis direction and the second direction is the X-axis direction.
[0123] For example, the first displacement amount in the first direction and the second displacement amount in the second direction of the second feature points can be obtained based on the following formula.
[0124] ptsTrans = prevptsStored - nextpts
[0125]
[0126] Wherein, ptsTrans is the displacement amount between any second feature point and the corresponding once-processed feature point, mean x is the first average displacement amount in the first direction of the processed image and the image to be processed, mean y is the second average displacement amount in the second direction of the processed image and the image to be processed, and N is a positive integer, and the maximum value of N is the number of once-processed feature points.
[0127] Step S605: Based on the first displacement amount in the first direction and the second displacement amount in the second direction of the once-processed feature points, obtain the first displacement matrix.
[0128] For example, the first displacement matrix can be obtained according to the following formula by combining the first average displacement amount and the second average displacement amount.
[0129]
[0130] Wherein, perspMatrix is the first displacement matrix, mean x is the first average displacement amount, mean y is the second average displacement amount.
[0131] Step S606: Obtain the secondarily processed feature points of the processed image based on the first displacement matrix, the second feature points, and the once-processed feature points.
[0132] In the embodiments of the present disclosure, step S606 can be implemented in any one of the embodiments of the present disclosure respectively. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further.
[0133] Step S607: Obtain the second displacement matrix based on the secondarily processed feature points and the second feature points.
[0134] In the embodiments of the present disclosure, step S607 can be implemented in any one of the embodiments of the present disclosure respectively. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further. For example, it can be implemented in the same way as steps S604 and S605.
[0135] Step S608: Process the image to be processed based on the second displacement matrix.
[0136] In the embodiments of the present disclosure, step S608 can be implemented in any one of the embodiments of the present disclosure respectively. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further.
[0137] By implementing the embodiments of the present disclosure, the second feature points in the image to be processed and the once-processed feature points in the processed image can be obtained based on the first feature points in the processed image, so as to obtain the first feature matrix by fitting based on the second feature points and the once-processed feature points, and correct the once-processed feature points based on the first feature matrix and the second feature points to obtain the secondarily processed feature points. Finally, fit the second feature points and the secondarily processed feature points to obtain the second feature matrix, so as to process the image to be processed according to the second feature matrix. It is possible to obtain more accurate feature points through multiple screenings of the obtained feature points, thereby obtaining a more accurate displacement matrix representing the displacement change between two consecutive images, and correcting the image based on this displacement matrix, so as to improve the user experience.
[0138] In one implementation, the second feature points can be corrected based on the first feature matrix to obtain the secondarily processed feature points. As an example, please refer to Figure 7 , Figure 7 is a flowchart of another image processing method shown according to an exemplary embodiment. As shown in Figure 7 , this method can include but is not limited to the following steps.
[0139] Step S701: Obtain the processed image and the image to be processed collected by the image acquisition device.
[0140] In an embodiment of the present disclosure, step S701 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not be elaborated further.
[0141] Step S702: Obtain the first feature points in the processed image.
[0142] In an embodiment of the present disclosure, step S702 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not be elaborated further.
[0143] Step S703: Based on the first feature points, obtain the second feature points of the image to be processed and the primary processed feature points of the processed image.
[0144] In an embodiment of the present disclosure, step S703 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not be elaborated further.
[0145] Step S704: Based on the second feature points and the primary processed feature points, obtain the first displacement matrix.
[0146] In an embodiment of the present disclosure, step S704 can be implemented in any one of the embodiments of the present disclosure. The embodiments of the present disclosure do not limit this and will not be elaborated further.
[0147] Step S705: Reproject the second feature points based on the first displacement matrix to obtain the fourth feature points corresponding to the first feature points.
[0148] Specifically, the following formula can be used to reproject the second feature points on the image to be processed onto the processed image based on the first displacement matrix to obtain the fourth feature points on the processed image.
[0149] ptsProject = perspMatrix · nextpts
[0150] Where ptsProject is the fourth feature point, perspMatrix is the first displacement matrix, and nextpts is the second feature point of the image to be processed.
[0151] In some embodiments of the present disclosure, the reprojection error can also be obtained according to the following formula as the credibility of the second displacement matrix.
[0152] ptsDiff = prevptsStored - ptsProject
[0153]
[0154]
[0155] Among them, ptsDiff is the pixel distance between the first feature point and the corresponding fourth feature point, prevptsStored is the first feature point, ptsProject is the fourth feature point, error is the reprojection error, and i and N are positive integers.
[0156] Step S706: Screen the once-processed feature points based on the fourth feature points to obtain the to-be-processed feature points of the processed image.
[0157] For example, obtain the pixel distance values between each once-processed feature point and the corresponding fourth feature point, sort the once-processed feature points based on the numerical relationship of the pixel distances corresponding to the once-processed feature points to obtain a feature point sequence, and use the once-processed feature points within a preset sequence interval (for example, the 10% - 90% interval) in the feature point sequence as the to-be-processed feature points.
[0158] Step S707: Obtain the RSC (Rolling Shutter Correction) information of the image acquisition device.
[0159] For example, obtain the RSC information of the image acquisition device between the acquisition of the processed image and the to-be-processed image.
[0160] Step S708: Correct the to-be-processed feature points based on the RSC information to obtain the twice-processed feature points.
[0161] Specifically, based on the RSC information of the image acquisition device between the acquisition of the processed image and the to-be-processed image, correct the to-be-processed feature points to obtain the twice-processed feature points.
[0162] Step S709: Obtain the second displacement matrix based on the twice-processed feature points and the second feature points.
[0163] In the embodiments of the present disclosure, step S709 can be implemented in any one of the various embodiments of the present disclosure. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further.
[0164] Step S710: Process the to-be-processed image based on the second displacement matrix.
[0165] In the embodiments of the present disclosure, step S710 can be implemented in any one of the various embodiments of the present disclosure. The embodiments of the present disclosure do not make any limitations in this regard and will not be elaborated further.
[0166] By implementing the embodiments of the present disclosure, the primary processed feature points can be corrected based on the first feature matrix and the second feature points to obtain the secondary processed feature points, so as to fit the second feature points and the secondary processed feature points to obtain the second feature matrix, and process the image to be processed according to the second feature matrix. By correcting the primary processed feature points, more accurate secondary processed feature points can be obtained, so as to obtain a more accurate displacement matrix representing the displacement change between two consecutive images, and correct the image based on the displacement matrix, thereby improving the user experience.
[0167] In some embodiments of the present disclosure, the processed image and the image to be processed are captured by an image acquisition device based on a telephoto lens.
[0168] In some embodiments of the present disclosure, the next frame image of the image to be processed can be obtained, and the image to be processed is used as the new processed image, and the next frame image of the image to be processed is used as the new image to be processed, and the new image to be processed is processed according to the method provided in any embodiment of the present disclosure.
[0169] Please refer to Figure 8 , Figure 8 which is a schematic diagram of an image processing solution shown according to an exemplary embodiment. As Figure 8 shown, the previous frame image PrevMat (i.e., the aforementioned processed image) in the video stream captured by the image acquisition device can be subjected to gray value equalization processing, and FAST corner detection is performed on the processed image to obtain FAST corners, and the FAST corners located outside the 5% boundary are removed; then the remaining FAST corners are merged with the corresponding feature points prevptsStored (if not present, it is empty) of the PrevMat in the previous processing cycle, and the FAST corners with a pixel distance less than 3 from the nearest point (FAST corner or prevptsStored) among the remaining FAST corners are removed to obtain the updated prevptsStored (i.e., the aforementioned first feature points); then forward sparse optical flow tracking is performed based on PrevMat and the updated prevptsStored to obtain the feature points nextpts (i.e., the aforementioned second feature points) corresponding to prevptsStored in the next frame image CurrMat (i.e., the aforementioned image to be processed) in the video stream, and backward sparse optical flow tracking is performed based on CurrMat and nextpts to obtain the prevptsRvs (i.e., the aforementioned third feature points) in PrevMat corresponding to nextpts.
[0170] After that, according to the distances between prevptsStored and the corresponding prevptsRvs, prevptsStored is sorted from small to large, and then the prevptsStored at the 10% and 90% positions of the sorting are found and used as the prevptsStored after the second update (i.e., the aforementioned feature points after the first processing).
[0171] After that, based on the prevptsStored after the second update and the corresponding nextpts, the average displacement amounts in the x-axis direction and y-axis direction of the previous frame image and the next frame image are obtained, and then the first displacement matrix perspMatrix is obtained. And the feature points ptsProject (i.e., the aforementioned fourth feature points) corresponding to the re-projection of nextpts on PrevMat are used to screen the prevptsStored after the second update to obtain the prevptsStored after the third update.
[0172] After that, the RSC information of the image acquisition device is obtained, and the prevptsStored after the third update is corrected using the RSC information. The second displacement matrix is recalculated using the corrected prevptsStored (i.e., the aforementioned feature points after the second processing) and the corresponding nextpts. Thus, CurrMat is corrected based on the second displacement matrix. And currMa is used as the new prevMat, and the feature points prevptsStored corresponding to currMat in the previous processing loop are used as the new prevMat. The next frame image is obtained as the new currMat to process the next frame image.
[0173] Please refer to Figure 9 , Figure 9 which is a block diagram of an image processing apparatus shown according to an exemplary embodiment. As Figure 9As shown in the figure, the device 900 includes: an acquisition module 901, configured to acquire a processed image and an image to be processed collected by an image acquisition device; wherein, the processed image and the image to be processed are consecutive frame images; a feature extraction module 902, configured to acquire first feature points in the processed image; a feature tracking module 903, configured to acquire second feature points of the image to be processed and first processed feature points of the processed image based on the first feature points; a first processing module 904, configured to acquire a first displacement matrix based on the second feature points and the first processed feature points; a second processing module 905, configured to acquire second processed feature points of the processed image based on the first displacement matrix, the second feature points and the first processed feature points; a third processing module 906, configured to acquire a second displacement matrix based on the second processed feature points and the second feature points; a fourth processing module 907, configured to process the image to be processed based on the second displacement matrix.
[0174] In one implementation, the feature extraction module 902 is specifically configured to: perform gray level equalization processing on the processed image, and perform FAST corner detection of feature points to be processed on the processed image after processing to obtain a plurality of initial corner points; based on the positions of the plurality of initial corner points in the processed image, obtain at least one corner point from the plurality of initial corner points; and obtain first feature points based on the at least one corner point.
[0175] In an optional implementation, the feature extraction module 902 is specifically configured to: based on the positions of the plurality of initial corner points in the processed image respectively, obtain the initial corner points within a preset area in the processed image as the at least one corner point.
[0176] In an optional implementation, the feature extraction module 902 is specifically configured to: obtain second feature points corresponding to the processed image when the processed image is processed; combine the second feature points corresponding to the processed image and the at least one corner point to obtain a feature point set; obtain a first pixel distance between each corner point in the feature point set and its adjacent points; remove the corner points with the first pixel distance less than a distance threshold from the feature point set, and use the remaining points in the feature point set as the first feature points.
[0177] In one implementation, the feature tracking module 903 is specifically configured to: perform forward sparse optical flow tracking on the image to be processed based on the first feature points to obtain second feature points corresponding to the first feature points; perform backward sparse optical flow tracking on the processed image based on the second feature points to obtain third feature points corresponding to the first feature points; and screen the first feature points based on the third feature points to obtain first processed feature points.
[0178] In an optional implementation, the feature tracking module 903 is specifically configured to: obtain a second pixel distance between the first feature points and the corresponding third feature points; screen the first feature points based on the second pixel distance to obtain first processed feature points.
[0179] Optionally, the feature tracking module 903 is specifically configured to: arrange the first feature points based on the numerical relationship between the second pixel distances to obtain a feature point sequence; and obtain the first feature points within a preset sequence interval in the feature point sequence as the first-time processed feature points.
[0180] In one implementation, the first processing module 904 is specifically configured to: obtain a first average displacement in the first direction and a second average displacement in the second direction of the processed image and the image to be processed based on the second feature points and the first-time processed feature points; and obtain a first displacement matrix based on the first displacement in the first direction and the second displacement in the second direction of the first-time processed feature points.
[0181] In one implementation, the second processing module 905 is specifically configured to: re-project the second feature points based on the first displacement matrix to obtain fourth feature points corresponding to the first feature points; screen the first-time processed feature points based on the fourth feature points to obtain the feature points to be processed in the processed image; obtain the rolling shutter correction (RSC) information of the image acquisition device; and correct the feature points to be processed based on the RSC information to obtain the second-time processed feature points.
[0182] In one implementation, the processed image and the image to be processed are captured by an image acquisition device using a telephoto lens.
[0183] Through the device according to the embodiments of the present disclosure, the second feature points in the image to be processed and the first-time processed feature points in the processed image can be obtained based on the first feature points in the processed image, so as to obtain a first feature matrix by fitting based on the second feature points and the first-time processed feature points, and correct the first-time processed feature points based on the first feature matrix and the second feature points to obtain the second-time processed feature points. Finally, the second feature matrix is obtained by fitting the second feature points and the second-time processed feature points, so as to process the image to be processed according to the second feature matrix. More accurate feature points can be obtained through multiple screenings of the acquired feature points, thereby obtaining a more accurate displacement matrix representing the displacement change between two consecutive images, and correcting the image based on the displacement matrix, thereby improving the user experience.
[0184] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.
[0185] Figure 10 is a block diagram of an electronic device for image processing according to an exemplary embodiment.
[0186] Refer to Figure 10, the electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power component 1006, a multimedia component 1008, an input / output (I / O) interface 1010, and a communication component 1012.
[0187] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, data communication, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the above-described method. In addition, the processing component 1002 may include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.
[0188] The memory 1004 is configured to store various types of data to support the operation of the device 1000. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.
[0189] The power component 1006 provides power to various components of the electronic device 1000. The power component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 1000.
[0190] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation.
[0191] The input / output interface 1010 provides an interface between the processing component 1002 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc.
[0192] The communication component 1012 is configured to facilitate communication between the electronic device 1000 and other devices in a wired or wireless manner. The electronic device 1000 can access a communication standard-based wireless network, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1012 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1012 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0193] In an exemplary embodiment, the electronic device 1000 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.
[0194] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 1004 including instructions, and the above instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0195] The present disclosure also provides a computer program product, and when the computer program product is executed by a computer, it implements the functions of any of the above method embodiments.
[0196] Those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present invention are pointed out by the following claims.
[0197] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.
Claims
1. An image processing method, characterized in that, Including: Obtain a processed image and an image to be processed collected by an image acquisition device; wherein, the processed image and the image to be processed are consecutive frame images; Obtain first feature points in the processed image; Based on the first feature points, obtain second feature points of the image to be processed and first processed feature points of the processed image; Based on the second feature points and the first processed feature points, obtain a first displacement matrix; Based on the first displacement matrix, the second feature points and the first processed feature points, obtain second processed feature points of the processed image; Based on the second processed feature points and the second feature points, obtain a second displacement matrix; Process the image to be processed based on the second displacement matrix.
2. The method according to claim 1, wherein The obtaining of the first feature points in the processed image includes: Perform gray level equalization processing on the processed image, and perform fast FAST corner detection on the processed image after processing to obtain a plurality of initial corner points; Based on the positions of the plurality of initial corner points in the processed image, obtain at least one corner point from the plurality of initial corner points; Based on the at least one corner point, obtain the first feature points.
3. The method according to claim 2, characterized in that, The obtaining of at least one corner point from the plurality of initial corner points based on the positions of the plurality of initial corner points in the processed image includes: Based on the positions of the plurality of initial corner points in the processed image respectively, obtain the initial corner points within a preset region in the processed image as the at least one corner point.
4. The method according to claim 2, wherein The obtaining of the first feature points based on the at least one corner point includes: Obtain second feature points corresponding to the processed image when the processed image is processed; Merge the second feature points corresponding to the processed image and the at least one corner point to obtain a feature point set; Obtain a first pixel distance between each corner point in the feature point set and its adjacent points; Eliminate the corner points in the feature point set whose first pixel distance is less than a distance threshold, and use the remaining points in the feature point set as the first feature points.
5. The method according to claim 1, characterized in that, The obtaining of the second feature points of the image to be processed and the first processed feature points of the processed image based on the first feature points includes: Perform forward sparse optical flow tracking on the image to be processed based on the first feature points to obtain the second feature points corresponding to the first feature points; Perform backward sparse optical flow tracking on the processed image based on the second feature points to obtain third feature points corresponding to the first feature points; Based on the third feature points, screen the first feature points to obtain the first processed feature points.
6. The method according to claim 5, wherein The screening of the first feature points based on the third feature points to obtain the first processed feature points includes: Obtain a second pixel distance between the first feature points and the corresponding third feature points; Based on the second pixel distance, screen the first feature points to obtain the first processed feature points.
7. The method according to claim 6, wherein The screening of the first feature points based on the second pixel distance to obtain the first processed feature points includes: Arrange the first feature points based on the numerical relationship between the second pixel distances to obtain a sequence of feature points; Obtain the first feature points within a preset sequence interval in the sequence of feature points as the first processed feature points.
8. The method according to claim 1, wherein The obtaining of the first displacement matrix based on the second feature points and the first processed feature points includes: Based on the second feature points and the first processed feature points, obtain the first average displacement in the first direction and the second average displacement in the second direction of the processed image and the image to be processed; Based on the first displacement in the first direction and the second displacement in the second direction of the first processed feature points, obtain the first displacement matrix.
9. The method according to claim 1, wherein The obtaining of the second processed feature points of the processed image based on the first displacement matrix, the second feature points and the first processed feature points includes: Reproject the second feature points based on the first displacement matrix to obtain the fourth feature points corresponding to the first feature points; Filter the first processed feature points based on the fourth feature points to obtain the feature points to be processed of the processed image; Obtain the rolling shutter correction (RSC) information of the image acquisition device; Correct the feature points to be processed based on the RSC information to obtain the second processed feature points.
10. The method according to claim 1, wherein The processed image and the image to be processed are captured by the image acquisition device based on a telephoto lens.
11. An image processing apparatus, characterized in that, It includes: An acquisition module, configured to acquire a processed image and an image to be processed acquired by an image acquisition device; wherein, the processed image and the image to be processed are consecutive frame images; A feature extraction module, configured to obtain the first feature points in the processed image; A feature tracking module, configured to obtain the second feature points of the image to be processed and the first processed feature points of the processed image based on the first feature points; A first processing module, configured to obtain a first displacement matrix based on the second feature points and the first processed feature points; A second processing module, configured to obtain the second processed feature points of the processed image based on the first displacement matrix, the second feature points and the first processed feature points; A third processing module, configured to obtain a second displacement matrix based on the second processed feature points and the second feature points; A fourth processing module, configured to process the image to be processed based on the second displacement matrix.
12. An electronic device, characterized in that, The device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 10.
13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.