Image processing method, video processing method, device and electronic equipment

Abstract

The present disclosure relates to an image processing method, a video processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The method obtains an alignment matrix corresponding to a to-be-processed image and a reference image, obtains target vertices after the vertices of the to-be-processed image are transformed to the reference image according to the alignment matrix, obtains a correction point set according to the sizes of the to-be-processed image and the reference image when there is a set of relative target vertices as the target angle, and constructs a target alignment matrix according to the correction point set, and performs alignment processing on the to-be-processed image using the target alignment matrix to obtain the target image after the alignment processing. Since this embodiment uses a correction point set to update the alignment matrix to obtain a target alignment matrix when it is recognized that the alignment estimation result is deformed, so as to resist erroneous transformation points, and then performs alignment processing through the target alignment matrix to achieve correction of the alignment estimation result, thereby achieving a better alignment effect.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, a video processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the advancement of image processing technology, video has gradually become a new medium for disseminating information. Compared with still images, videos contain the timing information of previous and next frames, which can coherently describe the environment and experience of the photographer at that time. However, when there is a large misalignment between adjacent frames, it will seriously affect the stability of the video.

[0003] In the related art, the latter frame can be transformed into the previous frame through a video stabilization algorithm to improve the stability of the video. Currently, the more commonly used video stabilization algorithm mainly adopts the homography matrix algorithm. However, since the traditional homography matrix is ​​based on the result of the feature extraction operator to construct a mapping matrix, when the video shooting environment is poor, it will seriously affect the shooting quality, thereby affecting the feature extraction. For example, in high-exposure scenes, low-light scenes, motion blur and other scenes, the feature points extracted from the picture will be insufficient or wrong. This insufficient or wrong information will cause errors in the constructed homography matrix, which will cause the aligned result to be severely distorted. Therefore, when the feature extraction is inaccurate, the current homography matrix algorithm will produce a large estimation error, thereby affecting the picture quality. Summary of the invention

[0004] The present disclosure provides an image processing method, a video processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem of homography transformation error caused by inaccurate feature extraction in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0006] Acquire an image to be processed and a reference image, and acquire a corresponding alignment matrix according to the image to be processed and the reference image;

[0007] According to the alignment matrix, obtaining target vertices of the image to be processed after the vertices are transformed to the reference image;

[0008] When there is a set of relative target vertex angles that are target angles, a correction point set is obtained according to the sizes of the image to be processed and the reference image;

[0009] A target alignment matrix is ​​constructed according to the correction point set, and the image to be processed is aligned using the target alignment matrix to obtain an aligned target image.

[0010] In one of the embodiments, the method further comprises: obtaining the coordinates of the target vertex after the vertex of the image to be processed is transformed into the reference image according to the alignment matrix; and calculating the angle corresponding to the target vertex using trigonometric functions according to the coordinates of the target vertex.

[0011] In one of the embodiments, after obtaining the target vertex after the vertex of the image to be processed is transformed into the reference image according to the alignment matrix, the method further includes: obtaining the coordinates of the target vertex, and calculating the cosine value of the target vertex using the cosine theorem according to the coordinates of the target vertex; when the cosine value of the target vertex meets a set threshold, determining that the angle of the target vertex is the target angle.

[0012] In one of the embodiments, the image to be processed is equal to the image size of the reference image, the image includes a first length in a first coordinate direction and a second length in a second coordinate direction, the first coordinate direction and the second coordinate direction are perpendicular to each other; obtaining the correction point set according to the sizes of the image to be processed and the reference image includes: within the first length range of the image, determining a first coordinate set in the first coordinate direction according to a first set length interval, the first coordinate set includes a plurality of first coordinates; within the second length range of the image, determining a second coordinate set in the second coordinate direction according to a second set length interval, the second coordinate set includes a plurality of second coordinates; and arranging and combining the plurality of first coordinates and the plurality of second coordinates respectively to obtain a coordinate point set after the arrangement and combination, and using the coordinate point set as the correction point set.

[0013] In one of the embodiments, the image to be processed is equal to the image size of the reference image, and the image includes a first length in a first coordinate direction and a second length in a second coordinate direction; obtaining the correction point set according to the sizes of the image to be processed and the reference image includes: determining a first coordinate set in a first coordinate direction within a first length range of the image according to a first set number, the first coordinate set including a plurality of first coordinates; determining a second coordinate set in a second coordinate direction within a second length range of the image according to a second set number, the second coordinate set including a plurality of second coordinates; and arranging and combining the plurality of first coordinates and the plurality of second coordinates respectively to obtain a coordinate point set after the arrangement and combination, and using the coordinate point set as the correction point set.

[0014] In one of the embodiments, obtaining a corresponding alignment matrix based on the image to be processed and the reference image includes: extracting feature points of the image to be processed to obtain a first feature point set, and extracting feature points of the reference image to obtain a second feature point set; matching feature points based on the first feature point set and the second feature point set to determine a plurality of matched feature point pairs, wherein the feature point pairs include corresponding first feature points and second feature points; and constructing a corresponding alignment matrix based on the plurality of feature point pairs.

[0015] In one of the embodiments, constructing a target alignment matrix based on the correction point set includes: performing feature matching processing on the image to be processed and the reference image according to the correction point set to obtain a feature matching result; and constructing a target alignment matrix based on the feature matching result.

[0016] In one embodiment, the target alignment matrix is ​​a homography matrix; the use of the target alignment matrix to align the image to be processed to obtain the aligned target image includes: according to the homography matrix, using a homography transformation method to calculate the original pixel coordinates of the image to be processed and transform them into the target coordinates of the reference image; according to the transformed target coordinates of each pixel of the image to be processed, obtaining the target image after the image to be processed is aligned with the reference image.

[0017] According to a second aspect of an embodiment of the present disclosure, a video processing method is provided, the method comprising:

[0018] Acquire a video to be processed, and determine at least one adjacent frame to be stabilized from the video to be processed, wherein the adjacent frame includes a reference image located in a previous frame and an image to be processed located in a subsequent frame;

[0019] The image to be processed in the adjacent frames is processed using the image processing method described in the first aspect to obtain a target image of the image to be processed in each of the adjacent frames;

[0020] The target image is replaced with a corresponding frame in the video to be processed to obtain a processed target video.

[0021] In one of the embodiments, determining at least one adjacent frame to be stabilized from the video to be processed includes: performing frame processing on the video to be processed to obtain each frame image of the video to be processed; performing feature extraction on each frame image to obtain image features of each frame image; and determining at least one adjacent frame to be stabilized based on the image features of each frame image.

[0022] In one of the embodiments, determining at least one adjacent frame to be stabilized based on the image features of each frame image includes: for any two adjacent frames of images, obtaining the feature matching degree between the image features of the latter frame image and the image features of the previous frame image; when the feature matching degree is less than a set value, determining that the two adjacent frames of images are adjacent frames to be stabilized.

[0023] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0024] A matrix acquisition module is configured to acquire an image to be processed and a reference image, and acquire a corresponding alignment matrix according to the image to be processed and the reference image;

[0025] A coordinate acquisition module is configured to acquire, according to the alignment matrix, target vertices of the image to be processed after transformation to the reference image;

[0026] A correction point set acquisition module is configured to acquire a correction point set according to the sizes of the image to be processed and the reference image when there is a set of relative target vertices with angles being the target angle;

[0027] The alignment processing module is configured to construct a target alignment matrix according to the correction point set, and use the target alignment matrix to align the image to be processed to obtain the target image after the alignment processing.

[0028] In one of the embodiments, the device further includes an angle acquisition module configured to execute: acquiring the coordinates of the target vertex; and calculating the angle corresponding to the target vertex using a trigonometric function according to the coordinates of the target vertex.

[0029] In one of the embodiments, the device also includes an angle determination module, which is configured to execute: obtaining the coordinates of the target vertex, and calculating the cosine value of the target vertex using the cosine theorem based on the coordinates of the target vertex; when the cosine value of the target vertex meets a set threshold, determining the angle of the target vertex as the target angle.

[0030] In one of the embodiments, the image to be processed is equal to the image size of the reference image, the image includes a first length in a first coordinate direction and a second length in a second coordinate direction, and the first coordinate direction and the second coordinate direction are perpendicular to each other; the correction point set acquisition module is configured to execute: within the first length range of the image, determine a first coordinate set in the first coordinate direction according to a first set length interval, the first coordinate set includes a plurality of first coordinates; within the second length range of the image, determine a second coordinate set in the second coordinate direction according to a second set length interval, the second coordinate set includes a plurality of second coordinates; arrange and combine the plurality of first coordinates and the plurality of second coordinates respectively to obtain a coordinate point set after arrangement and combination, and use the coordinate point set as the correction point set.

[0031] In one of the embodiments, the image to be processed is equal to the image size of the reference image, and the image includes a first length in the first coordinate direction and a second length in the second coordinate direction; the correction point set acquisition module is also configured to execute: determining a first coordinate set in the first coordinate direction within the first length range of the image according to a first set number, the first coordinate set including multiple first coordinates; determining a second coordinate set in the second coordinate direction within the second length range of the image according to a second set number, the second coordinate set including multiple second coordinates; arranging and combining the multiple first coordinates and the multiple second coordinates respectively to obtain a coordinate point set after the arrangement and combination, and using the coordinate point set as the correction point set.

[0032] In one of the embodiments, the matrix acquisition module is further configured to execute: extracting feature points from the image to be processed to obtain a first feature point set, and extracting feature points from the reference image to obtain a second feature point set; matching feature points based on the first feature point set and the second feature point set to determine a plurality of matching feature point pairs, wherein the feature point pairs include corresponding first feature points and second feature points; and constructing a corresponding alignment matrix based on the plurality of feature point pairs.

[0033] In one of the embodiments, the alignment processing module is further configured to execute: performing feature matching processing on the image to be processed and the reference image according to the correction point set to obtain a feature matching result; and constructing a target alignment matrix according to the feature matching result.

[0034] In one embodiment, the target alignment matrix is ​​a homography matrix; the alignment processing module is further configured to execute: according to the homography matrix, using a homography transformation method to calculate the transformation of the original pixel coordinates of the image to be processed to the target coordinates of the reference image; according to the transformed target coordinates of each pixel of the image to be processed, obtaining the target image after the image to be processed is aligned with the reference image.

[0035] According to a fourth aspect of an embodiment of the present disclosure, a video processing device is provided, the device comprising:

[0036] A to-be-processed video acquisition module is configured to acquire the to-be-processed video, and determine at least one adjacent frame to be stabilized from the to-be-processed video, wherein the adjacent frame includes a reference image located in a previous frame and an image to be processed located in a subsequent frame;

[0037] The image processing device as described in the third aspect above is configured to process the images to be processed in the adjacent frames to obtain a target image of the images to be processed in each of the adjacent frames;

[0038] The target video acquisition module is configured to replace the target image with a corresponding frame in the video to be processed to obtain a processed target video.

[0039] In one of the embodiments, the module for acquiring the video to be processed further includes: a frame segmentation unit, configured to perform frame segmentation processing on the video to be processed to obtain each frame image of the video to be processed; a feature extraction unit, configured to perform feature extraction on each frame image respectively to obtain image features of each frame image; and an adjacent frame determination unit, configured to determine at least one adjacent frame to be stabilized based on the image features of each frame image.

[0040] In one of the embodiments, the adjacent frame determination unit is configured to perform: for any two adjacent frames of images, obtain the feature matching degree between the image features of the latter frame image and the image features of the previous frame image; when the feature matching degree is less than a set value, determine that the two adjacent frames of images are adjacent frames to be stabilized.

[0041] According to the fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the image processing method as described in the first aspect above and the video processing method as described in the second aspect above.

[0042] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method described in the first aspect and the video processing method described in the second aspect.

[0043] According to the seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which includes instructions. When the instructions are executed by a processor of an electronic device, the electronic device can execute the image processing method described in the first aspect and the video processing method described in the second aspect.

[0044] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: by obtaining the image to be processed and the reference image, obtaining the corresponding alignment matrix according to the image to be processed and the reference image, obtaining the target vertices after the vertices of the image to be processed are transformed to the reference image according to the alignment matrix, and when there is a set of relative target vertices whose angles are the target angles, obtaining the correction point set according to the sizes of the image to be processed and the reference image, and constructing the target alignment matrix according to the correction point set, and using the target alignment matrix to align the image to be processed, thereby obtaining the target image after the alignment processing. Because this embodiment uses the correction point set to update the alignment matrix to obtain the target alignment matrix when it is recognized that the alignment estimation result is deformed, so as to resist the erroneous transformation points, and then performs alignment processing through the target alignment matrix to achieve correction of the alignment estimation result, so as to achieve a better alignment effect.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0047] Figure 1 The figure is a schematic diagram showing a homography transformation application according to an exemplary embodiment.

[0048] Figure 2 is a schematic diagram showing an image alignment effect according to an exemplary embodiment.

[0049] Figure 3 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0050] Figure 4 is a coordinate schematic diagram shown according to an exemplary embodiment.

[0051] Figure 5 is a flowchart of an image processing method according to another exemplary embodiment.

[0052] Figure 6 is a flowchart showing the steps of obtaining a correction point set according to an exemplary embodiment.

[0053] Figure 7 is a flowchart of steps for obtaining a correction point set according to another exemplary embodiment.

[0054] Figure 8 is a flowchart showing steps of obtaining an alignment matrix according to an exemplary embodiment.

[0055] Fig. 9 The figure is a flowchart showing the steps of obtaining a target image according to an exemplary embodiment.

[0056] Fig.10 The figure is a flowchart of a video processing method according to an exemplary embodiment.

[0057] Fig.11 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0058] Fig.12 The invention is a block diagram of a video processing device according to an exemplary embodiment.

[0059] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0060] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0063] Since traditional video stabilization algorithms mainly use algorithms based on homography matrices, such as Figure 1 As shown, the images of adjacent frames are defined as In and Ir, which are scenes of the same plane S taken by two cameras at different angles. In order to play In and Ir more stably, In and Ir are usually aligned, that is, one of the images is transformed into the other image. This embodiment is illustrated by aligning In to Ir as an example. First, the two-dimensional feature point m matrix of In and the two-dimensional feature point m' matrix of Ir are obtained through a feature extraction algorithm. Then the minimum mean square error is used to construct the mapping matrix H of the two planes. Here, H can be a homography transformation matrix, and the size of H is 3x3. After obtaining the transformation matrix H, it can be matrix multiplied with the coordinates of the image In, and the new coordinates obtained are the result of transforming In to Ir.

[0064] Since the homography matrix has the characteristic of edge preservation, there will be no texture artifacts in the transformed image. Therefore, under certain ideal conditions, the traditional homography matrix algorithm has good performance. For example, when the captured image is clear or there is only slight shaking of the camera as a whole, a better transformation effect can be obtained through the traditional homography matrix algorithm. However, since the homography transformation is a transformation from one plane to another, it relies on the two-dimensional feature points on the two planes to construct the transformation matrix of the plane. This dependency will lead to estimation errors of the transformation matrix when the feature extraction is inaccurate, which will cause serious distortion of the alignment result. Figure 2 As shown, the left side is the image before transformation, and the right side is the image after transformation using the traditional homography matrix algorithm. From the results, the transformed image has a distortion of an erroneous parallelogram.

[0065] Based on this, the present disclosure provides an image processing method. This embodiment uses the method applied to a terminal as an example, wherein the terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices that can store and process videos. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. In this embodiment, if Figure 3 As shown, the method may include the following steps:

[0066] In step S310, the image to be processed and the reference image are obtained, and the corresponding alignment matrix is ​​obtained according to the image to be processed and the reference image.

[0067] The image to be processed and the reference image may be frame images corresponding to adjacent frames in the video. For example, when adjacent frames in the video are significantly misaligned, the stability of the video will be affected. Therefore, in this embodiment, the stability of the video is improved by aligning the latter frame of the adjacent frames with the former frame. Specifically, the latter frame of the adjacent frames may be the image to be processed, that is, the image that needs to be aligned, and the former frame may be the reference image, that is, the reference during the alignment process.

[0068] The alignment matrix is ​​a matrix used to align images. The alignment matrix can reflect the transformation relationship from one image to the other of the two images. Usually, when two images are known, the corresponding alignment matrix can be solved by feature matching. Specifically, the alignment matrix can be a homography matrix.

[0069] In this embodiment, when alignment processing is required for images, the corresponding image to be processed and the reference image are first obtained, and then the corresponding alignment matrix is ​​obtained based on the image to be processed and the reference image.

[0070] In step S320, according to the alignment matrix, target vertices of the image to be processed after transformation to the reference image are obtained.

[0071] The vertices of the image to be processed refer to the common vertices of two boundary lines in the image to be processed. Usually, the image is a rectangular structure, so the vertices of the image to be processed are the positions of the four corners of the rectangular boundary lines of the image. The target vertex is the vertex of the image obtained by aligning the image to be processed to the reference image.

[0072] Specifically, the position of the target vertex can be determined based on the corresponding coordinates. In this embodiment, the alignment matrix obtained above is matrix-multiplied with the vertex coordinates of the image to be processed, so that the position of the target vertex after transformation to the reference image can be obtained. It should be noted that in the case of a known image, the coordinates of each pixel in the image can be easily and quickly read by a computer device.

[0073] In step S330, when there is a set of angles of relative target vertices that are target angles, a correction point set is obtained according to the sizes of the image to be processed and the reference image.

[0074] Among them, the relative target vertices can be understood as two target vertices without a common edge, that is, the vertices corresponding to the diagonal corners in the transformed image. The angle of the target vertex is the size of the angle corresponding to the target vertex. The target angle is a standard for judging whether the transformed image is distorted, which can be a pre-set angle range. Specifically, the target angle in this embodiment can be an acute angle greater than 0 degrees and less than 90 degrees.

[0075] The correction point set is a set of feature points used to correct the transformed image when it is determined that the transformed image is distorted. Specifically, the correction point set is feature points extracted from the image according to the size of the image to be processed and the reference image.

[0076] In this embodiment, the coordinates of the target vertex can be obtained, and the angle of the corresponding target vertex can be obtained by calculation according to the coordinates of the target vertex. Then, it is determined whether the angle of the target vertex is the target angle. When it is determined that there is a set of relative target vertex angles that are the target angles, it means that the transformed image is distorted and needs to be corrected. Therefore, according to the size of the image to be processed and the reference image, a correction point set is obtained, and correction processing is performed based on subsequent steps.

[0077] In step S340, a target alignment matrix is ​​constructed according to the correction point set, and the image to be processed is aligned using the target alignment matrix to obtain an aligned target image.

[0078] The target image is the result of the alignment process on the image to be processed, that is, the image after the image to be processed is aligned with the reference image. The target alignment matrix is ​​an alignment matrix reconstructed based on the feature points in the correction point set.

[0079] Specifically, the target alignment matrix is ​​reconstructed according to the correction point set obtained above, and the image to be processed is aligned using the target alignment matrix, that is, the image to be processed is aligned with the reference image through the target alignment matrix, so as to obtain the aligned target image.

[0080] In the above-mentioned image processing method, by obtaining the image to be processed and the reference image, the corresponding alignment matrix is ​​obtained according to the image to be processed and the reference image, and according to the alignment matrix, the target vertices after the vertices of the image to be processed are transformed to the reference image are obtained. When there is a set of relative target vertices whose angles are the target angles, a correction point set is obtained according to the sizes of the image to be processed and the reference image, and a target alignment matrix is ​​constructed according to the correction point set, and the image to be processed is aligned using the target alignment matrix to obtain the aligned target image. Since this embodiment uses the correction point set to update the alignment matrix to obtain the target alignment matrix when it is recognized that the alignment estimation result is deformed, so as to resist the erroneous transformation points, and then the alignment is performed through the target alignment matrix to achieve the correction of the alignment estimation result, so as to achieve a better alignment effect.

[0081] In an exemplary embodiment, in step S320, after obtaining the target vertex after the vertex of the image to be processed is transformed into the reference image according to the alignment matrix, the above method may also include: obtaining the coordinates of the target vertex, and calculating the angle corresponding to the target vertex using trigonometric functions according to the coordinates of the target vertex.

[0082] Specifically, the alignment matrix obtained above is matrix-multiplied with the vertex coordinates of the image to be processed, so that the coordinates of the target vertex after transformation to the reference image can be obtained. When the coordinates of the target vertex are obtained, the corresponding coordinate vector can be calculated based on the coordinates of each target vertex. Then, based on the trigonometric function law, the corresponding angle is solved according to the coordinate vector of each target vertex. For example, Figure 4 As shown, if the coordinates of the four target vertices obtained include: A (X1, Y1), B (X2, Y2), C (X3, Y3) and D (X4, Y4), then the AB vector is (X2-X1, Y2-Y1), the AC vector is (X3-X1, Y3-Y1), and similarly, the CD vector and the BD vector can be obtained. Then, the corresponding angle can be solved by trigonometric functions. For example, taking the arc cosine as an example, the angle A = Arccos {[(X2-X1) × (X3-X1) + (Y2-Y1) × (Y3-Y1)] / |AB||AC|}, where |AB| = [(X2-X1) 2 +(Y2-Y1) 2 ] / 2, |AC|=[(X3-X1) 2 +(Y3-Y1) 2 ] / 2. Then the specific angle of the target vertex A can be obtained by calculating this formula. Similarly, the angles corresponding to other target vertices can be obtained. Of course, other trigonometric functions can also be used to solve the angles corresponding to the target vertices, and this embodiment does not limit this.

[0083] In the above embodiment, by acquiring the coordinates of the target vertex, the angle corresponding to the target vertex can be conveniently and quickly calculated using trigonometric functions, and then the size of the angle of the target vertex and the target angle can be judged to determine whether correction is needed.

[0084] In an exemplary embodiment, if Figure 5 As shown, in step S320, after obtaining the coordinates of the target vertices of the vertices of the image to be processed after transformation to the reference image according to the alignment matrix, the method may further include:

[0085] In step S322, the coordinates of the target vertex are obtained, and the cosine value of the target vertex is calculated using the cosine theorem according to the coordinates of the target vertex.

[0086] The cosine value is the cosine parameter of the target vertex calculated based on the cosine theorem of trigonometric functions. Figure 4 As shown, based on the cosine law, cosA = [(X2-X1) × (X3-X1) + (Y2-Y1) × (Y3-Y1)] / |AB||AC|, and the obtained cosA is the cosine value corresponding to the target vertex A. Similarly, the function values ​​corresponding to other target vertices can be obtained. Of course, the function value of the corresponding target vertex can also be solved by using a sine function or other trigonometric functions, which is not limited in this embodiment.

[0087] In step S324, when the cosine value of the target vertex meets the set threshold, the angle of the target vertex is determined to be the target angle.

[0088] Among them, the set threshold is the range of cosine values ​​corresponding to the preset target angle. Since this embodiment mainly corrects the distortion of the parallelogram, and based on the characteristics of the parallelogram, it can be known that there is a set of acute angles in the parallelogram, that is, the target angle can be determined to be an acute angle, and the cosine value corresponding to the acute angle is greater than 0. Therefore, when the target angle is an acute angle, the set threshold in this embodiment can be 0. Then, when the cosine value of the target vertex is greater than 0, it means that the cosine value of the target vertex meets the set threshold, and the angle of the target vertex can be determined to be the target angle.

[0089] In the above embodiment, the coordinates of the target vertex are obtained, and the cosine value of the target vertex is calculated using the cosine theorem, and when the cosine value of the target vertex meets the set threshold, the angle of the target vertex is determined to be the target angle. Therefore, it is convenient and quick to judge whether the angle of the target vertex meets the target angle, that is, it is quick to determine whether correction is needed.

[0090] In an exemplary embodiment, the image to be processed is equal to the image size of the reference image, and the image includes a first length located in a first coordinate direction and a second length located in a second coordinate direction; then Figure 6 As shown, in step S330, according to the sizes of the image to be processed and the reference image, a correction point set is obtained, which may specifically include:

[0091] In step S610, within a first length range of the image, a first coordinate set in a first coordinate direction is determined according to a first set length interval.

[0092] The first coordinate direction and the second coordinate direction may be the horizontal coordinate direction and the vertical coordinate direction, respectively. For example, when the first coordinate direction is the horizontal coordinate direction, the second coordinate direction is the vertical coordinate direction, and when the first coordinate direction is the vertical coordinate direction, the second coordinate direction is the horizontal coordinate direction. Therefore, the first length and the second length are the width and height of the image, respectively. For example, when the first length is the width of the image, the second length is the height of the image, and when the first length is the height of the image, the second length is the width of the image.

[0093] The first coordinate set includes a plurality of first coordinates. The first length range refers to the range of the first length of the image. For example, if the first length is d1, the first length range of the image refers to the range of the length d1 of the image. The first set length interval is a preset interval for determining the first coordinate on the first length of the image.

[0094] In this embodiment, within the first length range of the image, the first coordinate set in the first coordinate direction is determined according to the first set length interval. For example, if the first length is d1 and the first set length interval is a, then a first coordinate is determined for each interval length a on the first length of the image to be processed, thereby obtaining the corresponding first coordinate set [0, a, 2a, ..., d1].

[0095] In step S620, within a second length range of the image, a second coordinate set in a second coordinate direction is determined according to a second set length interval.

[0096] The second coordinate set includes a plurality of second coordinates. The second length range refers to the range of the second length of the image. For example, if the second length is d2, the second length range of the image refers to the range of the length d2 of the image. The second set length interval is a preset interval for determining the second coordinate on the second length of the image.

[0097] In this embodiment, within the second length range of the image, the second coordinate set in the second coordinate direction is determined according to the second set length interval. For example, if the second length is d2 and the second set length interval is b, then a second coordinate is determined for each interval length b on the second length of the image to be processed, thereby obtaining a corresponding second coordinate set [0, b, 2b, ..., d2].

[0098] In step S630, the plurality of first coordinates and the plurality of second coordinates are arranged and combined respectively to obtain an arranged and combined coordinate point set, and the coordinate point set is used as a correction point set.

[0099] Specifically, by arbitrarily permuting and combining the first coordinates in the first coordinate set and the second coordinates in the second coordinate set, a plurality of coordinate points after permutation and combination are obtained, and the coordinate point set composed of the plurality of coordinate points is the correction point set.

[0100] In the above embodiment, a first coordinate set in a first coordinate direction is determined according to a first set length interval within a first length range of the image, and a second coordinate set in a second coordinate direction is determined according to a second set length interval within a second length range of the image, and a plurality of first coordinates and a plurality of second coordinates are arranged and combined respectively to obtain an arranged and combined coordinate point set, and the coordinate point set is used as a correction point set. Since the correction point set in this embodiment includes a plurality of coordinate points, the alignment matrix due to insufficient or erroneous feature point extraction can be corrected, thereby improving the image alignment effect.

[0101] In an exemplary embodiment, the image to be processed is equal to the image size of the reference image, and the image includes a first length located in a first coordinate direction and a second length located in a second coordinate direction; then Figure 7 As shown, in step S330, according to the sizes of the image to be processed and the reference image, a correction point set is obtained, which may specifically include:

[0102] In step S710, a first coordinate set in a first coordinate direction is determined within a first length range of the image according to a first set number.

[0103] The first coordinate set includes a plurality of first coordinates. The first length range refers to the range of the first length of the image. For example, if the first length is d1, the first length range of the image refers to the range of the length d1 of the image. The first set number is a preset number of first coordinates determined on the first length of the image.

[0104] In this embodiment, within the first length range of the image, the first coordinate set in the first coordinate direction is determined according to the first set number. For example, if the first length is d1 and the first set number is n1, the first length d1 can be evenly distributed based on the first set number n1, and the corresponding evenly distributed point is the first coordinate, such as obtaining the corresponding first coordinate set [0, d1 / n1, 2(d1 / n1), ..., d1], where the number of elements in the set is n1.

[0105] In step S720, a second coordinate set in a second coordinate direction is determined within a second length range of the image according to a second set number.

[0106] Similarly, the second coordinate set includes a plurality of second coordinates. The second length range refers to the range of the second length of the image. For example, if the second length is d2, the second length range of the image refers to the range of the length d2 of the image. The second set number is a preset number of second coordinates determined on the second length of the image.

[0107] In this embodiment, within the second length range of the image, the second coordinate set in the second coordinate direction is determined according to the second set number. For example, if the second length is d2 and the second set number is n2, the second length d2 can be evenly distributed based on the second set number n2, and the corresponding evenly distributed points are the second coordinates, such as the corresponding second coordinate set [0, d2 / n2, 2(d2 / n2), ..., d2], where the number of elements in the set is n2.

[0108] In step S730, the plurality of first coordinates and the plurality of second coordinates are arranged and combined respectively to obtain an arranged and combined coordinate point set, and the coordinate point set is used as a correction point set.

[0109] Specifically, by arbitrarily permuting and combining the first coordinates in the first coordinate set and the second coordinates in the second coordinate set, a plurality of coordinate points after permutation and combination are obtained, and the coordinate point set composed of the plurality of coordinate points is the correction point set.

[0110] In the above embodiment, a first coordinate set in the first coordinate direction is determined according to a first set number within the first length range of the image, and a second coordinate set in the second coordinate direction is determined according to a second set number within the second length range of the image, and the plurality of first coordinates and the plurality of second coordinates are arranged and combined respectively to obtain an arranged and combined coordinate point set, and the coordinate point set is used as a correction point set. Since the correction point set in this embodiment includes a plurality of coordinate points, the alignment matrix due to insufficient or erroneous feature point extraction can be corrected, thereby improving the image alignment effect.

[0111] In an exemplary embodiment, if Figure 8 As shown, in step S310, the corresponding alignment matrix is ​​obtained according to the image to be processed and the reference image, which may specifically include the following steps:

[0112] In step S312, feature points are extracted from the image to be processed to obtain a first feature point set, and feature points are extracted from the reference image to obtain a second feature point set.

[0113] Specifically, feature point extraction may be performed using SIFT (Scale-invariant feature transform) or ORB (Oriented Fast and Rotated Brief). In this embodiment, feature points are extracted from the processed image to obtain a first feature point set, and feature points are extracted from the reference image to obtain a second feature point set.

[0114] In step S314, feature point matching is performed based on the first feature point set and the second feature point set to determine a plurality of matched feature point pairs.

[0115] Among them, the feature point pair includes the corresponding first feature point and second feature point. In this embodiment, after the feature point sets of the two images are extracted respectively, the extracted feature points can be matched. Specifically, the Euclidean distance of the key point feature vector can be used as a similarity determination metric for the key points in the two images. For example, take a first feature point in the first feature point set, and find the two second feature points with the closest distance in the second feature point set by traversal. Among these two second feature points, if the closest distance divided by the next closest distance is less than a certain threshold, the first feature point and the second feature point corresponding to the closest distance are determined to be a matching feature point pair. Based on this, multiple matching feature point pairs can be determined.

[0116] In step S316, a corresponding alignment matrix is ​​constructed according to the plurality of feature point pairs.

[0117] Specifically, based on the multiple feature point pairs extracted above, the corresponding alignment matrix can be constructed by using the minimum mean square error method, or the corresponding alignment matrix can be solved by using the least square method.

[0118] In the above embodiment, feature points are extracted from the processed image to obtain a first feature point set, and feature points are extracted from the reference image to obtain a second feature point set. Feature point matching is performed based on the first feature point set and the second feature point set, multiple matching feature point pairs are determined, and a corresponding alignment matrix is ​​constructed based on the multiple feature point pairs. The coordinates of the target vertices after the image transformation can be obtained based on the alignment matrix, and then it is determined whether the transformed image needs to be corrected to improve the image alignment effect.

[0119] In an exemplary embodiment, in step S340, constructing a target alignment matrix based on the correction point set may specifically include: performing feature matching processing on the processed image and the reference image based on the correction point set to obtain feature matching results; and then constructing the target alignment matrix based on the feature matching results.

[0120] The feature matching process can be referred to as Figure 8 The embodiment shown is different from the above-mentioned embodiment in that the present embodiment performs feature matching processing based on the correction point set, that is, the present embodiment determines multiple matching feature point pairs, i.e., feature matching results, from the correction point set, and replaces the original alignment matrix, i.e., obtains the target alignment matrix, and then performs coarse alignment on the image, thereby improving the alignment effect.

[0121] Although the correction of this embodiment is only a simple alignment of the processed image, it is of great help to the continuity between frames of the video. Even in a video with a slight shake of the picture, as long as there is a parallelogram distortion caused by the matrix error, the picture will shake violently, which will bring a bad experience to the viewer. The method disclosed in this invention can improve the shaking of the picture and increase the continuity of the video. In particular, for images in special scenes with poor shooting environment, it can better resist wrong transformation points to improve the image alignment effect.

[0122] In an exemplary embodiment, the target alignment matrix is ​​a homography matrix, then Fig. 9 As shown, in step S340, the target alignment matrix is ​​used to align the image to be processed to obtain the aligned target image, which may specifically include:

[0123] In step S342, according to the homography matrix, a homography transformation method is used to calculate the pixel original coordinates of the image to be processed and transform them into target coordinates of the reference image.

[0124] The original pixel coordinates of the image to be processed refer to the coordinates of the pixels in the image to be processed in the image to be processed, and the target coordinates refer to the coordinates of the pixels in the image to be processed after being aligned with the reference image.

[0125] Specifically, the alignment matrix obtained above is matrix-multiplied with the original pixel coordinates of the image to be processed by a homography transformation method, and the result obtained is the transformation of the original pixel coordinates of the image to be processed to the target coordinates of the reference image.

[0126] In step S344, a target image is obtained after the image to be processed is aligned with the reference image according to the target coordinates of each pixel of the image to be processed after transformation.

[0127] The target image is the result obtained after the image to be processed is aligned with the reference image. Specifically, the target image after the image to be processed is transformed to the reference image is obtained according to the target coordinates after each pixel of the image to be processed is transformed.

[0128] In the above embodiment, according to the homography matrix, the homography transformation method is used to calculate the transformation of the original pixel coordinates of the image to be processed to the target coordinates of the reference image, and the target image after the image to be processed is aligned with the reference image is obtained according to the target coordinates after each pixel of the image to be processed is transformed. Since the homography matrix for image alignment in this embodiment is obtained after correction processing based on the correction point set, the matrix error in special scenarios can be optimized, thereby improving the alignment effect.

[0129] In an exemplary embodiment, if Fig.10 As shown, the present disclosure also provides a video processing method, comprising the following steps:

[0130] In step S1010, a video to be processed is obtained, and at least one adjacent frame to be stabilized is determined from the video to be processed.

[0131] The video to be processed refers to a video with poor stability and screen shaking. Generally, if there is a large misalignment between adjacent frames in a video, the stability of the video will be seriously affected. The adjacent frames to be stabilized refer to adjacent frames in the video that are greatly misaligned. Specifically, the adjacent frames include a reference image located in the previous frame and an image to be processed located in the next frame.

[0132] In this embodiment, when there is a large misalignment between adjacent frames in a video, it is necessary to process them to improve the stability of the video. Specifically, at least one adjacent frame to be stabilized is determined from the video to be processed.

[0133] In step S1020, the above-mentioned image processing method is used to process the images to be processed in adjacent frames to obtain a target image of the images to be processed in each adjacent frame.

[0134] For details, please refer to Figures 3 to 9 The images to be processed in adjacent frames are processed in a manner to obtain a target image of the images to be processed in each adjacent frame, which will not be described in detail in this embodiment.

[0135] In step S1030, the target image is replaced with the corresponding frame in the video to be processed to obtain a processed target video.

[0136] The target video refers to a video that can be stably played after stabilizing the video to be processed. Since the target image is obtained after aligning a frame of the image to be processed, a relatively stable target video can be obtained by replacing the target image obtained by the above processing with the corresponding frame in the video to be processed.

[0137] In the above-mentioned video processing method, a video to be processed is obtained, at least one adjacent frame to be stabilized is determined from the video to be processed, and the above-mentioned image processing method is used to process the image to be processed in the adjacent frames to obtain a target image of the image to be processed in each adjacent frame, and then the target image is replaced with the corresponding frame in the video to be processed, so that a relatively stable target video can be obtained to improve the stability of the video.

[0138] In an exemplary embodiment, in step S1010, at least one adjacent frame to be stabilized is determined from the video to be processed, including: performing frame processing on the video to be processed to obtain each frame image of the video to be processed, performing feature extraction on each frame image to obtain image features of each frame image, and determining at least one adjacent frame to be stabilized according to the image features of each frame image. The image feature is a feature that can express the content of the frame image after feature extraction of the frame image. Specifically, the image feature can be a feature vector extracted from the frame image. In this embodiment, by performing frame processing on the video to be processed, each frame image of the video to be processed is obtained, and feature extraction is performed on each frame image to obtain image features of each frame image, and then at least one adjacent frame to be stabilized is determined according to the image features of each frame image.

[0139] In an exemplary embodiment, at least one adjacent frame to be stabilized is determined based on the image features of each frame of image, specifically comprising: for any two adjacent frames of image, obtaining the feature matching degree between the image features of the latter frame of image and the image features of the previous frame of image, and when the feature matching degree is less than a set value, determining that the two adjacent frames of image are adjacent frames to be stabilized. The feature matching degree may be the distance or similarity of the image features. The set value may be a preset distance threshold or similarity threshold of the image features, and its size may be set based on the actual scene. For example, when a higher stabilization effect is required, a larger set value may be set, and when a lower stabilization effect is required, a smaller set value may be set.

[0140] Specifically, in this embodiment, for any two adjacent frames of images in the video to be processed, the feature matching degree between the image features of the latter frame of image and the image features of the previous frame of image is obtained. When the feature matching degree is less than the set value, it means that the two adjacent frames of images are not aligned, resulting in unstable video. Therefore, the two adjacent frames of images can be determined as adjacent frames to be stabilized, and then processed to improve the stability of the video.

[0141] It should be understood that although Figure 3-Figure 10The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 3-Figure 10 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0142] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0143] Fig.11 FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment. Fig.11 The device includes a matrix acquisition module 1102, a coordinate acquisition module 1104, a correction point set acquisition module 1106 and an alignment processing module 1108.

[0144] The matrix acquisition module 1102 is configured to acquire the image to be processed and the reference image, and acquire the corresponding alignment matrix according to the image to be processed and the reference image;

[0145] A coordinate acquisition module 1104 is configured to acquire the coordinates of the target vertices of the vertices of the image to be processed after transformation to the reference image according to the alignment matrix;

[0146] The correction point set acquisition module 1106 is configured to acquire a correction point set according to the sizes of the image to be processed and the reference image when there is a set of relative target vertices with angles being the target angle;

[0147] The alignment processing module 1108 is configured to construct a target alignment matrix according to the correction point set, and use the target alignment matrix to align the image to be processed to obtain the aligned target image.

[0148] In an exemplary embodiment, the device further includes an angle acquisition module configured to execute: acquiring the coordinates of a target vertex, and calculating an angle corresponding to the target vertex using a trigonometric function according to the coordinates of the target vertex.

[0149] In an exemplary embodiment, the device also includes an angle determination module, which is configured to perform: obtaining the coordinates of the target vertex, and calculating the cosine value of the target vertex using the cosine theorem based on the coordinates of the target vertex; when the cosine value of the target vertex meets a set threshold, determining that the angle of the target vertex is the target angle.

[0150] In an exemplary embodiment, the image to be processed is equal to the image size of the reference image, and the image includes a first length in the first coordinate direction and a second length in the second coordinate direction; the correction point set acquisition module is configured to execute: within the first length range of the image, determine a first coordinate set in the first coordinate direction according to a first set length interval, the first coordinate set includes multiple first coordinates; within the second length range of the image, determine a second coordinate set in the second coordinate direction according to a second set length interval, the second coordinate set includes multiple second coordinates; arrange and combine the multiple first coordinates and the multiple second coordinates respectively to obtain a set of arranged and combined coordinate points, and use the set of coordinate points as the correction point set.

[0151] In an exemplary embodiment, the image to be processed is equal to the image size of the reference image, and the image includes a first length in the first coordinate direction and a second length in the second coordinate direction; the correction point set acquisition module is also configured to execute: determining a first coordinate set in the first coordinate direction within the first length range of the image according to a first set number, the first coordinate set including multiple first coordinates; determining a second coordinate set in the second coordinate direction within the second length range of the image according to a second set number, the second coordinate set including multiple second coordinates; arranging and combining the multiple first coordinates and the multiple second coordinates respectively to obtain a set of arranged and combined coordinate points, and using the set of coordinate points as the correction point set.

[0152] In an exemplary embodiment, the matrix acquisition module is further configured to execute: extracting feature points of the image to be processed to obtain a first feature point set, and extracting feature points of the reference image to obtain a second feature point set; performing feature point matching based on the first feature point set and the second feature point set to determine a plurality of matching feature point pairs, wherein the feature point pairs include corresponding first feature points and second feature points; and constructing a corresponding alignment matrix based on the plurality of feature point pairs.

[0153] In an exemplary embodiment, the alignment processing module is further configured to execute: performing feature matching processing on the image to be processed and the reference image according to the correction point set to obtain a feature matching result; and constructing a target alignment matrix according to the feature matching result.

[0154] In an exemplary embodiment, the target alignment matrix is ​​a homography matrix; the alignment processing module is further configured to execute: according to the homography matrix, using a homography transformation method to calculate the transformation of the original pixel coordinates of the image to be processed to the target coordinates of the reference image; according to the transformed target coordinates of each pixel of the image to be processed, obtaining the target image after the image to be processed is aligned to the reference image.

[0155] Fig.12 is a block diagram of a video processing device according to an exemplary embodiment. Fig.12 The device includes a to-be-processed video acquisition module 1202 , an image processing device 1204 and a target video acquisition module 1206 .

[0156] The to-be-processed video acquisition module 1202 is configured to acquire the to-be-processed video, and determine at least one adjacent frame to be stabilized from the to-be-processed video, wherein the adjacent frame includes a reference image located in a previous frame and an image to be processed located in a subsequent frame;

[0157] like Fig.11 The image processing device 1204 shown is configured to process the images to be processed in the adjacent frames to obtain a target image of the images to be processed in each of the adjacent frames;

[0158] The target video acquisition module 1206 is configured to replace the target image with a corresponding frame in the video to be processed to obtain a processed target video.

[0159] In an exemplary embodiment, the module for acquiring the video to be processed also includes: a frame segmentation unit, configured to perform frame segmentation processing on the video to be processed to obtain each frame image of the video to be processed; a feature extraction unit, configured to perform feature extraction on each frame image respectively to obtain image features of each frame image; and an adjacent frame determination unit, configured to determine at least one adjacent frame to be stabilized based on the image features of each frame image.

[0160] In an exemplary embodiment, the adjacent frame determination unit is configured to perform: for any two adjacent frames of images, obtain the feature matching degree between the image features of the latter frame image and the image features of the previous frame image; when the feature matching degree is less than a set value, determine that the two adjacent frames of images are adjacent frames to be stabilized.

[0161] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0162] Fig.131 is a block diagram of an electronic device Z00 for image processing according to an exemplary embodiment. For example, the electronic device Z00 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0163] Reference Fig.13 The electronic device Z00 may include one or more of the following components: a processing component Z02, a memory Z04, a power component Z06, a multimedia component Z08, an audio component Z10, an input / output (I / O) interface Z12, a sensor component Z14, and a communication component Z16.

[0164] The processing component Z02 generally controls the overall operation of the electronic device Z00, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component Z02 may include one or more processors Z20 to execute instructions to complete all or part of the steps of the above-described method. In addition, the processing component Z02 may include one or more modules to facilitate interaction between the processing component Z02 and other components. For example, the processing component Z02 may include a multimedia module to facilitate interaction between the multimedia component Z08 and the processing component Z02.

[0165] The memory Z04 is configured to store various types of data to support operations on the electronic device Z00. Examples of such data include instructions for any application or method operating on the electronic device Z00, contact data, phone book data, messages, pictures, videos, etc. The memory Z04 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0166] The power supply assembly Z06 provides power to various components of the electronic device Z00. The power supply assembly Z06 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device Z00.

[0167] The multimedia component Z08 includes a screen that provides an output interface between the electronic device Z00 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component Z08 includes a front camera and / or a rear camera. When the electronic device Z00 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0168] The audio component Z10 is configured to output and / or input audio signals. For example, the audio component Z10 includes a microphone (MIC), and when the electronic device Z00 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory Z04 or sent via the communication component Z16. In some embodiments, the audio component Z10 also includes a speaker for outputting an audio signal.

[0169] I / O interface Z12 provides an interface between processing component Z02 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0170] The sensor assembly Z14 includes one or more sensors for providing various aspects of status assessment for the electronic device Z00. For example, the sensor assembly Z14 can detect the open / closed state of the electronic device Z00, the relative positioning of the components, such as the display and keypad of the electronic device Z00, and the sensor assembly Z14 can also detect the position change of the electronic device Z00 or the components of the electronic device Z00, the presence or absence of contact between the user and the electronic device Z00, the orientation or acceleration / deceleration of the device Z00 and the temperature change of the electronic device Z00. The sensor assembly Z14 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly Z14 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly Z14 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0171] The communication component Z16 is configured to facilitate wired or wireless communication between the electronic device Z00 and other devices. The electronic device Z00 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component Z16 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component Z16 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0172] In an exemplary embodiment, the electronic device Z00 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0173] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory Z04 including instructions, and the above instructions can be executed by a processor Z20 of the electronic device Z00 to complete the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0174] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor Z20 of the electronic device Z00 to implement the above method.

[0175] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0176] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0177] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed and a reference image, and acquire a corresponding alignment matrix according to the image to be processed and the reference image; According to the alignment matrix, obtaining target vertices of the image to be processed after the vertices are transformed to the reference image; When there is a set of relative target vertex angles that are preset target angles, a correction point set is obtained according to the sizes of the image to be processed and the reference image, and the target angle is an angle range for determining whether the image to be processed after the transformation has distortion or deformation; Constructing a target alignment matrix according to the correction point set, and using the target alignment matrix to align the image to be processed to obtain an aligned target image; The image to be processed is equal to the image size of the reference image, the image includes a first length in a first coordinate direction and a second length in a second coordinate direction, the first coordinate direction and the second coordinate direction are perpendicular to each other, and the image includes any one of the image to be processed and the reference image; acquiring the correction point set according to the sizes of the image to be processed and the reference image includes: Within a first length range of the image, a first coordinate set in a first coordinate direction is determined according to a first set length interval, the first coordinate set includes a plurality of first coordinates; within a second length range of the image, a second coordinate set in a second coordinate direction is determined according to a second set length interval, the second coordinate set includes a plurality of second coordinates; the plurality of first coordinates and the plurality of second coordinates are respectively arranged and combined to obtain a set of arranged and combined coordinate points, and the set of coordinate points is used as a correction point set; The constructing a target alignment matrix according to the correction point set includes: performing feature matching processing on the image to be processed and the reference image according to the correction point set to obtain a feature matching result, and constructing a target alignment matrix according to the feature matching result.

2. The method according to claim 1, characterized in that The method further comprises: obtaining, according to the alignment matrix, target vertices of the image to be processed after the vertices are transformed to the reference image; Obtaining the coordinates of the target vertex; According to the coordinates of the target vertex, a trigonometric function is used to calculate the angle corresponding to the target vertex.

3. The method according to claim 1, characterized in that After obtaining the target vertices of the image to be processed after the vertices are transformed to the reference image according to the alignment matrix, the method further includes: Acquire the coordinates of the target vertex, and calculate the cosine value of the target vertex using the cosine theorem according to the coordinates of the target vertex; When the cosine value of the target vertex meets a set threshold, the angle of the target vertex is determined to be a target angle.

4. The method according to claim 1, characterized in that: The image to be processed is equal to the image size of the reference image, the image includes a first length in a first coordinate direction and a second length in a second coordinate direction, and the image includes any one of the image to be processed and the reference image; acquiring the correction point set according to the sizes of the image to be processed and the reference image includes: Determine a first coordinate set in a first coordinate direction within a first length range of the image according to a first set number, wherein the first coordinate set includes a plurality of first coordinates; Determine a second coordinate set in a second coordinate direction within a second length range of the image according to a second set number, wherein the second coordinate set includes a plurality of second coordinates; The plurality of first coordinates and the plurality of second coordinates are arranged and combined respectively to obtain an arranged and combined coordinate point set, and the coordinate point set is used as a correction point set.

5. The method according to any one of claims 1 to 4, characterized in that: The acquiring a corresponding alignment matrix according to the image to be processed and the reference image comprises: Extracting feature points from the image to be processed to obtain a first feature point set, and extracting feature points from the reference image to obtain a second feature point set; Perform feature point matching according to the first feature point set and the second feature point set to determine a plurality of matched feature point pairs, wherein the feature point pairs include corresponding first feature points and second feature points; Construct the corresponding alignment matrix based on multiple feature point pairs.

6. The method according to any one of claims 1 to 4, characterized in that: The target alignment matrix is ​​a homography matrix; the step of using the target alignment matrix to align the image to be processed to obtain the aligned target image includes: According to the homography matrix, a homography transformation method is used to calculate the pixel original coordinates of the image to be processed and transform them into the target coordinates of the reference image; According to the transformed target coordinates of each pixel of the image to be processed, a target image after the image to be processed is aligned with the reference image is obtained.

7. A video processing method, characterized in that: The method comprises: Acquire a video to be processed, and determine at least one adjacent frame to be stabilized from the video to be processed, wherein the adjacent frame includes a reference image located in a previous frame and an image to be processed located in a subsequent frame; Processing the image to be processed in the adjacent frames using the image processing method according to any one of claims 1 to 6 to obtain a target image of the image to be processed in each of the adjacent frames; The target image is replaced with a corresponding frame in the video to be processed to obtain a processed target video.

8. The method according to claim 7, characterized in that The step of determining at least one adjacent frame to be stabilized from the video to be processed comprises: Performing frame processing on the video to be processed to obtain each frame image of the video to be processed; Perform feature extraction on each frame of image to obtain image features of each frame of image; At least one adjacent frame to be stabilized is determined according to the image features of each frame of image.

9. The method according to claim 8, characterized in that The step of determining at least one adjacent frame to be stabilized according to the image features of each frame of image comprises: For any two adjacent frames of images, obtaining a feature matching degree between an image feature of a later frame of image and an image feature of a previous frame of image; When the feature matching degree is less than a set value, the two adjacent frames of images are determined to be adjacent frames to be stabilized.

10. An image processing device, characterized in that: The device comprises: A matrix acquisition module is configured to acquire an image to be processed and a reference image, and acquire a corresponding alignment matrix according to the image to be processed and the reference image; A coordinate acquisition module is configured to acquire, according to the alignment matrix, target vertices of the image to be processed after transformation to the reference image; A correction point set acquisition module is configured to acquire a correction point set according to the sizes of the image to be processed and the reference image when there is a set of relative target vertices with an angle of a preset target angle, wherein the target angle is an angle range used to determine whether the image to be processed after the transformation has a distortion deformation; An alignment processing module is configured to construct a target alignment matrix according to the correction point set, and use the target alignment matrix to align the image to be processed to obtain a target image after alignment processing; The image to be processed is equal to the image size of the reference image, the image includes a first length in a first coordinate direction, and a second length in a second coordinate direction, the first coordinate direction and the second coordinate direction are perpendicular to each other, and the image includes any one of the image to be processed and the reference image; the correction point set acquisition module is further configured to execute: within the first length range of the image, determine a first coordinate set in the first coordinate direction according to a first set length interval, the first coordinate set includes a plurality of first coordinates; within the second length range of the image, determine a second coordinate set in the second coordinate direction according to a second set length interval, the second coordinate set includes a plurality of second coordinates; arrange and combine the plurality of first coordinates and the plurality of second coordinates respectively to obtain a coordinate point set after arrangement and combination, and use the coordinate point set as the correction point set; The alignment processing module is further configured to execute: performing feature matching processing on the image to be processed and the reference image according to the correction point set to obtain a feature matching result, and constructing a target alignment matrix according to the feature matching result.

11. The device according to claim 10, characterized in that The device further comprises an angle acquisition module configured to execute: Obtaining the coordinates of the target vertex; According to the coordinates of the target vertex, a trigonometric function is used to calculate the angle corresponding to the target vertex.

12. The device according to claim 10, characterized in that The apparatus further comprises an angle determination module configured to execute: Obtaining the coordinates of the target vertex, and calculating the cosine value of the target vertex using the cosine theorem according to the coordinates of the target vertex; When the cosine value of the target vertex meets a set threshold, the angle of the target vertex is determined to be a target angle.

13. The device according to claim 10, characterized in that The image to be processed is equal to the image size of the reference image, the image includes a first length in a first coordinate direction and a second length in a second coordinate direction, and the image includes any one of the image to be processed and the reference image; the correction point set acquisition module is further configured to execute: Determine a first coordinate set in a first coordinate direction within a first length range of the image according to a first set number, wherein the first coordinate set includes a plurality of first coordinates; Determine a second coordinate set in a second coordinate direction within a second length range of the image according to a second set number, wherein the second coordinate set includes a plurality of second coordinates; The plurality of first coordinates and the plurality of second coordinates are arranged and combined respectively to obtain an arranged and combined coordinate point set, and the coordinate point set is used as a correction point set.

14. The device according to any one of claims 10 to 13, characterized in that The matrix acquisition module is further configured to execute: Extracting feature points from the image to be processed to obtain a first feature point set, and extracting feature points from the reference image to obtain a second feature point set; Perform feature point matching according to the first feature point set and the second feature point set to determine a plurality of matched feature point pairs, wherein the feature point pairs include corresponding first feature points and second feature points; Construct the corresponding alignment matrix based on multiple feature point pairs.

15. The device according to any one of claims 10 to 13, characterized in that The target alignment matrix is ​​a homography matrix; the alignment processing module is further configured to perform: According to the homography matrix, a homography transformation method is used to calculate the pixel original coordinates of the image to be processed and transform them into the target coordinates of the reference image; According to the transformed target coordinates of each pixel of the image to be processed, a target image after the image to be processed is aligned with the reference image is obtained.

16. A video processing device, characterized in that: The device comprises: A to-be-processed video acquisition module is configured to acquire the to-be-processed video, and determine at least one adjacent frame to be stabilized from the to-be-processed video, wherein the adjacent frame includes a reference image located in a previous frame and an image to be processed located in a subsequent frame; The image processing device according to any one of claims 10 to 15, configured to process the images to be processed in the adjacent frames to obtain a target image of the images to be processed in each of the adjacent frames; The target video acquisition module is configured to replace the target image with a corresponding frame in the video to be processed to obtain a processed target video.

17. The device according to claim 16, characterized in that The to-be-processed video acquisition module further includes: A frame division unit is configured to perform frame division processing on the video to be processed to obtain each frame image of the video to be processed; The feature extraction unit is configured to perform feature extraction on each frame of image to obtain image features of each frame of image; The adjacent frame determination unit is configured to determine at least one adjacent frame to be stabilized according to the image features of each frame of image.

18. The device according to claim 17, characterized in that The adjacent frame determination unit is configured to perform: For any two adjacent frames of images, obtaining a feature matching degree between an image feature of a later frame of image and an image feature of a previous frame of image; When the feature matching degree is less than a set value, the two adjacent frames of images are determined to be adjacent frames to be stabilized.

19. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6 or the video processing method according to any one of claims 7 to 9.

20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in any one of claims 1 to 6 or implement the video processing method as described in any one of claims 7 to 9.

21. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method according to any one of claims 1 to 6 or implement the video processing method according to any one of claims 7 to 9.

Images