An image processing method, system and action camera

Abstract

The present application discloses an image processing method, system and action camera. Obtain a current image frame and Euler angles corresponding to the current image frame, and obtain an initial constraint range, where the initial constraint range is used to detect whether there are black edges in the image frame; if the Euler angles exceed the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; if the image frame detection result indicates that there are black edges in the current image frame, perform adjustment processing on the initial constraint range according to the Euler angles to obtain a constraint limit range, where there are no black edges in the image frame within the constraint limit range; perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame. In this way, in the present application, while realizing motion compensation for the image frames in the video, it can be ensured that there are no black edges in the finally output image frames, thereby improving the image processing effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and particularly to an image processing method, system, and action camera. Background Art

[0002] With the rapid development of information technology, action cameras have become common tools for people to record daily life. However, during the shooting process, due to the action camera being vulnerable to human or environmental influences, the captured video shakes, thus affecting the viewing experience. To solve this problem, video anti-shake technologies have been widely applied, including mechanical image stabilization technology, optical image stabilization technology, and electronic image stabilization technology.

[0003] Since the electronic image stabilization technology has advantages such as low cost and strong flexibility, it is widely used in camera technologies. However, in related technologies, after using the electronic image stabilization technology to perform motion compensation on the captured video, there are often black edges in the images of the motion-compensated video. These black edges in the images not only affect the visual perception of the video but also reduce the effective visible area.

[0004] Furthermore, in related technologies, the methods of cropping through a fixed rectangular frame and fusing image feature points are usually adopted to remove the black edges of the images. However, when the video shakes violently, the fixed rectangular frame cannot ensure that all croppings contain valid pixels, and if the image feature points are not fused properly, obvious fusion traces are likely to appear at the image edges, resulting in poor image processing effects.

[0005] Therefore, how to improve the image processing effect is a key issue that those skilled in the art focus on. Summary of the Invention

[0006] Based on the above problems, the present application provides an image processing method, system, and action camera to improve the image processing effect. The embodiments of the present application disclose the following technical solutions:

[0007] In a first aspect, the present application discloses an image processing method, including:

[0008] Obtaining a current image frame and the Euler angles corresponding to the current image frame, and obtaining an initial constraint range, where the initial constraint range is used to detect whether there are black edges in the image frame;

[0009] If the Euler angles exceed the initial constraint range, performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result;

[0010] If the image frame detection result indicates that there is a black edge in the current image frame, adjust the initial constraint range according to the Euler angles to obtain a constraint limit range, where there is no black edge in the image frame within the constraint limit range;

[0011] Perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

[0012] In an implementable embodiment, the Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle, where the first Euler angle includes an angle on the horizontal axis, the second Euler angle includes an angle on the vertical axis, and the third Euler angle includes an angle on the vertical axis;

[0013] Before adjusting the initial constraint range according to the Euler angles to obtain a constraint limit range, it further includes:

[0014] Obtain an expansion factor, a contraction factor, a minimum angle threshold, and a maximum angle threshold;

[0015] Determine whether the target Euler angle on the target axis exceeds the boundary of the initial constraint range, where the target Euler angle is any one of the first Euler angle, the second Euler angle, and the third Euler angle, and the target axis is determined based on the target Euler angle.

[0016] In an implementable embodiment, adjusting the initial constraint range according to the Euler angles to obtain a constraint limit range includes:

[0017] If the target Euler angle on the target axis exceeds the boundary of the initial constraint range, perform calculation processing on the target Euler angle according to the expansion factor to obtain a first calculated angle;

[0018] Perform calculation processing on the Euler angles other than the target Euler angle among the first Euler angle, the second Euler angle, and the third Euler angle according to the contraction factor to obtain a second calculated angle and a third calculated angle;

[0019] Adjust the initial constraint range according to the first calculated angle, the second calculated angle, and the third calculated angle to obtain a constraint limit range.

[0020] In an implementable embodiment, adjusting the initial constraint range according to the first calculated angle, the second calculated angle, and the third calculated angle to obtain a constraint limit range includes:

[0021] If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle;

[0022] Adjust the range of the initial constraint range according to the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle to obtain a constraint limit range.

[0023] In an implementable embodiment, the detecting and processing the current image frame according to the initial constraint range to obtain an image frame detection result includes:

[0024] Perform pre-correction processing on the current image frame to obtain a pre-corrected image frame;

[0025] Detect and process the pre-corrected image frame according to the initial constraint range to obtain an image frame detection result.

[0026] In an implementable embodiment, before the detecting and processing the pre-corrected image frame according to the initial constraint range to obtain an image frame detection result, it further includes:

[0027] Perform point sampling on each edge of the pre-corrected image frame to obtain n image sampling points corresponding to each edge of the pre-corrected image frame, where n is greater than or equal to 1;

[0028] The detecting and processing the pre-corrected image frame according to the initial constraint range to obtain an image frame detection result includes:

[0029] For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapped sampling point corresponding to the image sampling point;

[0030] For multiple image sampling points, perform sampling point detection on the mapped sampling points corresponding to the multiple image sampling points according to the initial constraint range to obtain an image frame detection result.

[0031] In an implementable embodiment, the obtaining the current image frame and the Euler angle corresponding to the current image frame includes:

[0032] Obtain the current image frame in real time based on a camera, and obtain the motion data corresponding to the current image frame in real time based on an IMU instrument;

[0033] Perform computational processing on the operating data to obtain the initial Euler angles corresponding to the current image frame;

[0034] Perform smoothing processing on the initial Euler angles corresponding to the current image frame to obtain the Euler angles corresponding to the current image frame.

[0035] In an implementable embodiment, it further includes:

[0036] Obtain the stabilized image frames corresponding to multiple image frames respectively, where the current image frame is any one of the multiple image frames;

[0037] Generate a target video according to the stabilized image frames corresponding to the multiple image frames respectively, where the target video is used to show the target object.

[0038] In a second aspect, the present application discloses an image processing system, including:

[0039] An image frame acquisition unit, configured to acquire a current image frame, the Euler angles corresponding to the current image frame, and an initial constraint range, where the initial constraint range is used to detect whether there is a black edge in the image frame;

[0040] An image frame detection unit, configured to, if the Euler angles exceed the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result;

[0041] A constraint range adjustment unit, configured to, if the image frame detection result indicates that there is a black edge in the current image frame, perform adjustment processing on the initial constraint range according to the Euler angles to obtain a constraint limit range, where there is no black edge in the image frame within the constraint limit range;

[0042] An image frame stabilization unit, configured to perform stabilization processing on the current image frame according to the constraint limit range to obtain the stabilized image frame corresponding to the current image frame.

[0043] In an implementable embodiment, the system further includes:

[0044] A factor threshold acquisition unit, configured to acquire an expansion factor, a contraction factor, a minimum angle threshold, and a maximum angle threshold;

[0045] An Euler angle judgment unit, configured to judge whether the target Euler angle exceeds the boundary of the initial constraint range on the target axis, where the target Euler angle is any one of the first Euler angle, the second Euler angle, and the third Euler angle, and the target axis is an axis determined based on the target Euler angle.

[0046] In an implementable embodiment, the constraint range adjustment unit includes:

[0047] A first angle calculation unit, configured to, if the target Euler angle on the target axis exceeds the boundary of the initial constraint range, perform calculation processing on the target Euler angle according to the expansion factor to obtain a first calculated angle;

[0048] A second angle calculation unit, configured to perform calculation processing on the Euler angles other than the target Euler angle among the first Euler angle, the second Euler angle, and the third Euler angle according to the contraction factor to obtain a second calculated angle and a third calculated angle;

[0049] A restricted range obtaining unit, configured to perform adjustment processing on the initial constraint range according to the first calculated angle, the second calculated angle, and the third calculated angle to obtain a constraint restricted range.

[0050] In an implementable embodiment, the restricted range obtaining unit is specifically configured to:

[0051] If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle;

[0052] Perform range adjustment on the initial constraint range according to the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle to obtain a constraint restricted range.

[0053] In an implementable embodiment, the image frame detection unit includes:

[0054] A pre-correction processing unit, configured to perform pre-correction processing on the current image frame to obtain a pre-corrected image frame;

[0055] A corrected frame detection unit, configured to perform detection processing on the pre-corrected image frame according to the initial constraint range to obtain an image frame detection result.

[0056] In an implementable embodiment, the system further includes:

[0057] The correction frame sampling point unit is used to perform sampling point processing on each edge of the pre-corrected image frame, and obtain n image sampling points corresponding to each edge of the pre-corrected image frame, where n is greater than or equal to 1;

[0058] The correction frame detection unit is specifically configured to:

[0059] For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapped sampling point corresponding to the image sampling point;

[0060] For multiple image sampling points, perform sampling point detection on the mapped sampling points corresponding to the multiple image sampling points according to the initial constraint range to obtain an image frame detection result.

[0061] In an implementable embodiment, the image frame acquisition unit is specifically configured to:

[0062] Obtain the current image frame in real time based on a camera, and obtain the motion data corresponding to the current image frame in real time based on an IMU instrument;

[0063] Perform calculation processing on the operation data to obtain the initial Euler angle corresponding to the current image frame;

[0064] Perform smoothing processing on the initial Euler angle corresponding to the current image frame to obtain the Euler angle corresponding to the current image frame.

[0065] In an implementable embodiment, the system further includes:

[0066] The image stabilization image frame acquisition unit is used to acquire the image stabilization image frames corresponding to multiple image frames, where the current image frame is any one of the multiple image frames;

[0067] The target video generation unit is used to generate a target video according to the image stabilization image frames corresponding to the multiple image frames, where the target video is used to show a target object.

[0068] In a third aspect, an embodiment of the present application provides an action camera, which is characterized by including:

[0069] An image processing system, where the image processing system is configured to execute the image processing method according to the first aspect.

[0070] Compared with the prior art, the present application has the following beneficial effects:

[0071] In the technical solution of this application, first, the current image frame and the Euler angles corresponding to the current image frame are obtained, and the initial constraint range is obtained. After that, if the Euler angles exceed the initial constraint range, the current image frame is detected and processed according to the initial constraint range to obtain an image frame detection result. And if the image frame detection result indicates that there are black edges in the current image frame, the initial constraint range is adjusted according to the Euler angles to obtain a constraint limit range. Finally, the current image frame is stabilized according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame. It should be noted that the initial constraint range is used to detect whether there are black edges in the image frame, and there are no black edges in the image frame within the constraint limit range.

[0072] It can be seen that in this application, under the action of the initial constraint range, it can be determined whether the Euler angles corresponding to the current image frame exceed the initial constraint range and whether there are black edges in the current image frame. If the Euler angles corresponding to the current image frame exceed the initial constraint range and there are black edges in the current image frame, then the initial constraint range can be adjusted based on the Euler angles to obtain a constraint limit range, and there are no black edges in the stabilized image frame obtained within the constraint limit range. In this way, in this application, while realizing the motion compensation for the image frames in the video, it can be ensured that there are no black edges in the finally output image frames, thereby improving the image processing effect. Description of the Drawings

[0073] In order to more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0074] Figure 1 It is a flowchart of an image processing method provided by an embodiment of this application;

[0075] Figure 2 It is a schematic diagram of black edges in an image processing method provided by an embodiment of this application;

[0076] Figure 3 It is a schematic diagram of image frame correction in an image processing method provided by an embodiment of this application;

[0077] Figure 4 It is a schematic diagram of the boundary of the initial constraint range in an image processing method provided by an embodiment of this application;

[0078] Figure 5 It is a full flowchart of image processing in an image processing method provided by an embodiment of this application;

[0079] Figure 6 The structural schematic diagram of an image processing system provided by an embodiment of the present application. Detailed implementation manners

[0080] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

[0081] It should be noted that an image processing method, system and action camera provided by the present application are used in the field of image processing technology. The above is only an example, and does not limit the application fields of the method and system names provided by the present application.

[0082] As described above, with the rapid development of information technology, action cameras have become common tools for people to record their daily lives. However, during the shooting process, due to the action camera being easily affected by humans or the environment, the captured video shakes, thus affecting the viewing experience. To solve this problem, video anti-shake technology has been widely used, and this video anti-shake technology includes mechanical image stabilization technology, optical image stabilization technology and electronic image stabilization technology.

[0083] Since the electronic image stabilization technology has advantages such as low cost and strong flexibility, this electronic image stabilization technology is widely used in camera technology. However, in the related technology, after using this electronic image stabilization technology to perform motion compensation on the captured video, there are often black edges in the images in the motion-compensated video. These black edges in the images not only affect the visual perception of the video, but also reduce the effective visible area.

[0084] Furthermore, in the related art, the method of removing the black edges of an image is usually achieved by combining the method of cropping with a fixed rectangular frame and the method of fusing image feature points. The method of cropping with a fixed rectangular frame usually involves setting a rectangular frame of a specific size and cropping the image based on this rectangular frame. However, when the video shakes violently, the fixed rectangular frame cannot ensure that all croppings contain valid pixels. The method of fusing image feature points usually involves registering the reference frame (the image after motion compensation) and the current frame (the image before motion compensation), that is, finding the feature points between the reference frame and the current frame and fusing the reference frame and the current frame through the feature points. However, if the image feature points are not properly fused, obvious fusion traces are likely to appear at the image edges, and the computational overhead brought by this is relatively high. Thus, both the method of cropping with a fixed rectangular frame and the method of fusing image feature points proposed in the related art will result in poor image processing effects, and both of these methods are offline tasks. Therefore, how to improve the image processing effect is a key issue that those skilled in the art focus on.

[0085] In another feasible implementation, for the image processing of online tasks in the related art, it is also proposed that the information of past frames can be used to perform multi-frame fusion and filtering on the current frame to expand the effective pixel area in the image and reduce the influence of black edges. However, multi-frame fusion will cause obvious fusion traces to easily appear at the image edges, affecting the visual quality, and the computational overhead of multi-frame fusion and filtering is relatively large, significantly increasing the processing delay and hardware risk, thereby resulting in poor image processing efficiency.

[0086] Therefore, the inventor proposes the technical solution of this application, aiming to improve the image processing effect. In the embodiment of this application, first, the current image frame and the Euler angle corresponding to the current image frame are obtained, and the initial constraint range is obtained. After that, if the Euler angle exceeds the initial constraint range, the current image frame is detected and processed according to the initial constraint range to obtain the image frame detection result. And if the image frame detection result indicates that there are black edges in the current image frame, the initial constraint range is adjusted according to the Euler angle to obtain the constraint limit range. Finally, the current image frame is stabilized according to the constraint limit range to obtain the stabilized image frame corresponding to the current image frame. It should be noted that the initial constraint range is used to detect whether there are black edges in the image frame, and there are no black edges in the image frame within the constraint limit range.

[0087] It can be seen that, in the present application, under the action of the initial constraint range, it is possible to determine whether the Euler angles corresponding to the current image frame exceed the initial constraint range and whether there are black edges in the current image frame. If the Euler angles corresponding to the current image frame exceed the initial constraint range and there are black edges in the current image frame, then the initial constraint range can be adjusted based on the Euler angles to obtain a constraint limit range, and there are no black edges in the stabilized image frames obtained within this constraint limit range. In this way, in the present application, while realizing real-time motion compensation for the image frames in the video, it is possible to ensure that there are no black edges in the finally output image frames, thereby improving the image processing effect, and further improving the image processing efficiency, so as to better meet the real-time processing requirements of the video in complex dynamic scenarios.

[0088] In order to enable those skilled in the art to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0089] The following describes, through an embodiment, an image processing method provided by the present application.

[0090] See Figure 1 , which is a flowchart of an image processing method provided by an embodiment of the present application. As Figure 1 shown, the method includes:

[0091] S101: Obtain the current image frame and the Euler angles corresponding to the current image frame, and obtain the initial constraint range.

[0092] It should be noted that, in this step, first, the current image frame can be obtained in real time based on a camera. The current image frame can be any frame in the video captured in real time by the object, and the motion data corresponding to the current image frame can be obtained in real time based on an IMU (Inertial Measurement Unit) instrument. The IMU instrument includes an accelerometer and a gyroscope, and the motion data can be used to calculate the attitude change of the motion.

[0093] After that, the running data can be calculated and processed to obtain the initial Euler angles corresponding to the current image frame. Specifically, in this step, first, a filtering technique can be used to reduce the influence of the noise generated by the IMU instrument on the motion data, so as to improve the accuracy of the motion attitude estimation. After that, the angular velocity data of the gyroscope in the IMU instrument can be used to calculate the initial Euler angles corresponding to the current image frame by means of numerical integration. And during the calculation of the initial Euler angles, it is also necessary to align the timestamps of the camera and the IMU data. It should be noted that the method of obtaining Euler angles mentioned in the present application is similar to the existing solutions, and will not be specifically described here.

[0094] In this step, in order to make the initial Euler angles corresponding to the current image frame obtained by calculating the motion data smoother and more stable, the initial Euler angles corresponding to the current image frame are smoothed by the Kalman filtering algorithm to obtain the Euler angles corresponding to the current image frame. It should be noted that the Kalman filtering algorithm mentioned in this application is an existing algorithm and will not be elaborated here.

[0095] Further, in this step, an initial constraint range can also be obtained, which is used to detect whether there are black edges in the image frame. As Figure 2 shown, Figure 2 is a schematic diagram of black edges in an image processing method provided by an embodiment of this application. The left black area in Figure 2 can be understood as the black edges existing in the image.

[0096] Next, the process of obtaining the initial constraint range will be introduced. In this application, first, random Euler angles can be obtained. These random Euler angles are obtained randomly and then screened to obtain the screened random Euler angles. The screened random Euler angles are the Euler angles that will not result in black edges after mapping the image at a specified field of view (such as the horizontal axis, vertical axis, etc.), and a point set is formed based on the screened random Euler angles, where i represents the serial number of the point, and y, p, r represent the three-dimensional coordinates corresponding to the screened random Euler angles. It should be noted that for the screening process of random Euler angles, the black edge detection algorithm mentioned later in this application can be used. For the introduction of the black edge detection algorithm, see the subsequent description.

[0097] After that, convex hull fitting can be performed on the screened random Euler angles, that is, the points with the smallest and the largest coordinates are determined from the point set, and the initial edge of the initial constraint range is determined based on the point with the smallest coordinate and the point with the largest coordinate. The initial constraint range can be understood as an initial rectangular box. Then, the point farthest from this initial edge is used as a new convex hull vertex to divide the point set into different subsets, and the same operation is recursively performed on each subset until all points are included in the convex hull, and finally a convex hull is formed.

[0098] Finally, for the ellipsoid equation (where x, y, z represent the horizontal axis, vertical axis, and vertical axis respectively, and a represents the coefficient), the points on the convex hull are used to construct a least-squares equation, and the minimum circumscribed ellipsoid of the convex hull is obtained based on this least-squares equation. And the maximum inscribed cuboid is obtained according to the lengths of the three axes of the ellipsoid. The maximum inscribed cuboid is the range that will not result in black edges after appropriate adjustment, that is, the initial constraint range. In this way, in this application, by using Euler angles without black edges and solving the initial range of Euler angle constraints through Euler angles, the dynamic initialization of the jitter constraint range is achieved.

[0099] S102: If the Euler angles exceed the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result.

[0100] It should be noted that before the present application executes step S102, it is also necessary to determine whether the current image frame is within the initial constraint range. If the current image frame is within the initial constraint range or the current image frame is not within the initial constraint range, determine whether the Euler angles exceed the initial constraint range. If the Euler angles do not exceed the initial constraint range, perform image stabilization processing on the current image frame; if the Euler angles exceed the initial constraint range, pre-correction processing can be performed on the current image frame to obtain an image pre-correction frame, and detection processing is performed on the image pre-correction frame according to the initial constraint range to obtain an image frame detection result, where the pre-correction processing can be image de-distortion, and the pre-correction processing can be image transformation.

[0101] It should also be noted that in the present application, a black edge detection algorithm can be used to detect whether there are black edges in the image pre-correction frame. The black edge detection algorithm is as follows: Before the present application executes the operation of performing detection processing on the image pre-correction frame according to the initial constraint range to obtain an image frame detection result, point sampling can also be performed on each edge of the image pre-correction frame to obtain n image sampling points corresponding to each edge of the image pre-correction frame, where n is greater than or equal to 1, and the points sampled on each edge can be the same, and the points sampled on each edge can also be different. For example, one edge of the image pre-correction frame can be 12 points, and another edge of the image pre-correction frame can be 8 points.

[0102] Specifically, for each image sampling point, inverse perspective mapping (inverse perspective projection) can be performed on the image sampling point to obtain a mapped sampling point corresponding to the image sampling point, and for multiple image sampling points, sampling point detection is performed on the mapped sampling points corresponding to the multiple image sampling points according to the initial constraint range to obtain an image frame detection result, where the image frame detection result includes a result that there are black edges in the current image frame or a result that there are no black edges in the current image frame. In this way, in the present application, by sampling points in the image pre-correction frame, it is possible to more quickly determine whether there are black edges in the initial constraint range.

[0103] As Figure 3 shown, Figure 3 is a schematic diagram of image frame correction in an image processing method provided by an embodiment of the present application. Specifically, Figure 3 in (a) is the current image frame, Figure 3 in (b) is the image pre-correction frame. Further, Figure 3 the black rectangular frame in (b) can be understood as the initial constraint range, Figure 3The black area in (b) can be understood as a black edge. Figure 3 The points on the region boundary of the grid area in (b) can be understood as image sampling points.

[0104] In an implementable manner, in this application, the black edge detection algorithm can be introduced by using steps (1) - (7). Steps (1) - (7) are specifically as follows:

[0105] Step (1): Uniformly take points on the four sides of the pre - corrected image frame , where i represents the serial number of the point and n represents the total number of points;

[0106] Step (2): For any image sampling point in the sampling point set P , convert it to the normalized plane: , and obtain the pixel point of this image sampling point on the normalized plane , where represents the position of the original pixel point in the pre - corrected image frame, is the new principal point of the pre - corrected image frame, respectively represent the new focal lengths on the x - axis and y - axis of the pre - corrected image frame, can represent the coordinate position in the horizontal dimension, can represent the coordinate position in the vertical dimension;

[0107] Step (3): Perform an inverse rotation transformation on the pixel point of this image sampling point on the normalized plane : , where W represents the dimension in the inverse perspective projection, R represents the rotation matrix, T represents the transpose operation, X and Y respectively represent the transformed abscissa and ordinate, and x and y respectively represent the original abscissa and original ordinate;

[0108] Step (4): Normalize the pixel point after the inverse rotation transformation: , , where represents the new pixel point of this image sampling point on the normalized plane;

[0109] Step (5): Perform a de - distortion operation on the new pixel point: , , where is the distortion coefficient, and the distance ;

[0110] Step (6): Map the new pixel point after de - distortion back to the current image frame to obtain the mapped sampling point : , . Where is the principal point of the current image frame, respectively represent the focal lengths on the x-axis and y-axis of the current image frame;

[0111] Step (7): Perform the operations of Step (2) - Step (6) on each image sampling point to determine whether each point satisfies in the current image frame. Meeting the requirements indicates that there is no black edge in the visible range of the pre-corrected image frame, where represents the width of the image frame, represents the height of the image frame.

[0112] It can be understood that if the judgment requirements are met, no black edge will appear when cropping the pre-corrected image frame using the initial constraint range. In this way, in this application, only simple matrix operations are performed on a few points to achieve the inverse transformation, enabling real-time black edge detection simply and efficiently, reducing the computational complexity and resource consumption, and the black edge detection algorithm can achieve efficient detection in practical applications (100 pictures of 720p (row pixels), 1080p, and 2160p are respectively taken for testing, and the times used are 17ms (milliseconds), 19ms, and 25ms respectively, that is, the detection time for each picture is about 0.2ms, and as the image resolution increases, the increase in the image detection time is not significant).

[0113] S103: If the image frame detection result indicates that there is a black edge in the current image frame, adjust the initial constraint range according to the Euler angles to obtain a constraint limit range.

[0114] In this step, the Euler angles include the first Euler angle, the second Euler angle, and the third Euler angle. The first Euler angle includes the angle on the horizontal axis, the second Euler angle includes the angle on the vertical axis, and the third Euler angle includes the angle on the vertical axis. It should be noted that there can be multiple Euler angles in an image frame, and the Euler angles shown in this application are only partial examples. And if the image frame detection result indicates that there is no black edge in the current image frame, the current image frame can be subjected to image stabilization processing to obtain a stabilized image frame.

[0115] In an implementable embodiment, before the present application performs the operation of adjusting the initial constraint range according to Euler angles to obtain the constraint limit range, it may also obtain an expansion factor, a contraction factor, a minimum angle threshold, and a maximum angle threshold. The expansion factor, the contraction factor, the minimum angle threshold, and the maximum angle threshold can be used to implement the adjustment of the initial constraint range. Among them, the expansion factor and the contraction factor are empirical values, which vary in different scenarios. The minimum angle threshold and the maximum angle threshold are preset values, and the specific settings of the expansion factor, the contraction factor, the minimum angle threshold, and the maximum angle threshold can also be adjusted according to the actual situation in practical applications. And it can be determined whether the target Euler angle on the target axis exceeds the boundary of the initial constraint range, where the target Euler angle is any one of the first Euler angle, the second Euler angle, and the third Euler angle, and the target axis is the axis determined based on the target Euler angle. For example, if the target Euler angle is the angle on a certain axis, then the target axis is that certain axis.

[0116] It can be understood that in the present application, it can be determined whether the first Euler angle exceeds the boundary of the initial constraint range on the horizontal axis, or whether the second Euler angle exceeds the boundary of the initial constraint range on the vertical axis, or whether the third Euler angle exceeds the boundary of the initial constraint range on the vertical axis. In this way, when the video jitters greatly, the axis where black edges may exist can be determined, so as to accurately adjust the initial constraint range.

[0117] As Figure 4 shown, Figure 4 It is a schematic diagram of the boundary of the initial constraint range in an image processing method provided by an embodiment of the present application. Figure 4 In (a), it means that the first Euler angle exceeds the boundary of the initial constraint range on the horizontal axis, Figure 4 In (b), it means that the second Euler angle approaches the boundary of the initial constraint range on the vertical axis, Figure 4 In (c), it means that the third Euler angle exceeds the boundary of the initial constraint range on the vertical axis.

[0118] Specifically, in the present application, if the target Euler angles on the target axis exceed the boundaries of the initial constraint range, the first Euler angle, the second Euler angle, and the third Euler angle can be calculated and processed according to the expansion factor and the contraction factor to obtain the first calculated angle, the second calculated angle, and the third calculated angle. Thereafter, the first calculated angle, the second calculated angle, and the third calculated angle can be calculated and processed according to the minimum angle threshold and the maximum angle threshold to obtain the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle. Finally, the initial constraint range can be adjusted according to the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle to obtain the constraint limit range.

[0119] Next, the process of adjusting the range of the initial constraint range to obtain the constraint limit range in the present application will be further introduced. If the target Euler angle on the target axis exceeds the boundary of the initial constraint range, the target Euler angle can be calculated and processed according to the expansion factor to obtain the first calculated angle. The process of obtaining the first calculated angle can be reflected by formula (1), and formula (1) is specifically as follows:

[0120] Formula (1)

[0121] Where, represents the first calculated angle, represents the target Euler angle, X represents the expansion factor, and the expansion factor can be 0.05. In this way, when the Euler angle of a certain axis approaches or exceeds the boundary of the initial constraint range, dynamic expansion can be performed based on the expansion factor.

[0122] Thereafter, the Euler angles other than the target Euler angle among the first Euler angle, the second Euler angle, and the third Euler angle can be calculated and processed separately according to the contraction factor to obtain the second calculated angle and the third calculated angle. The process of obtaining the second calculated angle / the third calculated angle can be reflected by formula (2), and formula (2) is specifically as follows:

[0123] Formula (2)

[0124] Where, represents the second calculated angle / the third calculated angle, represents the target Euler angle, X represents the contraction factor, and the contraction factor can be 0.05. In this way, when the Euler angle of a certain axis does not approach or exceed the boundary of the initial constraint range, appropriate dynamic contraction can be performed on it based on the contraction factor.

[0125] Finally, the initial constraint range can be adjusted according to the first calculated angle, the second calculated angle, and the third calculated angle to obtain a constraint limit range, where there is no black edge in the image frame within the constraint limit range. Specifically, if the first calculated angle is between the minimum angle threshold and the maximum angle threshold , and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, then the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle can be determined according to the first calculated angle, the second calculated angle, and the third calculated angle.

[0126] If any one of the first calculated angle, the second calculated angle, and the third calculated angle does not meet the requirements, the steps of expansion and contraction need to be repeated until each calculated angle meets the above requirements. Thereafter, the initial constraint range can be adjusted according to the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle to obtain a constraint limit range. In this way, the dynamic adjustment of the initial constraint range is realized in the present application.

[0127] S104: Perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

[0128] It can be understood that at this time, the adjusted Euler angles are constrained within the new constraint limit range, and the adjusted Euler angles can be applied to the current image frame through inverse geometric transformation for image stabilization processing, so that the stabilized image frame can contain valid pixel points, ensuring no distortion and no black edge in the process of stabilizing the corrected image.

[0129] After that, the present application can also obtain the stabilized image frames corresponding to multiple image frames respectively, and generate a target video according to the stabilized image frames corresponding to multiple image frames respectively, where the current image frame is any one of the multiple image frames, and the target video is used to show the target object (such as the shooting person, etc.). In this way, through the technical solution of the present application, the processing of images can be realized in real time and efficiently to output high-quality videos, improving the user experience.

[0130] As Figure 5 shown, Figure 5 is the full flow chart of image processing in an image processing method provided by an embodiment of the present application. In Figure 5First, the current image frame and the Euler angles corresponding to the current image frame can be obtained. Thereafter, under the action of the initial constraint range, it is determined whether the Euler angles corresponding to the current image frame exceed the initial constraint range. If the Euler angles corresponding to the current image frame exceed the initial constraint range, it is determined that black edge detection is performed on the current image frame to obtain an image frame detection result; if the Euler angles corresponding to the current image frame do not exceed the initial constraint range, the current image frame is subjected to image stabilization processing to obtain a stabilized image frame.

[0131] If the image frame detection result indicates that there are black edges in the current image frame, the initial constraint range can be adjusted based on the Euler angles to obtain a constraint limit range, so as to implement subsequent image stabilization processing based on this constraint limit range; if the image frame detection result indicates that there are no black edges in the current image frame, the current image frame is subjected to image stabilization processing to obtain a stabilized image frame. In this way, while the present application realizes real-time motion compensation for image frames in a video, it can ensure that the finally output image frames do not have black edges, thereby improving the image processing effect.

[0132] In summary, in the technical solution of the present application, under the action of the initial constraint range, it can be determined whether the Euler angles corresponding to the current image frame exceed the initial constraint range and whether there are black edges in the current image frame. If the Euler angles corresponding to the current image frame exceed the initial constraint range and there are black edges in the current image frame, the initial constraint range can be adjusted based on the Euler angles to obtain a constraint limit range, and the stabilized image frame obtained within this constraint limit range does not have black edges. In this way, in the present application, while realizing real-time motion compensation for image frames in a video, it can ensure that the finally output image frames do not have black edges, thereby improving the image processing effect, and further improving the image processing efficiency to better meet the real-time processing requirements of videos in complex dynamic scenarios.

[0133] Next, an image processing system provided by an embodiment of the present application will be introduced. The image processing system described below can be mutually corresponding and referred to with the image processing method described above.

[0134] See Figure 6 , this figure is a schematic structural diagram of an image processing system provided by an embodiment of the present application. As Figure 6 shown, the image processing system includes:

[0135] An image frame acquisition unit 601, configured to acquire a current image frame and the Euler angles corresponding to the current image frame, and acquire an initial constraint range, where the initial constraint range is used to detect whether there are black edges in the image frame;

[0136] An image frame detection unit 602, configured to, if the Euler angles exceed the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result;

[0137] A constraint range adjustment unit 603, configured to, if the image frame detection result indicates that there is a black edge in the current image frame, adjust the initial constraint range according to the Euler angles to obtain a constraint limit range, where there is no black edge in the image frame within the constraint limit range;

[0138] An image frame stabilization unit 604, configured to perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

[0139] In an implementable embodiment, the system further includes:

[0140] A factor threshold acquisition unit, configured to acquire an expansion factor, a contraction factor, a minimum angle threshold, and a maximum angle threshold;

[0141] An Euler angle determination unit, configured to determine whether a target Euler angle on a target axis exceeds a boundary of the initial constraint range, where the target Euler angle is any one of the first Euler angle, the second Euler angle, and the third Euler angle, and the target axis is an axis determined based on the target Euler angle.

[0142] In an implementable embodiment, the constraint range adjustment unit includes:

[0143] A first angle calculation unit, configured to, if the target Euler angle on the target axis exceeds the boundary of the initial constraint range, perform calculation processing on the target Euler angle according to the expansion factor to obtain a first calculated angle;

[0144] A second angle calculation unit, configured to perform calculation processing on the Euler angles other than the target Euler angle among the first Euler angle, the second Euler angle, and the third Euler angle according to the contraction factor to obtain a second calculated angle and a third calculated angle;

[0145] A limit range acquisition unit, configured to perform adjustment processing on the initial constraint range according to the first calculated angle, the second calculated angle, and the third calculated angle to obtain a constraint limit range.

[0146] In an implementable embodiment, the limit range acquisition unit is specifically configured to:

[0147] If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle;

[0148] Adjust the initial constraint range according to the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle to obtain a constraint limit range.

[0149] In an implementable embodiment, the image frame detection unit includes:

[0150] A pre-correction processing unit for performing pre-correction processing on the current image frame to obtain a pre-corrected image frame;

[0151] A corrected frame detection unit for performing detection processing on the pre-corrected image frame according to the initial constraint range to obtain an image frame detection result.

[0152] In an implementable embodiment, the system further includes:

[0153] A corrected frame point sampling unit for performing point sampling on each edge of the pre-corrected image frame to obtain n image sampling points respectively corresponding to each edge of the pre-corrected image frame, where n is greater than or equal to 1;

[0154] The corrected frame detection unit is specifically configured to:

[0155] For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapped sampling point corresponding to the image sampling point;

[0156] For multiple image sampling points, perform sampling point detection on the mapped sampling points respectively corresponding to the multiple image sampling points according to the initial constraint range to obtain an image frame detection result.

[0157] In an implementable embodiment, the image frame acquisition unit is specifically configured to:

[0158] Real-time obtain the current image frame based on a camera, and real-time obtain the motion data corresponding to the current image frame based on an IMU instrument;

[0159] Perform calculation processing on the operation data to obtain the initial Euler angle corresponding to the current image frame;

[0160] Smooth the initial Euler angles corresponding to the current image frame to obtain the Euler angles corresponding to the current image frame.

[0161] In an implementable embodiment, the system further includes:

[0162] A stabilized image frame acquisition unit, configured to acquire stabilized image frames corresponding to multiple image frames respectively, where the current image frame is any one of the multiple image frames;

[0163] A target video generation unit, configured to generate a target video according to the stabilized image frames corresponding to the multiple image frames respectively, where the target video is used to show a target object.

[0164] The image processing system provided in the embodiments of the present application has the same beneficial effects as the image processing method provided in the above embodiments, so details are not described herein again.

[0165] In addition, the present invention also discloses an action camera, including an image processing system, and the image processing system is the image processing system disclosed in the above embodiments, and it can be configured to execute the image processing method according to the above. Therefore, the action camera with this image processing system also has all the above technical effects, which are not described herein again. It should be noted that the action camera can be a GoPro, and the action camera can also be an Insta360.

[0166] It should be noted that the image processing method provided by the present invention can be used in the field of image processing technology. The above is only an example and does not limit the application field of the image processing method provided by the present invention. It should also be noted that the "first", "second" (if any) in the names mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first and second in order.

[0167] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0168] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination thereof. The software modules may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0169] The above has introduced in detail an image processing method, system, and action camera provided by the present application. Specific examples are used herein to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only for helping to understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Obtaining a current image frame and the Euler angles corresponding to the current image frame, and obtaining an initial constraint range, wherein the initial constraint range is used to detect whether there is a black edge in the image frame; If the Euler angle exceeds the initial constraint range, performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; If the image frame detection result indicates that there are black edges in the current image frame, adjusting the initial constraint range according to the Euler angle to obtain a constraint limit range, wherein no black edges exist in the image frame within the constraint limit range; The current image frame is subjected to image stabilization processing according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

2. The method according to claim 1, characterized in that: The Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle, wherein the first Euler angle includes an angle on the horizontal axis, the second Euler angle includes an angle on the vertical axis, and the third Euler angle includes an angle on the vertical axis; Before the initial constraint range is adjusted according to the Euler angle to obtain the constraint limit range, the method further includes: Get the expansion factor, contraction factor, minimum angle threshold and maximum angle threshold; Determine whether a target Euler angle on a target axis exceeds a boundary of the initial constraint range, wherein the target Euler angle is any one of the first Euler angle, the second Euler angle, and the third Euler angle, and the target axis is an axis determined based on the target Euler angle.

3. The method according to claim 2, characterized in that The adjusting process of the initial constraint range according to the Euler angle to obtain the constraint limit range includes: If the target Euler angle on the target axis exceeds the boundary of the initial constraint range, calculating and processing the target Euler angle according to the expansion factor to obtain a first calculated angle; According to the contraction factor, the Euler angles among the first Euler angle, the second Euler angle and the third Euler angle except the target Euler angle are calculated and processed to obtain a second calculated angle and a third calculated angle; The initial constraint range is adjusted according to the first calculation angle, the second calculation angle and the third calculation angle to obtain a constraint limit range.

4. The method according to claim 3, characterized in that The adjusting the initial constraint range according to the first calculation angle, the second calculation angle and the third calculation angle to obtain a constraint limit range includes: If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine a boundary threshold corresponding to the first calculated angle, a boundary threshold corresponding to the second calculated angle, and a boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle; The initial constraint range is adjusted according to the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle, and the boundary threshold corresponding to the third calculation angle to obtain a constraint limit range.

5. The method according to claim 1, characterized in that The performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result includes: Performing pre-correction processing on the current image frame to obtain an image pre-correction frame; The image pre-distortion frame is detected and processed according to the initial constraint range to obtain an image frame detection result.

6. The method according to claim 5, characterized in that Before the detection process is performed on the image pre-distortion frame according to the initial constraint range to obtain the image frame detection result, the method further includes: Performing point sampling processing on each edge in the image pre-correction frame to obtain n image sampling points corresponding to each edge in the image pre-correction frame, where n is greater than or equal to 1; The detecting and processing the image pre-distortion frame according to the initial constraint range to obtain an image frame detection result includes: For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapping sampling point corresponding to the image sampling point; For a plurality of image sampling points, sampling point detection is performed on mapping sampling points respectively corresponding to the plurality of image sampling points according to the initial constraint range to obtain an image frame detection result.

7. The method according to claim 1, characterized in that The obtaining of the current image frame and the Euler angles corresponding to the current image frame comprises: Acquire a current image frame in real time based on a camera, and acquire motion data corresponding to the current image frame in real time based on an IMU instrument; Calculating and processing the motion data to obtain an initial Euler angle corresponding to the current image frame; The initial Euler angle corresponding to the current image frame is smoothed to obtain the Euler angle corresponding to the current image frame.

8. The method according to claim 1, characterized in that Also includes: Acquire stabilized image frames corresponding to a plurality of image frames respectively, wherein the current image frame is any one of the plurality of image frames; A target video is generated according to the stabilized image frames respectively corresponding to the plurality of image frames, wherein the target video is used to show the target object.

9. An image processing system, characterized in that: include: An image frame acquisition unit, used to acquire a current image frame and the Euler angles corresponding to the current image frame, and to acquire an initial constraint range, wherein the initial constraint range is used to detect whether there is a black edge in the image frame; An image frame detection unit, configured to, if the Euler angle exceeds the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; a constraint range adjustment unit, configured to adjust the initial constraint range according to the Euler angle to obtain a constraint limit range if the image frame detection result indicates that a black border exists in the current image frame, wherein no black border exists in the image frame within the constraint limit range; The image frame stabilization unit is used to perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

10. A sports camera, characterized in that: include: An image processing system configured to execute the image processing method according to any one of claims 1 to 8.

Images