Image processing method and system and motion camera

Abstract

The invention discloses an image processing method and system and a motion camera. Obtaining a current image frame and an Euler angle corresponding to the current image frame, and obtaining an initial constraint range which is used for detecting whether a black edge exists in the image frame; if the Euler angle exceeds the initial constraint range, performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; if the image frame detection result indicates that the black edge exists in the current image frame, the initial constraint range is adjusted according to the Euler angle, a constraint limit range is obtained, and no black edge exists in the image frame within the constraint limit range; and carrying out image stabilization processing on the current image frame according to the constraint limit range to obtain an image-stabilized image frame corresponding to the current image frame. Therefore, the motion compensation of the image frame in the video is realized, and the finally output image frame does not have the black edge, so that the image processing effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, system and motion camera. Background Art

[0002] With the rapid development of information technology, action cameras have become a common tool for people to record their daily lives. However, during the shooting process, action cameras are easily affected by human or environmental factors, which causes the video to shake, thus affecting the viewing experience. To solve this problem, video stabilization technology has been widely used, including mechanical image stabilization technology, optical image stabilization technology, and electronic image stabilization technology.

[0003] Electronic image stabilization technology is widely used in camera technology due to its advantages such as low cost and high flexibility. However, in related technologies, after using the electronic image stabilization technology to perform motion compensation on the captured video, the images in the motion compensated video often have black edges. The black edges in these images not only affect the viewing experience of the video, but also reduce the effective visible area.

[0004] Furthermore, in the related art, the black edges of the image are usually removed by cropping with a fixed rectangular frame and fusing image feature points. However, when the video shakes violently, the fixed rectangular frame cannot guarantee that all crops contain valid pixels. If the image feature points are not fused properly, obvious fusion marks will easily appear on the edge of the image, resulting in poor image processing effect.

[0005] Therefore, how to improve the image processing effect is a key issue that technicians in this field are concerned about. Summary of the invention

[0006] Based on the above problems, the present application provides an image processing method, system and motion camera to improve the image processing effect. The embodiments of the present application disclose the following technical solutions: In a first aspect, the present application discloses an image processing method, comprising: Obtaining a current image frame and the Euler angles corresponding to the current image frame, and obtaining an initial constraint range, wherein the initial constraint range is used to detect whether there is a black edge in the image frame; If the Euler angle exceeds the initial constraint range, performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; If the image frame detection result indicates that there are black edges in the current image frame, adjusting the initial constraint range according to the Euler angle to obtain a constraint limit range, wherein no black edges exist in the image frame within the constraint limit range; The current image frame is subjected to image stabilization processing according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

[0007] In an achievable implementation, the Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle, wherein the first Euler angle includes an angle on a horizontal axis, the second Euler angle includes an angle on a vertical axis, and the third Euler angle includes an angle on a vertical axis; Before the initial constraint range is adjusted according to the Euler angle to obtain the constraint limit range, the method further includes: Get the expansion factor, contraction factor, minimum angle threshold and maximum angle threshold; Determine whether a target Euler angle on a target axis exceeds a boundary of the initial constraint range, wherein the target Euler angle is any one of the first Euler angle, the second Euler angle, and the third Euler angle, and the target axis is an axis determined based on the target Euler angle.

[0008] In an achievable implementation, the adjusting the initial constraint range according to the Euler angle to obtain the constraint limit range includes: If the target Euler angle on the target axis exceeds the boundary of the initial constraint range, calculating and processing the target Euler angle according to the expansion factor to obtain a first calculated angle; According to the contraction factor, the Euler angles among the first Euler angle, the second Euler angle and the third Euler angle except the target Euler angle are calculated and processed to obtain a second calculated angle and a third calculated angle; The initial constraint range is adjusted according to the first calculation angle, the second calculation angle and the third calculation angle to obtain a constraint limit range.

[0009] In an achievable implementation, adjusting the initial constraint range according to the first calculation angle, the second calculation angle, and the third calculation angle to obtain a constraint limit range includes: If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine a boundary threshold corresponding to the first calculated angle, a boundary threshold corresponding to the second calculated angle, and a boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle; The initial constraint range is adjusted according to the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle, and the boundary threshold corresponding to the third calculation angle to obtain a constraint limit range.

[0010] In an achievable implementation manner, performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result includes: Performing pre-correction processing on the current image frame to obtain an image pre-correction frame; The image pre-distortion frame is detected and processed according to the initial constraint range to obtain an image frame detection result.

[0011] In an achievable implementation manner, before performing detection processing on the image pre-distortion frame according to the initial constraint range to obtain the image frame detection result, the method further includes: Performing point sampling processing on each edge in the image pre-correction frame to obtain n image sampling points corresponding to each edge in the image pre-correction frame, where n is greater than or equal to 1; The detecting and processing the image pre-distortion frame according to the initial constraint range to obtain an image frame detection result includes: For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapping sampling point corresponding to the image sampling point; For a plurality of image sampling points, sampling point detection is performed on mapping sampling points respectively corresponding to the plurality of image sampling points according to the initial constraint range to obtain an image frame detection result.

[0012] In an achievable implementation manner, the obtaining the current image frame and the Euler angles corresponding to the current image frame includes: Acquire a current image frame in real time based on a camera, and acquire motion data corresponding to the current image frame in real time based on an IMU instrument; Calculating and processing the operation data to obtain an initial Euler angle corresponding to the current image frame; The initial Euler angle corresponding to the current image frame is smoothed to obtain the Euler angle corresponding to the current image frame.

[0013] In one achievable implementation, it further includes: Acquire stabilized image frames corresponding to a plurality of image frames respectively, wherein the current image frame is any one of the plurality of image frames; A target video is generated according to the stabilized image frames respectively corresponding to the plurality of image frames, wherein the target video is used to show the target object.

[0014] In a second aspect, the present application discloses an image processing system, comprising: An image frame acquisition unit, used to acquire a current image frame and the Euler angles corresponding to the current image frame, and to acquire an initial constraint range, wherein the initial constraint range is used to detect whether there is a black edge in the image frame; An image frame detection unit, configured to, if the Euler angle exceeds the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; a constraint range adjustment unit, configured to adjust the initial constraint range according to the Euler angle to obtain a constraint limit range if the image frame detection result indicates that a black border exists in the current image frame, wherein no black border exists in the image frame within the constraint limit range; The image frame stabilization unit is used to perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

[0015] In one achievable implementation, the system further includes: A factor threshold acquisition unit, used to acquire an expansion factor, a contraction factor, a minimum angle threshold, and a maximum angle threshold; An Euler angle judgment unit is used to judge whether a target Euler angle on a target axis exceeds a boundary of the initial constraint range, wherein the target Euler angle is any one of the first Euler angle, the second Euler angle and the third Euler angle, and the target axis is an axis determined based on the target Euler angle.

[0016] In an achievable implementation, the constraint range adjustment unit includes: A first angle calculation unit, configured to calculate the target Euler angle according to the expansion factor to obtain a first calculated angle if the target Euler angle on the target axis exceeds a boundary of the initial constraint range; a second angle calculation unit, configured to calculate and process the Euler angles of the first Euler angle, the second Euler angle, and the third Euler angle except the target Euler angle according to the contraction factor, to obtain a second calculated angle and a third calculated angle; The restriction range obtaining unit is used to adjust the initial constraint range according to the first calculation angle, the second calculation angle and the third calculation angle to obtain a constraint restriction range.

[0017] In an achievable implementation manner, the restriction range obtaining unit is specifically used to: If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine a boundary threshold corresponding to the first calculated angle, a boundary threshold corresponding to the second calculated angle, and a boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle; The initial constraint range is adjusted according to the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle, and the boundary threshold corresponding to the third calculation angle to obtain a constraint limit range.

[0018] In an achievable implementation, the image frame detection unit includes: A pre-correction processing unit, used for performing pre-correction processing on the current image frame to obtain an image pre-correction frame; The correction frame detection unit is used to perform detection processing on the image pre-correction frame according to the initial constraint range to obtain an image frame detection result.

[0019] In one achievable implementation, the system further includes: A correction frame sampling unit, used for performing a sampling process on each edge in the image pre-correction frame to obtain n image sampling points corresponding to each edge in the image pre-correction frame, wherein n is greater than or equal to 1; The correction frame detection unit is specifically used for: For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapping sampling point corresponding to the image sampling point; For a plurality of image sampling points, sampling point detection is performed on mapping sampling points respectively corresponding to the plurality of image sampling points according to the initial constraint range to obtain an image frame detection result.

[0020] In an achievable implementation manner, the image frame acquisition unit is specifically used to: Acquire a current image frame in real time based on a camera, and acquire motion data corresponding to the current image frame in real time based on an IMU instrument; Calculating and processing the operation data to obtain an initial Euler angle corresponding to the current image frame; The initial Euler angle corresponding to the current image frame is smoothed to obtain the Euler angle corresponding to the current image frame.

[0021] In one achievable implementation, the system further includes: A stabilized image frame acquisition unit, used to acquire stabilized image frames corresponding to a plurality of image frames, wherein the current image frame is any one of the plurality of image frames; The target video generating unit is used to generate a target video according to the stabilized image frames respectively corresponding to the plurality of image frames, wherein the target video is used to show the target object.

[0022] In a third aspect, an embodiment of the present application provides a motion camera, characterized in that it includes: An image processing system is configured to execute the image processing method according to the first aspect.

[0023] Compared with the prior art, this application has the following beneficial effects: In the technical solution of the present application, the current image frame and the Euler angles corresponding to the current image frame are first obtained, and the initial constraint range is obtained. After that, if the Euler angle exceeds the initial constraint range, the current image frame is detected and processed according to the initial constraint range to obtain the image frame detection result. If the image frame detection result indicates that there are black edges in the current image frame, the initial constraint range is adjusted according to the Euler angle to obtain the constraint limit range. Finally, the current image frame is stabilized according to the constraint limit range to obtain the stabilized image frame corresponding to the current image frame. It should be noted that the initial constraint range is used to detect whether there are black edges in the image frame, and there are no black edges in the image frame within the constraint limit range.

[0024] It can be seen that in this application, under the action of the initial constraint range, it can be determined whether the Euler angle corresponding to the current image frame exceeds the initial constraint range and whether there are black edges in the current image frame. If the Euler angle corresponding to the current image frame exceeds the initial constraint range and there are black edges in the current image frame, the initial constraint range can be adjusted based on the Euler angle to obtain a constraint limit range. The stabilized image frame obtained within the constraint limit range does not have black edges. In this way, in this application, while realizing motion compensation for image frames in the video, it can ensure that the final output image frame does not have black edges, thereby improving the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application; Figure 2 A schematic diagram of a black border in an image processing method provided in an embodiment of the present application; Figure 3 A schematic diagram of image frame correction in an image processing method provided in an embodiment of the present application; Figure 4 A schematic diagram of a boundary of an initial constraint range in an image processing method provided in an embodiment of the present application; Figure 5 A full flow chart of image processing in an image processing method provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an image processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] It should be noted that the image processing method, system and motion camera provided by the present application are used in the field of image processing technology. The above are only examples and do not limit the application fields of the method and system names provided by the present application.

[0029] As described above, with the rapid development of information technology, action cameras have become a common tool for people to record their daily lives. However, during the shooting process, action cameras are easily affected by human or environmental factors, which causes the video to shake, thus affecting the viewing experience. In order to solve this problem, video stabilization technology has been widely used, including mechanical image stabilization technology, optical image stabilization technology and electronic image stabilization technology.

[0030] Electronic image stabilization technology is widely used in camera technology due to its advantages such as low cost and high flexibility. However, in related technologies, after using the electronic image stabilization technology to perform motion compensation on the captured video, the images in the motion compensated video often have black edges. The black edges in these images not only affect the viewing experience of the video, but also reduce the effective visible area.

[0031] Furthermore, in the related art, the black edges of the image are usually removed by cropping with a fixed rectangular frame and fusing image feature points. The cropping with a fixed rectangular frame is usually reflected in setting a rectangular frame of a specific size and cropping the image based on the rectangular frame. However, when the video shakes violently, the fixed rectangular frame cannot guarantee that all crops contain valid pixels. And the image feature point fusion method is usually reflected in registering the reference frame (the image after motion compensation) and the current frame (the image before motion compensation), that is, finding the feature points between the reference frame and the current frame, and fusing the reference frame and the current frame through the feature points. However, if the image feature points are not properly fused, the image edge is prone to obvious fusion marks, and the computational overhead is also relatively high. In this way, the fixed rectangular frame cropping method and the image feature point fusion method proposed in the related art will lead to poor image processing effects, and both methods are offline tasks. Therefore, how to improve the image processing effect is a key issue that technicians in this field are concerned about.

[0032] In another feasible implementation, the related art also proposes that for image processing of online tasks, information from past frames can be used to perform multi-frame fusion and filtering on the current frame to expand the effective pixel area in the image and reduce the impact of black edges. However, multi-frame fusion will cause obvious fusion marks to appear on the edges of the image, affecting the visual quality. The computational overhead of multi-frame fusion and filtering is large, which significantly increases processing delays and hardware risks, thereby resulting in poor image processing efficiency.

[0033] Therefore, the inventors proposed the technical solution of the present application, aiming to improve the image processing effect. In the embodiment of the present application, the current image frame and the Euler angles corresponding to the current image frame are first obtained, and the initial constraint range is obtained. After that, if the Euler angles exceed the initial constraint range, the current image frame is detected and processed according to the initial constraint range to obtain the image frame detection result, and if the image frame detection result indicates that there are black edges in the current image frame, the initial constraint range is adjusted and processed according to the Euler angles to obtain the constraint restriction range, and finally the current image frame is stabilized according to the constraint restriction range to obtain a stabilized image frame corresponding to the current image frame. It should be noted that the initial constraint range is used to detect whether there are black edges in the image frame, and there are no black edges in the image frame within the constraint restriction range.

[0034] It can be seen that in this application, under the action of the initial constraint range, it can be determined whether the Euler angle corresponding to the current image frame exceeds the initial constraint range and whether there are black edges in the current image frame. If the Euler angle corresponding to the current image frame exceeds the initial constraint range and there are black edges in the current image frame, the initial constraint range can be adjusted based on the Euler angle to obtain a constraint limit range. The stabilized image frame obtained within the constraint limit range does not have black edges. In this way, in this application, while realizing real-time motion compensation for image frames in the video, it can ensure that the final output image frame does not have black edges, thereby improving the image processing effect, and then improving the image processing efficiency, so as to better adapt to the real-time processing requirements of video in complex dynamic scenes.

[0035] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] An image processing method provided by the present application is described below through an embodiment.

[0037] See also Figure 1 , which is a flow chart of an image processing method provided by an embodiment of the present application, such as Figure 1 As shown, the method includes: S101: Obtain a current image frame and the Euler angles corresponding to the current image frame, and obtain an initial constraint range.

[0038] It should be noted that in this step, firstly, the current image frame can be acquired in real time based on the camera, and the current image frame can be any frame in the real-time video of the object, and the motion data corresponding to the current image frame can be acquired in real time based on the IMU (Inertial Measurement Unit) instrument. The IMU instrument includes an accelerometer and a gyroscope, and the motion data can be used to calculate the posture changes of the movement.

[0039] After this, the running data can be calculated and processed to obtain the initial Euler angle corresponding to the current image frame. Specifically, in this step, filtering technology can be first used to reduce the impact of noise generated by the IMU instrument on the motion data, so as to improve the accuracy of motion posture estimation. After that, the angular velocity data of the gyroscope in the IMU instrument can be used to calculate the initial Euler angle corresponding to the current image frame by numerical integration. In the process of calculating the initial Euler angle, it is also necessary to align the camera and IMU data for timestamps. It should be noted that the method of obtaining the Euler angle mentioned in this application is similar to the existing solution, and will not be described in detail here.

[0040] In this step, in order to make the initial Euler angle corresponding to the current image frame obtained by calculating the motion data smoother and more stable, the initial Euler angle corresponding to the current image frame is smoothed by the Kalman filter algorithm to obtain the Euler angle corresponding to the current image frame. It should be noted that the Kalman filter algorithm mentioned in this application is an existing algorithm and will not be described in detail here.

[0041] Furthermore, in this step, an initial constraint range can also be obtained, and the initial constraint range is used to detect whether there is a black border in the image frame. Figure 2 As shown, Figure 2 A schematic diagram of a black border in an image processing method provided in an embodiment of the present application, in Figure 2 The black area on the left can be understood as the black edge in the image.

[0042] Next, the process of obtaining the initial constraint range is introduced. In this application, firstly, a random Euler angle can be obtained, which is obtained by random adoption, and the random Euler angle is screened to obtain the screened random Euler angle. The screened random Euler angle is the Euler angle that will not have a black edge after mapping the image under a specified field of view (such as the horizontal axis, the vertical axis, etc.), so as to form a point set based on the screened random Euler angle. , where i represents the point number, and y, p, and r represent the three-dimensional coordinates corresponding to the random Euler angles after screening. It should be noted that the black edge detection algorithm mentioned later in this application can be used for the screening process of random Euler angles. For the introduction of the black edge detection algorithm, see the subsequent description.

[0043] After that, the convex hull fitting can be performed on the selected random Euler angles, that is, the minimum coordinate can be determined from the point set. and coordinate maximum The initial edge of the initial constraint range is determined based on the point with the smallest coordinate and the point with the largest coordinate. The initial constraint range can be understood as an initial rectangular frame. The point farthest from the initial edge is used as the new convex hull vertex to divide the point set into different subsets, and the same operation is recursively performed on each subset until all points are included in the convex hull, and finally a convex hull is formed.

[0044] Finally, we can use the ellipsoid equation (where x, y, z represent the horizontal axis, the vertical axis, and the vertical axis, respectively, and a represents the coefficient), use the points on the convex hull to construct the least squares equation, and obtain the minimum circumscribed ellipsoid of the convex hull based on the least squares equation. And obtain its maximum inscribed cuboid according to the three-axis length of the ellipsoid. The maximum inscribed cuboid is the range where no black edges will appear even if it is properly adjusted, which is the initial constraint range. In this way, in this application, by using Euler angles without black edges and solving the initial range of Euler angle constraints through Euler angles, dynamic initialization of the jitter constraint range is achieved.

[0045] S102: If the Euler angle exceeds the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result.

[0046] It should be noted that before executing step S102, the present application also needs to determine whether the current image frame is within the initial constraint range. If the current image frame is within the initial constraint range or the current image frame is not within the initial constraint range, determine whether the Euler angle exceeds the initial constraint range. If the Euler angle does not exceed the initial constraint range, the current image frame is stabilized; if the Euler angle exceeds the initial constraint range, the current image frame can be pre-corrected to obtain an image pre-corrected frame, and the image pre-corrected frame is detected according to the initial constraint range to obtain an image frame detection result, wherein the pre-correction processing can be image dedistortion, and the pre-correction processing can be image transformation.

[0047] It should also be noted that in the present application, a black edge detection algorithm can be used to detect whether there are black edges in the image pre-correction frame. The black edge detection algorithm is as follows: before performing detection processing on the image pre-correction frame according to the initial constraint range to obtain the image frame detection result operation, the present application can also perform point sampling processing on each edge in the image pre-correction frame to obtain n image sampling points corresponding to each edge in the image pre-correction frame, where n is greater than or equal to 1, and the sampling points of each edge can be the same, or the sampling points of each edge can be different, for example, one edge in the image pre-correction frame can be 12 points, and the other edge in the image pre-correction frame can be 8 points.

[0048] Specifically, for each image sampling point, the image sampling point can be inversely mapped (inversely projected) to obtain the mapping sampling point corresponding to the image sampling point, and for multiple image sampling points, the mapping sampling points corresponding to the multiple image sampling points are respectively detected according to the initial constraint range to obtain the image frame detection result, wherein the image frame detection result includes the result of the existence of black edges in the current image frame, or the result of the absence of black edges in the current image frame. In this way, in the present application, by sampling the midpoint of the image pre-correction frame, it is possible to more quickly determine whether there is a black edge in the initial constraint range.

[0049] like Figure 3 As shown, Figure 3 A schematic diagram of image frame correction in an image processing method provided in an embodiment of the present application. Specifically, Figure 3 (a) in the figure is the current image frame. Figure 3 (b) in the figure is the pre-corrected frame. Further, Figure 3 The black rectangle in (b) can be understood as the initial constraint range. Figure 3The black area in (b) can be understood as the black edge. Figure 3 The points on the region boundary of the grid area in (b) can be understood as image sampling points.

[0050] In one achievable manner, in this application, steps (1) to (7) may be used to introduce a black edge detection algorithm, where steps (1) to (7) are specifically as follows: Step (1): Evenly select points on the four edges of the image pre-corrected frame , where i represents the serial number of the point and n represents the total number of points; Step (2): For any image sampling point in the sampling point set P , transformed to the normalized plane: , get the pixel point of the image sampling point on the normalized plane ,in Indicates the position of the original pixel in the pre-corrected frame of the image, is the new principal point of the image pre-distortion frame, Respectively represent the new focal lengths of the x-axis and y-axis on the image pre-correction frame, It can be expressed as the coordinate position in the horizontal dimension, It can be expressed as a coordinate position in the vertical dimension; Step (3): Calculate the pixel point of the image sampling point on the normalized plane Perform the inverse rotation transformation: , where W represents the dimension in the inverse perspective projection, R represents the rotation matrix, T represents the transpose operation, X and Y represent the transformed horizontal and vertical coordinates, respectively, and x and y represent the original horizontal and vertical coordinates, respectively; Step (4): Normalize the pixels after the inverse rotation transformation: , ,in Indicates the new pixel point of the image sampling point on the normalized plane; Step (5): Dedistort the new pixels: , ,in is the distortion coefficient, and the distance ; Step (6): The new pixel after dedistortion Map back to the current image frame and obtain the mapping sampling points : , .in is the principal point of the current image frame, Respectively represent the focal lengths of the x-axis and y-axis on the current image frame; Step (7): Perform the operations of steps (2) to (6) for each image sampling point to determine whether each point satisfies the condition in the current image frame. , meeting the requirements means that the image pre-correction frame has no black edges within the visible range, where Indicates the width of the image frame, Indicates the height of the image frame.

[0051] It is understandable that if the judgment requirements are met, no black edges will appear when the image pre-correction frame is cropped using the initial constraint range. In this way, in this application, only simple matrix operations are performed on a few points to achieve inverse transformation, which can simply and efficiently achieve real-time black edge detection, reduce computational complexity and resource consumption, and use this black edge detection algorithm in practical applications to achieve efficient detection (100 720p (row pixels), 1080p, and 2160p pictures were taken for testing, and the time used was 17ms (milliseconds), 19ms, and 25ms, respectively, that is, the detection time of each picture is about 0.2ms, and as the image resolution increases, the increase in image detection time is not significant).

[0052] S103: If the image frame detection result indicates that a black border exists in the current image frame, the initial constraint range is adjusted according to the Euler angle to obtain a constraint limit range.

[0053] In this step, the Euler angle includes a first Euler angle, a second Euler angle and a third Euler angle, wherein the first Euler angle includes an angle on the horizontal axis, the second Euler angle includes an angle on the vertical axis, and the third Euler angle includes an angle on the vertical axis. It should be noted that there may be multiple Euler angles in an image frame, and the Euler angles shown in this application are only some examples. And if the image frame detection result indicates that there is no black edge in the current image frame, the current image frame may be stabilized to obtain a stabilized image frame.

[0054] In an achievable implementation, before the application performs the adjustment process of the initial constraint range according to the Euler angle and obtains the constraint limit range operation, it can also obtain the expansion factor, the contraction factor, the minimum angle threshold and the maximum angle threshold, which can be used to adjust the initial constraint range, wherein the expansion factor and the contraction factor are empirical values, and the expansion factor and the contraction factor vary in different scenarios, the minimum angle threshold and the maximum angle threshold are preset values, and the specific settings of the expansion factor, the contraction factor, the minimum angle threshold and the maximum angle threshold can also be adjusted according to the actual situation in the actual application. And it can be judged whether the target Euler angle on the target axis exceeds the boundary of the initial constraint range, wherein the target Euler angle is any one of the first Euler angle, the second Euler angle and the third Euler angle, and the target axis is an axis determined based on the target Euler angle, such as the target Euler angle is an angle on a certain axis, then the target axis is a certain axis.

[0055] It is understandable that in the present application, it is possible to determine whether the first Euler angle on the horizontal axis exceeds the boundary of the initial constraint range, or to determine whether the second Euler angle on the vertical axis exceeds the boundary of the initial constraint range, or to determine whether the third Euler angle on the vertical axis exceeds the boundary of the initial constraint range. In this way, when the video shakes over a large range, the axis where the black edge may exist can be determined, so as to accurately adjust the initial constraint range.

[0056] like Figure 4 As shown, Figure 4 A schematic diagram of the boundary of an initial constraint range in an image processing method provided in an embodiment of the present application. Figure 4 (a) in the figure indicates that the first Euler angle on the horizontal axis exceeds the boundary of the initial constraint range. Figure 4 (b) in the figure shows that the second Euler angle on the vertical axis is close to the boundary of the initial constraint range. Figure 4 (c) in the figure shows that the third Euler angle on the vertical axis exceeds the boundary of the initial constraint range.

[0057] Specifically, in the present application, if the target Euler angle on the target axis exceeds the boundary of the initial constraint range, the first Euler angle, the second Euler angle and the third Euler angle can be calculated and processed according to the expansion factor and the contraction factor to obtain the first calculation angle, the second calculation angle and the third calculation angle. Thereafter, the first calculation angle, the second calculation angle and the third calculation angle can be calculated and processed according to the minimum angle threshold and the maximum angle threshold to obtain the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle and the boundary threshold corresponding to the third calculation angle. Finally, the initial constraint range can be adjusted according to the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle and the boundary threshold corresponding to the third calculation angle to obtain the constraint limit range.

[0058] Next, the process of adjusting the initial constraint range to obtain the constraint limit range in the present application is further introduced. If the target Euler angle on the target axis exceeds the boundary of the initial constraint range, the target Euler angle can be calculated according to the expansion factor to obtain a first calculated angle. The process of obtaining the first calculated angle can be reflected by formula (1). Formula (1) is specifically reflected as follows: Formula (1) in, represents the first calculated angle, represents the target Euler angle, and X represents the expansion factor, which can be 0.05. In this way, when the Euler angle of a certain axis approaches or exceeds the boundary of the initial constraint range, it can be dynamically expanded based on the expansion factor.

[0059] Thereafter, the Euler angles other than the target Euler angle in the first Euler angle, the second Euler angle, and the third Euler angle may be respectively calculated and processed according to the contraction factor to obtain the second calculated angle and the third calculated angle. The process of obtaining the second calculated angle / the third calculated angle may be embodied by formula (2). Formula (2) is specifically embodied as follows: Formula (2) in, Indicates the second calculation angle / third calculation angle, represents the target Euler angle, and X represents the shrinkage factor, which can be 0.05. In this way, when the Euler angle of a certain axis is not close to or exceeds the boundary of the initial constraint range, it can be appropriately dynamically shrunk based on the shrinkage factor.

[0060] Finally, the initial constraint range can be adjusted according to the first calculated angle, the second calculated angle, and the third calculated angle to obtain a constraint limit range, wherein the image frame has no black edges within the constraint limit range. Specifically, if the first calculated angle is between the minimum angle threshold and the maximum angle threshold , and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, then the boundary threshold corresponding to the first calculated angle, the boundary threshold corresponding to the second calculated angle, and the boundary threshold corresponding to the third calculated angle can be determined according to the first calculated angle, the second calculated angle, and the third calculated angle.

[0061] If any of the first calculation angle, the second calculation angle, and the third calculation angle does not meet the requirements, the expansion and contraction steps need to be repeated until each calculation angle meets the above requirements. Thereafter, the initial constraint range can be adjusted according to the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle, and the boundary threshold corresponding to the third calculation angle to obtain the constraint limit range. In this way, dynamic adjustment of the initial constraint range is achieved in the present application.

[0062] S104: Performing image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

[0063] It can be understood that at this time, the adjusted Euler angles are constrained within the new constraint range, and the adjusted Euler angles can be applied to the current image frame for stabilization through inverse geometric transformation, so that the stabilized image frame can contain valid pixels, ensuring that the corrected image is distortion-free and has no black edges during the stabilization process.

[0064] After that, the present application can also obtain stabilized image frames corresponding to multiple image frames, and generate a target video based on the stabilized image frames corresponding to the multiple image frames, wherein the current image frame is any one of the multiple image frames, and the target video is used to show the target object (such as the person being photographed, etc.). In this way, the technical solution of the present application can realize the processing of images in real time and efficiently, so as to output high-quality videos and improve the user experience.

[0065] like Figure 5 As shown, Figure 5 This is a full flow chart of image processing in an image processing method provided in an embodiment of the present application. Figure 5First, the current image frame and the Euler angle corresponding to the current image frame can be obtained. Then, under the action of the initial constraint range, it is determined whether the Euler angle corresponding to the current image frame exceeds the initial constraint range. If the Euler angle corresponding to the current image frame exceeds the initial constraint range, the current image frame is determined to perform black edge detection to obtain the image frame detection result; if the Euler angle corresponding to the current image frame does not exceed the initial constraint range, the current image frame is stabilized to obtain a stabilized image frame.

[0066] If the image frame detection result indicates that there are black edges in the current image frame, the initial constraint range can be adjusted based on the Euler angle to obtain the constraint limit range, so as to implement subsequent image stabilization based on the constraint limit range; if the image frame detection result indicates that there are no black edges in the current image frame, the current image frame is stabilized to obtain a stabilized image frame. In this way, the present application can ensure that the final output image frame does not have black edges while implementing motion compensation for the image frame in the video in real time, thereby improving the image processing effect.

[0067] In summary, in the technical solution of the present application, under the action of the initial constraint range, it can be determined whether the Euler angle corresponding to the current image frame exceeds the initial constraint range and whether there are black edges in the current image frame. If the Euler angle corresponding to the current image frame exceeds the initial constraint range and there are black edges in the current image frame, the initial constraint range can be adjusted based on the Euler angle to obtain a constraint limit range, and the stabilized image frame obtained within the constraint limit range does not have black edges. In this way, in the present application, while realizing real-time motion compensation for image frames in the video, it can ensure that the final output image frame does not have black edges, thereby improving the image processing effect, and then improving the image processing efficiency, so as to better adapt to the real-time processing requirements of video in complex dynamic scenes.

[0068] An image processing system provided in an embodiment of the present application is introduced below. The image processing system described below and the image processing method described above can refer to each other.

[0069] See also Figure 6 , which is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application, such as Figure 6 As shown, the image processing system includes: An image frame acquisition unit 601 is used to acquire a current image frame and the Euler angles corresponding to the current image frame, and to acquire an initial constraint range, wherein the initial constraint range is used to detect whether there is a black edge in the image frame; An image frame detection unit 602 is configured to, if the Euler angle exceeds the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; A constraint range adjustment unit 603 is configured to adjust the initial constraint range according to the Euler angle to obtain a constraint limit range if the image frame detection result indicates that a black border exists in the current image frame, wherein no black border exists in the image frame within the constraint limit range; The image frame stabilization unit 604 is used to perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

[0070] In one achievable implementation, the system further includes: A factor threshold acquisition unit, used to acquire an expansion factor, a contraction factor, a minimum angle threshold, and a maximum angle threshold; An Euler angle judgment unit is used to judge whether a target Euler angle on a target axis exceeds a boundary of the initial constraint range, wherein the target Euler angle is any one of the first Euler angle, the second Euler angle and the third Euler angle, and the target axis is an axis determined based on the target Euler angle.

[0071] In an achievable implementation, the constraint range adjustment unit includes: A first angle calculation unit, configured to calculate the target Euler angle according to the expansion factor to obtain a first calculated angle if the target Euler angle on the target axis exceeds a boundary of the initial constraint range; a second angle calculation unit, configured to calculate and process the Euler angles of the first Euler angle, the second Euler angle, and the third Euler angle except the target Euler angle according to the contraction factor, to obtain a second calculated angle and a third calculated angle; The restriction range obtaining unit is used to adjust the initial constraint range according to the first calculation angle, the second calculation angle and the third calculation angle to obtain a constraint restriction range.

[0072] In an achievable implementation manner, the restriction range obtaining unit is specifically used to: If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine a boundary threshold corresponding to the first calculated angle, a boundary threshold corresponding to the second calculated angle, and a boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle; The initial constraint range is adjusted according to the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle, and the boundary threshold corresponding to the third calculation angle to obtain a constraint limit range.

[0073] In an achievable implementation, the image frame detection unit includes: A pre-correction processing unit, used for performing pre-correction processing on the current image frame to obtain an image pre-correction frame; The correction frame detection unit is used to perform detection processing on the image pre-correction frame according to the initial constraint range to obtain an image frame detection result.

[0074] In one achievable implementation, the system further includes: A correction frame sampling unit, used for performing a sampling process on each edge in the image pre-correction frame to obtain n image sampling points corresponding to each edge in the image pre-correction frame, wherein n is greater than or equal to 1; The correction frame detection unit is specifically used for: For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapping sampling point corresponding to the image sampling point; For a plurality of image sampling points, sampling point detection is performed on mapping sampling points respectively corresponding to the plurality of image sampling points according to the initial constraint range to obtain an image frame detection result.

[0075] In an achievable implementation manner, the image frame acquisition unit is specifically used to: Acquire a current image frame in real time based on a camera, and acquire motion data corresponding to the current image frame in real time based on an IMU instrument; Calculating and processing the operation data to obtain an initial Euler angle corresponding to the current image frame; The initial Euler angle corresponding to the current image frame is smoothed to obtain the Euler angle corresponding to the current image frame.

[0076] In one achievable implementation, the system further includes: A stabilized image frame acquisition unit, used to acquire stabilized image frames corresponding to a plurality of image frames, wherein the current image frame is any one of the plurality of image frames; The target video generating unit is used to generate a target video according to the stabilized image frames respectively corresponding to the plurality of image frames, wherein the target video is used to show the target object.

[0077] The image processing system provided in the embodiment of the present application has the same beneficial effects as the image processing method provided in the above embodiment, so it will not be described in detail.

[0078] In addition, the present invention also discloses a sports camera, including an image processing system, and the image processing system is the image processing system disclosed in the above embodiment, which can be configured to execute the above image processing method. Therefore, the sports camera with the image processing system also has all the above technical effects, which will not be repeated here. It should be noted that the sports camera can be GoPro, and the sports camera can also be Insta360.

[0079] It should be noted that the image processing method provided by the present invention can be used in the field of image processing technology. The above is only an example and does not limit the application field of the image processing method provided by the present invention. It should also be noted that the "first" and "second" (if any) mentioned in the embodiments of the present application are only used as name identifications and do not represent the first and second in order.

[0080] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0081] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0082] The above is a detailed introduction to an image processing method, system and motion camera provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Obtaining a current image frame and the Euler angles corresponding to the current image frame, and obtaining an initial constraint range, wherein the initial constraint range is used to detect whether there is a black edge in the image frame; If the Euler angle exceeds the initial constraint range, performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; If the image frame detection result indicates that there are black edges in the current image frame, adjusting the initial constraint range according to the Euler angle to obtain a constraint limit range, wherein no black edges exist in the image frame within the constraint limit range; The current image frame is subjected to image stabilization processing according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

2. The method according to claim 1, characterized in that The Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle, wherein the first Euler angle includes an angle on the horizontal axis, the second Euler angle includes an angle on the vertical axis, and the third Euler angle includes an angle on the vertical axis; Before the initial constraint range is adjusted according to the Euler angle to obtain the constraint limit range, the method further includes: Get the expansion factor, contraction factor, minimum angle threshold and maximum angle threshold; Determine whether a target Euler angle on a target axis exceeds a boundary of the initial constraint range, wherein the target Euler angle is any one of the first Euler angle, the second Euler angle, and the third Euler angle, and the target axis is an axis determined based on the target Euler angle.

3. The method according to claim 2, characterized in that The adjusting process of the initial constraint range according to the Euler angle to obtain the constraint limit range includes: If the target Euler angle on the target axis exceeds the boundary of the initial constraint range, calculating and processing the target Euler angle according to the expansion factor to obtain a first calculated angle; According to the contraction factor, the Euler angles among the first Euler angle, the second Euler angle and the third Euler angle except the target Euler angle are calculated and processed to obtain a second calculated angle and a third calculated angle; The initial constraint range is adjusted according to the first calculation angle, the second calculation angle and the third calculation angle to obtain a constraint limit range.

4. The method according to claim 3, characterized in that The adjusting the initial constraint range according to the first calculation angle, the second calculation angle and the third calculation angle to obtain a constraint limit range includes: If the first calculated angle is between the minimum angle threshold and the maximum angle threshold, and the second calculated angle is between the minimum angle threshold and the maximum angle threshold, and the third calculated angle is between the minimum angle threshold and the maximum angle threshold, determine a boundary threshold corresponding to the first calculated angle, a boundary threshold corresponding to the second calculated angle, and a boundary threshold corresponding to the third calculated angle according to the first calculated angle, the second calculated angle, and the third calculated angle; The initial constraint range is adjusted according to the boundary threshold corresponding to the first calculation angle, the boundary threshold corresponding to the second calculation angle, and the boundary threshold corresponding to the third calculation angle to obtain a constraint limit range.

5. The method according to claim 1, characterized in that The performing detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result includes: Performing pre-correction processing on the current image frame to obtain an image pre-correction frame; The image pre-distortion frame is detected and processed according to the initial constraint range to obtain an image frame detection result.

6. The method according to claim 5, characterized in that Before the detection process is performed on the image pre-distortion frame according to the initial constraint range to obtain the image frame detection result, the method further includes: Performing point sampling processing on each edge in the image pre-correction frame to obtain n image sampling points corresponding to each edge in the image pre-correction frame, where n is greater than or equal to 1; The detecting and processing the image pre-distortion frame according to the initial constraint range to obtain an image frame detection result includes: For each image sampling point, perform inverse perspective mapping on the image sampling point to obtain a mapping sampling point corresponding to the image sampling point; For a plurality of image sampling points, sampling point detection is performed on mapping sampling points respectively corresponding to the plurality of image sampling points according to the initial constraint range to obtain an image frame detection result.

7. The method according to claim 1, characterized in that The obtaining of the current image frame and the Euler angles corresponding to the current image frame comprises: Acquire a current image frame in real time based on a camera, and acquire motion data corresponding to the current image frame in real time based on an IMU instrument; Calculating and processing the operation data to obtain an initial Euler angle corresponding to the current image frame; The initial Euler angle corresponding to the current image frame is smoothed to obtain the Euler angle corresponding to the current image frame.

8. The method according to claim 1, characterized in that Also includes: Acquire stabilized image frames corresponding to a plurality of image frames respectively, wherein the current image frame is any one of the plurality of image frames; A target video is generated according to the stabilized image frames respectively corresponding to the plurality of image frames, wherein the target video is used to show the target object.

9. An image processing system, characterized in that: include: An image frame acquisition unit, used to acquire a current image frame and the Euler angles corresponding to the current image frame, and to acquire an initial constraint range, wherein the initial constraint range is used to detect whether there is a black edge in the image frame; An image frame detection unit, configured to, if the Euler angle exceeds the initial constraint range, perform detection processing on the current image frame according to the initial constraint range to obtain an image frame detection result; a constraint range adjustment unit, configured to adjust the initial constraint range according to the Euler angle to obtain a constraint limit range if the image frame detection result indicates that a black border exists in the current image frame, wherein no black border exists in the image frame within the constraint limit range; The image frame stabilization unit is used to perform image stabilization processing on the current image frame according to the constraint limit range to obtain a stabilized image frame corresponding to the current image frame.

10. A sports camera, characterized in that: include: An image processing system configured to execute the image processing method according to any one of claims 1 to 8.

Images