Wireless image transmission system, video image stabilization method and computer program product

Abstract

The embodiment of the invention provides a wireless image transmission system, a video image stabilization method and a computer program product. In a wireless image transmission scene, aiming at a scene in which the camera device does not have a built-in anti-shake function and at least one of a transmitting device and a receiving device in a wireless image transmission system is provided with an attitude measurement unit, the camera device can be rigidly connected with at least one of the transmitting device and the receiving device; therefore, the camera device can multiplex the attitude data acquired by the attitude measurement unit of the transmitting device or the receiving device, and image stabilization processing is carried out on the video data acquired by the camera device based on the attitude data, so that image stabilization processing can be carried out on the video image acquired by the camera device in a scene without a built-in anti-shake function of the camera device; and stable, clear and high-quality video data can be obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more particularly, to a wireless video transmission system, a video stabilization method, and a computer program product. Background Art

[0002] When a camera device captures video images, due to the influence of the carrier of the camera device or the shooting environment, the captured video footage may exhibit jitter, affecting the viewing experience. Current video stabilization technologies typically rely on the built-in optical or electronic image stabilization functions of the camera device to achieve video stabilization. However, this approach requires the camera device itself to carry an attitude measurement unit such as an IMU that can measure the attitude changes of the camera device in real time. However, for some camera devices, they may not have a built-in attitude measurement unit and do not have an image stabilization function. In such a scenario, it is not possible to perform video stabilization processing on the video footage captured by the camera device, resulting in a poor viewing effect. Summary of the Invention

[0003] In view of this, the present application provides a wireless video transmission system, a video stabilization method, and a computer program product.

[0004] According to a first aspect of the present application, there is provided a wireless video transmission system including a transmitting device and a receiving device, at least one of the transmitting device and the receiving device being provided with an attitude measurement unit and being rigidly connected to a camera device;

[0005] The transmitting device is configured to obtain video data captured by the camera device and send the video data to the receiving device;

[0006] The receiving device is configured to obtain attitude data collected by the attitude measurement unit, perform video stabilization processing on the video data based on the attitude data, and output the processed video data.

[0007] According to a second aspect of the present application, there is provided a video stabilization method, the method comprising:

[0008] Obtaining video data captured by a camera device and obtaining attitude data collected by an attitude measurement unit provided on a target device rigidly connected to the camera device;

[0009] Performing video stabilization processing on the video data based on the attitude data and outputting the processed video data.

[0010] According to a third aspect of the present application, there is provided a computer program product including a computer program, which when executed implements the method mentioned in the second aspect.

[0011] According to a fourth aspect of the present application, there is provided an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, the method mentioned in the second aspect above is implemented.

[0012] According to a fifth aspect of the present application, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the method mentioned in the second aspect above is implemented.

[0013] Applying the solution provided by the present application, in the scenario of wireless video transmission, the camera device connected to the wireless video transmission system may not have an anti-shake function, resulting in potentially low video quality and affecting subsequent applications. Considering that an attitude measurement unit is usually provided in the transmitting device or receiving device of the wireless video transmission system, therefore, for the scenario where the camera device does not have a built-in anti-shake function and at least one of the transmitting device and receiving device in the wireless video transmission system is provided with an attitude measurement unit, the camera device can be rigidly connected to at least one of the transmitting device and receiving device. Thus, the camera device can reuse the attitude data collected by the attitude measurement unit of the transmitting device or receiving device, and perform video stabilization processing on the video data collected by the camera device based on this attitude data, so as to perform video stabilization processing on the video image collected by the camera device in the scenario where the camera device does not have a built-in anti-shake function, and obtain stable, clear, and high-quality video data.

[0014] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] FIG. 1(a) is a schematic structural diagram of a wireless video transmission system according to an embodiment of the present application.

[0017] FIG. 1(b) is a schematic structural diagram of a wireless video transmission system according to another embodiment of the present application.

[0018] Figure 2 is a schematic diagram of an application scenario according to an embodiment of the present application.

[0019] Figure 3 is a schematic diagram of an application scenario according to another embodiment of the present application.

[0020] Figure 4It is a schematic diagram of cropping the video picture after image stabilization processing according to an embodiment of the present application.

[0021] Figure 5 It is a flowchart of a video image stabilization method according to another embodiment of the present application.

[0022] Figure 6 It is a schematic diagram of the logical structure of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0024] When the imaging device captures a video image, due to the influence of the carrier of the imaging device or the shooting environment, the random jitter of the imaging device relative to the object to be captured is inevitable, which may cause the captured video picture to shake, greatly affecting the imaging quality and viewing experience. At present, mainly by built-in anti-shake function in the imaging device to eliminate or reduce this kind of jitter, so as to improve the stability and clarity of the video.

[0025] At present, the common anti-shake methods include optical anti-shake and electronic anti-shake. Optical anti-shake is to detect the pose change of the imaging device through an attitude measurement unit such as an IMU built in the imaging device, and then control the movement of the lens or sensor of the imaging device based on the detected pose change to offset the jitter.

[0026] Electronic anti-shake can use software algorithms to process the video images captured by the imaging device to reduce jitter. For example, the attitude measurement unit built in the imaging device can be used to detect the pose change of the imaging device, and based on this pose change, the image or video frame can be rotated, cropped, interpolated and synthesized to offset the jitter.

[0027] Whether it is optical anti-shake or electronic anti-shake, in order to accurately detect the jitter of the imaging device, it is required that the imaging device is built with an attitude measurement unit. However, some imaging devices may not have an attitude measurement unit itself. For such scenarios, it is currently impossible to accurately determine the jitter of the imaging device, and then perform image stabilization processing on the video images captured by the imaging device to improve the quality and viewing experience of the video images.

[0028] With the rapid development of wireless communication technology, wireless video transmission technology has been widely used in many fields, such as drone aerial photography, security monitoring, telemedicine, and intelligent transportation. Wireless video transmission technology is to transmit the video data collected by a camera device from one device to another. A wireless video transmission system usually includes a transmitting device and a receiving device. The transmitting device is wirelessly connected to the receiving device. The transmitting device can communicate with the camera device to obtain the video data collected by the camera device and wirelessly transmit it to the receiving device. The receiving device can display the video data or send it to other devices for post-production, etc.

[0029] In the scenario of wireless video transmission, the camera device connected to the wireless video transmission system may not have an anti-shake function, resulting in potentially low video quality and affecting subsequent applications. Considering that an attitude measurement unit is usually provided in the transmitting device or the receiving device of the wireless video transmission system, therefore, for the scenario where the camera device does not have a built-in anti-shake function and at least one of the transmitting device and the receiving device in the wireless video transmission system is provided with an attitude measurement unit, the camera device can be rigidly connected to at least one of the transmitting device and the receiving device. Thus, the camera device can reuse the attitude data collected by the attitude measurement unit of the transmitting device or the receiving device, and perform image stabilization processing on the video data collected by the camera device based on this attitude data. So that in the scenario where the camera device does not have a built-in anti-shake function, it is also possible to perform image stabilization processing on the video image collected by the camera device to obtain stable, clear, and high-quality video data.

[0030] Based on the above inventive concept, an embodiment of the present application provides a wireless video transmission system, as Figure 1(a) and 1(b) shown. The wireless video transmission system includes a transmitting device and a receiving device. The transmitting device is communicatively connected to the camera device. For example, it can be connected to the camera device through some video transmission interfaces (such as SDI, HDMI). The transmitting device and the receiving device are wirelessly connected, and the two can transmit video or image data through some wireless transmission protocols.

[0031] At least one of the transmitting device and the receiving device is provided with an attitude measurement unit and is rigidly connected to the imaging device. Among them, Fig. 1(a) takes the transmitting device as an example, which is provided with an attitude measurement unit and is rigidly connected to the imaging device, and Fig. 1(b) takes the receiving device as an example, which is provided with an attitude measurement unit and is rigidly connected to the imaging device. Among them, the attitude measurement unit can be various sensors capable of measuring the pose change of the transmitting device or the receiving device. For example, it can be an inertial measurement unit (IMU, usually including a gyroscope, an accelerometer, and a magnetometer), or other sensors for measuring the pose change of an object, etc., which are not limited in the embodiments of the present application. The fact that the transmitting device and / or the receiving device is rigidly connected to the imaging device means that during the process of the imaging device shooting video images, the relative pose of the transmitting device and / or the receiving device and the imaging device remains unchanged. Therefore, the pose change conditions collected by the attitude measurement units of the transmitting device and / or the receiving device are the pose change conditions of the imaging device.

[0032] In some embodiments, the transmitting device and / or the receiving device can be rigidly connected to the imaging device through some fixing components. For example, in some scenarios, the fixing component can be a pan-tilt head, which can be fixed on the imaging device, and the transmitting device and / or the receiving device can be fixed on the pan-tilt head.

[0033] During the process of the imaging device shooting video, the transmitting device can obtain the video data collected by the imaging device and send the video data to the receiving device, and the receiving device can obtain the attitude data collected by the attitude measurement unit and the video data sent by the transmitting device. Due to the rigid connection between the transmitting device and / or the receiving device and the imaging device, the pose data collected by the attitude measurement unit is the pose data of the imaging device. Furthermore, the video data can be subjected to image stabilization processing based on the attitude data and then output.

[0034] For example, the receiving device can directly display the stabilized video data obtained by the image stabilization processing to the user, or the receiving device can also send the stabilized video data to other devices for other devices to display to the user.

[0035] When performing image stabilization processing on the video data using the attitude data, the attitude transformation parameters corresponding to each original video frame in the video data can be determined (for example, the attitude transformation parameters of the original video frame relative to the previous frame), and then the original video frame can be subjected to image stabilization processing based on the attitude transformation parameters to obtain the corresponding stabilized video frame.

[0036] In some embodiments, the transmitting device is provided with an attitude measurement unit and is rigidly connected to the imaging device. The transmitting device is further configured to obtain the attitude data collected by the attitude measurement unit and send it to the receiving device. For example, as Figure 2As shown, the transmitting device is rigidly connected to the imaging device. An IMU is provided in the transmitting device. While the transmitting device sends the video data collected by the imaging device to the receiving device, it can also send the attitude data collected by its own IMU to the receiving device, so that the receiving device can use the attitude data to perform image stabilization processing on the video data.

[0037] In some embodiments, the receiving device is provided with an attitude measurement unit and is rigidly connected to the imaging device. Then the receiving device can directly obtain the attitude data from the attitude measurement unit provided in itself and use the attitude data to perform image stabilization processing on the video data. For example, Figure 3 As shown, both the transmitting device and the receiving device are rigidly connected to the imaging device, and an IMU is provided in the receiving device. The transmitting device can send the video data collected by the imaging device to the receiving device. The receiving device can obtain the attitude data collected by the IMU and then use the attitude data to perform image stabilization processing on the video data.

[0038] During the process of performing image stabilization processing on the video data, when the video picture shakes, in order to keep the picture stable, appropriate rotation adjustment can be performed on each frame. This adjustment will cause some parts of the video picture after image stabilization processing to exceed the boundary of the original video picture, resulting in black borders. In order to eliminate these black borders, the video picture can usually be cropped to retain the effective content area. In the related art, when cropping the video picture, usually a unified picture cropping ratio is used to crop the video picture. For example, in order to avoid black borders in the video picture, usually a maximum picture cropping ratio that ensures no black borders in the video picture is set. After performing rotation and other processing on each original video frame, the processed video picture is then cropped at this maximum picture cropping ratio to obtain the stabilized video frame after image stabilization processing. For example, Figure 4 As shown, after performing distortion and rotation and other processing on the original video frame, a stabilized image is obtained. In order to avoid black borders, the part outside the rectangular frame can be cropped off, and only the image within the rectangular frame is retained as the stabilized video frame. Although using this cropping method during the image stabilization processing can ensure that the obtained stabilized video frame has no black borders and ensure the image stabilization effect, since each video frame is cropped at the maximum picture cropping ratio, more original pictures cannot be retained, and a large amount of original picture content will be lost.

[0039] To avoid the above problems, in some embodiments, during the process of performing video stabilization processing on each original video frame in the video data, the receiving device may perform rotation and cropping processing on each original video frame based on the pose data to obtain a video-stabilized video frame, where the screen cropping ratio corresponding to each original video frame is positively correlated with the jitter degree of the original video frame. For example, the pose transformation parameters (such as a rotation matrix) between each original video frame and the video-stabilized video frame corresponding to the original video frame may be determined based on the pose data, and each original video frame may be rotated based on the rotation matrix. Moreover, the jitter degree of each original video frame may be determined based on the pose data, and the screen cropping ratio of each original video frame may be flexibly adjusted based on the jitter degree of each original video frame. The rotated original video frame may be cropped based on the screen cropping ratio to obtain a video-stabilized video frame. For example, if the screen jitter degree of the original video frame is large, a larger screen cropping ratio may be adopted; if the screen jitter degree of the original video frame is small, a smaller screen cropping ratio may be adopted. By dynamically adjusting the screen cropping ratio corresponding to each original video frame based on the jitter degree of each original video frame, the original screen information can be retained to the greatest extent while ensuring the video stabilization effect.

[0040] Since during the process of performing video stabilization on the original video frame, the pixel coordinates of the pixel points in the original video frame can be converted into three-dimensional coordinate points in three-dimensional space by using the internal parameters of the imaging device, then the three-dimensional coordinate points are rotated, and then the rotated coordinates are projected onto the image by using the internal parameters of the imaging device to obtain the pixel coordinates of the corresponding pixel points of the pixel points in the video-stabilized video. It can be seen that the internal parameters of the imaging device are required during the entire video stabilization process. Therefore, in some embodiments, when the receiving device performs rotation and cropping processing on each original video frame based on the pose data to obtain a video-stabilized video frame, a focal length multiplier may be used to control the screen cropping ratio. The focal length multiplier is used to adjust the focal length in the internal parameters of the imaging device. The larger the focal length multiplier, the larger the focal length in the internal parameters adjusted by the focal length multiplier, and thus the larger the original screen retained in the video-stabilized video frame obtained by performing video stabilization processing on the original video frame using the adjusted internal parameters, that is, the smaller the screen cropping ratio corresponding to the original video frame. Conversely, the smaller the focal length multiplier, the larger the screen cropping ratio finally corresponding to the original video frame. Therefore, for each original video frame, the focal length multiplier corresponding to the original video frame may be determined based on the pose data, and then the focal length in the internal parameters of the imaging device may be adjusted by using the focal length multiplier to obtain the adjusted internal parameters. Furthermore, video stabilization processing may be performed on the original video frame based on the adjusted internal parameters and the pose data to obtain a video-stabilized video frame.

[0041] For example, the internal parameters of the imaging device may be represented by an internal parameter matrix K: Among them, fx is the focal length in the horizontal direction, and fy is the focal length in the vertical direction. cx is the optical center coordinate in the horizontal direction, and cy is the optical center coordinate in the vertical direction. The focal length multiplier can be used to adjust K to control the picture cropping ratio during the image stabilization process. For example, the focal lengths in the adjusted internal parameter matrix are as follows: fx' = fx * fr, fy' = fy * fr. Increasing fr results in less cropping and more of the original picture being retained, while decreasing fr leads to more cropping and less of the original picture being retained. Since the calculation of fr takes into account the result of image stabilization rotation, there will be no black edges in the picture, and the rendered picture is the result after cropping, which is directly saved as a video as the output result.

[0042] In some embodiments, considering that if a focal length multiplier is determined for each original video frame to control the picture cropping ratio of the original video frame, there may be a large fluctuation in the focal length multipliers of adjacent frames, resulting in a large fluctuation in the picture cropping ratio, thereby affecting the overall visual effect of the video. To improve the overall visual effect of the video, after obtaining the focal length multiplier corresponding to each original video frame in the video data, a focal length multiplier sequence composed of the focal length multipliers respectively corresponding to each original video frame in the video data can be obtained, and then the focal length multiplier sequence can be smoothed to reduce the problem of large fluctuations in the focal length multipliers of adjacent frames.

[0043] In some embodiments, after obtaining the focal length multiplier sequence, a window with a preset length can be moved on the focal length multiplier sequence to traverse the focal length multiplier sequence. During the traversal, the focal length multipliers within each window can be subjected to a first smoothing process. Among them, the smoothed value of the focal length multipliers within each window after the first smoothing process is the minimum focal length multiplier within the window. Then, a second smoothing process can be performed on multiple focal length multipliers at the junctions of different windows in the focal length multiplier sequence after the first smoothing process to obtain a smoothed focal length multiplier sequence, so as to adjust the internal parameters using the focal length multipliers in the smoothed focal length multiplier sequence.

[0044] Generally, for a series of consecutive video frames within a short period of time, it is usually required that the video images be generally consistent. If the cropping ratios of the images of these video frames are different, there may be a problem that the content of the video image suddenly becomes larger or smaller within a short period of time, affecting the display effect. To avoid the above problem, after determining the focal length multiplier for each video frame based on the pose data and obtaining the focal length multiplier sequence, a sliding window can be used to slide within the focal length multiplier sequence. For each sliding window, the minimum focal length multiplier within the sliding window can be determined, and then all the focal length multipliers within the sliding window can be replaced with the minimum focal length multiplier. That is, for the focal length multipliers of multiple consecutive video frames within a short period of time, the minimum focal length multiplier can be taken as the focal length multiplier of these video frames, that is, the multiple consecutive video frames can be cropped uniformly according to the largest cropping ratio, so as to ensure that the cropping ratios of the images of these video frames are consistent, that is, the sizes of the objects in the images are the same, and the image is kept stable. In addition, by taking the minimum focal length multiplier, that is, cropping multiple consecutive video frames according to the largest image cropping ratio, it can be ensured that there will be no black edges in each cropped video frame. Among them, the length of the sliding window can be set according to actual needs. For example, if it is desired that the image ratios of 5 consecutive video frames be generally consistent, the length of the sliding window can be set to 5.

[0045] After the above smoothing process, it can be ensured that the cropping ratios of a series of consecutive video frames within a short period of time are consistent, and the image is kept stable within a short period of time. However, for the focal length multipliers at the junction of different windows, there may still be a problem of drastic changes. To avoid the above problem, a second smoothing process can be further performed on multiple focal length multipliers at the junction of different windows in the focal length multiplier sequence after the first smoothing process. For example, the second smoothing process can be Gaussian filtering, mean filtering, etc., so as to obtain a smoothed focal length multiplier sequence. Then, each focal length multiplier in the smoothed focal length multiplier sequence can be used as the focal length multiplier corresponding to each original video frame, and then the focal length in the internal parameters of the imaging device can be adjusted using the focal length multiplier, so as to perform image stabilization processing on the original video frame using the adjusted internal parameters.

[0046] In some embodiments, when the receiving device determines the focal length multiplier corresponding to the original video frame based on the attitude data, it may first determine the attitude transformation parameter between the original video frame and the stabilized video frame corresponding to the original video frame based on the attitude data. Among them, the attitude transformation parameter can describe the change in the pose of the imaging device when collecting the original video frame relative to the pose of the imaging device when collecting the reference frame (for example, the previous frame or multiple frames of the original video frame). To reduce the calculation amount and improve the processing efficiency, the stabilized video frame can be divided into multiple grid points and processed in units of grid points (instead of pixel points), significantly reducing the amount of data to be processed. For example, the pixel coordinates corresponding to each grid point on the original video frame can be determined based on the attitude transformation parameter. Then, the target pixel coordinates can be determined from the set of pixel coordinates corresponding to each of the multiple grid points. Among them, the target pixel coordinates are the closest to the center point of the stabilized video frame, and the target pixel coordinates do not exceed the boundary of the pixel coordinates of the original video frame, that is, the pixel points on the target pixel coordinates are valid pixels in the original video frame. Then, the focal length multiplier can be determined based on the pixel distance between the target pixel coordinates and the center point, and the resolution of the original video frame.

[0047] For example, assuming that the pixel coordinates of a certain grid point in the stabilized video frame are (x0, y0), the corresponding pixel coordinates (x1, y1) of the grid point in the original video frame can be determined based on the internal parameters of the imaging device and the attitude transformation parameter (such as the rotation matrix) between the original video frame and the corresponding stabilized video frame. Among them, considering the rotation processing, some grid points may have corresponding pixel coordinates outside the boundary of the original video frame. For the corresponding pixel coordinates of the grid points, it can be determined whether the pixel points on the corresponding pixel coordinates are valid points in the original video frame based on the pixel coordinates. A valid point means that the pixel coordinates of the pixel point are within the boundary of the pixel coordinates of the original video frame, that is, the corresponding pixel value can be collected from the original pixel coordinates. Then, the pixel coordinates that are the closest to the center point of the original video frame and are located in the valid pixel area of the original video frame can be determined from the pixel coordinates corresponding to each of the multiple grid points, that is, the above-mentioned target pixel coordinates. And the focal length multiplier fr is calculated based on the distance between the above-mentioned target pixel coordinates and the center point and the resolution of the original video frame. Among them, the focal length multiplier is negatively correlated with the width and height of the original video frame and positively correlated with the above distance. For example, assuming that the distance between the target pixel point and the center point is r, the focal length multiplier can be calculated based on the following method:

[0048] fr = min(2 * r / width, 2 * r / height)

[0049] where width is the width of the original video frame, height is the height of the original video frame, and r is the distance between the target pixel point and the center point.

[0050] To obtain an accurate r value, the grid can be refined near the pixel point, and the point closest to the center point can be searched again. Performing this operation 3 to 4 times can obtain an accurate r value.

[0051] Taking offline image stabilization as an example, assuming the attitude measurement unit is an IMU, after obtaining the attitude data collected by the IMU, the attitude data can be integrated to obtain the rotation matrices corresponding to different time points, that is, the cumulative rotation matrix sequence R1, R2, …, Rn. To ensure the stable change of the video image, the cumulative rotation sequence R1, R2, …, Rn can be smoothed to obtain the smoothed cumulative rotation sequence R1′, R2′, …, Rn′. For each original video frame in the video data, the following operations can be performed to perform image stabilization processing on it:

[0052] (1) Find the corresponding IMU data

[0053] Assume that the current frame being processed is the i-th frame with a timestamp of t. Find the two time points t1 and t2 closest to t in the IMU's cumulative rotation sequence, satisfying t1 ≤ t ≤ t2.

[0054] (2) Interpolate to calculate the rotation matrix

[0055] Use the slerp interpolation algorithm of quaternions to calculate the rotation matrix Ra of the i-th frame, Ra = slerp(Rt1, Rt2, (t - t1) / (t2 - t1)) (the rotation matrix is first converted to a quaternion, then interpolated, and then converted back to a rotation matrix)

[0056] (3) Determine the smoothed rotation matrix

[0057] Obtain the smoothed rotation Rs corresponding to the i-th frame from the smoothed cumulative rotation sequence. The rotation matrix for rotating the stabilized video frame to the original video frame: Rd = Ra * Rs -1

[0058] (4) Convert the image coordinate system to the camera's three-dimensional coordinate system and rotate

[0059] (sx, sy, sz) = Rd * Kd -1 *(dx, dy, 1.0)

[0060] (dx, dy) are the pixel coordinates on the stabilized video frame without distortion (if the output field of view is very large, it can also be with distortion, and after passing through Kd -1 it is de-distorted and then the rotation Rd is performed). Since the output resolution may be different, the internal parameter matrix Kd changes based on the camera's calibrated internal parameter K:

[0061] For example, the internal parameter matrix K of the camera can be expressed as:

[0062] For Kd, cx and cy in K can be adjusted based on the resolution of the output stabilized video frame to obtain Kd.

[0063] Among them,

[0064] Kd[cx] = out_width * 0.5; / / Image center change (output resolution change)

[0065] Kd[cy] = out_height * 0.5; / / Image center change (output resolution change)

[0066] (5) Add distortion

[0067] Finally, add distortion to the rotated coordinates to obtain the coordinates before stabilization (on the original video frame):

[0068] (sx′, sy′) = distort(sx, sy, sz)

[0069] (6) Project onto the image coordinate space

[0070] Apply the camera internal parameter Ks to convert the coordinates after adding distortion into image coordinates:

[0071] (spx, spy = Ks * (sx′, sy′, 1.0), where Ks is based on the calibrated camera internal parameters and is adjusted according to the focal length multiplier:

[0072] For example, the internal parameter matrix K of the camera can be expressed as:

[0073] For Ks, fx and fy in K can be adjusted based on the resolution of the output stabilized video frame to obtain Ks.

[0074] Among them,

[0075] Ks[fx] = fr * fx; / / Focal length multiplied by a coefficient (focal length multiplier fr)

[0076] Ks[fy] = fr * fy; / / Focal length multiplied by a coefficient (focal length multiplier fr)

[0077] For each pixel in the stabilized video frame, the position of the corresponding pixel point in the original video frame can be determined in the above manner, and then the stabilized video frame can be obtained by interpolation on the original video frame.

[0078] For online video stabilization, the overall video stabilization method is roughly the same as the above process, except that there are slight differences in the smooth calculation method of the rotation matrix. Online video stabilization can only be closest to the smoothed rotation position of the previous frame to the greatest extent under the limitation of the cropping ratio. The specific operation is to linearly search for the maximum scale factor that satisfies the cropping ratio limitation on the tangent space of the manifold space formed by rotation to approach the rotation smoothed in the previous frame. Generally, it is required that the resolution of the video data for video stabilization processing is the target resolution. Since the video frame is cropped during the video stabilization process, in order to ensure that the resolution of the finally output video frame is the target resolution, in some embodiments, after the receiving device performs cropping processing on each video frame based on the attitude data, it can perform enlargement (or reduction) processing on the cropped video frame to obtain a video stabilization frame with the target resolution.

[0079] Since the attitude data includes the attitude change of the imaging device at different times, and the video data includes the original video frames collected by the imaging device at different times, when using the attitude data to perform video stabilization processing on the video data, the pose change of the imaging device when collecting each original video frame can be determined, and then the original video frame can be stabilized based on the pose change. Therefore, the attitude data and the video data can be matched in time to determine the attitude data corresponding to each original video frame. Since the video data and the attitude data come from two different devices, generally, there is a certain deviation in the clocks of different devices, so there is also a certain deviation between the timestamps of the video data and the timestamps of the attitude data. Therefore, when matching the original video frames and the attitude data, the deviation between the timestamp of the video data and the timestamp of the attitude data can be determined first, and the timestamps of the video data and the timestamps of the attitude data can be synchronized based on this deviation. Based on the synchronized timestamps of the video data and the synchronized timestamps of the attitude data, the attitude transformation parameters corresponding to each original video frame in the video data can be determined. Among them, the attitude transformation parameters can be determined based on the attitude data. For example, assuming that the attitude data is the data collected by the IMU, the IMU data can be integrated to obtain the attitude transformation parameters corresponding to each original video frame. Then, the original video frame can be stabilized based on the attitude transformation parameters corresponding to each original video frame to obtain a stabilized video frame.

[0080] In some embodiments, if real-time image stabilization processing is performed on the video data collected by the imaging device, the video data of 1-2 minutes before formal shooting and the attitude data collected by the attitude measurement unit can be used to determine the deviation of the time stamps. In some embodiments, if offline image stabilization processing (i.e., post-production) is performed on the video data collected by the imaging device, the deviation between the time stamps can be determined using the entire recorded video data and the attitude data, and then the image stabilization operation can be performed. Considering that when determining the deviation between the time stamps, if the video data of the target to be photographed collected by the imaging device is directly used to determine the above deviation, there may be a problem that there are few feature points in the video frames of the target to be photographed, resulting in the inability to accurately extract the feature points and determine the above deviation. Therefore, in some embodiments, in order to more accurately determine the above deviation, before using the imaging device to collect video data, the deviation can be calibrated using a calibration object. Wherein, the calibration object may include a plurality of feature points with known relative position relationships. For example, the calibration object may be a checkerboard, a dot matrix, etc.

[0081] During the calibration process, while jittering the imaging device, the imaging device can be used to collect calibration video data of the calibration object. At the same time, the attitude measurement unit on the transmitting device or the receiving device can synchronously collect calibration attitude data. Then, the calibration video data collected by the imaging device for the calibration object and the calibration attitude data synchronously collected by the attitude measurement unit can be obtained, and feature points are extracted from each video frame in the calibration video data to extract the feature points in each video frame, and the feature points in different video frames are matched to obtain matching point pairs. Then, the first pose transformation parameters when the imaging device collects any two video frames in the calibration video data can be determined based on the matching point pairs, and a first pose transformation parameter sequence can be obtained. Since the matched feature point pairs are the images of the same point in the three-dimensional space collected by the imaging device from different perspectives, the pose transformation parameters when the imaging device collects different video frames can be solved based on the matching point pairs.

[0082] At the same time, the second pose transformation parameters at different time points during the process of the imaging device collecting the calibration video data can be determined based on the calibration attitude data, and a second pose transformation parameter sequence can be obtained. Then, the first pose transformation parameter sequence and the second position transformation parameter sequence can be matched, and based on the matching result, the time stamps of each video frame in the calibration video data and the time stamps of each second position transformation parameter in the second pose transformation parameter sequence, the above deviation can be determined.

[0083] For example, assume that the video frames captured by the imaging device at t1, t2, t3, t4... are video frame 1, video frame 2, video frame 3, video frame 4..., and the sequence of the first pose transformation parameters between adjacent video frames is R1, R2, R3.... The sequence of the second pose transformation parameters determined based on the pose data collected by the pose measurement unit is r1, r2, r3, r4..., and the corresponding timestamps are T1, T2, T3, T4.... Assume that the sampling frequencies of the two types of data are the same. Then, the sequence of the first pose transformation parameters and the sequence of the second pose transformation parameters can be matched to determine which timestamp in the sequence of t1 and T1, T2, T3, T4... is close, and then the deviation can be determined based on the matching result and the respective corresponding timestamps.

[0084] In some embodiments, if the sampling frequencies of the video frames and the pose data are inconsistent, the sampling frequencies of the two can also be unified by upsampling or downsampling, and then the matching can be performed.

[0085] In some embodiments, the calibration object includes a checkerboard, the feature points include the corner points of the checkerboard, and the receiving device is further configured to determine the internal parameters and distortion parameters of the imaging device based on the above-mentioned matching point pairs. Since the relative positional relationship of the feature points is known, the internal parameters of the imaging device can also be solved based on the relative pose relationship of the feature points. For example, if the internal parameters of the imaging device are unknown, during the process of calibrating the deviation between the timestamps of the video data captured by the imaging device and the timestamps of the pose data collected by the pose unit, the internal parameters of the imaging device can be calibrated synchronously.

[0086] By using the calibration object to calibrate the deviation between the timestamps, and the internal parameter K and distortion parameters (k1, k2, k3, k4) of the imaging device simultaneously, the accuracy of the calibrated timestamp deviation can be improved, and the calibration efficiency can also be improved.

[0087] For example, in order to obtain the best image stabilization effect, the internal parameters K of the camera and the distortion parameters coff (coff = [k1, k2, k3, k4]) need to be calibrated in advance. Each frame of the camera video has a timestamp (for videos decoded in formats such as mp4, each frame of the video has a timestamp. In the video transmission system, the timestamp of each frame of the video is the time recorded by the video transmission recording system closest to the camera and is passed to the receiver). Each record of the IMU data also has a timestamp, and there is a time deviation timestamp_offset between the two. The calibration of the internal parameters K and the distortion parameters coff only requires recording the checkerboard video from multiple angles to be calibrated. The timestamp deviation (timestamp_offset) requires the video and the IMU data. Here, when shooting the checkerboard video, the IMU data is also recorded. Then, the checkerboard corner points can be detected frame by frame to obtain a set of frame-by-frame matching points, and the rotation matrix can be calculated using the essential matrix method (opencv function), and the Euler angles are converted to obtain the rotation angle sequences of the x, y, and z axes, which are matched with the IMU data (the rotation angles of the x, y, and z axes) to obtain the timestamp deviation (timestamp_offset).

[0088] Therefore, K, coff, and timestamp_offset can be combined to perform calibration.

[0089] For online image stabilization, after calibration once, the subsequent video image stabilization directly uses timestamp_offset. For offline image stabilization, the calibrated timestamp deviation timestamp_offset can be directly used, or the recorded video can be used to estimate the set of matching points between video frames using the optical flow method, and then the timestamp deviation can be estimated.

[0090] In addition, the embodiment of the present application also provides a video image stabilization method. This video image stabilization method can be applied to the receiving device in the above wireless transmission system. Of course, it can also be applied to other electronic devices with image stabilization functions, such as user mobile phones, computers, cloud servers, etc. These electronic devices can obtain the video data collected by the imaging device and the attitude data collected by the attitude measurement unit on other devices rigidly connected to the imaging device, and use the attitude data to perform image stabilization processing on the video data. As Figure 5 shown, the video image stabilization method may include the following steps:

[0091] S502. Obtain the video data collected by the imaging device and obtain the attitude data collected by the attitude measurement unit provided on the target device rigidly connected to the imaging device;

[0092] S504. Perform image stabilization processing on the video data based on the attitude data and output it.

[0093] In some embodiments, performing image stabilization processing on the video data based on the pose data includes:

[0094] For each original video frame in the video data, performing rotation and cropping processing on each original video frame based on the pose data to obtain a stabilized video frame, wherein the screen cropping ratio corresponding to each original video frame is positively correlated with the jitter degree of the original video frame.

[0095] In some embodiments, performing rotation and cropping processing on each original video frame based on the pose data to obtain a stabilized video frame includes:

[0096] For each original video frame, determining a focal length multiplier corresponding to the original video frame based on the pose data, wherein the focal length multiplier is used to control the screen cropping ratio, and the smaller the focal length multiplier, the larger the screen cropping ratio;

[0097] Adjusting the focal length in the internal parameters of the imaging device based on the focal length multiplier to obtain adjusted internal parameters;

[0098] Performing rotation and cropping processing on the original video frame based on the adjusted internal parameters and the pose data to obtain a stabilized video frame.

[0099] In some embodiments, after determining the focal length multiplier corresponding to each original video frame based on the pose data for each original video frame, it further includes:

[0100] Obtaining a focal length multiplier sequence, which is composed of the focal length multipliers corresponding to each original video frame in the video data;

[0101] Moving a window with a preset length on the focal length multiplier sequence to traverse the focal length multiplier sequence. During the traversal, performing first smoothing processing on the focal length multipliers within each window, wherein the smoothed value of the focal length multipliers within each window after the first smoothing processing is the minimum focal length multiplier within the window;

[0102] Performing second smoothing processing on multiple focal length multipliers at the junctions of different windows in the focal length multiplier sequence after the first smoothing processing to obtain a smoothed focal length multiplier sequence, so as to adjust the focal length in the internal parameters using the focal length multipliers in the smoothed focal length multiplier sequence.

[0103] In some embodiments, determining the focal length multiplier corresponding to the original video frame based on the pose data includes:

[0104] Determining the pose transformation parameters between the original video frame and the stabilized video frame corresponding to the original video frame based on the pose data;

[0105] Divide the stabilized video frame into a plurality of grid points, and determine the pixel coordinates corresponding to each grid point in the original video frame based on the pose transformation parameters;

[0106] Determine target pixel coordinates from the set of pixel coordinates corresponding to each of the plurality of grid points, where the target pixel coordinates are the closest to the center point of the original video frame and the target pixel coordinates do not exceed the boundary of the pixel coordinates of the original video frame;

[0107] Determine the focal length multiplier based on the pixel distance between the target pixel coordinates and the center point of the original video frame, and the resolution of the original video frame.

[0108] In some embodiments, both the video data and the pose data carry timestamps, and the receiving device is used to perform stabilized image processing on the video data based on the pose data. Specifically, it is used to:

[0109] Determine the deviation between the timestamp of the video data and the timestamp of the pose data;

[0110] Synchronize the timestamp of the video data and the timestamp of the pose data based on the deviation;

[0111] Based on the timestamp of the video data after synchronization processing and the timestamp of the pose data after synchronization processing, determine the pose transformation parameters corresponding to each original video frame in the video data, where the pose transformation parameters are determined based on the pose data;

[0112] Perform stabilized image processing on the original video frame based on the pose transformation parameters corresponding to each original video frame to obtain a stabilized video frame.

[0113] In some embodiments, the deviation is determined based on the following method:

[0114] Obtain the calibration video data collected by the imaging device for the calibration object, and the calibration pose data synchronously collected by the pose measurement unit during the process of the imaging device collecting the calibration video data, where the calibration object includes a plurality of feature points with known relative position relationships;

[0115] Extract the feature points of each video frame in the calibration video data, and match the feature points in different video frames to obtain matching point pairs;

[0116] Determine the first pose transformation parameter when the imaging device collects two adjacent video frames in the calibration video data based on the matching point pairs to obtain a first pose transformation parameter sequence;

[0117] Determine a second pose transformation parameter sequence during the process of the camera device collecting the calibration video data based on the marked pose data;

[0118] Match the first pose transformation parameter sequence and the second position transformation parameter sequence, and determine the deviation based on the matching result, the timestamp of the calibration video data, and the timestamp of the pose data.

[0119] In some embodiments, the calibration object includes a checkerboard, the feature points include the corner points of the checkerboard, and the receiving device is further configured to determine the internal parameters and distortion parameters of the camera device based on the matching point pairs.

[0120] Among them, the specific details of this video stabilization method can refer to the descriptions of the embodiments in the above wireless transmission system, and will not be elaborated here.

[0121] The video stabilization method provided by the embodiments of the present application can use the calibration error to calibrate the internal parameters, distortion parameters of the camera device, and the deviation between the timestamp of the video frame collected by the camera device and the timestamp of the pose data collected by the attitude measurement unit at the same time, which can improve the calibration efficiency and the accuracy of the calibration result. At the same time, the focal length multiplier can be determined based on the jitter degree of each original video frame, and the focal length multiplier is used to control the picture cropping ratio, so that the picture cropping ratio can be dynamically adjusted based on the jitter degree of the original video frame, that is, the stabilization effect can be guaranteed and the original picture information can be retained to the greatest extent. Among them, for online stabilization and offline stabilization, this method can be used to determine the picture cropping ratio to render the stabilized video frame.

[0122] Among them, the solutions of the above embodiments can be freely combined to obtain new solutions without conflict. Due to space limitations, they will not be listed one by one here.

[0123] In addition, the embodiments of the present application further provide a computer program product, the computer program product includes a computer program, and when the computer program is executed, it implements the method mentioned in any of the above embodiments.

[0124] In addition, the embodiments of the present application further provide an electronic device, as Figure 6 shown, the electronic device 60 includes a processor 61, a memory 62, and a computer program stored in the memory 62 and executable by the processor 61. When the processor 61 executes the computer program, it implements the method mentioned in the first aspect above.

[0125] Correspondingly, the embodiments of the present application further provide a computer storage medium, and a program is stored in the storage medium. When the program is executed by a processor, it implements the method in any of the above embodiments.

[0126] Embodiments of the present application may be in the form of a computer program product implemented on one or more storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disks or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device.

[0127] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0128] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article, or device comprising the element.

[0129] The methods and devices provided by the embodiments of the present application have been introduced in detail above. Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of the present application should not be construed as a limitation to the present application.

Claims

1. A wireless video transmission system, characterized in that, The wireless video transmission system includes a transmitting device and a receiving device. At least one of the transmitting device and the receiving device is provided with an attitude measurement unit and is rigidly connected to the imaging device; The transmitting device is used to obtain the video data collected by the imaging device and send the video data to the receiving device; The receiving device is used to obtain the attitude data collected by the attitude measurement unit, perform image stabilization processing on the video data based on the attitude data, and output it.

2. The wireless video transmission system according to claim 1, wherein The transmitting device is provided with an attitude measurement unit and is rigidly connected to the imaging device. The transmitting device is also used to obtain the attitude data collected by the attitude measurement unit and send it to the receiving device; or The receiving device is provided with an attitude measurement unit and is rigidly connected to the imaging device. The receiving device is used to obtain the attitude data from the attitude measurement unit provided by itself.

3. The wireless video transmission system according to claim 1, wherein The receiving device is used to perform image stabilization processing on the video data based on the attitude data. Specifically, it is used for: For each original video frame in the video data, perform rotation and cropping processing on each original video frame based on the attitude data to obtain a stabilized video frame. Among them, the picture cropping ratio corresponding to each original video frame is positively correlated with the jitter degree of the original video frame.

4. The wireless video transmission system according to claim 3, wherein The receiving device is used to perform rotation and cropping processing on each original video frame based on the attitude data to obtain a stabilized video frame. Specifically, it is used for: For each original video frame, determine the focal length multiplier corresponding to the original video frame based on the attitude data. Among them, the focal length multiplier is used to control the picture cropping ratio, and the smaller the focal length multiplier, the larger the picture cropping ratio; Adjust the focal length in the internal parameters of the imaging device based on the focal length multiplier to obtain the adjusted internal parameters; Perform rotation and cropping processing on the original video frame based on the adjusted internal parameters and the attitude data to obtain a stabilized video frame.

5. The wireless video transmission system according to claim 4, wherein After the receiving device determines the focal length multiplier corresponding to the original video frame based on the attitude data for each original video frame, it is also used for: Obtain a sequence of focal length multipliers, which is composed of the focal length multipliers corresponding to each original video frame in the video data; Use a window with a preset length to move on the sequence of focal length multipliers to traverse the sequence of focal length multipliers. During the traversal, perform first smoothing processing on the focal length multipliers within each window. Among them, the smoothed value of the focal length multipliers within each window after the first smoothing processing is the minimum focal length multiplier within the window; Perform second smoothing processing on multiple focal length multipliers at the junction of different windows in the sequence of focal length multipliers after the first smoothing processing to obtain a smoothed sequence of focal length multipliers, so as to adjust the focal length in the internal parameters using the focal length multipliers in the smoothed sequence of focal length multipliers.

6. The wireless video transmission system according to claim 5, wherein, The receiving device is used to determine the focal length multiplier corresponding to the original video frame based on the attitude data. Specifically, it is used for: Determine the attitude transformation parameters between the original video frame and the stabilized video frame corresponding to the original video frame based on the attitude data; Divide the stabilized video frame into multiple grid points, and determine the pixel coordinates corresponding to each grid point in the original video frame based on the pose transformation parameters; Determine target pixel coordinates from the sets of pixel coordinates corresponding to the multiple grid points, where the target pixel coordinates are the closest to the center point of the original video frame and the target pixel coordinates do not exceed the boundaries of the pixel coordinates of the original video frame; Determine the focal length multiplier based on the pixel distance between the target pixel coordinates and the center point of the original video frame, and the resolution of the original video frame.

7. The wireless video transmission system according to claim 1, wherein Both the video data and the pose data carry timestamps, and the receiving device is used to perform stabilized image processing on the video data based on the pose data. Specifically, it is used to: Determine the deviation between the timestamp of the video data and the timestamp of the pose data; Synchronize the timestamp of the video data and the timestamp of the pose data based on the deviation; Based on the timestamp of the video data after synchronization processing and the timestamp of the pose data after synchronization processing, determine the pose transformation parameters corresponding to each original video frame in the video data, where the pose transformation parameters are determined based on the pose data; Perform stabilized image processing on the original video frame based on the pose transformation parameters corresponding to each original video frame to obtain a stabilized video frame.

8. The wireless video transmission system according to claim 7, wherein The deviation is determined based on the following method: Obtain the calibration video data collected by the imaging device for the calibration object, and the calibration pose data synchronously collected by the pose measurement unit during the process of the imaging device collecting the calibration video data, where the calibration object includes multiple feature points with known relative position relationships; Extract the feature points of each video frame in the calibration video data, and match the feature points in different video frames to obtain matching point pairs; Determine the first pose transformation parameters when the imaging device collects two adjacent video frames in the calibration video data based on the matching point pairs to obtain a sequence of first pose transformation parameters; Determine a sequence of second pose transformation parameters during the process of the imaging device collecting the calibration video data based on the calibration pose data; Match the sequence of first pose transformation parameters and the sequence of second pose transformation parameters, and determine the deviation based on the matching result, the timestamp of the calibration video data, and the timestamp of the calibration pose data.

9. The wireless video transmission system according to claim 8, wherein, The calibration object includes a checkerboard, the feature points include the corner points of the checkerboard, and the receiving device is also used to determine the internal parameters and distortion parameters of the imaging device based on the matching point pairs.

10. A video stabilization method, characterized in that, The method includes: Obtain the video data collected by the imaging device, and obtain the pose data collected by the pose measurement unit provided on the target device rigidly connected to the imaging device; Perform stabilized image processing on the video data based on the pose data and output it.

11. The video stabilization method according to claim 10, characterized in that, The performing stabilized image processing on the video data based on the pose data includes: For each original video frame in the video data, perform rotation and cropping processing on each original video frame based on the pose data to obtain a stabilized video frame, where the aspect ratio of the picture cropped for each original video frame is positively correlated with the degree of jitter of the original video frame.

12. The method according to claim 11, wherein The performing rotation and cropping processing on each original video frame based on the pose data to obtain a stabilized video frame includes: For each original video frame, determine the focal length multiplier corresponding to the original video frame based on the pose data, where the focal length multiplier is used to control the aspect ratio of the picture cropped, and the smaller the focal length multiplier, the larger the aspect ratio of the picture cropped; Adjust the focal length in the internal parameters of the imaging device based on the focal length multiplier to obtain the adjusted internal parameters; Perform rotation and cropping processing on the original video frame based on the adjusted internal parameters and the pose data to obtain a stabilized video frame.

13. The method according to claim 12, characterized in that, After determining the focal length multiplier corresponding to each original video frame based on the pose data for each original video frame, it further includes: Obtain a sequence of focal length multipliers, where the sequence of focal length multipliers is composed of the focal length multipliers corresponding to each original video frame in the video data; Use a window with a preset length to move on the sequence of focal length multipliers to traverse the sequence of focal length multipliers. During the traversal, perform first smoothing processing on the focal length multipliers within each window, where the smoothed value of the focal length multipliers within each window after the first smoothing processing is the minimum focal length multiplier within the window; Perform second smoothing processing on the multiple focal length multipliers at the junction of different windows in the sequence of focal length multipliers after the first smoothing processing to obtain a smoothed sequence of focal length multipliers, so as to adjust the focal length in the internal parameters using the focal length multipliers in the smoothed sequence of focal length multipliers.

14. The method according to claim 12, wherein The determining the focal length multiplier corresponding to the original video frame based on the pose data includes: Determine the pose transformation parameters between the original video frame and the stabilized video frame corresponding to the original video frame based on the pose data; Divide the stabilized video frame into multiple grid points, and determine the pixel coordinates corresponding to each grid point in the original video frame based on the pose transformation parameters; Determine the target pixel coordinates from the set of pixel coordinates corresponding to each of the multiple grid points, where the target pixel coordinates are the closest to the center point of the original video frame and the target pixel coordinates do not exceed the boundaries of the pixel coordinates of the original video frame; Determine the focal length multiplier based on the pixel distance between the target pixel coordinates and the center point of the original video frame, and the resolution of the original video frame.

15. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed, it implements the method according to any one of claims 10 - 14.

Images