Image electronic stability augmentation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an electronic stability augmentation method and device for an image, equipment and a storage medium, and the method comprises the steps: obtaining a frame queue, calculating the pixel translation amount of two adjacent frames of images in the frame queue, obtaining a motion parameter set, calculating the number of low-pass filtering frames, determining the number of to-be-extracted historical motion parameters according to the number of low-pass filtering frames and a preset rule, and carrying out the electronic stability augmentation of the image. Inputting the historical motion parameter and the current motion parameter into a preset observation model to obtain a current motion parameter observed quantity, and inputting a motion parameter optimal estimated value of a previous frame, which is calculated historically, into a preset prediction model to obtain a motion parameter predicted value of the current frame, the weighted calculation is performed according to the motion parameter predicted value of the current frame and the current motion parameter observed quantity to obtain the motion parameter optimal value of the current frame, and the stability augmentation image is obtained through the preset stability model, so that the stability augmentation can be performed on the low and medium frequency picture jitter, and the picture acquisition effect of the unmanned equipment is enhanced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image electronic stabilization method, device, equipment and storage medium. Background Art

[0002] With the development of intelligent agriculture, unmanned equipment has been widely used. For example, unmanned equipment can be used to fly and mark points on pre-planned routes. However, there is a problem of image shaking during the flight and marking process of agricultural drones.

[0003] In the existing technical solutions, low-pass filtering and Kalman filtering are usually used for electronic stabilization, but the above two solutions mainly solve high-frequency jitters, and the processing effect on medium and low-frequency jitters is not significant. In the process of unmanned equipment marking, in addition to high-frequency jitters during takeoff and hovering, there are also large medium and low-frequency shaking during flight, but the low-pass filtering solution and Kalman filtering solution cannot solve the problem of medium and low-frequency shaking in the unmanned equipment marking scene. Summary of the invention

[0004] The embodiments of the present application provide an image electronic stabilization method, device, equipment and storage medium, which solve the problem of large medium and low frequency image jitters in the unmanned equipment during the flight marking process, stabilize the medium and low frequency image jitters, and enhance the effect of unmanned equipment in collecting images.

[0005] In a first aspect, an embodiment of the present application provides an image electronic stabilization method, comprising:

[0006] Obtain a frame queue, calculate pixel translation amounts of two adjacent frame images in the frame queue, and obtain a set of motion parameters, wherein the frame queue includes multiple frame images taken during the navigation of the unmanned device;

[0007] Calculate the low-pass filtering frame number according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset frame number;

[0008] Determine the number of historical motion parameters to be extracted according to the number of low-pass filter frames and preset rules, and extract the historical motion parameters closest to the current frame from the motion parameter set according to the number of historical motion parameters;

[0009] Inputting historical motion parameters and current motion parameters into a preset observation model to obtain the current motion parameter observation value, and inputting the optimal estimated value of the motion parameter of the previous frame calculated historically into a preset prediction model to obtain the motion parameter prediction value of the current frame;

[0010] Perform weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame;

[0011] The optimal values ​​of the motion parameters are transformed into a matrix to obtain an optimal estimation value matrix, and the optimal estimation value matrix is ​​input into a preset stabilization model to obtain stabilization motion parameters, and image mapping is performed based on the stabilization motion parameters to obtain a stabilized image.

[0012] In a second aspect, an embodiment of the present application provides an image electronic stabilization device, comprising:

[0013] A motion parameter set building module is used to obtain a frame queue, respectively calculate the pixel translation amount of two adjacent frame images in the frame queue, and obtain a motion parameter set. The frame queue includes multiple frame images taken during the navigation of the unmanned equipment;

[0014] A low-pass filter frame number calculation module, used to calculate the low-pass filter frame number according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset frame number;

[0015] A historical motion parameter extraction module is used to determine the number of historical motion parameters to be extracted according to the number of low-pass filter frames and preset rules, and extract the historical motion parameters closest to the current frame from the motion parameter set according to the number of historical motion parameters;

[0016] An observation calculation module is used to input historical motion parameters and current motion parameters into a preset observation model to obtain the current motion parameter observation;

[0017] A prediction value calculation module, used to input the optimal estimated value of the motion parameter of the previous frame calculated historically into a preset prediction model to obtain the predicted value of the motion parameter of the current frame;

[0018] The optimal estimation value calculation module is used to perform weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame;

[0019] The stabilization image mapping module is used to perform matrix conversion on the optimal values ​​of motion parameters to obtain the optimal estimation value matrix, and input the optimal estimation value matrix into a preset stabilization model to obtain the stabilization motion parameters, and perform image mapping based on the stabilization motion parameters to obtain a stabilized image.

[0020] In a third aspect, an embodiment of the present application provides an image electronic stabilization device, including: a memory and one or more processors;

[0021] A memory for storing one or more programs;

[0022] When one or more programs are executed by one or more processors, the one or more processors implement the image electronic stabilization method as described in the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute the image electronic stabilization method as described in the first aspect.

[0024] In the above, by obtaining a frame queue, the pixel shift of two adjacent frames in the frame queue is calculated respectively to obtain a motion parameter set, the frame queue includes multiple frames of images taken during the navigation of the unmanned equipment, the low-pass filter frame number is calculated according to the pixel shift between the current frame and the previous frame in the motion parameter set and the preset frame number, the number of historical motion parameters to be extracted is determined according to the low-pass filter frame number and the preset rule, the historical motion parameters closest to the current frame are extracted from the motion parameter set according to the number of historical motion parameters, the historical motion parameters and the current motion parameters are input into the preset observation model to obtain the current motion parameter observation amount, the optimal estimated value of the motion parameter of the previous frame calculated historically is input into the preset prediction model to obtain the motion parameter prediction value of the current frame, a weighted calculation is performed based on the motion parameter prediction value of the current frame and the current motion parameter observation amount to obtain the optimal value of the motion parameter of the current frame, the optimal value of the motion parameter is input for matrix conversion to obtain the optimal estimated value matrix, and the optimal estimated value matrix is ​​input into the preset stabilization model to obtain the stabilization motion parameters, and image mapping is performed based on the stabilization motion parameters to obtain the stabilized image. It is possible to remove medium and low frequency motions by extracting historical motion parameters from the motion parameter set at intervals, and to have a lower intensity low pass for high frequency motions and a higher intensity low pass for medium and low frequency motions. This not only processes high frequency jitters during the flight of unmanned equipment, but also smoothes low frequency motions, allowing users to quickly identify targets and improving the efficiency of flight marking. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of an image electronic stabilization method provided in an embodiment of the present application;

[0026] Figure 2 This is a schematic diagram of an image electronic stabilization effect provided by an embodiment of the present application;

[0027] Figure 3 is a flow chart of a method for calculating a motion parameter prediction value provided by an embodiment of the present application;

[0028] Figure 4 is a structural schematic diagram of an image electronic stabilization device provided in an embodiment of the present application;

[0029] Figure 5 It is a structural schematic diagram of an image electronic stabilization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0031] Figure 1 A flow chart of an image electronic stabilization method provided by an embodiment of the present application is given. The embodiment of the present application can be applied to scenarios in which unmanned equipment performs agricultural operations such as precise sowing, spraying of pesticides and fertilization in a preset area, wherein the unmanned equipment can be a drone. The image electronic stabilization method can be performed by an image electronic stabilization device, which can be implemented by hardware and / or software and integrated in the image electronic stabilization device.

[0032] The following description is made by taking the image electronic stabilization device executing the image electronic stabilization method as an example. Figure 1 , the image electronic stabilization method comprises:

[0033] Step S101, obtaining a frame queue, respectively calculating the pixel translation amount of two adjacent frame images in the frame queue, and obtaining a motion parameter set, wherein the frame queue includes multiple frame images taken during the navigation of the unmanned equipment.

[0034] In one embodiment, the frame queue may be a queue for storing multiple frames of images, including multiple frames of images captured by the unmanned device during navigation, or multiple frames of images captured by the unmanned device during takeoff or landing. The frames of images in the frame queue are arranged in sequence according to the order in which the images are acquired. The frames of images in the frame queue may be all the frames of images acquired by the unmanned device, or some of the frames of images acquired by the unmanned device. The pixel translation may refer to the change in pixel points from one frame of image to another, wherein the change in pixel points may be the movement of the pixel points in the horizontal and vertical directions, and may also include rotation parameters. The rotation parameter may be the rotation amount of the z-axis in the pixel coordinate axis. The pixel translation may be represented by coordinates or a homography matrix. The homography matrix may be used to describe the transformation relationship of coplanar points from one frame of captured image to another frame of captured image. Exemplarily, the current frame motion parameter matrix H(t) is calculated as:

[0035]

[0036] α z = atan(a 21 ,a 11 )

[0037] where t x and t y are the pixel shift between two adjacent frames of images, α z is the amount of rotation about the z-axis, atan represents arctan, and a11, a12, a21, and a22 represent motions such as rotation and scaling. The motion parameter set may be a set of calculated pixel translations for each two adjacent frames. For example, if the frame queue stores 100 frames of images taken by the drone during flight, the changes in each pixel in the 1st frame and the 2nd frame, the changes in each pixel in the 2nd frame and the 3rd frame, to the changes in each pixel in the 99th frame and the 100th frame, i.e., the pixel translation, are calculated respectively, thereby calculating 50 pixel translations, which are combined into a motion parameter set.

[0038] In one embodiment, before acquiring the frame queue, it also includes: constructing a frame queue based on the current frame and preset queue information, and the preset queue information includes a preset interval time and a preset number of frames.

[0039] The current frame may be the latest frame image collected by the unmanned device. The preset queue information may be used to indicate the frame queue size and the construction rules of the frame queue. If only 10 frames of images can be stored in the frame queue, the size of the frame queue is 10. The preset rules may be used to indicate the rules for extracting frame images when constructing a frame queue, such as extracting one frame of image every preset number of frames from 100 frames of image collected by the unmanned device, or extracting one frame of image every preset time. Therefore, the preset queue information may include a preset interval time and a preset number of frames, which is the size of the frame queue. Exemplarily, if the frame queue stores 100 frames of images taken by the drone during flight, the preset interval time is 1S, the preset number of frames is 10, and one frame of image is extracted every 1S, for extracting 10 frames. Extract the 10 frames of images closest to the current frame. For example, if 10 frames of images are transmitted per second, extract the frame image 1 second before the current time, that is, the 90th frame image, and so on, extract the current 100th frame image, the 90th frame image, the 80th frame image, the 70th frame image, the 60th frame image, the 50th frame image, the 40th frame image, the 30th frame image, the 20th frame image and the 10th frame image respectively. A frame queue is constructed based on the extracted 10 frames, and the pixel translation between the 100th frame image and the 90th frame image, the pixel translation between the 90th frame image and the 80th frame image, the pixel translation between the 80th frame image and the 70th frame image, the pixel translation between the 70th frame image and the 60th frame image, the pixel translation between the 60th frame image and the 50th frame image, the pixel translation between the 50th frame image and the 40th frame image, the pixel translation between the 40th frame image and the 30th frame image, the pixel translation between the 30th frame image and the 20th frame image, and the pixel translation between the 20th frame image and the 10th frame image are calculated respectively, and a motion parameter collection is constructed based on the calculation results.

[0040] Step S102: Calculate the low-pass filtering frame number according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset frame number.

[0041] The number of low-pass filter frames can be the number of frames involved in low-pass filtering. Low-pass filtering is a signal filtering method. The rule of low-pass filtering is that low-frequency signals can pass normally, while high-frequency signals exceeding the set threshold are blocked or weakened. However, the degree of blocking and weakening will vary according to different frequencies and different filtering procedures. The preset number of frames can be the number of image frames in the frame queue.

[0042] In one embodiment, the formula for calculating the number of low-pass filtering frames according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset number of frames includes:

[0043]

[0044]

[0045] The symbol [] represents the rounding function, k is the low-pass filter frame number, t is the current frame, N is the preset frame number, t xy is the pixel shift between two adjacent frames.

[0046] Step S103: determining the number of historical motion parameters to be extracted according to the number of low-pass filtering frames and a preset rule, and extracting the historical motion parameters closest to the current frame from the motion parameter set according to the number of historical motion parameters.

[0047] The preset rule may be a rule for determining the number of historical motion parameters to be extracted based on the number of low-pass filter frames. If the number of low-pass filter frames is k, the preset rule may be to extract k-1 historical motion parameters. The historical motion parameters may be motion parameters in the motion parameter set other than the motion parameters calculated based on the current frame and the previous frame. Exemplarily, if the motion parameter set includes the pixel shift between the 100th frame image and the 90th frame image, the pixel shift between the 90th frame image and the 80th frame image, the pixel shift between the 80th frame image and the 70th frame image, the pixel shift between the 70th frame image and the 60th frame image, the pixel shift between the 60th frame image and the 50th frame image, the pixel shift between the 50th frame image and the 40th frame image, the pixel shift between the 40th frame image and the 30th frame image, the pixel shift between the 30th frame image and the 20th frame image, and the pixel shift between the 20th frame image and the 10th frame image, the pixel shift between the 100th frame image and the 90th frame image are current motion parameters, and the remaining translation parameters are historical motion parameters. If the number of low-pass filtering frames is k=3 and the preset rule is k-1, it can be determined that the number of historical motion parameters to be extracted is 2. Then, the two historical motion parameters closest to the current frame are extracted from the historical motion parameter set, which are the pixel translation amounts of the 90th frame image and the 80th frame image, and the pixel translation amounts of the 80th frame image and the 70th frame image.

[0048] Step S104: input the historical motion parameters and the current motion parameters into a preset observation model to obtain the current motion parameter observation value, and input the optimal estimated value of the motion parameter of the previous frame calculated historically into a preset prediction model to obtain the motion parameter prediction value of the current frame.

[0049] The preset observation model may be a pre-set model for calculating observation values. The current motion parameter observation value may be obtained by inputting the historical motion parameters and the current motion parameters into the preset observation model for low-pass filtering. The current motion parameter observation value may be a sample value of the current motion parameter obtained by measurement. In one embodiment, the sample value of the current motion parameter may be a value directly measured by a camera or sensor on an unmanned device. The optimal estimated value of the motion parameter of the previous frame may be the optimal motion parameter estimated by the previous frame image. Exemplarily, if the current frame is the 100th frame, the motion parameter prediction value of the current frame is the motion parameter prediction value of the 100th frame and the 90th frame, and the motion parameter optimal estimated value of the previous frame is the optimal estimated value of the 90th frame and the 80th frame. The preset prediction model may be a pre-set model that can predict the current motion parameter. The optimal estimated value of the motion parameter of the previous frame calculated historically may be input into the preset prediction model, and the motion parameter prediction value of the current frame may be determined by the optimal motion parameter obtained by the estimated value of the previous frame image. Exemplarily, the motion parameters of the 100th frame and the 90th frame are predicted by the optimal estimation values ​​of the 90th frame and the 80th frame, so as to obtain the predicted values ​​of the motion parameters of the 100th frame and the 90th frame.

[0050] In another embodiment, inputting the historical motion parameters and the current motion parameters into a preset observation model includes: performing cumulative multiplication calculation on the historical motion parameters and the current motion parameters through the preset observation model.

[0051] The preset observation model can be:

[0052]

[0053] Among them, M smooth It represents the cumulative transformation matrix after Kalman filtering, t is the current motion parameter, τ = 1 is the first historical motion parameter extracted, H τ are historical motion parameters.

[0054] The above preset observation model calculates and extracts the result of multiplying the first historical motion parameter to the τth historical motion parameter and the current parameter.

[0055] Step S105: Perform weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame.

[0056] In one embodiment, the observation noise and the processing noise can be predetermined. The processing noise can be determined from the pixels of the image taken by the unmanned device during flight, and the specific value of the processing noise is determined according to the degree to which the pixels satisfy the uniform motion model. The observation noise can be obtained by decomposing the motion parameters by taking two frames of images by a camera on the unmanned device, and the observation noise is determined according to the accuracy of the decomposed motion parameters. The weight of the current motion parameter observation is determined according to the observation noise. The smaller the observation noise, the greater the weight of the previous motion parameter observation, and vice versa. The weight of the motion parameter prediction value of the current frame is determined according to the processing noise. The smaller the processing noise, the greater the weight of the motion parameter prediction value of the current frame.

[0057] Based on the predetermined observation noise and processing noise, corresponding weights are assigned to the motion parameter prediction value of the current frame and the current motion parameter observation value and calculated to obtain the optimal estimated value of the pixel point of the current frame.

[0058] In one embodiment, weighted calculation is performed based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame, including:

[0059] P_(t)=A*P(t-1)A -1 +Q

[0060] Kg=P(t-1)*H T *(H*P(t-1)*H T +R) -1

[0061] M(t)=M_(t)+Kg*(Z(t)-H*M_(t))

[0062] Among them, Kg is the Kalman gain, R is the preset observation noise, Z(t) is the current motion parameter observation, H is the observation matrix, M(t) is the optimal estimate after correction, M_(t) is the motion parameter prediction value, T is the preset interval time, A is the state transfer matrix, P(t-1) is the prediction error at the previous moment t-1, Q is the processing noise, and P_(t) is the prediction covariance at the current moment t.

[0063] Step S106: performing matrix conversion on the optimal values ​​of the motion parameters to obtain an optimal estimation value matrix, and inputting the optimal estimation value matrix into a preset stabilization model to obtain stabilization motion parameters, and performing image mapping based on the stabilization motion parameters to obtain a stabilized image.

[0064] The process of converting the calculated optimal motion parameter value into the optimal estimation value matrix is ​​similar to the above process of determining the current frame motion parameter matrix H(t). The specific process can refer to the above process and will not be repeated in this embodiment.

[0065] The stabilization model can be used to quickly calculate the motion parameters of a frame of image after stabilization. In one embodiment, the preset stabilization model is:

[0066] I stable =warp(I input ,M smooth .(M source ) -1 )

[0067] Among them, I input is the current frame image input, I stable is the stabilization motion parameter, M source is the cumulative transformation matrix of the jittered video before stabilization, M smooth It represents the cumulative transformation matrix after Kalman filtering, M smooth .(M source ) -1 It is represented as an optimal estimate matrix.

[0068] Stabilization motion parameters can be motion parameters corresponding to a certain frame obtained by stabilization calculation for a shaky image. Stabilization image can be a stable image obtained by stabilization processing for a shaky image. Stabilization motion parameters are calculated by inputting the optimal value of motion parameters of the current frame calculated into a preset stabilization model. Figure 2 is a schematic diagram of an image electronic stabilization effect provided by an embodiment of the present application, such as Figure 2 As shown in the figure, the horizontal axis represents the frame number, and the vertical axis represents the motion parameter value. The curve with obvious high-frequency jitter is the motion parameter trajectory of the original input video, and the smooth curve is the stabilization motion parameter trajectory. Pixel value mapping is performed according to the stabilization parameter, that is, the calculated stabilization motion parameter is mapped to the corresponding position of another image according to the preset mapping relationship to obtain a new image, which is the stabilized image.

[0069] In the above, by obtaining a frame queue, the pixel shift of two adjacent frames in the frame queue is calculated respectively to obtain a motion parameter set, the frame queue includes multiple frames of images taken during the navigation of the unmanned equipment, the low-pass filter frame number is calculated according to the pixel shift between the current frame and the previous frame in the motion parameter set and the preset frame number, the number of historical motion parameters to be extracted is determined according to the low-pass filter frame number and the preset rule, the historical motion parameters closest to the current frame are extracted from the motion parameter set according to the number of historical motion parameters, the historical motion parameters and the current motion parameters are input into a preset observation model to obtain the current motion parameter observation amount, the optimal estimated value of the motion parameter of the previous frame calculated historically is input into a preset prediction model to obtain the motion parameter prediction value of the current frame, a weighted calculation is performed based on the motion parameter prediction value of the current frame and the current motion parameter observation amount to obtain the optimal value of the motion parameter of the current frame, the optimal value of the motion parameter is matrix-converted to obtain the optimal estimated value matrix, and the optimal estimated value matrix is ​​input into a preset stabilization model to obtain the stabilization motion parameters, and image mapping is performed based on the stabilization motion parameters to obtain the stabilized image. It is possible to remove medium and low frequency motions by extracting historical motion parameters from the motion parameter set at intervals, and to have a lower intensity low pass for high frequency motions and a higher intensity low pass for medium and low frequency motions. This not only processes high frequency jitters during the flight of unmanned equipment, but also smoothes low frequency motions, allowing users to quickly identify targets and improving the efficiency of flight marking.

[0070] Figure 3 is a flow chart of a motion parameter prediction value calculation method provided by an embodiment of the present application, such as Figure 3 As shown, including:

[0071] Step S301, obtain a frame queue, calculate the pixel translation amount of two adjacent frame images in the frame queue respectively, and obtain a motion parameter set, the frame queue includes multiple frame images taken during the navigation of the unmanned equipment.

[0072] Step S302: Calculate the number of low-pass filtering frames according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset number of frames.

[0073] Step S303: determine the number of historical motion parameters to be extracted according to the number of low-pass filtering frames and preset rules, and extract the historical motion parameters closest to the current frame from the motion parameter set according to the number of historical motion parameters.

[0074] Step S304: input the historical motion parameters and the current motion parameters into a preset observation model to obtain the current motion parameter observation value, and calculate according to the historically calculated optimal estimation value of the motion parameters of the previous frame and the state transfer matrix to obtain the motion parameter prediction value of the current frame.

[0075] The historical motion parameters and the current motion parameters are input into the preset observation model for low-pass filtering to obtain the current motion parameter observation. The state transfer matrix can be a unit matrix. The preset observation model includes the state rotation matrix, and the motion parameter prediction value of the current frame is output by multiplying the state transfer matrix in the prediction model with the motion parameter optimal estimation value output value of the previous frame in the prediction model. The motion parameter prediction value of the current frame can be the predicted motion parameter of the current frame relative to the previous frame.

[0076] In one embodiment, the calculation formula for obtaining the motion parameter prediction value of the current frame based on the historically calculated motion parameter optimal estimation value of the previous frame and the state transfer matrix includes:

[0077] M_(t)=A*M(t-1)

[0078] Among them, A is the state transfer matrix, M_(t) is the predicted value of the motion parameter at the current time t, and M(t-1) is the optimal estimated value of the motion parameter of the previous frame.

[0079] Step S305: Perform weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame.

[0080] Step S306: Perform matrix conversion on the optimal values ​​of the motion parameters to obtain an optimal estimation value matrix, and input the optimal estimation value matrix into a preset stabilization model to obtain stabilization motion parameters, and perform image mapping based on the stabilization motion parameters to obtain a stabilized image.

[0081] As described above, by calculating the motion parameter prediction value of the current frame based on the historically calculated optimal estimation value of the motion parameter of the previous frame and the state transfer matrix, the correlation in the video spatial domain is utilized and the current pixel is predicted using the adjacent pixels in the same frame image, which can effectively remove the video time domain redundancy.

[0082] Figure 4 is a schematic diagram of the structure of an image electronic stabilization device provided in an embodiment of the present application, such as Figure 4 As shown, specifically including:

[0083] The motion parameter set building module 41 is used to obtain a frame queue, respectively calculate the pixel translation amount of two adjacent frame images in the frame queue, and obtain a motion parameter set, wherein the frame queue includes multiple frame images taken during the navigation of the unmanned device;

[0084] A low-pass filter frame number calculation module 42, used to calculate the low-pass filter frame number according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset frame number;

[0085] The historical motion parameter extraction module 43 is used to determine the number of historical motion parameters to be extracted according to the number of low-pass filtering frames and a preset rule, and extract the historical motion parameters closest to the current frame from the motion parameter set according to the number of historical motion parameters;

[0086] The observation calculation module 44 is used to input the historical motion parameters and the current motion parameters into a preset observation model to obtain the current motion parameter observation;

[0087] A prediction value calculation module 45 is used to input the optimal estimated value of the motion parameter of the previous frame calculated historically into a preset prediction model to obtain the predicted value of the motion parameter of the current frame;

[0088] The optimal estimation value calculation module 46 is used to perform weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame;

[0089] The stabilization image mapping module 47 is used to perform matrix conversion on the optimal values ​​of the motion parameters to obtain an optimal estimation value matrix, and input the optimal estimation value matrix into a preset stabilization model to obtain stabilization motion parameters, and perform image mapping based on the stabilization motion parameters to obtain a stabilized image.

[0090] In the above, by obtaining a frame queue, the pixel shift of two adjacent frames in the frame queue is calculated respectively to obtain a motion parameter set, the frame queue includes multiple frames of images taken during the navigation of the unmanned equipment, the low-pass filter frame number is calculated according to the pixel shift between the current frame and the previous frame in the motion parameter set and the preset frame number, the number of historical motion parameters to be extracted is determined according to the low-pass filter frame number and the preset rule, the historical motion parameters closest to the current frame are extracted from the motion parameter set according to the number of historical motion parameters, the historical motion parameters and the current motion parameters are input into a preset observation model to obtain the current motion parameter observation amount, the optimal estimated value of the motion parameter of the previous frame calculated historically is input into a preset prediction model to obtain the motion parameter prediction value of the current frame, a weighted calculation is performed based on the motion parameter prediction value of the current frame and the current motion parameter observation amount to obtain the optimal value of the motion parameter of the current frame, the optimal value of the motion parameter is matrix-converted to obtain the optimal estimated value matrix, and the optimal estimated value matrix is ​​input into a preset stabilization model to obtain the stabilization motion parameters, and image mapping is performed based on the stabilization motion parameters to obtain the stabilized image. It is possible to remove medium and low frequency motions by extracting historical motion parameters from the motion parameter set at intervals, and to have a lower intensity low pass for high frequency motions and a higher intensity low pass for medium and low frequency motions. This not only processes high frequency jitters during the flight of unmanned equipment, but also smoothes low frequency motions, allowing users to quickly identify targets and improving the efficiency of flight marking.

[0091] In one embodiment, the formula for calculating the number of low-pass filtering frames according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset number of frames includes:

[0092]

[0093]

[0094] The symbol [] represents the rounding function, k is the low-pass filter frame number, t is the current frame, N is the preset frame number, t xy is the pixel shift between two adjacent frames.

[0095] In one embodiment, the prediction value calculation module is specifically used to calculate based on the historically calculated optimal estimation value of the motion parameter of the previous frame and the state transfer matrix to obtain the prediction value of the motion parameter of the current frame.

[0096] In one embodiment, the calculation formula for obtaining the motion parameter prediction value of the current frame based on the historically calculated motion parameter optimal estimation value of the previous frame and the state transfer matrix includes:

[0097] M_(t)=A*M(t-1)

[0098] Among them, A is the state transfer matrix, M_(t) is the predicted value of the motion parameter at the current time t, and M(t-1) is the optimal estimated value of the motion parameter of the previous frame.

[0099] In one embodiment, performing weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame includes:

[0100] P_(t)=A*P(t-1)A -1 +Q

[0101] Kg=P(t-1)*H T *(H*P(t-1)*H T +R) -1

[0102] M(t)=M_(t)+Kg*(Z(t)-H*M_(t))

[0103] Among them, Kg is the Kalman gain, R is the preset observation noise, Z(t) is the current motion parameter observation, H is the observation matrix, M(t) is the optimal estimate after correction, M_(t) is the motion parameter prediction value, T is the preset interval time, A is the state transfer matrix, P(t-1) is the prediction error at the previous moment t-1, Q is the processing noise, and P_(t) is the prediction covariance at the current moment t.

[0104] In one embodiment, the observation value calculation module is specifically used to perform cumulative multiplication calculation on the historical motion parameters and the current motion parameters through a preset observation model.

[0105] In one embodiment, the preset stability model is:

[0106] I stable =warp(I input ,M smooth .(M source ) -1 )

[0107] Among them, I input is the current frame image input, I stable is the stabilization motion parameter, M source is the cumulative transformation matrix of the jittered video before stabilization, M smooth It represents the cumulative transformation matrix after Kalman filtering, M smooth .(M source ) -1 It is represented as an optimal estimate matrix.

[0108] In one embodiment, the image electronic stabilization device further includes a frame queue construction module, which is used to construct a frame queue based on a current frame and preset queue information, wherein the preset queue information includes a preset interval time and a preset number of frames.

[0109] The embodiment of the present application also provides an image electronic stabilization device, which can be integrated with the image electronic stabilization device provided in the embodiment of the present application. Figure 5 Schematic diagram of the structure of an image electronic stabilization device provided in an embodiment of the present application. Figure 5 The image electronic stabilization device includes: an input device 53, an output device 54, a memory 52 and one or more processors 51; the memory 52 is used to store one or more programs; when one or more programs are executed by one or more processors 51, the one or more processors 51 implement the image electronic stabilization method provided by the above embodiment. The input device 53, the output device 54, the memory 52 and the processor 51 can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0110] The memory 52 is a computing device readable storage medium, which can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image electronic stabilization method provided in any embodiment of the present application (for example, motion parameter set construction module 41, low-pass filter frame number calculation module 42, historical motion parameter extraction module 43, observation value calculation module 44, prediction value calculation module 45, optimal estimation value calculation module 46 and stabilization image mapping module 47). The memory 52 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 52 may further include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0111] The input device 53 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 54 may include a display device such as a display screen.

[0112] The processor 51 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 52, that is, implements the above-mentioned image electronic stabilization method.

[0113] The image electronic stabilization device, equipment and computer provided above can be used to execute the image electronic stabilization method provided by any of the above embodiments, and have corresponding functions and beneficial effects.

[0114] The embodiment of the present application also provides a storage medium storing computer executable instructions, which are used to execute the image electronic stabilization method provided in the above embodiment when executed by a computer processor. The image electronic stabilization method includes: obtaining a frame queue, respectively calculating the pixel translation of two adjacent frames of images in the frame queue to obtain a motion parameter set, wherein the frame queue includes multiple frames of images taken during the navigation of the unmanned equipment; calculating the number of low-pass filtered frames according to the pixel translation between the current frame and the previous frame in the motion parameter set and the preset number of frames; determining the number of historical motion parameters to be extracted according to the number of low-pass filtered frames and the preset rules, and obtaining a motion parameter set according to the number of historical motion parameters in the motion parameter set. The historical motion parameters closest to the current frame are extracted from the motion parameter set; the historical motion parameters and the current motion parameters are input into a preset observation model to obtain the current motion parameter observation value, and the optimal estimated value of the motion parameter of the previous frame calculated historically is input into a preset prediction model to obtain the motion parameter prediction value of the current frame; weighted calculation is performed based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame; matrix conversion is performed on the optimal value of the motion parameter to obtain the optimal estimated value matrix, and the optimal estimated value matrix is ​​input into a preset stabilization model to obtain the stabilization motion parameters, and image mapping is performed based on the stabilization motion parameters to obtain the stabilized image.

[0115] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0116] Of course, the computer executable instructions of a storage medium storing computer executable instructions provided in an embodiment of the present application are not limited to the image electronic stabilization method provided above, and can also execute related operations in the image electronic stabilization method provided in any embodiment of the present application.

[0117] The image electronic stabilization devices, equipment and storage media provided in the above embodiments can execute the image electronic stabilization method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the image electronic stabilization method provided in any embodiment of the present application.

[0118] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments provided herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. An image electronic stabilization method, characterized in that: include: Acquire a frame queue, and respectively calculate pixel translation amounts of two adjacent frame images in the frame queue to obtain a set of motion parameters, wherein the frame queue includes multiple frame images taken during the navigation of the unmanned device; Calculating the number of low-pass filtering frames according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset number of frames; Determine the number of historical motion parameters to be extracted according to the number of low-pass filtering frames and a preset rule, and extract the historical motion parameters closest to the current frame from the motion parameter set according to the number of historical motion parameters; Inputting the historical motion parameters and the current motion parameters into a preset observation model to obtain the current motion parameter observation value, and inputting the optimal estimated value of the motion parameters of the previous frame calculated historically into a preset prediction model to obtain the motion parameter prediction value of the current frame; Performing weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame; The optimal values ​​of the motion parameters are transformed into a matrix to obtain an optimal estimation value matrix, and the optimal estimation value matrix is ​​input into a preset stabilization model to obtain stabilization motion parameters, and image mapping is performed based on the stabilization motion parameters to obtain a stabilized image.

2. The image electronic stabilization method according to claim 1, characterized in that: The formula for calculating the number of low-pass filtering frames according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset number of frames includes: The symbol [] represents the rounding function, k is the low-pass filter frame number, t is the current frame, N is the preset frame number, t xy is the pixel shift between two adjacent frames.

3. The image electronic stabilization method according to claim 1, characterized in that: The preset prediction model includes a state transfer matrix, and the optimal estimated value of the motion parameter of the previous frame calculated historically is input into the preset prediction model to obtain the predicted value of the motion parameter of the current frame, including: The motion parameter prediction value of the current frame is obtained by calculating the optimal estimated value of the motion parameter of the previous frame and the state transfer matrix calculated historically.

4. The image electronic stabilization method according to claim 3, characterized in that: The calculation formula for obtaining the motion parameter prediction value of the current frame by calculating the motion parameter prediction value of the previous frame based on the historically calculated motion parameter optimal estimation value and the state transfer matrix includes: M_(t)=A*M(t-1) Among them, A is the state transfer matrix, M_(t) is the predicted value of the motion parameter at the current time t, and M(t-1) is the optimal estimated value of the motion parameter of the previous frame.

5. The image electronic stabilization method according to claim 1, characterized in that: The step of performing weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame includes: P_(t)=A*P(t-1)A -1 +Q Kg=P(t-1)*H T *(H*P(t-1)*H T +R) -1 M(t)=M_(t)+Kg*(Z(t)-H*M_(t)) Among them, Kg is the Kalman gain, R is the preset observation noise, Z(t) is the current motion parameter observation, H is the observation matrix, M(t) is the optimal estimate after correction, M_(t) is the motion parameter prediction value, T is the preset interval time, A is the state transfer matrix, P(t-1) is the prediction error at the previous moment t-1, Q is the processing noise, and P_(t) is the prediction covariance at the current moment t.

6. The image electronic stabilization method according to claim 1, characterized in that: The step of inputting the historical motion parameters and the current motion parameters into a preset observation model includes: The historical motion parameters and the current motion parameters are cumulatively calculated through a preset observation model.

7. The image electronic stabilization method according to claim 1, characterized in that: The preset stability model is: I stable =warp(I input ,M smooth .(M source ) -1 ) Among them, I input is the current frame image input, I stable is the stabilization motion parameter, M source is the cumulative transformation matrix of the jittered video before stabilization, M smooth It represents the cumulative transformation matrix after Kalman filtering, M smooth .(M source ) -1 It is represented as an optimal estimate matrix.

8. The image electronic stabilization method according to claim 1, characterized in that: Before getting the frame queue, also include: A frame queue is constructed based on the current frame and preset queue information, where the preset queue information includes a preset interval time and a preset number of frames.

9. An image electronic stabilization device, characterized in that: include: A motion parameter set building module is used to obtain a frame queue, respectively calculate the pixel translation amount of two adjacent frame images in the frame queue, and obtain a motion parameter set, wherein the frame queue includes multiple frame images taken during the navigation of the unmanned device; A low-pass filter frame number calculation module, used to calculate the low-pass filter frame number according to the pixel translation amount between the current frame and the previous frame in the motion parameter set and the preset frame number; A historical motion parameter extraction module, used to determine the number of historical motion parameters to be extracted according to the low-pass filtering frame number and a preset rule, and extract the historical motion parameters closest to the current frame from the motion parameter set according to the number of historical motion parameters; An observation value calculation module, used for inputting the historical motion parameters and the current motion parameters into a preset observation model to obtain the current motion parameter observation value; A prediction value calculation module, used to input the optimal estimated value of the motion parameter of the previous frame calculated historically into a preset prediction model to obtain the predicted value of the motion parameter of the current frame; An optimal estimation value calculation module, used for performing weighted calculation based on the motion parameter prediction value of the current frame and the current motion parameter observation value to obtain the optimal value of the motion parameter of the current frame; The stabilization image mapping module is used to perform matrix conversion on the optimal values ​​of the motion parameters to obtain an optimal estimation value matrix, and input the optimal estimation value matrix into a preset stabilization model to obtain stabilization motion parameters, and perform image mapping based on the stabilization motion parameters to obtain a stabilized image.

10. An image electronic stabilization device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image electronic stabilization method as described in any one of claims 1 to 8.

11. A storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the image electronic stabilization method as described in any one of claims 1 to 8 when executed by a computer processor.

Images