Image processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring an original image frame and a background reference attitude corresponding to the original image frame; the background reference attitude is an expected attitude when a background image in the original image frame meets a first condition; determining a foreground reference attitude based on the background reference attitude; the foreground reference attitude is an expected attitude when a foreground image in the original image frame meets a second condition; fusing the foreground reference attitude and the background reference attitude to obtain a fused attitude; and correcting the original image frame based on the fusion attitude to obtain a final image frame.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, apparatus, device, and storage medium. Background Art

[0002] With the advancement of technology and the improvement of living standards, more and more users use mobile devices to record videos of their life and work and share them on social media, which puts higher demands on video stabilization technology.

[0003] The current electronic image stabilization (EIS) method mainly focuses on the stabilization of the background. When the foreground moves, such as faces and bodies, it is prone to shaking and instability. The shaking foreground features will also interfere with the stabilization effect of the background, resulting in a poor anti-shake effect. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and apparatus, a device, and a storage medium, which can improve the anti-shake effect.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Acquire an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets the first condition;

[0008] Determine a foreground reference posture based on the background reference posture; the foreground reference posture is an expected posture when the foreground image in the original image frame meets the second condition;

[0009] The foreground reference posture and the background reference posture are fused to obtain a fused posture; and the original image frame is corrected based on the fused posture to obtain a final image frame.

[0010] In the above image processing method, the step of determining the foreground reference posture based on the background reference posture comprises:

[0011] Obtaining the original posture of the original image frame when it is collected;

[0012] Determine a background pose difference between the original pose and the background reference pose, and determine a foreground pose difference based on the background pose difference;

[0013] The foreground reference pose is determined based on the foreground pose difference and the original pose.

[0014] In the above-mentioned image processing method, the step of determining the foreground posture difference according to the background posture difference comprises:

[0015] determining foreground key points from the foreground image;

[0016] Rotating the foreground key point based on the background posture difference to obtain the rotated foreground key point;

[0017] Based on the rotated foreground key points and the historical foreground key points, the foreground reference key points are determined; the foreground posture difference is determined according to the foreground reference key points and the rotated foreground key points; the historical foreground key points are the foreground key points in the historical image frame; the historical image frame is the image frame that is corrected from the previous frame of the original image frame.

[0018] In the above image processing method, determining the foreground key points from the foreground image includes:

[0019] Determining a target object included in the foreground image;

[0020] In the case that there are multiple target objects, determining a first object with the largest image area among the multiple target objects;

[0021] Feature extraction is performed on the first object to obtain object features of the first object, and a mean point of the object features is used as the foreground key point.

[0022] In the above-mentioned image processing method, fusing the foreground reference posture and the background reference posture to obtain a fused posture includes:

[0023] Determining a capture scene of the original image frame according to an original posture of the original image frame when it is captured;

[0024] Determine a preset weight value assigned to the acquisition scene; fuse the foreground reference posture and the background reference posture according to the preset weight value to obtain the fused posture.

[0025] In the above-mentioned image processing method, the correcting the original image frame based on the fusion posture to obtain the final image frame includes:

[0026] Determining the anti-shake strength according to the fusion posture and the pre-calibrated device internal parameters; the pre-calibrated device internal parameters are the internal parameters of the image acquisition device used when acquiring the original image;

[0027] Updating the fused posture based on the anti-shake strength to obtain an updated fused posture;

[0028] The original image frame is corrected based on the updated fusion posture to obtain the final image frame.

[0029] In the above-mentioned image processing method, the step of determining the anti-shake strength according to the fusion posture and the pre-calibrated device internal parameters includes:

[0030] determining a fused posture difference between the fused posture and the original posture;

[0031] Rotating the original image frame based on the fusion posture difference to obtain a rotated image frame;

[0032] Determine the corner point position of the rotated image frame according to the pre-calibrated device internal parameters; determine the minimum distance between the corner point position and a preset boundary; the preset boundary is the acquisition window boundary corresponding to the image acquisition device;

[0033] The anti-shake strength is determined based on the minimum distance.

[0034] In the above-mentioned image processing method, the correcting the original image frame based on the fusion posture to obtain the final image frame includes:

[0035] generating a first correction grid based on the fused posture and pre-calibrated device internal parameters;

[0036] The original image frame is subjected to image correction using the first correction grid to obtain the final image frame.

[0037] In the above image processing method, the method further includes:

[0038] Determining a first moving distance between a foreground key point of a foreground image in a historical image frame and a foreground key point of a foreground image in the final image frame;

[0039] Determining a second moving distance between a background key point in the background image in the historical image frame and a background key point in the background image in the final image frame;

[0040] A foreground anti-shake effect and a background anti-shake effect are determined based on the first moving distance and the second moving distance.

[0041] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0042] An acquisition unit, used to acquire an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets a first condition;

[0043] A determination unit, configured to determine a foreground reference posture based on the background reference posture; the foreground reference posture is an expected posture when the foreground image in the original image frame satisfies a second condition;

[0044] A fusion unit is used to fuse the foreground reference posture and the background reference posture to obtain a fused posture; based on the fused posture, the original image frame is corrected to obtain a final image frame. In a third aspect, an embodiment of the present application provides an image processing device, the device comprising: a processor, a memory and a communication bus; when the processor executes an operating program stored in the memory, an image processing method as described in any one of the above items is implemented.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method as described in any one of the above items.

[0046] The embodiment of the present application provides an image processing method and apparatus, device, and storage medium, the method comprising: obtaining an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is the expected posture when the background image in the original image frame meets the first condition; determining a foreground reference posture based on the background reference posture; the foreground reference posture is the expected posture when the foreground image in the original image frame meets the second condition; fusing the foreground reference posture and the background reference posture to obtain a fused posture; correcting the original image frame based on the fused posture to obtain a final image frame. With the above implementation scheme, since the foreground reference posture is a foreground reference posture that stabilizes the foreground image, and the background reference posture is a background reference posture that stabilizes the background image, the fused posture obtained after fusing the foreground reference posture with the background reference posture can be considered to be able to simultaneously keep both the foreground image and the background image in a stable posture, and therefore, the final image frame corrected based on the posture can simultaneously take into account the stability of both the foreground image and the background image, thereby improving the anti-shake effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A flow chart of an image processing method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of an exemplary method for determining a foreground reference posture provided in an embodiment of the present application;

[0050] Figure 3A flowchart of an exemplary method for determining a foreground posture difference provided in an embodiment of the present application;

[0051] Figure 4 A flowchart of an exemplary method for determining foreground key points provided in an embodiment of the present application;

[0052] Figure 5 A flowchart of an exemplary method for generating a fusion posture provided in an embodiment of the present application;

[0053] Figure 6 A flowchart of an exemplary method for updating a fusion posture provided in an embodiment of the present application;

[0054] Figure 7 A flowchart of an exemplary method for determining anti-shake strength provided in an embodiment of the present application;

[0055] Figure 8 A flowchart of an exemplary image correction method provided in an embodiment of the present application;

[0056] Fig. 9 A flowchart of an exemplary method for determining an anti-shake effect provided in an embodiment of the present application;

[0057] Fig.10 An exemplary shooting schematic diagram provided for an embodiment of the present application;

[0058] Fig.11 An exemplary data flow diagram provided for an embodiment of the present application;

[0059] Fig.12 A flowchart of an exemplary method for calculating fusion posture provided in an embodiment of the present application;

[0060] Fig.13 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0061] Fig.14 A schematic diagram of the composition structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] The present application provides an image processing method, which is applied to an image processing device. Figure 1A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the image processing method may include the following steps S100 to S102:

[0064] Step S100, obtaining an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets a first condition.

[0065] In an embodiment of the present application, the image processing device obtains an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets the first condition.

[0066] In some embodiments, the original image frame may be an image frame captured by an image acquisition device. For example, when taking photos by an image acquisition device, each photo is an image frame, or when taking photos by an image acquisition device, each frame of the image in the obtained video is an image frame. The specific original image frame may be determined based on actual conditions, and the embodiments of the present application do not make any specific limitations thereon.

[0067] In some embodiments, the original image frame is generally composed of a background image and a foreground image. For example, if the original image frame is a selfie of a user, then the user's portrait is the foreground image in the selfie, and the background of the user when taking the selfie is the background image in the selfie. The specific background image and foreground image can be determined based on actual conditions, and the embodiments of the present application do not make specific limitations here.

[0068] It should be noted that since the original image frames are captured image frames that have not yet been anti-shake processed, the original image frames usually have foreground shaking or background shaking. The solution of the present application is the process of anti-shake processing on the original image frames.

[0069] In some embodiments, after acquiring the original image frame, the image processing device can use a background stabilization method to obtain the background reference posture of the background image in the original image frame. The background reference posture can be understood as the background image being stable under this posture; wherein the first condition can be understood as a condition that characterizes the stability of the background image, for example, the shortest distance between a feature point of the background image in the original image frame and the corresponding feature point of the background image in the previous image frame is less than a threshold, at which time, the background image of the original image frame can be considered to be stable; the specific first condition can be specified according to actual conditions, and the embodiments of the present application do not make specific limitations here.

[0070] Step S101, determining a foreground reference posture based on a background reference posture; the foreground reference posture is an expected posture when the foreground image in the original image frame meets a second condition.

[0071] In an embodiment of the present application, after acquiring the original image frame and the background reference posture corresponding to the original image frame, the image processing device determines the foreground reference posture based on the background reference posture; the foreground reference posture is the expected posture when the foreground image in the original image frame meets the second condition.

[0072] In some embodiments, after acquiring the background reference posture corresponding to the original image frame, the image processing device can determine the foreground reference posture of the foreground image through the background reference posture of the background image. Similar to the background reference posture, the foreground reference posture can also be understood as the foreground image being stable under this posture; wherein the second condition can be understood as a condition that characterizes the stability of the foreground image, for example, the shortest distance between a feature point of the foreground image in the original image frame and the corresponding feature point of the foreground image in the previous image frame is less than a threshold value. At this time, the foreground image of the original image frame can be considered to be stable; the specific second condition can be specified according to actual conditions, and the embodiments of the present application are not specifically limited here.

[0073] Step S102: Fusing the foreground reference posture and the background reference posture to obtain a fused posture; and correcting the original image frame based on the fused posture to obtain a final image frame.

[0074] In an embodiment of the present application, after obtaining the foreground reference posture corresponding to the foreground image and the stable background reference posture of the background image, the image processing device fuses the foreground reference posture and the background reference posture to obtain a fused posture; and corrects the original image frame based on the fused posture to obtain a final image frame.

[0075] It can be understood that since the foreground reference posture is the foreground reference posture that stabilizes the foreground image, and the background reference posture is the background reference posture that stabilizes the background image, the fused posture obtained after fusing the foreground reference posture with the background reference posture can be considered to be able to simultaneously keep both the foreground image and the background image in a stable posture. Therefore, the final image frame corrected based on the posture can take into account the stability of both the foreground image and the background image.

[0076] In an optional embodiment, the image processing device determines a foreground reference posture based on the background reference posture. Figure 2 , including the following steps S200 to S202:

[0077] Step S200: obtaining the original posture of the original image frame when it is captured.

[0078] In the embodiment of the present application, the image processing device obtains the original posture of the original image frame when it is captured.

[0079] In some embodiments, since the original image frame is an image frame captured by an image acquisition device, and the image acquisition device includes an inertial measurement unit (IMU), the original posture is the IMU data of the image acquisition device when capturing the original image frame; the specific original posture can be determined based on actual conditions, and the embodiments of the present application do not make specific limitations here.

[0080] Step S201: determine a background posture difference between an original posture and a background reference posture, and determine a foreground posture difference according to the background posture difference.

[0081] In the embodiment of the present application, after obtaining the original posture of the original image frame at the time of acquisition, the image processing device determines the background posture difference between the original posture and the background reference posture, and determines the foreground posture difference based on the background posture difference.

[0082] In some embodiments, the background posture difference can be obtained by taking the difference between the background reference posture and the original posture; the specific method for determining the background posture difference can be selected according to actual conditions, and the embodiments of the present application do not make specific limitations here.

[0083] Step S202: determining a foreground reference pose based on the foreground pose difference and the original pose.

[0084] In the embodiment of the present application, after determining the foreground posture difference, the image processing device determines the foreground reference posture based on the foreground posture difference and the original posture.

[0085] In some embodiments, the background posture difference can be understood as the difference between the background reference posture and the original posture. Similarly, the foreground posture difference can be understood as the difference between the foreground reference posture and the original posture. After determining the foreground posture difference and the original posture, the sum of the foreground posture difference and the original posture can be determined as the foreground reference posture. The specific method for determining the foreground posture difference can be selected according to actual conditions, and the embodiments of the present application are not specifically limited here.

[0086] In an optional embodiment, the image processing device determines the foreground posture difference according to the background posture difference, referring to Figure 3 , including the following steps S300 to S302:

[0087] Step S300: Determine foreground key points from the foreground image.

[0088] In the embodiment of the present application, when the image processing device determines the foreground posture difference based on the background posture difference, it is necessary to first determine the foreground key points from the foreground image.

[0089] In some embodiments, the foreground key point can be understood as a feature point in the foreground image, which can be used to determine the foreground posture difference; for example, assuming that the original image frame is a selfie of the user, the foreground image in the original image frame is a portrait of the user, and the foreground key point can be selected from the portrait photo; the specific selection of the foreground key point can be determined according to actual conditions, and the embodiments of the present application are not specifically limited here.

[0090] Step S301: Rotate the foreground key points based on the background posture difference to obtain the rotated foreground key points.

[0091] In the embodiment of the present application, after determining the foreground key points from the foreground image, the image processing device rotates the foreground key points based on the background posture difference to obtain the rotated foreground key points.

[0092] In some embodiments, when the image processing device rotates the foreground key point, it rotates the posture corresponding to the foreground key point in the original posture. Therefore, it is necessary to first determine the posture corresponding to the foreground key point from the original posture, and then rotate the posture based on the background posture difference to obtain the rotated foreground key point. It should be noted that when the posture is rotated, it is rotated through quaternions; the specific posture rotation method can be rotated according to actual conditions, and the embodiments of the present application do not make specific limitations here.

[0093] Step S302, determine the foreground reference key points based on the rotated foreground key points and the historical foreground key points; determine the foreground posture difference based on the foreground reference key points and the rotated foreground key points; the historical foreground key points are the foreground key points in the historical image frame; the historical image frame is the image frame that is corrected before the original image frame.

[0094] In an embodiment of the present application, after obtaining the rotated foreground key points, the image processing device determines the foreground reference key points based on the rotated foreground key points and the historical foreground key points; determines the foreground posture difference based on the foreground reference key points and the rotated foreground key points; the historical foreground key points are the foreground key points in the historical image frames; and the historical image frames are the image frames that are the previous frame of the original image frame after correction.

[0095] In some embodiments, assuming that the original image frame is an image frame in a video in an image acquisition device, the image processing device usually processes the video frame by frame when performing anti-shake processing on the video. At this time, the original image frame is the unprocessed i-th frame image. After obtaining the rotated foreground key point in the i-th frame image, the image processing device can first obtain the processed i-1-th frame image, determine the position of the foreground key point in the i-1-th frame image, use the position as the historical foreground key point, and then perform weighted averaging of the historical foreground key point and the rotated foreground key point to obtain the foreground reference key point.

[0096] In an optional embodiment, the image processing device determines the foreground key points from the foreground image, referring to Figure 4 , including the following steps S400 to S402:

[0097] Step S400: Determine the target object included in the foreground image.

[0098] In the embodiment of the present application, when the image processing device determines the foreground key points from the foreground image, it is necessary to first determine the target object included in the foreground image.

[0099] In some embodiments, since the original image frame is taken by an image acquisition device, assuming that the object acquired by the image acquisition device is a human face, at this time, the target object is the portrait appearing in the original image frame; assuming that the object acquired by the image acquisition device is a dish, at this time, the target object is the dish appearing in the original image frame; the specific target object can be determined according to actual conditions, and the embodiments of the present application do not make specific limitations here.

[0100] Step S401: when there are multiple target objects, determine a first object having the largest image area among the multiple target objects.

[0101] In the embodiment of the present application, after determining the target object included in the foreground image, the image processing device determines a first object having the largest image area among the multiple target objects if there are multiple target objects.

[0102] In some embodiments, taking the target object as a portrait, one or more portraits may appear in the foreground image. If the number of target objects is one, that is, only one portrait appears in the foreground image, then the portrait can be directly determined as the first object. If the number of target objects is multiple, that is, multiple portraits appear in the foreground image, then the portrait with the largest area among the multiple portraits can be determined as the first object. The specific first object can be determined based on actual conditions, and the embodiments of the present application do not make specific limitations here.

[0103] Step S402: extract features from the first object to obtain object features of the first object, and use the mean point of the object features as the foreground key point.

[0104] In the embodiment of the present application, after determining the first object from the foreground image, the image processing device extracts features of the first object to obtain object features of the first object, and uses the mean point of the object features as the foreground key point.

[0105] In some embodiments, a target detection algorithm (You Only Look Once, YOLO) can be used to extract features of the first object, and then the mean point of the features is calculated as the foreground key point in the original image frame; the specific method for determining the foreground key point can be selected according to actual conditions, and the embodiments of the present application are not specifically limited here.

[0106] In an optional embodiment, the image processing device fuses the foreground reference posture and the background reference posture to obtain a fused posture. Figure 5 , including the following steps S500 to S501:

[0107] Step S500: determining the capture scene of the original image frame according to the original posture of the original image frame when it is captured.

[0108] In the embodiment of the present application, the image processing device fuses the foreground reference posture and the background reference posture to obtain the fused posture. It is necessary to first determine the acquisition scene of the original image frame according to the original posture of the original image frame when it was acquired.

[0109] In some embodiments, when the foreground reference posture and the background reference posture are fused, the weight ratios of the foreground reference posture and the background reference posture in the fused posture can be selected according to the shooting scene. Therefore, the acquisition scene, that is, the shooting scene, can be determined first according to the original posture; for example, when it is determined according to the original posture that the posture amplitude frequency of the original image frame is greater than a preset threshold, the acquisition scene is determined to be running; the specific judgment method can be formulated according to the actual situation, and the embodiments of the present application are not specifically limited here.

[0110] Step S501, determining a preset weight value assigned to a collection scene; fusing a foreground reference posture and a background reference posture according to the preset weight value to obtain a fused posture.

[0111] In an embodiment of the present application, after determining the acquisition scene of the original image frame, the image processing device determines a preset weight value assigned to the acquisition scene; and fuses the foreground reference posture and the background reference posture according to the preset weight value to obtain a fused posture.

[0112] In some embodiments, different weight ratios can be set for the foreground image and the background image according to different acquisition scenarios. For example, assuming that the acquisition scenario is a running scenario, due to the large shaking in the running scenario, a smaller weight can be assigned to the foreground image and a larger weight can be assigned to the background image. Alternatively, assuming that the acquisition scenario is a static scene, a larger weight can be assigned to the foreground image and a smaller weight can be assigned to the background image. Alternatively, assuming that the acquisition scenario is a circular shooting scene, the weight of the foreground image can be reduced and the weight of the background image can be increased during the circular shooting process. After the circular shooting, the weight of the foreground image can be increased and the weight of the background image can be reduced. The specific scenario and weight allocation method can be formulated according to actual conditions, and the embodiments of the present application are not specifically limited here.

[0113] In some embodiments, assuming that the original image frame is a frame in a video, and there are image frames in which a person enters and exits the screen, in order to prevent the video from freezing after anti-shake processing, increasing or decreasing transition weights can be designed when processing the image frames. For example, assuming that the person in the i-th frame is still in the screen, the corresponding weight A is assigned to the foreground image, and the i+1-th frame determines that the person is no longer in the screen. At this time, although the corresponding weight of the foreground image is B when the person is not in the screen, the weight of the foreground image does not need to be directly changed from A to B. The weight of the foreground image can be gradually transitioned from A to B in the i+2, i+3 and other frame images, thereby ensuring the smoothness of the video.

[0114] In an optional embodiment, the image processing device corrects the original image frame based on the fusion posture to obtain a final image frame, referring to Figure 6 , including the following steps S600 to S602:

[0115] Step S600, determining the anti-shake strength according to the fusion posture and the pre-calibrated device internal parameters; the pre-calibrated device internal parameters are the internal parameters of the image acquisition device used when acquiring the original image.

[0116] In an embodiment of the present application, the image processing device corrects the original image frame based on the fusion posture. When obtaining the final image frame, the anti-shake strength can be first determined based on the fusion posture and the pre-calibrated device internal parameters; the pre-calibrated device internal parameters are the internal parameters of the image acquisition device used when acquiring the original image.

[0117] In some embodiments, since the image acquisition device has a limit, the anti-shake strength set for the foreground image and the background image can be determined by fusing the posture with the internal parameters of the image acquisition device, thereby reducing the number of limit collisions and further enhancing the smoothness of the video.

[0118] Step S601: Update the fused posture based on the anti-shake strength to obtain an updated fused posture.

[0119] In the embodiment of the present application, after determining the anti-shake strength, the image processing device updates the fusion posture based on the anti-shake strength to obtain an updated fusion posture.

[0120] In some embodiments, after obtaining the anti-shake strength, the fused posture can be updated according to the anti-shake strength. Since the anti-shake strength includes the foreground anti-shake strength and the background anti-shake strength, the weight ratio of the foreground reference posture in the fused posture can be adjusted according to the foreground anti-shake strength, and the weight ratio of the background reference posture in the fused posture can be adjusted according to the background anti-shake strength, thereby obtaining an updated fused posture.

[0121] Step S602: Correct the original image frame based on the updated fusion posture to obtain a final image frame.

[0122] In the embodiment of the present application, after obtaining the updated fusion posture, the image processing device corrects the original image frame based on the updated fusion posture to obtain the final image frame.

[0123] In an optional embodiment, the image processing device determines the anti-shake strength according to the fusion posture and the pre-calibrated device internal parameters, referring to Figure 7 , including the following steps S700 to S703:

[0124] Step S700: Determine the fused posture difference between the fused posture and the original posture.

[0125] In the embodiment of the present application, the image processing device determines the anti-shake strength according to the fused posture and the pre-calibrated device internal parameters. First, it is necessary to determine the fused posture difference between the fused posture and the original posture.

[0126] Step S701: Rotate the original image frame based on the fused posture difference to obtain a rotated image frame.

[0127] In the embodiment of the present application, after determining the fusion posture difference between the fusion posture and the original posture, the image processing device rotates the original image frame based on the fusion posture difference to obtain a rotated image frame.

[0128] In some embodiments, the original image can be rotated based on the fusion posture difference. At this time, the rotation can be performed based on the center point of the original image frame, or based on the foreground key point in the original image frame. The specific rotation center can be determined based on actual conditions, and the embodiments of the present application do not make specific limitations here.

[0129] Step S702, determining the corner point position of the rotated image frame according to the pre-calibrated device internal parameters; determining the minimum distance between the corner point position and a preset boundary; the preset boundary is the acquisition window boundary corresponding to the image acquisition device.

[0130] In an embodiment of the present application, after the image processing device rotates the original image frame based on the fusion posture difference to obtain the rotated image frame, the image processing device determines the corner point position of the rotated image frame according to the pre-calibrated device internal parameters; determines the minimum distance between the corner point position and a preset boundary; the preset boundary is the acquisition window boundary corresponding to the image acquisition device.

[0131] In some embodiments, after obtaining the rotated image frame, the positions of the four corner points of the rotated image frame can be calculated according to the pre-calibrated device internal parameters, and then the shortest distances between the positions of the four corner points and the corresponding boundaries are calculated to obtain the four shortest distances corresponding to the four corner points, and then the minimum distance of the four shortest distances is determined, and the anti-shake strength is determined based on the minimum distance; another way is to determine the minimum ratio. For example, taking the upper right corner point as an example, the distance from the horizontal coordinate value to the image width is calculated and then divided by the image width as the x ratio, and the distance from the vertical coordinate value to 0 is calculated and then divided by the image height as the y ratio. The smaller value of the x ratio and the y ratio is taken as the ratio corresponding to the upper right corner point. Finally, the four ratios corresponding to the four corner points are obtained, the minimum ratio of the four ratios is determined, and the anti-shake strength is determined based on the minimum ratio.

[0132] Step S703: determine the anti-shake strength based on the minimum distance.

[0133] In the embodiment of the present application, the image processing device determines the anti-shake strength based on the minimum distance.

[0134] In some embodiments, the foreground anti-shake strength of the foreground image and the background anti-shake strength of the background image can be adjusted according to the minimum distance or the minimum ratio. The smaller the value, the smaller the strength. The specific adjustment method can be determined according to actual conditions, and the embodiments of the present application are not specifically limited here.

[0135] In an optional embodiment, the image processing device corrects the original image frame based on the fusion posture to obtain a final image frame, referring to Figure 8 , including the following steps S800 to S801:

[0136] Step S800: Generate a first correction grid based on the fused posture and pre-calibrated device internal parameters.

[0137] In the embodiment of the present application, the image processing device corrects the original image frame based on the fusion posture. When obtaining the final image frame, it is necessary to first generate a first correction grid based on the fusion posture and pre-calibrated device internal parameters.

[0138] In some embodiments, after the fusion posture is determined, a two-dimensional correction grid (first correction grid) for anti-shake processing of the original image frame can be generated according to the fusion posture involved in the pre-calibrated device, and the original image frame can be corrected using the two-dimensional correction grid to obtain the final image frame after anti-shake processing.

[0139] Step S801: Use a first correction grid to perform image correction on an original image frame to obtain a final image frame.

[0140] In the embodiment of the present application, after the image processing device generates a first correction grid based on the fusion posture and the pre-calibrated device internal parameters, the image processing device uses the first correction grid to perform image correction on the original image frame to obtain a final image frame.

[0141] In some embodiments, when performing image correction on the original image frame using a two-dimensional correction grid, the original image frame can be first grid-divided in a division method corresponding to the two-dimensional correction grid, and then the original image frame can be rendered as a final image frame by mesh-warping the grid on the original image frame using the two-dimensional correction grid.

[0142] In an optional embodiment, reference Fig. 9 , the image processing device may further perform the following steps S900 to S902:

[0143] Step S900: Determine a first moving distance between a foreground key point of a foreground image in a historical image frame and a foreground key point of a foreground image in a final image frame.

[0144] In an embodiment of the present application, the image processing device determines a first moving distance between a foreground key point of a foreground image in a historical image frame and a foreground key point of a foreground image in a final image frame.

[0145] In some embodiments, the effect of the anti-shake processing can be evaluated after each image frame in the video is anti-shake processed, including the foreground anti-shake effect and the background anti-shake effect. For the foreground anti-shake effect, the positions of the foreground key points in the final image frame and the historical image frame can be determined respectively, and then the first moving distance between the two can be calculated. The smaller the first moving distance is, the higher the foreground motion stability is, the smaller the foreground jitter is, and thus the better the foreground anti-shake effect is.

[0146] Step S901: Determine a second moving distance between a background key point in a background image in a historical image frame and a background key point in a background image in a final image frame.

[0147] In an embodiment of the present application, the image processing device determines a first moving distance between a foreground key point of a foreground image in a historical image frame and a foreground key point of a foreground image in a final image frame.

[0148] In some embodiments, the evaluation of the background anti-shake effect is similar to that of the foreground anti-shake effect. The positions of the background key points in the final image frame and the historical image frame can be determined respectively, and then the second moving distance between the two can be calculated. The smaller the second moving distance, the higher the background motion stability and the smaller the background jitter, thus the better the background anti-shake effect.

[0149] In some embodiments, when determining the background key points in the historical image frames and the final image frames, the foreground images in the historical image frames and the final image frames can be segmented out respectively, and the image feature extraction (Oriented FAST and Rotated BRIEF, ORB) algorithm or other feature point extraction methods can be used in the area outside the foreground image to detect the background feature points, and then the optical flow or other methods are used to track the feature points, and the mean point of the feature points is calculated as the background key point; the specific method for determining the background key points can be determined according to the actual situation, and the embodiments of the present application are not specifically limited here.

[0150] Step S902: Determine a foreground anti-shake effect and a background anti-shake effect based on the first moving distance and the second moving distance.

[0151] In the embodiment of the present application, after obtaining the first moving distance and the second moving distance, the image processing device can determine the foreground anti-shake effect and the background anti-shake effect based on the first moving distance and the second moving distance.

[0152] In some embodiments, the smaller the first moving distance, the higher the foreground motion stability and the smaller the foreground jitter, thus the better the foreground anti-shake effect; the smaller the second moving distance, the higher the background motion stability and the smaller the background jitter, thus the better the background anti-shake effect.

[0153] The embodiment of the present application provides an image processing method, which includes: obtaining an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is the expected posture when the background image in the original image frame meets the first condition; determining a foreground reference posture based on the background reference posture; the foreground reference posture is the expected posture when the foreground image in the original image frame meets the second condition; fusing the foreground reference posture and the background reference posture to obtain a fused posture; correcting the original image frame based on the fused posture to obtain a final image frame. With the above implementation scheme, since the foreground reference posture is a foreground reference posture that stabilizes the foreground image, and the background reference posture is a background reference posture that stabilizes the background image, the fused posture obtained after fusing the foreground reference posture with the background reference posture can be considered to be able to simultaneously keep both the foreground image and the background image in a stable posture, and therefore, the final image frame corrected based on the posture can simultaneously take into account the stability of both the foreground image and the background image, thereby improving the anti-shake effect.

[0154] The above introduces the image processing method provided by the embodiment of the present application. To facilitate understanding of the embodiment of the present application, the following introduces possible implementation schemes of the image processing method applicable to the embodiment of the present application based on actual application scenarios.

[0155] refer to Fig.10 , the above image processing method is explained by taking a user taking a photo with a mobile phone (the image acquisition device in the above embodiment) as an example.

[0156] Specifically, refer to Fig.11 , which is the hardware data flow of the technical solution of this application, the specific steps are as follows:

[0157] Step 1: When a user uses a mobile phone to take a photo, a complementary metal oxide semiconductor (CMOS) 1100 in the mobile phone obtains an original image and obtains an original image frame after being processed by an image signal processor (ISP) unit 1101.

[0158] Step 2: The ISP unit 1101 inputs the original image frame into the feature point detection unit 1102, and the feature point detection unit 1102 detects the feature points.

[0159] Step 3, the inertial measurement unit 1103 obtains the original posture when the original image frame is shot and inputs it into the filtering unit 1104 for noise filtering. Finally, the original posture after noise filtering is matched with the corresponding original image frame. At the same time, the original posture after noise filtering also needs to be input into the motion estimation unit 1105 to estimate the shooting scene.

[0160] Step 4: Input the shooting scene, feature points of the original image frame and the original posture of the original image frame into the fusion posture and correction grid calculation unit 1106 at the same time, use the fusion posture and correction grid calculation unit 1106 to calculate the two-dimensional correction grid (the first correction grid in the above embodiment), and then input the original image frame and the two-dimensional correction grid into the image processing unit 1107.

[0161] Step 5: The image processing unit 1107 uses the two-dimensional correction grid to correct the original image frame to obtain a final image frame, that is, a stabilized two-dimensional image.

[0162] Specifically, refer to Fig.12 , the implementation of the fusion posture and correction grid calculation unit 1106 is shown in steps S1200 to S1205:

[0163] Step S1200: determine the foreground key points of the original image frame.

[0164] Specifically, it is necessary to detect the face in the original image frame according to the image and extract the face key points, use the YOLO deep learning model to extract the face features (the method includes but is not limited to this), and then calculate the mean point of the face features as the center point, that is, the foreground key point; it should be emphasized that if the original image frame contains multiple faces, the face occupying the largest image area is set as the main face, and its center point is set as the foreground key point. The calculation formula is as follows:

[0165]

[0166] It should be noted that the foreground key point can be the average value of the face points, the centroid or the center position of the nose, etc.

[0167] Step S1201: Determine the background reference posture of the original image frame.

[0168] Specifically, a background stabilization method (including but not limited to weighted filtering using the posture of the previous image frame, filtering with a cache frame, or filtering based on image feature points) is used to obtain a background reference posture Q after background filtering. bg ,The background reference pose can be considered as the stabilized pose of the background image, and then the pose difference between the background reference pose and the original pose is calculated.

[0169] Step S1202: Calculate foreground reference key points according to the background reference posture.

[0170] Specifically, the foreground key point after rotation is obtained by rotating the foreground key point by posture difference, and the foreground key point after rotation and the foreground key point of the previous image frame (the previous image frame after correction) are weighted averaged to obtain the foreground reference key point after current foreground filtering.

[0171] Step S1203: Calculate the foreground reference posture according to the foreground reference key points.

[0172] Specifically, based on the foreground reference key points and the rotated foreground key points projected by the camera 3D model, the foreground pose difference can be calculated, and then the foreground reference pose Q after foreground filtering can be obtained based on the foreground pose difference and the original pose. fg ,The foreground reference pose can be considered as the foreground stable pose.

[0173] Step S1204: Fusing the foreground reference posture and the background reference posture into a fused posture.

[0174] Specifically, based on the foreground reference pose Q fg and background reference pose Q bg , weighted average can get the fusion posture Q res , the fusion posture can take into account both the foreground and the background and be stable at the same time. The calculation formula is as follows:

[0175] Q res =W fg ×Q fg +(1-W fg )×Q bg (2);

[0176] In formula (2), W fg is the foreground reference pose Q fg The corresponding fusion weight.

[0177] Step S1205: Calculate a two-dimensional correction grid according to the fusion posture, and use the two-dimensional correction grid to correct the original image frame.

[0178] Specifically, a two-dimensional correction grid can be obtained according to the fusion posture and the pre-calibrated camera internal parameters (the pre-calibrated device internal parameters in the above embodiment). The original image frame is directly rendered into a two-dimensional image after anti-shake, that is, the final image frame, through the grid distortion transformation.

[0179] It is understandable that the present application does not require the fusion of the foreground image and the background image. It only needs to finally fuse the complete original image frame with the two-dimensional correction grid to obtain a stable final image frame, thereby improving the anti-shake effect.

[0180] Furthermore, for different shooting scenes, the weight W fg It can be adjusted by yourself. For example, if the background shakes a lot when running, the foreground stability weight is reduced, and if it remains still, the foreground weight is increased. The foreground stability weight is reduced during the ring shooting process, and the foreground stability weight is increased after the ring shooting. Different scenes can be judged based on the posture data statistics, such as whether the posture amplitude frequency of the current frame is greater than the set fixed threshold. If both are greater, it is judged as running. For the foreground face entering and exiting the screen, an increasing or decreasing transition weight can be designed to prevent freezes and enhance the smoothness of the overall video.

[0181] In addition, this application can also adjust the anti-shake strength of the foreground and background by judging the distance from the fusion posture to the limit, so as to reduce the number of limit collisions and enhance the smoothness of the overall video. The specific steps are as follows:

[0182] Step 1: Rotate the original image frame according to the fusion posture to obtain the rotated image frame. Calculate the positions of the four corner points of the rotated image frame according to the fusion posture and the pre-calibrated camera internal parameters.

[0183] Step 2: Calculate the difference ratio between the four corner points and the border, and take the minimum ratio of the four points. For example, for the upper right corner, calculate the distance from the horizontal coordinate value to the image width and divide it by the image width as the x ratio, and calculate the distance from the vertical coordinate value to 0 and divide it by the image height as the y ratio. Take the smaller value. The calculation formula is as follows:

[0184]

[0185] Step 3: The anti-shake strength of the foreground and background can be adjusted according to a smaller ratio. The smaller the value, the smaller the strength. fg The background anti-shake strength can be adjusted according to the selected filtering method.

[0186] Furthermore, after the video or image is stabilized, the stabilization effect can be evaluated:

[0187] For the foreground anti-shake effect, the foreground stability index is set to the average of the moving distances of the foreground key points between the current frame and the previous frame in the video. The smaller the calculated average value, the higher the foreground motion stability and the smaller the foreground jitter. The specific average calculation formula is as follows:

[0188]

[0189] Similarly, for the background anti-shake effect, the background stability index is set to the average of the moving distances of the background key points between the current frame and the previous frame in the video. The smaller the calculated average value is, the higher the background motion stability is and the smaller the background jitter is. The calculation method of the background key points can be implemented by the following method: use portrait segmentation to segment the foreground (or detect the portrait area and estimate the overall foreground area), use ORB or other feature point extraction methods in the area outside the foreground to detect background feature points, and then use optical flow or other methods to track feature points, and calculate the mean point of the feature points as the background key points. The specific average calculation formula is as follows:

[0190]

[0191] It is understandable that the final video after the anti-shake processing of this application shows a significant improvement in the stability of the foreground and background compared to the original video. This application can further enhance the smoothness of the overall video by dynamically adjusting the anti-shake strength of the foreground and background, as well as the targeted parameter adjustment of each scene. In addition, this application performs fusion processing at the posture level and renders the image as a whole, effectively avoiding the image quality degradation and ghosting problems that may be caused by the fusion of the foreground and background images, thereby providing users with a better video anti-shake experience.

[0192] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of ​​the present application, they should also be regarded as the contents disclosed in the present application. For another example, under the premise of no conflict, the various embodiments and / or the technical features in the various embodiments described in the present application can be arbitrarily combined with the prior art, and the technical solution obtained after the combination should also fall within the protection scope of the present application.

[0193] Based on the above embodiment, in another embodiment of the present application, an image processing device 13 is provided. Fig.13 A schematic diagram of the structure of an image processing device provided in this application is shown in FIG. Fig.13 As shown, the image processing device 13 includes:

[0194] An acquisition unit 1300 is used to acquire an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets a first condition;

[0195] A determining unit 1301 is configured to determine a foreground reference posture based on the background reference posture; the foreground reference posture is an expected posture when the foreground image in the original image frame satisfies a second condition;

[0196] The fusion unit 1302 is used to fuse the foreground reference posture and the background reference posture to obtain a fused posture; and correct the original image frame based on the fused posture to obtain a final image frame.

[0197] Optionally, the acquisition unit 1300 is further used to acquire the original posture of the original image frame when it is acquired;

[0198] The determining unit 1301 is further configured to determine a background posture difference between the original posture and the background reference posture, determine a foreground posture difference according to the background posture difference, and determine the foreground reference posture based on the foreground posture difference and the original posture.

[0199] Optionally, the image processing device 13 further includes: a data processing unit;

[0200] The determining unit 1301 is further configured to determine a foreground key point from the foreground image;

[0201] The data processing unit is used to rotate the foreground key point based on the background posture difference to obtain the rotated foreground key point;

[0202] The determination unit 1301 is also used to determine the foreground reference key points based on the rotated foreground key points and the historical foreground key points; determine the foreground posture difference according to the foreground reference key points and the rotated foreground key points; the historical foreground key points are the foreground key points in the historical image frame; the historical image frame is the image frame that is corrected from the previous frame of the original image frame.

[0203] Optionally, the determining unit 1301 is further configured to determine a target object included in the foreground image; and when there are multiple target objects, determine a first object having the largest image area among the multiple target objects;

[0204] The data processing unit is further used to extract features from the first object to obtain object features of the first object, and use the mean point of the object features as the foreground key point.

[0205] Optionally, the determining unit 1301 is further configured to determine a capture scene of the original image frame according to an original posture of the original image frame when it is captured; and determine a preset weight value assigned to the capture scene;

[0206] The fusion unit 1302 is further configured to fuse the foreground reference posture and the background reference posture according to the preset weight value to obtain the fused posture.

[0207] Optionally, the image processing device 13 further includes: an updating unit and an image processing unit;

[0208] The determining unit 1301 is further configured to determine the anti-shake strength according to the fusion posture and the pre-calibrated device internal parameters; the pre-calibrated device internal parameters are the internal parameters of the image acquisition device used when acquiring the original image;

[0209] The updating unit is used to update the fused posture based on the anti-shake strength to obtain an updated fused posture;

[0210] The image processing unit is used to correct the original image frame based on the updated fusion posture to obtain the final image frame.

[0211] Optionally, the determining unit 1301 is further configured to determine a fusion posture difference between the fusion posture and the original posture;

[0212] The image processing unit is further used to rotate the original image frame based on the fusion posture difference to obtain a rotated image frame;

[0213] The determination unit 1301 is further used to determine the corner point position of the rotated image frame according to the pre-calibrated device internal parameters; determine the minimum distance between the corner point position and a preset boundary; the preset boundary is the acquisition window boundary corresponding to the image acquisition device; and determine the anti-shake strength based on the minimum distance.

[0214] Optionally, the image processing unit is further used to generate a first correction grid based on the fusion posture and pre-calibrated device internal parameters; and use the first correction grid to perform image correction on the original image frame to obtain the final image frame.

[0215] Optionally, the determination unit 1301 is also used to determine a first moving distance between a foreground key point of a foreground image in a historical image frame and a foreground key point of a foreground image in a final image frame; determine a second moving distance between a background key point in a background image in the historical image frame and a background key point in the background image in the final image frame; and determine a foreground anti-shake effect and a background anti-shake effect based on the first moving distance and the second moving distance.

[0216] The embodiment of the present application provides an image processing device, which includes: an acquisition unit, used to acquire an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is the expected posture when the background image in the original image frame meets the first condition; a determination unit, used to determine a foreground reference posture based on the background reference posture; the foreground reference posture is the expected posture when the foreground image in the original image frame meets the second condition; a fusion unit, used to fuse the foreground reference posture and the background reference posture to obtain a fused posture; based on the fused posture, the original image frame is corrected to obtain a final image frame. With the above implementation scheme, since the foreground reference posture is a foreground reference posture that makes the foreground image stable, and the background reference posture is a background reference posture that makes the background image stable, the fused posture obtained after fusing the foreground reference posture with the background reference posture can be considered to be able to simultaneously keep the foreground image and the background image in a stable posture. Therefore, the final image frame corrected based on the posture can simultaneously take into account the stability of the foreground image and the background image, thereby improving the anti-shake effect.

[0217] Fig.14 This is a schematic diagram of the composition structure of an image processing device provided in an embodiment of the present application. In practical applications, based on the same disclosed concept of the above embodiments, Fig.14As shown, the image processing device 14 of this embodiment includes: a processor 1400 , a memory 1401 and a communication bus 1402 .

[0218] In the process of a specific embodiment, the acquisition unit 1300, the determination unit 1301, the fusion unit 1302, the data processing unit, the update unit and the image processing unit can be implemented by a processor 1400 located on the image processing device 14, and the processor 1400 can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing image processing device (DSPD), a programmable logic image processing device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It can be understood that for different image processing devices, the electronic device used to implement the above-mentioned processor function can also be other, and this embodiment does not specifically limit it.

[0219] In the embodiment of the present application, the communication bus 1402 is used to realize the connection and communication between the processor 1400 and the memory 1401; when the processor 1400 executes the running program stored in the memory 1401, the following image processing method is realized:

[0220] Acquire an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets the first condition;

[0221] Determine a foreground reference posture based on the background reference posture; the foreground reference posture is an expected posture when the foreground image in the original image frame meets the second condition;

[0222] The foreground reference posture and the background reference posture are fused to obtain a fused posture; and the original image frame is corrected based on the fused posture to obtain a final image frame.

[0223] Optionally, the processor 1400 is also used to obtain the original posture of the original image frame when it is collected; determine the background posture difference between the original posture and the background reference posture, and determine the foreground posture difference based on the background posture difference; determine the foreground reference posture based on the foreground posture difference and the original posture.

[0224] Optionally, the processor 1400 is also used to determine foreground key points from the foreground image; rotate the foreground key points based on the background posture difference to obtain the rotated foreground key points; determine the foreground reference key points based on the rotated foreground key points and historical foreground key points; determine the foreground posture difference based on the foreground reference key points and the rotated foreground key points; the historical foreground key points are the foreground key points in the historical image frame; the historical image frame is the image frame that is a corrected frame before the original image frame.

[0225] Optionally, the processor 1400 is further used to determine the target objects included in the foreground image; when there are multiple target objects, determine the first object with the largest image area among the multiple target objects; perform feature extraction on the first object to obtain object features of the first object, and use the mean point of the object features as the foreground key point.

[0226] Optionally, the processor 1400 is further used to determine the capture scene of the original image frame based on the original posture of the original image frame when it was captured; determine a preset weight value assigned to the capture scene; and fuse the foreground reference posture and the background reference posture according to the preset weight value to obtain the fused posture.

[0227] Optionally, the processor 1400 is also used to determine the anti-shake strength based on the fusion posture and pre-calibrated device internal parameters; the pre-calibrated device internal parameters are internal parameters of the image acquisition device used when acquiring the original image; based on the anti-shake strength, the fusion posture is updated to obtain an updated fusion posture; based on the updated fusion posture, the original image frame is corrected to obtain the final image frame.

[0228] Optionally, the processor 1400 is further used to determine a fusion posture difference between the fusion posture and the original posture; rotate the original image frame based on the fusion posture difference to obtain a rotated image frame; determine the corner point position of the rotated image frame according to a pre-calibrated device internal parameter; determine the minimum distance between the corner point position and a preset boundary; the preset boundary is the acquisition window boundary corresponding to the image acquisition device; and determine the anti-shake strength based on the minimum distance.

[0229] Optionally, the processor 1400 is further configured to generate a first correction grid based on the fusion posture and pre-calibrated device internal parameters; and perform image correction on the original image frame using the first correction grid to obtain the final image frame.

[0230] Optionally, the processor 1400 is also used to determine a first moving distance between a foreground key point of a foreground image in a historical image frame and a foreground key point of a foreground image in a final image frame; determine a second moving distance between a background key point in a background image in the historical image frame and a background key point in the background image in the final image frame; and determine a foreground anti-shake effect and a background anti-shake effect based on the first moving distance and the second moving distance.

[0231] An embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the image processing method of any of the above embodiments.

[0232] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above-mentioned image processing method.

[0233] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0234] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0235] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0236] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0237] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0238] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0239] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0240] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0241] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets the first condition; Determine a foreground reference posture based on the background reference posture; the foreground reference posture is an expected posture when the foreground image in the original image frame meets the second condition; The foreground reference posture and the background reference posture are fused to obtain a fused posture; and the original image frame is corrected based on the fused posture to obtain a final image frame.

2. The method according to claim 1, characterized in that The determining of a foreground reference posture based on the background reference posture comprises: Obtaining the original posture of the original image frame when it is collected; Determine a background pose difference between the original pose and the background reference pose, and determine a foreground pose difference based on the background pose difference; The foreground reference pose is determined based on the foreground pose difference and the original pose.

3. The method according to claim 2, characterized in that The determining of the foreground posture difference according to the background posture difference comprises: determining foreground key points from the foreground image; Rotating the foreground key point based on the background posture difference to obtain the rotated foreground key point; Based on the rotated foreground key points and the historical foreground key points, the foreground reference key points are determined; the foreground posture difference is determined according to the foreground reference key points and the rotated foreground key points; the historical foreground key points are the foreground key points in the historical image frame; the historical image frame is the image frame that is corrected from the previous frame of the original image frame.

4. The method according to claim 3, characterized in that The step of determining a foreground key point from the foreground image comprises: Determining a target object included in the foreground image; In the case that there are multiple target objects, determining a first object with the largest image area among the multiple target objects; Feature extraction is performed on the first object to obtain object features of the first object, and a mean point of the object features is used as the foreground key point.

5. The method according to claim 1, characterized in that The fusing the foreground reference posture and the background reference posture to obtain a fused posture includes: Determining a capture scene of the original image frame according to an original posture of the original image frame when it is captured; Determine a preset weight value assigned to the acquisition scene; fuse the foreground reference posture and the background reference posture according to the preset weight value to obtain the fused posture.

6. The method according to claim 1 or 5, characterized in that: The correcting the original image frame based on the fusion posture to obtain a final image frame includes: Determining the anti-shake strength according to the fusion posture and the pre-calibrated device internal parameters; the pre-calibrated device internal parameters are the internal parameters of the image acquisition device used when acquiring the original image; Updating the fused posture based on the anti-shake strength to obtain an updated fused posture; The original image frame is corrected based on the updated fusion posture to obtain the final image frame.

7. The method according to claim 6, characterized in that The step of determining the anti-shake strength according to the fusion posture and the pre-calibrated device internal parameters includes: determining a fused posture difference between the fused posture and the original posture; Rotating the original image frame based on the fusion posture difference to obtain a rotated image frame; Determine the corner point position of the rotated image frame according to the pre-calibrated device internal parameters; determine the minimum distance between the corner point position and a preset boundary; the preset boundary is the acquisition window boundary corresponding to the image acquisition device; The anti-shake strength is determined based on the minimum distance.

8. The method according to claim 1, characterized in that The correcting the original image frame based on the fusion posture to obtain a final image frame includes: generating a first correction grid based on the fused posture and pre-calibrated device internal parameters; The original image frame is subjected to image correction using the first correction grid to obtain the final image frame.

9. The method according to claim 8, characterized in that The method further comprises: Determining a first moving distance between a foreground key point of a foreground image in a historical image frame and a foreground key point of a foreground image in the final image frame; Determining a second moving distance between a background key point in the background image in the historical image frame and a background key point in the background image in the final image frame; A foreground anti-shake effect and a background anti-shake effect are determined based on the first moving distance and the second moving distance.

10. An image processing device, characterized in that: The device comprises: An acquisition unit, used to acquire an original image frame and a background reference posture corresponding to the original image frame; the background reference posture is an expected posture when the background image in the original image frame meets a first condition; A determination unit, configured to determine a foreground reference posture based on the background reference posture; the foreground reference posture is an expected posture when the foreground image in the original image frame satisfies a second condition; The fusion unit is used to fuse the foreground reference posture and the background reference posture to obtain a fused posture; and correct the original image frame based on the fused posture to obtain a final image frame.

11. An image processing device, characterized in that: The device comprises: a processor, a memory and a communication bus; when the processor executes the running program stored in the memory, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images