Image processing method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure provides an image processing method, device, electronic device, storage medium and program product, the method comprising: in response to the electronic device acquiring an nth image frame in the process of executing a video shooting function, determining a first cropping parameter of the nth image frame based on a jitter parameter of the electronic device; wherein the jitter parameter is determined based on a motion parameter of the electronic device in a period from after acquiring the nth-m image frame to before acquiring the nth image frame; m is greater than or equal to 1; based on the nth image frame, obtaining an nth to-be-cropped image frame matched with the first cropping parameter, and cropping the nth to-be-cropped image frame based on the first cropping parameter to obtain an nth target image frame; in response to detecting a shooting end instruction, generating a target video based on each target image frame obtained in the process of executing the video shooting function. The embodiments of the present disclosure can reduce the influence of the jitter of the electronic device on the video obtained by shooting.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In related technologies, video stabilization can be performed during the video recording function of an electronic device. For example, related technologies can reduce or eliminate blurring and shaking when an electronic device captures or records images in a moving or vibrating environment through optical image stabilization, electronic image stabilization, and image sensor stabilization.

[0003] However, optical image stabilization and sensor-based image stabilization place high demands on the hardware of electronic devices, resulting in high hardware costs. Furthermore, electronic image stabilization technology cannot effectively eliminate the impact of electronic device shake on the captured image frames in all situations, leading to severe image shakiness still present in the captured video. Summary of the Invention

[0004] To overcome the problems in related technologies, this disclosure provides an image processing method, apparatus, electronic device, storage medium, and program product to effectively reduce the impact of electronic device shake on the captured video during video shooting and improve video stability.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising:

[0006] In response to the acquisition of the nth image frame by the electronic device during the video recording function, the first cropping parameter of the nth image frame is determined based on the jitter parameter of the electronic device; wherein, the jitter parameter is determined based on the motion parameter of the electronic device during the first time period from after the acquisition of the nmth image frame to before the acquisition of the nth image frame; m is greater than or equal to 1;

[0007] Based on the nth image frame, obtain the nth image frame to be cropped that matches the first cropping parameter, and perform cropping processing on the nth image frame to be cropped based on the first cropping parameter to obtain the nth target image frame;

[0008] In response to the detection of a shooting end command, a target video is generated based on each target image frame obtained during the execution of the video shooting function.

[0009] In one embodiment, obtaining the nth image frame to be cropped based on the nth image frame and matching the first cropping parameter includes:

[0010] If the nth image frame matches the first cropping parameter, the nth image frame is determined as the nth image frame to be cropped.

[0011] If the nth image frame does not match the first cropping parameter, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0012] In one embodiment, the nth image frame is subjected to image expansion processing to obtain the nth image frame to be cropped, including:

[0013] Based on the (n-1)th image frame and / or the (n+1)th image frame, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0014] In one embodiment, the method further includes:

[0015] If the nth image frame does not match the first cropping parameter, the expansion direction and / or expansion ratio are determined based on the first cropping parameter.

[0016] The nth image frame is expanded to obtain the nth image frame to be cropped, including:

[0017] Based on the expansion direction and / or expansion ratio, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0018] In one embodiment, the jitter parameters include the jitter amplitude and / or jitter direction of the electronic device; based on the jitter parameters of the electronic device, determining the first cropping parameters for the nth image frame includes:

[0019] Obtain the second cropping parameters for the nm-th image frame; where the cropping parameters are used to indicate the cropping position;

[0020] Based on the jitter amplitude and / or jitter direction, the cutting position indicated by the second cutting parameter is adjusted to obtain the first cutting parameter;

[0021] Among them, the amplitude of shaking is positively correlated with the adjustment amplitude of the cutting position, and the direction of shaking is opposite to the direction of the adjustment of the cutting position.

[0022] In one embodiment, the method further includes:

[0023] The jitter parameters are determined based on the first motion parameters and / or the second motion parameters;

[0024] The first motion parameter is determined based on the deviation between the first position of the same object in the nm-th image frame and the second position in the n-th image frame; the second motion parameter is determined based on the sensing data acquired by the inertial sensor of the electronic device within the first time period.

[0025] In one embodiment, determining the jitter parameters based on a first motion parameter and / or a second motion parameter includes:

[0026] The first motion parameters are weighted based on a preset first weight value to obtain the first weighted motion parameters.

[0027] The second motion parameters are weighted based on the preset second weight value to obtain the second weighted motion parameters.

[0028] The jitter parameters are determined based on the first weighted motion parameters and the second weighted motion parameters;

[0029] Specifically, if the brightness parameter of the nth image frame is within a preset range, the first weight value is greater than the second weight value; if the brightness parameter of the nth image frame is outside the preset range, the first weight value is less than the second weight value.

[0030] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0031] The determination module is configured to, in response to the electronic device acquiring the nth image frame during the video capture function, determine the first cropping parameter of the nth image frame based on the jitter parameter of the electronic device; wherein, the jitter parameter is determined based on the motion parameter of the electronic device during a first time period from after acquiring the nmth image frame to before acquiring the nth image frame;

[0032] The processing module is configured to obtain the nth image frame to be cropped based on the nth image frame, which matches the first cropping parameter, and to perform cropping processing on the nth image frame to be cropped based on the first cropping parameter to obtain the nth target image frame.

[0033] The generation module is configured to generate a target video based on each target image frame obtained during the execution of the video capture function in response to the detection of a shooting end command.

[0034] In one embodiment, the processing module is further configured as follows:

[0035] If the nth image frame matches the first cropping parameter, the nth image frame is determined as the nth image frame to be cropped.

[0036] If the nth image frame does not match the first cropping parameter, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0037] In one embodiment, the processing module is further configured as follows:

[0038] Based on the (n-1)th image frame and / or the (n+1)th image frame, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0039] In one embodiment, the determining module is further configured as follows:

[0040] If the nth image frame does not match the first cropping parameter, the expansion direction and / or expansion ratio are determined based on the first cropping parameter.

[0041] The processing module is also configured as follows:

[0042] Based on the expansion direction and / or expansion ratio, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0043] In one embodiment, the jitter parameters include the jitter amplitude and / or jitter direction of the electronic device; the determining module is further configured to:

[0044] Obtain the second cropping parameters for the nm-th image frame; where the cropping parameters are used to indicate the cropping position;

[0045] Based on the jitter amplitude and / or jitter direction, the cutting position indicated by the second cutting parameter is adjusted to obtain the first cutting parameter;

[0046] Among them, the amplitude of shaking is positively correlated with the adjustment amplitude of the cutting position, and the direction of shaking is opposite to the direction of the adjustment of the cutting position.

[0047] In one embodiment, the determining module is further configured as follows:

[0048] The jitter parameters are determined based on the first motion parameters and / or the second motion parameters;

[0049] The first motion parameter is determined based on the deviation between the first position of the same object in the nm-th image frame and the second position in the n-th image frame; the second motion parameter is determined based on the sensing data acquired by the inertial sensor of the electronic device during the first time period.

[0050] In one embodiment, the processing module is further configured as follows:

[0051] The first motion parameters are weighted based on a preset first weight value to obtain the first weighted motion parameters.

[0052] The second motion parameters are weighted based on the preset second weight value to obtain the second weighted motion parameters.

[0053] The determination module is also configured to: determine jitter parameters based on the first weighted motion parameters and the second weighted motion parameters;

[0054] Specifically, if the brightness parameter of the nth image frame is within a preset range, the first weight value is greater than the second weight value; if the brightness parameter of the nth image frame is outside the preset range, the first weight value is less than the second weight value.

[0055] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0056] processor;

[0057] Memory used to store computer programs or instructions;

[0058] The processor executes computer programs or instructions to implement the steps in any of the image processing methods in the first aspect described above.

[0059] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising:

[0060] When a computer program or instruction in a storage medium is executed by a processor, the steps in any of the image processing methods in the first aspect described above are implemented.

[0061] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of any of the image processing methods in the first aspect described above.

[0062] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0063] In this embodiment of the disclosure, during the video shooting function of the electronic device, the cropping parameters for each image frame can be determined based on the jitter parameters of the electronic device during the time period of acquiring each image frame. Furthermore, regardless of whether the image frame can match the cropping parameters, the image frame to be cropped that matches the cropping parameters can be obtained based on the captured image frame.

[0064] At this point, cropping parameters matching the image frame to be cropped can be used to compensate for image jitter, resulting in visually smooth transitions between target image frames. This allows for the generation of a stable target video based on these visually smooth transitions. This reduces the impact of electronic device jitter on the generated target video, improving its stability.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0067] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0068] Figure 2 This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0069] Figure 3 This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0070] Figure 4 This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0071] Figure 5 This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0072] Figure 6 This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0073] Figure 7 This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0074] Figure 8 This is a structural block of an image processing apparatus according to an exemplary embodiment.

[0075] Figure 9 This is a structural block diagram of an electronic device according to an exemplary embodiment.

[0076] Figure 10 This is a block diagram of an apparatus according to an exemplary embodiment.

[0077] Figure label:

[0078] 1. The nth image frame; 2. The cropping area indicated by the first cropping parameter; 3. The nth target image frame; 4. The nth image frame to be cropped; 5. The expanded image area; 6. The nmth image frame; 7. The cropping area indicated by the second cropping parameter; 8. The nmth target image frame. Detailed Implementation

[0079] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0080] The image processing method shown in this embodiment can be applied to electronic devices with video recording capabilities. Here, the electronic device can include either a mobile electronic device or a fixed electronic device. Mobile electronic devices can include devices such as mobile phones, tablets, laptops, and in-vehicle electronic devices. Fixed electronic devices can include desktop computers, smart TVs, etc.

[0081] In some embodiments, the operating system of the electronic device may include: an Input Output System (IOS) operating system, an Android operating system, etc.

[0082] It should be noted that electronic devices may include, but are not limited to, mobile communication electronic devices, portable entertainment devices, wearable devices, home appliances, augmented reality devices, virtual reality devices, and special-purpose devices. Among these, mobile communication electronic devices may include, but are not limited to, mobile phones, tablets, and smartwatches; portable entertainment devices may include, but are not limited to, digital cameras; wearable devices may include, but are not limited to, smart bracelets and smart glasses; home appliances may include, but are not limited to, televisions and video recorders; augmented reality (AR) devices may include, but are not limited to, AR glasses; virtual reality (VR) devices may include, but are not limited to, VR glasses; and special-purpose devices may include, but are not limited to, professional cameras (such as SLR cameras and point-and-shoot cameras).

[0083] It should be noted that the execution entity of the embodiments of this disclosure can be the central processing unit (CPU) in an electronic device in terms of hardware, and can be, for example, a related background service or application in an electronic device in terms of software, without limitation.

[0084] To better understand the technical solutions in the embodiments of this disclosure, the image processing methods in related technologies are described by way of example below:

[0085] Electronic Image Stabilization (EIS) is an advanced image processing technology specifically designed to reduce image shake caused by unsteady handholds or other external factors during video recording. EIS works by using image processing algorithms to reduce or eliminate video shake caused by instability in the shooting equipment (such as hand tremors or vibrations).

[0086] For example, the process of image processing based on EIS may include the following steps:

[0087] Step 1, Image Capture: When the user presses the shutter button or starts recording, the camera captures still or moving images.

[0088] Step 2, Image Sensor Data Acquisition: The camera's image sensor acquires the brightness information of each pixel in the image and converts it into a digital signal.

[0089] Step 3, Luminance Data Processing: By calculating and comparing luminance data from consecutive frames, EIS can identify camera motion during the capture process. This motion information forms the basis for subsequent image stabilization processing.

[0090] Step 4, Image Stabilization: Based on the results of brightness data processing, EIS uses mathematical algorithms to correct each frame of the image to a stable position. During processing, EIS fine-tunes the position of each pixel according to the raw data from the image sensor, thereby counteracting camera shake. This provides a stable image while preserving the image content.

[0091] Step 5, Image Output: After processing, the stable image will be transmitted to the monitor for the user to view.

[0092] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 1 As shown, the method includes:

[0093] Step 11: In response to the electronic device acquiring the nth image frame during the video capture function, the first cropping parameter of the nth image frame is determined based on the jitter parameter of the electronic device; wherein, the jitter parameter is determined based on the motion parameter of the electronic device during the first time period from after acquiring the nmth image frame to before acquiring the nth image frame; m is greater than or equal to 1.

[0094] In one embodiment, the jitter parameters include the jitter amplitude and / or jitter direction of the electronic device.

[0095] In one embodiment, the cropping parameters include at least one of the following: cropping area, cropping position, and / or cropping ratio. It should be noted that the cropping area can be a region in the image frame that needs to be retained. The cropping ratio can be used to indicate the proportion of the region to be cropped in the image frame. For example, the cropping ratio may include: a first proportion of the cropped area in the image frame, and / or a second proportion of the other regions in the image frame besides the cropped area.

[0096] In one embodiment, the cropping parameters are used to indicate cropping from the edge regions of the image frame. In this case, after cropping the image frame based on the cropping parameters, the central region of the image frame can be preserved.

[0097] For example, cropping parameters can be used to indicate the edge regions of an image frame that can be cropped, and the cropping ratio indicated by the cropping parameters can be 20%. For example, as Figure 2 As shown, for the nth image frame 1, 20% of the edge region can be cropped, retaining 80% of the central region (i.e., the cropping region 2 indicated by the first cropping parameter), resulting in the nth target image frame 3. In the nth target image frame, the human figure is located in the center position.

[0098] In one embodiment, the cropping ratio can be determined separately for each side of the image frame.

[0099] For example, the cropping ratio for the right side of the image frame can be 20%, and the cropping ratio for the left side of the image frame can be 10%. In this case, 20% of the area on the right side of the image frame can be cropped, and 10% of the area on the left side of the image frame can be cropped.

[0100] In one embodiment, a second cropping parameter for the nm-th image frame can be obtained; a first cropping parameter can be determined based on the jitter parameter of the electronic device and the second cropping parameter. Here, the second cropping parameter of the nm-th image frame can be used as a reference cropping parameter, and adapted to the jitter of the electronic device during the first time period from the nm-th image frame to the n-th image frame, as well as the second cropping parameter, the first cropping parameter used for cropping the n-th image frame can be accurately determined.

[0101] In this way, the cropped area in the nth image frame indicated by the first cropping parameter and the cropped area in the nmth image frame indicated by the second cropping parameter can be matched. That is, the content of the image area retained in the nth image frame and the content of the image area retained in the nmth image frame can be matched, thereby reducing image jitter between different image frames during video shooting.

[0102] In one embodiment, if it is determined that the electronic device is not jittering based on the jitter parameters of the electronic device, the second trimming parameter is determined as the first trimming parameter.

[0103] In one embodiment, if it is determined that the electronic device is jittering based on the jitter parameters of the electronic device, the second cutting parameter is adjusted based on the jitter parameters of the electronic device to obtain the first cutting parameter.

[0104] Here, the second cropping parameter of the nm-th image frame can be used as a reference cropping parameter. Furthermore, if the electronic device experiences jitter during the first time period from the nm-th to the n-th image frame, the second cropping parameter can be precisely adjusted based on the jitter parameter of the electronic device during that first time period. This allows the image content corresponding to the object in the image content retained in the nm-th image frame to be cropped from the n-th image frame based on the precisely determined first cropping parameter, even if the object shifts during that first time period. Thus, reducing image jitter between multiple image frames captured during video recording improves the stability of the same object's position in the video.

[0105] In one embodiment, m can be less than a preset threshold number. It should be noted that the fewer the number of image frames between the nm-th image frame and the n-th image frame, the less change there is in the jitter parameters during the first time period from acquiring the nm-th image frame to acquiring the n-th image frame. In this case, the adjustment range of the second cropping parameter based on the jitter parameters for the nm-th image frame is smaller. This simplifies the process of adjusting the second cropping parameter and increases the speed at which the first cropping parameter is obtained based on the jitter parameters.

[0106] In one embodiment, the jitter parameter can be determined based on the motion parameters of the electronic device during a first time period from after acquiring the (n-1)th image frame to before acquiring the nth image frame. That is, m can be 1, and any second cropping parameter in this disclosure can be the cropping parameter of the (n-1)th image frame.

[0107] In one embodiment, in response to the electronic device acquiring a first image frame during video capture, the first image frame is cropped based on preset cropping parameters to obtain a first target image frame. The preset cropping parameters may include at least one of the following: preset cropping position, preset cropping area, and preset cropping ratio.

[0108] For example, preset cropping parameters can instruct the cropping of 20% of the edge region of an image frame, retaining 80% of the center region. For instance, as... Figure 1 As shown, after cropping the first image frame according to the preset cropping parameters, the human figure is centered in the image frame.

[0109] In one embodiment, if the jitter parameters of the electronic device indicate that the jitter amplitude of the electronic device is less than a preset amplitude threshold, the first trimming parameter can be determined based on the jitter parameters of the electronic device and the second trimming parameter.

[0110] In one embodiment, if the jitter parameters of the electronic device indicate that the jitter amplitude of the electronic device is greater than a preset amplitude threshold, the preset cropping parameter can be determined as the first cropping parameter of the nth image frame.

[0111] It should be noted that if the vibration amplitude of the electronic device exceeds a preset amplitude threshold, the subject being photographed by the electronic device may have changed. In this case, it is not necessary to determine the first cropping parameter of the nth image frame based on the cropping parameters of the image frames acquired before the nth image frame, thus improving the accuracy of cropping the nth image frame.

[0112] In one embodiment, the nmth image frame can be the previous image frame that has been cropped using preset cropping parameters.

[0113] Step 12: Based on the nth image frame, obtain the nth image frame to be cropped that matches the first cropping parameter, and perform cropping processing on the nth image frame to be cropped based on the first cropping parameter to obtain the nth target image frame.

[0114] It should be noted that the nth image frame to be cropped that matches the first cropping parameter can mean that the cropping position indicated by the first cropping parameter is located within the nth image frame to be cropped. That is, the cropping position indicated by the first cropping parameter does not exceed the image boundary of the nth image frame to be cropped.

[0115] It should be noted that the nth target image frame can be the cropped region that was not cropped in the nth image frame to be cropped. All target image frames have the same size. That is, when the cropping parameters include a cropping region, the size of the cropping region is the same for all cropping parameters.

[0116] Step 13: In response to the detection of the end-of-capture command, generate the target video based on each target image frame obtained during the execution of the video capture function.

[0117] In one embodiment, the target image frames can be synthesized according to the time sequence in which they are obtained to obtain the target video.

[0118] In one embodiment, a target video with a first data volume can be encoded and / or compressed to obtain video data with a second data volume, wherein the second data volume is smaller than the first data volume. This reduces the data volume of the target video, facilitating its storage and transmission. It should be noted that the target video can be restored by decompressing and / or decoding the video with the second data volume.

[0119] In one embodiment, the target video can be displayed on the display interface. This allows the display of a stabilized target video, providing the user with a stable viewing experience.

[0120] In this embodiment of the disclosure, during the video shooting function of the electronic device, the cropping parameters for each image frame can be determined based on the jitter parameters of the electronic device during the time period of acquiring each image frame. Furthermore, regardless of whether the image frame can match the cropping parameters, the image frame to be cropped that matches the cropping parameters can be obtained based on the captured image frame.

[0121] At this point, cropping parameters matching the cropping parameters can be used to compensate for image jitter, resulting in visually smooth transitions between target image frames. This allows for the generation of a stable target video based on these visually smooth transitions. This reduces the impact of electronic device jitter on the generated target video.

[0122] In one embodiment, obtaining the nth image frame to be cropped based on the nth image frame and matching the first cropping parameter includes:

[0123] If the nth image frame matches the first cropping parameter, the nth image frame is determined as the nth image frame to be cropped.

[0124] If the nth image frame does not match the first cropping parameter, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0125] In related technologies, EIS technology can also accurately capture the device's motion state using built-in inertial sensors (such as gyroscopes and accelerometers), and combined with advanced image processing algorithms, it can effectively counteract the shaky effect by intelligently cropping and shifting stable areas of the image, thereby significantly reducing image blur and shakiness caused by unstable hand-held operation in the final video. However, when faced with significant shakiness, the edge cropping strategy that traditional EIS technology relies on may reach its limit.

[0126] Furthermore, among related technologies, traditional EIS technology lacks the ability to intelligently expand the edges of the image, fill in and optimize newly added pixels, which makes it inadequate when dealing with extreme shaky scenes.

[0127] In one embodiment, a reference image frame can be obtained by performing a geometric transformation on the nth image frame. The nth image frame can then be expanded based on the reference image frame to obtain the nth image frame to be cropped. The geometric transformation process can include at least one of the following: translation, rotation, and mirroring.

[0128] In one embodiment, the reference image frame and the nth image frame can be stitched together to obtain the nth image frame to be cropped.

[0129] It should be noted that although a reference image frame can be obtained by performing geometric transformations on the nth image frame, and then the edges of the nth image frame can be filled based on the reference image frame to expand the image edges and widen the cropping space, the filled pixels do not match the image content of the nth image frame. Furthermore, the image expansion process does not perform semantic information-based expansion on the nth image frame; instead, it simply performs mechanical filling. Therefore, the effect of the filled image frame has a low degree of matching with the actual scene.

[0130] Therefore, in one embodiment, the nth image frame can be subjected to image expansion processing based on an algorithm model. This algorithm model can be an Artificial Intelligence (AI) model.

[0131] Here, an AI-based video stabilization method is introduced. When the shaking amplitude exceeds the conventional cropping range, it can intelligently expand the edge of the image and seamlessly fill and optimize these newly added pixels through AI algorithms, thereby widening the cropping space. Even in extreme shaking scenarios, it can maintain the stability and integrity of the video, achieving a superior stabilization effect that surpasses the limits of traditional cropping.

[0132] For example, such as Figure 3 As shown, when the nth image frame 1 matches the first cropping parameter, that is, when the cropping position calculated by the EIS video stabilization technology does not exceed the image boundary of the nth image frame 1, the nth image frame 1 can be directly determined as the nth image frame to be cropped, and the nth image frame 1 is directly cropped based on the cropping area 2 indicated by the first cropping parameter to obtain the nth target image frame 3 where the person's image is located in the center. Here, the traditional EIS method can be used to crop the nth image frame 1 to maintain video stability.

[0133] For example, such as Figure 4 and Figure 5 As shown, when the nth image frame 1 does not match the first cropping parameter, that is, when the cropping position calculated by EIS technology exceeds the original image boundary, AI image expansion processing can be performed on the nth image frame 1 to obtain the nth image frame to be cropped 4. Then, based on the first cropping parameter, the nth image frame 1 obtained after the image expansion processing is cropped to obtain the nth target image frame 3 where the person is located in the center. In other words, when the cropping position calculated by EIS technology exceeds the original image boundary, the electronic device can automatically execute the AI ​​image expansion function. The gray area in the figure is the expanded area 5 intelligently filled after AI image expansion.

[0134] Here, because the pixels generated by AI can accurately simulate the actual scene, when these AI-enlarged parts are incorporated into the final video frame, users will experience a visual experience that is highly consistent with the actual situation.

[0135] It should be noted that the AI-based video stabilization method disclosed herein has the following advantages:

[0136] 1. Intelligent adaptation to large-scale shaking: This AI-based video stabilization method can effectively address the limitations of traditional EIS technology in handling large-scale shaking. By intelligently expanding the edges of the image, it ensures that the video remains stable and intact even in extreme shaking scenarios.

[0137] 2. AI-Powered Intelligent Image Augmentation: When faced with significant camera shake, traditional methods may lose some image content due to edge cropping reaching its limit. AI-based methods, however, can intelligently augment the image edges. This edge information contains semantic information, rather than being achieved through simple mirroring or flipping. The algorithm seamlessly fills in the newly added pixels, thus maintaining the integrity and continuity of the video.

[0138] 3. Superior image stabilization: By intelligently adjusting the image and filling in new pixels, this technology can maintain video stability even in extremely shaky scenes, achieving superior image stabilization that surpasses the limits of traditional cropping.

[0139] 4. Innovation and Foresight: This method combines advanced AI and image processing technologies, demonstrating technological innovation and foresight, and providing a new direction for the future development of video stabilization technology.

[0140] In one embodiment, if the nth image frame and the first cropping parameter do not match, the nth image frame can be expanded based on the first image information extracted from the nth image frame to obtain the nth image frame to be cropped.

[0141] For example, image information can be used to indicate at least one of the following: texture features, color features, and contour features of an image frame. Image information can also be used to indicate the type of individual objects in an image frame and / or the relative positional relationships between individual objects.

[0142] In one embodiment, if the nth image frame does not match the first cropping parameter, at least one reference image frame that matches the cropping parameter can be determined from the candidate image frames; the nth image frame can be enlarged based on at least one reference image frame. The candidate image frame can be an image frame acquired by the electronic device before the nth image frame during the video capture function, or the candidate image frame can be a target image frame acquired before the nth image frame.

[0143] It should be noted that for each image frame captured during the video recording process, its corresponding cropping parameters can be determined. Image frames that match these cropping parameters without requiring image enlargement can be selected as reference image frames.

[0144] Here, the reference image frame matched with the cropping parameters contains crucial image information that was missing from the nth image frame during the electronic device's jitter. Therefore, by performing image expansion processing on the nth image frame based on the reference frame matched with the cropping parameters, it can be ensured that the newly filled image content in the resulting nth image to be cropped matches the content actually captured during video recording.

[0145] In one embodiment, a first quantity can be determined based on the performance parameters of the electronic device; wherein the performance parameters of the electronic device may be positively correlated with the processing speed of the electronic device; the first quantity is positively correlated with the performance parameters; a first quantity of reference image frames is determined from the candidate image frames; and based on the first quantity of reference image frames, the nth image frame is subjected to image expansion processing.

[0146] Here, the performance parameters of the electronic device can be adapted to accurately determine the first number of reference image frames identified from the candidate image frames. Then, based on the rich image information in the sufficient number of reference image frames, the nth image frame can be expanded, ensuring the accuracy of the expansion process and ensuring that the performance of the electronic device can support the processing of the first number of reference image frames, thus reducing the possibility of slow expansion processing speed.

[0147] In this embodiment of the disclosure, if the nth image frame does not match the first cropping parameter, the nth image frame can be enlarged to increase the size of the image frame and fill in the missing image content in the nth image frame, thereby obtaining the nth image frame to be cropped that can match the first cropping parameter.

[0148] At this point, regardless of whether the nth image frame matches the first cropping parameter, an nth image frame to be cropped that matches the first cropping parameter can be obtained based on the nth image frame. Therefore, the nth image frame to be cropped can be cropped based on the first cropping parameter to obtain a target image frame capable of inversely compensating for the jitter of the electronic device. This reduces image jitter between adjacent image frames, and allows for the generation of a more stable target video based on the acquired target image frames.

[0149] In one embodiment, the nth image frame is subjected to image expansion processing to obtain the nth image frame to be cropped, including:

[0150] Based on the (n-1)th image frame and / or the (n+1)th image frame, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0151] In one embodiment, the nth image frame can be expanded based on the second image information extracted from the (n-1)th image frame and / or the third image information extracted from the (n+1)th image frame to obtain the nth image frame to be cropped.

[0152] In one embodiment, in response to the deviation between the time when the first cropping parameter is determined and the time when the (n+1)th image frame is acquired being less than a preset deviation threshold, the nth image frame can be expanded based on the nth image frame and / or the (n+1)th image frame to obtain the nth image frame to be cropped. Here, it is unnecessary to spend excessive time waiting to acquire the (n+1)th image frame after determining the first cropping parameter, thereby improving the speed of expanding the nth image frame based on image frames adjacent to the nth image frame.

[0153] In one embodiment, the nth image frame can be expanded based on the target image frame obtained before the nth image frame to obtain the nth image frame to be cropped.

[0154] For example, the nth image frame can be expanded based on the (n-1)th target image frame to obtain the nth image frame to be cropped.

[0155] For example, the nth image frame can be expanded based on the fourth image information extracted from the (n-1)th target image frame to obtain the nth image frame to be cropped.

[0156] In one embodiment, a comparison can be made between the nth image frame and the (n-1)th and / or (n+1)th image frames to obtain a comparison result. This comparison result can be used to indicate a target image region, which can be an image region present in the (n-1)th and / or (n+1)th image frames, and can also be an image region not present in the nth image frame. Based on the target image region indicated by the comparison result, the nth image frame can be expanded to obtain the nth image frame to be cropped.

[0157] For example, the nth image frame can be expanded based on the fifth image information extracted from the target image region to obtain the nth image frame to be cropped.

[0158] It should be noted that the target image region may contain key image information missing from each of the target image frames in the nth image frame. By performing image expansion processing on the nth image frame based on the target region, the key image information missing in the nth image frame can be completed, resulting in the nth image frame to be cropped that matches the image information in the target image frame.

[0159] In this embodiment, the nth image frame can be expanded based on adjacent image frames to improve the accuracy of supplementing missing image information in the nth image frame, resulting in an nth image frame to be cropped that matches the first cropping parameter. Thus, by cropping the nth image frame, which is highly adapted to the actual shooting scene, based on the first cropping parameter, compensation for electronic device shake can be achieved, thereby reducing image shake and ensuring the authenticity of the obtained nth target image frame.

[0160] In one embodiment, the method further includes:

[0161] If the nth image frame does not match the first cropping parameter, the expansion direction and / or expansion ratio are determined based on the first cropping parameter.

[0162] The nth image frame is expanded to obtain the nth image frame to be cropped, including:

[0163] Based on the expansion direction and / or expansion ratio, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0164] In one embodiment, the expansion direction and / or expansion ratio can be determined based on the cropping position indicated by the first cropping parameter. For example, the cropping position indicated by the first cropping parameter extends beyond the left edge of the nth image frame. In this case, the expansion direction can be the leftward direction of the nth image frame. The left region of the nth image frame can be expanded based on the expansion direction.

[0165] In one embodiment, when the cropping position indicated by the first cropping parameter exceeds the image boundary of the nth image frame, the image expansion ratio can be positively correlated with the extent to which the cropping position indicated by the first cropping parameter exceeds the image boundary of the nth image frame. For example, the cropping position indicated by the first cropping parameter exceeds the left image boundary of the nth image frame by f pixels. In this case, the image expansion ratio can be used to indicate the expansion of the left region of the nth image frame by f pixels.

[0166] In one embodiment, the image expansion ratio can be determined based on the size of the region where the cropping area, indicated by the first cropping parameter, extends beyond the image boundary of the nth image frame. For example, as... Figure 5As shown, the size of the cropped region 2, indicated by the first cropping parameter, which extends beyond the image boundary of the nth image frame 1, can be the same as the size of the expanded region 5, indicated by the expansion ratio. In this case, there is no need to waste computing resources on expanding the non-cropped region.

[0167] In one embodiment, the nth image frame can be expanded based on a preset expansion direction and / or a preset expansion ratio to obtain the nth image frame to be cropped. This eliminates the need to consume computational resources to calculate the expansion direction and / or expansion ratio.

[0168] For example, the preset expansion ratio can be 20% of the nth image frame. The preset expansion direction can indicate that the expansion processing is performed outside the image boundaries on all sides of the image frame. For example, the expansion processing can be performed above, to the left, below, and to the right of the image frame.

[0169] In one embodiment, an algorithm model can be used to expand the image of the nth image frame. A preset expansion ratio can be determined based on the model parameters of the algorithm model. The preset expansion ratio can be positively correlated with the amount of data in the model parameters and / or the accuracy of the algorithm model. That is, the preset expansion ratio can be positively correlated with the expansion capability of the algorithm model. Here, the expansion ratio can be precisely determined according to the expansion capability of the algorithm model, thereby ensuring the reliability of the expansion processing of the nth image frame while reducing the possibility of low accuracy in the nth image frame to be cropped after the algorithm model has performed the expansion processing.

[0170] In one embodiment, the image expansion ratio can be less than or equal to a preset first ratio threshold. For example, the first ratio threshold can be 20% of the image frame size. This can reduce the possibility of the expanded image content not matching the actual scene due to an excessively large expansion ratio.

[0171] In one embodiment, in response to detecting a preset input operation, the image expansion ratio can be determined based on the preset input operation. Here, the image expansion ratio can be customized based on the content of user input to improve the flexibility of image frame expansion processing.

[0172] In this embodiment of the disclosure, an expansion direction and / or expansion ratio adapted to the first cropping parameter can be determined, thereby improving the accuracy of the expansion processing of the nth image frame, reducing the invalid expansion of areas that do not need to be cropped, and increasing the speed of the expansion processing of the nth image frame.

[0173] In one embodiment, the jitter parameters include the jitter amplitude and / or jitter direction of the electronic device; based on the jitter parameters of the electronic device, determining the first cropping parameters for the nth image frame includes:

[0174] Obtain the second cropping parameters for the nm-th image frame; where the cropping parameters are used to indicate the cropping position;

[0175] Based on the jitter amplitude and / or jitter direction, the cutting position indicated by the second cutting parameter is adjusted to obtain the first cutting parameter;

[0176] Among them, the amplitude of shaking is positively correlated with the adjustment amplitude of the cutting position, and the direction of shaking is opposite to the direction of the adjustment of the cutting position.

[0177] For example, such as Figure 6 As shown, in the nm-th image frame 6, the human figure is located in the center position. The second cropping parameter can instruct that 20% of the edge region of the nm-th image frame 6 be cropped, while retaining 80% of the central region of the nm-th image frame 6 (i.e., as shown). Figure 6 The cropping region 7 is indicated by the second cropping parameter shown. The dashed box represents the nm-th image frame 6, and the solid box represents the nm-th target image frame 8 obtained after cropping. At this point, the human figure is centered in the nm-th target image frame 8.

[0178] When the electronic device vibrates, the second position of the portrait in the nth image frame 1 will shift compared to the first position of the portrait in the nmth image frame 6. For example, the second position of the portrait in the nth image frame 1 may shift to the left compared to the first position of the portrait in the nmth image frame 6. In this case, the second cropping parameter can be adjusted based on the vibration parameter of the electronic device. Compared to the cropping position of the cropping area 7 indicated by the second cropping parameter, the cropping position of the cropping area 2 indicated by the first cropping parameter may shift to the left. Here, after cropping the nth image frame 1 based on the first cropping parameter, the portrait can still be located at the center position of the nth target image frame 3.

[0179] For example, such as Figure 7 As shown, in the nm-th image frame 6, the human figure is located in the center. The second cropping parameter can instruct the cropping of 20% of the right edge region of the nm-th image frame 6, while retaining 80% of the left edge region of the nm-th image frame 6 (i.e., as shown). Figure 7 The cropping region 7 is indicated by the second cropping parameter shown. After cropping the nm-th image frame 6 based on the second cropping parameter, the nm-th target image frame 8 can be obtained. In the nm-th target image frame 8, the human figure is centered.

[0180] When the electronic device shakes to the right, the second position of the portrait in the nth image frame 1 will shift to the left compared to the first position of the portrait in the nmth image frame 6. At this time, the first cropping parameter can be determined based on the shaking parameter of the electronic device and the second cropping parameter, and the cropping position of the cropping region 2 indicated by the first cropping parameter can shift to the left compared to the cropping position of the cropping region 7 indicated by the second cropping parameter. If the cropping position of the cropping region 2 exceeds the image boundary of the nth image frame 1, the nth image frame 1 can be expanded to obtain the nth image frame to be cropped 4. The nth image frame to be cropped 4 has an expanded region 5. Here, after cropping the nth image frame to be cropped based on the first cropping parameter, the nth target image frame 3 is obtained. The portrait can still be located at the center position of the nth target image frame 3.

[0181] like Figure 6 and Figure 7 As shown, although the video was captured with unstable images due to the shaking of the electronic device, after video stabilization, the image remained firmly centered in every frame, achieving a visually smooth and shaky experience.

[0182] In this embodiment of the disclosure, the second cropping parameter of the nm-th image frame can be precisely adjusted based on the jitter amplitude and / or jitter direction during a first time period from when the electronic device acquires the nm-th image frame to when it acquires the n-th image frame, so as to obtain the first cropping parameter for compensating for the jitter generated by the electronic device. Then, after cropping the n-th image to be cropped based on the first cropping parameter, the nm-th target image frame and the n-th target image frame with visually stable transition can be obtained.

[0183] In one embodiment, the method further includes:

[0184] The jitter parameters are determined based on the first motion parameters and / or the second motion parameters;

[0185] The first motion parameter is determined based on the deviation between the first position of the same object in the nm-th image frame and the second position in the n-th image frame; the second motion parameter is determined based on the sensing data acquired by the inertial sensor of the electronic device during the first time period.

[0186] In one embodiment, after acquiring the nth image frame, a preprocessing operation can be performed on the nth image frame to obtain a processed nth image frame. Based on the processed nth image frame, an nth image frame to be cropped matching the first cropping parameter can be obtained. And / or, the first motion parameter can be determined based on the deviation between the first position of the same object in the processed nmth image frame and the second position in the processed nth image frame. It should be noted that the first motion parameter can indicate the inter-frame motion information between the nmth and nth image frames.

[0187] In one embodiment, the first motion parameters may include translation parameters and / or rotation parameters. Translation parameters include translation direction and / or translation distance. Translation parameters can be used to indicate the translation direction and / or translation distance required to translate from a first position to a second position corresponding to the same object. Rotation parameters include rotation angles. Rotation parameters can be used to indicate the rotation angle required to rotate from a first position to a second position corresponding to the same object.

[0188] In one embodiment, preprocessing can be performed on each image frame acquired during the video capture function of the electronic device to obtain processed image frames.

[0189] In one embodiment, the preprocessing operation may include, but is not limited to, at least one of the following: noise reduction, white balance adjustment, and resolution adjustment. Thus, by preprocessing each image frame acquired during video capture, the accuracy of cropping the processed image frames can be improved.

[0190] In one embodiment, the inertial sensor can be calibrated when the electronic device first drives it. For example, a driver program can be integrated into the electronic device to drive the inertial sensor. This ensures that the inertial sensor can be driven normally using the driver program. Furthermore, the inertial sensor can be calibrated after it has been driven. This ensures that the inertial sensor can accurately detect the motion state of the device.

[0191] In one embodiment, the inertial sensor may include a gyroscope sensor and / or an accelerometer.

[0192] In one embodiment, sensing data can be acquired in real time based on an inertial sensor. It should be noted that the real-time acquired sensing data can be stored in the electronic device. This ensures the accuracy and real-time nature of the data, facilitates subsequent retrieval of sensing data from the electronic device, and enables accurate analysis of the electronic device's historical second motion parameters based on the sensing data.

[0193] In one embodiment, a second motion parameter of the electronic device during the first time period is determined based on sensing data acquired by an inertial sensor during the first time period. The sensing data includes the acceleration and / or angular velocity of the electronic device. The second motion parameter may include the displacement parameter and / or rotation angle of the electronic device. The displacement parameter may include the displacement direction and / or displacement distance of the electronic device.

[0194] In this embodiment, the jitter parameters of the electronic device can be determined by analyzing the sensing data acquired by the inertial sensor in the first time period and the difference between the nm-th and n-th image frames acquired during video recording. This allows for the quantization of motion vectors between different image frames acquired during video recording based on the jitter parameters. In this way, traditional electronic image stabilization algorithms can be applied to intelligently crop each image frame according to the jitter parameters of the electronic device, ensuring that the main content is located in a stable area of ​​the image, thereby offsetting the impact of device jitter.

[0195] In one embodiment, the first motion parameter is weighted based on a preset first weight value to obtain a first weighted motion parameter; the second motion parameter is weighted based on a preset second weight value to obtain a second weighted motion parameter; and the jitter parameter is determined based on the first weighted motion parameter and the second weighted motion parameter.

[0196] In one embodiment, the first weight value and the second weight value are different. Here, in the process of determining the jitter parameters of an electronic device based on various motion parameters, different weight values ​​can be configured for different motion parameters to flexibly determine the proportion of each motion parameter in the process of determining the jitter parameters, thereby ensuring the accuracy of the determined jitter parameters.

[0197] In one embodiment, determining the jitter parameters based on a first motion parameter and / or a second motion parameter includes:

[0198] The first motion parameters are weighted based on a preset first weight value to obtain the first weighted motion parameters.

[0199] The second motion parameters are weighted based on the preset second weight value to obtain the second weighted motion parameters.

[0200] The jitter parameters are determined based on the first weighted motion parameters and the second weighted motion parameters;

[0201] Specifically, if the brightness parameter of the nth image frame is within a preset range, the first weight value is greater than the second weight value; if the brightness parameter of the nth image frame is outside the preset range, the first weight value is less than the second weight value.

[0202] In one embodiment, the brightness parameter of the nth image frame can be the brightness parameter of the shooting environment in which the electronic device is located during the acquisition of the nth image frame. Alternatively, the image feature parameters of the image frame can be extracted from the nth image frame, and the brightness parameter of the nth image frame can be determined based on these image feature parameters.

[0203] In this embodiment, on the one hand, when the brightness parameter of the nth image frame is within a preset range, the probability of the nth image frame captured by the electronic device being overexposed or underexposed is relatively small. At this time, the image quality of the nth image frame is good, and the loss of detail information of each object in the nth image frame will not occur due to overexposure or underexposedness. Therefore, the accuracy of the first motion parameter determined based on the high-quality nth image frame is relatively high. This can be achieved by configuring a higher first weight value for the first motion parameter, increasing its proportion in the process of determining the jitter parameters based on each motion parameter, thereby improving the accuracy of the determined jitter parameters.

[0204] On the other hand, if the brightness parameter of the nth image frame is outside the preset range, the probability of the nth image frame being overexposed or underexposed is relatively high. In this case, the image quality of the nth image frame is poor, and the overexposure or underexposure may cause a loss of detail information about objects in the nth image frame. Therefore, the accuracy of the first motion parameter determined based on the poor-quality nth image frame may be lower than the accuracy of the second motion parameter determined based on the sensing data obtained from the inertial sensor. In this situation, by assigning a higher second weight value to the second motion parameter, the proportion of the second motion parameter in the process of determining the jitter parameter based on each motion parameter can be increased, thereby improving the accuracy of the determined jitter parameter.

[0205] In one embodiment, a first weight value and / or a second weight value can be determined based on the shooting scene in which the electronic device is located. The shooting scene indicates at least one of the following: the shooting time, the subject being photographed, and / or the shooting weather.

[0206] It should be noted that the shooting scene and the quality of the image frames acquired during video recording are related, and the quality of the image frames is related to the accuracy of the first motion parameters determined based on the image frames. Here, we can adapt to the shooting scene of the electronic device, that is, adapt to the accuracy of the first motion parameters determined based on the image frames, and flexibly determine the proportion of each motion parameter in the process of determining the shake parameters, thereby ensuring the accuracy of the determined shake parameters.

[0207] It should be noted that the sum of the first weight value and the second weight value can be 1. In this case, as long as the first weight value or the second weight value is determined, the other weight value can be obtained.

[0208] For example, in a shooting scenario where an electronic device is taking pictures in hazy weather, the second weight value can be greater than the first weight value. It is understood that the quality of image frames acquired in hazy weather may be poor, causing the accuracy of the first motion parameter obtained based on that image frame to be less than the accuracy of the second motion parameter determined based on sensing data from an inertial sensor. In this case, the first weight configured for the first motion parameter can be less than the second weight value configured for the second motion parameter, thereby improving the accuracy of the determined jitter parameters when determining jitter parameters based on each motion parameter.

[0209] In one embodiment, when the type of the object being photographed is the target type, the first weight value is less than the second weight value. When the type of the object being photographed is not the target type, the second weight value is greater than the first weight value. In the nth image frame, the proportion of solid color areas in the image region where the target type object is located is greater than a preset second proportion threshold.

[0210] For example, the target type object can be a white wall. When the electronic device is shooting a scene where the target object is a white wall, the second weight value can be greater than the first weight value. It should be noted that the proportion of solid color areas in the image frame obtained for the target type object is relatively large, resulting in less detail information in the image frame. Therefore, the accuracy of the first motion parameter determined based on the image frame with less detail information may be lower than the accuracy of the second motion parameter determined based on the sensing data of the inertial sensor. In this case, the first weight value configured for the first motion parameter can be smaller than the second weight value configured for the second motion parameter, thereby improving the accuracy of the determined jitter parameter when determining the jitter parameter based on each motion parameter.

[0211] In one embodiment, a first weight value can be determined based on the resolution of the nth image frame and / or the nmth image frame. The first weight value can be positively correlated with the resolution of the image frame.

[0212] In one embodiment, if the resolution of the nth image frame and / or the nmth image frame is greater than a preset resolution threshold, the first weight value is greater than the second weight value. If the resolution of the nth image frame and / or the nmth image frame is less than the preset resolution threshold, the first weight value is less than the second weight value.

[0213] It should be noted that the higher the resolution of the image frame, the more detailed information it contains, and the more accurate the first motion parameters obtained based on the image frame. Therefore, appropriate weight values ​​can be flexibly configured for the first motion parameters determined based on the image frame resolution. In this way, the accuracy of the determined jitter parameters can be improved when determining jitter parameters based on various motion parameters.

[0214] To better understand the embodiments of this disclosure, this disclosure provides an image processing method, which includes the following steps:

[0215] Step 21: Based on the sensing data detected by the inertial sensor and the inter-frame motion information between the (n-1)th image frame and the nth image frame, determine the jitter parameters of the electronic device.

[0216] Step 22: Determine the first cutting parameters based on the jitter parameters of the electronic device.

[0217] Step 22: If the cropping range indicated by the first cropping parameter does not exceed the image boundary of the nth image frame, crop the nth image frame based on the first cropping parameter to obtain the nth target image frame.

[0218] Step 23: If the cropping range indicated by the first cropping parameter exceeds the image boundary of the nth image frame, activate the AI-assisted module.

[0219] Step 24: Based on the algorithm in the AI-assisted module, evaluate the differences between the nth image frame and the (n-1)th image frame, as well as the differences between the nth image frame and the (n+1)th image frame, and identify key visual information regions that may be lost due to the jitter of electronic devices.

[0220] Step 25: The AI-assisted module may include an AI image expansion model. A deep learning model (i.e., the AI ​​image expansion model) can be used to expand the image of the nth image frame based on the image content of the (n-1)th and / or the (n+1)th image frame, obtaining the nth image region to be cropped. At this point, by expanding the image of the nth image frame, key image information missing in the nth image frame can be supplemented.

[0221] Here, an AI-powered image expansion model can intelligently predict and generate missing edge pixels in the nth image frame, expanding the image edges and ensuring content continuity and naturalness. The expansion ratio can be customized, for example, set to 20% of the nth image size. It's important to note that an excessively large expansion ratio may produce results that don't reflect reality. The expansion ratio can be determined based on the AI ​​image expansion model's capabilities. If the AI ​​model has strong expansion capabilities, a larger expansion ratio can be set.

[0222] Step 26: Based on the first cropping parameter, crop the nth image region to be cropped to obtain the nth target image frame.

[0223] Here, the nth image to be cropped, obtained after image expansion processing using the AI ​​image expansion model, can be reintroduced into the EIS process. Combined with intelligent cropping and translation, residual jitter can be further eliminated.

[0224] In this disclosure, the cropping space can be expanded based on the filled and optimized image to accommodate situations with significant shaking. This maintains video stability and integrity even in extremely shaky scenarios.

[0225] The image processing method in this disclosure has the following advantages:

[0226] 1. Significantly improves image stabilization performance:

[0227] Compared to traditional EIS technology, this technology introduces an AI-based video stabilization method that can intelligently expand the edges of the image and seamlessly fill and optimize newly added pixels through AI algorithms, thereby maintaining the stability and integrity of the video even when faced with significant shaking.

[0228] 2. Enhance user experience:

[0229] When users are recording videos, various factors (such as unstable handheld shooting or external environmental interference) can cause image shakiness, affecting the viewing experience. This technical solution effectively reduces image blurring and shaking caused by shakiness, resulting in a more stable and clearer final video, greatly improving the user's viewing experience.

[0230] 3. Expanding application scenarios:

[0231] Traditional EIS technology may have limited effectiveness in stabilizing large amounts of shaking. However, this solution, through the introduction of AI technology, achieves superior stabilization even in extremely shaky scenarios, thus expanding the application scenarios of video stabilization technology to occasions requiring high levels of image stabilization, such as sports photography and outdoor adventure.

[0232] 4. Improve video quality:

[0233] This technical solution uses AI algorithms to seamlessly fill in and optimize newly added pixels, ensuring the image maintains high quality even after expansion. This not only enhances the visual effects of the video but also provides greater flexibility and convenience for post-production editing and processing.

[0234] 5. Cost-effectiveness:

[0235] Compared to optical image stabilization (OIS) technology, which relies entirely on expensive optical stabilization components, AI-based electronic image stabilization (EIS) technology can provide stabilization effects that are close to or even surpass those of some OIS systems without significantly increasing hardware costs, making it more cost-effective for consumers.

[0236] In one embodiment, in response to the electronic device acquiring the nth image frame during video recording and enabling video stabilization, a first cropping parameter for the nth image frame is determined based on the shake parameters of the electronic device. The shake parameters are determined based on motion parameters of the electronic device during a first time period from acquiring the (n-m1)th image frame to acquiring the nth image frame, where m is greater than or equal to 1. Based on the nth image frame, an nth image frame to be cropped matching the first cropping parameter is obtained, and the nth image frame to be cropped is cropped based on the first cropping parameter to obtain the nth target image frame. In response to detecting a recording end command, a target video is generated based on each target image frame acquired during video recording. It is understood that the video stabilization function can be a function that performs stabilization processing on each image frame acquired by the electronic device during video recording based on the shake parameters of the electronic device. When the video stabilization function is enabled, any image processing method of this disclosure will be executed.

[0237] In one embodiment, video stabilization is activated in response to the electronic device being in a preset shooting scene.

[0238] For example, the preset shooting scene may include, but is not limited to, at least one of the following shooting scenes: sports shooting scene, aerial shooting scene, surveillance scene, and live streaming scene.

[0239] For example, sports shooting scenarios can include sports videography and outdoor adventure scenes. In high-intensity sports shooting scenarios such as skiing, mountain biking, and hiking, conventional EIS is difficult to handle the shooting situation of severe turbulence, while AI image stabilization technology can ensure that clear and stable video footage can still be obtained in the midst of violent movement, recording every dynamic detail.

[0240] For example, the aerial photography scene could be a scene captured by a drone. In such a scene, the drone is susceptible to wind during flight, causing it to shake. AI-based electronic image stabilization technology can effectively reduce image shake caused by airflow, resulting in smoother and more natural aerial landscape footage.

[0241] For example, a live streaming scenario could be a shooting scene for news reporting or live broadcasting. In a live streaming scenario, even if the reporter or anchor moves quickly and reports live, this technology can still ensure stable footage, enhancing the professionalism of instant communication and the viewing experience for the audience.

[0242] For example, in surveillance scenarios where monitoring and security management are carried out through surveillance cameras, especially in outdoor environments or environments where electronic devices vibrate frequently, AI-based video stabilization technology can enhance the clarity and tracking accuracy of video surveillance and improve the efficiency of security monitoring.

[0243] For example, video stabilization can also be applied to shooting scenarios involving mobile photography and video log (Vlog) production. As smartphones become the primary tool for everyday shooting, AI stabilization technology allows ordinary users to easily shoot professional-grade stable videos without the need for external stabilizers, making it suitable for users who need to shoot while moving.

[0244] For example, video stabilization can also be applied to film and television production and advertising shooting scenarios. In professional film and television production, especially in handheld shooting and alternatives to professional camera equipment, this technology can provide photographers with more creative freedom while maintaining high-quality image output.

[0245] In conclusion, this AI-based electronic image stabilization technology, due to its unique advantages, can be widely applied in various scenarios to improve video quality and viewing experience. Furthermore, by overcoming the limitations of traditional EIS under significant shaking, AI-based video stabilization not only improves video stability in various shooting environments but also brings broader creative possibilities and higher video quality standards to different industries and application scenarios.

[0246] Figure 8 This is an image processing apparatus exemplarily shown according to an embodiment of the present disclosure, the apparatus comprising:

[0247] The determining module 81 is configured to, in response to the electronic device acquiring the nth image frame during the video capture function, determine the first cropping parameter of the nth image frame based on the jitter parameter of the electronic device; wherein, the jitter parameter is determined based on the motion parameter of the electronic device during a first time period from after acquiring the nmth image frame to before acquiring the nth image frame;

[0248] The processing module 82 is configured to obtain the nth image frame to be cropped based on the nth image frame and match the first cropping parameter, and to perform cropping processing on the nth image frame to be cropped based on the first cropping parameter to obtain the nth target image frame.

[0249] The generation module 83 is configured to generate a target video based on each target image frame obtained during the execution of the video shooting function in response to the detection of a shooting end command.

[0250] In one embodiment, the processing module 82 is further configured to:

[0251] If the nth image frame matches the first cropping parameter, the nth image frame is determined as the nth image frame to be cropped.

[0252] If the nth image frame does not match the first cropping parameter, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0253] In one embodiment, the processing module 82 is further configured to:

[0254] Based on the (n-1)th image frame and / or the (n+1)th image frame, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0255] In one embodiment, the determining module 81 is further configured as follows:

[0256] If the nth image frame does not match the first cropping parameter, the expansion direction and / or expansion ratio are determined based on the first cropping parameter.

[0257] Processing module 82 is also configured as follows:

[0258] Based on the expansion direction and / or expansion ratio, the nth image frame is expanded to obtain the nth image frame to be cropped.

[0259] In one embodiment, the jitter parameters include the jitter amplitude and / or jitter direction of the electronic device; the determining module 81 is further configured to:

[0260] Obtain the second cropping parameters for the nm-th image frame; where the cropping parameters are used to indicate the cropping position;

[0261] Based on the jitter amplitude and / or jitter direction, the cutting position indicated by the second cutting parameter is adjusted to obtain the first cutting parameter;

[0262] Among them, the amplitude of shaking is positively correlated with the adjustment amplitude of the cutting position, and the direction of shaking is opposite to the direction of the adjustment of the cutting position.

[0263] In one embodiment, the determining module 81 is further configured as follows:

[0264] The jitter parameters are determined based on the first motion parameters and / or the second motion parameters;

[0265] The first motion parameter is determined based on the deviation between the first position of the same object in the nm-th image frame and the second position in the n-th image frame; the second motion parameter is determined based on the sensing data acquired by the inertial sensor of the electronic device during the first time period.

[0266] In one embodiment, the processing module 82 is further configured to:

[0267] The first motion parameters are weighted based on a preset first weight value to obtain the first weighted motion parameters.

[0268] The second motion parameters are weighted based on the preset second weight value to obtain the second weighted motion parameters.

[0269] The determination module 81 is further configured to: determine the jitter parameters based on the first weighted motion parameters and the second weighted motion parameters;

[0270] Specifically, if the brightness parameter of the nth image frame is within a preset range, the first weight value is greater than the second weight value; if the brightness parameter of the nth image frame is outside the preset range, the first weight value is less than the second weight value.

[0271] Figure 9 This is a structural block diagram illustrating an electronic device 900 according to an exemplary embodiment. For example, the electronic device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0272] Reference Figure 9 The electronic device 900 may include one or more of the following components: processing component 902, memory 904, power supply component 906, multimedia component 908, audio component 910, input / output (I / O) interface 912, sensor component 914, and communication component 916.

[0273] Processing component 902 typically controls the overall operation of electronic device 900, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0274] Memory 904 is configured to store various types of data to support the operation of electronic device 900. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 900, contact data, phonebook data, messages, pictures, and videos. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0275] Power supply component 906 provides power to various components of electronic device 900. Power supply component 906 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 900.

[0276] Multimedia component 908 includes a screen that provides an output interface between electronic device 900 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0277] In some embodiments, the multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and rear-facing camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0278] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when electronic device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0279] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0280] Sensor assembly 914 includes one or more sensors for providing state assessment of various aspects of electronic device 900. For example, sensor assembly 914 can detect the on / off state of electronic device 900, the relative positioning of components such as the display and keypad of electronic device 900, changes in position of electronic device 900 or one of its components, the presence or absence of user contact with electronic device 900, orientation or acceleration / deceleration of electronic device 900, and temperature changes of electronic device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include an optical sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications.

[0281] In some embodiments, the sensor assembly 914 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, and a temperature sensor.

[0282] Communication component 916 is configured to facilitate wired or wireless communication between electronic device 900 and other devices. Electronic device 900 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 9G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0283] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0284] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including executable instructions or a computer program, which can be executed by a processor 920 of an electronic device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0285] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile electronic device, enables the mobile electronic device to perform any of the image processing methods described above in the embodiments of this disclosure. For example, the image processing method includes:

[0286] In response to the acquisition of the nth image frame by the electronic device during the video recording function, the first cropping parameter of the nth image frame is determined based on the jitter parameter of the electronic device; wherein, the jitter parameter is determined based on the motion parameter of the electronic device during the first time period from after the acquisition of the nmth image frame to before the acquisition of the nth image frame; m is greater than or equal to 1;

[0287] Based on the nth image frame, obtain the nth image frame to be cropped that matches the first cropping parameter, and perform cropping processing on the nth image frame to be cropped based on the first cropping parameter to obtain the nth target image frame;

[0288] In response to the detection of a shooting end command, a target video is generated based on each target image frame obtained during the execution of the video shooting function.

[0289] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image processing methods described above in this disclosure.

[0290] Figure 10 This is a block diagram illustrating a display device 1000 according to an exemplary embodiment. For example, device 1000 may be provided as a server. (Refer to...) Figure 10 The apparatus 1000 includes a processing component 1022, which further includes one or more processors, and memory resources represented by memory 1032 for storing instructions, such as application programs, that can be executed by the processing component 1022. The application programs stored in memory 1032 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1022 is configured to execute instructions to perform the aforementioned image processing method:

[0291] In response to the acquisition of the nth image frame by the electronic device during the video recording function, the first cropping parameter of the nth image frame is determined based on the jitter parameter of the electronic device; wherein, the jitter parameter is determined based on the motion parameter of the electronic device during the first time period from after the acquisition of the nmth image frame to before the acquisition of the nth image frame; m is greater than or equal to 1;

[0292] Based on the nth image frame, obtain the nth image frame to be cropped that matches the first cropping parameter, and perform cropping processing on the nth image frame to be cropped based on the first cropping parameter to obtain the nth target image frame;

[0293] In response to the detection of a shooting end command, a target video is generated based on each target image frame obtained during the execution of the video shooting function.

[0294] Device 1000 may also include a power supply component 1026 configured to perform power management of device 1000, a wired or wireless network interface 1050 configured to connect device 1000 to a network, and an input / output (I / O) interface 1058. Device 1000 can operate an operating system stored in memory 1032, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0295] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0296] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: In response to the electronic device acquiring the nth image frame during the video recording process, a first cropping parameter for the nth image frame is determined based on the jitter parameters of the electronic device; wherein, the jitter parameters are determined based on the motion parameters of the electronic device during the time period from acquiring the nmth image frame to acquiring the nth image frame; m is greater than or equal to 1; Based on the nth image frame, an nth image frame to be cropped that matches the first cropping parameter is obtained, and the nth image frame to be cropped is cropped based on the first cropping parameter to obtain the nth target image frame; In response to the detection of a shooting end command, a target video is generated based on each target image frame obtained during the execution of the video shooting function.

2. The image processing method according to claim 1, characterized in that, The step of obtaining the nth image frame to be cropped based on the nth image frame and matching the first cropping parameters includes: If the nth image frame matches the first cropping parameter, the nth image frame is determined as the nth image frame to be cropped. If the nth image frame does not match the first cropping parameter, the nth image frame is enlarged to obtain the nth image frame to be cropped.

3. The image processing method according to claim 2, characterized in that, The step of performing image expansion processing on the nth image frame to obtain the nth image frame to be cropped includes: Based on the (n-1)th image frame and / or the (n+1)th image frame, the nth image frame is expanded to obtain the nth image frame to be cropped.

4. The image processing method according to claim 2 or 3, characterized in that, The method further includes: If the nth image frame does not match the first cropping parameter, the expansion direction and / or expansion ratio are determined based on the first cropping parameter. The step of performing image expansion processing on the nth image frame to obtain the nth image frame to be cropped includes: Based on the expansion direction and / or the expansion ratio, the nth image frame is subjected to expansion processing to obtain the nth image frame to be cropped.

5. The image processing method according to claim 1, characterized in that, The jitter parameters include the jitter amplitude and / or jitter direction of the electronic device; Determining the first cropping parameter of the nth image frame based on the jitter parameters of the electronic device includes: Obtain the second cropping parameter of the nm-th image frame; wherein the cropping parameter is used to indicate the cropping position; Based on the jitter amplitude and / or the jitter direction, the cutting position indicated by the second cutting parameter is adjusted to obtain the first cutting parameter; The amplitude of the shaking is positively correlated with the adjustment amplitude of the cutting position, and the direction of shaking is opposite to the direction of adjustment of the cutting position.

6. The image processing method according to claim 1, characterized in that, The method further includes: The jitter parameters are determined based on the first motion parameters and / or the second motion parameters; The first motion parameter is determined based on the deviation between the first position of the same object in the nm-th image frame and the second position in the n-th image frame; the second motion parameter is determined based on the sensing data acquired by the inertial sensor of the electronic device during the time period.

7. The image processing method according to claim 6, characterized in that, Determining the jitter parameters based on the first motion parameter and / or the second motion parameter includes: The first motion parameters are weighted based on a preset first weight value to obtain the first weighted motion parameters. The second motion parameters are weighted based on a preset second weight value to obtain the second weighted motion parameters. The jitter parameters are determined based on the first weighted motion parameters and the second weighted motion parameters; Wherein, if the brightness parameter of the nth image frame is within a preset range, the first weight value is greater than the second weight value; if the brightness parameter of the nth image frame is outside the preset range, the first weight value is less than the second weight value.

8. An image processing apparatus, characterized in that, The device includes: The determination module is configured to, in response to the electronic device acquiring the nth image frame during the video recording process, determine a first cropping parameter for the nth image frame based on the jitter parameters of the electronic device; wherein the jitter parameters are determined based on the motion parameters of the electronic device during the time period from acquiring the nmth image frame to acquiring the nth image frame; m is greater than or equal to 1; The processing module is configured to obtain a nth image frame to be cropped that matches the first cropping parameter based on the nth image frame, and to perform cropping processing on the nth image frame to be cropped based on the first cropping parameter to obtain the nth target image frame; The generation module is configured to generate a target video based on each target image frame obtained during the execution of the video shooting function in response to the detection of a shooting end command.

9. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images