Image processing method and electronic device

Abstract

Embodiments of the present application provide an image processing method and an electronic device, the method comprising: acquiring a plurality of captured images captured by an image capturing device and a motion displacement of the image capturing device; determining a motion type of the image capturing device and a target image in the plurality of captured images based on the motion displacement; performing spatial alignment on the plurality of captured images based on the motion type to obtain a plurality of aligned images and an alignment confidence of each aligned image in the plurality of aligned images; and correcting the target image based on the alignment confidence of each aligned image to obtain an output image of the image capturing device.

Description

Technical Field

[0001] This application relates to the field of computer technology, and includes, but is not limited to, an image processing method and an electronic device. Background Technology

[0002] Currently, in traditional multi-frame composite photography schemes, several frames of images are typically captured consecutively before the user presses the shutter button, and one frame is selected as the main frame for composite output. This scheme can effectively improve the image signal-to-noise ratio and dynamic range under normal shooting conditions.

[0003] However, in extreme shooting scenarios such as ultra-high zoom, low light, and dynamic shooting, the relatively long exposure time means that even slight hand shake can cause noticeable motion blur in a single frame. Therefore, there is an urgent need for a solution that can effectively suppress motion blur. Summary of the Invention

[0004] Based on the problems existing in related technologies, this application provides an image processing method and an electronic device.

[0005] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide an image processing method, the image processing method comprising: Acquire multiple frames of images captured by the image acquisition device and the motion displacement of the image acquisition device; Based on motion displacement, determine the motion type of the image acquisition device and the target image in multiple frames of acquired images; Based on motion type, spatial alignment is performed on multiple frames of acquired images to obtain multi-frame aligned images and alignment confidence of each frame in the multi-frame aligned images; Based on the alignment confidence of each frame of aligned images, the target image is corrected to obtain the output image of the image acquisition device.

[0006] In some embodiments, the method further includes: within each sampling period, acquiring the displacement amount and displacement confidence level of the image acquisition device determined by the motion sensing module in the image acquisition device; and based on the displacement confidence level, filtering the displacement amount of the image acquisition device in multiple sampling periods to obtain the motion displacement of the image acquisition device when acquiring multiple frames of images.

[0007] In some embodiments, determining the motion type of the image acquisition device based on motion displacement includes: determining the image translation amount between two adjacent frames of the multi-frame acquisition image based on the motion displacement of the image acquisition device when acquiring multiple frames of acquisition images, wherein the image translation amount includes at least one of translation jitter amplitude, rotation angle, or scaling ratio; determining the motion change between two adjacent frames of acquisition images based on at least one of a first comparison result between the translation jitter amplitude and jitter threshold, a second comparison result between the rotation angle and rotation angle threshold, and a third comparison result between the scaling ratio and scaling ratio threshold; wherein the motion change includes at least displacement change, deformation change, or mixed change; accumulating the motion change to obtain the motion type of the image acquisition device; wherein the motion type includes at least translational motion, deformation motion, or mixed motion.

[0008] In some embodiments, spatial alignment is performed on multiple frames of acquired images based on motion type to obtain multi-frame aligned images and alignment confidence of each frame of aligned images in the multi-frame images. This includes: selecting an alignment model corresponding to two adjacent frames of acquired images based on motion changes between them; the alignment model includes at least a translational alignment model, an affine alignment model, and a local optical flow correction model, and the alignment model is pre-trained; aligning adjacent frames of acquired images sequentially based on the acquisition order of the multi-frame images according to the corresponding alignment model to obtain multi-frame aligned images; and determining the alignment confidence of each frame of aligned images based on the similarity between each frame of aligned images and a reference frame; the reference frame is the first frame or the center frame in the multi-frame aligned images.

[0009] In some embodiments, spatial alignment is performed on multiple frames of acquired images based on motion type to obtain multiple aligned images and the alignment confidence of each aligned image in the multiple frames, including: selecting an alignment model corresponding to the multiple frames of acquired images based on motion type; the alignment model includes at least a displacement alignment model, an affine alignment model, and a local optical flow correction model, and the alignment model is pre-trained; based on the acquisition order of the multiple frames of acquired images, taking the first frame of acquired images as a reference image, and based on the alignment model, aligning each frame of acquired images with the reference image starting from the second frame of acquired images to obtain multiple aligned images; and determining the alignment confidence of each aligned image based on the similarity between each aligned image and the reference image.

[0010] In some embodiments, determining a target image in multiple acquired images based on motion displacement includes: determining the target image in the multiple acquired images based on motion displacement or image quality of each acquired image; correcting the target image based on the alignment confidence of each aligned image to obtain an output image of the image acquisition device, including: extracting weights from each region of the other aligned images based on the alignment confidence of the other aligned images besides the target image and the motion displacement between each other aligned image and the target image during alignment, to obtain a region weight map corresponding to each aligned image; wherein each region includes at least a static region and a dynamic region; the region weight map includes the weight of each region; filtering the regions in each region weight map based on the weight of each region to obtain a usable region; and correcting the target image based on the usable region to obtain an output image.

[0011] In some embodiments, the target image is corrected based on the available area to obtain the output image of the image acquisition device, including: determining the region to be optimized in the target image, the region to be optimized including at least the afterimage region and the blurred region; and covering the region to be optimized with the available area corresponding to the location of the region to be optimized to obtain the output image.

[0012] In some embodiments, acquiring multiple frames of acquired images and the motion displacement of the image acquisition device includes: acquiring multiple frames of acquired images and the motion displacement of the image acquisition device in response to the ambient light intensity of the image acquisition device being lower than a preset light threshold and / or the motion displacement being greater than a preset displacement threshold.

[0013] In some embodiments, the image processing method further includes: adjusting the exposure time or acquisition frame rate of the image acquisition device in response to the motion displacement detected by the motion sensing module being greater than a preset displacement threshold; the motion sensing module includes at least one of the following: an event visual sensor and an optical flow sensor.

[0014] Secondly, embodiments of this application further provide an image processing apparatus, comprising: an acquisition module for acquiring multiple frames of acquired images and the motion displacement of the image acquisition device; a determination module for determining the motion type of the image acquisition device and a target image in the multiple frames of acquired images based on the motion displacement; a spatial alignment module for spatially aligning the multiple frames of acquired images based on the motion type to obtain a multi-frame aligned image and the alignment confidence of each aligned image in the multi-frame aligned image; and a correction module for correcting the target image based on the alignment confidence of each aligned image to obtain an output image of the image acquisition device.

[0015] Thirdly, embodiments of this application provide an electronic device, including: an image acquisition device, comprising at least an image acquisition unit and a motion sensing module, wherein the image acquisition unit is used to acquire multiple frames of acquired images, and the motion sensing module is used to determine the motion displacement of the image acquisition device; a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to perform the following steps: acquiring multiple frames of acquired images and the motion displacement of the image acquisition device; determining the motion type of the image acquisition device and a target image in the multiple frames of acquired images based on the motion displacement; spatially aligning the multiple frames of acquired images based on the motion type to obtain a multi-frame aligned image and an alignment confidence level of each aligned image in the multi-frame aligned image; and correcting the target image based on the alignment confidence level of each aligned image to obtain an output image of the image acquisition device.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the aforementioned image processing method.

[0017] Fifthly, embodiments of this application provide a computer program product, which includes executable instructions stored in a computer-readable storage medium; when the processor of the control unit reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-described image processing method is implemented. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 2 This is an optional flowchart illustrating the image processing method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the multi-frame image synthesis method provided in the embodiments of this application. Detailed Implementation

[0019] To more clearly illustrate the purpose, technical solutions, and advantages of the embodiments of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the following description of the embodiments is intended to explain and illustrate the overall concept of the embodiments of this application, and should not be construed as a limitation of the embodiments of this application. In the specification and drawings, the same or similar reference numerals refer to the same or similar parts or components. For clarity, the drawings are not necessarily drawn to scale, and some well-known parts and structures may be omitted from the drawings.

[0020] In some embodiments, unless otherwise defined, the technical or scientific terms used in the embodiments of this application shall have the ordinary meaning understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The word "a" or "an" does not exclude multiple components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," "right," "top," or "bottom" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. When an element such as a layer, film, region, or substrate is referred to as being "above" or "below" another element, the element may be "directly" located "above" or "below" the other element, or there may be intermediate elements present.

[0021] Traditional multi-frame synthesis (MFS) relies on the image itself to estimate inter-frame motion. However, in low-light / texture-deficient scenes, image optical flow is prone to failure, resulting in misalignment and causing image ghosting, ghosting, and blurring.

[0022] Related technologies attempt to select frames with less motion and higher alignment quality from a sequence for synthesis by using forward frame extraction, backward frame extraction, or combination methods. However, these methods typically require more complex inter-frame analysis and filtering, or acquiring image sequences within a longer time window, leading to increased shutter lag. This means that the waiting time from pressing the shutter to the final image output is prolonged, affecting the immediacy and smoothness of the shooting experience. Therefore, how to effectively suppress motion blur and other artifacts while ensuring the quality of multi-frame synthesized images and minimizing shutter lag has become a key technical challenge for improving the shooting experience in high-magnification zoom and low-light scenes.

[0023] To address the problems existing in related technologies, this application proposes an image processing method that acquires multiple frames of images acquired by an image acquisition device and the motion displacement of the image acquisition device; based on the motion displacement, determines the motion type of the image acquisition device and the target image in the multiple frames of images; based on the motion type, spatially aligns the multiple frames of images to obtain a multi-frame aligned image and the alignment confidence of each aligned image in the multi-frame aligned image; based on the alignment confidence of each aligned image, corrects the target image to obtain the output image of the image acquisition device.

[0024] This application embodiment determines the motion type and target image of the image acquisition device during image acquisition by measuring the motion displacement of the image acquisition device, thereby realizing motion perception and intelligent frame capture during the image acquisition process. The adaptive image alignment strategy based on motion type can dynamically match the alignment algorithm according to the motion type, which reduces the amount of computation and processing time while ensuring alignment accuracy, and significantly shortens the shutter delay. Through multi-frame image synthesis guided by alignment confidence, abnormal information introduced by alignment failure or scene motion is effectively identified and suppressed, which improves the image quality while reducing ghosting and local blurring, and realizes the high-definition imaging effect of instant capture in moving scenes.

[0025] The image processing methods provided in the embodiments of this application can be executed by electronic devices. Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 1 The illustrated electronic device 10 may include at least one controller 110, a memory 150, at least one network interface 120, a user interface 130, and an image acquisition device 160. The various components in the first electronic device are coupled together via a bus system 140. It is understood that the bus system 140 is used to implement communication between these components. In addition to a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 1 The general labeled all buses as Bus System 140.

[0026] The image acquisition device 160 may include at least an image acquisition unit and a motion sensing module. The image acquisition unit is used to acquire multiple frames of images, and the motion sensing module is used to acquire dynamic data to determine the motion displacement of the image acquisition device. The motion sensing module includes at least one of the following: an event vision sensor and an optical flow sensor.

[0027] The controller 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose controller, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose controller can be a microcontroller or any conventional controller.

[0028] User interface 130 includes one or more output devices 131 that enable the presentation of media content, and one or more input devices 132.

[0029] Memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 150 may optionally include one or more storage devices physically located remote from controller 110. Memory 150 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 150 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0030] Operating system 151 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. The input processing module 153 is used to detect one or more inputs or interactions from one or more input devices 132.

[0031] In some embodiments, the image processing apparatus provided in this application can be implemented in software. Figure 1 An image processing apparatus 154 stored in memory 150 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: an acquisition module 1541, a determination module 1542, a spatial alignment module 1543, and a correction module 1544. These modules can be logically linked and therefore can be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0032] In other embodiments, the apparatus provided in this application can also be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0033] The technical solution of this application will now be described in detail with reference to the accompanying drawings.

[0034] Figure 2 This is an optional flowchart illustrating an image processing method provided in an embodiment of this application, such as... Figure 2 As shown, the image processing method provided in this application embodiment can be implemented through steps S201 to S204: S201, acquire multiple frames of images acquired by the image acquisition device and the motion displacement of the image acquisition device.

[0035] In this embodiment, the image acquisition device may include at least two functional modules: an image acquisition unit and a motion sensing module. The image acquisition unit may be a camera module used to acquire multiple frames of images for subsequent image synthesis. The motion sensing module is used to acquire dynamic motion data of the image acquisition device during the image acquisition process, and its output can be used to characterize the motion displacement or attitude changes of the image acquisition device in three-dimensional space.

[0036] In some embodiments, the motion sensing module may be an optical flow sensor, an inertial measurement unit, or an event-based vision sensor (EVS). The EVS can output an asynchronous event stream to record pixel-level brightness changes with high temporal resolution, thereby indirectly reflecting the instantaneous motion displacement of the image acquisition device. The optical flow sensor can output the motion displacement of the image acquisition device by detecting position changes of pixels or feature points in continuously acquired image frames, and can also output confidence information of the motion displacement at the same time.

[0037] In some embodiments, if the optical flow sensor is a single-point sensor, it can only output global translation (Δx, Δy), and rotation / scaling needs to be supplemented by gyroscope or multi-point estimation.

[0038] In this embodiment of the application, when the user triggers the shooting command, the image acquisition device can output multiple frames of acquired images in continuous frame mode, and the motion sensing module can continuously output optical flow data. After processing, the motion displacement of the image acquisition device when acquiring multiple frames of images can be obtained.

[0039] The motion displacement of the image acquisition device can be calculated based on optical flow data using methods such as correlation matching or optical flow equation. It can be represented as a sequence of displacement vectors associated with each frame of acquired images. This sequence is temporally aligned with multiple frames of acquired images, forming a set of one-to-one data pairs, which provides the input basis for subsequent motion type analysis and frame selection.

[0040] The embodiments of this application can perform spatial position calibration and alignment during the assembly of the mobile phone using the event vision sensor and the RGB image sensor in the camera module, so that the motion displacement output by the optical flow sensor can accurately reflect the image offset.

[0041] S202, based on motion displacement, determine the motion type of the image acquisition device and the target image in the multi-frame acquired images.

[0042] In this embodiment, analyzing the motion displacement yields a displacement sequence of the image acquisition device during the acquisition of multiple frames of images. This displacement sequence characterizes the motion parameters in three-dimensional space during image acquisition and can be represented by displacement amounts in three-dimensional directions, such as translational components along the xyz directions or rotational components around the xyz axes. The motion parameters may also include at least one of the following: planar motion trajectory in the xy plane, depth variation information along the z-axis, motion velocity in three-dimensional space, and acceleration.

[0043] After obtaining the displacement sequence, the motion type of the image acquisition device can be determined based on the characteristics of the motion parameters. For example, when the translational components in the X and Y axes of the displacement sequence are the main changing components and are greater than a preset threshold, while the translational component in the Z axis and all rotational components are less than a preset threshold, the motion type is determined to be translational motion. For example, when shooting a static scene with a handheld device, the user's slight hand tremors usually manifest as random translations in the XY plane.

[0044] In some embodiments, the motion type may include translational motion with small-amplitude movement in a plane, deformation motion with rotation or scaling, and a hybrid motion with both.

[0045] In some embodiments, based on motion displacement, the cumulative motion error or motion consistency score corresponding to each frame of the acquired image can be calculated. The frame with the smallest cumulative motion error or the highest motion consistency score can be determined as the target image, i.e., the main frame. The cumulative motion error characterizes the total displacement compensation required for motion alignment between this frame and the other frames; a smaller error indicates a closer spatial relationship between this frame and other frames in the sequence. The motion consistency score characterizes the degree of matching between the motion displacement of this frame and the average motion displacement of the entire sequence; a higher score indicates a more consistent motion state between this frame and the mainstream motion mode of the image acquisition device during the overall acquisition process, resulting in a smaller geometric transformation amplitude required for subsequent multi-frame alignment when this frame is used as the main frame.

[0046] In this way, in scenarios involving displacement motion, the frame with the least jitter can be selected as the target image, effectively suppressing the composite blur caused by hand tremors. In scenarios involving deformation motion, the frame that best matches the user's zoom intention can be selected as the target image, maintaining the stability of the image composition. Thus, the target image can serve as the most stable reference, effectively reducing the cumulative error caused by global motion compensation in subsequent multi-frame synthesis, thereby improving the quality of the synthesized image.

[0047] S203, based on motion type, spatially aligns multiple acquired images to obtain multi-frame aligned images and the alignment confidence of each frame aligned image in the multi-frame aligned images.

[0048] In the embodiments of this application, spatial alignment can refer to the process of mapping each frame of a multi-frame acquired image to a common coordinate system based on the target image (main frame). The mapping of pixel coordinates can be achieved through a geometric transformation model to compensate for the relative displacement, rotation and scale changes between images caused by camera shake or movement during shooting, so that the acquired images of each frame are consistent in spatial position, providing a geometric basis for subsequent multi-frame fusion or super-resolution reconstruction.

[0049] In some embodiments, different geometric transformation models can be used for alignment depending on the type of motion. For example, when the motion is planar translation, a translational alignment model can be used, which only includes displacement parameters in the horizontal and vertical directions. When the motion involves depth changes or scaling, a similarity transformation model or an affine transformation model can be used to compensate for scale changes and rotations. When the motion is complex 3D motion or large-angle rotation, a local optical flow correction model can be used to correct projection distortion and achieve more accurate spatial registration. This motion-adaptive alignment model selection mechanism optimizes computational efficiency while ensuring alignment accuracy, avoiding the use of overly complex transformation models for simple motions.

[0050] In some embodiments, alignment confidence is used to characterize the spatial alignment quality between the current frame and the reference frame. The value can range from 0 to 1, and a higher value indicates that the alignment result is more reliable.

[0051] S204, Based on the alignment confidence of each frame of aligned images, the target image is corrected to obtain the output image of the image acquisition device.

[0052] In some embodiments, the target image can be used as a reference. Based on the alignment confidence of each frame of aligned images, the contribution weight of each frame of aligned images in the fusion process is determined. A weighted fusion algorithm is used to fuse the details, textures, or dynamic range information of high-confidence regions in each frame of aligned images into the reference layer to correct the target image and finally generate a quality-enhanced output image.

[0053] In this embodiment, high-alignment-confidence frames may include clear details, better local exposure, or richer textures, which can be integrated into the target image to improve image quality. Low-alignment-confidence frames are considered unreliable due to poor alignment (e.g., motion blur, object movement), and their weight can be reduced or ignored to prevent their blurriness or erroneous information from misleading the final image.

[0054] In some embodiments, the available area in the high alignment confidence frame can be used to cover the corresponding area of ​​the main frame to improve the image quality of the main frame.

[0055] The embodiments of this application determine the motion displacement of the image acquisition device during image acquisition, identifying the motion type and target image, thus realizing motion perception and intelligent frame capture during the image acquisition process. An adaptive image alignment strategy based on motion type dynamically matches the alignment algorithm according to the motion type, ensuring alignment accuracy while reducing computational load and processing time, significantly shortening shutter delay. Through multi-frame image synthesis guided by alignment confidence, abnormal information introduced by alignment failure or scene motion is effectively identified and suppressed, improving image quality while reducing ghosting and local blurring, achieving high-definition imaging effects for instant capture in moving scenes.

[0056] In some embodiments, the image processing method provided in this application can determine the motion displacement of the image acquisition device through steps S1 and S2: S1, within each sampling period, acquire the displacement and displacement confidence of the image acquisition device determined by the motion sensing module in the image acquisition device.

[0057] In some embodiments, the motion sensing module may be an optical flow sensor or an event camera (EVS), or a combination of both. The optical flow sensor analyzes the brightness variation patterns (optical flow field) of pixels between consecutively acquired images, calculating the displacement (Δx, Δy) of the image acquisition device in a two-dimensional plane. The displacement confidence level can be generated based on the feature matching degree of the optical flow field or the texture richness of the computational region. The event camera asynchronously outputs events where the brightness change exceeds a threshold. By accumulating events within a specific time window, the displacement of the image acquisition device can be determined.

[0058] Within each sampling period, the motion sensing module can output the displacement of the image acquisition device relative to the previous frame in the image acquisition plane when acquiring the frame, and at the same time output the displacement confidence level characterizing the reliability of the displacement.

[0059] Here, the displacement can be represented by a two-dimensional vector (Δx, Δy), where Δx represents the horizontal displacement component and Δy represents the vertical displacement component. The unit of displacement can be pixels or millimeters, depending on the sensor calibration.

[0060] The displacement confidence level can be a value between 0 and 1, and can be determined based on at least one of the following: signal quality within the motion sensing module, feature matching degree, or ambient lighting conditions.

[0061] S2, based on displacement confidence, the displacement of the image acquisition device in multiple sampling periods is filtered to obtain the motion displacement of the image acquisition device when acquiring multiple frames of images.

[0062] In this embodiment, from all sampling periods corresponding to multiple frames of acquired images, valid sampling periods with displacement confidence levels higher than a confidence threshold (which can be set empirically) can be selected. Based on the displacement obtained within the valid sampling periods, the motion displacement of the image acquisition device when acquiring each frame of the acquired image is reconstructed through interpolation, filtering, or state estimation algorithms.

[0063] For example, a Kalman filter or a moving average filter can be used to process the high-confidence displacement sequence to smooth noise and compensate for the information loss caused by the removal of low-confidence data, ultimately obtaining continuous motion displacement corresponding to the time sequence of multiple acquired images.

[0064] This application's embodiments introduce displacement confidence to filter displacement amounts, effectively eliminating erroneous displacement estimates caused by sensor noise or sudden violent movements. This provides a highly reliable and accurate input data foundation, improving the robustness of image synthesis and the final output image quality.

[0065] In some embodiments, the motion type of the image acquisition device during the entire shooting process can be determined by the geometric differences between adjacent frames, and by performing temporal accumulation and threshold judgment. Step S202 can be implemented by steps S2021 to S2023: S2021, based on the motion displacement of the image acquisition device when acquiring multiple frames of images, determine the image translation amount between two adjacent frames of images in the multiple frames of images, wherein the image translation amount includes at least one of translation jitter amplitude, rotation angle or scaling ratio.

[0066] In some embodiments, a correspondence between the acquisition timestamp of the acquired image and the sampling timestamp of the motion sensing module can be established first. In this way, for each pair of adjacent acquired image frames (the Nth frame and the N+1th frame), the image translation amount between the two adjacent acquired images can be determined from the motion displacement of the image acquisition device when acquiring multiple acquired images within their time interval.

[0067] In this embodiment, the motion sensing module and the image acquisition device are spatially calibrated and aligned during assembly, and a geometric mapping model is constructed from the coordinate system of the motion sensing module to the coordinate system of the image plane of the image acquisition device. The geometric mapping model may include a rotation matrix and a translation vector. Based on the rotation matrix, the translation vector and the intrinsic parameters of the image acquisition device, the motion displacement obtained by the motion sensing module can be reflected in the image offset through geometric calculation and transformation. That is, the translational motion in three-dimensional space is converted into pixel-level offset on the two-dimensional image plane, thereby obtaining the image translation amount between two adjacent frames of acquired images.

[0068] In some embodiments, the image translation amount includes at least one of translation jitter amplitude, rotation angle, or scaling ratio. Translation jitter amplitude represents the amount of offset between two adjacent captured images in the horizontal (X-axis) and vertical (Y-axis) directions of the image, reflecting minor, unintentional translation jitter of the device. Rotation angle represents the angle of rotation about the Z-axis of the image, such as slight internal or external rotation of the wrist. Scaling ratio represents the magnification or reduction ratio of the subsequent image relative to the previous image (e.g., 1.01 represents 1% magnification, 0.99 represents 1% reduction), reflecting perspective scaling movement of the device in the Z-axis direction (forward or backward), or user-initiated zoom operation.

[0069] S2022, based on at least one of the following: a first comparison result between the translation jitter amplitude and jitter threshold between two adjacent acquired images; a second comparison result between the rotation angle and rotation angle threshold between two adjacent acquired images; and a third comparison result between the scaling ratio and scaling ratio threshold between two adjacent acquired images, determine the motion change between two adjacent acquired images; wherein the motion change includes at least displacement change, deformation change, or mixed change.

[0070] In this embodiment, the calculated translation jitter amplitude is compared with a preset jitter threshold (which can be a multi-level threshold comparison, such as 1, 3, 5, 15 pixels) to obtain a first comparison result, determining whether there is a significant displacement change between two frames; the rotation angle is compared with a preset rotation angle threshold (which can be a multi-level threshold comparison, such as 1°, 3°) to obtain a second comparison result, determining whether there is a rotation change; the scaling ratio is compared with a preset scaling ratio threshold (which can be a multi-level threshold comparison, such as 1.0±0.02, 1.0±0.05) to obtain a third comparison result, determining whether there is a scaling change. By setting different levels of thresholds, motion changes can be classified in multiple levels to accurately match the subsequent alignment model.

[0071] Analyzing at least one comparison result can determine the motion type of the image acquisition device. If the translation jitter exceeds a threshold, while rotation and scaling are not significant (i.e., translation components in the X and Y axes are the main changing components, and the translation component in the Z axis and all rotation components are less than a preset threshold), it indicates that the movement between these two frames is mainly device translation, a displacement change. If the rotation angle or scaling ratio exceeds a threshold, while translation is not significant (i.e., the translation component in the Z axis exceeds a preset threshold, or the feature points in the XY plane show a uniform scaling trend), it indicates that the movement between these two frames is mainly device rotation or scaling, a deformation change. If two or three of the translation, rotation, and scaling values ​​significantly exceed the threshold simultaneously, it indicates complex motion, such as simultaneous translation and rotation, a mixed change.

[0072] In some embodiments, when the translation jitter amplitude, rotation angle, and scaling ratio are all less than the threshold, it can be considered that no displacement has occurred between the two image frames.

[0073] S2023, accumulate the motion changes to obtain the motion type of the image acquisition device; the motion type includes at least translational motion, deformation motion or mixed motion.

[0074] To reduce misjudgments caused by a single severe jitter, temporal cumulative analysis of motion changes can be performed. That is, starting from the first frame, motion changes between each pair of adjacent frames can be continuously tracked and recorded. Sliding window statistics or state machine models can be used for accumulation. Based on the accumulation results, the motion type of the image acquisition device can be determined.

[0075] If, throughout the entire sequence or a sufficiently long observation window, the vast majority of instantaneous motion changes (e.g., over 80%) are determined to be displacement changes, while deformation and mixed changes occur very rarely, it indicates that the user's hand is primarily moving and shaking during the entire shooting process, without any intention to zoom or rotate significantly. In this case, the motion type of the image acquisition device can be considered translational motion. If deformation changes (rotation or scaling) dominate, it indicates that the user may be actively zooming, or the device may be continuously rotating (e.g., shooting a panorama). In this case, the motion type of the image acquisition device can be considered deformation motion. If displacement and deformation changes frequently alternate, it indicates that the user's movements are very unstable, or the shooting scene is complex (e.g., zooming while walking and shooting). In this case, the motion type of the image acquisition device can be considered mixed motion.

[0076] This application embodiment converts the three-dimensional physical displacement of the image acquisition device into a two-dimensional image translation on the image plane, reliably reflecting the actual jitter of the image acquisition device during image acquisition, improving the accuracy of motion type estimation, providing high-quality input for subsequent image alignment, and improving the final image quality and stability of multi-frame composite images.

[0077] In some embodiments, adjacent frames can be aligned using corresponding alignment models; that is, different adjacent frames with different motion changes can be aligned using different alignment models. Step S203 can be implemented through steps S2031 to S2033: S2031, based on the motion changes between two adjacent acquired images, select the alignment model corresponding to the two adjacent acquired images; the alignment model includes at least a translation alignment model, an affine alignment model and a local optical flow correction model, and the alignment model is pre-trained.

[0078] In some embodiments, the translation alignment model can be used for alignment of translational motions without rotation, scaling, or other deformations; the affine alignment model can be used for alignment of deformable motions (including translation, rotation, uniform scaling, and slight shearing); and the local optical flow correction model can be used for alignment of mixed motions, performing dense / sparse optical flow estimation on image blocks or extracted feature points, and can handle non-rigid motions. Selecting the appropriate alignment model based on different motion variations can reduce unnecessary complex calculations while ensuring alignment accuracy.

[0079] The model can be trained using deep learning, specifically by learning the mapping relationship between sensor displacement and pixel offset under different focal lengths through a large amount of data. Model parameters are then determined using loss functions (including at least geometric and photometric losses), resulting in a trained model used for image alignment. The data can consist of sample image pairs with various focal lengths, lighting conditions, and motion modes, along with their actual motion change labels. These labels can be obtained through manual annotation.

[0080] S2032, based on the acquisition order of multi-frame acquired images, align two adjacent acquired images sequentially based on the corresponding alignment model to obtain multi-frame aligned images.

[0081] In this embodiment, images are processed sequentially according to their timestamps, and a chain-like alignment method with pairwise transmission is used to align multiple frames of acquired images. First, alignment is performed using the corresponding alignment models of each pair of adjacent frames. For example, the second frame is aligned to the coordinate system of the first frame. Then, through the accumulation of transformation matrices, the local alignment results are uniformly mapped to a specified reference coordinate system, which can be the first frame. For example, the third frame is first aligned to the second frame, and then, based on the transformation from the second frame to the first frame, the third frame is transformed to the coordinate system of the first frame.

[0082] S2033, Based on the similarity between each frame aligned image and the reference frame, determine the alignment confidence of each frame aligned image; the reference frame is the first frame or the center frame in the multi-frame aligned images.

[0083] In some embodiments, the first frame or the center frame with the highest quality and least motion can be selected as the reference frame, and all aligned images are compared with this frame.

[0084] The degree of matching between the aligned image and the reference frame can be determined using metrics such as Structural Similarity Index Measure (SSIM), feature point matching, or normalized cross-correlation. A higher matching degree indicates a more successful alignment, higher alignment confidence, and a higher weight for that frame in subsequent corrections; low confidence indicates that there is large motion causing matching errors or that there is independent motion in the scene (such as a pedestrian walking by).

[0085] Alignment confidence can eliminate bad frames and improve the quality of the output image.

[0086] This application embodiment determines the corresponding alignment model by the motion changes of two adjacent frames, and unifies multiple frames of images into the reference frame coordinate system in sequence to obtain the alignment confidence. In this way, while ensuring alignment accuracy, the computational load is reduced and the image output time is reduced by dynamically selecting the model. At the same time, the confidence can reduce the situation of abnormal frames being synthesized, and suppress ghosting and blurring in the output image.

[0087] In some embodiments, an alignment model can be determined for multiple captured images based on the motion type of the image acquisition device, and the multiple captured images can be aligned using this alignment model. Step S203 can also be implemented through steps S2034 to S2036: S2034, Based on the motion type, select the alignment model corresponding to the multi-frame acquired images; the alignment model includes at least a displacement alignment model, an affine alignment model, and a local optical flow correction model, and the alignment model is pre-trained.

[0088] In some embodiments, translational motion can be selected from a displacement alignment model, deformation motion can be selected from an affine alignment model, and hybrid motion can be selected from a local optical flow correction model.

[0089] S2035, based on the acquisition order of multiple frames of acquired images, taking the first frame of acquired images as the reference image, and based on the alignment model, aligning each frame of acquired images with the reference image starting from the second frame of acquired images to obtain multi-frame aligned images.

[0090] In this embodiment, the first frame can be selected as the reference image, and a one-to-many star alignment strategy is adopted to align the multi-frame acquired images. Starting from the second frame, each frame is directly aligned with the first frame reference image using the corresponding alignment model to obtain the aligned image in the coordinate system of the first frame reference image. This reduces the problem of error accumulation in alignment frame by frame and ensures that all frames are unified to the same coordinate system.

[0091] S2036, Based on the similarity between each frame's aligned image and the reference image, determine the alignment confidence of each frame's aligned image.

[0092] In this embodiment, confidence can be calculated by comparing the degree of matching between the aligned image and the reference image in the overlapping region. For example, the degree of matching between the aligned image and the reference frame can be determined by indicators such as structural similarity (SSIM), feature point matching, or normalized cross-correlation. High-confidence frames will receive high weights, and their clear details and better exposure will be used during synthesis; low-confidence frames will receive low weights or be directly discarded to prevent their erroneous information from polluting the final synthesized image, thereby effectively suppressing ghosting.

[0093] This application embodiment achieves adaptive calculation for image alignment through motion classification and geometric consistency of aligned images through star-shaped alignment. In this way, in complex and ever-changing real shooting scenarios, while saving computing power, it can stably output high-definition, ghost-free multi-frame composite images, achieving the best balance between image quality, speed and power consumption.

[0094] In some embodiments, determining the target image in the multi-frame acquired images based on motion displacement in step S202 can be achieved through step S11: S11, determine the target image from multiple acquired images based on motion displacement or image quality of each frame.

[0095] In this embodiment, hardware motion data and image quality can be combined to intelligently determine the target image as the best reference frame, i.e. the main frame, from multiple captured images.

[0096] Motion displacement quantifies the instantaneous motion state (such as translation speed and rotational angular velocity) of the image acquisition device when acquiring each frame. Frames with small cumulative motion errors can be used as target images, indicating that the camera was in a relatively stable state when capturing the frame. Frames with high motion consistency scores can also be used as target images, as the motion trajectory of the frame best matches the average motion type of the entire shooting process, and the total compensation required to align other frames with the frame as a reference is the smallest.

[0097] In some embodiments, image quality can be the image quality of each frame of the captured image, including sharpness, exposure, focus, and noise, and the captured image with the highest image quality can be used as the target image.

[0098] In some embodiments, motion displacement and image quality can be combined for weighted or two-stage filtering. Two-stage filtering refers to eliminating frames with significantly excessive cumulative motion errors (such as those suddenly exacerbated by hand tremors) based on motion displacement to ensure the geometric stability of the target image, and then selecting the frame with the highest overall image quality score (sharpness, noise, exposure) as the target image. Alternatively, motion scores and quality scores can be assigned to each frame, weighted according to the actual scene, and the frame with the highest overall score can be selected as the target image.

[0099] Correspondingly, step S204 can be achieved through steps S2041 to S2043: S2041, based on the alignment confidence of other aligned images besides the target image in the multi-frame aligned images and the motion displacement between each other aligned image and the target image during alignment, weights are extracted from each region of the other aligned images to obtain a region weight map corresponding to each aligned image; wherein, each region includes at least a static region and a dynamic region; the region weight map includes the weight of each region.

[0100] In the embodiments of this application, the alignment confidence of the aligned image indicates whether the frame image is trustworthy. The motion displacement between each other aligned image and the target image during alignment represents the amount of motion between each other aligned image and the target image. Based on these two values, each aligned image can be partitioned and the weight of each region can be determined to obtain the region weight map of each aligned image. The value of each image block in the map represents the trustworthiness of the position information in subsequent synthesis.

[0101] Image partitioning can refer to dividing an image into static and dynamic regions. Static regions can refer to the background or stationary objects, while dynamic regions can refer to the areas where moving objects (such as pedestrians or vehicles) are located in the image.

[0102] The weights of static regions can be calculated based on alignment confidence. If the alignment confidence of the entire frame is high, the weight of the static region is also high because the background has been stably aligned, and its clear details can be used to enhance the target image; conversely, the weight is low.

[0103] The calculation of dynamic region weights is highly dependent on local motion analysis. If the local motion displacement of a region is consistent with the global motion displacement (e.g., the object moves with the camera), it may still be aligned and can be given a higher weight to enhance the texture. If the local motion displacement of a region is inconsistent with the global motion displacement (e.g., a person walks in the opposite direction), the weight of the region is lower regardless of the overall alignment confidence, in order to reduce ghosting of moving objects in the composite image.

[0104] In some embodiments, motion segmentation algorithms (to distinguish between dynamic and static regions) and residual analysis (to compare the differences between the aligned image and the target image) can be combined to complete the weight allocation of each region.

[0105] S2042, based on the weight of each region, filters the regions in the weight map of each region to obtain the usable regions.

[0106] In this embodiment, multiple weight thresholds can be set to filter regions. Regions with weights below a first threshold are marked as unusable, suitable for dynamic regions with extremely low weights due to severe misalignment or independent movement. Regions with weights above a second threshold are marked as usable regions; regions with weights between the first and second thresholds are reserved. Here, the second threshold is greater than the first threshold.

[0107] In this way, usable areas can be obtained in each frame of the aligned image, which can be used in subsequent image synthesis.

[0108] S2043, Based on the available area, the target image is corrected to obtain the output image.

[0109] Correction can refer to the process of enhancing a target image by using available regions from multiple frames as a base. It can be the extraction of better information than the target image from the available regions of all other aligned images. For example, extracting clearer texture details from static regions of another frame; extracting better-exposed pixels from static regions of another frame; or extracting objects with less motion blur from available dynamic regions of another frame.

[0110] In the embodiments of this application, even if the overall alignment confidence of some frames is low due to severe jitter, as long as the local area (such as a small background) is successfully aligned, the useful information in that area can still be utilized, thereby improving the fault tolerance and output rate of image synthesis.

[0111] Correction can be achieved using multi-scale fusion (such as Laplacian pyramid fusion) or weighted averaging. During fusion, information from high-weight regions is prioritized, while information from low-weight or discarded regions remains unaffected, resulting in the final output image.

[0112] This application embodiment can identify misaligned dynamic regions by partitioning other aligned images and determining available areas. Image correction is performed using high-weighted regions, reducing ghosting caused by moving objects during image synthesis. In scenes containing complex motion (such as street photography), it can stably output high-quality synthesized images with extremely clear backgrounds, accurate capture of dynamic subjects, and no artifacts.

[0113] In some embodiments, step S2043 can be implemented by steps S21 to S22: S21, Determine the region to be optimized in the target image. The region to be optimized includes at least the afterimage region and the blurred region.

[0114] In some embodiments, the afterimage region can refer to a semi-transparent or ghosted object trace caused by a moving object during shooting, and the blurred region can refer to a local decrease in sharpness caused by camera shake, object movement, or inaccurate focusing at the moment of shooting. The afterimage region or blurred region in the target image, i.e., the region to be optimized, can be identified through image recognition or any feasible method.

[0115] S22, the region to be optimized is covered by the available region corresponding to the location of the region to be optimized, and the output image is obtained.

[0116] Here, for each region to be optimized, the best-quality, geometrically matching local image patch can be found in the corresponding available regions of the remaining frames to replace the region to be optimized on the target image.

[0117] In some embodiments, since all images are aligned to a coordinate system based on the target image, the coordinate positions of the region to be optimized and the corresponding regions in other frames are related. It is possible to directly check whether the corresponding position of the region to be optimized is marked as a usable region in the region of other frames, and determine the usable image block as a candidate block.

[0118] From all candidate blocks, the optimal block can be selected to cover the region to be optimized based on quality score (sharpness, noise level) or image consistency (natural transition of color and texture with the surrounding area of ​​the target image). For example, for blurred regions, the sharpest candidate block can be selected; for ghosting regions, the candidate block that does not contain any moving objects or has the cleanest object outline can be selected.

[0119] The embodiments of this application can employ image fusion techniques such as gradient domain fusion to cover the defect location of the target image with the selected optimal candidate block, ensuring a smooth transition of color, brightness and texture at the boundary without leaving any repair traces.

[0120] This application embodiment identifies regions with poor quality in the target image, and then selects high-quality local information from the corresponding available regions of other aligned frames for precise replacement, thereby efficiently and seamlessly repairing defects in the main frame and obtaining an output image with the required quality.

[0121] In some embodiments, the image processing method of this application can be activated when the output image quality is at risk or the shooting scene is dynamically complex, thereby achieving a balance between power consumption and image quality. Step S201 can be implemented through step S2011: S2011, in response to the ambient light intensity of the image acquisition device being lower than a preset light threshold and / or the motion displacement being greater than a preset displacement threshold, acquire multiple frames of acquired images and the motion displacement of the image acquisition device.

[0122] In this embodiment, ambient light intensity below a preset light threshold can refer to a situation where the ambient light dims (e.g., at night, in low indoor light) and a single-frame shot cannot meet the image requirements. Motion displacement greater than a preset displacement threshold indicates that severe camera shake is detected (e.g., unstable handheld shooting, shooting while walking) or objects in the frame are moving too fast. If the single-frame exposure time is too long, motion blur will occur; if it is too short, underexposure will occur. In such cases, multi-frame image synthesis is required to improve image quality. In this way, in well-lit and stable scenes (such as shooting landscapes during the day), single-frame shooting can be maintained, avoiding the computational overhead of multiple frames and extending the device's battery life; in night scenes or motion scenes, the scene can be automatically identified and the method provided in the embodiments of this application can be invoked to achieve adaptive adjustment, so that images that meet the quality requirements can be output in any scene.

[0123] In some embodiments, the image processing method provided in this application may further include step S31: S31, in response to the motion displacement detected by the motion sensing module being greater than a preset displacement threshold, the exposure time or acquisition frame rate of the image acquisition device is adjusted; the motion sensing module includes at least one of the following: an event visual sensor and an optical flow sensor.

[0124] In this embodiment, when the motion sensing module detects that the camera is in a state of violent motion (such as violent camera shaking or high-speed movement of the subject), it can actively adjust the image acquisition strategy. For example, it can shorten the exposure time of the image acquisition device to reduce motion blur, and also increase the acquisition frame rate of the image acquisition device to acquire more frames in the same time window, providing richer material for subsequent multi-frame synthesis, and more accurately capturing motion trajectories to reduce the estimation error of inter-frame displacement.

[0125] The preset displacement threshold can be dynamically adjusted according to the scene (for example, telephoto lenses are more sensitive to shaking, so the threshold should be smaller) to achieve fine-grained scene adaptation.

[0126] In some embodiments, only one parameter, exposure time or frame rate, may be adjusted, or both may be adjusted simultaneously. This application does not impose any limitations on these embodiments.

[0127] In some embodiments, the motion sensing module includes at least one of an event vision sensor and an optical flow sensor, which will not be described in detail here.

[0128] In this application embodiment, motion blur is reduced by adjusting the acquisition parameters during the acquisition stage, thus reducing the workload of post-processing. By employing a short exposure and high frame rate strategy, high-quality, low-deformation multi-frame acquired images are provided, with accurate geometric information, ensuring inter-frame continuity and laying the foundation for generating clear, motion-free output images.

[0129] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0130] To address the problems existing in related technologies, this application provides a multi-frame synthesis scheme assisted by an optical flow sensor. By introducing an optical flow sensor, motion detection is performed by activating the optical flow sensor in a specific scene, and motion compensation is performed based on the motion detection results, thereby reducing low-light photography issues such as ghosting and motion blur.

[0131] In this embodiment, on the one hand, the frame with the smallest motion amplitude can be selected as the main frame based on the detection results of the optical flow sensor; on the other hand, the motion trajectory of moving objects in multi-frame acquired images can be accurately analyzed. Based on the value of the position offset vector between the current frames, position correction parameters are dynamically selected to help align and synthesize multi-frame images. The optical flow data can more effectively compensate for hand shakiness and camera vibration, reducing image blur.

[0132] Here, the position correction parameters were determined through extensive experiments conducted under different environmental conditions and the movement of the captured object. It was found that when the scene and the optical flow vector were x, applying automatic exposure, automatic focus or automatic white balance image signal processing (3A ISP, Auto Focus, Auto Exposure, Auto White Balance Image Signal Processor) and position correction parameters could achieve better imaging results, improving image details while maintaining file size, and reducing blur and noise.

[0133] This application embodiment can add a low-latency, subpixel-level displacement output optical flow sensor to the original multi-frame composite photography process to indicate the camera movement direction, provide inter-frame displacement vector (i.e., the image translation amount between two adjacent captured images) and confidence (i.e. displacement confidence) in real time, dynamically select the alignment model (translation alignment model / affine alignment model / local optical flow correction model), adjust the synthesis weight, and drive the lightweight depth deblurring / denoising module, ultimately significantly reducing blur and motion blur caused by hand shake and camera vibration, while meeting low power consumption and low latency requirements.

[0134] Introducing an optical flow sensor can provide the camera's motion trajectory in real time, which is more stable and has lower latency compared to image estimation.

[0135] In this embodiment, the optical flow sensor (i.e., motion sensing module) provides data on how much the camera (i.e., image acquisition device) has moved. The camera uses this motion data to guide frame selection, alignment, and compositing. When needed, the strategy is dynamically adjusted (e.g., selecting different alignment models, weighted fusion methods, or even controlling exposure time). Ultimately, a clearer, motion-free composite image is obtained.

[0136] This application embodiment incorporates an optical flow sensor (similar to the optical sensor in a miniature mouse in a mobile phone) into the camera module, enabling high-frequency output of inter-frame displacement vectors. A data path is established with the phone's main camera chip (such as an ISP / System-on-a-Chip (SoC)) to ensure timestamp synchronization. During phone assembly, the Event-based Vision Sensor (EVS) and the RGB image sensor in the camera module undergo spatial calibration and alignment. This ensures data spatial alignment between the EVS and the image sensor, allowing accurate correspondence between the EVS's coordinate system, viewpoint, and imaging geometry with the RGB camera. In other words, the same scene point seen by both sensors can be mapped to the same spatial location through geometric transformation. The displacement data from the optical flow sensor can be calculated and transformed to align with the image sensor's position (including pixel space), ensuring that the motion data output by the optical flow sensor accurately reflects image offsets during subsequent use.

[0137] Figure 3 This is a flowchart illustrating the multi-frame image synthesis method provided in the embodiments of this application, as shown below. Figure 3 As shown, the multi-frame image synthesis method can be implemented through steps S301 to S308: The S301 optical flow sensor outputs the camera's motion displacement and confidence level in the x and y directions in real time.

[0138] Optical flow sensors collect data at different times, which may differ from the times at which a camera captures multiple frames of images. They output in real time the camera's displacement in the x and y directions (i.e., the displacement of the image acquisition device) and confidence level (i.e., displacement confidence) within each sampling period, thus obtaining an optical flow displacement sequence. The optical flow sensor can output at a fixed frequency (e.g., 200 to 1000 Hz) and continuously record these motion vectors, stitching them together to form a time series, i.e., an optical flow displacement sequence.

[0139] S302, the motion detection section performs filtering and statistical data processing on the optical flow data to obtain the overall motion mode of the camera.

[0140] Here, the motion detection section performs filtering and statistical data processing on the optical flow data, and aligns it with the camera in time to obtain the overall motion pattern (i.e., motion type) of the camera. Time alignment refers to obtaining the motion displacement and confidence level between two frames acquired by the camera, based on the time interval between the two frames, and discarding frames with low confidence levels.

[0141] The optical flow displacement sequence is acquired, the magnitude and direction of motion are analyzed, and the optical flow data is filtered and statistically analyzed to obtain the overall motion mode of the camera.

[0142] Here, the optical flow displacement sequence can be a sequence of camera motion vectors measured each time within a continuous time period, representing the translation direction and magnitude of the camera in each sampling period.

[0143] The main purpose of filtering is to remove noise and extract the true motion trend, including: translational jitter (i.e. displacement change) in the x and y directions, rotation (calculated by the displacement difference between different regions) and scaling (calculated by the displacement ratio between different regions). Rotation and scaling are deformation motions.

[0144] The following statistical analyses are performed on the optical flow displacement sequence to determine the motion mode. For example, the overall offset direction is determined by the average displacement; the stability is determined by the variance of the jitter amplitude; the peak displacement is detected to determine if there is a sudden vibration; the camera trajectory is accumulated by time integration; and the overall motion mode of the camera is obtained based on the camera trajectory.

[0145] In some embodiments, it can be determined whether the overall motion mode of the camera is small motion (i.e., translational motion), medium rotation / zoom (i.e., deformation motion), or complex shaking (mixed motion).

[0146] Among them, the average translation between two adjacent frames If μ is less than 0.2 pixels (px), it can be considered as no jitter; if μ is between 0.2 and 1, it can be considered as small jitter; if μ is between 1 and 3 pixels, it can be considered as medium jitter; if μ is between 3 and 15 pixels, it can be considered as large jitter, which can be corrected by gyroscope fusion, multi-frame accumulation, and rolling shutter; if μ is greater than 15 pixels, it can be considered as severe irregular jitter, which can be corrected by enabling optical image stabilization (OIS), gyroscope, and inter-frame cropping / electronic image stabilization (EIS).

[0147] For multi-pixel array optical flow sensors, the optical flow differences at different points (such as clockwise / counterclockwise differences) can be analyzed to derive the rotation angle θ and determine whether there is rotation; based on the distance changes at multiple points, the scaling factor s can be derived to determine whether there is magnification / zoom; the jitter pattern can be calculated to determine whether it is stable.

[0148] Based on the above data, the overall motion mode of the camera is comprehensively judged. For example, small motion: μ<1px, θ<1° and s≈1.0±0.02 are all met; medium motion: 1px≤μ<3px, 1°≤θ<3° and s≈1.0±0.05 are at least one; complex motion: μ≥3px or θ≥3° or σ²>threshold.

[0149] S303: The camera continuously acquires multiple frames of images, which are used as input image frames.

[0150] In this embodiment, the camera acquires multiple frames of images (i.e., acquired images) through short and long exposures, and these images need to be aligned and composited.

[0151] S304 selects the appropriate image space alignment model for different motion modes.

[0152] In some embodiments, a translational alignment model can be selected for small movements; an affine alignment model can be selected for medium rotations / scalings; and a local optical flow correction model can be selected for complex jitters.

[0153] Image spatial alignment models can be pre-trained models or calculated based on the type of motion between two frames.

[0154] S305 performs spatial alignment on each frame based on the model selection result, and outputs the aligned image frame and alignment confidence.

[0155] Optical flow displacement is used for alignment, allowing for the accurate overlay of multiple image frames. One frame can be selected as a reference frame (usually the first frame or a frame with intermediate brightness), and the remaining frames are aligned to this reference frame based on optical flow displacement information. The output is the aligned image frame and the alignment confidence score (which can be determined using image matching similarity or optical flow correlation matching methods). This solves the misalignment problem between multiple frames, laying a solid foundation for image fusion.

[0156] S306, Image fusion is performed based on alignment confidence to obtain a fused image.

[0157] Image synthesis can be achieved through adaptive fusion, introducing motion weights (i.e., the weight of each region). Regions with large optical flow displacement amplitude / low confidence (less than 0.2) (i.e., dynamic regions), such as moving cats and dogs, can have their weights reduced to avoid motion blur. The confidence threshold can be calibrated through experimental testing or determined based on pixel offset, and can be adjusted in increments of 0.1.

[0158] The offset of the optical flow sensor output is less than a fixed value (e.g., 0.2) in a stable region (static region), such as a sofa, which increases the fusion weight and improves noise reduction and detail.

[0159] The image with the least blurry afterimage can be selected as the main frame (i.e., the target image). The weights of each region on other frames are determined, and the regions with higher weights are used as the regions that can be corrected in the main frame. If the main frame needs to be corrected, it can be corrected through these regions to obtain the fused image.

[0160] S307 optimizes the fused image to obtain the output image.

[0161] Traditional parameter tuning combined with lightweight neural networks can be used to repair edge ghosting and texture blur, restoring sharpness, color, brightness, and noise levels to achieve results close to a single-frame clear photo. Edge ghosting repair and balancing sharpness and noise can also be achieved through small neural networks.

[0162] In some embodiments, when a large motion amplitude is detected at the current moment (which may be an average displacement > 5px, a peak displacement > 7px), the exposure time can be automatically shortened, the number of short frames increased, the probability of ghosting reduced, and the success rate of the final image improved.

[0163] S308 outputs the output image.

[0164] The embodiments of this application significantly reduce ghosting in low-light and moving scenes; no need to introduce a large amount of algorithm processing time, resulting in minimal image lag.

[0165] Please continue to refer to Figure 1 The image processing device 154 may include an acquisition module 1541, a determination module 1542, a spatial alignment module 1543, and a correction module 1544. The acquisition module 1541 is used to acquire multiple frames of images acquired by the image acquisition device and the motion displacement of the image acquisition device. The determination module 1542 is used to determine the motion type of the image acquisition device and the target image in the multiple frames of images based on the motion displacement. The spatial alignment module 1543 is used to perform spatial alignment on the multiple frames of images based on the motion type, obtaining a multi-frame aligned image and the alignment confidence of each aligned image in the multi-frame aligned image. The correction module 1544 is used to correct the target image based on the alignment confidence of each aligned image, obtaining the output image of the image acquisition device.

[0166] It should be noted that the description of the device embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments; therefore, it will not be repeated. For technical details not disclosed in the device embodiments, please refer to the description of the method embodiments in this application for understanding.

[0167] It should be noted that, in the embodiments of this application, if the above-described image processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0168] This application provides a storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the image processing method provided in this application.

[0169] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0170] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0171] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0172] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application. It should be understood that "an embodiment" or "one embodiment" mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence number of the above-described processes does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments of this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments.

[0173] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0174] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, the method comprising: Acquire multiple frames of images captured by the image acquisition device and the motion displacement of the image acquisition device; Based on the motion displacement, the motion type of the image acquisition device and the target image in the multi-frame acquired images are determined; Based on the motion type, spatial alignment is performed on the multi-frame acquired images to obtain multi-frame aligned images and the alignment confidence of each frame aligned image in the multi-frame aligned images; Based on the alignment confidence of each frame of aligned images, the target image is corrected to obtain the output image of the image acquisition device.

2. The image processing method according to claim 1, further comprising: Within each sampling period, the displacement and displacement confidence of the image acquisition device, determined by the motion sensing module in the image acquisition device, are acquired. Based on the displacement confidence level, the displacement of the image acquisition device in multiple sampling periods is filtered to obtain the motion displacement of the image acquisition device when acquiring the multiple frames of images.

3. The image processing method according to claim 2, wherein determining the motion type of the image acquisition device based on the motion displacement includes: Based on the motion displacement of the image acquisition device when acquiring the multi-frame acquired images, the image translation amount between two adjacent acquired images in the multi-frame acquired images is determined. The image translation amount includes at least one of translation jitter amplitude, rotation angle or scaling ratio. Based on at least one of the following: a first comparison result between the translation jitter amplitude and jitter threshold between two adjacent acquired images; a second comparison result between the rotation angle and rotation angle threshold between two adjacent acquired images; and a third comparison result between the scaling ratio and scaling ratio threshold between two adjacent acquired images, the motion change between the two adjacent acquired images is determined; wherein, the motion change includes at least displacement change, deformation change, or a mixture of changes. The motion changes are accumulated to obtain the motion type of the image acquisition device; the motion type includes at least translational motion, deformation motion, or mixed motion.

4. The image processing method according to claim 3, wherein spatial alignment of the multi-frame acquired images based on the motion type to obtain multi-frame aligned images and alignment confidence of each frame aligned image in the multi-frame aligned images includes: Based on the motion changes between two adjacent acquired images, an alignment model corresponding to the two adjacent acquired images is selected. The alignment model includes at least a translation alignment model, an affine alignment model, and a local optical flow correction model, and the alignment model is pre-trained. Based on the acquisition order of multiple frames of acquired images, adjacent frames of acquired images are aligned sequentially based on the corresponding alignment models to obtain the multi-frame aligned images; The alignment confidence of each frame is determined based on the similarity between the aligned image and the reference frame; the reference frame is the first frame or the center frame in the multi-frame aligned image.

5. The image processing method according to claim 3, wherein spatial alignment of the multi-frame acquired images based on the motion type to obtain a multi-frame aligned image and the alignment confidence of each frame aligned image in the multi-frame aligned image includes: Based on the motion type, select the alignment model corresponding to the multi-frame acquired images; The alignment model includes at least a displacement alignment model, an affine alignment model, and a local optical flow correction model, and the alignment model is pre-trained. Based on the acquisition order of the multi-frame acquired images, taking the first frame acquired image as a reference image, and based on the alignment model, starting from the second frame acquired image, each frame acquired image is aligned with the reference image to obtain the multi-frame aligned image; The alignment confidence of each frame-aligned image is determined based on the similarity between each frame-aligned image and the reference image.

6. The image processing method according to any one of claims 1 to 3, wherein determining the target image in the multi-frame acquired images based on the motion displacement comprises: Based on the motion displacement or the image quality of each frame of acquired images, the target image is determined from the multiple frames of acquired images; The step of correcting the target image based on the alignment confidence of each frame of aligned images to obtain the output image of the image acquisition device includes: Based on the alignment confidence of the other aligned images (excluding the target image) in the multi-frame aligned images and the motion displacement between each other aligned image and the target image during alignment, weights are extracted for each region of the other aligned images to obtain a region weight map corresponding to each aligned image; wherein each region includes at least a static region and a dynamic region; the region weight map includes the weight of each region; Based on the weight of each region, the regions in the weight map of each region are filtered to obtain the usable regions; Based on the available area, the target image is corrected to obtain the output image.

7. The image processing method according to claim 6, wherein correcting the target image based on the available area to obtain the output image comprises: Determine the region to be optimized in the target image, wherein the region to be optimized includes at least a ghosting region and a blurred region; The output image is obtained by covering the region to be optimized with the available area corresponding to the location of the region to be optimized.

8. The image processing method according to claim 7, wherein acquiring multiple frames of acquired images acquired by the image acquisition device and the motion displacement of the image acquisition device comprises: In response to the ambient light intensity of the image acquisition device being lower than a preset light threshold and / or the motion displacement being greater than a preset displacement threshold, the image acquisition device acquires multiple frames of images and the motion displacement of the image acquisition device.

9. The image processing method according to any one of claims 1 to 3, further comprising: In response to the motion displacement detected by the motion sensing module being greater than a preset displacement threshold, the exposure time or acquisition frame rate of the image acquisition device is adjusted. The motion sensing module includes at least one of the following: an event vision sensor and an optical flow sensor.

10. An electronic device, comprising: An image acquisition device includes at least an image acquisition unit and a motion sensing module, wherein the image acquisition unit is used to acquire multiple frames of images, and the motion sensing module is used to determine the motion displacement of the image acquisition device. Memory is used to store executable instructions or computer programs. When a processor executes computer-executable instructions or computer programs stored in the memory, it performs the following steps: Acquire multiple frames of images captured by the image acquisition device and the motion displacement of the image acquisition device; Based on the motion displacement, the motion type of the image acquisition device and the target image in the multi-frame acquired images are determined; Based on the motion type, spatial alignment is performed on the multi-frame acquired images to obtain multi-frame aligned images and the alignment confidence of each frame aligned image in the multi-frame aligned images; Based on the alignment confidence of each frame of aligned images, the target image is corrected to obtain the output image of the image acquisition device.

Images